Setting the pagination method to
multiple allows you to use multiple scrollers on a single API.
This type of pagination contains the definition of all scrollers used in the entire configuration.
Each job is then assigned a
scroller in its configuration.
This is useful mainly if the API has inconsistent pagination methods among various API calls. It may be also useful in case you need to vary parameters — for instance, set different page sizes for different endpoints.
The following configuration parameters are supported for the
multiple method of pagination:
scrollers(required, object) — object with configuration of the scrollers (see below)
default(optional, string) — name of a scroller used for all jobs without a specified scroller; if not specified, then the jobs with no assigned scroller will not use any type of pagination.
scrollers configuration is an object whose keys are arbitrary scroller names. The values of the
keys are standard scroller configurations. Any of the supported
paging strategies can be used, and
multiple paging strategies can be mixed. The configurations are the same as if there was a single scroller.
The name of the scroller must be used in a specific job
default scroller can be set (must be one of the names defined in
scrollers). In that case, all jobs
without an assigned scroller will use the default one.
There are no specific stopping conditions for the
multiple pagination. Each scroller acts upon its
normal stopping conditions.
Assume you have an API with several endpoints (
/search, etc.). Most endpoints
use the offset pagination strategy (meaning that the results are split into pages of the same size, and
the next page is obtained by setting offset to a multiple of the page size). The
/search endpoint uses the
page number pagination strategy because the retrieved pages are not of equal size. The following
configuration extracts from two endpoints with different paging strategies.
api.pagination.scrollers defines both pagination methods:
It is then important to actually use the scroller in the
job.scroller configuration for the endpoint
/users has no assigned scroller, therefore it uses the default one, which is
See example [EX062].