The Response URL Scroller can be used with APIs that provide the URL of the next page in the response. This scroller is suitable for APIs supporting the JSON API specification.
The following configuration parameters are supported for the
response.url pagination method:
urlKey(optional, string) — path in the response to the field which contains the URL of the next request; the default value is
delimiter(optional, string) — char used as the delimiter of the nested keys in the
urlKey; the default value is
true— job parameters are added to the parameters of the URL provided in the response; the default value is
See the examples below.
The pagination ends when the value of the
urlKey parameter is empty — the key is not present at all, is null,
is an empty string or is
false. Be careful when configuring the
urlKey parameter. If you, for example, misspell the
key name, the extraction will not go beyond the first page.
Common stopping conditions also apply.
This section provides three API pagination examples where the Response URL Scroller is used.
To configure pagination for an API that supports the JSON API specification, use the configuration below:
The configuration expects a response to contain a
links.next field with the URL of the next page, e.g.:
The URL may be either an absolute link (
http://example.com/users?page=2) or an absolute path (
If the URL is relative (
users?page=2), it is appended to the endpoint URL.
See example [EX054].
To pass additional parameters to each of the page URLs, use the
A sample response:
In the above configuration, the
account parameter is sent with every API request. If it were not for the
includeParams option, it would be sent only with the first request. Note that adding
jobs.params.page parameter would overwrite the
page parameter in the response URL and thus
would probably break the paging.
See example [EX055].
Sometimes the API does not pass the entire URL, but only the query string parameters which should be used for querying the next page.
Then use the
paramsIsQuery configuration so that your Generic Extractor can produce a
Also notice that with the above
page parameter specified in the job is used only for the first page because it
is overridden by the
page parameter given in the response. That is to say that the first request is sent to
/users?account=123&page=start and the second request is sent to
See example [EX056].