Pagination breaks a result with a large number of items into separate pages and is used very commonly in many API calls.
In the previous part of the tutorial, you fetched campaigns from the MailChimp API. If you created a new account, chances are that you probably have only one campaign. You should now create some more campaigns (you do not have to configure them anyhow).
If the API has consistent pagination for all resources (which the
MailChimp API has),
then the pagination is defined in the
api section of the configuration.
The MailChimp API uses the
offset pagination method;
it means that each page has a fixed
limit (by default 10 items), and you need to use the offset to move
that fixed-size page over the next set of results. For the first page, the
offset is 0, for the second
offset is 10. This is the same kind of pagination as in SQL.
The offset pagination method is configured with the following basic properties:
method — for MailChimp, set this property to
offsetParam — name of the API parameter which defines the page offset
limitParam – name of the API parameters which define the page size (limit)
So for MailChimp, configure the pagination this way:
Now make sure that you have more than one campaign in your account. The entire Generic Extractor
configuration will look like this:
Because you probably have less than ten (the default page size) campaigns in your MailChimp account,
there is no way to tell whether the pagination works or not. Let’s make sure then, by setting the
to 1 and turning the
debug mode on, that you can see all the requests sent by Generic Extractor.
Run the configuration and review the events produced by the job. You should see something like this:
The oldest events are at the bottom, so you can see that the extractor started by sending an HTTP request:
Then it continued with
and so on. You should also see a warning that the
dataField 'campaigns' contains no data.
This is expected because Generic Extractor tries bigger offsets until the number of returned items is
less than the page size. With the page size set to 1, this means that the last page will contain no data.
In this part of the tutorial, you learned how to set up simple pagination. This is very important because most APIs use some sort of pagination and without proper setting you would be getting incomplete data. The next two parts of our tutorial deal with setting up jobs and mapping: