iterations section allows you to execute a configuration multiple times, each time with different
values. The most typical use for
iterations is extraction of the same data from multiple accounts.
Iterations can always be replaced by creating multiple complete configurations of Generic Extractor.
Iterations are specified as an array of objects, where each object contains the same properties
config section. All properties of the object are optional.
Consider the following example of an
iterations configuration defining that the entire Generic Configuration
will be executed twice: the first time with the username
JohnDoe, and the second time with the username
iterations properties override the
config properties, they are accessible
as configuration attributes
Keep in mind that
iterations can refer directly only to the things specified in the
api section, you must use functions.
Also, it is not possible to iterate over values returned in the response.
The number of iterations and their values must be defined in the configuration.
Because the values defined in
iterations override those in the
everything that can be in the
config section is allowed as well
(including arbitrary user attributes used in functions).
mappings in iterations does not make much sense though.
Also, if you use
userData in iterations, they must result in the same columns; otherwise the resulting
table cannot be imported into Storage. If you use
incrementalOutput, only the last value of
Suppose, you have an API which takes a URL parameter
account_id, which restricts the returned data to a
certain account. The following configuration executes the entire configuration for two accounts —
iterations section overrides the values in the
config section, the below configuration
yields the exact same results as the configuration above:
It looks as if the first execution is with
account_id=123, but it is not the case. The configuration
will be executed only twice: the first time with
account_id=345 and the second time with
See example [EX112].
Suppose you have an API from which you want to extract data from two accounts (
API uses the HTTP Basic Authentication method, and in addition,
each user has their own API token, which must be provided in the
Even if the above parameters relate to the
api configuration, which cannot
be directly included in
iterations, we can specify them as
http.headers and therefore it is still possible to use them.
#password from the
config section, the above configuration overrides also
http.headers.X-Api-Token setting. The configuration can be simplified by using
functions and references:
config section specifies the part of the token authentication which is common to both iterations.
Each iteration then specifies only the token. By writing
"attr": "apiToken" under
X-Api-Token we say that
X-Api-Token will have the value of the
apiToken property from the
config section. That property is in
turn specified in the iteration objects.
See example [EX113].