It is possible to register a Generic Extractor configuration and create a completely new extractor based on it. This enables sharing the API extractor between various projects and simplifies its further configuration.
Even though the registration process is common to all components, there are some specifics for Generic Extractor.
Before converting your configuration to a universally available component, consider
what values in the configuration should be provided by the end-user (typically authentication values).
Then design a configuration schema for setting
those values. You can test the schema online.
The values obtained from the end-user will be stored in the
Modify your configuration to read those values from there.
The following settings should be used when registering a new component based on Generic Extractor:
Because the UI is assumed to be
genericTemplatesUI, provide a
configuration schema and
a template to be used in conjunction with the schema.
Optionally, the templates UI may contain interface to negotiate OAuth authentication.
An example of the templates UI is shown in the picture below.
Config section of the templates UI is defined by the configuration schema you provide.
Template section contains at least one template. A template is simply a configuration of
For example, you might want to provide one configuration for incremental loading
and a different configuration for full loading. The template UI also has the option to
Switch to JSON editor, which displays the configuration JSON and allows the end-user to modify it.
Notice that the JSON editor allows modification only to the
section. Other sections, such as
authorization.oauth_api may not be modified by the end-user.
You can review existing templates in their Github repository. If you feel confident, you can send a pull request with your templates, otherwise contact us on support to activate templates for your extractor.
Let’s say you have the following working API configuration (see example [EX111]):
and you identify that four values of that configuration need to be specified by the end-user:
For each of them, create a parameter of the appropriate type:
JohnDoe— a string parameter
TopSecret— a string parameter
#password(it will be encrypted)
123— a numeric parameter
active— a enumeration parameter
The parameters names are completely arbitrary. However, they must not conflict with existing
configuration properties of Generic Extractor (e.g.,
Now create a configuration schema for the four parameters.
When you test the schema online, it will produce a configuration JSON:
The argument to the
base64_encode function is now the
concat function which joins together the
values of the
#password fields. The
accountId parameter needs to be moved to the
jobs section because the
http.defaultOptions.params section does not support function calls (yet!).
type parameter was changed to a reference to the
(see example [EX111]).
When you handled the configuration parameters, turn the configuration into a template. Place
api section to a separate, individual
Once you make sure that the extractor works as it did before,
remove the user provided values (
config section, put it in a
data section and add
description to it.
Save the file into a separate
template.json file. The template file therefore contains
Create as many
template.json files as you wish. However, all of them need to share the same
configuration. When you want to register your component, attach the
api.json and all