It is possible to publish a Generic Extractor configuration as a completely separate component. This enables sharing the API extractor between various projects and simplifies its further configuration.
Before converting your configuration to a universally available component, consider
what values in the configuration should be provided by the end-user (typically authentication values).
Then design a configuration schema for setting
those values. You can test the schema online (alternative).
The values obtained from the end user will be stored in the
Modify your configuration to read those values from there.
Do not forget that if you prefix a value with a hash
#, it will be
encrypted once the configuration is saved.
Also, try to make the extractor work incrementally
To publish your Generic Extractor configuration, you need to create a new component in
the Developer Portal. Choose an appropriate name and the type
extractor. Once you
have created the component, edit it, and fill in the following details:
For a list of available tags, see the Generic Extractor GitHub repository or
Generic Extractor Quay repository, both of which contain the same tags
as the above AWS ECR repository. It is also possible to use the
latest tag, which points to the highest available tag. However,
we recommend that you configure your component with a specific tag and update it manually to avoid problems with breaking changes
in future Generic Extractor releases.
Because the UI is assumed to be
genericTemplatesUI, provide a
configuration schema and
a template to be used in conjunction with the schema. Optionally, the template UI may also contain an interface to
negotiate OAuth authentication.
An example of the template UI is shown in the picture below.
Config section of the templates UI is defined by the configuration schema you provide.
Template section contains at least one template. A template is simply a configuration of
For example, you might want to provide one configuration for incremental loading
and a different configuration for full loading. The template UI also has the option to
Switch to JSON editor, which displays the configuration JSON and allows the end user to modify it.
Notice that the JSON editor allows modification only to the
section. Other sections, such as
authorization.oauth_api, may not be modified by the end user.
You can review existing templates in their GitHub repository. If you feel confident, you can send a pull request with your templates, otherwise submit it when requesting the publication of your component.
Let’s say you have the following working API configuration (see example [EX111]):
and you identify that four values of that configuration need to be specified by the end user:
For each of the values, create a parameter of the appropriate type:
JohnDoe— a string parameter
TopSecret— a string parameter
#password(it will be encrypted)
123— a numeric parameter
active— an enumeration parameter
The parameter names are completely arbitrary. However, they must not conflict with existing
configuration properties of Generic Extractor (e.g.,
Now create a configuration schema for the four parameters.
When you test the schema online (alternative), it will produce a configuration JSON:
The above properties will be merged into the
config section. Now
modify the configuration so that it reads them from there using functions and references.
The argument to the
base64_encode function is now the
concat function, which joins together the
values of the
#password fields. The
accountId parameter needs to be moved to the
jobs section because the
http.defaultOptions.params section does not support function calls (yet!).
type parameter was changed to a reference to the
(see example [EX111]).
When you handled the configuration parameters, turn the configuration into a template. Place
api section to a separate, individual
Once you make sure that the extractor works as it did before,
remove the user provided values (
config section, put it in a
data section and add
description to it.
Save the file into a separate
template.json file. The template file therefore contains
Create as many
template.json files as you wish. However, all of them need to share the same
configuration. When you want to publish your component, attach the
api.json and all