As described in the architecture overview, Keboola Connection (KBC) consists of many different components. Only those components that are registered in our Component List are generally available in KBC. The list is provided by our Storage Component API in the dedicated Components section. The list of Components is managed using the Keboola Developer portal.
That being said, any KBC user can use any registered component, unless
To register your application, you need to have an account in the Keboola Developer Portal, which manages the list of components available in KBC.
The portal uses different credentials than KBC. Creating an account is free and quick; it requires a working email address (to which a confirmation email will be sent) and a mobile phone for a mandatory two-factor authorization.
When you log in to the developer portal, you have to join a vendor — an organization of developers. Every KBC application has to have a vendor assigned. If you join an existing vendor, an administrator of that vendor has to approve your request. If you do not work for a company, create a vendor with your name (even a single component has to be assigned to a vendor).
In order to create a new vendor, a Keboola Administrator has to approve your request, and you will receive a development project in KBC. Also, you need to provide us with a channel for receiving internal errors from your applications. Basically anything supported by Papertrail notifications is available, though e-mail or a Slack channel is most commonly used.
When you are confirmed as a member of a vendor, you may proceed to creating your own applications.
To add an application, use the Create App button, and fill in the application name and ID:
Important: Do not use the words ‘extractor’, ‘writer’ or ‘application’ in the application name.
When creating an application, you will get a Component ID (in the form
vendor.app-id, for instance,
Once you have the Component ID, you can create configurations of the application in KBC. You can also review the
application in KBC by visiting the following URL:
Note that the configuration will not be runnable until you configure the Repository section of the application. When you register an application, it will have assigned a memory limit of 64MB and run timeout of 1 hour. If you need to change those limits, please contact our support.
Important: Changes made in the Developer Portal take up to 5 minutes to propagate to all Keboola Connection instances in all regions.
The Application Repository is a crucial part of the application registration, because it actually defines what Docker image will be used when running the application. We offer free hosting of your docker images in the Amazon Container Registry (AWS ECR) under our own account. All repositories in AWS ECR are private. When registering your component, you will receive credentials for deployment to the repository, and you can either push the images manually or use an automated script to do it.
We also support the DockerHub and Quay.io registries, both public and private. However, we recommend that you use AWS ECR unless you require DockerHub or Quay for some reason (e.g., you want the image to be public). The main benefit of our AWS ECR is its reliability, as Quay.io and DockerHub are more prone to outages and are beyond our control.
For registering a component based on the Generic Extractor, use the following repository:
For a list of available tags, see the Generic Extractor Github repository
or Generic Extractor Quay Repository, both
of which contain the same tags as the above AWS ECR repository.
It is also possible to use the
latest tag, which points to the highest available tag. However, we recommend that you
register your component with a specific tag and update it manually to avoid problems with breaking changes in future Generic
Extractor releases. For more details on registering components based on Generic Extractor, see
the dedicated page.
When registering Custom Science applications, one of our images should be used. The registration of Custom Science applications is not yet supported in the Developer Portal, so please contact our support. If you are registering a Custom Science extension and want to use a private git repository, provide us with encrypted credentials to the git repository.
Each extension needs to specify how its user interface (UI) will look. Without any configuration, the component cannot
be configured via the UI (it can still be configured using the API though). The most basic configuration
genericDockerUI. The generic UI will always show a text field for entering the
component configuration in JSON format. Other parts of the UI are turned on using other flags
genericDockerUI-tableOutput). All of the flags may be combined freely.
This provides a basic text area for setting extension parameters as a JSON; the text area has JSON validation and syntax highlighting.
Defining a configuration schema will replace the JSON text area with a form.
This flag provides a UI for setting the table input mapping. With this UI, you can set:
With this UI, you can set:
This flag provides a UI for the Processor configuration. It provides a basic text area for setting the processors and their parameters as a JSON; the text area has JSON validation and syntax highlighting.
This flag provides a UI for setting the file input mapping. With this UI, you can set:
This flag provides a UI for setting the file output mapping. With this UI, you can set:
When you register an extension (a Docker extension, Custom Science extension, and Generic Extractor), it is not published. A non-published component can be used without limitations, but it is not offered in the KBC UI. It can only be used by directly visiting a link with the specific component ID or via the API. Unpublished components are also not part of the Public Component list. An existing configuration of a non-public component is accessible the same way as a configuration of any other component.
Before your application can be published, it must be approved by Keboola. Request the approval from the application list in the Keboola Developer portal. A member of our staff will review your application and either publish it or contact you with the required changes.
The goal of the application review is to maintain reasonable end-user experience and application reliability. Before applying for application registration, make sure that the same application does not exist already. If there is a similar one (e.g., an extractor for the same service), clearly state the differences in the new application’s description. During our application review, the best practices in the next sections are followed.
propertyOrderto explicitly define the order of the fields in the form.
titlewithout a colon, period, etc.
descriptionto provide an explanatory sentence if needed.
some introduction textWRONG:
### Input Description
description of input tables
#### First Table
some other text
## Configuration Description
some introduction text
#### Input Description
description of input tables
master) in production is not allowed.