As described in the architecture overview, Keboola Connection (KBC) consists of many different components. Only those components that are registered in our Component List are generally available in KBC. The list is provided by our Storage Component API in the dedicated Components section. The list of Components is managed using the Keboola Developer portal.
That being said, any KBC user can use any registered component, unless
To register and application, you need to have an account in Keboola Developer Portal. The Developer Portal is a completely separate application which manages the list of components available in KBC. As it is a separate application, it uses different credentials than KBC and you have to obtain an account. Creating an account is free and quick, it requires a working email (to which a confirmation email will be sent) and a mobile phone for a mandatory two-factor authorization. When you log in to the developer portal, you have to join a vendor. A vendor is an organization of developers. Every application in KBC has to belong to a vendor. If you don’t work for a company, we suggest you create a vendor with your name. Even if you want to create a single component, it still has to belong to a vendor.
If you join an existing vendor, an administrator of that vendor has to approve your request. If you want to create a new vendor, a Keboola Administrator has to approve your request. When you create a new vendor you will receive a development project in KBC and you should provide us with a channel for receiving internal errors from your applications. Basically anything supported by Papertrail notifications is available, though e-mail or Slack channel is most commonly used.
When you are confirmed as a member of a vendor, you may proceed to creating your own applications.
To add an application, use the Create App button and fill in the application name and ID:
Do not use the words ‘extractor’, ‘writer’ or ‘application’ in the application name.
When creating an application, you will obtain the Component ID (in the form
vendor.app-id) – e.g.
Once you have the Component ID, you can create configurations of the application in KBC. You can also review the
application in KBC by visiting an URL:
Note that the configuration will not be runnable until you configure the Repository section of the application.
Important: changes made in the Developer Portal take up to 5 minutes to propagate to all Keboola Connection instances in all regions.
Application Repository is crucial part of the application registration, because it actually defines what Docker image will be used when running the application. We offer free hosting of your docker images in Amazon Container Registry (AWS ECR) under our own account. All repositories in AWS ECR are private. When registering your component, you will receive credentials for deployment to the repository and you can either push the images manually or use an automated script to push images.
We also support the DockerHub and Quay.io registries, both public and private. However, we recommend that you use AWS ECR unless you require DockerHub or Quay for some reason (e.g. you require that the image is public). The main benefit of our AWS ECR is its reliability, as Quay.io and DockerHub are more prone to outages and are beyond our control.
For registering a component based on the Generic Extractor, use the following repository:
For a list of available tags, see the Generic Extractor Github repository
or Generic Extractor Quay Repository, both
of which contain the same tags as the above AWS ECR repository.
It is also possible to use the
latest tag, which points to the highest available tag. However we recommend that you
register your component with a specific tag and update manually to avoid problems with breaking changes in future Generic
Extractor releases. For more details on registering components based on Generic Extractor, see
the dedicated page.
When registering Custom science applications, one of our images should be used. The registration of Custom Science applications is not supported yet in the Developer Portal, so please contact us on support. If you are registering a Custom Science extension and want to use a private git repository, provide us with encrypted credentials to the git repository.
Each extension needs to specify how its user interface (UI) will look. Without any configuration, the component cannot
be configured via UI (it can still be configured using the API though). The most basic configuration
genericDockerUI. The generic UI will always show a text field for entering the
component configuration in JSON format. Other components of the UI are turned on using other flags
genericDockerUI-tableOutput). All of the flags may combined freely.
This provides a basic textarea for setting extension parameters as a JSON; the textarea has JSON validation and syntax highlighting.
Defining a configuration schema will replace the JSON textarea with a form.
This flag provides a UI for setting the table input mapping. With this UI, you can set:
With this UI, you can set:
This flag provides a UI for the Processor configuration. It provides a basic textarea for setting the processors and their parameters as a JSON; the textarea has JSON validation and syntax highlighting.
This flag provides a UI for setting the file input mapping. With this UI, you can set:
This flag provides a UI for setting the file output mapping. With this UI, you can set:
When you register an extension (be it either Docker Extension or Custom Science or Generic Extractor), it is not published. A non-published component can be used without limitations but it is not offered in the KBC UI. This means that it can only be used by directly visiting a link with the specific component ID or via the API. Unpublished components are also not part of the Public Component list. An existing configuration of a non-public component is accessible the same way as a configuration of any other component.
To publish an application, you have to request application approval from Keboola. This is done in Keboola Developer portal by requesting approval from the application list. A member of our staff will review your application and publish the application or contact you with the required changes.
The goal of the application review is to maintain reasonable end-user experience and application reliability. Before applying for application registration, make sure that the same application does not exist yet. If a similar application exists (e.g. an extractor for the same service), make sure that application description clearly states the differences. During application review, the following best practices are checked:
writer. (Good: Cloudera Impala, Bad: Cloudera Extractor)
propertyOrderto explicitly define the order of the fields in the form.
titlewithout a colon.
descriptionto provide an explanatory sentence if needed.
some introduction textBad:
### Input Description
description of input tables
#### First Table
some other text
## Configuration Description
some introduction text
#### Input Description
description of input tables
master) in production is not allowed.