Component Registration

Introduction

As described in the architecture overview, Keboola Connection (KBC) consists of many different components. Only those components that are registered in our Component List are generally available in KBC. The list is provided by our Storage Component API in the dedicated Components section.

While a Custom Science extension requires registration only when offered to all KBC users, registering a Docker extension is mandatory at all times.

That being said, any KBC user can use any registered component, unless

  • the KBC user (or their token) has a limited access to the component,
  • the component itself limits where it can run (in what projects and for which users).

Using a Private Repository

If your image cannot be public, you can use a private Docker repository on both DockerHub, Quay.io and AWS ECR. If you are registering a Custom Science extension and want to use a private git repository, provide us with encrypted credentials to the git repository.

DockerHub

An account has to be created in order to access your private DockerHub repositories. Please provide us with your username, password and server properties from the login credentials. Better yet — create a new user that has access to the desired repository and surrender it to us.

DockerHub offers 1 free private repository per account.

Quay.io

Quay.io offers convenient robot accounts. Provide us with a robot username and token with read-only access to the desired repository and we are good to go.

Quay.io screenshot

Private Quay.io repositories are paid.

AWS ECR

We offer free hosting of your images in AWS ECR under our account. Repositories in AWS ECR are private and our application has read-only access. When registering your component, you will receive push credentials to the repository and you can either push the images manually or integrate the push in your CI pipeline.

The main benefit of AWS ECR is its reliability, as Quay.io and Dockerhub are more prone to outages.

Registration

The registration process is simple, but it must be done by Keboola. To register your extension, please fill in the checklist and contact us. When your component is registered, you will receive its Component ID. If you need to make any changes to your application (e.g., changes in the JSON schema), contact us with a modified checklist.

UI Options

Each extension will receive a Generic UI. The generic UI will always show a text field for entering the component configuration in JSON format. Additionally, you can request other parts of the generic UI by adding any of the tableInput, tableOutput, fileInput, fileOutput, genericDockerUI flags in the checklist.

Each of the options is shown below:

genericDockerUI

This provides a basic textarea for setting extension parameters as a JSON; the textarea has JSON validation and syntax highlighting.

Generic configuration screenshot

Defining a configuration schema will replace the JSON textarea with a form.

tableInput

This flag provides a UI for setting the table input mapping. With this UI, you can set:

  • Source — the name of the table in Storage
  • Destination file name — the name of the .csv file passed to the application
  • Columns — select only some columns of the source table
  • Days — load only rows modified in the specified number of days; useful for incremental loads; set to 0 to load all data
  • Data filter — a simple filter for selecting specified rows only

Table input screenshot

Table input detail screenshot

tableOutput

This flag provides a UI for setting the table output mapping. With this UI, you can set:

  • Source — the name of the .csv file retrieved from the application
  • Destination — the name of the table in Storage, the destination bucket should exist already
  • Incremental — if checked, the loaded data will be appended to the contents of the destination table
  • Primary key — set the primary key for your destination table — multiple columns are allowed
  • Delete rows — delete some rows from the destination table using a simple filter

Table output screenshot

Table output detail screenshot

fileInput

This flag provides a UI for setting the file input mapping. With this UI, you can set:

File input screenshot

File input detail screenshot

fileOutput

This flag provides a UI for setting the file output mapping. With this UI, you can set:

  • Source — the name of the file produced by the application
  • File tags — the file tags assigned to the produced file
  • Is public — the file is accessible to anyone knowing its URL
  • Is permanent — the file will not be deleted after 180 days

File output screenshot

File output detail screenshot

Hide your Extension from KBC App Store

If you want to test your extension before making it available to all users, it can be hidden from the list of all components (UI flag excludeFromNewList). However, this does not really prevent anyone from using it. To use a non-published extension, create the configuration with the component API. Or visit directly the following URL: https://connection.keboola.com/admin/projects/{projectId}/applications/{componentId}. The projectid is your KBC project ID. The componentId (e.g. myCompany.acmeApplication) is provided upon your extension registration.