In this tutorial, you will create a simple “Hello, World!” component which runs in Keboola Connection (KBC).
You need to have a computer with working Docker to develop the KBC component code. To be able to create new components, you also need to have an account in the Keboola Developer Portal, which manages the list of components available in KBC.
The Developer Portal uses different credentials than KBC. Creating an account is free; it requires a working email address (to which a confirmation email will be sent) and a mobile phone for a mandatory two-factor authorization.
When you log in to the Developer Portal, you have to join a vendor — an organization of developers. Every KBC component has to have a vendor assigned. If you join an existing vendor, a vendor administrator has to approve your request. If you do not work for a company, create a vendor with your name (even a single developer has to be assigned to a vendor). When you join or create a vendor you should also receive access to a development KBC project.
In order to create a new vendor, a Keboola administrator has to approve your request, and you will receive a development project in KBC. In addition to that, you need to provide us with a channel for receiving internal errors from your components. Anything supported by Papertrail notifications is available, though e-mail or a Slack channel is most commonly used.
When you are confirmed as a member of a vendor, you may proceed to creating your own component. The example component is written in the Python language, but no knowledge of Python is required. Before you continue with this tutorial, make sure you
Note: Even though the tutorial assumes using GitHub + Travis services, they are not required for extending KBC. We use them because we like them the most. The deployment documentation shows how to configure, for example, Bitbucket and GitLab integrations.
To add a component, use the Add a component button on the main page, and fill in the component name and type:
Important: Do not use the words ‘extractor’, ‘writer’, or ‘application’ in the component name.
Choose the appropriate component type:
extractor– brings data into KBC
writer– sends data out of KBC
application– does some transformation of the data, or something completely different
The above does not mean technically that, for example, an extractor cannot send data out of KBC or an application cannot bring new data into KBC. It is a matter of user perception, so use your judgement to select the correct type.
When you fill the form in, you will obtain a component ID (in the
vendor-id.component-name, for instance,
keboola-test.ex-docs-tutorial). Make a note of the ID.
To be able to deploy the component to KBC, you will need service credentials. For security reasons, we strongly advice against using your own credentials in any deployment service. To create new deployment credentials, click the Create a service account button on the Service accounts page.
Fill in a name (e.g.,
ex_docs_tutorial_travis) and description (e.g.,
Travis deployment credentials) and confirm:
Take a note of the username and password.
Once you have the component ID and the service account username and password, you can use our component generator tool to create a component skeleton for you in your favorite programming language.
Create an empty Github repository. The name of the repository is arbitrary, but using the component is probably a good idea to avoid confusion.
Checkout the repository on your local computer and execute the following from the command line:
docker run -i -t --volume=/path/to/repository/:/code/ quay.io/keboola/component-generator
/path/to/repository/ with an absolute local path to your empty repository. Follow
the on-screen instructions:
When done, you will have an initialized repository with a “Hello, World!” component.
In the above example, we chose the
simple-python template, which contains the following:
You will also obtain a path to your Travis CI configuration (in the above example https://travis-ci.org/keboola/ex-docs-tutorial).
When done exploring, push to the repository.
This will automatically trigger a build on the Travis service; you can view the build
progress by visiting the provided link. In fact, two builds will be triggered: one
master branch, and one for the
We are more interested in the latter because that is going to trigger the deployment to KBC.
If the deployment passes without errors, the component will become available in KBC. You can verify that in the component details in the Developer Portal:
This means that the component deployment is fully automated. If you change the component source code, all you need to do is push the changes to the git repository and tag them with the normal version tag.
Once the component is deployed, it becomes available in KBC. Note that it takes up to 5 minutes for the changes to propagate to all KBC instances. After that, you can configure the component by visiting the following URL:
You can then run the configuration without any settings.
And you should see the “Hello, World” message in the events:
When you create a component, it will have assigned a memory limit of 256MB and run timeout of 1 hour. If you need to change those limits, please contact our support.
The component repository is a crucial part of the component setting because it actually defines what Docker image will be used when running the component. We offer free hosting of your Docker images in the Amazon Container Registry (AWS ECR) under our own account. All repositories in AWS ECR are private. When you create your component using the method shown above, we have just provisioned you with the Docker image hosting and you do not need to worry about it any more.
We also support the DockerHub and Quay.io registries, both public and private. However, as they are more prone to outages and beyond our control, we recommend using our reliable AWS ECR. Use DockerHub or Quay.io only if you, for instance, want the image to be public.
You have just created your own KBC component. Although it does not do much, it shows the easiest path to bringing your own application logic to KBC. You can now continue with other parts of the tutorial:
Although you rarely need all of the above parts (e.g., you do not need input mapping when building an extractor), we suggest you go through all of them to gain a general overview of the available options. You can also read all the details in the respective parts of the documentation: