Component Quick Start

In this tutorial, you will create a simple “Hello, World!” component which runs in Keboola Connection (KBC).

Before you start

You need to have a computer with working Docker to develop the KBC component code. To be able to create new components, you also need to have an account in the Keboola Developer Portal, which manages the list of components available in KBC.

The Developer portal uses different credentials than KBC. Creating an account is free; it requires a working email address (to which a confirmation email will be sent) and a mobile phone for a mandatory two-factor authorization.

When you log in to the developer portal, you have to join a vendor — an organization of developers. Every KBC application has to have a vendor assigned. If you join an existing vendor, an administrator of that vendor has to approve your request. If you do not work for a company, create a vendor with your name (even a single developer has to be assigned to a vendor).

Screenshot -- Join a vendor

In order to create a new vendor, a Keboola Administrator has to approve your request, and you will receive a development project in KBC. Also, you need to provide us with a channel for receiving internal errors from your components. Basically anything supported by Papertrail notifications is available, though e-mail or a Slack channel is most commonly used.

When you are confirmed as a member of a vendor, you may proceed to creating your own component. The example component is written in Python language, but no knowledge of Python is required. Before you continue with this tutorial, make sure you:

Creating a Component

To add a component, use the Create Component button on the main page, and fill in the component name and type:

Screenshot -- Create application

Important: Do not use the words ‘extractor’, ‘writer’ or ‘application’ in the component name.

To choose the appropriate type:

  • extractor – brings data into KBC
  • writer – sends data out of KBC
  • application – does some transformation of the data, or does something completely different.

The above does not mean technically that for example an extractor can’t send data out of KBC, or that an application cannot bring new data into KBC. It is a matter of user perception, so use your judgement to select the correct type.

When you fill the form, you will obtain a Component ID (in the form vendor-id.component-name, for instance, keboola-test.ex-docs-tutorial). Take note of the Component ID.

Creating a deployment account

To be able to deploy the application to KBC, you will need Service credentials. For security reasons we strongly advice against using your own credentials in any deployment service. To create new deployment credentials, click the Create a service account button on the Service accounts page.

Screenshot -- Create account

Fill in name (e.g. ex_docs_tutorial_travis) and description (e.g. Travis deployment credentials) and confirm:

Screenshot -- Account details

Take note of the Username and Password.

Screenshot -- Account credentials

Initializing the Component

Once you have the Component Id and service account Username and Password, you can user our Component generator tool to create a component skeleton for you in your favorite programming language.

Create an empty Github repository. The name of the repository is arbitrary, but using the component is probably a good idea to avoid confusion.

Screenshot -- Github Repository

Checkout the repository on your local computer and execute the following from command line:

docker run -i -t --volume=/path/to/repository/:/code/ quay.io/keboola/component-generator

Replace /path/to/repository/ with an absolute local path to your empty repository. Follow the on-screen instructions:

Screenshot -- Component Generator

When done, you will have an initialized repository with a “Hello, World!” application. In the above example, I choose the simple-python template which contains:

  • template.md – description of the template files,
  • main.py – a “Hello, World!” Python script,
  • Dockerfile – a Dockerfile defining the environment in which the script runs,
  • deploy.sh – a Bash script to deploy the component to KBC,
  • .travis.yml – a configuration file for Travis CI to automate the deployment.

You will also obtain a path to your Travis CI configuration (in the above example https://travis-ci.org/keboola/ex-docs-tutorial).

Building the Component

When done exploring, push to the repository. This will automatically trigger a build on the Travis service, you can view the build progress by visiting the provided link. In fact, two builds will be triggered, one for the master branch, and one for the 0.0.1 tag.

Screenshot -- Travis Build

We are more interested in the latter, because that is going to trigger the deployment to KBC.

Screenshot -- Travis Build Detail

If the deployment passed without errors, the component will become available in KBC. You can verify that in the component details in the Developer Portal:

Screenshot -- Component Deployed

This means that the application deployment is fully automated. If you change the component source code, all you need to do is push the changes to the git repository and tag them with normal version tag.

Running the Component

Once the application is deployed, it becomes available in KBC. Note that it takes up to 5 minutes to propagate the changes to all KBC instances. Once propagated, you can configure the application by visiting the following URL:

https://connection.keboola.com/admin/projects/{PROJECT_ID}/extractors/{COMPONENT_ID}

You can then run the configuration without any settings.

Screenshot -- Component Configuration

And you should see the “Hello, World” message in the events:

Screenshot -- Component Events

Summary

You have just created your own KBC component. Although it does not do much, it show the easiest path to bring your own application logic to KBC. You can now continue with other parts of the tutorial:

Although you usually don’t need everything (e.g. you don’t need input mapping when building an extractor), we suggest you go through all of the above to gain a general overview of the available options. You can also read all the details in the respective parts of the documentation: