- Extending KBC
Components allow you to extend Keboola Connection (KBC).
The data interface to components is very similar to Transformations — data is exchanged as
CSV files in designated directories.
Intro to Component Creation
As a developer, you implement the application logic in a language of your choice and store it in a
git repository. The component must adhere to our common interface.
To start quickly, use our component generator that can generate a skeleton of the component for you. We also provide libraries to help you with implementation in
Check our example component in PHP.
The main part of the common interface is the specification how
CSV files and designated folders are used to exchange data between KBC and components:
- Applications process input tables stored in CSV files and generate result tables in CSV files.
- Extractors write results in the same way as applications, but instead of reading their
input from KBC tables, they get it from an external source (usually an API).
- Writers, on the other hand, access their input tables in the same way as applications, but push their results into external systems and do not generate any KBC tables.
Apart from this basic usage, the common interface offers many more features:
- Passing parameters
- Error control
- Working with metadata
- OAuth support
- Working with non-CSV files
- Logging configuration
Our Docker Runner component makes sure that the common interface is honoured
from our side. It also takes care of executing your component in its own isolated environment.
Before you start developing a new component, you should
- have a KBC project where you can test your code.
- get yourself acquainted with Docker. You should be
able to run
docker commands. Strictly speaking, you can get away
with not using them, but it will certainly speed things up for you.
- be able to send API requests. Although you can use the Apiary client console, we
recommend using Postman as it is
more convenient. A list of sample requests
- have a git repository (Github or Bitbucket are recommended,
although any other host should work as well).
You can work with your component in your KBC projects immediately as soon as you
create it. However, to make the component publicly available to all users,
it must be published.
The following component types are currently allowed:
- Extractor – a component designed to bring data into Keboola Connection Storage
- Writer – a component designed to bring data to an external system
- Application – another arbitrary component
- Processor – a processor
- Code Pattern – a code pattern
- Transformation – a special type of component for transformations
- Other – a completely special component
The type of a component has no effect on the component internals – i.e., all components regardless of their type
share the identical features of the common interface. The component type just tells
the end-user what behavior to expect from a given component. For example, an extractor is expected to extract data
from an external system. Nothing prevents it from processing data from Storage
(e.g., Geocoding does that),
but its primary reason of existence is to bring data into Storage.
The expected behavior of the above component types can be described in more detail as follows:
- Extractor – extracts data from an external system into Storage, uses no Storage tables on input, has a UI.
- Writer – loads data from Storage into an external system, generates no Storage tables on output, has a UI.
- Application – processes data in Storage, either something like a wrapped transformation or using an external service. Or, it does not work with data in Storage at all. This component has a UI.
- Processor – is designed for post-processing or pre-processing data of other components. It is not designed to run alone and has no UI.
- Code Pattern – generates code for the transformation’s component. This component has a UI.
- Transformation – represents a transformation engine. The UI treats these components specially and expects that they have similar capabilities
and configuration options. These are created by Keboola. If you wish to bring your own, please contact us first.
- Other – this component type has a special role in the UI, it has no standard component UI. Notable “other” components are: