Common Interface

To exchange data between your component and Keboola, use

a predefined set of input and output folders for tables and files,
a configuration file,
environment variables and return values.

Optionally, you can use

logging,
manifest files for working with table and file meta-data,
the OAuth part of the configuration file, and
actions for quick synchronous tasks.

In addition to that, Docker Runner provides tools for encryption and OAuth2 authorization.

To quickly get the picture, have a look a random sample data folder.

Component Limits

Even though you can define your own limits for your component, all components are also subject to the following service limits:

Both memory and swap sizes are set to an equal value
Docker overlay2 size is set to 10 GB

The size allocated for overlay2 is consumed by memory swapping, and all other operations in the component (for instance, ad hoc module installations); only input and output folders (/data/) and /tmp/ are excluded. As the swap size cannot be larger than the allocated disk space, we cannot safely increase the memory limit over 8 GB.

If you need more than 8 GB of memory/swap or larger disk space, get in touch with us to discuss possible solutions.