Creating Sandbox

In order to run and debug KBC dockerized applications (including Custom Science, R and Python Transformations) on your own computer, you need to manually supply the application with a data folder and configuration file.

To create a sample data folder, use a Docker Runner API. There are three calls available:

The API calls will resolve and validate the input mapping and create a configuration file. Then they will archive the whole /data/ folder and upload it to your KBC project. None of these API calls will write any tables or files other than the archive, so they are very safe to run.

The Sandbox API call is useful for obtaining a sample environment configuration when starting with development of a new Docker Extension or Custom Science extension.

The Input API call is useful for obtaining an environment configuration for a registered Docker extension (without encryption) or Transformations.

The Dry run API call is the last step. It will do everything except the output mapping and is therefore useful for debugging an existing application in production without modifying files and tables in a KBC project.

The body structure of the first two API calls is the same. Before you start, you need a KBC project. We recommend that you use Apiary or Postman to call the API. A collection of examples of the Sandbox API calls is available in Postman Docs.

Create Sandbox API Call

Prepare

Create a table in KBC Storage which contains a column named number. You can use the sample table. In the following example, the table is stored in the in.c-main bucket and the table name is sample. The table ID is therefore in.c-main.sample.

Storage Screenshot

You also need a Storage API token.

Send the API Request

In the collection of sample requests, there is an Introduction example with the the following JSON in its body:

{
    "configData": {
        "storage": {
            "input": {
                "tables": [
                    {
                        "source": "in.c-main.sample",
                        "destination": "source.csv"
                    }
                ]
            }
        },
        "parameters": {
            "multiplier": 4
        }
    }
}

The node configData.storage.input.tables.source refers to the existing table ID (the table created in the previous step) in Storage. The configData.storage.input.tables.destination node refers to the destination to which the table will be downloaded for the application; it will therefore be the source for the application. For registered components with a UI, the entire configData.storage node is generated by the UI. The node parameters contains arbitrary parameters which are passed to the application.

The URL of the request is https://syrup.keboola.com/docker/sandbox. The request body is in JSON. Enter your Storage API token into X-StorageAPI-Token header and run the request.

Getting the Result

When running the request with valid parameters, you should receive a response similar to this:

{
    "id": "176883685",
    "url": "https://syrup.keboola.com/queue/job/176883685",
    "status": "waiting"
}

This means an asynchronous job which will prepare the sandbox has been created. If curious, view the job progress under Jobs in KBC:

Job progress screenshot

The job will be usually executed very quickly, so you might as well go straight to StorageFiles in KBC. There you will find a data.zip file with a sample data folder. You can now use this folder with your Docker extension or Custom Science

Input Data API Call

The input API call differs in that it must be used with an existing component. It requires componentId obtained from the component registration. This also means that this call can be used both with existing configurations as well as ad-hoc configurations (as in the above sandbox request.

Prepare

We assume you have the same in.c-main.test source table as in the previous request. You can then create configuration for our sample component keboola.docs-docker-example-parameters by visiting the following URL:

https://connection.keboola.com/admin/projects/{projectId}/applications/keboola.docs-docker-example-parameters

Where you replace {projectId} with the Id of the project in KBC (you can find that in the URL). Then create the configuration. The equivalent to what we have used in the Sandbox above call would be

Configuration screenshot

Run the API Request

When you created the configuration, it was assigned a configuration Id — 328831433 — in our example. Use this Id instead of manually crafting the request body.

You can see an Introduction sample request in our collection of requests.

The following is the request body:

{
    "config": "328831433"
}

Where you need to replace 328831433 with your own configuration ID. The request URL is

https://syrup.keboola.com/docker/keboola.docs-docker-example-parameters/input/

Where keboola.docs-docker-example-parameters is the component ID (you can replace that with your own component if you like). Again, do not forget to enter your Storage API token into X-StorageAPI-Token header.

As with the Sandbox call, running the API call will create a job which will execute and produce a data.zip file in StorageFiles.

Important: If you actually want to run the above 328831433 configuration, you also need to set the output mapping from destination.csv to some table.

Summary

  • For unregistered components, use the Sandbox call:
    • the whole configuration must be passed as the body (configData node) of the API call in JSON format
    • the source data is limited to 50 rows
  • For registered components, use the Input data call:
    • the configuration can be either passed as the body (configData node), or it can refer to an existing configuration (config node)
    • the source data is exported unlimited — this can lead to large data folders!
  • Both the Sandbox and Input calls create a job (automatically executed) which produces a data.zip file in your Storage - File Uploads
    • the data.zip folder can be extracted and mapped to your dockerized application
    • the data.zip contains the input tables and files, their manifests and configuration file
    • the data.zip does not contain any data in the out folder, your application has to produce it