Component Configurations Tutorial

Configurations are an important part of a KBC project. Most operations are available in the UI, use the API if you want to manipulate them programmatically.

When working with the component configurations API, you usually need to know the componentId. You can get a list of all available components with the API index call, see example.

It will give you something like this:

{
    "host": "6f151bc57e25",
    "api": "storage",
    "version": "v2",
    "revision": "11dc5566bfe7d09ec93dfa7c252423904408891e",
    "documentation": "http://docs.keboola.apiary.io/",
    "components": [
        {
            "id": "keboola.ex-db-snowflake",
            "type": "extractor",
            "name": "Snowflake",
            "description": "Cloud-Native Elastic Data Warehouse Service",
            "data": {
                "definition": {
                    "type": "aws-ecr",
                    "uri": "147946154733.dkr.ecr.us-east-1.amazonaws.com/developer-portal-v2/keboola.ex-db-snowflake",
                    "tag": "1.2.5"
                },
                "vendor": {
                    "contact": [
                        "Keboola",
                        "Křižíkova 488/115, Praha, CZ",
                        "support@keboola.com"
                    ]
                }
            },
            "flags": [
                "genericDockerUI",
                "encrypt"
            ],
            "uri": "https://syrup.keboola.com/docker/keboola.ex-db-snowflake",
            "documentationUrl": "https://github.com/keboola/db-extractor-snowflake/blob/master/README.md"
        }
    ],
    "services": [
        {
            "id": "docker-runner",
            "url": "https://docker-runner.keboola.com"
        },
        {
            "id": "import",
            "url": "https://import.keboola.com"
        },
        {
            "id": "syrup",
            "url": "https://syrup.keboola.com"
        }
    ],
    "urlTemplates": {
        "orchestrationJob": "/admin/projects/&&projectId&&/orchestrations/&&orchestrationId&&/jobs/&&jobId&&"
    }
}

From here, you can select a particular component. In the following examples, we will use keboola.ex-db-snowflake — the Snowflake Database Extractor.

Inspecting Configuration

To obtain configuration details, use the List Configs call, which will return all the configuration details. This means

  • the configuration itself (configuration) — section on configuration follows;
  • configuration rows (rows) — additional data of the configuration; and
  • configuration state (state) — component state.

Please note that the contents of the configuration, rows and state section depend purely on the component itself. See an example

A sample result for Snowflake extractor looks like this:

[
    {
        "id": "328864809",
        "name": "Sample database",
        "description": "",
        "created": "2017-11-06T13:28:48+0100",
        "creatorToken": {
            "id": 27865,
            "description": "ondrej.popelka@keboola.com"
        },
        "version": 3,
        "changeDescription": "Create query account",
        "isDeleted": false,
        "configuration": {
            "parameters": {
                "db": {
                    "port": 443,
                    "ssh": {
                        "sshPort": 22
                    },
                    "host": "kebooladev.snowflakecomputing.com",
                    "user": "HELP_TUTORIAL",
                    "#password": "KBC::ComponentProjectEncrypted==r1C314c+6+W50aKQp6yyKjgaN31q+2gyo28R+L5u8cmbf2taSWlrzR5AfhDOIYFMH5b8XUj2K16iHWMHLrUWaA==",
                    "database": "HELP_TUTORIAL",
                    "schema": "HELP_TUTORIAL",
                    "warehouse": "DEV"
                },
                "tables": [
                    {
                        "enabled": true,
                        "incremental": false,
                        "outputTable": "in.c-keboola-ex-db-snowflake.account",
                        "primaryKey": [],
                        "query": "SELECT * FROM account;",
                        "id": 25998,
                        "name": "account"
                    }
                ]
            }
        },
        "rowsSortOrder": [],
        "rows": [],
        "state": [],
        "currentVersion": {
            "created": "2017-11-06T13:30:12+0100",
            "creatorToken": {
                "id": 27865,
                "description": "ondrej.popelka@keboola.com"
            },
            "changeDescription": "Create query account"
        }
    }
]

The important part is the configuration id 328864809, which is required in the following examples.

Configuration Versions

When you update a configuration, actually a new configuration version is created. In the above calls, only the last (active/published) configuration is returned. To obtain a list of all recorded versions, use the list versions call. See an example which would give you the following output:

[
  {
    "version": 2,
    "created": "2016-05-30T18:04:04+0200",
    "creatorToken": {
      "id": 53044,
      "description": "ondrej.popelka@keboola.com"
    },
    "changeDescription": "",
    "name": "Sample database",
    "description": ""
  },
  {
    "version": 1,
    "created": "2016-05-30T18:01:42+0200",
    "creatorToken": {
      "id": 53044,
      "description": "ondrej.popelka@keboola.com"
    },
    "changeDescription": "",
    "name": "Sample database",
    "description": ""
  }
]

The field version represents the version_id in the following API example.

Creating a Configuration Copy

When you have chosen a particular version, you can create a new independent configuration copy out of it. See an example how to create a new configuration called test-copy from version 3 of the 328864809 configuration for the keboola.ex-db-snowflake component.

It will return the ID of the newly created configuration:

{
    "id": "328874097"
}

Modifying a configuration

Modifying a configuration version means that a new version is created. For modifying a configuration, use the Update configuration API call.

See an example for modifying a configuration to {"foo": "bar"}. Notice that configuration must be sent in a form field as the endpoint does not accept pure JSON (yet). You should always check the component configuration documentation for the supported configuration options. Using unsupported options may lead to funny results.