Code Pattern Interface

This page describes how the code patterns works internally, as a part of Keboola Connection.

Common Interface

Code pattern is a special type of the Component, therefore, common interface applies to it. In the links below you can find the basic information to integrate your own components into Keboola Conneciton.

Let’s highlight the most important information.

Code Generation Process

This section shows how the code generation process works from start to end.

  • First, there must be a published code pattern component, for example keboola.example-pattern.
  • The component must have configured Supported Transformations.
    • For example, it supports keboola.snowflake-transformation.
  • Create a transformation with the code pattern in the user interface.
  • Click the Generate Code button.
  • User interface calls the Generate Action on the keboola.example-pattern component.
  • The action finishes with the correct exit code:
    • If success, exit code = 0
      • The component stdout contains JSON in the Output Format.
      • The code blocks in the parameters are stored to the transformation.
    • If failure exit code = 1 or 2
      • The error is processed according to the exit code, see Return Values.
      • The previous version of the generated code remains in the transformation.
  • The generated code is read-only displayed in the user interface.

Generate Action

There are two types of the component’s actions:

Code patterns:

Expected behavior of the generate action :

  • The action is started by the Run component action API call.
  • The CMD process defined in the Dockerfile is started in the container.
  • The component’s generates a transformation code based on the Configration.
  • The result is written in the Output Format to stdout.
  • The process ends successfully with exit code = 0 (or with other Return Value in an error occured).
  • API returns the result of the action.
  • The user interface modifies the transformation’s configuration and saves it.

Configuration

The Configuration File config.json file in the KBC_DATADIR contains:

  • action key, with the generate value, name of the Action.
  • storage key contains current input and output mapping from the transformation.
  • parameters key modifies generated code.
    • _componentId key contains the id of the target transformation component.
      • For example keboola.snowflake-transformation.
      • Based on this, it is possible to modify the generated code, e.g. for various SQL dialects.
    • The other keys are provided from the Parameters Form, filled in by the user.

Note: Read more about KBC_DATADIR environment variable in the Environment.

Example configuration. Examples of the storage key can be found here.

{ 
  "action": "generate",
  "storage": {
    "input": {
      "tables": ["..."]
     },
     "output": {
       "tables": ["..."]
     }
  },
  "parameters": {
    "_componentId": "keboola.snowflake-transformation",
    "form_parameter_1": "value 1",
    "form_parameter_2": "value 2"
  }
}

Output Format

The component must write the generated code to stdout in the following JSON format:

  • storage key contains a new transformation’s input and output mapping.
    • It is optional, if absent, the mapping remains unchanged.
    • It is copied into the transformation’s configuration storage key.
    • In the Configuration File - Tables is a schema and examples.
  • parameters key with the generated code.
    • It is copied into the transformation’s configuration parameters key.
    • Schema blocks -> codes -> script must be used. See below.
    • Each statement must be a separate item in the script array.

Example output. Examples of the storage key can be found here.

{
  "storage": {
    "input": {
      "tables": ["..."]
     },
     "output": {
       "tables": ["..."]
     }
  },
  "parameters": {
    "blocks": [
      {
        "name": "Generated block",
        "codes": [
          {
            "name": "Generated code",
            "script": [
              "CREATE TABLE table1;",
              "SELECT foo1, foo2 FROM table2 INTO bar;"
            ]
          }
        ]
      }
    ]   
  }
}

Developer Portal

Component must be registered in the Keboola Developer Portal, see Component Quick Start.

For code pattern component it is necessary to take following extra steps.

First, create a component with the Code Pattern type.

Screenshot -- Add component

Open the component edit page and modify the settings described in the following sections.

Screenshot -- Edit component page

Configuration Schema

Screenshot -- Configuration schema

Supported Components

Each code pattern can generate a code for one or more types of the transformation’s component.

  • An array of these component ids must be entered in the Configration Schema.
  • At the root level, to the supported_components key.

Screenshot -- List of the supported components

Now, when creating one of the listed transformation’s component,
then the published code pattern will be available in the select box.

Screenshot -- Create a new transformation

The code pattern’s Configuration contains parameters._componentId key, so it is possible to distinguish for which transformation’s component the code is generated.

Next Steps

  • Tutorial helps you to implement your first code pattern.
  • Code Patterns Help shows the code patterns from the user’s point of view.