1. Home
2. Extending Keboola
3. Common Interface
4. Manifest Files
5. OUT tables with Native Types

/data/out/tables manifests with Native Types

Native Types provide a structured way for components to define their handling of data types, referred to as “Native Types.”

The level of type handling is specified by the dataTypeSupport property, which can take one of three values:

• Authoritative: The component reliably enforces specific data types.
• Hints: The component provides type suggestions that may not always be reliable.
• None: Represents the legacy state with no explicit type handling.

This design overcomes limitations in current settings, where all components automatically produce typed tables when a project switches to Native Types. For instance, Data Sources that output unreliable type hints (e.g., an int column containing values like N/A) can now explicitly signal their limitations, reducing downstream issues.

An output table manifest sets options for transferring a table to Storage. The following examples list available manifest fields; all of them are optional. The destination field overrides the table name generated from the file name; it can (and commonly is) overridden by the end-user configuration.

{
    "destination": "out.c-main.Leads",
    "incremental": true,
    "delimiter": "\t",
    "enclosure": "\"",
    "manifest_type": "output",
    "has_header": true,
    "description": "Best table",
    "table_metadata": ...
    "schema": ...
}

The table_metadata fields allow you to set Metadata for the table. The table_metadata field corresponds to the Table Metadata API call. The key and value of the object are passed directly to the API; the provider value is filled by the Id of the running component (e.g., keboola.ex-db-snowflake).

{
    ...
    "table_metadata": {
        "something else": "a value"
    }
}

Additionally, the following options will cause the specified rows to be deleted from the source table before the new table is imported. See an example. Using this option makes sense only with incremental loads.

{
    ...
    "delete_where": [
        {
            "where_filters": [
                {
                    "column": "column name",
                    "operator": "eq",
                    "values_from_set": ["value1", "value2"]
                }
            ]
        }
    ]
}

The schema [optional] field allow you to create a table with Native Data Types columns. Each object in the schema array represents one column:

• The name [required] field specifies the column name.
• The data_type [optional] field defines the data type for different storage backends, referred to as “Native Types”.
  • The base [required] type is always required and can have values specified in the Base Types documentation.
  • Other types like Snowflake and BigQuery are optional and allows you to specify settings for a particular database backend.
• The nullable [optional] field indicates if the column can be null.
• The primary_key [optional] field specifies if the column is a primary key.
• The description [optional] field provides a description of the column.
• The metadata [optional] field allows setting additional metadata for the column.

{
    "schema": [
        {
            "name": "id",
            "data_type": {
                "base": {
                    "type": "INTEGER",
                    "length": "11",
                    "default": "123"
                },
                "snowflake": {
                    "type": "GEOMETRY",
                    "length": "123,123,4455",
                    "default": "POINT(1 1)"
                },
                "bigquery": {
                    "type": "VARCHAR",
                    "length": "123",
                    "default": null
                }
            },
            "nullable": false,
            "primary_key": true,
            "description": "Optional description of the column",
            "metadata": {
                "KBC.someColumnMetadata": "value1"
                "KBC.someOther": "value2"
            }
        }
    ]
}

Base Types

Source data types are mapped to a destination using a base type. The current base types are STRING, INTEGER, NUMERIC, FLOAT, BOOLEAN, DATE, and TIMESTAMP. This means that, for example, a MySQL extractor may store the value BIGINT as a type of column; that type maps to the INTEGER base type. When the Snowflake writer consumes this value, it will read the base type INTEGER and choose a corresponding type for Snowflake, which happens to be also INTEGER. This ensures high interoperability between the components. Please take a look at the conversion table below.

View the extracted data types in the storage tables detail:

You can also override the data type:

When you use the table (e.g., in the Snowflake writer), you’ll see the data type you have configured:

The data is converted only when writing or copying (e.g., to a transformation or a writer). That means that you can extract an integer column, mark it as a timestamp in storage and write it as an integer into a target database (though you’ll be offered to write it as a timestamp).

You access both the source and base data type through the corresponding API.

Nullable Conversion

Nullable conversion, which transforms an empty string originating from data into a null value, refers to the process where a textual value consisting solely of an empty string "" is replaced with the value null.

Data Type Conversions

As described above, the source data type is converted to a base data type stored in metadata storage. The base type is then converted to the target data type. The following tables show mappings for each base type. The mapping causes possible information loss (e.g., assigning SMALLINT to INTEGER). To minimize this, we also keep track of the data type size and transfer that if possible. For example, a SMALLINT column would be stored as base type INTEGER with size 2. If the target database supports integer sizes, you will be offered to set the type in the target database as INTEGER(2).

STRING

Base type STRING represents any textual type; both CHARACTER VARYING (or VARCHAR) and TEXT types are included. Also, the string base type is used for any other unrecognized type on input. It means that the source type column is not an exhaustive list in the following table. It’s a list of suitable string types converted to a string. All other unknown types are converted to a string as well.

INTEGER

The INTEGER base type represents data types for whole numbers.

NUMERIC

The NUMERIC base type represents fixed-point fractional numbers (real, numeric or decimal data types).

FLOAT

The FLOAT base type represents floating-point fractional numbers (float or double data types).

BOOLEAN

The BOOLEAN base type represents a true or false value.

DATE

The DATE base type represents a date value without a time portion.

TIMESTAMP

The TIMESTAMP base type represents a date value with a time portion.