Warning: βtemplatesβ is an experimental feature.
Templates repository is a directory stored in:
The repository contains a manifest and directories with templates.
π .keboola - metadata directory
β π¦ repository.json - manifest, paths and versions
π [template] - template directory, see below
β π [template version]
β ...
...
All templates are listed in the repository manifest file .keboola/repository.json
.
Repository manifest structure:
version
- current major version, now 2
author
- repository author
name
- author nameurl
- url to authorβs websitetemplates
(array) - information about the project
id
- template IDname
- a human-readable namedescription
- short description of the templatecategories
- string[] - list of template categories, e.g., ["Data Extraction", "E-Commerce"]
Other
category.path
- path to the template directoryversions
(array)
version
- semantic versiondescription
- short description of the versionstable
- is the template ready for production use?path
- path to the template version directorycomponents
(array) - list of components used by the templateSnowflake Writer (data destination) component ID differs on AWS and Azure stacks, because staging storage differs.
keboola.wr-db-snowflake
is used for AWS stacks.keboola.wr-snowflake-blob-storage
is used for Azure stacks.<keboola.wr-db-snowflake>
in the repository.json
, in the components
list.SnowflakeWriterComponentId()
in Jsonnet Files.{
"version": 2,
"author": {
"name": "Keboola",
"url": "https://keboola.com"
},
"templates": [
{
"id": "my-template",
"name": "My Template",
"description": "Full workflow to ...",
"path": "my-template",
"versions": [
{
"version": "0.0.1",
"description": "",
"stable": false,
"path": "v0"
},
{
"version": "1.2.3",
"description": "Version notes.",
"stable": true,
"path": "v1"
}
]
}
]
}
Creating template
git
command.Using template
templates.repositories
key.A template directory is stored in the repository and contains directories with template versions.
π [template]
β π [template version]
β£ π src
β β πͺ inputs.jsonnet - definition of user inputs
β β£ πͺ manifest.jsonnet - template manifest, object IDs and paths
β β£ π© description.md - description which is displayed on the template detail page
β β£ π© README.md - detailed information, changelog, ...
β β π [component-type]
β β π [component-id]
β β π [config-name] - structure is similar to the project structure,
β β£ πͺ config.jsonnet but instead of JSON files there are JSONNET templates
β β£ πͺ meta.jsonet
β β£ π© description.md
β ...
β π tests - tests directory
β£ π [test name]
β β£ π expected-out - expected structure of the project directory
β β after applying the template in the test
β β πͺ inputs.json - sample inputs used to apply the template in the test
β ...
...
There are three types of template description:
description
field in the repository.json:
description.md
file in the template src directory:
README.md
file in the template src directory:
More Details
section.A template is identified by <repository>/<template-id>/<version>
, e.g., keboola/my-template/1.2.3
.
Each template version is stored in a separate directory, see the directory structure.
Templates use semantic versioning:
<major>.<minor>.<patch>
.v1.2.3
, prefix v
is optional.my-template/v1
references the latest available version 1.x.x
.my-template/v1.4
references the latest available version 1.4.x
.<major>
version.3.2.1
the directory name should be v3
.For small changes in the template, it is recommended to update the existing version.
<minor>
or <patch>
part of the version in the repository manifest.For larger changes in the template, it is recommended to create a new <major>
version.
v3
-> v4
.4.0.0
.v3
-> v4
.v4
-> v3
.All configurations and configuration rows are defined in the manifest file manifest.jsonnet
.
Each template should have a mainConfig
that can be started in the UI by pressing the Run button after the template is used.
This is usually the main orchestration/flow.
Template manifest structure:
mainConfig
- main configuration
componentId
- ID of the componentid
- human-readable ID of the configuration defined by ConfigId
functionconfigurations
- array of component configurations
componentId
- ID of the componentid
- human-readable ID of the configuration defined by ConfigId
functionpath
- path to the configuration from the template version directoryrows
- array of configuration rows (if the component supports rows)
id
- human-readable ID of the row defined by ConfigRowId
functionpath
- path to the row from the configuration directory{
mainConfig: {
componentId: "keboola.orchestrator",
id: ConfigId("orchestrator"),
},
configurations: [
{
componentId: "keboola.ex-db-mysql",
id: ConfigId("country"),
path: "extractor/keboola.ex-db-mysql/country",
rows: [
{
id: ConfigRowId("people"),
path: "rows/people",
},
{
id: ConfigRowId("cities"),
path: "rows/cities",
},
],
},
],
}
All user inputs are defined in the inputs.jsonnet
.
Read more in Template Inputs.
Files saved in the _common
directory in the repository directory can be accessed by every template using the <common>/
prefix.
Use of the _common
directory in manifest.jsonnet
:
{
configurations: [
{
componentId: "ex-generic-v2",
id: ConfigId("myconfig"),
path: "<common>/foo/bar/extractor/ex-generic-v2/myconfig",
rows: [],
}
],
}
Use of the _common
directory in a Jsonnet file (config.jsonnet
):
local part1 = import "lib/part1.jsonnet";
local part2 = import "/<common>/foo/bar/extractor/ex-generic-v2/myconfig/lib/part2.jsonnet";
std.mergePatch(part1, part2)