Automated Build

An important part of the Docker ecosystem is a Docker registry. It acts as a folder of images, taking care of their storing and building. Docker Hub is the official Docker registry. There are also alternative registries, such as Quay, or completely private registries such as AWS ECR where we are keen to host your images for you.

We support public and private images on both Docker Hub and Quay registries, as well as private images on AWS ECR. Other registries are not yet supported.

For reliability reasons, we strongly recommend to use the AWS ECR provisioned by the Keboola Developer Portal.

Working with a Registry

In order to run an image, pull (docker pull) that image to your machine. The docker run command does that automatically for you. So, when you do:

docker run -i quay.io/keboola/base-php70

you will see something like this:

Unable to find image 'quay.io/keboola/base-php70:latest' locally
latest: Pulling from keboola/base-php70
a3ed95caeb02: Pull complete

When you build an image locally, push (docker push) it to the Docker registry. Then the image can be shared with anyone (if public), or with your organization (if private).

Because the image is defined only by the Dockerfile instructions (and optionally build context), you can take a shortcut and give the registry only the Dockerfile. The registry will then build the image on its own infrastructure. This is best done by setting up an automated build which links a git repository containing the Dockerfile (and usually the application code) with the registry.

Setting up a Repository on Quay

This may get slightly confusing because we will create a new Image Repository and link that to an existing Github Repository. Use the sample repository created in our tutorial.

Create an account and organization, and then Create a New Repository:

Create Repository

In the repository configuration, select Link to a Github Repository Push:

Repository configuration

Then link the image repository to a Github repository (you can use our sample repository):

Link repositories

After that, configure the build trigger. The easiest way to do that is setting the trigger to All Branches and Tags. It will trigger an image rebuild on every commit to the repository. You can also set the build trigger only to a specific branch, for example, heads/master:

Configure build trigger for branch

An alternative option is to configure the trigger to a specific tag. For Semantic versioning, the following regular expression ^tags/[0-9]+\.[0-9]+\.[0-9]+$ ensures the image is rebuilt only when you create a new tag.

Configure build trigger for tag

Regardless of your chosen approach, finish setting up the trigger by completing the wizard:

Configure build trigger

Pushing a new commit into a git repository or creating a new tag (depending on the trigger setting) will now trigger a new build of the Docker Image. Also note that the image automatically inherits the git repository tag or branch name. So, when you push a commit to the master branch, you will get an image with a tag master (which will move away from any older image builds). When creating a 1.0.0 tag, you will get an image with a 1.0.0 tag.

Setting up a Repository on AWS ECR

If you do not have a Keboola Developer Portal account yet, head to the API documentation, create a new account and register your vendor.

The Get credentials to ECR repository API call will create a repository and temporary credentials for logging into the AWS ECR registry and uploading your Docker image.

If you want to integrate this process into a CI tool like Travis or CircleCI, do not use your Keboola Developer Portal credentials to log in. For this case, the Generate credentials for a service account API call will create a service user whose credentials are safe to share with Travis or the CI tool of your choice.

Sample Integration with Travis CI

Generate the service username and password using the Generate credentials for service account API call and save these to environment variables:

  • KBC_DEVELOPERPORTAL_USERNAME with the login
  • KBC_DEVELOPERPORTAL_PASSWORD with the password
  • KBC_DEVELOPERPORTAL_URL with the string https://apps-api.keboola.com
  • KBC_DEVELOPERPORTAL_VENDOR with the vendor of the app
  • KBC_DEVELOPERPORTAL_APP the appId
  • KBC_APP_REPOSITORY the app repository name (ex. keboola/docker-demo-app)

Environment variables in Travis CI

Then simply paste this code in your deploy script:

# Deploy to repository provided by Keboola Developer Portal
docker pull quay.io/keboola/developer-portal-cli-v2:latest
export REPOSITORY=`docker run --rm \
  -e KBC_DEVELOPERPORTAL_USERNAME \
  -e KBC_DEVELOPERPORTAL_PASSWORD \
  -e KBC_DEVELOPERPORTAL_URL \
  quay.io/keboola/developer-portal-cli-v2:latest ecr:get-repository \
  $KBC_DEVELOPERPORTAL_VENDOR $KBC_DEVELOPERPORTAL_APP`
docker tag $KBC_APP_REPOSITORY:latest $REPOSITORY:$TRAVIS_TAG
docker tag $KBC_APP_REPOSITORY:latest $REPOSITORY:latest
eval $(docker run --rm \
  -e KBC_DEVELOPERPORTAL_USERNAME \
  -e KBC_DEVELOPERPORTAL_PASSWORD \
  -e KBC_DEVELOPERPORTAL_URL \
  quay.io/keboola/developer-portal-cli-v2:latest ecr:get-login \
  $KBC_DEVELOPERPORTAL_VENDOR $KBC_DEVELOPERPORTAL_APP)
docker push $REPOSITORY:$TRAVIS_TAG
docker push $REPOSITORY:latest

# Deploy the application to KBC
docker run --rm \
  -e KBC_DEVELOPERPORTAL_USERNAME \
  -e KBC_DEVELOPERPORTAL_PASSWORD \
  -e KBC_DEVELOPERPORTAL_URL \
  quay.io/keboola/developer-portal-cli-v2:latest update-app-repository \
  $KBC_DEVELOPERPORTAL_VENDOR $KBC_DEVELOPERPORTAL_APP $TRAVIS_TAG

This code will tag your image with the relevant tags (latest and the tag of the build) and push them to our registry. Finally, it will update the application in Developer Portal to use the current tag. This means that the new version of the application is immediately deployed into KBC.

This sample deployment script uses the Developer Portal CLI tool. The CLI (delivered as a Docker image) provides the deploy script with simple commands to retrieve the repository and credentials to our AWS ECR registry. The ecr:get-repository command returns the repository associated with the user in the variable KBC_DEVELOPERPORTAL_USERNAME. The ecr:get-login command returns a docker login ... command to authenticate to that repository.

Run test jobs of your new image against live configurations

Before deploying your new image to the developer portal, you may want to try it out on some ‘real’ configurations in your project. You can do this by adding some environment variables to Travis with an appropriate storage token and configurationId. (It is highly recommended to create a dedicated token for this task.)

The commands will need two extra environment variables including the ones listed above:

  • KBC_STORAGE_TOKEN — the storage token that the test(s) will run under
  • KBC_APP_TEST_CONFIG — the configuration to test

Note that if you want to run multiple test jobs, simply repeat the command with the different configuration IDs that you would like to test.

Then simply add the following steps to your script section in .travis.yml to run the test jobs.

# Push master image to ECR
docker pull quay.io/keboola/developer-portal-cli-v2:latest
export REPOSITORY=`docker run --rm \
  -e KBC_DEVELOPERPORTAL_USERNAME \
  -e KBC_DEVELOPERPORTAL_PASSWORD \
  -e KBC_DEVELOPERPORTAL_URL \
  quay.io/keboola/developer-portal-cli-v2:latest ecr:get-repository \
  $KBC_DEVELOPERPORTAL_VENDOR $KBC_DEVELOPERPORTAL_APP`
docker tag $KBC_APP_REPOSITORY:latest $REPOSITORY:master
eval $(docker run --rm \
  -e KBC_DEVELOPERPORTAL_USERNAME \
  -e KBC_DEVELOPERPORTAL_PASSWORD \
  -e KBC_DEVELOPERPORTAL_URL \
  quay.io/keboola/developer-portal-cli-v2:latest ecr:get-login \
  $KBC_DEVELOPERPORTAL_VENDOR $KBC_DEVELOPERPORTAL_APP)
docker push $REPOSITORY:master

# Run live test job on new master image
# env requires: KBC_STORAGE_TOKEN and KBC_APP_TEST_CONFIG (a token to a test project where the app has a configuration)
docker pull quay.io/keboola/syrup-cli:latest
docker run --rm \
  -e KBC_STORAGE_TOKEN \
  quay.io/keboola/syrup-cli:latest run-job \
  $KBC_DEVELOPERPORTAL_APP $KBC_APP_TEST_CONFIG master

The commands above do as follows:

  • Pull the developer portal cli client Developer Portal CLI
  • Get the application’s KBC repository url from the developer portal
  • Tag the image as master
  • Push the image to the repository
  • Pull the job runner cli client Syrup PHP CLI
  • Run the specified test job on KBC using the /{component}/{config}/run/tag/{tag}Keboola Docker API

You can see both .travis.yml and the deploy script (deploy.sh) in our Docker Demo App GitHub repository.