Logging Specification

There are two main, mutually exclusive, ways in which your application can display events to KBC end-users:

  1. Using standard output and standard error
  2. Using Graylog GELF compatible logger

Using the standard output option requires no extra work from you or your application. You just print all informational messages to the standard output and all error messages to the standard error. These will be forwarded to Storage Events as informational or error messages. As the simplest approach, it is also recommended for Custom Science extensions.

Using a GELF compatible logger requires implementing or including such a logger in your application. However, it offers much greater flexibility: you can send different kinds of messages (such as error, informational, warning, debug) and the messages can contain additional structured information (not only a plain text string)..

Standard Output and Standard Error

By default, the Docker Runner listens to STDOUT and STDERR of the application and forwards any content live to Storage API Events (log levels info and error). The events are displayed in a Job detail.

Make sure your application does not use any output buffering, otherwise all events will be cached after the application finishes. In R applications, the outputs printed in rapid succession are sometimes joined into a single event; this is a known behavior of R and it has no workaround. In Python applications, the output is buffered, but the output buffering may be switched off. The easiest solution is to run your script with the -u option – you would use CMD python -u ./main.py in your Dockerfile.

The events serve to pass only informational and error messages; no data can be passed through. The event message size is limited (about 64KB). If live events are turned off, the amount of complete application output is also limited (about 1MB). If the limit is exceeded, the message will be trimmed.

Gelf Logger

GELF is a log format allowing you to send structured event messages. The messages can be sent over several transports and you can specify whether they will be silenced or displayed based on their level.

Setting Up

If you turn on GELF logging in the component registration, our Docker Runner will listen for messages on the transport you specify (UDP, TCP and HTTP are supported). We suggest using TCP as it offers a nice compromise between transport overhead and reliability, but the final choice is up to you. If you choose UDP as a transport, make sure that there is a little delay between your application start and the first message sent (about 1s) to give the network sockets some time to initialize.

Additionally, you can set the visibility of each event message as follows:

  • none - Message is ignored entirely.
  • camouflage - Generic error message is shown to the end-user instead of the real message content; the full message is logged internally.
  • normal - Event message (GELF short_message field) is shown to the end-user; the full message is logged internally.
  • verbose - Full message is shown to the user including GELF additional fields.

Default settings for message visibilities:

KBC Level Gelf Log Method Syslog Level Default KBC Verbosity
100 debug() 7 none
200 info() 6 normal
250 notice() 5 normal
300 warning() 4 normal
400 error() 3 normal
500 critical() 2 camouflage
550 alert() 1 camouflage
600 emergency() 0 camouflage

Examples

Since GELF is sort of a standard format for structured logs, there are a number of libraries available for client implementation. The following examples show how to use the GELF logger in some common languages. Always use the KBC_LOGGER_ADDR and KBC_LOGGER_PORT environment variables in your client, which will be injected into your component by our Docker Runner.

Important: Never rely on the default logger settings.

When developing your application, you need the GELF server for development.

PHP

For PHP, use the official GELF client library. To install it, use

composer require graylog2/gelf-php

Test the logging with this simple script:

<?php

require("vendor/autoload.php");

$transport = new Gelf\Transport\TcpTransport(getenv('KBC_LOGGER_ADDR'), getenv('KBC_LOGGER_PORT'));
$logger = new \Gelf\Logger($transport);

$logger->emergency("A sample emergency message", ["some" => ["structured" => "data"]]);

You can see a complete component in a sample repository. For other transports, use the UdpTransport or HttpTransport class (AMQP transport is not supported yet). For additional examples on using the library, see its official documentation.

Python

For Python, there are a number of libraries available. For example, the logging-gelf library. To install it, use

pip3 install logging_gelf

Test the logging with the following simple script:

import logging_gelf.handlers
import logging_gelf.formatters
import logging
import os

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
logging_gelf_handler = logging_gelf.handlers.GELFTCPSocketHandler(host=os.getenv('KBC_LOGGER_ADDR'), port=int(os.getenv('KBC_LOGGER_PORT')))
logging_gelf_handler.setFormatter(logging_gelf.formatters.GELFFormatter(null_character=True))
logger.addHandler(logging_gelf_handler)

# remove default logging to stdout
logger.removeHandler(logger.handlers[0])

logging.critical('A sample emergency message')

Due to the nature of Python logging, only some error levels are permitted.

Node.js

There are a number of libraries available for NodeJS. For example, the GrayGelf library.

npm install graygelf

Test the logging with this simple script:

var log = require('graygelf')({
  host: process.env.KBC_LOGGER_ADDR,
  port: process.env.KBC_LOGGER_PORT
})

log.info('hello', 'world')
log.info.a('short', 'full', { foo: 'bar' })

Note that the library supports only the UDP transport.