1. Home
2. Extending Keboola
3. Common Interface
4. Logging

Logging Specification

• Standard Output and Standard Error
• GELF Logger
  • Setting Up
  • Examples
    • PHP
    • Python
    • Node.js

There are two main, mutually exclusive, ways in which your component can display events Keboola end-users:

1. Using standard output and standard error
2. Using Graylog GELF compatible logger

Using the standard output option requires no extra work from you or your component. You just print all informational messages to standard output and all error messages to standard error. These will be forwarded to Storage Events as informational or error messages. See implementation notes for best practices in logging.

Using a GELF compatible logger requires implementing or including such a logger in your component. However, it offers much greater flexibility: you can send different kinds of messages (such as error, informational, warning, debug), and they can contain additional structured information (not only a plain text string).

Standard Output and Standard Error

By default – unless you have turned on GELF logging in the component configuration, Docker Runner listens to STDOUT and STDERR of the component and forwards the STDOUT content live to Storage API Events (log level info). The content of STDERR is collected and added (if not empty) as the last event of the job with level error. The events are displayed in a Job detail.

The entire output from a component is filter for sensitive values. The Runner keeps track of all encrypted values and if it encounters them in the component output, it replaces them by [hidden] placeholder. This prevents accidental leaking of sensitive information for example in exception traces.

GELF Logger

GELF is a log format allowing you to send structured event messages. The messages can be sent over several transports and you can specify whether they will be silenced or displayed based on their level.

Note: The size of the messages is limited. Sending a message larger than 200KB will cause the component job to fail.

Setting Up

If you turn on GELF logging in the component configuration, our Docker Runner will listen for messages on the transport you specify (UDP, TCP and HTTP are supported). We suggest using TCP as it offers a nice compromise between transport overhead and reliability, but the final choice is up to you. If you choose UDP as a transport, make sure that there is a little delay between your component start and the first message sent (about 1s) to give the network sockets some time to initialize.

Additionally, you can set the visibility of each event message as follows:

• none – Message is ignored entirely.
• camouflage – Generic error message is shown to the end-user instead of the real message content; the full message is logged internally.
• normal – Event message (GELF short_message field) is shown to the end-user; the full message is logged internally.
• verbose – Full message is shown to the user including GELF additional fields.

Default settings for message visibilities:

Examples

Since GELF is sort of a standard format for structured logs, there are a number of libraries available for client implementation. The following examples show how to use the GELF logger in some common languages. Always use the KBC_LOGGER_ADDR and KBC_LOGGER_PORT environment variables in your client, which will be injected into your component by our Docker Runner.

Important: Never rely on the default logger settings.

When developing your component, you may want to use the GELF server for development to test the logging.

PHP

For PHP, use the official GELF client library. To install it, use

composer require graylog2/gelf-php

Then test that logging works with this simple script:

<?php

require("vendor/autoload.php");

$transport = new Gelf\Transport\TcpTransport(getenv('KBC_LOGGER_ADDR'), getenv('KBC_LOGGER_PORT'));
$logger = new \Gelf\Logger($transport);

$logger->emergency("A sample emergency message", ["some" => ["structured" => "data"]]);

You can see the complete component in a sample repository. For other transports, use the UdpTransport or HttpTransport class (AMQP transport is not supported yet). For additional examples on using the library, see its official documentation.

Python

For Python, we strongly suggest using the prepared Component package which takes care of the setup automatically.

If you want to set a GELF logger yourself, you need to choose from a number of libraries available. For example, the logging-gelf library. To install it, use

pip3 install logging_gelf

Then test that logging works with this simple script:

import logging_gelf.handlers
import logging_gelf.formatters
import logging
import os

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
logging_gelf_handler = logging_gelf.handlers.GELFTCPSocketHandler(host=os.getenv('KBC_LOGGER_ADDR'), port=int(os.getenv('KBC_LOGGER_PORT')))
logging_gelf_handler.setFormatter(logging_gelf.formatters.GELFFormatter(null_character=True))
logger.addHandler(logging_gelf_handler)

# remove default logging to stdout
logger.removeHandler(logger.handlers[0])

logging.critical('A sample emergency message')

Due to the nature of Python logging, only some error levels are permitted.

Node.js

There are a number of libraries available for NodeJS. For example, the GrayGelf library.

npm install graygelf

Then test that logging works with this simple script:

var log = require('graygelf')({
  host: process.env.KBC_LOGGER_ADDR,
  port: process.env.KBC_LOGGER_PORT
})

log.info('hello', 'world')
log.info.a('short', 'full', { foo: 'bar' })

Note that the library supports only the UDP transport.