Skip to main content

Docker

Logs​

Docker collector​

Docker is a set of platform-as-a-service products that deliver software in containers. This integration is a Docker container that can use either Filebeat or Fluent Bit to collect logs from other Docker containers and forward them to your Logz.io account.

To use docker-collector-logs, you'll set environment variables when you run the container. The Docker logs directory and docker.sock are mounted to the container, enabling Filebeat or Fluent Bit to collect logs and metadata.

note

Project's GitHub repo

Upgrading to a newer version​

  • When upgrading to a newer version of docker-collector-logs while it is already running, logs within the ignoreOlder timeframe may be resent. To minimize log duplicates, configure the ignoreOlder parameter of the new container to a lower value (e.g., 20m).

  • Version 0.1.0 of docker-collector-logs includes breaking changes. Refer to the change log for further details.

Pull the Docker image​

Download the logzio/docker-collector-logs image.

docker pull logzio/docker-collector-logs

Run the Docker image​

For a complete list of options, see the parameters below the code block.👇

Docker​
docker run --name docker-collector-logs \
--env LOGZIO_TOKEN="<<LOG-SHIPPING-TOKEN>>" \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /var/lib/docker/containers:/var/lib/docker/containers \
logzio/docker-collector-logs
Docker Swarm​
docker service create --name docker-collector-logs \
--env LOGZIO_TOKEN="<<LOG-SHIPPING-TOKEN>>" \
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
--mount type=bind,source=/var/lib/docker/containers,target=/var/lib/docker/containers \
--mode global logzio/docker-collector-logs

Parameters​

ParameterDescriptionField Type
LOGZIO_TOKENYour Logz.io account token. Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.Required
LOGZIO_REGIONLogz.io region code to ship the logs to. This region code changes depending on the region your account is hosted in. For example, accounts in the EU region have region code eu. If you don't specify this parameter, the default value will be used. For more information, see Account region on the Logz.io Docs.Default: (US region)
LOGZIO_TYPEThe log type you'll use with this Docker. Declare your log type for parsing purposes. Logz.io applies default parsing pipelines to the following list of built-in log types. If you declare another type, contact support for assistance with custom parsing. Can't contain spaces.Default: Docker image name
LOGZIO_CODECSet to json if shipping JSON logs. Otherwise, set to plain for plain text format.Default: plain
ignoreOlderSet a time limit on back shipping logs. Upgrading to a newer version of docker-collector-logs while it is already running will cause it to resend logs that are within the ignoreOlder timeframe. You can minimize log duplicates by setting the ignoreOlder parameter of the new docker to a lower value (for example, 20m).Default: 3h
LOGZIO_URLURL for your account listener host. The URL changes depending on the region your account is hosted in. You can skip this parameter if you specify LOGZIO_REGION. If neither LOGZIO_URL nor LOGZIO_REGION is specified, the default value will be used. For more information, see Account region on the Logz.io Docs.Default: listener.logz.io:5015
additionalFieldsInclude additional fields with every message sent, formatted as "fieldName1=fieldValue1;fieldName2=fieldValue2". To use an environment variable, format as "fieldName1=fieldValue1;fieldName2=$ENV_VAR_NAME". In that case, the environment variable should be the only value in the field. If the environment variable can't be resolved, the field is omitted.optional
matchContainerNameComma-separated list of containers you want to collect the logs from. If a container's name partially matches a name on the list, that container's logs are shipped. Otherwise, its logs are ignored. Note: Can't be used with skipContainerNameoptional
skipContainerNameComma-separated list of containers you want to ignore. If a container's name partially matches a name on the list, that container's logs are ignored. Otherwise, its logs are shipped. Note: Can't be used with matchContainerNameoptional
includeLinesComma-separated list of regular expressions to match the lines that you want to include. Note: Regular expressions in this list should not contain commas.optional
excludeLinesComma-separated list of regular expressions to match the lines that you want to exclude. Note: Regular expressions in this list should not contain commas.optional
renameFieldsRename fields with every message sent, formatted as "oldName,newName;oldName2,newName2". To use an environment variable, format as "oldName,newName;oldName2,$ENV_VAR_NAME". When using an environment variable, it should be the only value in the field. If the environment variable can't be resolved, the field will be omitted.optional
HOSTNAMEInclude your host name to display it for the field agent.name. If no value is entered, agent.namedisplays the container id.Default: ''
multilinePatternInclude your regex pattern. See Filebeat's official documentation for more information.Default: ''
multilineNegateInclude 'true' to negate the pattern. Note: Cannot be used without multilinePattern. See Filebeat's official documentation for more information.'false'
multilineMatchSpecifies how Filebeat combines matching lines into an event. The settings are after or before. The behavior of these settings depends on what you specify for negate. Note: Cannot be used without multilinePattern. See Filebeat's official documentation for more information.Default: 'after'
LOG_LEVELSet log level for the collector. Allowed values are: debug, info, warning, errorDefault: info
INPUT_ENCODINGHere is a full list of valid encodings you can use.Default: utf-8
note

By default, logs from docker-collector-logs and docker-collector-metrics containers are ignored.

Check Logz.io for your logs​

Spin up your Docker containers if you haven't done so already. Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don't see your logs, see log shipping troubleshooting.

The logzio-logging plugin​

Docker is a set of platform as a service products that deliver software in containers. Deploy this integration to send your Docker logs to your Logz.io account using a dedicated Logz.io plugin.

Before you begin, you'll need: Docker Engine 17.05 or later, Docker Community Edition (Docker CE) 18.03 or later

Install the plugin from the Docker store​

docker plugin install logzio/logzio-logging-plugin:1.0.2 \
--alias logzio/logzio-logging-plugin

Check to see if logzio-logging-plugin is enabled.

docker plugin ls --filter enabled=true

If logzio-logging-plugin isn't on the list, enable it now.

docker plugin enable logzio/logzio-logging-plugin

Configure global settings with daemon.json​

You can configure all containers with the same options using daemon.json.

For a complete list of options, see the configuration parameters below the code sample.👇

Code sample​
{
  "log-driver": "logzio/logzio-logging-plugin",
  "log-opts": {
    "logzio-url": "<<LISTENER-HOST>>",
    "logzio-token": "<<LOG-SHIPPING-TOKEN>>",
    "logzio-dir-path": "<dir_path_to_logs_disk_queue>"
  }
}
Parameters​
ParameterDescriptionRequired/Default
logzio-tokenYour Logz.io account token. Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.Required
logzio-urlListener URL and port. Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.Required
logzio-dir-pathUnsent logs are saved to this location on the disk.Required
logzio-sourceEvent source.optional
logzio-formatLog message format, either json or text.text
logzio-tagLog tag for the Container ID. For more information, see Log tags for logging driver from Docker.\{\{.ID\}\}
labelsComma-separated list of labels to include in the log message.optional
envComma-separated list of environment variables to include in the log message.optional
env-regexA regular expression to match logging-related environment variables. Used for advanced log tag options. If there is collision between the label and env keys, env wins. Both options add additional fields to the attributes of a logging message.optional
logzio-attributesJSON-formatted metadata to include in the log message.optional

(Optional) Set environment variables​

Some logzio-logging-plugin options are controlled using environment variables. Each of these variables has a default value, so you can skip this step if you're comfortable with the defaults.

Environment variables​
ParameterDescriptionRequired/Default
LOGZIO_DRIVER_LOGS_DRAIN_TIMEOUTTime to wait between sending attempts.5s
LOGZIO_DRIVER_DISK_THRESHOLDThreshold, as % of disk usage, over which plugin will start dropping logs.70
LOGZIO_DRIVER_CHANNEL_SIZEThe number of pending messages that can be in the channel before adding them to the disk queue.10000
LOGZIO_MAX_MSG_BUFFER_SIZEAppends logs that are segmented by Docker with 16kb limit. Specifies the biggest message, in bytes, that the system can reassemble. 1048576 (1 MB) maximum.1048576 (1 MB)
LOGZIO_MAX_PARTIAL_BUFFER_DURATIONHow long the buffer keeps the partial logs before flushing them.500ms

(Optional) Override global settings for an individual container​

You can configure the plugin separately for each container when using the docker run command.

Code sample​
docker run --log-driver=logzio/logzio-logging-plugin \
--log-opt logzio-token=<<LOG-SHIPPING-TOKEN>> \
--log-opt logzio-url=https://<<LISTENER-HOST>>:8071 \
--log-opt logzio-dir-path=./docker_logs \
--log-opt logzio-tag="{{Name}}/{{FullID}}" \
--log-opt labels=region \
--log-opt env=DEV \
--env "DEV=true" \
--label region=us-east-1 \
<<DOCKER-IMAGE-NAME>>

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.

For a complete list of options, see the configuration parameters above. 👆

Check Logz.io for your logs​

Spin up your Docker containers if you haven't done so already. Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don't see your logs, see log shipping troubleshooting.

Metrics​

Deploy this integration to ship metrics from your Docker network using containerized Telegraf agent.

Pull the Docker image​

docker pull logzio/docker-metrics-collector:latest

Run the Docker image​

For a complete list of options, see the parameters below the code block.👇

Docker​
docker run --name telegraf-docker-collector-metrics \
 --env METRICS_TOKEN="<<PROMETHEUS-METRICS-SHIPPING-TOKEN>>" \
 --env LOGZIO_LISTENER="https://<<LISTENER-HOST>>:8053" \
 -v /var/run/docker.sock:/var/run/docker.sock \
 logzio/docker-metrics-collector:latest
Docker Swarm​
docker service create --name telegraf-docker-collector-metrics \
 --env METRICS_TOKEN="<<PROMETHEUS-METRICS-SHIPPING-TOKEN>" \
 --env LOGZIO_LISTENER="https://<<LISTENER-HOST>>:8053" \
 --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
 --mode global logzio/docker-metrics-collector:latest

Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):

  • Replace <<LISTENER-HOST>> with the Logz.io Listener URL for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic.
  • Replace <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with a token for the Metrics account you want to ship to. Look up your Metrics token.

If you prefer to keep these environment variables in an .env file, run the following command:

docker run -d --env-file=docker.env -v /var/run/docker.sock:/var/run/docker.sock logzio/docker-metrics-collector:latest

Parameters​

Below is a list of all environment variables available with this integration. If required, add a variable to the docker run command using the --env flag.

NameDescriptionRequired/Default
METRICS_TOKENYour Logz.io metrics account token.Required
LOGZIO_LISTENERYour Logz.io listener address followed by port 8053.Required/Default: https://listener.logz.io:8053.
DOCKER_ENDPOINTAddress to reach the required Docker Daemon.Default: unix:///var/run/docker.sock.
TIMEOUTThe request timeout for any Docker Daemon query.Default: 5s.
EXCLUDED_IMAGESA list of strings, regexes, or globs, the container image names of which, will not be among the queried containers. !-prefixed negations are possible for all item types to signify that only unmatched container image names should be monitored. For example: imageNameToExclude1,imageNameToExclude2)Default: nil.
GLOBAL_TAGSA comma separated list of key-value pairs that will be added to every metric. For example - key1=value1,key2=value2Default: nil.

Check Logz.io metrics​

Install the pre-built dashboards to enhance the observability of your metrics.

To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.