Docker

Logs

Docker collector

Docker is a set of platform-as-a-service products that deliver software in containers. This integration is a Docker container that uses Fluent Bit to collect logs from other Docker containers and forward them to your Logz.io account.

To use docker-logs-collector, you'll set environment variables when you run the container. The Docker logs directory and docker.sock are mounted to the container, enabling Fluent Bit to collect logs and metadata.

Fluent Bit

note

Project's GitHub repo

Pull the Docker image

Download the logzio/docker-logs-collector image.

docker pull logzio/docker-logs-collector:latest

Run the Docker image

For a complete list of options, see the parameters below the code block.👇

Docker

docker run --name docker-logs-collector \
--env LOGZIO_LOGS_TOKEN="<LOGS-SHIPPING-TOKEN>" \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /var/lib/docker/containers:/var/lib/docker/containers \
-e HEADERS="user-agent:logzio-docker-logs" \
logzio/docker-logs-collector:latest

Parameters

Parameter	Description	Field Type
LOGZIO_LOGS_TOKEN	Your Logz.io logs account token. Replace `<LOGS-SHIPPING-TOKEN>` with the token associated with the account you want to use for shipping logs.	Required
LOGZIO_URL	The full URL to send logs to, including your region if needed. For example, for the EU region, use `https://listener-eu.logz.io:8071`.	Default: `https://listener.logz.io:8071`
LOGZIO_TYPE	Sets the log type.	Default: `logzio-docker-logs`
LOGS_PATH	The path to docker container logs, supports globs.	Default: `/var/lib/docker/containers//.log`
MATCH_CONTAINER_NAME	Specify a container to collect logs from. If the container's name matches, its logs are shipped; otherwise, its logs are ignored. Note: This option cannot be used with SKIP_CONTAINER_NAMES. Use regular expressions to keep records that match a specific field.	optional (Example: `^my-app-container.*`)
SKIP_CONTAINER_NAMES	Comma-separated list of containers to ignore. If a container's name matches a name on this list, its logs are ignored; otherwise, its logs are shipped. Note: This option cannot be used with MATCH_CONTAINER_NAME. Use regular expressions to exclude records matching a specific field.	optional (Example: `test-container,debug-container`)
MATCH_IMAGE_NAME	Specify an image to collect logs from. If the image's name matches, its logs are shipped; otherwise, its logs are ignored. Note: This option cannot be used with SKIP_IMAGE_NAMES. Use regular expressions to keep records that match a specific field.	optional (Example: `my-app-image:v1.2.3`)
SKIP_IMAGE_NAMES	Comma-separated list of images to ignore. If an image's name matches a name on this list, its logs are ignored; otherwise, its logs are shipped. Note: This option cannot be used with MATCH_IMAGE_NAME. Use regular expressions to exclude records that match a specific field.	optional (Example: `test-image,debug-image`)
INCLUDE_LINE	Regular expression to match which lines Fluent Bit should include.	optional (Example: `^ERROR.*`)
EXCLUDE_LINES	Regular expression to match which lines Fluent Bit should exclude.	optional (Example: `.DEBUG.`)
ADDITIONAL_FIELDS	Include additional fields with every message sent, formatted as `"fieldName1:fieldValue1,fieldName2:fieldValue2"`.	optional
SET_FIELDS	Set fields with every message sent, formatted as `"fieldName1:fieldValue1,fieldName2:fieldValue2"`.	optional
LOG_LEVEL	Set log level for Fluent Bit. Allowed values: `debug`, `info`, `warning`, `error`.	Default: `info`
MULTILINE_START_STATE_RULE	Regular expression for the start state rule of multiline parsing. See Fluent Bit's official documentation for further info.	optional (Example: `^Exception`)
MULTILINE_CUSTOM_RULES	Custom rules for multiline parsing, separated by semicolons `;`.	optional (Example: `^\\sat\\s;^Caused by:`)
READ_FROM_HEAD	Specify if Fluent Bit should read logs from the beginning.	Default: `true`
OUTPUT_ID	Specify the output ID for Fluent Bit logs.	Default: `output_id`
HEADERS	Custom headers for Fluent Bit logs.	optional (Example: `x-custom-header:myCustomValue,x-other-header:someOtherValue`)

Check Logz.io for your logs

Spin up your Docker containers if you haven't done so already. Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don't see your logs, see log shipping troubleshooting.

The logzio-logging plugin

Docker is a set of platform as a service products that deliver software in containers. Deploy this integration to send your Docker logs to your Logz.io account using a dedicated Logz.io plugin.

Before you begin, you'll need: Docker Engine 17.05 or later, Docker Community Edition (Docker CE) 18.03 or later

Install the plugin from the Docker store

docker plugin install logzio/logzio-logging-plugin:1.0.2 \
--alias logzio/logzio-logging-plugin

Check to see if logzio-logging-plugin is enabled.

docker plugin ls --filter enabled=true

If logzio-logging-plugin isn't on the list, enable it now.

docker plugin enable logzio/logzio-logging-plugin

Configure global settings with daemon.json

You can configure all containers with the same options using daemon.json.

For a complete list of options, see the configuration parameters below the code sample.👇

Code sample

{
  "log-driver": "logzio/logzio-logging-plugin",
  "log-opts": {
    "logzio-url": "<<LISTENER-HOST>>",
    "logzio-token": "<<LOG-SHIPPING-TOKEN>>",
    "logzio-dir-path": "<dir_path_to_logs_disk_queue>"
  }
}

Parameters

Parameter	Description	Required/Default
logzio-token	Your Logz.io account token. Replace `<<LOG-SHIPPING-TOKEN>>` with the token of the account you want to ship to.	Required
logzio-url	Listener URL and port. Replace `<<LISTENER-HOST>>` with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.	Required
logzio-dir-path	Unsent logs are saved to this location on the disk.	Required
logzio-source	Event source.	optional
logzio-format	Log message format, either `json` or `text`.	`text`
logzio-tag	Log tag for the Container ID. For more information, see Log tags for logging driver from Docker.	`\{\{.ID\}\}`
labels	Comma-separated list of labels to include in the log message.	optional
env	Comma-separated list of environment variables to include in the log message.	optional
env-regex	A regular expression to match logging-related environment variables. Used for advanced log tag options. If there is collision between the `label` and `env` keys, `env` wins. Both options add additional fields to the attributes of a logging message.	optional
logzio-attributes	JSON-formatted metadata to include in the log message.	optional

(Optional) Set environment variables

Some logzio-logging-plugin options are controlled using environment variables. Each of these variables has a default value, so you can skip this step if you're comfortable with the defaults.

Environment variables

Parameter	Description	Required/Default
LOGZIO_DRIVER_LOGS_DRAIN_TIMEOUT	Time to wait between sending attempts.	`5s`
LOGZIO_DRIVER_DISK_THRESHOLD	Threshold, as % of disk usage, over which plugin will start dropping logs.	`70`
LOGZIO_DRIVER_CHANNEL_SIZE	The number of pending messages that can be in the channel before adding them to the disk queue.	`10000`
LOGZIO_MAX_MSG_BUFFER_SIZE	Appends logs that are segmented by Docker with 16kb limit. Specifies the biggest message, in bytes, that the system can reassemble. `1048576` (1 MB) maximum.	`1048576` (1 MB)
LOGZIO_MAX_PARTIAL_BUFFER_DURATION	How long the buffer keeps the partial logs before flushing them.	`500ms`

(Optional) Override global settings for an individual container

You can configure the plugin separately for each container when using the docker run command.

Code sample

docker run --log-driver=logzio/logzio-logging-plugin \
--log-opt logzio-token=<<LOG-SHIPPING-TOKEN>> \
--log-opt logzio-url=https://<<LISTENER-HOST>>:8071 \
--log-opt logzio-dir-path=./docker_logs \
--log-opt logzio-tag="{{Name}}/{{FullID}}" \
--log-opt labels=region \
--log-opt env=DEV \
--env "DEV=true" \
--label region=us-east-1 \
<<DOCKER-IMAGE-NAME>>

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.

For a complete list of options, see the configuration parameters above. 👆

Check Logz.io for your logs

Spin up your Docker containers if you haven't done so already. Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don't see your logs, see log shipping troubleshooting.

Metrics

Deploy this integration to ship metrics from your Docker network using containerized Telegraf agent.

Pull the Docker image

docker pull logzio/docker-metrics-collector:latest

Run the Docker image

For a complete list of options, see the parameters below the code block.👇

Docker

docker run --name telegraf-docker-collector-metrics \
 --env METRICS_TOKEN="<<PROMETHEUS-METRICS-SHIPPING-TOKEN>>" \
 --env LOGZIO_LISTENER="https://<<LISTENER-HOST>>:8053" \
 -v /var/run/docker.sock:/var/run/docker.sock \
 logzio/docker-metrics-collector:latest

Docker Swarm

docker service create --name telegraf-docker-collector-metrics \
 --env METRICS_TOKEN="<<PROMETHEUS-METRICS-SHIPPING-TOKEN>" \
 --env LOGZIO_LISTENER="https://<<LISTENER-HOST>>:8053" \
 --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
 --mode global logzio/docker-metrics-collector:latest 

Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):

Replace <<LISTENER-HOST>> with the Logz.io Listener URL for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic.
Replace <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with a token for the Metrics account you want to ship to. Look up your Metrics token.

If you prefer to keep these environment variables in an .env file, run the following command:

docker run -d --env-file=docker.env -v /var/run/docker.sock:/var/run/docker.sock logzio/docker-metrics-collector:latest

Parameters

Below is a list of all environment variables available with this integration. If required, add a variable to the docker run command using the --env flag.

Name	Description	Required/Default
METRICS_TOKEN	Your Logz.io metrics account token.	Required
LOGZIO_LISTENER	Your Logz.io listener address followed by port `8053`.	Required/Default: `https://listener.logz.io:8053`.
DOCKER_ENDPOINT	Address to reach the required Docker Daemon.	Default: `unix:///var/run/docker.sock`.
TIMEOUT	The request timeout for any Docker Daemon query.	Default: `5s`.
EXCLUDED_IMAGES	A list of strings, regexes, or globs, the container image names of which, will not be among the queried containers. !-prefixed negations are possible for all item types to signify that only unmatched container image names should be monitored. For example: `imageNameToExclude1,imageNameToExclude2)`	Default: `nil`.
GLOBAL_TAGS	A comma separated list of key-value pairs that will be added to every metric. For example - `key1=value1,key2=value2`	Default: `nil`.

Check Logz.io metrics

Install the pre-built dashboards to enhance the observability of your metrics.

To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.

Logs​

Docker collector​

Pull the Docker image​

Run the Docker image​

Docker​

Parameters​

Check Logz.io for your logs​

The logzio-logging plugin​

Install the plugin from the Docker store​

Configure global settings with daemon.json​

Code sample​

Parameters​

(Optional) Set environment variables​

Environment variables​

(Optional) Override global settings for an individual container​

Code sample​

Check Logz.io for your logs​

Metrics​

Pull the Docker image​

Run the Docker image​

Docker​

Docker Swarm​

Parameters​

Check Logz.io metrics​

Logs

Docker collector

Pull the Docker image

Run the Docker image

Docker

Parameters

Check Logz.io for your logs

The logzio-logging plugin

Install the plugin from the Docker store

Configure global settings with daemon.json

Code sample

Parameters

(Optional) Set environment variables

Environment variables

(Optional) Override global settings for an individual container

Code sample

Check Logz.io for your logs

Metrics

Pull the Docker image

Run the Docker image

Docker

Docker Swarm

Parameters

Check Logz.io metrics