Overview

You can use a Helm chart to ship k8s logs to Logz.io via Filebeat.

Helm is a tool for managing packages of pre-configured Kubernetes resources using Charts. Logzio-k8s-logs allows you to ship logs from your Kubernetes cluster to Logz.io. You can either deploy this Daemonset with the standard Filebeat configuration or with Filebeat Autodiscover. (Learn more about Filebeat Autodiscover from Elastic.)

Filebeat’s basic configuration tends to split longer, multiline logs into multiple logs - 1 log per line. See the relevant tab for details and options for controlling this setting.

Helm 2 reached EOL on November 2020. This document follows the command syntax recommended for Helm 3, but the Chart will work with both Helm 2 and Helm 3.

Standard configuration

Before you begin, you’ll need:

  • Helm CLI installed
  • Outgoing traffic to destination port 5015 allowed
Add logzio-k8s-logs repo to your helm repo list
helm repo add logzio-helm https://logzio.github.io/logzio-helm
helm repo update

Deploy

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-REGION>> with your region’s code (for example, eu). For more information on finding your account’s region, see Account region.

Replace <<CLUSTER-NAME>> with your cluster’s name.

helm install --namespace=kube-system \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
logzio-k8s-logs logzio-helm/logzio-k8s-logs
Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Logz.io.

Autodiscover configuration

Autodiscover allows you to adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.

Add logzio-k8s-logs repo to your helm repo list
helm repo add logzio-helm https://logzio.github.io/logzio-helm
helm repo update
Deploy

In the following commands, make the following changes:

  • Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

  • Replace <<LISTENER-REGION>> with your region’s code (for example, eu). For more information on finding your account’s region, see Account region.

  • Replace <<CLUSTER-NAME>> with your cluster’s name.

This Daemonset’s default autodiscover configuration is hints based. If you wish to deploy it use:

helm install --namespace=kube-system \
--set configType='autodiscover' \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

If you have a custom configuration, deploy with:

helm install --namespace=kube-system \
--set configType='auto-custom' \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
--set-file filebeatConfig.autoCustomConfig=/path/to/your/config.yaml \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

If you’re using a custom config, please make sure that you’re using a .yaml file with the following structure:

filebeat.yml: |-
  filebeat.autodiscover:
  #....
    # your autodiscover config
    # ...
  processors:
    - add_cloud_metadata: ~
  fields:
    logzio_codec: ${LOGZIO_CODEC}
    token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
    cluster: ${CLUSTER_NAME}
    type: ${LOGZIO_TYPE}
  fields_under_root: ${FIELDS_UNDER_ROOT}
  ignore_older: ${IGNORE_OLDER}
  output:
    logstash:
      hosts: ["${LOGZIO_LOGS_LISTENER_HOST}:5015"]
      ssl:
        certificate_authorities: ['/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt']
Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Kibana.

If you still don’t see your logs, see log shipping troubleshooting.

Uninstalling the Chart

The command removes all the k8s components associated with the chart and deletes the release.
To uninstall the logzio-k8s-logs deployment:

helm uninstall --namespace=kube-system logzio-k8s-logs

Changing a default

If you wish to change the default values, specify each parameter using the --set key=value argument to helm install. For example,

helm install --namespace=kube-system logzio-k8s-logs logzio-helm/logzio-k8s-logs \
  --set imageTag=7.7.0 \
  --set terminationGracePeriodSeconds=30

Configurations & defaults

Parameter Description Default
image The Filebeat docker image. docker.elastic.co/beats/filebeat
imageTag The Filebeat docker image tag. 7.8.1
nameOverride Overrides the Chart name for resources. ""
fullnameOverride Overrides the full name of the resources. filebeat
apiVersions.configMap ConfigMap API version. v1
apiVersions.daemonset Daemonset API version. apps/v1
apiVersions.clusterRoleBinding ClusterRoleBinding API version. rbac.authorization.k8s.io/v1
apiVersions.clusterRole ClusterRole API version. rbac.authorization.k8s.io/v1
apiVersions.serviceAccount ServiceAccount API version. v1
apiVersions.secret Secret API version. v1
managedServiceAccount Specifies whether the serviceAccount should be managed by this Helm Chart. Set this to false to manage your own service account and related roles. true
clusterRoleRules Configurable cluster role rules that Filebeat uses to access Kubernetes resources. See values.yaml
logzioCert Logzio public SSL certificate. See values.yaml
configType Specifies which configuration to use for Filebeat. Set to autodiscover to use autodiscover. standard
filebeatConfig.standardConfig Standard Filebeat configuration, using filebeat.input. See values.yaml
filebeatConfig.autodiscoverConfig Autodiscover Filebeat configuration, using filebeat.autodiscover. See values.yaml
filebeatConfig.autoCustomConfig Autodiscover Filebeat custom configuration, using filebeat.autodiscover. Required when you’re using your customized autodiscover config {}
serviceAccount.create Specifies whether a service account should be created. true
serviceAccount.name Name of the service account. filebeat
terminationGracePeriod Termination period (in seconds) to wait before killing Filebeat pod process on pod shutdown. 30
hostNetwork Controls whether the pod may use the node network namespace. true
dnsPolicy Specifies pod-specific DNS policies. ClusterFirstWithHostNet
daemonset.ignoreOlder Logs older than this will be ignored. 3h
daemonset.logzioCodec Set to json if shipping JSON logs. Otherwise, set to plain. json
daemonset.logzioType The log type you’ll use with this Daemonset. This is shown in your logs under the type field in Kibana. Logz.io applies parsing based on type. filebeat
daemonset.fieldsUnderRoot If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. "true"
daemonset.securityContext Configurable securityContext for Filebeat DaemonSet pod execution environment. See values.yaml
daemonset.resources Sets the resources for Filebeat Daemonset. See values.yaml
daemonset.volumes Templatable string of additional volumes to be passed to the DaemonSet. See values.yaml
daemonset.volumeMounts Templatable string of additional volumeMounts to be passed to the DaemonSet. See values.yaml
daemonset.tolerations Set tolerations for all DaemonSet pods. {}
secrets.logzioShippingToken Secret with your logzio shipping token. ""
secrets.logzioRegion Secret with your logzio region. Defaults to US East. " "
secrets.clusterName Secret with your cluster name. ""

Some values, such as daemonset.tolerations, should be set as follows:

--set daemonset.tolerations[0].key='value' \
--set daemonset.tolerations[0].operator='Equal' \

Configuring Filebeat to concatenate multiline logs

Filebeat splits multiline logs by default. If your original logs span multiple lines, you may find that they arrive in your Logz.io account split into several partial logs.

Filebeat offers configuration options that can be used to concatenate multiline logs. The configuration is managed differently, depending on your deployment method:

  • Standard configuration: If you are using a standard configuration (but not autodiscover), use an explicit configuration. Configuration options from Filebeat’s official documentation.

    When using an explicit configuration, you will need to create a single regex expression that covers all of your pods. It also means that Filebeat will need to be reconfigured more often, with the introduction of every new use case.

  • Autodiscover configuration: If you are using autodiscover hints & annotations, add an annotation to your deployment. Configuration options from Filebeat’s official documentation.

    Hints and annotations support the option to manage regex expressions separately for each component. This greatly simplifies the process, making it possible to add a dedicated regex expression to each pod, without needing to change anything on Filebeat itself.

Example

The following is an example of a multiline log sent from a deployment on a k8s cluster:

2021-02-08 09:37:51,031 - errorLogger - ERROR - Traceback (most recent call last):
File "./code.py", line 25, in my_func
1/0
ZeroDivisionError: division by zero

Filebeat’s default configuration will split the above log into 4 logs, 1 for each line of the original log. In other words, each line break (\n) causes a split.

You can overcome this behavior by configuring Filebeat to meet your needs.

Example of an explicit configuration for concatenating multiline logs

To add an explicit configuration to your Filebeat, edit your filebeat.yml file in a text editor and make the appropriate changes under the filebeat.input section.

For the above example, we could use the following regex expression to demarcate the start of our example log. This configuration example is set to identify the first log in a multiline log and concatenate the log lines that follow until it identifies the next log that matches the regex expression. In other words, there is no explicit regex expression to demarcate the end of a multiline log.

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

See Filebeat’s official documentation for additional configuration options.

Example for using hints & annotations to concatenate multiline logs

If you’re using Filebeat autodiscover hints, you can use annotations to identify multiline logs and concatenate them.

You will need to first configure Filebeat to enable the hints system, and add annotations to the relevant components when you deploy them to your cluster.

Enable Filebeat’s hints system

First, enable Filebeat’s hints system. In your filebeat.yml file, set hints.enabled: true under the filebeat.autodiscover section. For example:

filebeat.autodiscover:
	providers:
	  - type: kubernetes
	    node: ${NODE_NAME}
	    hints.enabled: true # This part enables the hints
	    hints.default_config:
	      type: container
	      paths:
	        - /var/log/containers/*-${data.kubernetes.container.id}.log
Add multiline annotations to your deployment

Whenever you plan to deploy a component to your cluster and want the hints system to detect the multiline logs, you’ll need to add multiline annotations.

For the above log example, you can add the following annotations to your deployment:

annotations:
  co.elastic.logs/multiline.type: 'pattern'
  co.elastic.logs/multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  co.elastic.logs/multiline.negate: 'true'
  co.elastic.logs/multiline.match: 'after'

The above configuration ensures that Filebeat will look for log lines that match the regex under multiline.pattern and concatenate all subsequent lines, until it reaches the next regex match.

See Filebeat’s official documentation for additional configuration options.