For Kubernetes, a DaemonSet ensures that some or all nodes run a copy of a pod. This implementation is uses a Fluentd DaemonSet to collect Kubernetes logs. Fluentd is flexible enough and has the proper plugins to distribute logs to different third parties such as Logz.io.

The logzio-k8s image comes pre-configured for Fluentd to gather all logs from the Kubernetes node environment and append the proper metadata to the logs.

Deploy logzio-k8s with default configuration

For most environments, we recommend using the default configuration. However, you can deploy a custom configuration if your environment needs it.

To deploy logzio-k8s
  1. Store your Logz.io credentials

    Save your Logz.io shipping credentials as a Kubernetes secret.

    Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to.

    Replace <<LISTENER-HOST>> with your region’s listener host (for example, listener.logz.io). For more information on finding your account’s region, see Account region.

    kubectl create secret generic logzio-logs-secret \
      --from-literal=logzio-log-shipping-token='<<SHIPPING-TOKEN>>' \
      --from-literal=logzio-log-listener='https://<<LISTENER-HOST>>:8071' \
      -n kube-system
    
  2. Deploy the DaemonSet

    kubectl apply -f https://raw.githubusercontent.com/logzio/logzio-k8s/master/logzio-daemonset-rbac.yaml
    
  3. Check Logz.io for your logs

    Give your logs some time to get from your system to ours, and then open Kibana.

    If you still don’t see your logs, see log shipping troubleshooting.

Deploy logzio-k8s with custom configuration

You can customize the configuration of the Fluentd container. This is done using a ConfigMap that overwrites the default DaemonSet.

To deploy logzio-k8s
  1. Store your Logz.io credentials

    Save your Logz.io shipping credentials as a Kubernetes secret.

    Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to.

    Replace <<LISTENER-HOST>> with your region’s listener host (for example, listener.logz.io). For more information on finding your account’s region, see Account region.

    kubectl create secret generic logzio-logs-secret \
      --from-literal=logzio-log-shipping-token='<<SHIPPING-TOKEN>>' \
      --from-literal=logzio-log-listener='https://<<LISTENER-HOST>>:8071' \
      -n kube-system
    
  2. Configure Fluentd

    Download either the RBAC DaemonSet or the non-RBAC DaemonSet and open the file in your text editor.

    Customize the Fluentd configuration with the parameters shown below. The Fluentd configuration is below the fluent.conf: |- line, at the bottom of the file.

    Parameters

    output_include_time true
    To append a timestamp to your logs when they’re processed, true. Otherwise, false.
    buffer_type file
    Specifies which plugin to use as the backend.
    buffer_path /var/log/Fluentd-buffers/stackdriver.buffer
    Path of the buffer.
    buffer_queue_full_action block
    Controls the behavior when the queue becomes full.
    buffer_chunk_limit 2M
    Maximum size of a chunk allowed
    buffer_queue_limit 6
    Maximum length of the output queue.
    flush_interval 5s
    Interval, in seconds, to wait before invoking the next buffer flush.
    max_retry_wait 30s
    Maximum interval, in seconds, to wait between retries.
    num_threads 2
    Number of threads to flush the buffer.
  3. Deploy the DaemonSet

    For the RBAC DaemonSet:

    kubectl apply -f /path/to/logzio-daemonset-rbac.yaml
    

    For the non-RBAC DaemonSet:

    kubectl apply -f /path/to/logzio-daemonset.yaml
    
  4. Check Logz.io for your logs

    Give your logs some time to get from your system to ours, and then open Kibana.

    If you still don’t see your logs, see log shipping troubleshooting.

Disabling systemd input

To suppress Fluentd system messages, set the FLUENTD_SYSTEMD_CONF environment variable to disable in your Kubernetes environment.