Linux Operating System

Before you begin, you'll need:

• Root access

Send Linux logs and metrics with OpenTelemetry

note

For a simpler and more efficient way to collect and send metrics, use the Logz.io telemetry collector.

1. Create a Logz.io directory:

sudo mkdir /opt/logzio-agent

2. Download OpenTelemetry tar.gz:

For amd64 architecture:

curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.138.0/otelcol-contrib_0.138.0_linux_amd64.tar.gz" >./otelcol-contrib.tar.gz

For arm64 architecture:

curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.138.0/otelcol-contrib_0.138.0_linux_arm64.tar.gz" >./otelcol-contrib.tar.gz

For s390x architecture:

curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.138.0/otelcol-contrib_0.138.0_linux_s390x.tar.gz" >./otelcol-contrib.tar.gz

3. Extract the OpenTelemetry binary:

sudo tar -zxf ./otelcol-contrib.tar.gz --directory /opt/logzio-agent otelcol-contrib

4. Create the OpenTelemetry config file:

sudo touch /opt/logzio-agent/otel_config.yaml

5. Copy the following into the config file:

Replace <<LOG-SHIPPING-TOKEN>>, <<LISTENER-HOST>>, and <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with the relevant parameters from your Logz.io account.

receivers:
  filelog/localhost_linux_system:
    include:
      - /var/log/*.log
    include_file_path: true
    operators:
      - type: move
        from: attributes["log.file.name"]
        to: attributes["log_file_name"]
      - type: move
        from: attributes["log.file.path"]
        to: attributes["log_file_path"]
    attributes:
      type: agent-linux
  hostmetrics/localhost_linux_system:
    collection_interval: 15s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      load:
      filesystem:
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network:
      paging:
      process:
        mute_process_name_error: true
        mute_process_exe_error: true
        mute_process_io_error: true
processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
  filter:
    metrics:
      include:
        match_type: strict
        metric_names: ["system.cpu.time", "system.cpu.load_average.1m", "system.cpu.load_average.5m", "system.cpu.load_average.15m", "system.cpu.utilization", "system.memory.usage", "system.memory.utilization", "system.filesystem.usage", "system.disk.io", "system.disk.io_time", "system.disk.operation_time", "system.network.connections", "system.network.io", "system.network.packets", "system.network.errors", "process.cpu.time", "process.memory.usage", "process.disk.io", "process.memory.usage", "process.memory.virtual"]
exporters:
  debug:
  logzio/logs:
    account_token: <<LOG-SHIPPING-TOKEN>>
    region: <<LOGZIO_ACCOUNT_REGION_CODE>> # Default is US
    headers:
      user-agent: logzio-linux-logs
  prometheusremotewrite:
    endpoint: https://<<LISTENER-HOST>>:8053
    headers:
      Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
      user-agent: logzio-linux-metrics
    resource_to_telemetry_conversion:
      enabled: true
    target_info:
      enabled: false
    add_metric_suffixes: false

service:
  pipelines:
    logs:
      receivers:
        - filelog/localhost_linux_system
      processors:
        - resourcedetection/system
      exporters: [logzio/logs]
    metrics:
      receivers:
        - hostmetrics/localhost_linux_system
      processors:
        - resourcedetection/system
        - filter
      exporters: [prometheusremotewrite]
  telemetry:
    logs:
      level: "info"
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '0.0.0.0'
                port: 8888

note

Ensure that your service pipeline includes the debug exporter in the exporters section. See the OpenTelemetry Debug Exporter documentation for more details.

Important

If OpenTelemetry metrics are already running on port 8888, edit the address field in the config file.

6. Create the service file:

sudo touch /etc/systemd/system/logzioOTELCollector.service

And copy the service file's content:

[Unit]

Description=OTEL collector for collecting logs/metrics and exporting them to Logz.io.

[Service]

ExecStart=/opt/logzio-agent/otelcol-contrib --config /opt/logzio-agent/otel_config.yaml

[Install]

WantedBy=multi-user.target

Optional - Add basic filtering

You can use the filter processor to remove telemetry data based on OTTL (OpenTelemetry Transformation Language) rules. Any telemetry that meets at least one of the specified conditions will be dropped.

For example, to allow span data from services app1, app2, and app3 and drop data from all other services:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - |
        resource.attributes["service.name"] != "app1" and
        resource.attributes["service.name"] != "app2" and
        resource.attributes["service.name"] != "app3"
    metrics:
      datapoint:
        - metric.name == "k8s.pod.phase" and value_int == 4
    logs:
      log_record:
        - 'IsMatch(body, ".*password.*")'
        - 'severity_number < SEVERITY_NUMBER_WARN'

To only drop spans from a service called service1 while keeping all other spans:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - resource.attributes["service.name"] == "service1"

Learn more about filtering options in OpenTelemetry's Transforming telemetry documentation.

Manage your OpenTelemetry on Localhost

Manage OpenTelemetry on your machine using the following commands:

Description	Command
Start service	sudo systemctl start logzioOTELCollector
Stop service	sudo systemctl stop logzioOTELCollector
Service logs	sudo systemctl status -l logzioOTELCollector
Delete service #1	sudo systemctl stop logzioOTELCollector
Delete service #2	sudo systemctl reset-failed logzioOTELCollector 2>/dev/null
Delete service #3	sudo rm /etc/systemd/system/logzioOTELCollector.service 2>/dev/null
Delete service #4	sudo rm /usr/lib/systemd/system/logzioOTELCollector.service 2>/dev/null
Delete service #5	sudo rm /etc/init.d/logzioOTELCollector 2>/dev/null