Linux Operating System

Before you begin, you'll need:

• Root access

Send Linux logs and metrics with OpenTelemetry

note

For a simpler and more efficient way to collect and send metrics, use the Logz.io telemetry collector.

1. Create a Logz.io directory:

sudo mkdir /opt/logzio-agent

2. Download OpenTelemetry tar.gz:

curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.111.0/otelcol-contrib_0.111.0_linux_amd64.tar.gz" >./otelcol-contrib.tar.gz

3. Extract the OpenTelemetry binary:

sudo tar -zxf ./otelcol-contrib.tar.gz --directory /opt/logzio-agent otelcol-contrib

4. Create the OpenTelemetry config file:

sudo touch /opt/logzio-agent/otel_config.yaml

5. Copy the following into the config file:

Replace <<LOG-SHIPPING-TOKEN>>, <<LISTENER-HOST>>, and <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with the relevant parameters from your Logz.io account.

receivers:
  filelog/localhost_linux_system:
    include:
      - /var/log/*.log
    include_file_path: true
    operators:
      - type: move
        from: attributes["log.file.name"]
        to: attributes["log_file_name"]
      - type: move
        from: attributes["log.file.path"]
        to: attributes["log_file_path"]
    attributes:
      type: agent-linux
  hostmetrics/localhost_linux_system:
    collection_interval: 15s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      load:
      filesystem:
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network:
      paging:
      process:
        mute_process_name_error: true
        mute_process_exe_error: true
        mute_process_io_error: true
processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
  filter:
    metrics:
      include:
        match_type: strict
        metric_names: ["system.cpu.time", "system.cpu.load_average.1m", "system.cpu.load_average.5m", "system.cpu.load_average.15m", "system.cpu.utilization", "system.memory.usage", "system.memory.utilization", "system.filesystem.usage", "system.disk.io", "system.disk.io_time", "system.disk.operation_time", "system.network.connections", "system.network.io", "system.network.packets", "system.network.errors", "process.cpu.time", "process.memory.usage", "process.disk.io", "process.memory.usage", "process.memory.virtual"]
exporters:
  debug:
  logzio/logs:
    account_token: <<LOG-SHIPPING-TOKEN>>
    region: <<LOGZIO_ACCOUNT_REGION_CODE>> # Default is US
    headers:
      user-agent: logzio-linux-logs
  prometheusremotewrite:
    endpoint: https://<<LISTENER-HOST>>:8053
    headers:
      Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
      user-agent: logzio-linux-metrics
    resource_to_telemetry_conversion:
      enabled: true
    target_info:
      enabled: false
service:
  pipelines:
    logs:
      receivers:
        - filelog/localhost_linux_system
      processors:
        - resourcedetection/system
      exporters: [logzio/logs]
    metrics:
      receivers:
        - hostmetrics/localhost_linux_system
      processors:
        - resourcedetection/system
        - filter
      exporters: [prometheusremotewrite]
  telemetry:
    logs:
      level: "info"
    metrics:
      address: localhost:8888

note

Ensure that your service pipeline includes the debug exporter in the exporters section. See the OpenTelemetry Debug Exporter documentation for more details.

Important

If OpenTelemetry metrics are already running on port 8888, edit the address field in the config file.

6. Create the service file:

sudo touch /etc/systemd/system/logzioOTELCollector.service

And copy the service file's content:

[Unit]

Description=OTEL collector for collecting logs/metrics and exporting them to Logz.io.

[Service]

ExecStart=/opt/logzio-agent/otelcol-contrib --config /opt/logzio-agent/otel_config.yaml

[Install]

WantedBy=multi-user.target

Optional - Add basic filtering

You can use the filter processor to remove telemetry data based on OTTL (OpenTelemetry Transformation Language) rules. Any telemetry that meets at least one of the specified conditions will be dropped.

For example, to allow span data from services app1, app2, and app3 and drop data from all other services:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - |
        resource.attributes["service.name"] != "app1" and
        resource.attributes["service.name"] != "app2" and
        resource.attributes["service.name"] != "app3"
    metrics:
      datapoint:
        - metric.name == "k8s.pod.phase" and value_int == 4
    logs:
      log_record:
        - 'IsMatch(body, ".*password.*")'
        - 'severity_number < SEVERITY_NUMBER_WARN'

To only drop spans from a service called service1 while keeping all other spans:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - resource.attributes["service.name"] == "service1"

Learn more about filtering options in OpenTelemetry's Transforming telemetry documentation.

Manage your OpenTelemetry on Localhost

Manage OpenTelemetry on your machine using the following commands:

Description	Command
Start service	sudo systemctl start logzioOTELCollector
Stop service	sudo systemctl stop logzioOTELCollector
Service logs	sudo systemctl status -l logzioOTELCollector
Delete service	sudo systemctl stop logzioOTELCollector sudo systemctl reset-failed logzioOTELCollector 2>/dev/null sudo rm /etc/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /usr/lib/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /etc/init.d/logzioOTELCollector 2>/dev/null

Send data through rsyslog

Before you begin, you'll need:

• Root access
• Port 5000 open

Run the rsyslog configuration script

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz \
  && tar xzf logzio-rsyslog.tar.gz \
  && sudo rsyslog/install.sh -t linux -a "<<LOG-SHIPPING-TOKEN>>" -l "<<LISTENER-HOST>>"

The above assumes the following defaults:

• Log location - /var/log/
• Log type - syslog

Check Logz.io for your logs

Allow some time for data ingestion, then open your metrics dashboard.

Encounter an issue? See our log shipping troubleshooting guide.

Troubleshooting

This section provides guidelines for handling errors when collecting logs for Rsyslog with SELinux configuration.

SELinux is a Linux feature for implementing access control security policies. In distributions like Fedora and RHEL, SELinux is enabled in Enforcing mode by default.

Rsyslog, a system process protected by SELinux, is restricted by default to sending data only to port 514/udp (the standard syslog port) and has limited access to files and directories beyond its initial configuration.

To send data to Logz.io in a SELinux environment, you need to add exceptions to allow:

• rsyslog to communicate with logz.io through the desired port.
• rsyslog to access the necessary files and directories.

Issue not related to SELinux

The issue may not be caused by SELinux.

Suggested remedy

Disable SELinux temporarily and see if that solves the problem.

Run the following command to check the current status of SELinux:

$ getenforce

SElinux's status can be in any of the following states:

• Enforcing: SELinux is active and blocking the actions that do not match the policy
• Permissive: SELinux is active but is not blocking the actions that do not match the policy -- it only leaves logs indicating which actions had been performed
• Disable: SELinux is disabled

If SELinux is not in Enforcing mode, no other action is needed because it is not blocking communication to Logz.io

If SELinux is Enforced, try to disable it temporally and then restart rsyslog:

$ sudo setenforce 0
$ sudo service rsyslog restart

Check if rsyslog is working and that you see the logs in you account.

To re-enable SELinux, run:

$ sudo setenforce 1
$ sudo service rsyslog restart

The above command only disables SELinux temporarily. To disable it completely, you will have to edit its configuration file. Although from a security point of view it's not recommended, if you want the changes to be permanent, edit the /etc/selinux/config file and restart the machine:

SELINUX=disabled 
SELINUX=permissive 

Need to add exceptions

You may need to add exception to SELinux configuration to enable Logz.io.

Suggested remedy

1. Install the policycoreutils and the setroubleshoot packages:

# Installing policycoreutils & setroubleshoot packages
$ sudo yum install policycoreutils setroubleshoot

2. Check which syslog ports are allowed by SELinux:

$ sudo semanage port -l| grep syslog

output:
syslogd_port_t udp 514

3. Add a new port to policy for Logz.io:

# Adding a port to SELinux policies
$ sudo semanage port -m -t syslogd_port_t -p tcp 5000

4. Authorize Rsyslog directory:

# instructing se to authorize the /var/spool/rsyslog directory
$ sudo semanage fcontext -a -t syslogd_var_lib_t "/var/spool/rsyslog/*"
$ sudo restorecon -R -v /var/spool/rsyslog

5. Depending on the distribution, run the following command:

# instructing se to authorize /etc/rsyslog.d/*
$ sudo semanage fcontext -a -t syslog_conf_t "/etc/rsyslog.d/"

$ sudo restorecon -R -v /etc/rsyslog.d/

$ sudo semanage fcontext -a -t etc_t "/etc/rsyslog.d"

$ sudo restorecon -v /etc/rsyslog.d

6. Restart Rsyslog:

$ sudo service rsyslog restart