Skip to main content

Linux Operating System

Before you begin, you'll need:

  • Root access

Send your Linux machine logs and metrics using OpenTelemetry service​

note

For a much easier and more efficient way to collect and send metrics, consider using the Logz.io telemetry collector.

Create a Logz.io directory: 

sudo mkdir /opt/logzio-agent

Download OpenTelemetry tar.gz: 

curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.82.0/otelcol-contrib_0.82.0_linux_amd64.tar.gz" >./otelcol-contrib.tar.gz

Extract the OpenTelemetry binary:

sudo tar -zxf ./otelcol-contrib.tar.gz --directory /opt/logzio-agent otelcol-contrib

Create the OpenTelemetry config file:

sudo touch /opt/logzio-agent/otel_config.yaml

And copy the following OpenTelemetry content into the config file.

Replace <<LOG-SHIPPING-TOKEN>>, <<LISTENER-HOST>>, and <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with the relevant parameters from your Logz.io account.

receivers:
  filelog/localhost_linux_system:
    include:
      - /var/log/*.log
    include_file_path: true
    operators:
      - type: move
        from: attributes["log.file.name"]
        to: attributes["log_file_name"]
      - type: move
        from: attributes["log.file.path"]
        to: attributes["log_file_path"]
    attributes:
      type: agent-linux
  hostmetrics/localhost_linux_system:
    collection_interval: 15s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      load:
      filesystem:
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network:
      paging:
      process:
        mute_process_name_error: true
        mute_process_exe_error: true
        mute_process_io_error: true
processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
  filter:
    metrics:
      include:
        match_type: strict
        metric_names: ["system.cpu.time", "system.cpu.load_average.1m", "system.cpu.load_average.5m", "system.cpu.load_average.15m", "system.cpu.utilization", "system.memory.usage", "system.memory.utilization", "system.filesystem.usage", "system.disk.io", "system.disk.io_time", "system.disk.operation_time", "system.network.connections", "system.network.io", "system.network.packets", "system.network.errors", "process.cpu.time", "process.memory.usage", "process.disk.io", "process.memory.usage", "process.memory.virtual"]
exporters:
  logging:
  logzio/logs:
    account_token: <<LOG-SHIPPING-TOKEN>>
    region: us
  prometheusremotewrite:
    endpoint: https:<<LISTENER-HOST>>:8053
    headers:
      Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
    resource_to_telemetry_conversion:
      enabled: true
    target_info:
        enabled: false
service:
  pipelines:
    logs:
      receivers:
        - filelog/localhost_linux_system
      processors:
        - resourcedetection/system
      exporters: [logzio/logs]
    metrics:
      receivers:
        - hostmetrics/localhost_linux_system
      processors:
        - resourcedetection/system
        - filter
      exporters: [prometheusremotewrite]
  telemetry:
    logs:
      level: "info"
    metrics:
      address: localhost:8888
Important

If you already running OpenTelemetry metrics on port 8888, you will need to edit the address field in the config file.

Next, create the service file:

sudo touch /etc/systemd/system/logzioOTELCollector.service

And copy the service file's content:

[Unit]

Description=OTEL collector for collecting logs/metrics and exporting them to Logz.io.

[Service]

ExecStart=/opt/logzio-agent/otelcol-contrib --config /opt/logzio-agent/otel_config.yaml

[Install]

WantedBy=multi-user.target

Manage your OpenTelemetry on Localhost​

DescriptionCommand
Start servicesudo systemctl start logzioOTELCollector
Stop servicesudo systemctl stop logzioOTELCollector
Service logssudo systemctl status -l logzioOTELCollector
Delete servicesudo systemctl stop logzioOTELCollector sudo systemctl reset-failed logzioOTELCollector 2>/dev/null sudo rm /etc/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /usr/lib/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /etc/init.d/logzioOTELCollector 2>/dev/null

Send your logs to Logz.io through rsyslog​

Before you begin, you'll need:

  • Root access
  • Port 5000 open

Run the rsyslog configuration script​

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-HOST>> with the host for your region. For example, listener.logz.io if your account is hosted on AWS US East, or listener-nl.logz.io if hosted on Azure West Europe. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071. 

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz \
  && tar xzf logzio-rsyslog.tar.gz \
  && sudo rsyslog/install.sh -t linux -a "<<LOG-SHIPPING-TOKEN>>" -l "<<LISTENER-HOST>>"

The above assumes the following defaults:

  • Log location - /var/log/
  • Log type - syslog

Check Logz.io for your logs​

Give your logs some time to get from your system to ours, and then open Open Search Dashboards. You can search for type:syslog to filter for your logs.

If you still don't see your logs, see log shipping troubleshooting.

This section contains some guidelines for handling errors that you may encounter when trying to collect logs for Rsyslog - SELinux configuration.

SELinux is a Linux feature that allows you to implement access control security policies in Linux systems. In distributions such as Fedora and RHEL, SELinux is in Enforcing mode by default.

Rsyslog is one of the system processes protected by SELinux. This means that rsyslog by default is not allowed to send to a port other than 514/udp (the standard syslog port) has limited access to other files and directories outside of their initial configurations.

To send information to Logz.io properly in a SELinux environment, it is necessary to add exceptions to allow:

  • rsyslog to communicate with logz.io through the desired port
  • rsyslog to access the files and directories needed for it to work properly
Possible cause - issue not related to SELinux​

The issue may not be caused by SELinux.

Suggested remedy​

Disable SELinux temporarily and see if that solves the problem.

Run the following command to check the current status of SELinux:

$ getenforce

SElinux's status can be in any of the following states:

  • Enforcing: SELinux is active and blocking the actions that do not match the policy
  • Permissive: SELinux is active but is not blocking the actions that do not match the policy -- it only leaves logs indicating which actions had been performed
  • Disable: SELinux is disabled

If SELinux is not in Enforcing mode, no other action is needed because it is not blocking communication to Logz.io

If SELinux is Enforced, try to disable it temporally and then restart rsyslog:

$ sudo setenforce 0
$ sudo service rsyslog restart

Check if rsyslog is working and that you see the logs in you account.

To re-enable SELinux, run: 

$ sudo setenforce 1
$ sudo service rsyslog restart

The above command only disables SELinux temporarily. To disable it completely, you will have to edit its configuration file. Although from a security point of view it's not recommended, if you want the changes to be permanent, edit the /etc/selinux/config file and restart the machine:

SELINUX=disabled 
SELINUX=permissive
Possible cause - need exceptions to SELinux for Logz.io​

You may need to add exception to SELinux configuration to enable Logz.io.

Suggested remedy​
Install the policycoreutils and the setroubleshoot packages​
# Installing policycoreutils & setroubleshoot packages
$ sudo yum install policycoreutils setroubleshoot
Check which syslog ports are allowed by SELinux​

Run the command as in the example below:

$ sudo semanage port -l| grep syslog

output:
syslogd_port_t udp 514
Add a new port to policy for Logz.io​
# Adding a port to SELinux policies
$ sudo semanage port -m -t syslogd_port_t -p tcp 5000
Authorize Rsyslog directory​
# instructing se to authorize the /var/spool/rsyslog directory
$ sudo semanage fcontext -a -t syslogd_var_lib_t "/var/spool/rsyslog/*"
$ sudo restorecon -R -v /var/spool/rsyslog

Depending on the distribution, run the following command:

# instructing se to authorize /etc/rsyslog.d/*
$ sudo semanage fcontext -a -t syslog_conf_t "/etc/rsyslog.d/"

$ sudo restorecon -R -v /etc/rsyslog.d/

$ sudo semanage fcontext -a -t etc_t "/etc/rsyslog.d"

$ sudo restorecon -v /etc/rsyslog.d
Restart Rsyslog​
$ sudo service rsyslog restart