Skip to main content

Linux Operating System

Before you begin, you'll need:

  • Root access

Send Linux logs and metrics with OpenTelemetry

note

For a simpler and more efficient way to collect and send metrics, use the Logz.io telemetry collector.

1. Create a Logz.io directory:

sudo mkdir /opt/logzio-agent

2. Download OpenTelemetry tar.gz:

curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.82.0/otelcol-contrib_0.82.0_linux_amd64.tar.gz" >./otelcol-contrib.tar.gz

3. Extract the OpenTelemetry binary:

sudo tar -zxf ./otelcol-contrib.tar.gz --directory /opt/logzio-agent otelcol-contrib

4. Create the OpenTelemetry config file:

sudo touch /opt/logzio-agent/otel_config.yaml

5. Copy the following into the config file:

Replace <<LOG-SHIPPING-TOKEN>>, <<LISTENER-HOST>>, and <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with the relevant parameters from your Logz.io account.

receivers:
filelog/localhost_linux_system:
include:
- /var/log/*.log
include_file_path: true
operators:
- type: move
from: attributes["log.file.name"]
to: attributes["log_file_name"]
- type: move
from: attributes["log.file.path"]
to: attributes["log_file_path"]
attributes:
type: agent-linux
hostmetrics/localhost_linux_system:
collection_interval: 15s
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
disk:
load:
filesystem:
memory:
metrics:
system.memory.utilization:
enabled: true
network:
paging:
process:
mute_process_name_error: true
mute_process_exe_error: true
mute_process_io_error: true
processors:
resourcedetection/system:
detectors: ["system"]
system:
hostname_sources: ["os"]
filter:
metrics:
include:
match_type: strict
metric_names: ["system.cpu.time", "system.cpu.load_average.1m", "system.cpu.load_average.5m", "system.cpu.load_average.15m", "system.cpu.utilization", "system.memory.usage", "system.memory.utilization", "system.filesystem.usage", "system.disk.io", "system.disk.io_time", "system.disk.operation_time", "system.network.connections", "system.network.io", "system.network.packets", "system.network.errors", "process.cpu.time", "process.memory.usage", "process.disk.io", "process.memory.usage", "process.memory.virtual"]
exporters:
logging:
logzio/logs:
account_token: <<LOG-SHIPPING-TOKEN>>
region: us
headers:
user-agent: logzio-linux-logs
prometheusremotewrite:
endpoint: https://<<LISTENER-HOST>>:8053
headers:
Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
user-agent: logzio-linux-metrics
resource_to_telemetry_conversion:
enabled: true
target_info:
enabled: false
service:
pipelines:
logs:
receivers:
- filelog/localhost_linux_system
processors:
- resourcedetection/system
exporters: [logzio/logs]
metrics:
receivers:
- hostmetrics/localhost_linux_system
processors:
- resourcedetection/system
- filter
exporters: [prometheusremotewrite]
telemetry:
logs:
level: "info"
metrics:
address: localhost:8888
Important

If OpenTelemetry metrics are already running on port 8888, edit the address field in the config file.

6. Create the service file:

sudo touch /etc/systemd/system/logzioOTELCollector.service

And copy the service file's content:

[Unit]

Description=OTEL collector for collecting logs/metrics and exporting them to Logz.io.

[Service]

ExecStart=/opt/logzio-agent/otelcol-contrib --config /opt/logzio-agent/otel_config.yaml

[Install]

WantedBy=multi-user.target

Manage your OpenTelemetry on Localhost

Manage OpenTelemetry on your machine using the following commands:

DescriptionCommand
Start servicesudo systemctl start logzioOTELCollector
Stop servicesudo systemctl stop logzioOTELCollector
Service logssudo systemctl status -l logzioOTELCollector
Delete servicesudo systemctl stop logzioOTELCollector sudo systemctl reset-failed logzioOTELCollector 2>/dev/null sudo rm /etc/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /usr/lib/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /etc/init.d/logzioOTELCollector 2>/dev/null

Send data through rsyslog

Before you begin, you'll need:

  • Root access
  • Port 5000 open

Run the rsyslog configuration script

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz \
&& tar xzf logzio-rsyslog.tar.gz \
&& sudo rsyslog/install.sh -t linux -a "<<LOG-SHIPPING-TOKEN>>" -l "<<LISTENER-HOST>>"

The above assumes the following defaults:

  • Log location - /var/log/
  • Log type - syslog

Check Logz.io for your logs

Allow some time for data ingestion, then open your metrics dashboard.

Encounter an issue? See our log shipping troubleshooting guide.

Troubleshooting

This section provides guidelines for handling errors when collecting logs for Rsyslog with SELinux configuration.

SELinux is a Linux feature for implementing access control security policies. In distributions like Fedora and RHEL, SELinux is enabled in Enforcing mode by default.

Rsyslog, a system process protected by SELinux, is restricted by default to sending data only to port 514/udp (the standard syslog port) and has limited access to files and directories beyond its initial configuration.

To send data to Logz.io in a SELinux environment, you need to add exceptions to allow:

  • rsyslog to communicate with logz.io through the desired port.
  • rsyslog to access the necessary files and directories.

The issue may not be caused by SELinux.

Suggested remedy

Disable SELinux temporarily and see if that solves the problem.

Run the following command to check the current status of SELinux:

$ getenforce

SElinux's status can be in any of the following states:

  • Enforcing: SELinux is active and blocking the actions that do not match the policy
  • Permissive: SELinux is active but is not blocking the actions that do not match the policy -- it only leaves logs indicating which actions had been performed
  • Disable: SELinux is disabled

If SELinux is not in Enforcing mode, no other action is needed because it is not blocking communication to Logz.io

If SELinux is Enforced, try to disable it temporally and then restart rsyslog:

$ sudo setenforce 0
$ sudo service rsyslog restart

Check if rsyslog is working and that you see the logs in you account.

To re-enable SELinux, run:

$ sudo setenforce 1
$ sudo service rsyslog restart

The above command only disables SELinux temporarily. To disable it completely, you will have to edit its configuration file. Although from a security point of view it's not recommended, if you want the changes to be permanent, edit the /etc/selinux/config file and restart the machine:

SELINUX=disabled 
SELINUX=permissive

Need to add exceptions

You may need to add exception to SELinux configuration to enable Logz.io.

Suggested remedy

  1. Install the policycoreutils and the setroubleshoot packages:
# Installing policycoreutils & setroubleshoot packages
$ sudo yum install policycoreutils setroubleshoot
  1. Check which syslog ports are allowed by SELinux:
$ sudo semanage port -l| grep syslog

output:
syslogd_port_t udp 514
  1. Add a new port to policy for Logz.io:
# Adding a port to SELinux policies
$ sudo semanage port -m -t syslogd_port_t -p tcp 5000
  1. Authorize Rsyslog directory:
# instructing se to authorize the /var/spool/rsyslog directory
$ sudo semanage fcontext -a -t syslogd_var_lib_t "/var/spool/rsyslog/*"
$ sudo restorecon -R -v /var/spool/rsyslog
  1. Depending on the distribution, run the following command:
# instructing se to authorize /etc/rsyslog.d/*
$ sudo semanage fcontext -a -t syslog_conf_t "/etc/rsyslog.d/"

$ sudo restorecon -R -v /etc/rsyslog.d/

$ sudo semanage fcontext -a -t etc_t "/etc/rsyslog.d"

$ sudo restorecon -v /etc/rsyslog.d
  1. Restart Rsyslog:
$ sudo service rsyslog restart