Linux Operating System
Before you begin, you'll need:
- Root access
Send Linux logs and metrics with OpenTelemetry
For a simpler and more efficient way to collect and send metrics, use the Logz.io telemetry collector.
1. Create a Logz.io directory:
sudo mkdir /opt/logzio-agent
2. Download OpenTelemetry tar.gz:
curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.82.0/otelcol-contrib_0.82.0_linux_amd64.tar.gz" >./otelcol-contrib.tar.gz
3. Extract the OpenTelemetry binary:
sudo tar -zxf ./otelcol-contrib.tar.gz --directory /opt/logzio-agent otelcol-contrib
4. Create the OpenTelemetry config file:
sudo touch /opt/logzio-agent/otel_config.yaml
5. Copy the following into the config file:
Replace <<LOG-SHIPPING-TOKEN>>
, <<LISTENER-HOST>>
, and <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
with the relevant parameters from your Logz.io account.
receivers:
filelog/localhost_linux_system:
include:
- /var/log/*.log
include_file_path: true
operators:
- type: move
from: attributes["log.file.name"]
to: attributes["log_file_name"]
- type: move
from: attributes["log.file.path"]
to: attributes["log_file_path"]
attributes:
type: agent-linux
hostmetrics/localhost_linux_system:
collection_interval: 15s
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
disk:
load:
filesystem:
memory:
metrics:
system.memory.utilization:
enabled: true
network:
paging:
process:
mute_process_name_error: true
mute_process_exe_error: true
mute_process_io_error: true
processors:
resourcedetection/system:
detectors: ["system"]
system:
hostname_sources: ["os"]
filter:
metrics:
include:
match_type: strict
metric_names: ["system.cpu.time", "system.cpu.load_average.1m", "system.cpu.load_average.5m", "system.cpu.load_average.15m", "system.cpu.utilization", "system.memory.usage", "system.memory.utilization", "system.filesystem.usage", "system.disk.io", "system.disk.io_time", "system.disk.operation_time", "system.network.connections", "system.network.io", "system.network.packets", "system.network.errors", "process.cpu.time", "process.memory.usage", "process.disk.io", "process.memory.usage", "process.memory.virtual"]
exporters:
logging:
logzio/logs:
account_token: <<LOG-SHIPPING-TOKEN>>
region: us
headers:
user-agent: logzio-linux-logs
prometheusremotewrite:
endpoint: https://<<LISTENER-HOST>>:8053
headers:
Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
user-agent: logzio-linux-metrics
resource_to_telemetry_conversion:
enabled: true
target_info:
enabled: false
service:
pipelines:
logs:
receivers:
- filelog/localhost_linux_system
processors:
- resourcedetection/system
exporters: [logzio/logs]
metrics:
receivers:
- hostmetrics/localhost_linux_system
processors:
- resourcedetection/system
- filter
exporters: [prometheusremotewrite]
telemetry:
logs:
level: "info"
metrics:
address: localhost:8888
If OpenTelemetry metrics are already running on port 8888, edit the address
field in the config file.
6. Create the service file:
sudo touch /etc/systemd/system/logzioOTELCollector.service
And copy the service file's content:
[Unit]
Description=OTEL collector for collecting logs/metrics and exporting them to Logz.io.
[Service]
ExecStart=/opt/logzio-agent/otelcol-contrib --config /opt/logzio-agent/otel_config.yaml
[Install]
WantedBy=multi-user.target
Manage your OpenTelemetry on Localhost
Manage OpenTelemetry on your machine using the following commands:
Description | Command |
---|---|
Start service | sudo systemctl start logzioOTELCollector |
Stop service | sudo systemctl stop logzioOTELCollector |
Service logs | sudo systemctl status -l logzioOTELCollector |
Delete service | sudo systemctl stop logzioOTELCollector sudo systemctl reset-failed logzioOTELCollector 2>/dev/null sudo rm /etc/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /usr/lib/systemd/system/logzioOTELCollector.service 2>/dev/null sudo rm /etc/init.d/logzioOTELCollector 2>/dev/null |
Send data through rsyslog
Before you begin, you'll need:
- Root access
- Port 5000 open
Run the rsyslog configuration script
Replace <<LOG-SHIPPING-TOKEN>>
with the token of the account you want to ship to.
Replace <<LISTENER-HOST>>
with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.
curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz \
&& tar xzf logzio-rsyslog.tar.gz \
&& sudo rsyslog/install.sh -t linux -a "<<LOG-SHIPPING-TOKEN>>" -l "<<LISTENER-HOST>>"
The above assumes the following defaults:
- Log location -
/var/log/
- Log type -
syslog
Check Logz.io for your logs
Allow some time for data ingestion, then open your metrics dashboard.
Encounter an issue? See our log shipping troubleshooting guide.
Troubleshooting
This section provides guidelines for handling errors when collecting logs for Rsyslog with SELinux configuration.
SELinux is a Linux feature for implementing access control security policies. In distributions like Fedora and RHEL, SELinux is enabled in Enforcing mode by default.
Rsyslog, a system process protected by SELinux, is restricted by default to sending data only to port 514/udp (the standard syslog port) and has limited access to files and directories beyond its initial configuration.
To send data to Logz.io in a SELinux environment, you need to add exceptions to allow:
- rsyslog to communicate with logz.io through the desired port.
- rsyslog to access the necessary files and directories.
Issue not related to SELinux
The issue may not be caused by SELinux.
Suggested remedy
Disable SELinux temporarily and see if that solves the problem.
Run the following command to check the current status of SELinux:
$ getenforce
SElinux's status can be in any of the following states:
- Enforcing: SELinux is active and blocking the actions that do not match the policy
- Permissive: SELinux is active but is not blocking the actions that do not match the policy -- it only leaves logs indicating which actions had been performed
- Disable: SELinux is disabled
If SELinux is not in Enforcing mode, no other action is needed because it is not blocking communication to Logz.io
If SELinux is Enforced, try to disable it temporally and then restart rsyslog:
$ sudo setenforce 0
$ sudo service rsyslog restart
Check if rsyslog is working and that you see the logs in you account.
To re-enable SELinux, run:
$ sudo setenforce 1
$ sudo service rsyslog restart
The above command only disables SELinux temporarily. To disable it completely, you will have to edit its configuration file. Although from a security point of view it's not recommended, if you want the changes to be permanent, edit the /etc/selinux/config file and restart the machine:
SELINUX=disabled
SELINUX=permissive
Need to add exceptions
You may need to add exception to SELinux configuration to enable Logz.io.
Suggested remedy
- Install the policycoreutils and the setroubleshoot packages:
# Installing policycoreutils & setroubleshoot packages
$ sudo yum install policycoreutils setroubleshoot
- Check which syslog ports are allowed by SELinux:
$ sudo semanage port -l| grep syslog
output:
syslogd_port_t udp 514
- Add a new port to policy for Logz.io:
# Adding a port to SELinux policies
$ sudo semanage port -m -t syslogd_port_t -p tcp 5000
- Authorize Rsyslog directory:
# instructing se to authorize the /var/spool/rsyslog directory
$ sudo semanage fcontext -a -t syslogd_var_lib_t "/var/spool/rsyslog/*"
$ sudo restorecon -R -v /var/spool/rsyslog
- Depending on the distribution, run the following command:
# instructing se to authorize /etc/rsyslog.d/*
$ sudo semanage fcontext -a -t syslog_conf_t "/etc/rsyslog.d/"
$ sudo restorecon -R -v /etc/rsyslog.d/
$ sudo semanage fcontext -a -t etc_t "/etc/rsyslog.d"
$ sudo restorecon -v /etc/rsyslog.d
- Restart Rsyslog:
$ sudo service rsyslog restart