Fluent Bit

Run Fluent Bit as a standalone app

Fluent Bit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources. This integration allows you to send logs from Fluent Bit running as a standalone app and forward them to your Logz.io account.

note

Project's GitHub repo

Install Fluent Bit

If you haven't installed Fluent Bit yet, you can build it from source according to the instructions from Fluent Bit.

Install and configure the Logz.io plugin

For Linux:

wget -O /fluent-bit/plugins/out_logzio.so \
https://github.com/logzio/fluent-bit-logzio-output/raw/master/build/out_logzio-linux.so

For MacOS:

wget -O /fluent-bit/plugins/out_logzio.so \
    https://github.com/logzio/fluent-bit-logzio-output/raw/master/build/out_logzio-macOS.so

For Windows:

wget https://github.com/logzio/fluent-bit-logzio-output/raw/master/build/out_logzio-windows.so

In your Fluent Bit configuration file (fluent-bit.conf by default), add Logz.io as an output.

note

Logz.io-Out Plugin for Fluent Bit supports one output stream to Logz.io. We recommend running a new instance for each output stream you need.

For a list of options, see the configuration parameters below the code block. 👇

[OUTPUT]
    Name  logzio
    Match *
    Workers 1
    logzio_token <<LOG-SHIPPING-TOKEN>>
    logzio_url   https://<<LISTENER-HOST>>:8071
    id <<YOUR-OUTPUT-ID>>

Parameters

Parameter	Description	Required/Default
logzio_token	Your Logz.io log shipping token directs the data securely to your Logz.io Log Management account. The default token is auto-populated in the examples when you're logged into the Logz.io app as an Admin. Manage your tokens.	Required
logzio_url	Listener URL and port. Replace `<<LISTENER-HOST>>` with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.	`https://listener.logz.io:8071`
logzio_type	Declare your log type for parsing purposes. Logz.io applies default parsing pipelines to the following list of built-in log types. If you declare another type, contact support for assistance with custom parsing. Can't contain spaces.	`logzio-fluent-bit`
logzio_debug	Set to `true` to print debug messages to stdout.	`false`
workers	Enables dedicated thread(s) for this output.	Default value is 1. To support more traffic, adjust this value accordingly.
proxy_host	Support HTTP proxy processing.	Optional, supported format: `<PROXY_HOST>:<PROXY_PORT>`
proxy_user	Support HTTP proxy user authentication.	Optional, default is `""`
proxy_pass	Support HTTP proxy password authentication.	Optional, default is `""`
id	Replace `<<YOUR-OUTPUT-ID>>` with your output ID. e.g: `logzio_output_1` . Recommended to set explicitly.	Required

Run Fluent Bit with the Logz.io plugin

Linux and MacOS:

fluent-bit -e /fluent-bit/plugins/out_logzio.so \
-c /fluent-bit/etc/fluent-bit.conf

Windows:

C:\PROGRA~1\td-agent-bit\bin\fluent-bit.exe -c C:\PROGRA~1\td-agent-bit\conf\fluent-bit.conf -e <<PATH_TO_PLUGIN>>\out_logzio-windows.so

Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don't see your logs, see log shipping troubleshooting.

Run Fluent Bit in a Docker container

Fluent Bit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources. This integration allows you to send logs from Fluent Bit running in a Docker container and forward them to your Logz.io account.

Make the configuration file

To run in a container, create a configuration file named fluent-bit.conf.

note

Logz.io-Out Plugin for Fluent Bit supports one output stream to Logz.io. We recommend running a new instance for each output stream you need.

For a list of options, see the configuration parameters below the code block. 👇

[SERVICE]
    # Include your remaining SERVICE configuration here.
    Plugins_File plugins.conf

[OUTPUT]
    Name  logzio
    Match *
    logzio_token <<LOG-SHIPPING-TOKEN>>
    logzio_url   https://<<LISTENER-HOST>>:8071
    id <<YOUR-OUTPUT-ID>>

Parameters

Parameter	Description	Required/Default
logzio_token	Your Logz.io log shipping token directs the data securely to your Logz.io Log Management account. The default token is auto-populated in the examples when you're logged into the Logz.io app as an Admin. Manage your tokens. Replace `<<LOG-SHIPPING-TOKEN>>` with the token of the account you want to ship to.	Required
logzio_url	Listener URL and port. Replace `<<LISTENER-HOST>>` with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.	`https://listener.logz.io:8071`
logzio_type	Declare your log type for parsing purposes. Logz.io applies default parsing pipelines to the following list of built-in log types. If you declare another type, contact support for assistance with custom parsing. Can't contain spaces.	`logzio-fluent-bit`
logzio_debug	Set to `true` to print debug messages to stdout.	`false`
id	Replace `<<YOUR-OUTPUT-ID>>` with your output ID. e.g: `logzio_output_1` . Recommended to set explicitly.	Required

Run the Docker image

Run the Docker image using the fluent-bit.conf file you made in step 1.

docker run -it --rm \
-v /path/to/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \
logzio/fluent-bit-output

Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don't see your logs, see log shipping troubleshooting.

Run Fluent Bit in Kubernetes

Fluent Bit is an open source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources. Helm is a tool for managing packages of preconfigured Kubernetes resources using Charts. You can use this Helm chart to ship Kubernetes logs to Logz.io with Fluent Bit.

note

This chart is based on the fluent-bit Helm chart.

Sending logs from nodes with taints

If you want to ship logs from any of the nodes that have a taint, make sure that the taint key values are listed in your in your daemonset/deployment configuration as follows:

tolerations:
- key: 
  operator: 
  value: 
  effect: 

To determine if a node uses taints as well as to display the taint keys, run:

kubectl get nodes -o json | jq ".items[]|{name:.metadata.name, taints:.spec.taints}"

For example:

--set logzio-fluentd.daemonset.tolerations[0].key=node-role.kubernetes.io/master --set logzio-fluentd.daemonset.tolerations[0].effect=NoSchedule

note

You need to use Helm client with version v3.9.0 or above.

Enabling multiline logs parser

If you want to enable parsing for multiline logs, add the following code to values.yaml under the customParsers parameter:

customParsers:
  [MULTILINE_PARSER]
      name          multiline-regex-test
      type          regex
      flush_timeout 1000

Standard configuration

Add `logzio-helm` repo

helm repo add logzio-helm https://logzio.github.io/logzio-helm
helm repo update

Run the Helm deployment code

helm install -n monitoring --create-namespace \
--set logzio.token="<<LOG-SHIPPING-TOKEN>>" \
--set logzio.listenerHost="<<LISTENER-HOST>>" \
--set logzio.logType="<<LOG-TYPE>>" \
logzio-fluent-bit logzio-helm/logzio-fluent-bit

Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071. Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to. If required, replace <<LOG_TYPE>> with the desired name for the log type, the default value is fluentbit.

Check Logz.io for your logs

Give your logs some time to get from your system to ours, then open Logz.io.

Customizing Helm chart parameters

Configure customization options

You can use the following options to update the Helm chart parameters:

Specify parameters using the --set key=value[,key=value] argument to helm install
Edit the values.yaml
Override default values with your own my_values.yaml and apply it in the helm install command.

Example

helm install logzio-fluent-bit logzio-helm/logzio-fluent-bit -f my_values.yaml

To modify fluentbit configuration, edit the config section in values.yaml.

Uninstalling the Chart

The Uninstall command is used to remove all the Kubernetes components associated with the chart and to delete the release.

To uninstall the logzio-fluent-bit deployment, use the following command:

helm uninstall logzio-fluent-bit

Run Fluent Bit as a standalone app​

Install Fluent Bit​

Install and configure the Logz.io plugin​

Parameters​

Run Fluent Bit with the Logz.io plugin​

Check Logz.io for your logs​

Run Fluent Bit in a Docker container​

Make the configuration file​

Parameters​

Run the Docker image​

Check Logz.io for your logs​

Run Fluent Bit in Kubernetes​

Sending logs from nodes with taints​

Enabling multiline logs parser​

Standard configuration​

Add logzio-helm repo​

Run the Helm deployment code​

Check Logz.io for your logs​

Customizing Helm chart parameters​

Configure customization options​

Example​

Uninstalling the Chart​

Run Fluent Bit as a standalone app

Install Fluent Bit

Install and configure the Logz.io plugin

Parameters

Run Fluent Bit with the Logz.io plugin

Check Logz.io for your logs

Run Fluent Bit in a Docker container

Make the configuration file

Parameters

Run the Docker image

Check Logz.io for your logs

Run Fluent Bit in Kubernetes

Sending logs from nodes with taints

Enabling multiline logs parser

Standard configuration

Add `logzio-helm` repo

Run the Helm deployment code

Check Logz.io for your logs

Customizing Helm chart parameters

Configure customization options

Example

Uninstalling the Chart