Step by step

Before you begin, you’ll need:

  • Filebeat 7 installed
  • Port 5015 open.
  • Root access
Download the public certificate to your credentials server

For HTTPS shipping, download the public certificate to your certificate authority folder.

sudo curl --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt
Find Apache Storm log location

Run the following command to find your Apache Storm logs directory:

ps -o args= -C java | grep -Po -- '-Dstorm.log.dir=\K[^\s]+'
Add Apache Storm as an input

In the Filebeat configuration file (/etc/filebeat/filebeat.yml), add Apache to the filebeat.inputs section.

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to. Replace <<LOGS_DIRECTORY>> with the path to your Apache Storm logs directory mentioned in the step above.

# ...
- type: log

    - <<LOGS_DIRECTORY>>/*.log
    - <<LOGS_DIRECTORY>>/workers-artifacts/*/*/*.log*

  exclude_files: ['.gz$']

    logzio_codec: plain

    # You can manage your tokens at
    token: <<LOG-SHIPPING-TOKEN>>
    type: apache_storm
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h

If you’re running Filebeat 7, paste this code block. Otherwise, you can leave it out.

# For Filebeat 7 and higher
filebeat.registry.path: /var/lib/filebeat
# The following processors are to ensure compatibility with version 7
- rename:
    - from: "agent"
      to: "beat_agent"
    ignore_missing: true
- rename:
    - from: "log.file.path"
      to: "source"
    ignore_missing: true
Set as the output

If is not an output, add it now. Remove all other outputs.

Replace <<LISTENER-HOST>> with the host for your region. For example, if your account is hosted on AWS US East, or if hosted on Azure West Europe.

# ...
  hosts: ["<<LISTENER-HOST>>:5015"]
    certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']
Start Filebeat

Start or restart Filebeat for the changes to take effect.

Check for your logs

Give your logs some time to get from your system to ours, and then open Kibana. You can search for type:apache_storm to filter for your Apache Storm logs.

If you still don’t see your logs, see log shipping troubleshooting.