Skip to main content

Apache Storm

Apache Storm is a free and open source distributed realtime computation system. This integration allows you to send logs from your Apache Storm server instances to your Logz.io account.

Configure Apache Storm to send metrics to Logz.io​

Before you begin, you'll need:

  • Filebeat installed
  • Port 5015 open.
  • Root access
Download the Logz.io public certificate to your credentials server​

For HTTPS shipping, download the Logz.io public certificate to your certificate authority folder.

sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/AAACertificateServices.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt

Find Apache Storm log location​

Run the following command to find your Apache Storm logs directory:

ps -o args= -C java | grep -Po -- '-Dstorm.log.dir=\K[^\s]+'

Add Apache Storm as an input​

In the Filebeat configuration file (/etc/filebeat/filebeat.yml), add Apache to the filebeat.inputs section.

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to. Replace <<LOGS_DIRECTORY>> with the path to your Apache Storm logs directory mentioned in the step above.

note

Filebeat requires a file extension specified for the log input.

# ...
filebeat.inputs:
- type: filestream

  paths:
    - <<LOGS_DIRECTORY>>/*.log
    - <<LOGS_DIRECTORY>>/workers-artifacts/*/*/*.log*


  fields:
    logzio_codec: plain

    # You can manage your tokens at
    #  https://app.logz.io/#/dashboard/settings/manage-tokens/log-shipping
    token: <<LOG-SHIPPING-TOKEN>>
    type: apache_storm
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h

If you're running Filebeat 7 to 8.1, paste the code block below instead:

# ...
filebeat.inputs:
- type: log

  paths:
    - <<LOGS_DIRECTORY>>/*.log
    - <<LOGS_DIRECTORY>>/workers-artifacts/*/*/*.log*


  fields:
    logzio_codec: plain

    # You can manage your tokens at
    #  https://app.logz.io/#/dashboard/settings/manage-tokens/log-shipping
    token: <<LOG-SHIPPING-TOKEN>>
    type: apache_storm
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h

Set Logz.io as the output​

If Logz.io is not an output, add it now. Remove all other outputs.

Replace <<LISTENER-HOST>> with the host for your region. For example, listener.logz.io if your account is hosted on AWS US East, or listener-nl.logz.io if hosted on Azure West Europe. 

# ...
output.logstash:
  hosts: ["<<LISTENER-HOST>>:5015"]
  ssl:
    certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

Start Filebeat​

Start or restart Filebeat for the changes to take effect.

Check Logz.io for your logs​

Give your logs some time to get from your system to ours, and then open Open Search Dashboards. You can search for type:apache_storm to filter for your Apache Storm logs.

If you still don't see your logs, see Filebeat troubleshooting.