This page covers methods for shipping Jenkins system logs and build console output.

  • To ship build console output (build logs), use the Jenkins plugin.
  • To ship Jenkins system logs, use Filebeat.

Filebeat setup for Jenkins

Configuration tl;dr

Files
Sample configuration
Encryption certificate
Listener
Port 5015. For help finding your region’s listener host, see Account region.
Default log location
/var/log/jenkins/jenkins.log
Log type (for preconfigured parsing)
jenkins
Guided configuration

You’ll need: Filebeat 7 or Filebeat 6, root access

  1. Download the Logz.io certificate

    For HTTPS shipping, download the Logz.io public certificate to your certificate authority folder.

    sudo wget https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt -P /etc/pki/tls/certs/
    
  2. Add Jenkins as an input

    In the Filebeat configuration file (/etc/filebeat/filebeat.yml), add Jenkins to the filebeat.inputs section.

    Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to.

    # ...
    filebeat.inputs:
    - type: log
      paths:
      - /var/log/jenkins/jenkins.log
      fields:
        logzio_codec: plain
    
        # Your Logz.io account token. You can find your token at
        #  https://app.logz.io/#/dashboard/settings/manage-accounts
        token: <<SHIPPING-TOKEN>>
        type: jenkins
      fields_under_root: true
      encoding: utf-8
      ignore_older: 3h
      multiline:
        pattern: '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2}, [0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
        negate: true
        match: after
    

    If you’re running Filebeat 7, paste this code block. Otherwise, you can leave it out.

    # ... For Filebeat 7 only ...
    filebeat.registry.path: /var/lib/filebeat
    processors:
    - rename:
        fields:
        - from: "agent"
          to: "filebeat_agent"
        ignore_missing: true
    - rename:
        fields:
        - from: "log.file.path"
          to: "source"
        ignore_missing: true
    

    If you’re running Filebeat 6, paste this code block.

    # ... For Filebeat 6 only ...
    registry_file: /var/lib/filebeat/registry
    
  3. Add Logz.io as an output

    If Logz.io is not an output, add it now.

    Replace <<LISTENER-HOST>> with your region’s listener host (for example, listener.logz.io). For more information on finding your account’s region, see Account region.

    # ...
    output.logstash:
      hosts: ["<<LISTENER-HOST>>:5015"]
      ssl:
        certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']
    
  4. Start Filebeat

    Start or restart Filebeat for the changes to take effect.

  5. Check Logz.io for your logs

    Give your logs a few minutes to get from your system to ours, and then open Kibana.

    If you still don’t see your logs, see log shipping troubleshooting.

Jenkins Logstash Plugin setup

This is a temporary fork of a Jenkins-maintained project named Jenkins Logstash Plugin. We’re working toward merging our implementation in the Jenkins repo. For full documentation and all configuration options, see the original Jenkins Logstash Plugin repo on GitHub.

Jenkins Logstash Plugin sends Jenkins build logs to your Logz.io account. The plugin is configured per project. You can choose to stream a project’s build logs or to send only the last logs of each build.

Configuration

You’ll need: JDK 8, Maven

  1. Clone the Jenkins Logstash Plugin repo

    Clone the Jenkins Logstash Plugin repo and cd into the logstash-plugin folder.

    git clone https://github.com/logzio/logstash-plugin
    cd logstash-plugin
    
  2. Load the plugin in Jenkins

    Set Maven to use JDK 8, and then build the plugin.

    JAVA_HOME=/path/to/jdk/8/ mvn package
    

    Copy logstash-plugin/target/logstash.hpi to your Jenkins plugins folder on your Jenkins server.

    cp /path/to/repo/logstash-plugin/target/logstash.hpi $JENKINS_HOME/plugins
    

    Restart Jenkins for the changes to take effect. You can do this by browsing to http://<<JENKINS-SERVER>>/restart or http://<<JENKINS-SERVER>>/safeRestart.

  3. Configure the plugin in Jenkins

    Log in to the Jenkins UI and navigate to Manage Jenkins > Configure System.

    In the Logstash section, select Enable sending logs to an Indexer, and then set these options:

    • In the Indexer Type list, choose Logz.io.
    • Logz.io Host: Your region’s listener host. For more information on finding your account’s region, see Account region.
    • Logz.io Token: The token of the account you want to ship to.

    Click Save.

  4. Enable the plugin in your Jenkins jobs

    In each Jenkins job, click Configure in the left menu to set your logging preferences.

    Make sure you enable only one of these options. If both options are enabled, Jenkins Logstash Plugin will send duplicate logs Logz.io.

    To stream all logs

    In the General section, select Send console log to Logstash, and click Save.

    To send only the last logs of each build

    In the Post-build Actions section (at the bottom of the page), select Add post-build action > Send console log to Logstash. In the Max lines box, type the number of logs you want to send per build, and then click Save.

  5. Check Logz.io for your logs

    Give your logs some time to get from your system to ours, and then open Kibana.

    If you still don’t see your logs, see log shipping troubleshooting.