Ship Jenkins logs

This page covers methods for shipping Jenkins system logs and build console output.

To ship build console output (build logs), use the Jenkins plugin.
To ship Jenkins system logs, use Filebeat.

Filebeat
Jenkins plugin

Filebeat setup for Jenkins

Configuration tl;dr

Files: Sample configuration
Encryption certificate
Listener: Port 5015. For help finding your region’s listener host, see Account region.
Default log location: /var/log/jenkins/jenkins.log
Log type (for preconfigured parsing): jenkins

Guided configuration

You’ll need: Filebeat 7 or Filebeat 6, root access

Download the Logz.io certificate

For HTTPS shipping, download the Logz.io public certificate to your certificate authority folder.

sudo wget https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomainValidationSecureServerCA.crt -P /etc/pki/tls/certs/

Add Jenkins as an input

In the Filebeat configuration file (/etc/filebeat/filebeat.yml), add Jenkins to the filebeat.inputs section.

Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to.

Filebeat 7
Filebeat 6

# Filebeat 7 configuration

filebeat.inputs:
- type: log
  paths:
  - /var/log/jenkins/jenkins.log
  fields:
    logzio_codec: plain

    # Your Logz.io account token. You can find your token at
    #  https://app.logz.io/#/dashboard/settings/manage-accounts
    token: <<SHIPPING-TOKEN>>
    type: jenkins
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h
  multiline:
    pattern: '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2}, [0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
    negate: true
    match: after

filebeat.registry.path: /var/lib/filebeat
processors:
- rename:
    fields:
    - from: "agent"
      to: "filebeat_agent"
    ignore_missing: true
- rename:
    fields:
    - from: "log.file.path"
      to: "source"
    ignore_missing: true

# Filebeat 6 configuration

filebeat.inputs:
- type: log
  paths:
  - /var/log/jenkins/jenkins.log
  fields:
    logzio_codec: plain

    # Your Logz.io account token. You can find your token at
    #  https://app.logz.io/#/dashboard/settings/manage-accounts
    token: <<SHIPPING-TOKEN>>
    type: jenkins
  fields_under_root: true
  encoding: utf-8
  ignore_older: 3h
  multiline:
    pattern: '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2}, [0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
    negate: true
    match: after

registry_file: /var/lib/filebeat/registry

Add Logz.io as an output

If Logz.io is not an output, add it now.

Replace <<LISTENER-HOST>> with your region’s listener host (for example, listener.logz.io). For more information on finding your account’s region, see Account region.
```
output.logstash:
  hosts: ["<<LISTENER-HOST>>:5015"]
  ssl:
    certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']
```
Start Filebeat

Start or restart Filebeat for the changes to take effect.
Check Logz.io for your logs

Give your logs a few minutes to get from your system to ours, and then open Kibana.

If you still don’t see your logs, see log shipping troubleshooting.

Jenkins Logstash Plugin setup

This is a temporary fork of a Jenkins-maintained project named Jenkins Logstash Plugin. We’re working toward merging our implementation in the Jenkins repo. For full documentation and all configuration options, see the original Jenkins Logstash Plugin repo on GitHub.

Jenkins Logstash Plugin sends Jenkins build logs to your Logz.io account. The plugin is configured per project. You can choose to stream a project’s build logs or to send only the last logs of each build.

Configuration

You’ll need: JDK 8, Maven

Clone the Jenkins Logstash Plugin repo

Clone the Jenkins Logstash Plugin repo and cd into the logstash-plugin folder.
```
git clone https://github.com/logzio/logstash-plugin
cd logstash-plugin
```
Load the plugin in Jenkins

Set Maven to use JDK 8, and then build the plugin.
```
JAVA_HOME=/path/to/jdk/8/ mvn package
```
Copy logstash-plugin/target/logstash.hpi to your Jenkins plugins folder on your Jenkins server.
```
cp /path/to/repo/logstash-plugin/target/logstash.hpi $JENKINS_HOME/plugins
```
Restart Jenkins for the changes to take effect. You can do this by browsing to http://<<JENKINS-SERVER>>/restart or http://<<JENKINS-SERVER>>/safeRestart.
Configure the plugin in Jenkins

Log in to the Jenkins UI and navigate to Manage Jenkins > Configure System.

In the Logstash section, select Enable sending logs to an Indexer, and then set these options:
- In the Indexer Type list, choose Logz.io.
- Logz.io Host: Your region’s listener host. For more information on finding your account’s region, see Account region.
- Logz.io Token: The token of the account you want to ship to.
Click Save.
Enable the plugin in your Jenkins jobs

In each Jenkins job, click Configure in the left menu to set your logging preferences.

Make sure you enable only one of these options. If both options are enabled, Jenkins Logstash Plugin will send duplicate logs Logz.io.

To stream all logs

In the General section, select Send console log to Logstash, and click Save.

To send only the last logs of each build

In the Post-build Actions section (at the bottom of the page), select Add post-build action > Send console log to Logstash. In the Max lines box, type the number of logs you want to send per build, and then click Save.
Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Kibana.

If you still don’t see your logs, see log shipping troubleshooting.