To simplify shipping metrics from one or many sources, we created a Docker Metrics Collector. The Docker Metrics Collector is a container that runs Metricbeat with the modules you enable at runtime.

Configuration

If you’re not already running Docker Metrics Collector, follow these steps.

Otherwise, stop the container, add aws to the LOGZIO_MODULES environment variable, and restart. You can find the run command and all parameters in this procedure.

Set up your IAM user

You’ll need an IAM user with these permissions:

  • cloudwatch:GetMetricData
  • cloudwatch:ListMetrics
  • ec2:DescribeInstances
  • ec2:DescribeRegions
  • iam:ListAccountAliases
  • sts:GetCallerIdentity

If you don’t have one, set that up now.

Create an Access key ID and Secret access key for the IAM user, and paste them in your text editor.

You’ll need these for your Metricbeat configuration later.

Get your metrics region

You’ll need to specify the AWS region you’re collecting metrics from.

AWS region menu

Find your region’s slug in the region menu (in the top menu, on the right side).

For example: The slug for US East (N. Virginia) is “us-east-1”, and the slug for Canada (Central) is “ca-central-1”.

Paste your region slug in your text editor. You’ll need this for your Metricbeat configuration later.

Pull the Docker image

Download the Docker Metrics Collector image:

docker pull logzio/docker-collector-metrics
Run the container

You’ll set your configuration using environment variables in the docker run command. Each parameter is formatted like this: --env ENV_VARIABLE_NAME="value".

For a complete list of options, see the parameters below the code block.👇

docker run --name docker-collector-metrics \
--env LOGZIO_TOKEN="<<SHIPPING-TOKEN>>" \
--env LOGZIO_MODULES="aws" \
--env AWS_ACCESS_KEY="<<ACCESS-KEY>>" \
--env AWS_SECRET_KEY="<<SECRET-KEY>>" \
--env AWS_REGION="<<AWS-REGION>>" \
--env AWS_NAMESPACES="<<NAMESPACES>>" \
logzio/docker-collector-metrics
Parameters for all modules
Parameter Description
LOGZIO_TOKEN Your Logz.io account token. Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to.
LOGZIO_MODULES Comma-separated list of Metricbeat modules to enable on this container (formatted as "module1,module2,module3"). To use a custom module configuration file, mount its folder to /logzio/modules.
LOGZIO_REGION Blank (US East) Two-letter region code, or blank for US East (Northern Virginia). This determines your listener URL (where you’re shipping the logs to) and API URL.
You can find your region code in the Regions and URLs table.
LOGZIO_TYPE docker-collector-metrics This field is needed only if you’re shipping metrics to Kibana and you want to override the default value.
In Kibana, this is shown in the type field. Logz.io applies parsing based on type.
LOGZIO_LOG_LEVEL "INFO" The log level the module startup scripts will generate.
LOGZIO_EXTRA_DIMENSIONS Semicolon-separated list of dimensions to be included with your metrics (formatted as dimensionName1=value1;dimensionName2=value2).
To use an environment variable as a value, format as dimensionName=$ENV_VAR_NAME. Environment variables must be the only value in the field. If an environment variable can’t be resolved, the field is omitted.
HOSTNAME `` Insert your host name if you want it to appear in the metrics’ host.name. If null, host.name will show the container’s ID.
DEBUG "false" Set to true if you want Metricbeat to run in debug mode.
Note: Debug mode tends to generate a lot of debugging output, so you should probably enable it temporarily only when an error occurs while running the docker-collector in production.
Parameters for the AWS module
Parameter Description
AWS_ACCESS_KEY Your IAM user’s access key ID.
AWS_SECRET_KEY Your IAM user’s secret key.
AWS_REGION Your region’s slug. You can find this in the AWS region menu (in the top menu, to the right).
AWS_NAMESPACES Comma-separated list of namespaces of the metrics you want to collect.
For Application ELB, this is AWS/ApplicationELB. For the complete list of all valid namespaces, see this resource.
Open your Cloudwatch AWS/Application ELB dashboard in Logz.io

Give your metrics a few minutes to get from your system to ours, and then open Logz.io.

Your metrics should appear in the preconfigured dashboard in your Metrics account. To open it, click > Manage in the left menu, then click Logz.io Dashboards > Cloudwatch AWS/Application ELB.

You have the option to ship CloudWatch metrics directly over Metricbeat, without a Docker container.

Recommended version: Metricbeat version 7.5.x

Configuration

Set up your IAM user

You’ll need an IAM user with these permissions:

  • cloudwatch:GetMetricData
  • cloudwatch:ListMetrics
  • ec2:DescribeInstances
  • ec2:DescribeRegions
  • iam:ListAccountAliases
  • sts:GetCallerIdentity

If you don’t have one, set that up now.

Create an Access key ID and Secret access key for the IAM user, and paste them in your text editor.

You’ll need these for your Metricbeat configuration later.

Get your metrics region

You’ll need to specify the AWS region you’re collecting metrics from.

AWS region menu

Find your region’s slug in the region menu (in the top menu, on the right side).

For example: The slug for US East (N. Virginia) is “us-east-1”, and the slug for Canada (Central) is “ca-central-1”.

Paste your region slug in your text editor. You’ll need this for your Metricbeat configuration later.

Download the Logz.io public certificate to your Metricbeat server

For HTTPS shipping, download the Logz.io public certificate to your certificate authority folder.

sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/AAACertificateServices.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt
(Optional) Disable the system module

By default, Metricbeat ships system metrics from its host. If you don’t need these metrics, disable the system module:

sudo metricbeat modules disable system
Configure Metricbeat

If you’re working with the default configuration file, (/etc/metricbeat/metricbeat.yml), clear the contents and start with a fresh file.

This code block lays out the default options for collecting metrics from Application ELB. Paste the code block. Then adjust it to match your AWS environment.

metricbeat.modules:
- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics: #specify aws namespaces you want to monitor, just add namspaces from AWS list
    - namespace: AWS/ApplicationELB

  access_key_id: '<<access_key_id>>'
  secret_access_key: '<<secret_access_key>>'

fields:
  logzio_codec: json
  token: <<SHIPPING-TOKEN>>
fields_under_root: true
ignore_older: 3hr
type: metrics
output.logstash:
  hosts: ["<<LISTENER-HOST>>:5015"]
  ssl.certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']
Replace the placeholders in your Metricbeat configuration

Still in the same configuration file, replace the placeholders to match your specifics.

  • Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to.

  • Replace <<LISTENER-HOST>> with your region’s listener host (for example, listener.logz.io). For more information on finding your account’s region, see Account region.

One last validation - make sure Logz.io is the only output and appears only once. If the file has other outputs, remove them.

Start Metricbeat

Start or restart Metricbeat for the changes to take effect.

Open your Cloudwatch AWS/Application ELB dashboard in Logz.io

Give your metrics a few minutes to get from your system to ours, and then open Logz.io.

Your metrics should appear in the preconfigured dashboard in your Metrics account. To open it, click > Manage in the left menu, then click Logz.io Dashboards > Cloudwatch AWS/Application ELB.