To simplify shipping metrics from one or many sources, we created a Docker Metrics Collector. The Docker Metrics Collector is a container that runs Metricbeat with the modules you enable at runtime.
Docker versions
- Docker Metrics Collector v0.2 runs Metricbeat v7.10.
- Docker Metrics Collector v0.1.6 runs Metricbeat v7.5.2.
The Metricbeat version determines which pre-built AWS Metrics dashboards you can use for your data. Filter by the Metricbeat version tag to find relevant AWS dashboards in your Logz.io Metrics library.
Prerequisites
This integration leverages AWS CloudWatch to collect EC2 metrics. You will need to install the CloudWatch agent on your EC2 servers to enable it.
Configuration
If you’re not already running Docker Metrics Collector, follow these steps.
Otherwise, stop the container, add
aws
to the LOGZIO_MODULES environment variable, and restart.
You can find the run command and all parameters
in this procedure.
Set up your IAM user
You’ll need an IAM user with these permissions:
cloudwatch:GetMetricDatacloudwatch:ListMetricsec2:DescribeInstancesec2:DescribeRegionsiam:ListAccountAliasessts:GetCallerIdentity
If you don’t have one, set that up now.
Create an Access key ID and Secret access key for the IAM user, and paste them in your text editor.
You’ll need these for your Metricbeat configuration later.
Get your metrics region
You’ll need to specify the AWS region you’re collecting metrics from.
Find your region’s slug in the region menu (in the top menu, on the right side).
For example: The slug for US East (N. Virginia) is “us-east-1”, and the slug for Canada (Central) is “ca-central-1”.
Paste your region slug in your text editor. You’ll need this for your Metricbeat configuration later.
Pull the Docker image
Download the Docker Metrics Collector image:
docker pull logzio/docker-collector-metrics
Run the container
You’ll set your configuration using environment variables
in the docker run command.
Each parameter is formatted like this:
--env ENV_VARIABLE_NAME="value".
For a complete list of options, see the parameters below the code block.👇
docker run --name docker-collector-metrics \
--env LOGZIO_TOKEN="<<METRICS-SHIPPING-TOKEN>>" \
--env LOGZIO_MODULES="aws" \
--env AWS_ACCESS_KEY_ID="<<ACCESS-KEY>>" \
--env AWS_SECRET_ACCESS_KEY="<<SECRET-KEY>>" \
--env AWS_DEFAULT_REGION="<<AWS-REGION>>" \
--env AWS_NAMESPACES="<<NAMESPACES>>" \
logzio/docker-collector-metrics
Parameters for all modules
| Parameter | Description | Default/Required |
|---|---|---|
| LOGZIO_TOKEN | Your Logz.io Metrics account token. Replace <<METRICS-SHIPPING-TOKEN>> with a token for the Metrics account you want to ship to. Look up your Metrics token. |
Required |
| LOGZIO_MODULES | Comma-separated list of Metricbeat modules to enable on this container (formatted as "module1,module2,module3"). To use a custom module configuration file, mount its folder to /logzio/modules. |
Required |
By default, the LOGZIO_MODULES parameter supports only the following prebuilt modules: aws, system and docker.
| LOGZIO_REGION | Two-letter region code, or blank for US East (Northern Virginia). This determines your listener URL (where you’re shipping the logs to) and API URL. You can find your region code in the Regions and URLs table. | US East |
| LOGZIO_TYPE | This field is needed only if you’re shipping metrics to Kibana and you want to override the default value. In Kibana, this is shown in the type field. Logz.io applies parsing based on type. |
docker-collector-metrics |
| LOGZIO_LOG_LEVEL | The log level the module startup scripts will generate. | "INFO" |
| LOGZIO_EXTRA_DIMENSIONS | Semicolon-separated list of dimensions to be included with your metrics (formatted as dimensionName1=value1;dimensionName2=value2). To use an environment variable as a value, format as dimensionName=$ENV_VAR_NAME. Environment variables must be the only value in the field. If an environment variable can’t be resolved, the field is omitted. |
-- |
| HOSTNAME | Insert your host.name for it to appear in the metrics. If null, host.name will show the container’s ID. |
`` |
| DEBUG | Set to true if you want Metricbeat to run in debug mode. Note: Debug mode tends to generate a lot of debugging output, so you should probably enable it temporarily only when an error occurs while running the docker-collector in production. |
false |
Parameters for the AWS module
Use these parameters if you prefer to explicitly name your access key ID and secret access key. If you are using a credentials file, skip to the relevant section.
These AWS module parameters are the preferred default. We recommend that you use a credentials file (and the AWS module parameters associated with the credentials file) only if there is a specific use case for doing so.
| Parameter | Description | Required |
|---|---|---|
| AWS_ACCESS_KEY_ID | Your IAM user’s access key ID. | Required |
| AWS_SECRET_ACCESS_KEY | Your IAM user’s secret key. | Required |
| AWS_DEFAULT_REGION | Your region’s slug. You can find this in the AWS region menu (in the top menu, to the right). | Required |
| AWS_NAMESPACES | Comma-separated list of namespaces of the metrics you want to collect. For EC2, this is AWS/EC2. See AWS Docs 🔗 for a complete list of valid namespaces. |
Required |
Parameters for the AWS module when using a credentials file
You can use a credential file instead of explicitly naming your access key ID and secret access key. You’ll also need to set an ARN role either on your credential file or explicitly using the parameter AWS_ROLE_ARN.
Learn more about Metricbeat configuration options for AWS Credentials 🔗.
| Parameter | Description | Required |
|---|---|---|
| AWS_DEFAULT_REGION | Your region’s slug. You can find this in the AWS region menu (in the top menu, to the right). | Required |
| AWS_NAMESPACES | Comma-separated list of namespaces of the metrics you want to collect. For EC2, this is AWS/EC2. See AWS Docs 🔗 for a complete list of valid namespaces. |
Required |
| AWS_CREDENTIAL_PROFILE_NAME | Your profile name in shared credentials file. | Required |
| AWS_SHARED_CREDENTIAL_FILE | Your directory of the shared credentials file, in your docker instance. Note: If you’re using this parameter, you’ll have to mount your credentials folder. To mount your credentials folder, add the following line to your docker run command: -v /path/to/your/aws/credentials/folder:<<path/to/shared/credential/file/in/docker>>:ro |
Required |
| AWS_SESSION_TOKEN | Your IAM user’s session token. | -- |
| AWS_ROLE_ARN | Your IAM role to assume. Add this parameter only if your credential file doesn’t include an ARN role. | -- |
Monitor advanced EC2 metrics (Optional)
If you want, you can also monitor advanced EC2 metrics, including disk memory and swap memory.
To enable this option, you’ll need to install and configure a CloudWatch agent on your machine and specify the CWAgent namespace under the AWS_NAMESPACES parameter. For example:
--env AWS_NAMESPACES="CWAgent,AWS/EC2"
For additional instructions, see more about installing the CloudWatch agent and configuring it.
Open your Cloudwatch AWS/EC2 dashboard in Logz.io
Give your metrics a few minutes to get from your system to ours, and then open Logz.io.
Your metrics should appear in the preconfigured dashboard in your Metrics account. To open it, search your Logz.io Metrics dashboards for Cloudwatch AWS/EC2.
You have the option to ship CloudWatch metrics directly over Metricbeat, without a Docker container.
Recommended version: Metricbeat version 7.5.x
Prerequisites
This integration leverages AWS CloudWatch to collect EC2 metrics. You will need to install the CloudWatch agent on your EC2 servers to enable it.
Configuration
Set up your IAM user
You’ll need an IAM user with these permissions:
cloudwatch:GetMetricDatacloudwatch:ListMetricsec2:DescribeInstancesec2:DescribeRegionsiam:ListAccountAliasessts:GetCallerIdentity
If you don’t have one, set that up now.
Create an Access key ID and Secret access key for the IAM user, and paste them in your text editor.
You’ll need these for your Metricbeat configuration later.
Get your metrics region
You’ll need to specify the AWS region you’re collecting metrics from.
Find your region’s slug in the region menu (in the top menu, on the right side).
For example: The slug for US East (N. Virginia) is “us-east-1”, and the slug for Canada (Central) is “ca-central-1”.
Paste your region slug in your text editor. You’ll need this for your Metricbeat configuration later.
Download the Logz.io public certificate to your Metricbeat server
For HTTPS shipping, download the Logz.io public certificate to your certificate authority folder.
sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/AAACertificateServices.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt
You’ll need to run this command on the server that hosts Metricbeat.
(Optional) Disable the system module
By default, Metricbeat ships system metrics from its host. If you don’t need these metrics, disable the system module:
sudo metricbeat modules disable system
Configure Metricbeat
If you’re working with the default configuration file,
(/etc/metricbeat/metricbeat.yml), clear the contents and start with a fresh file.
This code block lays out the default options for collecting metrics from EC2. Paste the code block. Then adjust it to match your AWS environment.
metricbeat.modules:
- module: aws
period: 300s
metricsets:
- cloudwatch
metrics: #specify aws namespaces you want to monitor, just add namspaces from AWS list
- namespace: AWS/EC2
access_key_id: '<<access_key_id>>'
secret_access_key: '<<secret_access_key>>'
fields:
logzio_codec: json
token: <<METRICS-SHIPPING-TOKEN>>
fields_under_root: true
ignore_older: 3hr
type: metrics
output.logstash:
hosts: ["<<LISTENER-HOST>>:5015"]
ssl.certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']
Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):
-
Replace
<<METRICS-SHIPPING-TOKEN>>with a token for the Metrics account you want to ship to.
Look up your Metrics token. -
Replace
<<LISTENER-HOST>>with the host for your region. For example,listener.logz.ioif your account is hosted on AWS US East, orlistener-nl.logz.ioif hosted on Azure West Europe. -
Replace
<<METRICS-SHIPPING-TOKEN>>with a token for the Metrics account you want to ship to.
Look up your Metrics token.
One last validation - make sure Logz.io is the only output and appears only once. If the file has other outputs, remove them.
Monitor advanced EC2 metrics (Optional)
If you want, you can also monitor advanced EC2 metrics, including disk memory and swap memory.
To enable this option, you’ll need to install and configure a CloudWatch agent on your machine and specify the CWAgent namespace under the metrics section. For example:
metricsets:
- cloudwatch
metrics: #specify aws namespaces you want to monitor, just add namspaces from AWS list
- namespace: AWS/EC2
- namespace: CWAgent
For additional instructions, see more about installing the CloudWatch agent and configuring it.
Start Metricbeat
Start or restart Metricbeat for the changes to take effect.
Open your Cloudwatch AWS/EC2 dashboard in Logz.io
Give your metrics a few minutes to get from your system to ours, and then open Logz.io.
Your metrics should appear in the preconfigured dashboard in your Metrics account. To open it, search your Logz.io Metrics dashboards for Cloudwatch AWS/EC2.