Within the applications included in the Docker compose file:

  1. The logzio/cloudwatch-config-builder writes your environment data to configuration files shared with both the CloudWatch exporter and the OpenTelemetry collector containers.

    The configuration files specify:

    • Which data is collected
    • Which data is exposed
    • To which endpoints the data is sent
  2. The CloudWatch exporter and OpenTelemetry collector containers use the configuration files to export and move the metrics data to Logz.io as follows:

    a. Via API, the CloudWatch exporter fetches the requested metrics data collected by CloudWatch from the AWS services in your platform and exposes these metrics to the designated endpoint.

    b. The OpenTelemetry collector receives the data via a metrics pipeline and sends it to Logz.io for ingestion.

Prometheus-CloudFront-OpenTelemetry schema

Note:

  • Extra calls and charges on AWS API requests may be generated by this project.

  • You can track the number of AWS API requests with the cloudwatch_requests_total metric, which provides labels with the API name and namespace specification. For example : cloudwatch_requests_total{action="getMetricStatistics",namespace="AWS/EC2",} 876.0.

  • For more information about the prom/cloudwatch-exporter cost, refer to the relevant Prometheus documentation and to the AWS cloudwatch API pricing page.

Before you begin

Set up your IAM user

You’ll need an IAM user with the following permissions:

  • cloudwatch:ListMetrics
  • cloudwatch:GetMetricStatistics
  • tag:GetResources

If you don’t have one, set that up now.

Create an Access key ID and Secret access key for the IAM user, and paste them in your text editor.

Get your metrics region

You’ll need to specify the AWS region you’re collecting metrics from.

AWS region menu

Find your region’s slug in the region menu (in the top menu, on the right side).

For example: The slug for US East (N. Virginia) is “us-east-1” and the slug for Canada (Central) is “ca-central-1”.

Collect CloudWatch metrics

To collect CloudWatch metrics, you’ll deploy a docker compose file that uses containers with the following images:

Download the docker-compose.yml file
curl https://raw.githubusercontent.com/logzio/logzio-aws-metrics/main/docker-compose.yml -o docker-compose.yml
Define the environment variables
AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY_ID>> \
AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>> \
AWS_DEFAULT_REGION=<<AWS_DEFAULT_REGION>> \
LOGZIO_REGION=<<LOGZIO_REGION>> \
LOGZIO_TOKEN=<<PROMETHEUS-METRICS-SHIPPING-TOKEN>> \
AWS_NAMESPACES=<<AWS_NAMESPACES>> \
SCRAPE_INTERVAL=<<SCRAPE_INTERVAL>> \
P8S_LOGZIO_NAME=<<P8S_LOGZIO_NAME>> \
docker-compose up
Set environment variables for the logzio/cloudwatch-config-builder container
Environment variable Description Required/Default
AWS_DEFAULT_REGION Your region’s slug. You can find this in the AWS Console region menu (in the top menu, to the right). Note: This is the region that you will collect metrics from. Required
LOGZIO_REGION Your Logz.io region code. For example if your region is US, then your region code is us. Look up your region code. Required
LOGZIO_TOKEN Token for shipping Prometheus metrics to your Logz.io account. Find it under Settings > Manage accounts. Look up your Metrics account token Required
SCRAPE_INTERVAL The time interval (in seconds) during which the Cloudwatch exporter retrieves metrics from Cloudwatch, and the Opentelemtry collector scrapes and sends the metrics to Logz.io. Default = 300. Note: This value must be a multiple of 60. Required
AWS_NAMESPACES Comma-separated list of namespaces of the metrics you want to collect. For Network ELB, this is AWS/Network ELB. You can find a complete list of namespaces at AWS Services That Publish CloudWatch Metrics. Note: This Environment variable is required unless you define the CUSTOM_CONFIG_PATH Environment variable Required
P8S_LOGZIO_NAME The value of the p8s_logzio_name external label. This variable identifies which Prometheus environment the metrics arriving at Logz.io came from. Default = logzio-cloudwatch-metrics.
CUSTOM_CONFIG_PATH Path to your Cloudwatch exporter configuration file. For more information refer to the documentation. Note: Set the period_seconds parameter according to your SCRAPE_INTERVAL  
CUSTOM_LISTENER Set a custom URL to ship metrics to (for example, http://localhost:9200). This overrides the LOGZIO_REGION Environment variable.  
Set environment variables for the prom/cloudwatch-exporter container
Environment variable Description
AWS_ACCESS_KEY_ID (Required) Your IAM user’s access key ID.
AWS_SECRET_ACCESS_KEY (Required) Your IAM user’s secret key.
Run Docker

Run the command: docker-compose up.

View service configurations

You can view the exporter and collector configurations in the browser:

Check Logz.io for your metrics

Give your metrics some time to get from your system to ours.

Log in to your Logz.io account and navigate to the current instructions page inside the Logz.io app. Install the pre-built dashboard to enhance the observability of your metrics.

To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.