Within the applications included in the Docker compose file:
The logzio/cloudwatch-config-builder writes your environment data to configuration files shared with both the CloudWatch exporter and the OpenTelemetry collector containers.
The configuration files specify:
- Which data is collected
- Which data is exposed
- To which endpoints the data is sent
The CloudWatch exporter and OpenTelemetry collector containers use the configuration files to export and move the metrics data to Logz.io as follows:
a. Via API, the CloudWatch exporter fetches the requested metrics data collected by CloudWatch from the AWS services in your platform and exposes these metrics to the designated endpoint.
b. The OpenTelemetry collector receives the data via a metrics pipeline and sends it to Logz.io for ingestion.
Extra calls and charges on AWS API requests may be generated by this project.
You can track the number of AWS API requests with the
cloudwatch_requests_totalmetric, which provides labels with the API name and namespace specification. For example :
Before you begin
Set up your IAM user
You’ll need an IAM user with the following permissions:
If you don’t have one, set that up now.
Create an Access key ID and Secret access key for the IAM user, and paste them in your text editor.
Get your metrics region
You’ll need to specify the AWS region you’re collecting metrics from.
Find your region’s slug in the region menu (in the top menu, on the right side).
For example: The slug for US East (N. Virginia) is “us-east-1” and the slug for Canada (Central) is “ca-central-1”.
Collect CloudWatch metrics
To collect CloudWatch metrics, you’ll deploy a docker compose file that uses containers with the following images:
Download the docker-compose.yml file
curl https://raw.githubusercontent.com/logzio/logzio-aws-metrics/main/docker-compose.yml -o docker-compose.yml
Define the environment variables
AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY_ID>> \ AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>> \ AWS_DEFAULT_REGION=<<AWS_DEFAULT_REGION>> \ LOGZIO_REGION=<<LOGZIO_REGION>> \ LOGZIO_TOKEN=<<PROMETHEUS-METRICS-SHIPPING-TOKEN>> \ AWS_NAMESPACES=<<AWS_NAMESPACES>> \ SCRAPE_INTERVAL=<<SCRAPE_INTERVAL>> \ P8S_LOGZIO_NAME=<<P8S_LOGZIO_NAME>> \ docker-compose up
Set environment variables for the logzio/cloudwatch-config-builder container
|AWS_DEFAULT_REGION||Your region’s slug. You can find this in the AWS Console region menu (in the top menu, to the right). Note: This is the region that you will collect metrics from.||Required|
|LOGZIO_REGION||Your Logz.io region code. For example if your region is US, then your region code is
|LOGZIO_TOKEN||Token for shipping Prometheus metrics to your Logz.io account. Find it under Settings > Manage accounts. Look up your Metrics account token||Required|
|SCRAPE_INTERVAL||The time interval (in seconds) during which the Cloudwatch exporter retrieves metrics from Cloudwatch, and the Opentelemtry collector scrapes and sends the metrics to Logz.io. Default = 300. Note: This value must be a multiple of 60.||Required|
|AWS_NAMESPACES||Comma-separated list of namespaces of the metrics you want to collect. For
|P8S_LOGZIO_NAME||The value of the
|CUSTOM_CONFIG_PATH||Path to your Cloudwatch exporter configuration file. For more information refer to the documentation. Note: Set the
|CUSTOM_LISTENER||Set a custom URL to ship metrics to (for example, http://localhost:9200). This overrides the
Set environment variables for the
|AWS_ACCESS_KEY_ID (Required)||Your IAM user’s access key ID.|
|AWS_SECRET_ACCESS_KEY (Required)||Your IAM user’s secret key.|
Run the command:
View service configurations
You can view the exporter and collector configurations in the browser:
Check Logz.io for your metrics
Give your metrics some time to get from your system to ours.
Log in to your Logz.io account and navigate to the current instructions page inside the Logz.io app. Install the pre-built dashboard to enhance the observability of your metrics.
To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.