AWS ECS Fargate

AWS Fargate is a serverless compute engine for building applications without managing servers. This template will create a new collector container to send your AWS ECS Fargate traces and metrics to Logz.io using AWS OTel Collector. ECS logs will be sent to logz.io with aws firehose

Only awsecscontainermetrics receiver metrics are collected by default

Make your ecs fargate logs are being sent to cloudwatch

Prerequisites

• An Amazon ECS Fargate cluster
• Applications instrumented with OpenTelemetry SDK (for Traces)

Next, you'll need to configure your CloudFormation template and point the OTLP exporter.

Choose the CloudFormation template

We support two types of CloudFormation stacks:

• Option 1 collects all logs from ECS clusters and also gathers metrics and traces from a single cluster.
• Option 2 is designed only for metrics and traces. To collect metrics and traces from multiple clusters without logs, deploy multiple instances of Option 2.

Option 1: Logs, Metrics, and Traces

Click Launch Stack below to deploy the CloudFormation template. This will automatically create the required resources and configurations for the AWS OTel Collector.

Region	Deployment
us-east-1
us-east-2
us-west-1
us-west-2
eu-central-1
eu-north-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
ap-northeast-1
ap-northeast-2
ap-northeast-3
ap-south-1
ap-southeast-1
ap-southeast-2
ca-central-1

Option 2: Metrics and Traces Only

Region	Deployment
us-east-1
us-east-2
us-west-1
us-west-2
eu-central-1
eu-north-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
ap-northeast-1
ap-northeast-2
ap-northeast-3
ap-south-1
ap-southeast-1
ap-southeast-2
ca-central-1

Point the OTLP exporter to the new collector

Update your application's OTLP exporter configuration to send data to the new collector container running in your ECS Fargate tasks.

Check Logz.io for your data

Allow some time for your data to be processed and appear in Logz.io.

Required parameters

The CloudFormation template requires the following parameters:

Traces and metrics

Parameter	Description
ClusterName	The name of your ECS cluster from which you want to collect metrics.
CreateIAMRoles	Specifies whether to create default IAM roles (True or False).
ExecutionRoleArn	The role ARN you want to use as the ECS task role. (Optional, only required if you set CreateIAMRoles to False)
SecurityGroups	The role ARN you want to use as the ECS execution role. (Optional, only required if you set CreateIAMRoles to False)
Subnets	The list of SecurityGroupIds in your Virtual Private Cloud (VPC).
LogzioTracingToken	Your Logz.io tracing account token.
LogzioMetricsToken	Your Logz.io metrics account token.
LogzioRegion	Your Logz.io region. for example: us
LogzioListenerUrl	Your Logz.io listener URL. for example: https://listener.logz.io:8053

Logs

Parameter	Description
logzioLogsToken	Your Logz.io logs account token.
LogzioListener	Your Logz.io listener URL. for example: https://aws-firehose-logs-listener.logz.io
logzioType	The log type you'll use with this shipping method. This can be a built-in log type, or your custom log type.
services	A comma-separated list of services you want to collect logs from.
customLogGroups	A comma-separated list of custom log groups to collect logs from, or the ARN of the Secret parameter (explanation below) storing the log groups list if it exceeds 4096 characters.
useCustomLogGroupsFromSecret	If you want to provide list of customLogGroups which exceeds 4096 characters, set to true and configure your customLogGroups as defined below.
triggerLambdaTimeout	The amount of seconds that Lambda allows a function to run before stopping it, for the trigger function.
triggerLambdaMemory	Trigger function's allocated CPU proportional to the memory configured, in MB.
triggerLambdaLogLevel	Log level for the Lambda function. Can be one of: debug, info, warn, error, fatal, panic.
httpEndpointDestinationIntervalInSeconds	The length of time, in seconds, that Kinesis Data Firehose buffers incoming data before delivering it to the destination.
httpEndpointDestinationSizeInMBs	The size of the buffer, in MBs, that Kinesis Data Firehose uses for incoming data before delivering it to the destination.

Resources and Configuration

The CloudFormation template creates several resources to set up the AWS OTel Collector and send metrics and traces to Logz.io. Here is a summary of the resources created by the template and the permissions granted to the IAM policies:

AWS::ECS::TaskDefinition

The ECSTaskDefinition resource defines the container properties, such as the image, command, environment variables, and log configuration for the AWS OTel Collector container. It also sets the task and execution roles, network mode, and required compatibilities.

AWS::ECS::Service

The ECSReplicaService resource creates an Amazon ECS service with the specified task definition, desired count, scheduling strategy, and network configuration. It associates the service with the provided ECS cluster.

AWS::SSM::Parameter

The CollectorConfigParameter resource creates an AWS Systems Manager (SSM) parameter to store the collector's configuration. The configuration defines the receivers, processors, exporters, and service pipelines for traces and metrics. The applied configuration will be:

receivers:
  awsxray:
    endpoint: 0.0.0.0:2000
    transport: udp
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"
  awsecscontainermetrics:
processors:
  batch:
    send_batch_size: 10000
    timeout: 1s
exporters:
  logzio/traces:
    account_token: ${LOGZIO_TRACING_TOKEN}
    region: ${LOGZIO_REGION}
  prometheusremotewrite:
    endpoint: ${LOGZIO_LISTENER_URL}
    external_labels:
      aws_env: fargate-ecs
    headers:
      Authorization: "Bearer ${LOGZIO_METRICS_TOKEN}"
    resource_to_telemetry_conversion:
      enabled: true
    target_info:
        enabled: false
service:
  pipelines:
    traces:
      receivers: [ awsxray,otlp ]
      processors: [ batch ]
      exporters: [ logzio/traces ]
    metrics:
      receivers: [ otlp, awsecscontainermetrics ]
      exporters: [ prometheusremotewrite ]
  telemetry:
    logs:
      level: "info"

AWS::Lambda::Function

The logzioFirehoseSubscriptionFiltersFunction Lambda function is designed to handle the process of filtering log data for Logz.io, and add and remove cloudwatch logs subscription filters

AWS::Events::Rule

The logGroupCreationEvent rule listens for AWS API calls via CloudTrail, specifically focusing on CreateLogGroup events from the logs.amazonaws.com event source. When such an event occurs, it triggers the Logz.io subscription filter function.

AWS::KinesisFirehose::DeliveryStream

A Kinesis Firehose delivery stream configured to send logs to Logz.io. Logs that fail to deliver are backed up to an S3 bucket.

AWS::S3::Bucket

An S3 bucket used to store backup logs that fail to deliver via Kinesis Firehose.

IAM Roles

1. ECSTaskRole: This role is assumed by the ECS tasks and allows them to call AWS services on your behalf. The role is created with the following properties:
   • RoleName: AWSOTelRole
2. ECSExecutionRole: The ECS container agent assumes this role, allowing it to make calls to the Amazon ECS API on your behalf. The role is created with the following properties:
   • RoleName: AWSOTelExecutionRole
3. firehoseSubscriptionFilterLambdaRole: This IAM role is assumed by the Lambda function when it runs. It has permissions to:
   • Create, access, and manage CloudWatch logs.
   • Manage CloudWatch Logs Subscription Filters.
   • Assume other roles.
4. logzioS3DestinationFirehoseRole: IAM role assumed by Kinesis Firehose when interacting with the S3 bucket for backup purposes. The role has permissions to perform basic S3 operations like PutObject, GetObject, and ListBucket.

IAM Policies

1. AWSOpenTelemetryPolicy: This policy is attached to the ECSTaskRole and grants the following permissions:

   • Allows the collector to create, describe, and put log events into CloudWatch Logs.
   • Allows the collector to send trace data to AWS X-Ray using PutTraceSegments and PutTelemetryRecords actions.
   • Allows the collector to fetch sampling rules, targets, and statistic summaries from AWS X-Ray.
   • Allows the collector to get parameters from AWS Systems Manager (SSM) Parameter Store.

2. AmazonECSTaskExecutionRolePolicy: This managed policy is attached to the ECSExecutionRole and grants the following permissions:

   • Allows the collector to register and deregister tasks and task definitions in Amazon ECS.
   • Allows the collector to access and manage container instances, clusters, and services.
   • Allows the collector to access and manage Elastic Load Balancing resources.

3. CloudWatchLogsFullAccess: This managed policy is attached to the ECSExecutionRole and grants full access to Amazon CloudWatch Logs.

4. AmazonSSMReadOnlyAccess: This managed policy is attached to the ECSExecutionRole and grants read-only access to the AWS Systems Manager services and resources.

By creating and attaching these IAM roles and policies, the AWS OTel Collector is granted the necessary permissions to collect and send metrics and traces from your ECS Fargate cluster to Logz.io. The template also configures the collector to use the specified Logz.io tokens, region, and listener URL.

Ensuring Connectivity Between the Application Containers and the Collector Container

To ensure seamless connectivity between your application containers and the AWS OTel Collector container, you need to properly configure your Amazon VPC, subnets, and security groups.

Amazon VPC and Subnets

The application containers and the AWS OTel Collector container must be launched in the same Amazon VPC and share at least one common subnet. This ensures that they can communicate with each other using private IP addresses. When deploying the CloudFormation template, make sure to provide the correct VPC subnet IDs in the Subnets parameter.

Security Groups

To allow your application containers to send traces and metrics to the AWS OTel Collector container, you need to configure the security groups for both sets of containers.

1. Application Containers Security Group: Modify the security group associated with your application containers to allow outbound traffic to the OTel Collector's ports (4317 for gRPC and 4318 for HTTP). Ensure that the rules are restricted to the IP address range of the VPC or the security group associated with the OTel Collector container. This configuration allows your application containers to send data to the collector container.

2. AWS OTel Collector Container Security Group: Create or modify the security group for the AWS OTel Collector container to allow inbound traffic on ports 4317 (gRPC) and 4318 (HTTP) from the application containers' security group. This configuration ensures the collector container accepts incoming data from your application containers.

When deploying the CloudFormation template, provide the collector container's security group ID in the SecurityGroups parameter.

By properly configuring your Amazon VPC, subnets, and security groups, you can ensure that your application containers can send metrics and traces to the AWS OTel Collector container, which in turn forwards the data to Logz.io.

Manual configuration: Send logs from AWS Fargate to Logz.io

Build the task execution role

In the IAM Console, create a new role.

• Under AWS Service, keep the default selection.
• Under Choose a use case:
  • Select Elastic Container Service.
  • Select Elastic Container Service Task (scroll to the bottom).
  • Click Next: Permissions and select AmazonECSTaskExecutionRolePolicy.
  • Click Next: Tags, then Next: Review.
• Set the Role Name to logzioEcsTaskExecutionRole, then click Create Role.

Once created, open the role's Summary page, copy the Role ARN, and save it for later.

• Click the newly created role to go to its Summary page.
• Copy the Role ARN (at the top of the page) and save it for later. You will need it for the deployment JSON.

Create a Fluent Bit task definition

In the ECS Console, go to the Task Definitions page.

• Click Create new Task Definition.
• Choose Fargate and click Next step.
• Scroll to the Volumes section and select Configure via JSON.
• Replace the default JSON with the following:

{
  "family": "logzio-fargate-task",
  "requiresCompatibilities": [ "FARGATE" ],
  "containerDefinitions": [
    {
      "name": "logzio-log-router",
      "image": "amazon/aws-for-fluent-bit:latest",
      "essential": true,
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": { "enable-ecs-log-metadata": "true" }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/aws/ecs/logzio-fargate-logs",
          "awslogs-region": "<<AWS-REGION>>",
          "awslogs-stream-prefix": "aws/ecs"
        }
      }
    },
    {
      "name": "app",
      "essential": true,
      "image": "<<YOUR-APP-IMAGE>>",
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "Http",
          "Host": "<<LISTENER-HOST>>",
          "URI": "/?token=<<LOG-SHIPPING-TOKEN>>&type=fargate",
          "Port": "8071",
          "tls": "on",
          "tls.verify": "off",
          "Format": "json_lines"
        }
      }
    }
  ],
  "cpu": "256",
  "executionRoleArn": "arn:aws:iam::<<AWS-ACCOUNT-ID>>:role/logzioEcsTaskExecutionRole",
  "memory": "512",
  "volumes": [ ],
  "placementConstraints": [ ],
  "networkMode": "awsvpc",
  "tags": [ ]
}

Replace the placeholders (<< >>) with your values:

Parameters in logzio-log-router

Parameter	Description
logConfiguration.options.awslogs-group	In the CloudWatch left menu, select Logs > Log groups, and then click Actions > Create log group. Give the Log Group Name /aws/ecs/logzio-fargate-logs.
logConfiguration.options.awslogs-region	The AWS region of your cluster.

Parameters in app

Parameter	Description
image	Replace <<YOUR-APP-IMAGE>> with the name of the image you want to ship logs from.
logConfiguration.options.Host	The host for your region.
logConfiguration.options.URI	Your Logz.io account token. Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Remaining parameters

Parameter	Description
executionRoleArn	Replace <<AWS-ACCOUNT-ID>> with your AWS account Id.

When you're done, click Save, and then click Create.

Run the task on your cluster

Open your Task Definition page and click Actions > Run Task.

Click Switch to launch type, and fill in these details:

• For Launch Type, select FARGATE.
• For Cluster, select your cluster.
• For Subnets, select a subnet from your cluster.

Click Run task.

The logs created by the Fluent Bit shipper are in Cloudwatch under the /aws/ecs/logzio-fargate-logs log group.

Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Explore.

You'll be able to find these logs by searching for type:{{fargate}}.

Manual configuration: Send metrics from AWS Fargate to Logz.io

There are a few alternatives to collect metrics for your ECS Fargate cluster:

Attach OpenTelemetry Collector Sidecar to Your Application Task

You can attach an OpenTelemetry collector sidecar to your ECS application task. This method allows you to collect a variety of metrics from your containerized applications. Note that this requires a deployment per task.

Collect ECS Metrics from CloudWatch

You can use the CloudWatch Metrics Stream integration to collect ECS metrics such as CPU and memory utilization directly from CloudWatch. This integration also includes an option to enable AWS Container Insights for enhanced ECS container metrics at the cluster level.

Understanding where your Metrics come from

The source of your metrics in Logz.io depends on the integration method you select:

• OpenTelemetry Collector Sidecar - Uses the awsecscontainermetricsreceiver to collect and send container-level metrics from within your Fargate tasks. View the available metrics.
• ECS CloudWatch - Collects cluster and service-level metrics from the AWS/ECS namespace, providing insights into CPU, memory utilization, and scaling activity. Container Insights CloudWatch – Extends CloudWatch monitoring by gathering per-task metrics, including CPU, memory, and EBS filesystem utilization, from the ECS/ContainerInsights namespace.

Each integration provides different data, so choose the option that best fits your monitoring needs.

note

Enabling AWS Container Insights may incur additional charges from AWS.

Adding extra configuration to your Fluent Bit task

Create an extra.conf file

Create an extra.conf file with the extra configuration. For example:

[FILTER]
    Name record_modifier
    Match *
    Record app-version ${APP_VERSION}

Upload the extra.conf file

• Upload the extra.conf file to S3 (if your Fluent Bit is on EC2).
• Upload the extra.conf file to the container image or on a volume that's mounted in the container.

Update the task definition file

Add the path to the extra.conf file to the task definition file as follows.

For the configuration file on S3:

"firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "s3",
                    "config-file-value": "arn:aws:s3:::<<PATH-TO-YOUR-EXTRA-CONF>>/extra.conf"
                }
            }

For the configuration file on the container image or on a volume mounted in the container:

"firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "file",
                    "config-file-value": "<<PATH-TO-YOUR-EXTRA-CONF>>/extra.conf"
                }
            }

Replace <<PATH-TO-YOUR-EXTRA-CONF>> with the path to the extra.conf file. For example, for S3, yourbucket/yourdirectory/extra.conf.