AWS ECS

note

For a much easier and more efficient way to collect and send telemetry, consider using the Logz.io telemetry collector.

Configure AWS to forward logs to Logz.io

This integration uses Fluentd in a Docker container to forward logs from your Amazon Elastic Container Service (ECS) cluster to Logz.io.

note

This integration refers to an EC2-based cluster. For Fargate-based cluster see our Fargate documentation.

Important

Fluentd will fetch all existing logs, as it is not able to ignore older logs.

Automated CloudFormation deployment

Configure and create your stack

Click the button that matches your AWS region, then follow the instructions below:

AWS Region	Launch button
`us-east-1`

note

If your region is not on the list, let us know in the repo's issues, or reach out to Logz.io support team!

####### In screen Step 1 Specify template:

Keep the defaults and click Next.

Screen_1

####### In screen Step 2 Specify stack details:

For Stack name you can either keep the default, or change the stack name.
For LogzioListener - choose your Logz.io listener from the list.
For LogzioToken - insert your Logz.io logs shipping token.
Click Next.

Screen_2

####### In screen Step 3 Configure stack options (Optional):

If you want to, you can add your custom tags, or not. Click on Next.

####### In screen Step 4 Review:

Scroll down and click on Create stack.

Give your stack a few moments to launch.

Run the task

Go to your AWS ECS page, and on the left menu, click on Task Definitions, then choose the task you just created.
Click on the Actions button, then choose Run Task.
In the Run Task screen, choose EC2 as your Launch type.
Choose the cluster you want to ship logs from.
For Placement Templates choose One Task Per Host.
Click on Run Task.

Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

Manual deployment (AWS Classic Console)

Download the task definition JSON

wget https://raw.githubusercontent.com/logzio/logzio-aws-ecs/master/task-definition.json

Configure the task

In your prefered text editor, open the JSON you downloaded in the previous step and replace the following:

Paramater	Details
`<<LOG-SHIPPING-TOKEN>>`	Required. Your Logz.io account token. Replace with the token of the account you want to ship to.
`<<LISTENER-HOST>>`	Required. Your Logz.io listener URL. Replace with your region's listener URL.

Advanced settings (optional)

Since the Docker image is based on Logz.io's fluentd-docker-logs image, any of the environment variables mentioned here can be added to the task definition JSON.

Add your task definition

In your Amazon ECS Classic Console menu, go to Task Definitions and click on Create new Task Definition.
In the Step 1: Select launch type compatibility screen, choose EC2 and click Next step.
In the Step 2: Configure task and container definitions screen, scroll down and click on the Configure via JSON button.
In the text-box, delete the existing text and paste your configured task definition JSON. Press Save, then press Create.

Run the task

After the task creation, click on the Actions button, then choose Run Task.
In the Run Task screen, choose EC2 as your Launch type.
Choose the cluster you want to ship logs from.
For Placement Templates, choose One Task Per Host.
Click on Run Task.

Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

Configure AWS to forward metrics to Logz.io

Deploy this integration to send your Amazon ECS metrics to Logz.io.

This integration creates a Kinesis Data Firehose delivery stream that links to your Amazon ECS metrics stream and then sends the metrics to your Logz.io account. It also creates a Lambda function that adds AWS namespaces to the metric stream, and a Lambda function that collects and ships the resources' tags.

Install the pre-built dashboard to enhance the observability of your metrics.

To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.

Before you begin, you'll need:

An active Logz.io account

Configure AWS to forward metrics to Logz.io

1. Set the required minimum IAM permissions

configured the minimum required IAM permissions as follows:

Amazon S3:
- s3:CreateBucket
- s3:DeleteBucket
- s3:PutObject
- s3:GetObject
- s3:DeleteObject
- s3:ListBucket
- s3:AbortMultipartUpload
- s3:GetBucketLocation
AWS Lambda:
- lambda:CreateFunction
- lambda:DeleteFunction
- lambda:InvokeFunction
- lambda:GetFunction
- lambda:UpdateFunctionCode
- lambda:UpdateFunctionConfiguration
- lambda:AddPermission
- lambda:RemovePermission
- lambda:ListFunctions
Amazon CloudWatch:
- cloudwatch:PutMetricData
- cloudwatch:PutMetricStream
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- logs:DeleteLogGroup
- logs:DeleteLogStream
AWS Kinesis Firehose:
- firehose:CreateDeliveryStream
- firehose:DeleteDeliveryStream
- firehose:PutRecord
- firehose:PutRecordBatch
IAM:
- iam:PassRole
- iam:CreateRole
- iam:DeleteRole
- iam:AttachRolePolicy
- iam:DetachRolePolicy
- iam:GetRole
- iam:CreatePolicy
- iam:DeletePolicy
- iam:GetPolicy
Amazon CloudFormation:
- cloudformation:CreateStack
- cloudformation:DeleteStack
- cloudformation:UpdateStack
- cloudformation:DescribeStacks
- cloudformation:DescribeStackEvents
- cloudformation:ListStackResources

2. Create stack in the relevant region

To deploy this project, click the button that matches the region you wish to deploy your stack to:

Region	Deployment
`us-east-1`
`us-east-2`
`us-west-1`
`us-west-2`
`eu-central-1`
`eu-central-2`
`eu-north-1`
`eu-west-1`
`eu-west-2`
`eu-west-3`
`eu-south-1`
`eu-south-2`
`sa-east-1`
`ap-northeast-1`
`ap-northeast-2`
`ap-northeast-3`
`ap-south-1`
`ap-south-2`
`ap-southeast-1`
`ap-southeast-2`
`ap-southeast-3`
`ap-southeast-4`
`ap-east-1`
`ca-central-1`
`ca-west-1`
`af-south-1`
`me-south-1`
`me-central-1`
`il-central-1`

3. Specify stack details

Specify the stack details as per the table below, check the checkboxes and select Create stack.

Parameter	Description	Required/Default
`logzioListener`	Logz.io listener URL for your region. (For more details, see the regions page. e.g., `https://listener.logz.io:8053`	Required
`logzioToken`	Your Logz.io metrics shipping token.	Required
`awsNamespaces`	Comma-separated list of AWS namespaces to monitor. See this list of namespaces. Use value `all-namespaces` to automatically add all namespaces.	At least one of `awsNamespaces` or `customNamespace` is required
`customNamespace`	A custom namespace for CloudWatch metrics. Used to specify a namespace unique to your setup, separate from the standard AWS namespaces.	At least one of `awsNamespaces` or `customNamespace` is required
`logzioDestination`	Your Logz.io destination URL. Choose the relevant endpoint from the drop down list based on your Logz.io account region.	Required
`httpEndpointDestinationIntervalInSeconds`	Buffer time in seconds before Kinesis Data Firehose delivers data.	`60`
`httpEndpointDestinationSizeInMBs`	Buffer size in MBs before Kinesis Data Firehose delivers data.	`5`
`debugMode`	Enable debug mode for detailed logging (true/false).	false

4. View your metrics

Allow some time for data ingestion, then open your Logz.io metrics account.

Install the pre-built dashboard to enhance the observability of your metrics.

To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.

Configure AWS to forward traces to Logz.io

Before you begin, you'll need:

An application instrumented with an OpenTelemetry instrumentation or any other supported instrumentations based on OpenTracing, Zipkin or Jaeger.

Configure the OpenTelemetry Collector

The OpenTelemetry Collector receives traces from the application and exports them to Logz.io.

1. Create AWS SSM Parameter

Go to AWS Systems Manager >> Parameter Store >> Create parameter:

Set Name to logzioOtelConfig.yaml
Keep Type as String and Data type as text
In Value paste the below configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

exporters:
  logzio/traces:
    account_token: "<<TRACING-SHIPPING-TOKEN>>"
    region: "<<LOGZIO_ACCOUNT_REGION_CODE>>"
    headers:
      user-agent: logzio-opentelemetry-traces

processors:
  batch:
  tail_sampling:
    policies:
      [
        {
          name: policy-errors,
          type: status_code,
          status_code: {status_codes: [ERROR]}
        },
        {
          name: policy-slow,
          type: latency,
          latency: {threshold_ms: 1000}
        }, 
        {
          name: policy-random-ok,
          type: probabilistic,
          probabilistic: {sampling_percentage: 10}
        }        
      ]

extensions:
  pprof:
    endpoint: :1777
  zpages:
    endpoint: :55679
  health_check:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [logzio/traces]
  telemetry:
    logs:
      level: info

2. Create Role to allow the ECS task to access the SSM Parameter

Copy the ARN of the SSM parameter that was created in step [1].

Create an IAM Policy and add it to the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:GetParameters",
      "Resource": [
        "<ARN_FROM_STEP_1>"
      ]
    }
  ]
}

Add the policy to your existing task IAM Role or create a new one and attach the policy to it.
If you created a new role for the ECS task, copy the Role ARN, as you'll need it later.

3. Create log groups for your OpenTelemetry Collector in CloudWatch.

You can either do so from AWS Console, or via AWS CLI:

aws logs create-log-group --log-group-name /ecs/otel-collector

Define ECS Task

Create a task definition for ECS that defines both your application container and the OpenTelemetry Collector container:

{
  "family": "<APP>-app-task",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/ecsTaskExecutionRole",
  "taskRoleArn": "<TASK_ROLE_ARN>",
  "containerDefinitions": [
    {
        <Your existing application container definitions>
    },
    {
      "name": "otel-collector",
      "image": "otel/opentelemetry-collector-contrib",
      "cpu": 128,      
      "essential": false,
      "command": [
          "--config",
          "env:OTEL_CONFIG"
      ],
      "environment": [],
      "secrets": [
          {
              "name": "OTEL_CONFIG",
              "valueFrom": "logzioOtelConfig.yaml"
          }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/otel-collector",
          "awslogs-region": "<AWS-REGION>",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

note

Replace:

<AWS-REGION> with your AWS account region name
<AWS_ACCOUNT_ID> with your AWS account ID
<TASK_ROLE_ARN> with the ARN of the role that was created in step [2]

Verify Application and Tracing

After deploying, run your application to generate activity that will create tracing data. Wait a few minutes, then check the Logz.io dashboard to confirm that traces are being sent correctly.

Configure AWS to forward logs to Logz.io​

Automated CloudFormation deployment​

Configure and create your stack​

Run the task​

Check Logz.io for your logs​

Manual deployment (AWS Classic Console)​

Download the task definition JSON​

Configure the task​

Advanced settings (optional)​

Add your task definition​

Run the task​

Check Logz.io for your logs​

Configure AWS to forward metrics to Logz.io​

Configure AWS to forward metrics to Logz.io​

Configure AWS to forward traces to Logz.io​

Configure the OpenTelemetry Collector​

1. Create AWS SSM Parameter​

2. Create Role to allow the ECS task to access the SSM Parameter​

3. Create log groups for your OpenTelemetry Collector in CloudWatch.​

Define ECS Task​

Verify Application and Tracing​

Configure AWS to forward logs to Logz.io

Automated CloudFormation deployment

Configure and create your stack

Run the task

Check Logz.io for your logs

Manual deployment (AWS Classic Console)

Download the task definition JSON

Configure the task

Advanced settings (optional)

Add your task definition

Run the task

Check Logz.io for your logs

Configure AWS to forward metrics to Logz.io

Configure AWS to forward metrics to Logz.io

Configure AWS to forward traces to Logz.io

Configure the OpenTelemetry Collector

1. Create AWS SSM Parameter

2. Create Role to allow the ECS task to access the SSM Parameter

3. Create log groups for your OpenTelemetry Collector in CloudWatch.

Define ECS Task

Verify Application and Tracing