AWS Kinesis Data Firehose
Logs
This project deploys instrumentation that allows shipping Cloudwatch logs to Logz.io, with a Firehose Delivery Stream. It uses a Cloudformation template to create a Stack that deploys:
- Firehose Delivery Stream with Logz.io as the stream's destination.
- Lambda function that adds Subscription Filters to Cloudwatch Log Groups, as defined by user's input.
- Roles, log groups, and other resources that are necessary for this instrumentation.
If you want to send logs from specific log groups, use customLogGroups
instead of services
. Since specifying services
will automatically send all logs from those services, regardless of any custom log groups you define.
Auto-deploy the Stack
To deploy this project, click the button that matches the region you wish to deploy your Stack to:
Specify stack details
Specify the stack details as per the table below, check the checkboxes and select Create stack.
Parameter | Description | Required/Default |
---|---|---|
logzioToken | The token of the account you want to ship logs to. | Required |
logzioListener | Listener host. | Required |
logzioType | The log type you'll use with this Lambda. This can be a built-in log type, or a custom log type. | logzio_firehose |
services | A comma-seperated list of services you want to collect logs from. Supported options are: apigateway , rds , cloudhsm , cloudtrail , codebuild , connect , elasticbeanstalk , ecs , eks , aws-glue , aws-iot , lambda , macie , amazon-mq . | - |
customLogGroups | A comma-separated list of custom log groups to collect logs from, or the ARN of the Secret parameter (explanation below) storing the log groups list if it exceeds 4096 characters. | - |
useCustomLogGroupsFromSecret | If you want to provide list of customLogGroups which exceeds 4096 characters, set to true and configure your customLogGroups as defined below. | false |
triggerLambdaTimeout | The amount of seconds that Lambda allows a function to run before stopping it, for the trigger function. | 60 |
triggerLambdaMemory | Trigger function's allocated CPU proportional to the memory configured, in MB. | 512 |
triggerLambdaLogLevel | Log level for the Lambda function. Can be one of: debug , info , warn , error , fatal , panic | info |
httpEndpointDestinationIntervalInSeconds | The length of time, in seconds, that Kinesis Data Firehose buffers incoming data before delivering it to the destination | 60 |
httpEndpointDestinationSizeInMBs | The size of the buffer, in MBs, that Kinesis Data Firehose uses for incoming data before delivering it to the destination | 5 |
AWS limits every log group to have up to 2 subscription filters. If your chosen log group already has 2 subscription filters, the trigger function won't be able to add another one.
Custom Log Group list exceeds 4096 characters limit
If your customLogGroups
list exceeds the 4096 characters limit, follow the below steps:
- Open AWS Secret Manager
- Click
Store a new secret
- Choose
Other type of secret
- For
key
uselogzioCustomLogGroups
- In
value
store your comma-separated custom log groups list - Name your secret, for example as
LogzioCustomLogGroups
- Copy the new secret's ARN
- Choose
- In your stack, Set:
customLogGroups
to your secret ARN that you copied in step 2useCustomLogGroupsFromSecret
totrue
Send logs
Give the stack a few minutes to be deployed.
Once new logs are added to your chosen log group, they will be sent to your Logz.io account.
If you've used the services
field, you'll have to wait 6 minutes before creating new log groups for your chosen services. This is due to cold start and custom resource invocation, that can cause the Lambda to behave unexpectedly.
Check Logz.io for your logs
Give your logs some time to get from your system to ours, and then open Open Search Dashboards.
If you still don't see your logs, see log shipping troubleshooting.
Metrics
For a much easier and more efficient way to collect and send metrics, consider using the Logz.io telemetry collector.
Deploy this integration to send your Amazon Kinesis Data Firehose metrics to Logz.io.
This integration creates a Kinesis Data Firehose delivery stream that links to your Amazon Kinesis Data Firehose metrics stream and then sends the metrics to your Logz.io account. It also creates a Lambda function that adds AWS namespaces to the metric stream, and a Lambda function that collects and ships the resources' tags.
Before you begin, you'll need:
- An active Logz.io account
Configure AWS to forward metrics to Logz.io
1. Set the required minimum IAM permissions
configured the minimum required IAM permissions as follows:
- Amazon S3:
s3:CreateBucket
s3:DeleteBucket
s3:PutObject
s3:GetObject
s3:DeleteObject
s3:ListBucket
s3:AbortMultipartUpload
s3:GetBucketLocation
- AWS Lambda:
lambda:CreateFunction
lambda:DeleteFunction
lambda:InvokeFunction
lambda:GetFunction
lambda:UpdateFunctionCode
lambda:UpdateFunctionConfiguration
lambda:AddPermission
lambda:RemovePermission
lambda:ListFunctions
- Amazon CloudWatch:
cloudwatch:PutMetricData
cloudwatch:PutMetricStream
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents
logs:DeleteLogGroup
logs:DeleteLogStream
- AWS Kinesis Firehose:
firehose:CreateDeliveryStream
firehose:DeleteDeliveryStream
firehose:PutRecord
firehose:PutRecordBatch
- IAM:
iam:PassRole
iam:CreateRole
iam:DeleteRole
iam:AttachRolePolicy
iam:DetachRolePolicy
iam:GetRole
iam:CreatePolicy
iam:DeletePolicy
iam:GetPolicy
- Amazon CloudFormation:
cloudformation:CreateStack
cloudformation:DeleteStack
cloudformation:UpdateStack
cloudformation:DescribeStacks
cloudformation:DescribeStackEvents
cloudformation:ListStackResources
2. Create Stack in the relevant region
To deploy this project, click the button that matches the region you wish to deploy your Stack to:
3. Specify stack details
Specify the stack details as per the table below, check the checkboxes and select Create stack.
Parameter | Description | Required/Default |
---|---|---|
logzioListener | Logz.io listener URL for your region. (For more details, see the regions page. e.g., https://listener.logz.io:8053 | Required |
logzioToken | Your Logz.io metrics shipping token. | Required |
awsNamespaces | Comma-separated list of AWS namespaces to monitor. See this list of namespaces. Use value all-namespaces to automatically add all namespaces. | At least one of awsNamespaces or customNamespace is required |
customNamespace | A custom namespace for CloudWatch metrics. Used to specify a namespace unique to your setup, separate from the standard AWS namespaces. | At least one of awsNamespaces or customNamespace is required |
logzioDestination | Your Logz.io destination URL. Choose the relevant endpoint from the drop down list based on your Logz.io account region. | Required |
httpEndpointDestinationIntervalInSeconds | Buffer time in seconds before Kinesis Data Firehose delivers data. | 60 |
httpEndpointDestinationSizeInMBs | Buffer size in MBs before Kinesis Data Firehose delivers data. | 5 |
debugMode | Enable debug mode for detailed logging (true/false). | false |
4. View your metrics
Allow some time for data ingestion, then open your Logz.io metrics account.
Install the pre-built dashboard to enhance the observability of your metrics.
To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.