This project deploys instrumentation that allows shipping Cloudwatch logs to, with a Firehose Delivery Stream. It uses a Cloudformation template to create a Stack that deploys:

  • Firehose Delivery Stream with as the stream’s destination.
  • Lambda function that adds Subscription Filters to Cloudwatch Log Groups, as defined by user’s input.
  • Roles, log groups, and other resources that are necessary for this instrumentation.
Auto-deploy the Stack

To deploy this project, click the button that matches the region you wish to deploy your Stack to:

Region Deployment
us-east-1 Deploy to AWS
us-east-2 Deploy to AWS
us-west-1 Deploy to AWS
us-west-2 Deploy to AWS
eu-central-1 Deploy to AWS
eu-north-1 Deploy to AWS
eu-west-1 Deploy to AWS
eu-west-2 Deploy to AWS
eu-west-3 Deploy to AWS
sa-east-1 Deploy to AWS
ap-northeast-1 Deploy to AWS
ap-northeast-2 Deploy to AWS
ap-northeast-3 Deploy to AWS
ap-south-1 Deploy to AWS
ap-southeast-1 Deploy to AWS
ap-southeast-2 Deploy to AWS
ca-central-1 Deploy to AWS
Specify stack details

Specify the stack details as per the table below, check the checkboxes and select Create stack.

Parameter Description Required/Default
logzioToken The token of the account you want to ship logs to. Required
logzioListener Listener host. Required
logzioType The log type you’ll use with this Lambda. This can be a built-in log type, or a custom log type. logzio_firehose
services A comma-seperated list of services you want to collect logs from. Supported options are: apigateway, rds, cloudhsm, cloudtrail, codebuild, connect, elasticbeanstalk, ecs, eks, aws-glue, aws-iot, lambda, macie, amazon-mq. -
customLogGroups A comma-seperated list of custom log groups you want to collect logs from -
triggerLambdaTimeout The amount of seconds that Lambda allows a function to run before stopping it, for the trigger function. 60
triggerLambdaMemory Trigger function’s allocated CPU proportional to the memory configured, in MB. 512
triggerLambdaLogLevel Log level for the Lambda function. Can be one of: debug, info, warn, error, fatal, panic info
httpEndpointDestinationIntervalInSeconds The length of time, in seconds, that Kinesis Data Firehose buffers incoming data before delivering it to the destination 60
httpEndpointDestinationSizeInMBs The size of the buffer, in MBs, that Kinesis Data Firehose uses for incoming data before delivering it to the destination 5

AWS limits every log group to have up to 2 subscription filters. If your chosen log group already has 2 subscription filters, the trigger function won’t be able to add another one.

Send logs

Give the stack a few minutes to be deployed.

Once new logs are added to your chosen log group, they will be sent to your account.

If you’ve used the services field, you’ll have to wait 6 minutes before creating new log groups for your chosen services. This is due to cold start and custom resource invocation, that can cause the Lambda to behave unexpectedly.

Check for your logs

Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don’t see your logs, see log shipping troubleshooting.