AWS MSK
Logs
Auto-deploy the Stack in the relevant region
This integration will deploy a Firehose connection with your AWS services to forward logs to Logz.io To deploy this project, click the button that matches the region you wish to deploy your Stack to:
| Region | Deployment |
|---|---|
us-east-1 |  |
us-east-2 | |
us-west-1 | |
us-west-2 | |
eu-central-1 | |
eu-north-1 | |
eu-west-1 | |
eu-west-2 | |
eu-west-3 | |
sa-east-1 | |
ap-northeast-1 | |
ap-northeast-2 | |
ap-northeast-3 | |
ap-south-1 | |
ap-southeast-1 | |
ap-southeast-2 | |
ca-central-1 | |
Specify stack details
Specify the stack details as per the table below, check the checkboxes and select Create stack.
| Parameter | Description | Required/Default |
|---|---|---|
logzioToken | The token of the account you want to ship logs to. | Required |
logzioListener | Listener host. | Required |
logzioType | The log type you'll use with this Lambda. This can be a built-in log type, or a custom log type. | logzio_firehose |
services | A comma-seperated list of services you want to collect logs from. Supported options are: apigateway, rds, cloudhsm, cloudtrail, codebuild, connect, elasticbeanstalk, ecs, eks, aws-glue, aws-iot, lambda, macie, amazon-mq. | - |
customLogGroups | A comma-seperated list of custom log groups you want to collect logs from | - |
triggerLambdaTimeout | The amount of seconds that Lambda allows a function to run before stopping it, for the trigger function. | 60 |
triggerLambdaMemory | Trigger function's allocated CPU proportional to the memory configured, in MB. | 512 |
triggerLambdaLogLevel | Log level for the Lambda function. Can be one of: debug, info, warn, error, fatal, panic | info |
httpEndpointDestinationIntervalInSeconds | The length of time, in seconds, that Kinesis Data Firehose buffers incoming data before delivering it to the destination | 60 |
httpEndpointDestinationSizeInMBs | The size of the buffer, in MBs, that Kinesis Data Firehose uses for incoming data before delivering it to the destination | 5 |
AWS limits every log group to have up to 2 subscription filters. If your chosen log group already has 2 subscription filters, the trigger function won't be able to add another one.
Send logs
Give the stack a few minutes to be deployed.
Once new logs are added to your chosen log group, they will be sent to your Logz.io account.
If you've used the services field, you'll have to wait 6 minutes before creating new log groups for your chosen services. This is due to cold start and custom resource invocation, that can cause the Lambda to behave unexpectedly.
Check Logz.io for your logs
Give your logs some time to get from your system to ours, and then open Open Search Dashboards.
If you still don't see your logs, see log shipping troubleshooting.
Metrics
For a much easier and more efficient way to collect and send metrics, consider using the Logz.io telemetry collector.
Deploy this integration to send your Amazon MSK metrics to Logz.io.
This integration creates a Kinesis Data Firehose delivery stream that links to your Amazon MSK metrics stream and then sends the metrics to your Logz.io account. It also creates a Lambda function that adds AWS namespaces to the metric stream, and a Lambda function that collects and ships the resources' tags.
Log in to your Logz.io account and navigate to the current instructions page inside the Logz.io app. Install the pre-built dashboard to enhance the observability of your metrics.
To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.
Before you begin, you'll need:
- An active account with Logz.io
Configure AWS to forward metrics to Logz.io
Create Stack in the relevant region
To deploy this project, click the button that matches the region you wish to deploy your Stack to:
| Region | Deployment |
|---|---|
us-east-1 | |
us-east-2 | |
us-west-1 | |
us-west-2 | |
eu-central-1 | |
eu-north-1 | |
eu-west-1 | |
eu-west-2 | |
eu-west-3 | |
sa-east-1 | |
ap-northeast-1 | |
ap-northeast-2 | |
ap-northeast-3 | |
ap-south-1 | |
ap-southeast-1 | |
ap-southeast-2 | |
ca-central-1 | |
Specify stack details
Specify the stack details as per the table below, check the checkboxes and select Create stack.
| Parameter | Description | Required/Default |
|---|---|---|
logzioListener | The Logz.io listener URL for your region. (For more details, see the regions page. For example - https://listener.logz.io:8053 | Required |
logzioToken | Your Logz.io metrics shipping token. | Required |
awsNamespaces | Comma-separated list of the AWS namespaces you want to monitor. See this list of namespaces. If you want to automatically add all namespaces, use value all-namespaces. | Required |
logzioDestination | Your Logz.io destination URL. | Required |
httpEndpointDestinationIntervalInSeconds | The length of time, in seconds, that Kinesis Data Firehose buffers incoming data before delivering it to the destination. | 60 |
httpEndpointDestinationSizeInMBs | The size of the buffer, in MBs, that Kinesis Data Firehose uses for incoming data before delivering it to the destination. | 5 |
Check Logz.io for your metrics
Give your data some time to get from your system to ours, then log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.
Log in to your Logz.io account and navigate to the current instructions page inside the Logz.io app. Install the pre-built dashboard to enhance the observability of your metrics.
To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.