Add alerts to graphs in your Metrics dashboards to be notified when values are outside of their acceptable range.
Alerts can be configured to send notifications via your preferred channels and email. You can also add a condition so an alert holds off for a certain time before it sends out notifications. During this time, the alert will appear on the graph in your dashboard as orange, before it switches to red. This will prevent triggering alerts for short-lived issues that are resolved on their own and help reduce alert fatigue.
If you’re running multiple servers for load balancing purposes, you’ll be happy to know that Logz.io Metrics won’t send duplicte alerts. A deduplication mechanism protects against that.
On this page
- Adding Alerts
Navigate to the Logz.io Infrastructure Monitoring Metrics tab.
To add an alert
Pick your dashboard
Before you begin, you’ll need to make sure you have permissions to edit the dashboard.
If you are using any of the pre-configured dashboards provided by Logz.io, you’ll need to duplicate it first. (They are read-only, by default.)
Click the gear in the top menu, then select Save as > Save . If you don’t rename the dashboard, it will have the same name as the original dashboard, with Copy appended at the end.
Edit your graph
Select the graph panel you’ll be using for the alert. Only the graph panel is supported at this point.
Hover over the graph panel name, and click Edit.
Alerts can’t run if you have any variables in your graph, so you’ll need to remove them first.
One option, is to duplicate the query and remove the variables from it. That way you can add alerts to the same graph you use to monitor your dashboard.
Click the copy icon to duplicate the query. Then click to hide the query in the dashboard. You’ll only be using it for the alert, and don’t want it to appear on the graph.
Remove any variables. Variables are indicated by a
$ sign, as seen in the example below. These will block you from saving your alert.
Select a fixed source and erase the query or rewrite the query so it doesn’t contain variables. You can also leave the query blank.
Add an alert
Click the bell in the left panel > Create alert to add an alert.
Next, configure the alert.
- In the Name field, name your alert. Preferrably, use a name that will be clear to anyone on the team and future newcomers.
- Set an Evaluation interval to decide how often the rule should be evaluated.
- In the field For, set a time for the “silent alert”. This is the time that the alert can go off before it sends a notification. For example, you could send an alert if the rule is breached for 5 minutes or longer. Use this to allow for some leeway, or “wiggle room”.
Next, set your alert conditions.
- First, make sure the condition points to the right query. That’s the one that doesn’t contain variables.
- Define the rule’s condition. You can type in the value or drag the heart directly on the graph to make your selection.
- You can click the condition to decide whether to set it for values above/below a threshold, within/outside a range, or when it has no value.
- You have the option to set up multiple conditions and queries for more advanced alerts.
Grafana tags for alert Severity levels
Use Grafana tags as a Severity field for your alerts when integrating Grafana with PagerDuty as an alert notification endpoint. Follow the link for more information on the syntax for setting up Grafana tags for PagerDuty.
If you want to send notifications or emails when the alert is triggered, choose notification endpoints.
Endpoints can be Slack channels, email addresses, or other places your team communicates.
To limit how often recipients are notified - you’ll need to scroll up to the Evaluation interval > For rule above. (See the instructions above).
When notifications are suppressed, your Metrics dashboard will show the orange marker on the graph, indicating that an alert is triggered, but not yet reached its notification threshold.
Additional alert notifications
For better alerts management and to help you focus only on active alerts, when a tracked metric returns to its accepted values (below the current alert threshold), you’ll get notified automatically that the triggered alert was resolved.
Resolved alert notifications are available for Slack, PagerDuty, Opsgenie, and email notification endpoints.
See Opsgenie notifications for resolved metrics alerts for more information.
To help you reduce the noise associated with receiving notifications for multiple instances of the same alert, you can use the Acknowledge status to assign the alert to a team member or to yourself, as well as manage alerts that are already under investigation (as determined by the ACK policy you manage in your notification software).
When you receive an alert with the same alert ID, if the alert has the Acknowledge status, your notification endpoint will not repeat the notification.
The Acknowledge status is available for the PagerDuty notification endpoint.
Click Save in the top-right panel to save your dashboard (and the alert). If the thresholds are passed and the alert is triggered, you’ll see the markers on your dashboard and receive your notifications.