Skip to main content

Hashicorp Consul

This project lets you configure the OpenTelemetry collector to send your Prometheus-format metrics from Hashicorp Consul to Logz.io.

Before you begin, you'll need:

  • Consul installed on your host
  • Golang

Update Consul configuration​

  1. Navigate to the Consul server and create a new file called prometheus.json under the Consul config library at /etc/consul.d.

  2. Add the following telemetry stanza to the prometheus.json:

    {
      "telemetry": {
        "disable_hostname": true,
        "prometheus_retention_time": "72h"

      }
    }

Check the Consul server’s metrics format​

Check that the Consul server’s metrics are in Prometheus format at this endpoint:

http://127.0.0.1:8500/v1/agent/metrics?format=prometheus

Download OpenTelemetry collector​

note

If you already have OpenTelemetry, proceed to the next step.

Create a dedicated directory on your host and download the OpenTelemetry collector that is relevant to the operating system of your host.

After downloading the collector, create a configuration file config.yaml.

Configure the receivers​

Open the configuration file and make sure that it states the receivers required for your source:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'consul-metrics'
          scrape_interval: 15s
          metrics_path: "/v1/agent/metrics"
          static_configs:
            - targets: ['localhost:8500']
  hostmetrics:
    collection_interval: 1m
    scrapers:
      cpu: ##All execpt Mac
      disk: ##All execpt Mac
      load:
      filesystem:
      memory:
      network:
      process: ##Linux & Windows

Configure the processors​

In the same configuration file, add the following to the processors section:

processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

Configure the exporters​

In the same configuration file, add the following to the exporters section:

exporters:
  prometheusremotewrite:
    endpoint: https://<<LISTENER-HOST>>:8053
    headers:
      Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>
    target_info:
        enabled: false

Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):

  • Replace <<LISTENER-HOST>> with the Logz.io Listener URL for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic.
  • Replace <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with a token for the Metrics account you want to ship to. Look up your Metrics token.

Configure the service pipeline​

In the service section of the configuration file, add the following configuration

service:
  pipelines:
    metrics:
      receivers: [prometheus,hostmetrics]
      processors: [resourcedetection/system]
      exporters:
        - prometheusremotewrite
  telemetry:
    logs:
      level: "info"

Start the collector​

Run the following command:

<path/to>/otelcol-contrib --config ./config.yaml
  • Replace <path/to> with the path to the directory where you downloaded the collector. If the name of your configuration file is different to config, adjust name in the command accordingly.

Check Logz.io for your metrics​

Install the pre-built dashboard to enhance the observability of your metrics.

To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.

Optional - Add basic filtering​

You can use the filter processor to remove telemetry data based on OTTL (OpenTelemetry Transformation Language) rules. Any telemetry that meets at least one of the specified conditions will be dropped.

For example, to allow span data from services app1, app2, and app3 and drop data from all other services:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - |
        resource.attributes["service.name"] != "app1" and
        resource.attributes["service.name"] != "app2" and
        resource.attributes["service.name"] != "app3"
    metrics:
      datapoint:
        - metric.name == "k8s.pod.phase" and value_int == 4
    logs:
      log_record:
        - 'IsMatch(body, ".*password.*")'
        - 'severity_number < SEVERITY_NUMBER_WARN'

To only drop spans from a service called service1 while keeping all other spans:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - resource.attributes["service.name"] == "service1"

Learn more about filtering options in OpenTelemetry's Transforming telemetry documentation.