Skip to main content

Python

Logs​

note

Project's GitHub repo

The Logz.io Python Handler sends logs in bulk over HTTPS to Logz.io, grouping them based on size. If the main thread quits, the handler attempts to send any remaining logs before exiting. If unsuccessful, the logs are saved to the local file system for later retrieval.

Setup Logz.io Python Handler​

Supported versions: Python 3.5 or newer.

Install dependency​

Navigate to your project's folder and run:

pip install logzio-python-handler

For Trace context, install the OpenTelemetry logging instrumentation dependency by running:

pip install logzio-python-handler[opentelemetry-logging]

Configure Python Handler for a standard project​

Replace placeholders with your details. You must configure these parameters by this exact order. i.e. you cannot set Debug to true, without configuring all of the previous parameters as well.

ParameterDescriptionRequired/Default
<< LOG-SHIPPING-TOKEN >>Your Logz.io account log shipping token.Required
<< LOG-TYPE >>Log type, for searching in logz.io.python
<<TIMEOUT>>Time to sleep between draining attempts3
<< LISTENER-HOST >>Logz.io listener host, as described here.https://listener.logz.io:8071
<<DEBUG-FLAG>>Debug flag. If set to True, will print debug messages to stdout.false
<<BACKUP-LOGS>>If set to False, disables the local backup of logs in case of failure.true
<<NETWORK-TIMEOUT>>Network timeout, in seconds, int or float, for sending the logs to logz.io.10
<<RETRY-LIMIT>>Retries number4
<<RETRY-TIMEOUT>>Retry timeout (retry_timeout) in seconds2
[handlers]
keys=LogzioHandler

[handler_LogzioHandler]
class=logzio.handler.LogzioHandler
formatter=logzioFormat

args=('<<LOG-SHIPPING-TOKEN>>', '<<LOG-TYPE>>', <<TIMEOUT>>, 'https://<<LISTENER-HOST>>:8071', <<DEBUG-FLAG>>,<<NETWORKING-TIMEOUT>>,<<RETRY-LIMIT>>,<<RETRY-TIMEOUT>>)

[formatters]
keys=logzioFormat

[loggers]
keys=root

[logger_root]
handlers=LogzioHandler
level=INFO

[formatter_logzioFormat]
format={"additional_field": "value"}

Dictionary configuration:​

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'logzioFormat': {
            'format': '{"additional_field": "value"}',
            'validate': False
        }
    },
    'handlers': {
        'logzio': {
            'class': 'logzio.handler.LogzioHandler',
            'level': 'INFO',
            'formatter': 'logzioFormat',
            'token': '<<LOG-SHIPPING-TOKEN>>',
            'logzio_type': '<<LOG-TYPE>>',
            'logs_drain_timeout': 5,
            'url': 'https://<<LISTENER-HOST>>:8071'
            'retries_no': 4,
            'retry_timeout': 2,
        }
    },
    'loggers': {
        '': {
            'level': 'DEBUG',
            'handlers': ['logzio'],
            'propagate': True
        }
    }
}

Django configuration​

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'logzioFormat': {
            'format': '{"additional_field": "value"}',
            'validate': False
        }
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'DEBUG',
            'formatter': 'verbose'
        },
        'logzio': {
            'class': 'logzio.handler.LogzioHandler',
            'level': 'INFO',
            'formatter': 'logzioFormat',
            'token': '<<LOG-SHIPPING-TOKEN>>',
            'logzio_type': "django",
            'logs_drain_timeout': 5,
            'url': 'https://<<LISTENER-HOST>>:8071'
            'debug': True,
            'network_timeout': 10,
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console', ],
            'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO')
        },
        '': {
            'handlers': ['console', 'logzio'],
            'level': 'INFO'
        }
    }
}

Serverless platforms​

When using a serverless function, import and add LogzioFlusher annotation before your sender function. In the code example below umcomment import and the @LogzioFlusher(logger) annotation line. Next, ensure the Logz.io handler is added to the root logger.

Be sure to replace superAwesomeLogzioLoggers with the name of your logger.

'loggers': {
     'superAwesomeLogzioLogger': {
         'level': 'DEBUG',
         'handlers': ['logzio'],
         'propagate': True
     }
 }

For example:

import logging
import logging.config
# from logzio.flusher import LogzioFlusher
# from logzio.handler import ExtraFieldsLogFilter

# Say I have saved my configuration as a dictionary in a variable named 'LOGGING' - see 'Dict Config' sample section
logging.config.dictConfig(LOGGING)
logger = logging.getLogger('superAwesomeLogzioLogger')

# @LogzioFlusher(logger)
def my_func():
    logger.info('Test log')
    logger.warning('Warning')

    try:
        1/0
    except:
        logger.exception("Supporting exceptions too!")

Dynamic extra fields​

You can dynamically add extra fields to your logs without predefining them in the configuration. This allows each log to have unique extra fields.


logger.info("Test log") 

extra_fields = {"foo":"bar","counter":1}
logger.addFilter(ExtraFieldsLogFilter(extra_fields))
logger.warning("Warning test log") 

error_fields = {"err_msg":"Failed to run due to exception.","status_code":500}
logger.addFilter(ExtraFieldsLogFilter(error_fields))
logger.error("Error test log")  

# If you'd like to remove filters from future logs using the logger.removeFilter option:
logger.removeFilter(ExtraFieldsLogFilter(error_fields))
logger.debug("Debug test log")

To add dynamic metadata to a specific log rather than to the logger, use the "extra" parameter. All key-value pairs in the dictionary passed to "extra" will appear as new fields in Logz.io. Note that you cannot override default fields set by the Python logger (e.g., lineno, thread).

For example:

logger.info('Warning', extra={'extra_key':'extra_value'})

Trace context​

You can correlate your logs with the trace context by installing the OpenTelemetry logging instrumentation dependency:

pip install logzio-python-handler[opentelemetry-logging]

Enable this feature by setting add_context parameter to True in your handler configuration:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'logzioFormat': {
            'format': '{"additional_field": "value"}',
            'validate': False
        }
    },
    'handlers': {
        'logzio': {
            'class': 'logzio.handler.LogzioHandler',
            'level': 'INFO',
            'formatter': 'logzioFormat',
            'token': '<<LOG-SHIPPING-TOKEN>>',
            'logzio_type': '<<LOG-TYPE>>',
            'logs_drain_timeout': 5,
            'url': 'https://<<LISTENER-HOST>>:8071'
            'retries_no': 4,
            'retry_timeout': 2,
            'add_context': True
        }
    },
    'loggers': {
        '': {
            'level': 'DEBUG',
            'handlers': ['logzio'],
            'propagate': True
        }
    }
}

Truncating logs​

To create a Python logging filter that truncates log messages to a specific number of characters before processing, use the following code:

class TruncationLoggerFilter(logging.Filter):
    def __init__(self):
        super(TruncationLoggerFilter, self).__init__()

    def filter(self, record):
        record.msg = record.msg[:32700]
        print(record.msg)
        return True

logger = logging.getLogger("logzio")
logger.addFilter(TruncationLoggerFilter())

The default limit is 32,700, but you can adjust this value as required.

Metrics​

Send custom metrics to Logz.io from your Python application. This example uses OpenTelemetry Python SDK and the OpenTelemetry remote write exporter.

Code configuration setup​

1. Install the snappy c-library

DEB: sudo apt-get install libsnappy-dev

RPM: sudo yum install libsnappy-devel

OSX/Brew: brew install snappy

Windows: pip install python_snappy-0.5-cp36-cp36m-win_amd64.whl

2. Install the exporter and opentelemtry sdk

pip install opentelemetry-exporter-prometheus-remote-write

3. Add instruments to your application

Replace the placeholders in the exporter section to match your specifics.

ParameterDescription
LISTENER-HOSTThe Logz.io Listener URL for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic. Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071. and add http/https protocol (https://listener.logz.io:8053).
PROMETHEUS-METRICS-SHIPPING-TOKENYour Logz.io Prometheus Metrics account token. Replace <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with a token for the Metrics account you want to ship to. Look up your Metrics token.
from time import sleep
from typing import Iterable

from opentelemetry.exporter.prometheus_remote_write import (
    PrometheusRemoteWriteMetricsExporter,
)
from opentelemetry.metrics import (
    CallbackOptions,
    Observation,
    get_meter_provider,
    set_meter_provider,
)
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

# configure the Logz.io listener endpoint and Prometheus metrics account token
exporter = PrometheusRemoteWriteMetricsExporter(
    endpoint="https://<<LISTENER-HOST>>:8053",
    headers={
        "Authorization": "Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>",
    },
)

reader = PeriodicExportingMetricReader(exporter)
provider = MeterProvider(metric_readers=[reader])
set_meter_provider(provider)


def observable_counter_func(options: CallbackOptions) -> Iterable[Observation]:
    yield Observation(1, {})


def observable_up_down_counter_func(
    options: CallbackOptions,
) -> Iterable[Observation]:
    yield Observation(-10, {})


def observable_gauge_func(options: CallbackOptions) -> Iterable[Observation]:
    yield Observation(9, {})


meter = get_meter_provider().get_meter("getting-started", "0.1.2")

# Counter
counter = meter.create_counter("counter")
counter.add(1)

# Async Counter
observable_counter = meter.create_observable_counter(
    "observable_counter",
    [observable_counter_func],
)

# UpDownCounter
updown_counter = meter.create_up_down_counter("updown_counter")
updown_counter.add(1)
updown_counter.add(-5)

# Async UpDownCounter
observable_updown_counter = meter.create_observable_up_down_counter(
    "observable_updown_counter", [observable_up_down_counter_func]
)

# Histogram
histogram = meter.create_histogram("histogram")
histogram.record(99.9)

# Async Gauge
gauge = meter.create_observable_gauge("gauge", [observable_gauge_func])
sleep(6)

Types of metric instruments​

See OpenTelemetry documentation for more details.

NameBehaviorDefault aggregation
CounterMetric value can only go up or be reset to 0, calculated per counter.add(value,labels) request.Sum
UpDownCounterMetric value can arbitrarily increment or decrement, calculated per updowncounter.add(value,labels) request.Sum
ValueRecorderMetric values captured by the valuerecorder.record(value) function, calculated per request.TBD
SumObserverMetric value can only go up or be reset to 0, calculated per push interval.Sum
UpDownSumObserverMetric value can arbitrarily increment or decrement, calculated per push interval.Sum
ValueObserverMetric values captured by the valuerecorder.observe(value) function, calculated per push interval.LastValue
Counter​
# create a counter instrument
counter = meter.create_counter(
    name="MyCounter",
    description="Description of MyCounter",
    unit="1",
    value_type=int
)
# add labels
labels = {
    "dimension": "value"
}
# provide the first data point
counter.add(25, labels)
UpDownCounter​
# create an updowncounter instrument
requests_active = meter.create_updowncounter(
    name="requests_active",
    description="number of active requests",
    unit="1",
    value_type=int,
)
# add labels
labels = {
    "dimension": "value"
}
# provide the first data point
requests_active.add(-2, labels)
ValueRecorder​
# create a valuerecorder instrument
requests_size = meter.create_valuerecorder(
    name="requests_size",
    description="size of requests",
    unit="1",
    value_type=int,
)
# add labels
labels = {
    "dimension": "value"
}
# provide the first data point
requests_size.record(85, labels)
SumObserver​
import psutil
# Callback to gather RAM usage
def get_ram_usage_callback(observer):
    ram_percent = psutil.virtual_memory().percent
    # add labels
    labels = {
        "dimension": "value"
    }
    observer.observe(ram_percent, labels)
# create a sumobserver instrument
meter.register_sumobserver(
    callback=get_ram_usage_callback,
    name="ram_usage",
    description="ram usage",
    unit="1",
    value_type=float,
)
UpDownSumObserver​
# Callback to gather RAM usage
def get_ram_usage_callback(observer):
    ram_percent = psutil.virtual_memory().percent
    # add labels
    labels = {
        "dimension": "value"
    }
    observer.observe(ram_percent, labels)
# create an updownsumobserver instrument
meter.register_updownsumobserver(
    callback=get_ram_usage_callback,
    name="ram_usage",
    description="ram usage",
    unit="1",
    value_type=float,
)
ValueObserver​
import psutil
def get_cpu_usage_callback(observer):
    for (number, percent) in enumerate(psutil.cpu_percent(percpu=True)):
        labels = {"cpu_number": str(number)}
        observer.observe(percent, labels)
# create a valueobserver instrument
meter.register_valueobserver(
    callback=get_cpu_usage_callback,
    name="cpu_percent",
    description="per-cpu usage",
    unit="1",
    value_type=float,
)

5. Check Logz.io for your metrics

Allow some time for your data to transfer. Then log in to your Logz.io Metrics account and open the Metrics dashboard.

Traces​

Deploy this integration to enable automatic instrumentation of your Python application using OpenTelemetry.

Architecture overview​

This integration includes:

  • Installing the OpenTelemetry Python instrumentation packages on your application host
  • Installing the OpenTelemetry collector with Logz.io exporter
  • Running your Python application in conjunction with the OpenTelemetry instrumentation

On deployment, the Python instrumentation automatically captures spans from your application and forwards them to the collector, which exports the data to your Logz.io account.

Local host Python application auto instrumentation​

Requirements:

  • A Python application without instrumentation
  • An active Logz.io account
  • Port 4317 available on your host system
  • A name defined for your tracing service
note

This integration uses OpenTelemetry Collector Contrib, not the OpenTelemetry Collector Core.

Install OpenTelemetry components for Python​

pip3 install opentelemetry-distro
pip3 install opentelemetry-instrumentation
opentelemetry-bootstrap --action=install
pip3 install opentelemetry-exporter-otlp

Set environment variables​

After installation, configure the exporter with this command:

export OTEL_TRACES_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES="service.name=<<YOUR-SERVICE-NAME>>"

Download and configure OpenTelemetry collector​

Create a directory on your Python application and download the relevant OpenTelemetry collector. Create a config.yaml with the following parameters:

  • Replace <<TRACING-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LOGZIO_ACCOUNT_REGION_CODE>> with the applicable region code.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

exporters:
  logzio/traces:
    account_token: "<<TRACING-SHIPPING-TOKEN>>"
    region: "<<LOGZIO_ACCOUNT_REGION_CODE>>"
    headers:
      user-agent: logzio-opentelemetry-traces

processors:
  batch:
  tail_sampling:
    policies:
      [
        {
          name: policy-errors,
          type: status_code,
          status_code: {status_codes: [ERROR]}
        },
        {
          name: policy-slow,
          type: latency,
          latency: {threshold_ms: 1000}
        }, 
        {
          name: policy-random-ok,
          type: probabilistic,
          probabilistic: {sampling_percentage: 10}
        }        
      ]

extensions:
  pprof:
    endpoint: :1777
  zpages:
    endpoint: :55679
  health_check:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [logzio/traces]
  telemetry:
    logs:
      level: info

tail_sampling defines which traces to sample after all spans in a request are completed. By default, it collects all traces with an error span, traces slower than 1000 ms, and 10% of all other traces.

Additional policy configurations can be added to the processor. For more details, refer to the OpenTelemetry Documentation.

The configurable parameters in the Logz.io default configuration are:

ParameterDescriptionDefault
threshold_msThreshold for the span latency - traces slower than this value will be included.1000
sampling_percentagePercentage of traces to sample using the probabilistic policy.10

Start the collector​

Run:

<path/to>/otelcontribcol_<VERSION-NAME> --config ./config.yaml
  • Replace <path/to> with the collector's directory.
  • Replace <VERSION-NAME> with the version name, e.g. otelcontribcol_darwin_amd64.

Run OpenTelemetry with your Python application​

Run this code from the directory of your Python application script:

opentelemetry-instrument python3 <YOUR-APPLICATION-SCRIPT>.py

Replace <YOUR-APPLICATION-SCRIPT> with your Python application script name.

Viewing Traces in Logz.io​

Give your traces time to process, after which they'll be available in your Tracing dashboard.

Troubleshooting​