Skip to main content

Logz.io API Fetcher Configuration Guide

The Logz.io API Fetcher supports both auth and OAuth APIs and includes specific implementations for Azure Graph, Office365 Message Trace reports, Cloudflare and 1Password.

This guide outlines the steps for configuring the Logz.io API Fetcher to fetch and send data to Logz.io. Our aim is to develop the API Fetcher as a generic tool capable of fetching data from any API endpoint. However, this presents significant challenges. If you encounter difficulties configuring the API Fetcher with a particular API endpoint, please reach out to our support team for assistance.

Below is a sample configuration template, as found in our documentation and on GitHub:

logzio:
url: https://<<LISTENER-HOST>>:8071
token: <<LOG-SHIPPING-TOKEN>>

apis:
- name: azure graph example
type: azure_graph
azure_ad_tenant_id: <<AZURE_AD_TENANT_ID>>
azure_ad_client_id: <<AZURE_AD_CLIENT_ID>>
azure_ad_secret_value: <<AZURE_AD_SECRET_VALUE>>
data_request:
url: https://graph.microsoft.com/v1.0/auditLogs/signIns
additional_fields:
type: azure_graph
field_to_add_to_my_logs: 123
scrape_interval: 1
days_back_fetch: 30

- name: mail reports example
type: azure_mail_reports
azure_ad_tenant_id: <<AZURE_AD_TENANT_ID>>
azure_ad_client_id: <<AZURE_AD_CLIENT_ID>>
azure_ad_secret_value: <<AZURE_AD_SECRET_VALUE>>
data_request:
url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace
additional_fields:
type: azure_mail_reports
scrape_interval: 60 # for mail reports we suggest no less than 60 minutes
days_back_fetch: 8 # for mail reports we suggest up to 8 days

- name: cloudflare example
type: cloudflare
cloudflare_account_id: <<CLOUDFLARE_ACCOUNT_ID>>
cloudflare_bearer_token: <<CLOUDFLARE_BEARER_TOKEN>>
url: https://api.cloudflare.com/client/v4/accounts/{account_id}/alerting/v3/history
next_url: https://api.cloudflare.com/client/v4/accounts/{account_id}/alerting/v3/history?since={res.result.[0].sent}
days_back_fetch: 7
scrape_interval: 5
additional_fields:
type: cloudflare

- name: 1Password example
type: 1password
onepassword_bearer_token: <<1PASSWORD_BEARER_TOKEN>>
url: https://events.1password.com/api/v1/auditevents
method: POST
days_back_fetch: 7
scrape_interval: 5
additional_fields:
type: 1password

- name: general example
type: general
url: https://first/request/url
headers:
CONTENT-TYPE: application/json
another-header: XXX
body: {
"size": 1000
}
method: POST
additional_fields:
type: my_fetcher
another_field: 123
pagination:
type: url
url_format: ?page={res.info.page+1}
update_first_url: True
stop_indication:
field: result
condition: empty
response_data_path: result

Configuration

Create a local config file config.yaml.

Add Your Logz.io Listener and Token

Insert your Logz.io listener URL and token into the config to begin the setup process.

logzio:
url: https://<<LISTENER-HOST>>:8071
token: <<LOG-SHIPPING-TOKEN>>

Your Logz.io log shipping token directs the data securely to your Logz.io Log Management account. The default token is auto-populated in the examples when you're logged into the Logz.io app as an Admin. Manage your tokens.

Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.

Configure your APIs

General API Settings

Configuration

Parameter NameDescriptionRequired/OptionalDefault
nameName of the API (custom name)Optionalthe defined url
urlThe request URLRequired-
headersThe request HeadersOptional{}
bodyThe request bodyOptional-
methodThe request method (GET or POST)OptionalGET
paginationPagination settings if needed (see options below)Optional-
next_urlIf needed to update the URL in next requests based on the last response. Supports using variables (see below)Optional-
response_data_pathThe path to the data inside the responseOptionalresponse root
additional_fieldsAdditional custom fields to add to the logs before sending to logzioOptionalAdd type as api-fetcher
scrape_intervalTime interval to wait between runs (unit: minutes)Optional1 (minute)

Pagination Configuration Options

If needed, you can configure pagination.

Parameter NameDescriptionRequired/OptionalDefault
typeThe pagination type (url, body or headers)Required-
url_formatIf pagination type is url, configure the URL format used for the pagination. Supports using variables (see below).Required if pagination type is url-
update_first_urlTrue or False; If pagination type is url, and it's required to append new params to the first request URL and not reset it completely.Optional if pagination type is urlFalse
headers_formatIf pagination type is headers, configure the headers format used for the pagination. Supports using variables (see below).Required if pagination type is headers-
body_formatIf pagination type is body, configure the body format used for the pagination. Supports using variables (see below).Required if pagination type is body-
stop_indicationWhen should the pagination end based on the response. (see options below).Optional (if not defined will stop on max_calls)-
max_callsMax calls that the pagination can make. (Supports up to 1000)Optional1000

Pagination Stop Indication Configuration

Parameter NameDescriptionRequired/OptionalDefault
fieldThe name of the field in the response body, to search the stop indication atRequired-
conditionThe stop condition (empty, equals or contains)Required-
valueIf condition is equals or contains, the value of the field that we should stop atRequired if condition is equals or contains-

Using Variables

Using variables allows taking values from the response of the first request, to structure the request after it.
Mathematical operations + and - are supported, to add or reduce a number from the variable value.

Use case examples for variable usage:

  1. Update a date filter at every call
  2. Update a page number in pagination

To use variables:

  • Wrap the variable name in curly brackets
  • Provide the full path to that variable in the response
  • Add res. prefix to the path.

Example: Say this is my response:

{
"field": "value",
"another_field": {
"nested": 123
},
"num_arr": [1, 2, 3],
"obj_arr": [
{
"field2": 345
},
{
"field2": 567
}
]
}

Paths to fields values are structured like so:

  • {res.field} = "value"
  • {res.another_field.nested} = 123
  • {res.num_arr.[2]} = 3
  • {res.obj_arr.[0].field2} = 345

Using the fields values in the next_url for example like the below:

next_url: https://logz.io/{res.field}/{res.obj_arr[0].field2}

Would update the URL at every call to have the value of the given fields from the response. In our example the url for the next call would be:

https://logz.io/value/345

And in the call after it, it would update again according to the response and the next_url structure, and so on.

Launch the Docker Container

Use Docker to run the Logz.io API Fetcher with the provided command in the path where you saved your config.yaml:

docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher
info

To run in Debug mode add --level flag to the command:

docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher \
--level DEBUG

Available Options: INFO, WARN, ERROR, DEBUG

Stop the Docker container

When you stop the container, the code will run until the iteration is completed. To make sure it will finish the iteration on time, please give it a grace period of 30 seconds when you run the docker stop command.

docker stop -t 30 logzio-api-fetcher