Skip to main content

Logz.io API Fetcher Configuration Guide

The Logz.io API Fetcher supports both auth and OAuth APIs and includes specific implementations for Azure Graph, Office365 Message Trace reports, Cloudflare and 1Password.

This guide outlines the steps for configuring the Logz.io API Fetcher to fetch and send data to Logz.io. Our aim is to develop the API Fetcher as a generic tool capable of fetching data from any API endpoint. However, this presents significant challenges. If you encounter difficulties configuring the API Fetcher with a particular API endpoint, please reach out to our support team for assistance.

Below is a sample configuration template, as found in our documentation and on GitHub:

logzio:
  url: https://<<LISTENER-HOST>>:8071
  token: <<LOG-SHIPPING-TOKEN>>

apis:
  - name: azure graph example
    type: azure_graph
    azure_ad_tenant_id: <<AZURE_AD_TENANT_ID>>
    azure_ad_client_id: <<AZURE_AD_CLIENT_ID>>
    azure_ad_secret_value: <<AZURE_AD_SECRET_VALUE>>
    data_request:
      url: https://graph.microsoft.com/v1.0/auditLogs/signIns
    additional_fields:
      type: azure_graph
      field_to_add_to_my_logs: 123
    scrape_interval: 1
    days_back_fetch: 30

  - name: mail reports example
    type: azure_mail_reports
    azure_ad_tenant_id: <<AZURE_AD_TENANT_ID>>
    azure_ad_client_id: <<AZURE_AD_CLIENT_ID>>
    azure_ad_secret_value: <<AZURE_AD_SECRET_VALUE>>
    data_request:
      url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace
    additional_fields:
      type: azure_mail_reports
    scrape_interval: 60  # for mail reports we suggest no less than 60 minutes
    days_back_fetch: 8  # for mail reports we suggest up to 8 days

  - name: cloudflare example
    type: cloudflare
    cloudflare_account_id: <<CLOUDFLARE_ACCOUNT_ID>>
    cloudflare_bearer_token: <<CLOUDFLARE_BEARER_TOKEN>>
    url: https://api.cloudflare.com/client/v4/accounts/alerting/v3/history
    next_url: https://api.cloudflare.com/client/v4/accounts/{account_id}/alerting/v3/history?since={res.result.[0].sent}
    days_back_fetch: 7
    scrape_interval: 5
    additional_fields:
      type: cloudflare

  - name: 1Password example
    type: 1password
    onepassword_bearer_token: <<1PASSWORD_BEARER_TOKEN>>
    url: https://events.1password.com/api/v1/auditevents
    method: POST
    days_back_fetch: 7
    scrape_interval: 5
    additional_fields:
      type: 1password

  - name: Dockerhub audit logs example
    type: dockerhub
    dockerhub_token: <<DOCKERHUB_PASSWORD>>
    dockerhub_user: <<DOCKERHUB_USERNAME>>
    url: https://hub.docker.com/v2/auditlogs/<<dockerhub_account>>
    next_url: https://hub.docker.com/v2/auditlogs/logzio?from={res.logs.[0].timestamp}
    days_back_fetch: 7
    scrape_interval: 1
    refresh_token_interval: 20
    additional_fields:
      type: dockerhub-audit
      eventType: auditevents

  - name: google activity example
    type: google_activity
    google_ws_sa_file_name: credentials_file.json
    google_ws_delegated_account: user@example.com
    application_name: login
    additional_fields:
      type: google_activity
    days_back_fetch: 7
    scrape_interval: 5

  - name: general example
    type: general
    url: https://first/request/url
    headers:
      CONTENT-TYPE: application/json
      another-header: XXX
    body: {
            "size": 1000
          }
    method: POST
    additional_fields:
      type: my_fetcher
      another_field: 123
    pagination:
      type: url
      url_format: ?page={res.info.page+1}
      update_first_url: True
      stop_indication:
        field: result
        condition: empty
    response_data_path: result
note

You can customize the endpoints to collect data by adding or modifying the configurations under the apis section. Refer to the relevant API documentation for more details.

Configuration​

Create a local config file config.yaml.

Add Your Logz.io Listener and Token​

Insert your Logz.io listener URL and token into the config to begin the setup process.

logzio:
  url: https://<<LISTENER-HOST>>:8071 
  token: <<LOG-SHIPPING-TOKEN>>

Your Logz.io log shipping token directs the data securely to your Logz.io Log Management account. The default token is auto-populated in the examples when you're logged into the Logz.io app as an Admin. Manage your tokens.

Replace <<LISTENER-HOST>> with the host for your region. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.

Configure your APIs​

General API Settings​

To configure a general API, set type to general in the API configuration.

Parameter NameDescriptionRequired/OptionalDefault
nameName of the API (custom name)Optionalthe defined url
urlThe request URLRequired-
headersThe request HeadersOptional{}
bodyThe request bodyOptional-
methodThe request method (GET or POST)OptionalGET
paginationPagination settings if needed (see options below)Optional-
next_urlIf needed to update the URL in next requests based on the last response. Supports using variables (see below)Optional-
response_data_pathThe path to the data inside the responseOptionalresponse root
additional_fieldsAdditional custom fields to add to the logs before sending to Logz.ioOptionalAdd type as api-fetcher
scrape_intervalTime interval to wait between runs (unit: minutes)Optional1 (minute)

Pagination Configuration Options​

If needed, you can configure pagination.

Parameter NameDescriptionRequired/OptionalDefault
typeThe pagination type (url, body or headers)Required-
url_formatIf pagination type is url, configure the URL format used for the pagination. Supports using variables (see below).Required if pagination type is url-
update_first_urlTrue or False; If pagination type is url, and it's required to append new params to the first request URL and not reset it completely.Optional if pagination type is urlFalse
headers_formatIf pagination type is headers, configure the headers format used for the pagination. Supports using variables (see below).Required if pagination type is headers-
body_formatIf pagination type is body, configure the body format used for the pagination. Supports using variables (see below).Required if pagination type is body-
stop_indicationWhen should the pagination end based on the response. (see options below).Optional (if not defined will stop on max_calls)-
max_callsMax calls that the pagination can make. (Supports up to 1000)Optional1000

Pagination Stop Indication Configuration​

Parameter NameDescriptionRequired/OptionalDefault
fieldThe name of the field in the response body, to search the stop indication atRequired-
conditionThe stop condition (empty, equals or contains)Required-
valueIf condition is equals or contains, the value of the field that we should stop atRequired if condition is equals or contains-

Using Variables​

Using variables allows taking values from the response of the first request, to structure the request after it.
Mathematical operations + and - are supported, to add or reduce a number from the variable value.

Use case examples for variable usage:

  1. Update a date filter at every call
  2. Update a page number in pagination

To use variables:

  • Wrap the variable name in curly brackets
  • Provide the full path to that variable in the response
  • Add res. prefix to the path.

Example: Say this is my response:

{
  "field": "value",
  "another_field": {
    "nested": 123
  },
  "num_arr": [1, 2, 3],
  "obj_arr": [
    {
      "field2": 345
    },
    {
      "field2": 567
    }
  ]
}

Paths to fields values are structured like so:

  • {res.field} = "value"
  • {res.another_field.nested} = 123
  • {res.num_arr.[2]} = 3
  • {res.obj_arr.[0].field2} = 345

Using the fields values in the next_url for example like the below:

next_url: https://logz.io/{res.field}/{res.obj_arr[0].field2}

Would update the URL at every call to have the value of the given fields from the response. In our example the url for the next call would be:

https://logz.io/value/345

And in the call after it, it would update again according to the response and the next_url structure, and so on.

Launch the Docker Container​

Use Docker to run the Logz.io API Fetcher with the provided command in the path where you saved your config.yaml:

docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher
info

To run in Debug mode add --level flag to the command:

docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher \
--level DEBUG

Available Options: INFO, WARN, ERROR, DEBUG

Stop the Docker container​

When you stop the container, the code will run until the iteration is completed. To make sure it will finish the iteration on time, please give it a grace period of 30 seconds when you run the docker stop command.

docker stop -t 30 logzio-api-fetcher