Microsoft Azure Office365 Message Trace Reports (mail reports)
You can ship logs available from the Microsoft Graph APIs with Logzio-api-fetcher. Microsoft Graph is a RESTful web API that enables you to access Microsoft Cloud service resources. This integration allows you to collect data from Microsoft Graph API and send it to your Logz.io account.
Logzio-api-fetcher supports many API endpoints, including but not limited to:
- Azure Active Directory audit logs
- Azure Active Directory sign-in logs
There are many other APIs available through Microsoft Graph.
Register a new app in Azure Active Directory
In the Azure portal, go to App registration and select New registration from the top menu.
Name your app and click Register.
Create a client secret
Choose Certificates & secrets from the side menu, and click on New client secret.
Add a Description. We recommend something specific, such as "secret for Logzio-MSGraph integration".
In the Expires list, choose Never.
Click Add.
Copy the value of the generated secret to your text editor. You'll need this later.
You won't be able to retrieve the secret's value after you leave this page.
Set the app's permissions
Choose API permissions from the side menu, and click Add a permission.
Select Office 365 exchange online > Application permissions.
Search for the following permssion: ReportingWebService.Read.All
Click Add permissions.
Click Grant admin consent for Default Directory, and then click Yes to confirm.
Only Azure administrators can grant consent for Default Directory. If the Grant admin consent button is disabled, ask your Azure admin to update the setting for you.
Configure application to have one of the following IAM roles:
IAM roles are showen in least permissive order:
- Global Reader role
- Exchange Administrator
- Global Administrator
Pull the Docker image of the Logz.io API fetcher
docker pull logzio/logzio-api-fetcher
Create a local directory for this integration
You will need a dedicated directory to use it as mounted directory for the Docker container of the Logz.io API fetcher.
mkdir logzio-api-fetcher
cd logzio-api-fetcher
Create a configuration file
In the directory created in the previous step, create a file config.yaml
using the example configuration below:
logzio:
url: https://<<LISTENER-HOST>>:8071
token: <<LOG-SHIPPING-TOKEN>>
oauth_apis:
- type: azure_mail_reports
name: mail_reports
credentials:
id: <<AZURE_AD_SECRET_ID>>
key: <<AZURE_AD_SECRET_VALUE>>
token_http_request:
url: https://login.microsoftonline.com/abcd-efgh-abcd-efgh/oauth2/v2.0/token
body: client_id=<<AZURE_AD_CLIENT_ID>>
&scope=https://outlook.office365.com/.default
&client_secret=<<AZURE_AD_SECRET_VALUE>>
&grant_type=client_credentials
headers:
method: POST
data_http_request:
url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace
method: GET
headers:
json_paths:
data_date: EndDate
next_url:
data:
filters:
format: Json
settings:
time_interval: 60 # for mail reports we suggest no less than 60 minutes
days_back_fetch: 8 # for mail reports we suggest up to 8 days
start_date_name: StartDate
end_date_name: EndDate
Parameter | Description | Required/Default |
---|---|---|
URL | Use the listener URL specific to the region where your Logz.io account is hosted. Click to look up your listener URL. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071. | Required |
TOKEN | Your Logz.io account token. Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to. | Required |
type | The type of the OAuth API. Currently we support the following types: azure_graph, general. | Required |
name | The name of the OAuth API. Please make names unique. | Required |
credentials.id | The OAuth API credentials id. | Required |
credentials.key | The OAuth API credentials key. | Required |
data_http_request.method | The HTTP method. Can be GET or POST. | Required |
data_http_request.url | The OAuth API url. Make sure the url is without ? at the end. | Required |
data_http_request.headers | Pairs of key and value the represents the headers of the HTTP request. | Optional |
data_http_request.body | The body of the HTTP request. Will be added to HTTP POST requests only. | Optional |
token_http_request.method | The HTTP method. Can be GET or POST. | Required |
token_http_request.url | The OAuth API token request url. Make sure the url is without ? at the end. | Required |
token_http_request.headers | Pairs of key and value the represents the headers of the HTTP request. | Optional |
token_http_request.body | The body of the HTTP request. Will be added to HTTP POST requests only. | Optional |
json_paths.data_date | The json path to the data's date value inside the response of the OAuth API. | Required |
settings.time_interval | The OAuth API time interval between runs. | Required |
settings.days_back_fetch | The max days back to fetch from the OAuth API. | Optional. Default value is 14 days. |
filters | Pairs of key and value of parameters that can be added to the OAuth API url. Make sure the keys and values are valid for the OAuth API. | Optional |
custom_fields | Pairs of key and value that will be added to each data and be sent to Logz.io. | Optional |
Important notes and limitations
- We recommend setting the
days_back_fetch
parameter to no more than8d
(~192 hours) as this might cause unexpected errors with the API. - We recommend setting the
time_interval
parameter to no less than60
, to avoid short time frames in which messages trace will be missed. - Microsoft may delay trace events for up to 24 hours, and events are not guaranteed to be sequential during this delay. For more information, see the Data granularity, persistence, and availability section of the MessageTrace report topic in the Microsoft documentation: MessageTrace report API
Create a Last Start Dates text file
Create an empty text file named last_start_dates.txt in the same directory as the config file:
$ touch last_start_dates.txt
After every successful iteration of an API, the last start date of the next iteration will be written to last_start_dates.txt. Each line starts with the API name and ends with the last start date.
If you stopped the container, you can continue from the exact place you stopped, by adding the date to the API filters in the configuration.
Run the Docker container
docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher
Stop the Docker container
When you stop the container, the code will run until the iteration is completed. To make sure it will finish the iteration on time, please give it a grace period of 30 seconds when you run the docker stop
command.
docker stop -t 30 logzio-api-fetcher
Check Logz.io for your logs
Give your logs some time to get from your system to ours,
and then open Open Search Dashboards. You can filter for data of your custom field type value or type api_fetcher
to see the incoming Microsoft Graph logs.
If you still don't see your logs, see log shipping troubleshooting.
You can see a full list of the possible configuration values in the logzio-api-fetcher github repository.