Field Mapping

Field mapping is the process of defining how a value and the fields it contains are stored and indexed.

OpenSearch mappings are important whenever you want to perform any action on a field, such as visualize it, aggregate by it, or use it in an alert.

Default mapping

You might have noticed that the particular fields mapped by OpenSearch Dashboards tend to vary. This is because mapping is dynamic and responds to the particular dataset you've selected. The larger the dataset, the more likely it is for fields to be unmapped by OpenSearch Dashboards.

By default, OpenSearch Dashboards maps only 1,000 fields to keep querying and filtering performance at top speed.

Here's how OpenSearch Dashboards does it. First, it finds every field that your account is actively using - in visualizations, dashboards, saved searches, alerts, and optimizers - and makes sure that those fields are mapped.

Let's say you have 10k fields in your database index, but are actively using 300 fields. Then OpenSearch Dashboards will first map your 300 required fields and then map another 700 random fields.

OpenSearch Dashboards will always make sure that all of your required fields are mapped by default. So even if you have more than 1,000 required fields, OpenSearch Dashboards will cover them all and ensure that all of them are mapped every time.

Field mapping types

To make your search engine queries and analytics are more effective, OpenSearch Dashboards maps each field by a data type, so it knows how to display it according to its capabilities. There are two types of mapping fields:

• Dynamic - This is the default mapping type, determined by the value of the log fields mapped at the beginning of each day.
• Explicit - This is a forced mapping type, and when chosen, OpenSearch will always map this field as the same data type.

For example, if the value of the log field is "yourField":123, OpenSearch will map it as a number (Long).

“yourField”:”abc” will be mapped as a Keyword (String).

“yourField”:{“someField”:”someValue”} will be mapped as an Object.

yourField.someField will be mapped as a Keyword (String).

If a field is mapped as a string, OpenSearch won’t allow you to run any mathematical queries on the field. If it's an analyzed field, such as message, tags, or geoip_location, OpenSearch won't let you use it in an alert, a visualization, or a group by rule.

Field data type determines how each field is indexed and shown in OpenSearch Dashboards. Account admins can change the data types according to a predefined set of options:

Changing a field's data type may affect any dashboards, visualizations, searches, alerts, optimizers, and integrations using that field.

Managing field mappings

Use field mapping to override dynamic mapping and resolve mapping errors for your ingested logs. The changes you make on the Field mappings page won't affect the logs that are already ingested.

You can make up to 5 mapping changes in a day, but be cautious. Changing a field mapping may impact the components and integrations that use the field: Your dashboards, visualizations, searches, alerts, and optimizers may not perform as expected if you change field mappings.

If you’re not sure of the impact of changing a mapping, contact the Logz.io Support team for advice.

Date data fields

note

Before you change, edit, and send date data fields, contact the Logz.io Support team.

There are additional restrictions for date data field types:

• Automatic date detection is disabled by default in dynamic mapping, which detects values as string instead of date.
• To avoid conflict between the mapping of date fields, the data type must be identical across all indices.

Therefore, to change the mapping of any field to a date field, contact Logz.io Support team before sending the fields.

How to identify when a field is not mapped

If you are trying to filter by a field but the field doesn't appear in the dropdown list, this is a good indication that the field is not mapped in OpenSearch Dashboards.

OpenSearch's capabilities are most powerful for mapped fields. Fields that aren't mapped can be searched and queried. But they will not appear in filters and do not support 1-click visualizations.

Action	Mapped field	Unmapped field
Filtering	✔	✖️
Appears in filtering menu	✔	✖️
Can be visualized	✔	✖️
Searchable	✔	✔

About array fields

Arrays are not natively supported by the OpenSearch Dashboards interface. When an array is included in a log, the full array is displayed as a single field marked with the icon next to the field name.

In the example below, the array ["a","b"] = [{"a":"1"}, {"b":"2"}] becomes a single field.

Depending on the array, you may be able to seach for the string elements inside an array, as in the filter syntax example below.

In general, the more organized and consitent your log structure is (especially if the structure includes unique keys), the more accurate the result of transforming the data in the array will be.

Manage your mapping

Refresh mapping

If you find that many of the fields you are interested in exploring aren't mapped, you can refresh your mapping via the navigation menu. Click Settings> General settings > Refresh mapping.

Add specific fields to your default mapping

Instead of refreshing the mapping in bulk, you can add specific fields to your default mapping. Click Field not indexed on an unmapped field. Learn more

Explicitly map a field

To manually edit a field mapping, Select Data Hub > Field mappings from the navigation menu.

To change the field mapping type, hover over the field, click edit , make your changes, and click Save.

Resolving mapping errors

If you encounter an issue or an error when mapping your fields, check out the common mapping errors and why they happen.