Skip to content

Onboard observability data

The managed platform uses an OpenTelemetry-first architecture for observability.

For new observability workloads:

Applications / Hosts / Kubernetes / Azure Services
                         ↓
                    Elastic Agent
                         ↓
              Fleet-managed policy
                         ↓
       OpenTelemetry logs, metrics, traces
                         ↓
                    Elasticsearch
                         ↓
                       Kibana

Use Elastic Agent as the standard collection and telemetry gateway layer and Fleet to manage agent configuration centrally.

The goal is for logs, metrics, and traces from different sources to use a consistent OpenTelemetry model.

Before you begin

Make sure:

  • platform access is working;
  • Fleet is available for the deployment;
  • at least one Elastic Agent can connect to Fleet;
  • the source environment can reach the required Elastic Agent or platform endpoints;
  • DNS and TLS trust are configured;
  • appropriate ingestion credentials are available.

If you have not configured Fleet and Elastic Agent yet, start with Manage Elastic Agents with Fleet.

Choose the telemetry source

Elastic Agent can collect or receive observability data from different sources.

Source Typical collection method
Instrumented applications OTLP to Elastic Agent
Hosts and virtual machines Elastic Agent host integrations
Kubernetes Elastic Agent Kubernetes and OpenTelemetry integrations
Azure services Elastic Agent Azure integrations
Custom applications OpenTelemetry instrumentation and OTLP
Existing telemetry pipelines Integrate with Elastic Agent or retain the existing supported pipeline where required

For new application telemetry, prefer native OpenTelemetry instrumentation.

Application telemetry

Instrument applications using OpenTelemetry SDKs or supported auto-instrumentation.

Applications can produce:

  • logs;
  • metrics;
  • traces.

Send OTLP telemetry to the Elastic Agent endpoint configured for the workload.

Application
    ↓
   OTLP
    ↓
Elastic Agent
    ↓
Elasticsearch

Use the OTLP endpoint and protocol configured in the Fleet policy.

Do not guess endpoints or send telemetry directly to Kibana.

Hosts and virtual machines

Use Elastic Agent to collect operating-system and application telemetry.

Typical signals include:

  • system logs;
  • CPU and memory metrics;
  • filesystem metrics;
  • network metrics;
  • application logs;
  • custom telemetry.

Where supported, normalize the resulting telemetry around OpenTelemetry conventions.

Kubernetes

For Kubernetes, Elastic Agent can combine infrastructure collection with application OpenTelemetry telemetry.

Typical data includes:

  • node metrics;
  • pod and container metrics;
  • container logs;
  • Kubernetes metadata;
  • application logs;
  • application metrics;
  • distributed traces.

Use Fleet policies to manage the Kubernetes collection configuration consistently across clusters.

Azure services

Use supported Elastic integrations through Elastic Agent to collect telemetry from Azure services.

Examples can include:

  • Azure Monitor metrics;
  • platform logs;
  • activity logs;
  • supported service-specific telemetry.

The resulting data can then be analyzed alongside application, host, and Kubernetes telemetry in Elasticsearch.

OpenTelemetry semantic conventions

For new observability data, preserve meaningful OpenTelemetry resource and service attributes.

Important examples include:

  • service.name;
  • service version;
  • deployment environment;
  • cloud provider and region;
  • host identity;
  • Kubernetes cluster, namespace, pod, and container identity.

Consistent attributes make it possible to correlate telemetry across logs, metrics, and traces.

OTel-native data streams

OpenTelemetry-native telemetry uses signal-specific Elasticsearch data streams.

Typical patterns include:

Signal Data stream pattern
Logs logs-*.otel-*
Metrics metrics-*.otel-*
Traces traces-*.otel-*

Existing Elastic integrations may continue to use ECS-compatible data streams.

Both can coexist, but new observability implementations should prefer OpenTelemetry-native conventions where supported.

Concrete OpenTelemetry integration examples

Elastic provides OpenTelemetry-native integration assets for many common infrastructure and application technologies.

These packages illustrate the preferred direction for new observability onboarding: collect telemetry using OpenTelemetry receivers and conventions, then use Elastic-provided assets to visualize and analyze the resulting OTel data.

Examples include:

Technology Elastic package OTel dataset examples Typical use
Host / operating system system_otel hostmetricsreceiver.otel CPU, memory, filesystem, network, and host metrics
NGINX nginx_otel nginxreceiver.otel, nginx.access.otel, nginx.error.otel Web-server metrics, access logs, and error logs
MySQL mysql_otel mysqlreceiver.otel Database availability, performance, queries, and resource metrics
Apache Airflow airflow_otel airflow.otel Scheduler, DAG processing, task, capacity, and error metrics
Apache Kafka kafka_otel kafkametricsreceiver.otel Broker, topic, partition, replication, and consumer-group metrics

Host monitoring

For servers and virtual machines, the Elastic system_otel package provides dashboards for telemetry collected with the OpenTelemetry hostmetrics receiver.

A typical flow is:

Linux / Windows host
        ↓
OpenTelemetry hostmetrics receiver
        ↓
Elastic Agent
        ↓
hostmetricsreceiver.otel
        ↓
Elasticsearch + Kibana

This provides a standard OpenTelemetry representation for infrastructure metrics instead of relying on a source-specific schema.

NGINX

Elastic provides the nginx_otel package for NGINX telemetry.

It recognizes OTel datasets including:

nginxreceiver.otel
nginx.access.otel
nginx.error.otel

This allows NGINX server metrics, access logs, and error logs to participate in the same OpenTelemetry-based observability model as application and infrastructure telemetry.

MySQL

The mysql_otel package provides OpenTelemetry assets for MySQL telemetry collected through the corresponding OpenTelemetry receiver.

The primary dataset is:

mysqlreceiver.otel

Elastic includes visualization assets for areas such as:

  • database overview;
  • performance;
  • queries;
  • availability.

Apache Airflow

The airflow_otel package provides OpenTelemetry assets specifically for Apache Airflow.

Its OTel dataset is:

airflow.otel

The package includes observability content for:

  • scheduler health;
  • capacity;
  • task execution;
  • errors;
  • DAG processing.

This is a good example of a platform application producing domain-specific telemetry while still using the common OpenTelemetry model.

Apache Kafka

For Kafka, Elastic provides the kafka_otel package using:

kafkametricsreceiver.otel

The provided assets cover operational areas such as:

  • Kafka overview;
  • consumer groups;
  • topics and partitions;
  • replication health.

Other OpenTelemetry packages

The Elastic integrations repository contains additional OTel-native packages such as:

  • apache_otel;
  • iis_otel;
  • redis_otel;
  • docker_otel;
  • oracle_otel;
  • etcd_otel;
  • activemq_otel;
  • ibmmq_otel;
  • aws_elb_otel.

The available integrations continue to evolve with Elastic releases.

For new onboarding, check the current Elastic integrations catalog and prefer an OpenTelemetry-native integration when one is available.

When an OTel-native integration exists, use this sequence:

Source technology
      ↓
OpenTelemetry receiver / instrumentation
      ↓
Elastic Agent
      ↓
Fleet-managed configuration
      ↓
OTel-native dataset
      ↓
Elastic integration assets
      ↓
Elasticsearch + Kibana

This is preferred over building custom parsing and dashboards for each technology because the integration provides a tested telemetry model and reusable Elastic content.

When no dedicated OTel-native integration exists, continue using Elastic Agent with the best supported integration while preserving OpenTelemetry semantic conventions wherever possible.

Start with one source

Before onboarding a full environment:

  1. choose one representative application, host, or workload;
  2. assign the appropriate Fleet policy;
  3. confirm the Elastic Agent is Healthy;
  4. send or collect a small amount of telemetry;
  5. confirm logs, metrics, or traces arrive in Elasticsearch;
  6. verify OpenTelemetry attributes and resource metadata;
  7. confirm the telemetry is usable in Kibana.

Then expand the same pattern to additional sources.

Validate logs

For logs, confirm:

  • timestamps are correct;
  • service.name or source identity is present;
  • log severity is represented correctly;
  • environment and infrastructure metadata are available;
  • multiline events are handled correctly where required;
  • sensitive information is not being collected unintentionally.

Validate metrics

For metrics, confirm:

  • expected metric names are present;
  • units are correct;
  • resource attributes identify the source;
  • collection intervals are appropriate;
  • unnecessary high-cardinality dimensions are avoided.

Validate traces

For traces, confirm:

  • services appear with the expected service.name;
  • traces contain expected spans;
  • service-to-service relationships are visible;
  • environment and resource metadata are present;
  • trace and span identifiers can correlate with application logs where configured.

Expand onboarding

After the first source is validated:

  1. add additional Elastic Agents or workloads;
  2. reuse Fleet policies where collection requirements are the same;
  3. create separate policies only when the requirements differ materially;
  4. verify agent health after each major policy change;
  5. monitor ingestion volume as additional sources are added.

Existing pipelines

Existing Logstash, ECS-based, or OpenTelemetry Collector pipelines do not need to be replaced immediately if they are already operating successfully.

For new observability onboarding, however, use the OpenTelemetry-first Elastic Agent and Fleet architecture unless there is a specific compatibility or integration requirement.

Troubleshooting

If telemetry does not arrive:

  • confirm the Elastic Agent is healthy in Fleet;
  • confirm the correct policy is assigned;
  • verify the relevant integration is enabled;
  • check OTLP endpoint configuration for application telemetry;
  • verify DNS, networking, and TLS;
  • review Elastic Agent logs;
  • check the expected OTel-native data streams.

For assistance:

  • AI chat: https://copilot.opsflw.io
  • Support portal: https://support.ivedha.com/

Provide the deployment reference, affected source type, and Elastic Agent information. Do not include API keys, enrollment tokens, passwords, or other secrets.

Next step

After telemetry is flowing, use Kibana to verify and explore the data, then continue with search and visualize data.