Default configuration of the EDOT Collector (Standalone)

Elastic Stack Serverless Observability

The default configuration of the Elastic Distribution of OpenTelemetry (EDOT) Collector includes pipelines for the collection of logs, host metrics, and data from OpenTelemetry SDKs.

The EDOT Collector can run in Agent or Gateway mode:

• Agent mode: The EDOT Collector ingests data from infrastructure and SDKs and forwards it to Elastic or to another collector running in Gateway mode.
• Gateway mode: The EDOT Collector ingests data from other collectors running in Agent mode and forwards it to Elastic.

The following sample config files for Agent mode are available:

Use the previous example configurations as a reference when configuring your upstream Collector or customizing your EDOT Collector configuration.

The following sections describe the default pipelines by use cases.

For self-managed and Elastic Cloud Hosted stack deployment use cases, ingest OpenTelemetry data from the EDOT Collector directly into Elasticsearch using the elasticsearch exporter.

Learn more about the configuration options for the elasticsearch exporter in the corresponding documentation.

The elasticsearch exporter comes with two relevant data ingestion modes:

• ecs: Writes data in backwards compatible Elastic Common Schema (ECS) format. Original attribute names and semantics might be lost during translation.
• otel: OTel attribute names and semantics are preserved.

The goal of EDOT is to preserve OTel data formats and semantics as much as possible, so otel is the default mode for the EDOT Collector. Some use cases might require data to be exported in ECS format for backwards compatibility.

For logs collection, the default configuration uses the filelog receiver to read log entries from files. In addition, the resourcedetection processor enriches the log entries with metadata about the corresponding host and operating system.

Data is exported directly to Elasticsearch using the elasticsearch exporter in OTel-native mode.

The application pipeline in the EDOT Collector receives data from OTel SDKs through the OTLP receiver. While logs and metrics are exported verbatim into Elasticsearch, traces require two additional components.

The elastictrace processor enriches trace data with additional attributes that improve the user experience in the Elastic Observability UIs. In addition, the elasticapm connector generates pre-aggregated APM metrics from tracing data.

Application-related OTel data is ingested into Elasticsearch in OTel-native format using the elasticsearch exporter.

The host metrics pipeline uses the hostmetrics receiver to collect disk, filesystem, cpu, memory, process and network metrics for the corresponding host.

For backwards compatibility, host metrics are translated into ECS-compatible system metrics using the elasticinframetrics processor. Finally, metrics are ingested in ecs format through the elasticsearch exporter.

The resourcedetection processor enriches the metrics with meta information about the corresponding host and operating system. The attributes and resource processor are used to set some fields for proper routing of the ECS-based system metrics data into corresponding Elasticsearch data streams.

When ingesting OTel data through Elastics Managed OTLP endpoint, all the enrichment that is required for an optimal experience in the Elastic solutions happens at the managed OTLP endpoint level and is transparent to users.

The Collector configuration for all the use cases involving the Managed OTLP endpoint is only concerned with local data collection and context enrichment.

Platform logs are scraped with the filelog receiver, host metrics are collected through the hostmetrics receiver and both signals are enriched with meta information through the resourcedetection processor.

Data from OTel SDKs is piped through the OTLP receiver directly to the OTLP exporter that sends data for all the signals to the managed OTLP endpoint.

With the managed OTLP Endpoint there is no need for configuring any Elastic-specific components, such as elasticinframetrics, elastictrace processors, elasticapm connector or the elasticsearch exporter. Edge setup and configuration can be 100% vendor agnostic.

The following sample configuration file is available for the Gateway mode:

• Gateway mode

Use the previous example configuration as a reference when configuring your Gateway Collector or customizing your EDOT Collector configuration.

The EDOT Collector in Gateway mode collects data from other Collectors using the OTLP protocol. By default, the sample Gateway configuration listens on port 4317 for gRPC and port 4318 for HTTP.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

1. Listen on all interfaces
2. Listen on all interfaces

The routing connector splits infrastructure metrics from other metrics and routes them to the appropriate Elastic Common Schema pipelines. Other metrics are exported in OTel-native format through the elasticsearch exporter.

connectors:
  routing:
    default_pipelines: [metrics/otel]
    error_mode: ignore
    table:
      - context: metric
        statement: route() where IsMatch(instrumentation_scope.name, "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/*")
        pipelines: [metrics/infra/ecs, metrics/otel]
  elasticapm: {}

The Gateway configuration includes several processors to transform and optimize the collected data:

processors:
  elasticinframetrics:
    add_system_metrics: true
    drop_original: true
  attributes/dataset:
    actions:
      - key: event.dataset
        from_attribute: data_stream.dataset
        action: upsert
  resource/process:
    attributes:
      - key: process.executable.name
        action: delete
      - key: process.executable.path
        action: delete
  batch:
    send_batch_size: 1000
    timeout: 1s
    send_batch_max_size: 1500
  batch/metrics:
    send_batch_max_size: 0
    timeout: 1s
  elastictrace: {}

1. Prevents splitting metrics requests

The Gateway exports data to Elasticsearch in two formats:

• OTel-native format using the elasticsearch/otel exporter.
• Elastic Common Schema (ECS) format using the elasticsearch/ecs exporter.

exporters:
  elasticsearch/otel:
    endpoints:
      - ${ELASTIC_ENDPOINT}
    api_key: ${ELASTIC_API_KEY}
    mapping:
      mode: otel
  elasticsearch/ecs:
    endpoints:
      - ${ELASTIC_ENDPOINT}
    api_key: ${ELASTIC_API_KEY}
    mapping:
      mode: ecs

The service section defines separate pipelines for different telemetry types:

• Metrics pipelines for infrastructure and OTel metrics
• Logs pipeline
• Traces pipeline
• Aggregated OTel metrics pipeline

Each pipeline connects specific receivers, processors, and exporters to handle different data types appropriately.