Skip to content

Collector pipelines

The OpenTelemetry Collector's configuration is deploy/otel-collector/config.yaml, mounted read-only into the otel-collector service (Observability). Values such as ${env:STACKR_ENVIRONMENT} are read from the service's environment, which compose.yaml sets from .env.

Pipelines

Pipeline Receivers Processors Exporters
traces otlp memory_limiter, resource/environment, batch ${env:STACKR_TRACES_EXPORTERS}: set by make up from the profiles
metrics otlp memory_limiter, resource/environment, batch otlp_http/prometheus
logs otlp memory_limiter, resource/environment, batch otlp_http/loki

The traces pipeline's exporters come from STACKR_TRACES_EXPORTERS, which make up sets from the profiles: [otlp_grpc/tempo, otlp_http/langfuse] when the langfuse profile runs, and [otlp_grpc/tempo] when it doesn't. Compose's default, for plain docker compose, is both.

Components

Receivers

Receiver Protocol Listens on Notes
otlp grpc 0.0.0.0:4317
otlp http 0.0.0.0:4318 CORS from http://localhost:*, http://127.0.0.1:*

Processors

Processor Settings
memory_limiter {check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25}
resource/environment {attributes: [{key: deployment.environment.name, value: '${env:STACKR_ENVIRONMENT}', action: insert}]}
batch {}

Exporters

Exporter Endpoint Sends Headers
otlp_grpc/tempo tempo:4317 OTLP gRPC, without TLS
otlp_http/prometheus http://prometheus:9090/api/v1/otlp OTLP HTTP, without TLS
otlp_http/loki http://loki:3100/otlp OTLP HTTP, without TLS
otlp_http/langfuse http://langfuse-web:3000/api/public/otel OTLP HTTP, without TLS Authorization: Basic ${env:LANGFUSE_OTLP_AUTH}, x-langfuse-ingestion-version: 4

Extensions

Extension Settings
health_check {endpoint: '0.0.0.0:13133'}

The Collector's own telemetry

Signal Settings
metrics {level: detailed, readers: [{pull: {exporter: {prometheus: {host: 0.0.0.0, port: 8888}}}}]}
logs {level: info}

LANGFUSE_OTLP_AUTH is base64 of the Langfuse project's public and secret keys, which make env derives. Prometheus scrapes the Collector's own metrics on port 8888 for the Collector health dashboard; neither 8888 nor the health check's 13133 is published.

The file

The whole file: deploy/otel-collector/config.yaml 
# The OpenTelemetry Collector: every application, and the LiteLLM proxy, sends
# OTLP here, and the Collector fans it out:
#   traces  -> Tempo, and Langfuse when the langfuse profile runs
#   metrics -> Prometheus, through its OTLP receiver
#   logs    -> Loki, through its OTLP endpoint
# See docs/architecture.md and ADR-0007.

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          # Browser applications served from this machine can send OTLP too.
          allowed_origins: ["http://localhost:*", "http://127.0.0.1:*"]

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Telemetry that doesn't say which environment it comes from is labelled
  # with STACKR_ENVIRONMENT (`local` by default).
  resource/environment:
    attributes:
      - key: deployment.environment.name
        value: ${env:STACKR_ENVIRONMENT}
        action: insert
  batch: {}

exporters:
  otlp_grpc/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlp_http/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp
    tls:
      insecure: true
  otlp_http/loki:
    endpoint: http://loki:3100/otlp
    tls:
      insecure: true
  # Langfuse takes traces only, over OTLP HTTP. The ingestion version header
  # makes them visible as soon as they are written.
  otlp_http/langfuse:
    endpoint: http://langfuse-web:3000/api/public/otel
    headers:
      Authorization: Basic ${env:LANGFUSE_OTLP_AUTH}
      x-langfuse-ingestion-version: "4"

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource/environment, batch]
      # [otlp_grpc/tempo, otlp_http/langfuse], or Tempo alone when the
      # langfuse profile isn't running; set in compose.yaml and the Makefile.
      exporters: ${env:STACKR_TRACES_EXPORTERS}
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource/environment, batch]
      exporters: [otlp_http/prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource/environment, batch]
      exporters: [otlp_http/loki]
  telemetry:
    # The Collector's own metrics, scraped by Prometheus for the Collector
    # dashboard.
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
    logs:
      level: info