APAC Observability Pipeline Guide 2026: OpenTelemetry Collector, Vector, and Telegraf for Platform Teams

Why APAC Telemetry Pipelines Matter More Than Backends

APAC platform and SRE teams commonly debate which observability backend to run — Grafana vs. Datadog, Loki vs. Elasticsearch, Prometheus vs. InfluxDB. The more consequential architectural decision is the telemetry pipeline layer that sits between APAC instrumented services and these backends: how APAC telemetry is collected, transformed, filtered, and routed.

Getting the APAC pipeline layer right matters for three reasons:

1. APAC vendor portability: Services instrumented to emit to a pipeline layer (OTel SDK → OTel Collector) can switch APAC backends without re-instrumentation. Services writing directly to Datadog APIs are locked in.

2. APAC cost control: Filtering and sampling APAC telemetry before it reaches paid APAC backends (Datadog, Elastic Cloud) is the primary lever for APAC observability cost reduction. Pipelines handle this; backends don't.

3. APAC data enrichment: Adding APAC Kubernetes metadata, APAC environment tags, and APAC business context to raw telemetry before storage makes APAC debugging queries productive. Pipelines enrich in-flight; re-enriching in backends is expensive.

Three tools serve the APAC observability pipeline spectrum:

OpenTelemetry Collector — CNCF standard vendor-neutral APAC telemetry pipeline for OTLP-instrumented APAC services.

Vector — High-performance Rust-based APAC log and metric pipeline with VRL transformation language.

Telegraf — Plugin-driven APAC metrics collection agent from InfluxData with 300+ source integrations.

APAC Observability Pipeline Architecture Patterns

Pattern 1: OTel Collector as APAC telemetry gateway

APAC Service A (Java, OTel SDK)  ──┐
APAC Service B (Go, OTel SDK)    ──┤──→ OTel Collector (gateway) ──→ Grafana Tempo (traces)
APAC Service C (Node.js, OTel)   ──┘         │                    ──→ Prometheus (metrics)
                                              │                    ──→ Loki (logs)
                                              ↓
                                        APAC processing:
                                        - Tail sampling (keep 100% errors, 1% success)
                                        - Attribute filtering (strip PII fields)
                                        - Metric aggregation (reduce cardinality)
                                        - Batch export (reduce APAC backend API calls)

Pattern 2: Vector as APAC log pipeline

APAC Kubernetes pods (stdout/stderr) ──→ Vector DaemonSet ──→ enrich (K8s metadata)
APAC Nginx access logs (file)        ──┘                  ──→ parse (VRL)
                                                           ──→ filter (drop health checks)
                                                           ──→ route:
                                                               Loki (APAC hot storage)
                                                               S3 (APAC cold archive)
                                                               Datadog (APAC alerting)

Pattern 3: Telegraf as APAC metrics collection agent

APAC Host (CPU/mem/disk)     ──┐
APAC Docker containers       ──┤──→ Telegraf agent ──→ Prometheus remote write → Grafana
APAC MySQL/PostgreSQL        ──┤                   ──→ InfluxDB v2
APAC Redis/MongoDB           ──┘                   ──→ Datadog
APAC SNMP network devices ───┘

OpenTelemetry Collector: APAC Standard Telemetry Pipeline

OTel Collector YAML configuration — APAC gateway deployment

# otel-collector-config.yaml — APAC gateway collector

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317     # APAC services send OTLP here
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: apac-services
          scrape_interval: 30s
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: [apac-payments, apac-identity, apac-notifications]

processors:
  batch:
    timeout: 10s                   # APAC batch export — reduces backend API calls
    send_batch_size: 1000

  memory_limiter:
    limit_mib: 512                 # APAC OOM protection
    spike_limit_mib: 128

  attributes:
    actions:
      - key: "apac.environment"
        value: "production"
        action: insert
      - key: "apac.region"
        value: "sea-singapore"
        action: insert
      - key: "credit_card_number"   # APAC PII field removal
        action: delete
      - key: "user_password"
        action: delete

  tail_sampling:
    decision_wait: 10s
    policies:
      - name: apac-errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: apac-slow-traces
        type: latency
        latency: { threshold_ms: 2000 }
      - name: apac-probabilistic-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }   # 1% of APAC success traces

exporters:
  otlp/tempo:
    endpoint: https://apac-tempo.internal:4317
  prometheusremotewrite:
    endpoint: https://apac-prometheus.internal/api/v1/write
  loki:
    endpoint: https://apac-loki.internal/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, attributes, tail_sampling]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch, attributes]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, attributes]
      exporters: [loki]

OTel Collector Kubernetes deployment — DaemonSet + gateway

# daemonset: collect APAC node-level metrics and container logs
# gateway: aggregate APAC OTLP from all services before export

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: apac-daemonset
  namespace: apac-observability
spec:
  mode: daemonset
  config: |
    receivers:
      kubeletstats:             # APAC node + pod resource metrics
        auth_type: serviceAccount
        endpoint: "https://${env:K8S_NODE_NAME}:10250"
        insecure_skip_verify: true
        metric_groups: [node, pod, container]
    exporters:
      otlp:
        endpoint: apac-gateway-collector:4317
    service:
      pipelines:
        metrics:
          receivers: [kubeletstats]
          exporters: [otlp]

Vector: High-Performance APAC Log Transformation

Vector VRL — APAC nginx log parsing

# vector.toml — APAC log pipeline configuration

[sources.apac_kubernetes_logs]
type = "kubernetes_logs"
namespace_labels_key = "apac_namespace"
pod_labels_key = "apac_pod_labels"

[transforms.apac_parse_nginx]
type = "remap"
inputs = ["apac_kubernetes_logs"]
source = '''
  # Parse APAC nginx combined log format
  parsed, err = parse_nginx_log(.message, format: "combined")
  if err != null {
    .apac_parse_error = err
  } else {
    . = merge(., parsed)
  }

  # Enrich with APAC business context
  .apac_environment = "production"
  .apac_region = get_env_var!("APAC_REGION")

  # APAC PII masking — hash IP addresses
  if exists(.client) {
    .client = md5(.client)
  }

  # APAC filter — drop APAC health check noise
  if .request == "GET /health HTTP/1.1" {
    abort
  }
'''

[transforms.apac_route]
type = "route"
inputs = ["apac_parse_nginx"]
route.apac_errors = '.status >= 500'
route.apac_slow = '.apac_request_time > 2.0'
route.apac_normal = 'true'

[sinks.apac_loki]
type = "loki"
inputs = ["apac_parse_nginx"]
endpoint = "https://apac-loki.internal"
labels.apac_app = "{{ apac_pod_labels.app }}"
labels.apac_namespace = "{{ apac_namespace }}"
labels.apac_env = "production"

[sinks.apac_s3_archive]
type = "aws_s3"
inputs = ["apac_parse_nginx"]
bucket = "apac-logs-archive"
region = "ap-southeast-1"
key_prefix = "nginx/{{ now() | strftime(\"%Y/%m/%d\") }}/"
compression = "gzip"

Telegraf: APAC Plugin-Driven Metrics Collection

Telegraf configuration — APAC multi-source metrics

# telegraf.conf — APAC metrics collection agent

[agent]
  interval = "10s"
  hostname = "$APAC_HOSTNAME"
  omit_hostname = false

# APAC host system metrics
[[inputs.cpu]]
  percpu = false
  totalcpu = true
  collect_cpu_time = false

[[inputs.mem]]
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs"]

[[inputs.net]]
  interfaces = ["eth0", "eth1"]

# APAC Docker container metrics
[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  gather_services = false
  total = true

# APAC PostgreSQL database metrics
[[inputs.postgresql]]
  address = "host=apac-postgres.internal user=telegraf dbname=postgres sslmode=require"
  databases = ["apac_payments", "apac_identity"]

# APAC Redis cache metrics
[[inputs.redis]]
  servers = ["tcp://apac-redis.internal:6379"]
  password = "$APAC_REDIS_PASSWORD"

# APAC custom HTTP endpoint metrics (APAC payment API /metrics)
[[inputs.http]]
  urls = ["https://apac-payments.internal/apac/metrics"]
  method = "GET"
  data_format = "json"
  json_query = "apac_metrics"

# Output to Prometheus remote write (APAC Grafana stack)
[[outputs.prometheus_remote_write]]
  url = "https://apac-prometheus.internal/api/v1/write"
  http_headers = {"Authorization" = "Bearer $APAC_PROM_TOKEN"}

# Output to InfluxDB v2 (APAC time-series archive)
[[outputs.influxdb_v2]]
  urls = ["https://apac-influxdb.internal"]
  token = "$APAC_INFLUX_TOKEN"
  organization = "apac-platform"
  bucket = "apac-infrastructure"

APAC Observability Pipeline Tool Selection

APAC Pipeline Need                    → Tool              → Why

APAC OTel-instrumented services       → OTel Collector    OTLP native; CNCF standard;
(traces + metrics + logs, vendor-free) →                  tail sampling; multi-backend

APAC log transformation pipeline      → Vector            VRL expressive transforms;
(PII masking, parsing, routing)       →                   10-100x Logstash throughput;
                                                          fan-out to APAC multi-sinks

APAC host and database metrics        → Telegraf          300+ input plugins; covers
(heterogeneous APAC infrastructure)   →                   APAC SNMP + proprietary DB;
                                                          InfluxDB-native integration

APAC K8s pod log collection           → Vector            DaemonSet K8s source;
(Kubernetes-native APAC logging)      →                   metadata enrichment; Loki sink

APAC mixed instrumentation            → OTel Collector    OTLP + Prometheus scrape;
(OTel + Prometheus APAC services)     →                   unifies APAC telemetry streams

APAC cost reduction (Datadog volume)  → Vector or OTel    Filter before APAC paid sink;
(sampling, filtering before billing)  →                   significant APAC cost savings

Related APAC Platform Engineering Resources

For the observability backends that receive telemetry from these APAC pipeline tools, see the APAC AIOps and observability guide covering Dynatrace, PagerDuty, and Datadog.

For the distributed tracing backends (Jaeger, OpenTelemetry, SigNoz) that the OTel Collector exports APAC traces to, see the APAC GitOps and observability platform guide.

For the SLO management tools that consume the APAC metrics these pipelines collect, see the APAC SLO management guide covering Pyrra, Sloth, and OpenSLO.