Log Aggregation Strategies

How do you implement centralized logging in a distributed system? What are the key components?

Monitoringintermediate

// interview question

How do you implement centralized logging in a distributed system? What are the key components?

Sample answer

Centralized logging collects logs from all services into one searchable system. Key components: 1) Collection - agents like Fluentd, Fluent Bit, or Filebeat. 2) Transport - message queues (Kafka) for buffering. 3) Processing - parsing, filtering, enriching (Logstash). 4) Storage - Elasticsearch, Loki, or cloud services. 5) Visualization - Kibana, Grafana. Best practices: use structured logging (JSON), include correlation IDs for tracing requests, set retention policies, and implement log levels appropriately.

Why this matters

In distributed systems, logs scattered across hundreds of containers are useless. Centralized logging enables searching across all services, correlating events, and debugging issues. The ELK stack (Elasticsearch, Logstash, Kibana) is traditional; newer options like Loki (Grafana) are more cost-effective. Structured logging is crucial - parsing unstructured text at scale is expensive.

Code examples

Fluent Bit DaemonSet

yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:latest
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: config
          mountPath: /fluent-bit/etc/
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: config
        configMap:
          name: fluent-bit-config

Structured log format

json

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123xyz",
  "span_id": "def456",
  "user_id": "user_789",
  "message": "Payment processing failed",
  "error": {
    "type": "PaymentGatewayError",
    "message": "Connection timeout",
    "stack": "..."
  },
  "duration_ms": 5023
}

Common mistakes to avoid

Logging sensitive data (passwords, PII) that violates compliance
Using DEBUG level in production, creating massive storage costs
Not including correlation IDs, making distributed tracing impossible

Likely follow-ups

How do you handle high-volume logging without impacting application performance?
What is the difference between logs, metrics, and traces?
How do you implement log retention and comply with data regulations?

Answer out loud first, then check yourself against the model answer.

Practice all Mid-Level questions More Monitoring questions

#logging#observability#elk#fluentd#monitoring

Also worth your time on this topic

Article

Datadog Alternatives in 2026: 12 Tools, Their Real Prices, and What 300 GB of Logs Costs on Each

We read the pricing page of Datadog and 12 alternatives on the same day and priced one workload on each: 300 GB of logs a month. Datadog with standard indexing of every event is $780 to $1,155 a month; the alternatives we could price run from about $19 to $180. Here is each tool, who it suits, and the catch on its pricing page.

Checklist

Monitoring & Observability Checklist

Comprehensive checklist for implementing monitoring, logging, tracing, and alerting across your infrastructure and applications.

60-90 minutes

Interview

Four Golden Signals of Monitoring

What are the four golden signals of monitoring and why are they important?

junior

Log Aggregation Strategies

More Monitoring interview questions

Also worth your time on this topic

Datadog Alternatives in 2026: 12 Tools, Their Real Prices, and What 300 GB of Logs Costs on Each

Monitoring & Observability Checklist

Four Golden Signals of Monitoring