Asked in: Akamai, Olyv

Centralized Distributed Logging System

1. Requirements

Functional Requirements

1. Centralized log collection
   • Logs generated by all services — S1, S2, S3 — should be collected into a centralized logging system.
2. Log querying and filtering
   • Users should be able to query logs using multiple filters such as:
     • Service
     • Date/time range
     • Pod/instance
     • Log level
     • Request ID / Trace ID, if available.
3. Near real-time log availability
   • Logs should become available in the centralized store with minimal latency, ideally within a few seconds.
4. Log retention and archival
   • Recent logs should remain in fast, searchable storage.
   • Older logs should be moved to cheaper archival storage according to retention policies.

Non-Functional Requirements

1. High throughput
   • The system should support a very large volume of logs generated by many services and pods.
2. Fault tolerance / no single point of failure
   • Failure of an individual pod, collector, Kafka broker, consumer, or storage node should not bring down the entire logging system.
3. Service isolation
   • A sudden log spike from one service should not significantly affect the logging pipeline of other services.
4. Durability
   • Logs should not be lost simply because the downstream database or consumer is temporarily unavailable.
5. Backpressure handling
   • If log storage becomes slower than the incoming log rate, the system should be able to buffer the excess logs instead of immediately losing them.
6. Horizontal scalability
   • Producers, Kafka consumers, and storage should be independently scalable as log volume increases.

2. Architecture

The basic architecture is:

                         SERVICE S1
                 ┌─────────────────────┐
                 │ Pod 1                │
                 │ ┌───────┐ ┌───────┐ │
                 │ │  App  │ │Sidecar│ │
                 │ │       │→│Logger │ │
                 │ └───────┘ └───┬───┘ │
                 └────────────────┼─────┘
                                  │
                 ┌────────────────┼─────┐
                 │ Pod 2           │     │
                 │ App → Sidecar ──┘     │
                 └───────────────────────┘
                                  │
                                  ▼
                         ┌────────────────┐
                         │ Kafka S1 Topic │
                         │ P0 P1 P2 ...Pn │
                         └───────┬────────┘
                                 │
                           Consumer Group
                         ┌───────┴────────┐
                         │ C1 C2 C3 ... Cn│
                         └───────┬────────┘
                                 │
                                 ▼
                         ┌─────────────────┐
                         │ Centralized Log │
                         │     Storage     │
                         └────────┬────────┘
                                  │
                                  ▼
                            Query / Search