<aside>
🌊
Буллеты про Kafka, event-driven architecture и streaming: producers/consumers, schemas, outbox, Kafka Streams, Flink, эксплуатация кластера и альтернативы.
</aside>
Продюсеры и консьюмеры
- Built Kafka producers and consumers handling X million events/day across X topics with at-least-once delivery.
- Scaled consumer throughput from Xk to Xk msg/sec by increasing partitions to X and parallelising in-partition processing.
- Cut end-to-end event latency from Xs to Xms by tuning linger.ms, batch.size and fetch parameters.
- Eliminated consumer-lag incidents by adding X autoscaled consumer replicas driven by lag metrics.
- Implemented manual offset commits after successful processing, removing X classes of message-loss defects.
- Built a dead-letter topic with X-attempt retry and exponential backoff, recovering X% of transiently failed messages.
- Reduced rebalance storms X% by tuning session timeouts and adopting cooperative sticky assignment.
- Implemented exactly-once semantics with transactional producers and read-committed consumers across X pipelines.
- Cut duplicate charge incidents X% by combining idempotency keys with the outbox pattern over Kafka.
- Processed X TB/day of telemetry with a consumer group of X instances, holding p99 processing latency at Xms.
- Introduced per-key ordering guarantees via partition keys, fixing X out-of-order processing defects.
- Built backpressure handling with pause/resume, keeping memory flat under Xx traffic spikes.
Схемы и контракты
- Introduced Avro schemas in a Schema Registry across X topics, eliminating X breaking-change incidents per quarter.
- Migrated X topics from JSON to Protobuf, shrinking message size X% and broker storage by X TB.
- Enforced backward-compatibility checks in CI, blocking X incompatible schema changes before release.
- Documented X event contracts as the integration surface between X teams, cutting cross-team integration time X%.
- Versioned event payloads with additive-only evolution, allowing X consumers to upgrade independently.