Scaling Stateful Connection Management: Distributed Sharding for a Real-Time Mailbox Platform

An engineering case study on transforming a single-instance, in-memory connection manager into a horizontally scalable, deterministically sharded distributed system on Kubernetes.

1. Executive Summary

A Real-Time Mailbox Connection Service maintained one persistent IMAP connection per email account to deliver live mailbox notifications and message access. The original design stored all connection state in memory, on a single process — which made the service impossible to scale horizontally without creating duplicate connections and corrupting request routing.

I owned the redesign end to end: converting a stateful single-instance service into a deterministically sharded, horizontally scalable distributed system running on Kubernetes.

Key technical achievements:

• Introduced hash-based deterministic ownership (MurmurHash3) so each account maps to exactly one pod — eliminating duplicate connections at the source.
• Built stateful request routing via Kubernetes StatefulSets + a headless service, with middleware-based rerouting to the owning pod.
• Designed a route classification model (stateful / cron / stateless) so each request type is handled with the correct consistency guarantees.

Impact: Reduced redundant connections by ~35%, and scaled the system from ~1.5K active connections on a single instance to 10K+ connections distributed across pods, with predictable, controllable rollouts.

This is the kind of problem where the hard part is not the code — it is making a fundamentally stateful workload behave correctly inside an orchestration layer that assumes statelessness.

2. Business Problem

The platform’s core value was real-time mailbox awareness — knowing immediately when a message arrived, moved, or changed in a connected mailbox. IMAP’s IDLE model makes this possible, but it requires a long-lived, persistent connection held open per account. That connection is inherently stateful: it lives in a process’s memory, tied to socket descriptors and session context.

The original service held all of these connections in-memory on one instance. As the number of connected accounts grew, this created a hard ceiling:

• Vertical scaling only. The only way to handle more accounts was a bigger box. There was no path to spread load across instances.
• Duplicate connections on scale-out. Naively running multiple replicas caused every replica to open a connection for every account. This is not just wasteful — duplicate IMAP connections to the same mailbox can trip provider-side connection limits and trigger throttling or lockouts.
• No connection–request affinity. A request that needed live mailbox state had to be served by the specific process holding that account’s connection. Kubernetes load balancing routed requests to any pod, so requests frequently landed on a pod that had no knowledge of the connection.

Why this was hard: Kubernetes is optimized for stateless, fungible workloads — any replica can serve any request. This service was the opposite: each unit of state (a connection) had to live in exactly one place, and traffic for that state had to find it. Reconciling a stateful workload with a stateless orchestration model is the central engineering tension of this project.