Chat system FAQ
Questions that come up around the chat system design but don’t fit neatly into one page. Try answering each one out loud before opening it: that’s the interview skill. Each answer is short and links to the full explanation.
Architecture
Does Redis replace the load balancer, or complement it?
They solve different problems, so they complement each other.
- The load balancer is the front door: it decides which server a new connection lands on, and handles TLS, health checks and rate limits at the edge.
- Redis pub/sub is the back channel between servers: when Alice on server
amessages a room,apublishes to Redis and every server with members of that room receives it. Browsers never connect to Redis.
With one server you need neither. Both become necessary at two or more servers.
Is a "where should I connect?" lookup API worth having?
Yes, it’s a standard answer. The client asks an HTTP endpoint which chat server to use and gets back a server ID plus a short-lived signed ticket.
- Pros: smart placement (load, region, keeping group members together); reconnecting just means asking again; tickets let chat servers verify users without the auth database.
- Cons: an extra round trip before connecting, and load numbers that can be slightly stale.
Real systems have this step: Discord clients call GET /gateway and later get a session-specific resume URL; Slack clients make a small HTTP call before opening the WebSocket.
Why power of two choices rather than least connections for chat?
A chat server’s load is its number of open connections, so least-connections-style placement fits. But plain least-connections sends every new connection to the emptiest server: a newly added server, or the survivors after a crash, get a pile-up. Several load balancers (or lookup instances) with slightly stale counts make it worse, since they all pick the same server. Picking the better of two random servers spreads those waves out and is nearly as good as checking every server.
More: Algorithms in practice
Without a load balancer in front of chat servers, what do you lose?
The edge services: TLS termination near users, rate limiting, authentication at the door, DDoS protection and firewall rules, hiding internal server addresses, central logs, a single place to drain servers. With a lookup API as the gate and an L4 load balancer in front, most of these are covered another way; what’s really lost comes down to unauthenticated handshakes reaching chat servers and no edge near users to end handshakes quickly.
Once a WebSocket is established, does the connection switch to an L4 load balancer?
No. There’s no handoff: a TCP connection can’t move between machines mid-stream. The same L7 proxy works at L7 during the handshake (reads the URL, headers and ticket, picks the backend), then after 101 Switching Protocols just copies bytes for that connection, behaving like L4. Separately, at scale an L4 tier sits in front of the L7 proxies to spread connections across them.
More: WebSockets through an L7 proxy, Passing WebSockets through a proxy
Do you need to balance load across the load balancers themselves?
At scale, yes. One machine can’t hold millions of connections and would be a single point of failure. DNS (or anycast) picks a region, routers hash each connection to one of several L4 machines (ECMP), the L4 tier spreads connections across a fleet of L7 proxies. For chat this means: keep the routing decision in the request (server ID + ticket) so any proxy can route it; per-proxy rate limits leak (N proxies → N× the limit); a dying proxy drops every WebSocket through it.
Which approach do Slack, Meta, WhatsApp and Discord actually use?
- Slack: DNS → AWS NLB (L4) → Envoy (L7) → WebSocket services; Flannel, an application-aware edge, and gateway servers subscribing to channel servers.
- Meta Messenger: Katran (L4) → edge Proxygen → origin Proxygen → MQTT backends, with socket takeover so restarts don’t drop users.
- WhatsApp: famously direct-to-server in 2012 (2M+ connections per Erlang server); today clients connect through a WhatsApp chat edge host on Meta’s network.
- Discord: lookup (
GET /gateway) plus a session-specific resume URL; what sits in front of the gateway isn’t public.
The pattern: large services converge on an edge tier near users in front of the core.
More: What real systems do, Where large services are in 2026
Sources: Slack: Migrating Millions of Concurrent Websockets to Envoy, Slack: Real-time Messaging, Meta zero downtime release (APNIC), Discord Gateway docs
Finding the recipient’s server
Does a user → chat server map fit on one Redis server?
By memory, yes: 200M entries at roughly 100 bytes each, Redis overhead included, is about 20 GB. By request rate, not if every message looks up its recipients: a few million lookups per second at WhatsApp scale, against roughly 100k–200k unpipelined operations per second for one instance. Batching, a local cache in each chat server, or more instances close the gap. Or skip lookups entirely with pub/sub (next question).
With Redis pub/sub for routing, do you still need a user → server map?
Mostly not. Each chat server subscribes to a channel per connected user (or per conversation), and senders publish, so the subscriptions are the map. The load moves rather than vanishing: a lookup per recipient becomes a publish per recipient. At that scale, use sharded pub/sub (SPUBLISH, Redis 7), because classic PUBLISH in a cluster is sent to every node. PUBLISH returns how many subscribers received the message, so 0 suggests “offline: send a push notification.” In a cluster, though, that count covers only the publisher’s node.
More: Redis pub/sub, PUBLISH
When does a user's entry in that map change? Only when a server dies?
No, on every connect and disconnect. With load-based placement (power of two choices), a reconnecting user can land on a different server, and phones reconnect constantly: network switches, backgrounding, deploys. The rate is about concurrent connections ÷ average connection lifetime, for connects and again for disconnects. A crash is the case where nobody can update, so track liveness per chat server (one heartbeat key or lease each), not with a per-user TTL that would cost a refresh write per user.
Could you drop the map by hashing users to chat servers?
Yes: server = consistent_hash(user_id) needs only the server list. You lose load-aware placement, and every membership change (crash, deploy, scaling) forces the affected users to move. All servers must also agree on the ring, so the server list needs a strongly consistent store such as etcd. Power of two choices keeps placement flexible and tolerates a slightly stale server list, at the cost of a per-user map or pub/sub subscriptions.
More: Partitioning and hot keys, Why power of two choices for chat
Connections and scale
If ports are limited to ~64k, how can WhatsApp hold 2 million connections on one server?
The port limit applies to the side that opens connections, not the side that accepts them. A server keeps one listening port (443); each connection is told apart by the client’s IP and port: (client IP, client port, server IP, 443). Accepting uses up no ports. What limits a server is open files (a setting), memory (roughly 30–50 KB per connection all-in), CPU for traffic and heartbeats, and blast radius.
More: Why a server can hold millions
Sources: WhatsApp: 1 million is so 2011, Phoenix: The Road to 2 Million Websocket Connections
Where does a company actually hit its connection limit: the load balancer?
Mostly not.
- L4 tier: forwards packets, holds no sockets; limited by packets per second.
- L7 proxy tier: holds 2 sockets per user, and its upstream side hits the port limit per proxy–server pair. Users per pair = total ÷ (proxies × servers), so it only bites when very dense servers sit behind few proxies.
- Chat servers: the number you plan around (“connections per server”), limited by memory, CPU and blast radius.
Why does an L7 proxy need 2 sockets per user? Isn't one WebSocket enough?
A TCP connection runs between exactly two endpoints, and the client’s ends at the proxy. To reach the chat server, the proxy must open a second connection. It’s one upstream per user because a WebSocket frame has no “from” field: the connection itself identifies the user.
Couldn't the proxy keep one shared WebSocket to each chat server and let the server route locally?
Yes: that’s multiplexing. The proxy wraps each frame with a label ({from: Alice, …}), keeps a user → client connection map for replies, and reports connects and disconnects. It removes the port limit and most server sockets. The price: a custom, app-aware proxy; slow clients must be buffered or dropped by the proxy; one lost packet delays every user on the shared connection (head-of-line blocking); one broken connection affects everyone through it. HTTP/2 streams (RFC 8441) are the standards-based variant.
More: One shared WebSocket from the proxy, Comparing the architectures
What's the difference between a proxy and a gateway?
- Proxy (Envoy, Nginx): generic; after the upgrade it copies bytes without understanding them.
- Gateway (e.g. Slack’s gateway servers): app-specific; understands the chat protocol, keeps per-user state, fans messages out to local users, subscribes to channel servers.
In L4 → L7 → chat server, the L7 is a proxy. Multiplexing with app-specific labels pushes it toward being a gateway.
More: Terms: proxy vs gateway
WebSockets
Do WebSockets use keep-alive by default?
“Keep-alive” means three things. HTTP keep-alive (reusing a connection for the next request) doesn’t apply: a WebSocket is one long connection. TCP keepalive exists but waits 2 hours by default on Linux before probing. WebSocket ping/pong is built into the protocol, but someone must send pings: in practice the server, every 20–30 seconds.
Why use a WebSocket library instead of implementing ping/pong yourself?
The library handles protocol mechanics (handshake, framing, answering pings). Your code owns the policy: how often to ping, how long to wait for a pong, what to do when none comes. Go’s standard library has no WebSocket server, and hand-writing frame parsing teaches little about chat systems.
Does the load balancer need to understand WebSockets?
It needs to pass them through, not understand them. A plain HTTP proxy waits for request/response pairs and breaks once the connection switches protocol. Passthrough means: spot the Upgrade header, forward the handshake, then hijack the raw connection and copy bytes both ways. No frames are read.
Security and compliance
Are Slack and Messenger end-to-end encrypted?
- Slack: no. Encrypted in transit and at rest, but Slack’s servers can read messages (search, compliance exports, integrations need it).
- Messenger: personal chats yes, by default since a rollout starting December 2023; group chats were opt-in as of 2024.
- WhatsApp: yes, everything by default. Discord: calls yes (DAVE protocol), text no.
If L7 proxies terminate TLS, can the service still be end-to-end encrypted?
Yes. TLS protects each hop; end-to-end encryption protects the message itself, applied by the sender’s app with keys only the recipient’s devices hold. A proxy opening TLS finds routing metadata (to, conversation) and a sealed body it can’t read. Meta’s edge proxies terminate connections, yet Messenger personal chats are end-to-end encrypted. Conversely, connecting directly to a chat server doesn’t make anything end-to-end: the server decrypts TLS and reads.
More: TLS vs end-to-end
Is there one public/private key pair per chat?
No. Keys are layered: an identity key pair and prekeys per device (public halves in a key directory); a shared secret per pair of devices agreed with Diffie-Hellman (X3DH), even if the recipient is offline; a new key per message (Double Ratchet) for forward secrecy; sender keys for groups.
More: How the keys work
Sources: Signal: X3DH, Signal: Double Ratchet
Which architecture suits a HIPAA-compliant messaging system?
HIPAA doesn’t prescribe a topology: every component that handles PHI is in scope and needs a BAA, encryption in transit and at rest, access control and audit logs. A managed L7 load balancer under a BAA that re-encrypts to the chat servers is a good default; direct connections through L4 decrypt PHI in fewer places. Usually not end-to-end encrypted, because records must be retained and produced. Keep PHI out of push notifications and logs.
Should AI conversations be a separate system from peer conversations?
Share the platform, separate the service. AI traffic is different (streamed responses over seconds, model rate limits and cost, outages that mustn’t affect peer messaging), but a fully separate system would duplicate identity, storage, retention and audit. The pattern: an AI service that joins conversations as a bot participant via the same pub/sub, so AI-only chats and AI inside peer chats share one audited store.