Design an embedding-based system that serves similarity...

At a billion users, several decisions that don’t matter at smaller scale become the entire design.

• Embedding model: two-tower architecture. Users and items are encoded into the same vector space by two independent towers, so a user embedding can be computed without needing any item data present at the same time, and vice versa. This decoupling is what makes everything downstream — batch generation, sharding, independent refresh cadences — actually tractable at this scale; a single joint model that needs both sides present to score anything doesn’t factor this way.
• Batch generation, not on-demand. Computing a billion embeddings can’t happen synchronously per request. A scheduled batch job (Spark or equivalent) generates embeddings for the full user base on a cadence — daily or weekly — while a separate, cheaper incremental path updates only users who were active recently, so freshness doesn’t require a full recompute.
• Approximate nearest-neighbor index. Exact nearest-neighbor search stops being fast well before a billion vectors — an ANN structure (a managed vector database or a self-hosted FAISS index) is mandatory, trading a small amount of recall for the latency the scale demands.
• Sharding by user ID hash. Partitioning the index this way is what makes queries parallelizable across many machines rather than all landing on one unsharded structure that can’t keep up regardless of how it’s indexed internally.
• Caching for popular lookups. A layer (Redis) holding embeddings for the users or items that get looked up repeatedly cuts both latency and load on the index for the traffic that dominates in practice.
• Two-speed update strategy. Incremental refresh handles anything time-sensitive; a full periodic recompute corrects whatever drift the incremental path accumulates over time, since incremental-only updates gradually diverge from a from-scratch computation.

The realistic latency target at this scale is sub-50ms, not sub-5ms — and being explicit about that distinction, rather than promising single-digit-millisecond latency at a billion vectors, is itself a signal of having actually operated a system at this scale.