How would you design a real-time feature pipeline for a...

A fraud model that recomputes “transactions in the last hour” from a database query at request time will never hit a 200ms budget once you add network round-trips and aggregation cost — the aggregation has to already be done and sitting in a fast store by the time the request arrives. That’s what pushes this to a streaming architecture: transaction events publish to Kafka as they happen, a Flink job maintains rolling window aggregates continuously as events stream in, and each update writes the current value straight to Redis, so the serving path is a single sub-5ms key lookup rather than a live computation.

The genuinely hard part isn’t the happy path, it’s correctness under real-world event timing: transactions can arrive out of order or late, so the stream processor needs watermarks to decide how long to wait for stragglers before finalizing a window, and the write path needs exactly-once semantics so a retried event doesn’t get double-counted into a user’s spend total. A nightly or hourly batch job recomputing the same aggregates from the full transaction history acts as a correctness backstop, quietly fixing any drift the streaming path accumulated from late arrivals — end-to-end, a well-built version of this pipeline gets from transaction to fraud decision in well under 200ms, Kafka and Redis contributing single-digit milliseconds each and the model inference itself dominating the budget.