Enter your search text in the box above

Select a result to preview

Real-Time Feedback Loop

#tiktok #recommendation-systems #systems-design #streaming

Real-Time Feedback Loop

TikTok's feedback loop is a continuous, real-time pipeline that runs on every user interaction. It is not a batch process that recalculates recommendations periodically. It is an always-on system where the time from "user watched something" to "that information changes what they see next" is measured in milliseconds.

The loop

User swipes to next video
    |
    v
Event emitted (watch time, completion, interactions)
    |
    v
Streaming ingestion (Kafka / Kinesis)
    |
    +--> User embedding updated (incrementally, async)
    |
    +--> Candidate retrieval pulls ~500 new videos (ANN search)
    |
    +--> Ranking model scores candidates (pre-ranking during current video)
    |
    v
Top result cached and ready
    |
    v
User swipes
    |
    v
Cached result served (delivery, not computation)
Pipelining, not sequential

The naive mental model is sequential: watch -> update -> rank -> serve. The real system pipelines these steps so that by the time you're three seconds into a video, the system is already scoring candidates for what comes next. The swipe triggers delivery, not ranking.

Infrastructure

The feedback loop relies on several infrastructure components working in parallel:

Event ingestion (Kafka / Kinesis)

Every watch event, skip, replay, and interaction flows through a streaming ingestion layer. At TikTok's scale, this pipeline absorbs millions of events per minute without creating backpressure on the serving layer.

User profile storage (DynamoDB + DAX)

User embeddings and behavioral profiles are stored in a fast key-value store. DAX (DynamoDB Accelerator) provides caching in front of DynamoDB, so most profile lookups never reach the database. Target latency: single-digit milliseconds.

Model serving (SageMaker / custom endpoints)

The ranking model runs as online inference, not batch prediction. The endpoint must be warm at all times, low-latency, and versioned so ranking experiments can run in parallel. Cold starts are not a performance nuisance -- they are a product failure.

Vector search (Faiss / OpenSearch ANN index)

Candidate retrieval uses approximate nearest neighbor search across billions of pre-cached video embeddings. This is the single architectural decision that accounts for more latency savings than any model optimization.

The five latency levers

TikTok's engineering team identifies five key optimizations that keep the loop under 200ms:

1. Pre-computation

You don't rank on every swipe. You rank during the current video. While the user is three seconds into what they're watching, the system is already scoring candidates for what comes next.

2. Candidate retrieval via ANN

The ranking model is fast when you feed it 200 candidates. It becomes the bottleneck when you ask it to score millions. ANN search narrows the pool from millions to hundreds before the ranker runs.

3. Warm endpoints

A cold SageMaker endpoint adds hundreds of milliseconds to every request. Provisioned concurrency and persistent endpoints eliminate cold starts.

4. Write/read path separation

The embedding update should never block the ranking step. The event is captured immediately, the update happens asynchronously, and the ranking step reads the last known good state from cache without waiting for the write to complete.

5. Model size

The model you train offline does not have to be the model you serve. Quantization and distillation can cut inference time significantly. A smaller, faster model that updates on every swipe will outperform a larger, slower one that updates once per session.

Speed of learning beats sophistication of model

This is the same tradeoff that defines the whole system. A fast, slightly imprecise ranking engine that updates on every swipe will outperform a more sophisticated one that recalculates daily. The compound effect of thousands of micro-updates per session is what creates the feeling that TikTok already knows what you want.

Write path vs read path

Path Components Latency Blocking?
Write path Kafka -> Workers -> Parameter Servers -> Serving Nodes Async (seconds) No
Read path Cache lookup -> ANN search -> Ranking -> Re-ranking < 200ms No

The two paths are completely decoupled via streaming infrastructure. The write path updates the model in the background. The read path uses the last known good state. This separation is critical -- if the write path blocked the read path, every model update would introduce latency spikes.

Why this matters

Most recommendation systems update user profiles on a schedule (hourly, daily). TikTok updates on every interaction. This creates a fundamentally different experience:

The tightness of the feedback loop is the single most important technical factor in why TikTok's feed feels so responsive and "magical."

See also