Two-Tower Neural Network
Two-Tower Neural Network
The two-tower neural network is the core retrieval architecture behind TikTok's recommendation system. It solves the problem of matching a user to relevant content from a pool of billions of videos in under 50 milliseconds.
The problem
Every time a user opens TikTok, the system must select approximately 500 candidate videos from a library of billions. Running a complex ranking model on every video in the library is computationally impossible. The two-tower architecture solves this by pre-computing video representations and using fast vector similarity search.
Architecture
The model consists of two completely separate neural networks (towers) that run in parallel:
User tower
- Input: watch history, completed videos, liked content, search queries, device type, time of day, current session signals
- Output: a single 256-dimensional user embedding vector -- a mathematical fingerprint of this user's content taste
- Computed: fresh at every request (reflects real-time behavior)
Video tower
- Input: visual frame features (computer vision), audio features, transcript, captions, hashtags, engagement velocity, upload recency
- Output: a 256-dimensional video embedding vector -- a mathematical fingerprint of this video's content
- Computed: once at upload time, cached in a vector database (Faiss, Milvus, or similar)
User Tower Video Tower
| |
v v
[watch history] [visual features]
[liked content] [audio features]
[session signals] [captions/hashtags]
| |
v v
256-dim user embedding 256-dim video embedding
| |
+----------> ANN Search <------+
|
v
~500 candidate videos
How matching works
The two towers produce embeddings in the same vector space. Matching is done via dot product or cosine similarity: the user embedding is compared against all pre-cached video embeddings to find the nearest neighbors.
This is performed using Approximate Nearest Neighbor (ANN) search, typically with an HNSW (Hierarchical Navigable Small World) index. ANN search trades a small amount of accuracy for massive speed gains -- it can search billions of vectors in sub-millisecond time.
Video embeddings are computed once at upload and never recomputed (unless the video's content changes). Only the user embedding is computed at request time. This means the most expensive computation happens offline. TikTok is essentially pre-indexing the entire content library for instant retrieval.
Why the split matters
Separating user and video processing into two towers is not just an architectural convenience. It enables fundamentally different update cadences:
| Tower | Update frequency | Data source |
|---|---|---|
| User tower | Every request (real-time) | Behavioral signals (implicit feedback) |
| Video tower | Once at upload (batch) | Content features (visual, audio, text) |
This split is what makes the Real-Time Feedback Loop possible. Your user embedding updates with every swipe, while video embeddings remain stable and pre-computed.
Cold start advantage
The two-tower architecture is TikTok's primary weapon against the Cold Start Problem:
- For a new user with zero history, the user tower generates a weak initial embedding, but the video tower still works perfectly. Even one completed video updates the user embedding enough to find similar content.
- For a new video, the video tower generates a content embedding at upload. The video can be matched to interested users immediately, without waiting for engagement data.
The cold start problem is solved at the architecture level, not the product level. The video tower does the heavy lifting for new content, and the user tower does the heavy lifting for new users. Neither requires the social graph.
Candidate retrieval sources
The two-tower model is the primary retrieval strategy, but TikTok uses multiple sources to ensure diversity:
| Source | Share of candidates | Purpose |
|---|---|---|
| Two-tower ANN search | ~25-30% | Interest-based discovery |
| Followee videos | ~25% | Social graph signal |
| Trending content | ~25% | Popularity signal |
| Similar-user recommendations | ~20-25% | Collaborative filtering |
See Multi-Stage Ranking Pipeline for how these candidates are refined.
Scaling considerations
- The ANN index must be updated as new videos are uploaded (thousands per minute globally)
- Vector dimensionality (256) is a trade-off: higher = more expressiveness, slower search
- Index sharding distributes the search across multiple machines
- Quantization (reducing float32 to int8) reduces memory and speeds search with minimal accuracy loss
See also
- Collisionless Embedding Tables - How unique embeddings are maintained at scale
- Multi-Stage Ranking Pipeline - What happens after candidate retrieval
- Cold Start Problem - How the two-tower architecture enables zero-history recommendations
- Real-Time Feedback Loop - How user embeddings update in real-time
- Vector Database - Infrastructure for ANN search
- TikTok vs YouTube vs Instagram - How YouTube and Meta use similar architectures differently