Related Pins at Pinterest: The Evolution of a Real-World Recommender System

Related Pins is the Web-scale recommender system that powers over 40% of user engagement on Pinterest. This paper is a longitudinal study of three years of its development, exploring the evolution of the system and its components from prototypes to present state. Each component was originally built with many constraints on engineering effort and computational resources, so we prioritized the simplest and highest-leverage solutions. We show how organic growth led to a complex system and how we managed this complexity. Many challenges arose while building this system, such as avoiding feedback loops, evaluating performance, activating content, and eliminating legacy heuristics. Finally, we offer suggestions for tackling these challenges when engineering Web-scale recommender systems.

• Publication year

  2017

• Venue

  The Web Conference

• Publication date

  2017-02-26

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3041021.3054202arXiv 1702.07969
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Visual Discovery at Pinterest

  2017cited by this paper
• Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time

  2017cited by this paper
• Power of Human Curation in Recommendation System

  2016cited by this paper
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

  2016cited by this paper
• Wide & Deep Learning for Recommender Systems

  2016cited by this paper
• Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images

  2016cited by this paper
• Deep Neural Networks for YouTube Recommendations

  2016cited by this paper
• The Netflix Recommender System

  2015cited by this paper
• Human Curation and Convnets: Powering Item-to-Item Recommendations on Pinterest

  2015cited by this paper
• Hidden Technical Debt in Machine Learning Systems

  2015influential reference
• Visual Search at Pinterest

  2015cited by this paper
• WTF: the who to follow service at Twitter

  2013cited by this paper
• Efficient Estimation of Word Representations in Vector Space

  2013cited by this paper
• The YouTube video recommendation system

  2010cited by this paper
• Learning to rank for information retrieval

  2009cited by this paper
• Collaborative Filtering for Implicit Feedback Datasets

  2008cited by this paper
• Video suggestion and discovery for youtube: taking random walks through the view graph

  2008cited by this paper
• Pagerank for product image search

  2008cited by this paper
• Scalable music recommendation by search

  2007cited by this paper
• Comparing Click Logs and Editorial Labels for Training Query Rewriting

  2007cited by this paper
• Learning to rank using gradient descent

  2005cited by this paper
• Optimizing search engines using clickthrough data

  2002cited by this paper

• Warmer for Less: A Cost-Efficient Strategy for Cold-Start Recommendations at Pinterest

  2025influential citation
• Selecting User Histories to Generate LLM Users for Cold-Start Item Recommendation

  2025cites this paper
• Save, Revisit, Retain: A Scalable Framework for Enhancing User Retention in Large-Scale Recommender Systems

  2025influential citation
• Synergizing Implicit and Explicit User Interests: A Multi-Embedding Retrieval Framework at Pinterest

  2025cites this paper
• Quake: Adaptive Indexing for Vector Search

  2025cites this paper
• The Limits of Graph Samplers for Training Inductive Recommender Systems

  2025cites this paper
• MicroNN: An On-device Disk-resident Updatable Vector Database

  2025cites this paper
• Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery Marketplace

  2025cites this paper
• Reducing Re-Indexing for Top-k Personalized PageRank Computation on Dynamic Graphs

  2025cites this paper
• HMR-LLM: Hybrid Multimodal Sequential Recommendation with LLM Reranking

  2025cites this paper
• Mitigating Sample Selection Bias with Robust Domain Adaption in Multimedia Recommendation

  2024cites this paper
• On the Analysis of Two-Stage Stochastic Bandit

  2024cites this paper
• Topology-monitorable Contrastive Learning on Dynamic Graphs

  2024cites this paper
• Personalized PageRanks over Dynamic Graphs - The Case for Optimizing Quality of Service

  2024cites this paper
• Recommender System-Induced Eating Disorder Relapse: Harmful Content and the Challenges of Responsible Recommendation

  2024cites this paper
• Position bias in features

  2024cites this paper
• Incremental IVF Index Maintenance for Streaming Vector Search

  2024cites this paper
• Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models

  2023cites this paper
• Beyond Similarity: Relation-Based Collaborative Filtering

  2023cites this paper
• High-Throughput Vector Similarity Search in Knowledge Graphs

  2023cites this paper
• Enhancing Image Recommendations in Social Contexts using Hierarchical Attention

  2023cites this paper
• Characterizing Impression-Aware Recommender Systems

  2023cites this paper
• Incorporating Impressions to Graph-Based Recommenders

  2023cites this paper
• On Gradient Boosted Decision Trees and Neural Rankers: A Case-Study on Short-Video Recommendations at ShareChat

  2023cites this paper
• Interdisciplinary approach to defining outdoor places of knowledge work: quantified photo analysis

  2023cites this paper
• Prior-Guided Accuracy-Bias Tradeoff Learning for CTR Prediction in Multimedia Recommendation

  2023cites this paper
• Criterion-based Heterogeneous Collaborative Filtering for Multi-behavior Implicit Recommendation

  2023cites this paper
• Neural_BPR: Multi-processing popularity bias mitigating method in recommendation systems

  2023cites this paper
• Workshop on Learning and Evaluating Recommendations with Impressions (LERI)

  2023cites this paper
• Transfer learning for collaborative recommendation with biased and unbiased data

  2023cites this paper
• Impression-Aware Recommender Systems

  2023influential citation
• Impressions in Recommender Systems: Present and Future

  2023cites this paper
• Online Matching: A Real-time Bandit System for Large-scale Recommendations

  2023cites this paper
• Streaming CTR Prediction: Rethinking Recommendation Task for Real-World Streaming Data

  2023cites this paper
• TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest

  2023cites this paper
• Addressing Fast Changing Fashion Trends in Multi-Stage Recommender Systems

  2023cites this paper
• KDCRec: Knowledge Distillation for Counterfactual Recommendation via Uniform Data

  2023influential citation
• Multi-Task Processing in Vertex-Centric Graph Systems: Evaluations and Insights

  2023cites this paper
• Single-Source Personalized PageRanks With Workload Robustness

  2023cites this paper
• Building and Deploying a Multi-Stage Recommender System with Merlin

  2022cites this paper
• GDSRec: Graph-Based Decentralized Collaborative Filtering for Social Recommendation

  2022cites this paper
• Understanding Scaling Laws for Recommendation Models

  2022cites this paper
• KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos

  2022cites this paper
• AutoShard: Automated Embedding Table Sharding for Recommender Systems

  2022cites this paper
• VC-Tune: Tuning and Exploring Distributed Vertex-Centric Graph Systems

  2022cites this paper
• Learning to Rank For Push Notifications Using Pairwise Expected Regret

  2022cites this paper
• Recommendation Systems with Distribution-Free Reliability Guarantees

  2022cites this paper
• Directional Adversarial Training for Robust Ownership-Based Recommendation System

  2022cites this paper
• kNN-Embed: Locally Smoothed Embedding Mixtures For Multi-interest Candidate Retrieval

  2022cites this paper
• Revisiting Negative Sampling vs. Non-sampling in Implicit Recommendation

  2022cites this paper
• SuperCone: Modeling Heterogeneous Experts with Concept Meta-learning for Unified Predictive Segments System

  2022cites this paper
• MetaCon: Unified Predictive Segments System with Trillion Concept Meta-Learning

  2022cites this paper
• Bagpipe: Accelerating Deep Recommendation Model Training

  2022cites this paper
• Homogeneous-Heterogeneous Interaction Graph for Deep Learning-based Recommendation Systems

  2022cites this paper
• Biomedical Multi-hop Question Answering Using Knowledge Graph Embeddings and Language Models

  2022cites this paper
• DreamShard: Generalizable Embedding Table Placement for Recommender Systems

  2022cites this paper
• Personalized Search Engine Using Binary Tree Traversal (BTT) - A Survey

  2021cites this paper
• Factor-level Attentive ICF for Recommendation

  2021cites this paper
• Regularized Categorical Embedding for Effective Demand Forecasting of Bike Sharing System

  2021cites this paper
• Path-based Deep Network for Candidate Item Matching in Recommenders

  2021cites this paper
• Criterion-based Heterogeneous Collaborative Filtering for Multi-behavior Implicit Recommendation

  2021cites this paper
• Bootstrapping Recommendations at Chrome Web Store

  2021influential citation
• Compound Target Identification in Tissue-Specific Interaction Networks

  2021cites this paper
• CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation

  2021cites this paper
• Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations

  2021cites this paper
• Transfer Learning in Collaborative Recommendation for Bias Reduction

  2021cites this paper
• CuRL: Coupled Representation Learning of Cards and Merchants to Detect Transaction Frauds

  2021cites this paper
• Studying Emerging New Contexts for Museum Digitisations on Pinterest

  2021cites this paper
• Agenda: Robust Personalized PageRanks in Evolving Graphs

  2021cites this paper
• Vandalism Detection in OpenStreetMap via User Embeddings

  2021cites this paper
• Persia: A Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters

  2021cites this paper
• Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters

  2021cites this paper
• Item-Based Collaborative Memory Networks for Recommendation

  2020cites this paper
• Recommendations for Different Tasks Based on the Uniform Multimodal Joint Representation

  2020cites this paper
• A Matrix Chernoff Bound for Markov Chains and Its Application to Co-occurrence Matrices

  2020cites this paper
• Self-supervised Learning for Large-scale Item Recommendations

  2020cites this paper
• BATON: Batch One-Hop Personalized PageRanks with Efficiency and Accuracy

  2020cites this paper
• Concentration Bounds for Co-occurrence Matrices of Markov Chains

  2020cites this paper
• Self-supervised Learning for Deep Models in Recommendations

  2020cites this paper
• A General Knowledge Distillation Framework for Counterfactual Recommendation via Uniform Data

  2020influential citation
• Attribute-based Propensity for Unbiased Learning in Recommender Systems: Algorithm and Case Studies

  2020cites this paper
• Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations

  2020cites this paper
• Improving Recommendation Quality in Google Drive

  2020cites this paper
• The Difference Between a Click and a Cart-Add: Learning Interaction-Specific Embeddings

  2020cites this paper
• Expanding Taxonomies with Implicit Edge Semantics

  2020cites this paper
• Personalized Page Rank on Knowledge Graphs: Particle Filtering is all you need!

  2020cites this paper
• Handling Position Bias for Unbiased Learning to Rank in Hotels Search

  2020cites this paper
• Efficient Neural Matrix Factorization without Sampling for Recommendation

  2020cites this paper
• Deep learning techniques for rating prediction: a survey of the state-of-the-art

  2020cites this paper
• Comprehensive Empirical Evaluation of Deep Learning Approaches for Session-Based Recommendation in E-Commerce

  2020cites this paper
• An Intelligent Recommendation for Social Contextual Image Using Hybrid Model

  2020cites this paper
• Approach to auto-recognition of design elements for the intelligent management of interior pictures

  2019cites this paper
• Efficient Batch One-Hop Personalized PageRanks

  2019cites this paper
• A Novel Ensemble Approach for Click-Through Rate Prediction Based on Factorization Machines and Gradient Boosting Decision Trees

  2019cites this paper
• Improving Relevance Prediction with Transfer Learning in Large-scale Retrieval Systems

  2019cites this paper
• Expanding the Usage of Web Archives by Recommending Archived Webpages Using Only the URI

  2019cites this paper
• Beyond Similarity: Relation Embedding with Dual Attentions for Item-based Recommendation

  2019cites this paper
• Learning to be Relevant: Evolution of a Course Recommendation System

  2019cites this paper
• Learning a Unified Embedding for Visual Search at Pinterest

  2019influential citation
• Parallelizing approximate single-source personalized PageRank queries on shared memory

  2019cites this paper