How to Build a Recommendation Engine That Converts, Not Just Scores Well

You lead product, growth or data at a business with a catalogue: an online retailer, a marketplace, a listings site, or a subscription service. You have browsing and purchase history in your systems, and a nagging sense you should be recommending better. Maybe someone has pitched you a recommender. Maybe you already run a quiet customers also bought that nobody trusts. A vendor waves a high accuracy score, and your board asks a simpler question: does it sell more? A recommendation engine that converts answers that question. It earns its keep by showing people things they actually want, and by moving clicks, baskets and retention. Accuracy on a benchmark means nothing if it does not move those numbers.

The prize is real. At Netflix, recommendations influence about 80% of hours streamed, and the remaining 20% comes from search. But a strong offline score is not the same as a business result. As Jannach and Jugovac put it in their 2019 review, the ways recommender systems create business value are much less clear than the accuracy leaderboards suggest, and offline experiments do not settle the question. So the job is not to win a benchmark. The job is to move a number your finance team cares about.

Start with the outcomes, not the algorithm. A recommender can lift clicks on a homepage, raise average order value in a basket, and keep subscribers watching or reading long enough to renew. The size of that prize varies, but the reported figures are worth knowing before you scope a project.

~80%

of hours streamed at Netflix are influenced by recommendations (2015)

$1B+ / year

estimated combined value of personalisation and recommendations at Netflix (2015)

~12%

engagement drop if Netflix personalisation were replaced by a popularity-based ranking (2025 causal study)

10 to 15%

typical revenue lift reported for personalisation done well (McKinsey, 2021)

Treat the older numbers with care. The Netflix figure of more than $1B a year is a combined estimate for personalisation and recommendations, reported by Netflix rather than independently audited. McKinsey has estimated that about 35% of what people buy on Amazon comes from recommendations, again an estimate, not an audited fact. The newer, more rigorous work is a 2025 causal study from Netflix researchers. It found that replacing the current system with matrix factorisation would cut engagement by roughly 4%, and swapping in a popularity-based ranking would cut it by about 12%. Smaller than the folklore, and more honest about what personalisation is really worth.

There are three well-established families, defined in the canonical 2005 survey by Adomavicius and Tuzhilin. You do not need to pick a favourite. You need the one that fits your data.

• Collaborative filtering: recommend items that people with similar tastes liked in the past. People like you also liked this. It needs a decent history of interactions to work.
• Content-based: recommend items similar to the ones a person already engaged with. More of what you already read, watched or bought. It leans on item attributes, so it copes better with thin interaction data.
• Hybrid: combine the two, plus sensible business rules. Hybrids exist because they avoid the specific limitations of each pure method, which is why they usually win in practice.

Every recommender hits the same wall: it cannot recommend well without data, and at the start it has none. Adomavicius and Tuzhilin name three versions of this. The new-user problem: the system must learn a person's preferences from ratings they have not given yet. The new-item problem: a collaborative system cannot surface a fresh product until enough people have interacted with it. And sparsity: the number of ratings you already have is tiny next to the number you need to predict.

The fixes are unglamorous and they work. Lean on content signals and item attributes for new users and new items, so you can recommend before you have behaviour to learn from. Add business rules for the cases that matter: new arrivals, stock you need to move, margin, editorial picks. Fall back to sensible popularity when you know nothing, then hand over to the personalised model as soon as you have signal. None of this is exotic. It is the difference between a recommender that works on day one and one that sulks until it has a year of data.

Left alone, a recommender tends to over-recommend what is already popular. A 2024 survey of popularity bias sets out the pattern. Recommenders often focus on popular items, which limits their value for both customers and providers, and can cause reinforcement effects over time. In one large field test on a North American retailer, the recommender did lift sales of niche items, but it lifted popular items even more, and aggregate sales diversity actually fell. Already popular items get more exposure, which earns them still more exposure: a feedback loop where the rich get richer.

This is often described as filter bubbles, though the evidence for real harm is mixed, and individual choice can matter more than the algorithm. So do not overclaim the danger, but do build the guardrails. Inject diversity so the same ten items do not fill every slot. Add freshness so the feed does not go stale. Keep business rules in the loop. The aim is a recommender that widens what people discover rather than narrowing it.

Here is the discipline that separates a recommender that converts from one that only scores well. Tune against real outcomes: conversion, order value and retention. Then keep testing, because tastes and inventory never stop changing. Offline metrics are a useful filter, not a verdict.

The Netflix team gives a concrete warning. They compared two sets of similar titles for a show. One set looked more relevant to a human eye. In an A/B test it lost to the set with a stronger popularity influence. Their conclusion is blunt: intuition, even collective intuition, often picks the wrong variant, and it is frequently impossible to tell good recommendations from great ones by eye. That is why they run A/B tests focused on member retention and medium-term engagement, not on offline accuracy.

If you have a catalogue and history sitting in your systems, you are closer than you think. The hard part is not the model. It is choosing the number to move, picking the simplest approach that moves it, and proving the lift with a test tied to revenue and retention.

See how we ship outcome-first machine learning on the MERIT Vision Case Studies page, then tell us the one number you need to move.

1. The Netflix Recommender System: Algorithms, Business Value, and Innovation (Gomez-Uribe and Hunt, ACM TMIS, 2015) (accessed 2026-09-07)
2. How retailers can keep up with consumers (MacKenzie, Meyer and Noble, McKinsey, 2013) (accessed 2026-09-07)
3. The value of getting personalization right or wrong is multiplying (McKinsey, 2021) (accessed 2026-09-07)
4. The Value of Personalized Recommendations: Evidence from Netflix (Zielnicki et al., arXiv:2511.07280, 2025) (accessed 2026-09-07)
5. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions (Adomavicius and Tuzhilin, IEEE TKDE, 2005) (accessed 2026-09-07)
6. A survey on popularity bias in recommender systems (Klimashevskaia et al., User Modeling and User-Adapted Interaction, 2024) (accessed 2026-09-07)
7. Measuring the Business Value of Recommender Systems (Jannach and Jugovac, arXiv:1908.08328, 2019) (accessed 2026-09-07)
8. Filter Bubbles in Recommender Systems: Fact or Fallacy, A Systematic Review (arXiv:2307.01221) (accessed 2026-09-07)