PAPIs 2018

In this talk, we describe recent work on the offline estimation of recommender system A/B tests using counterfactual reasoning techniques. We can determine whether our customers would have clicked on what we would have recommended by adding stochasticity to our recommendations. This ensures non-zero probability of having shown our new recommendations at some point in the past, which can leverage using a technique known as Pareto-smoothed importance sampling. This allows us to create a low-bias, low-variance estimator of how our recommender systems would have performed had they been deployed.