Decomposition and Interleaving for Variance Reduction of Post-click Metrics

by   Kojiro Iizuka, et al.

In this study, we propose an efficient method for comparing the post-click metric (e.g., dwell time and conversion rate) of multiple rankings in online experiments. The proposed method involves (1) the decomposition of the post-click metric measurement of a ranking into a click model estimation and a post-click metric measurement of each item in the ranking, and (2) interleaving of multiple rankings to produce a single ranking that preferentially exposes items possessing a high population variance. The decomposition of the post-click metric measurement enables the free layout of items in a ranking and focuses on the measurement of the post-click metric of each item in the multiple rankings. The interleaving of multiple rankings reduces the sample variance of the items possessing a high population variance by optimizing a ranking to be presented to the users so that those items received more samples of the post-click metric. In addition, we provide a proof that the proposed method leads to the minimization of the evaluation error in the ranking comparison and propose two practical techniques to stabilize the online experiment. We performed a comprehensive simulation experiment and a real service setting experiment. The experimental results revealed that (1) the proposed method outperformed existing methods in terms of efficiency and accuracy, and the performance was especially remarkable when the input rankings shared many items, and (2) the two stabilization techniques successfully improved the evaluation accuracy and efficiency.


Consensus measure of rankings

A ranking is an ordered sequence of items, in which an item with higher ...

Fairness and Diversity for Rankings in Two-Sided Markets

Ranking items by their probability of relevance has long been the goal o...

Modelling rankings in R: the PlackettLuce package

This paper presents the R package PlackettLuce, which implements a gener...

Developing a Ranking Problem Library (RPLIB) from a data-oriented perspective

We present an improved library for the ranking problem called RPLIB. RPL...

Personalization in E-Grocery: Top-N versus Top-k Rankings

Business success in e-commerce depends on customer perceived value. A cu...

Dimensionality Reduction and (Bucket) Ranking: a Mass Transportation Approach

Whereas most dimensionality reduction techniques (e.g. PCA, ICA, NMF) fo...

Overlaying Quantitative Measurement on Networks: An Evaluation of Three Positioning and Nine Visual Marker Techniques

We report results from an experiment on ranking visual markers and node ...

Please sign up or login with your details

Forgot password? Click here to reset