Cluster-Adaptive Network A/B Testing: From Randomization to Estimation

by   Yifan Zhou, et al.

A/B testing is an important decision-making tool in product development for evaluating user engagement or satisfaction from a new service, feature or product. The goal of A/B testing is to estimate the average treatment effects (ATE) of a new change, which becomes complicated when users are interacting. When the important assumption of A/B testing, the Stable Unit Treatment Value Assumption (SUTVA), which states that each individual's response is affected by their own treatment only, is not valid, the classical estimate of the ATE usually leads to a wrong conclusion. In this paper, we propose a cluster-adaptive network A/B testing procedure, which involves a sequential cluster-adaptive randomization and a cluster-adjusted estimator. The cluster-adaptive randomization is employed to minimize the cluster-level Mahalanobis distance within the two treatment groups, so that the variance of the estimate of the ATE can be reduced. In addition, the cluster-adjusted estimator is used to eliminate the bias caused by network interference, resulting in a consistent estimation for the ATE. Numerical studies suggest our cluster-adaptive network A/B testing achieves consistent estimation with higher efficiency. An empirical study is conducted based on a real world network to illustrate how our method can benefit decision-making in application.


A Method for Measuring Network Effects of One-to-One Communication Features in Online A/B Tests

A/B testing is an important decision making tool in product development ...

Minimizing Interference and Selection Bias in Network Experiment Design

Current approaches to A/B testing in networks focus on limiting interfer...

An Online Sequential Test for Qualitative Treatment Effects

Tech companies (e.g., Google or Facebook) often use randomized online ex...

Experimentation for Homogenous Policy Change

When the Stable Unit Treatment Value Assumption (SUTVA) is violated and ...

LinkLouvain: Link-Aware A/B Testing and Its Application on Online Marketing Campaign

A lot of online marketing campaigns aim to promote user interaction. The...

Using Ego-Clusters to Measure Network Effects at LinkedIn

A network effect is said to take place when a new feature not only impac...

Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Background: The Expected Value of Sample Information (EVSI) calculates t...

Please sign up or login with your details

Forgot password? Click here to reset