arXiv2019
Randomized experiments (A/B testings) have become the standard way for web-facing companies to guide innovation, evaluate new products, and prioritize ideas. There are times, however, when running an experiment is too complicated (e.g., we have not built the infrastructure), costly (e.g., the intervention will have a substantial negative impact on revenue), and time-consuming (e.g., the effect may take months to materialize). Even if we can run an experiment, knowing the magnitude of the impact will significantly accelerate the product development life cycle by helping us prioritize tests and determine the appropriate traffic allocation for different treatment groups. In this setting, we should leverage observational data to quickly and cost-efficiently obtain a reliable estimate of the causal effect. Although causal inference from observational data has a long history, its adoption by data scientist in technology companies has been slow. In this paper, we rectify this by providing a brief introduction to the vast field of causal inference with a specific focus on the tools and techniques that data scientist can directly leverage. We illustrate how to apply some of these methodologies to measure the effect of contributions (e.g., post, comment, like or send private messages) on engagement metrics. Evaluating the impact of contributions on engagement through an A/B test requires encouragement design and the development of non-standard experimentation infrastructure, which can consume a tremendous amount of time and financial resources. We present multiple efficient strategies that exploit historical data to accurately estimate the contemporaneous (or instantaneous) causal effect of a user's contribution on her own and her neighbors' (i.e., the users she is connected to) subsequent visitation frequency. We apply these tools to LinkedIn data for several million members.