Pushing to the Limit: A Pre-Analysis Plan

We study the effects of mobile push frequency on app visits, in-app behavior, and opt-outs in a large-scale randomized field experiment at a European provider of customer loyalty programs. We randomly assign one million app users to 6 groups with a maximum of 0, 1, 2, 3, 4, and 5 weekly push notifications, respectively, whereby notifications are allocated according to a personalized relevance score. We further include push-specific randomized control groups to explore the causal effects of individual notifications. We hypothesize that a higher frequency of push notifications strongly increases app visits and moderately affects in-app coupon usage, both of which increase business value. We further hypothesize that a higher frequency strongly increases user opt-outs so that users are unavailable for future notifications, thus decreasing business value. We expect considerable heterogeneity in the optimal notification frequency, suggesting that a personalized approach enabled by causal machine learning outperforms any strategy that notifies users with indiscriminate frequency.