Abstract: Although large-scale online learning increasingly succeeds in attracting learners worldwide, to date it fails to deliver on its promise. We first show the immense popularity of online learning and discuss its (unsatisfactory) effectiveness. We then discuss large-scale online randomized controlled experiments (A/B tests) as a powerful complimentary means to enable the desired leap forward. Although these experiments are widely and intensively used for web page optimization, and are slowly being adopted by the online learning community, their use, benefits, and challenges have only limitedly seeped through to the larger learning community. We summarize existing efforts in employing A/B tests in online learning, argue that such tests should take into account the typical nature of (online) learning, and encourage the use of knowledge from the various learning sciences to identify interventions that promise improved learning. We finally discuss both the limitations and promises of A/B tests, and show how such tests can ultimately contribute to learning that is tailored to each individual learner. The insights and priorities that arise from this overview and synthesis of A/B tests in online learning may help advance and direct the field.

Keywords: a/b test, learning intervention, online experimentation, online learning, online randomized controlled field experiment, research methodology