Real-world large-scale study on adaptive notification scheduling on smartphones

Human attention has bottlenecked today's ubiquitous computing environment where users are consuming increasing amounts of information from numerous applications and services. Since the system-to-user provision of information is becoming more proactive, mainly via push notifications that often cause interruption at the users' side, attention management is becoming very important. Despite the many existing studies concerned with detecting opportune moments to present such push information to the users (in a way that preserves the user's attention and lowers their cognitive load and frustration), there is little evaluation of such systems in the real-world production environments. Overlooked areas of study also include the examination of real users and notification contents. In this paper, we present various results from the first extensive evaluation on user's interruptibility and engagement in the real-world environment with a market-leading smartphone application that boasts a large number of users, including real notification content on the application. Following our previous mobile-sensing and machine learning-based interruptibility estimation approach, which was an effective study in its own right (Okoshi et al., 2015 [18,19]), we embedded a logic with the same approach in the “Yahoo! JAPAN” Android app (one of the most popular applications on the national market). The results from our large-scale in-the-wild user study (that included more than 680,000 users and spanned three weeks) indicate that, in most cases, delaying the notification delivery until an interruptible moment is detected is beneficial to users. The practice results in significant reduction of user response time (49.7%) when compared to delivering the notifications immediately. We observed a higher number of notifications opened in our system and constant improvement in user engagement levels throughout the entire study period. We also observed differences in click rates among different days and time among users with different attributes (e.g., age, gender, and occupation). Additional evaluation of our revised system, which can train and distribute different models for weekdays and weekends, improved the click rates during weekends. This negated the performance degradation previously observed during weekends.