Evaluating Expert Curation in a Baby Milestone Tracking App

摘要

Early childhood developmental screening is critical for timely detection and intervention. babyTRACKS is a free, live, interactive developmental tracking mobile app with over 3,000 children's diaries. Parents write or select short milestone texts, like "began taking first steps", to record their babies' developmental achievements, and receive crowd-based percentiles to evaluate development and catch potential delays. Currently, an expert-based Curated Crowd Intelligence (CCI) process manually groups incoming novel parent-authored milestone texts according to their similarity to existing milestones in the database (for example, "starting to walk"), or determining that the milestone represents a new developmental concept not seen before in another child's diary. CCI cannot scale well, however, and babyTRACKS is mature enough, with a rich enough database of existing milestone texts, to now consider machine learning tools to replace or assist the human curators. Three new studies explore (1) the usefulness of automation, by analyzing the human cost of CCI and how the work is currently broken down; (2) the validity of automation, by testing the inter-rater reliability of curators; and (3) the value of automation, by appraising the "real world" clinical value of milestones when assessing child development. We conclude that automation can indeed be appropriate and helpful for a large percentage, though not all, of CCI work. We further establish realistic upper bounds for algorithm performance; confirm that the babyTRACKS milestones dataset is valid for training and testing purposes; and verify that it represents clinically meaningful developmental information.