Assistant Professor Oregon Health & Science University Portland, Oregon, United States
Background: Mobile apps and websites designed to screen children for autism may be useful in improving autism detection. Parents may find a range of resources when using their mobile devices to search for ways to evaluate their child for signs of autism. Given that website and mobile app content is largely unregulated, the quality of such resources may be highly variable. Objective: To evaluate the quality of available digital tools that screen for autism using a comprehensive rubric and 2 standardized instruments. Design/Methods: A previous study identified available digital tools that screen for autism, defined as a mobile app or website for which a parent inputs data, and which is offered in English. Google Search and Apple and Google app stores were searched in the US, UK, India, Australia, and Canada. 15 websites and 8 apps were identified in all. In this study, we evaluated 21 of these tools (1 not available, 1 insufficient reviews). The tools received evaluations by a median of 4 (IQR 4) of the 9 authors, using 43 questions across 5 domains: Scientific/Clinical Basis, Functionality, Usability, Accountability, and Accessibility. Authors assigned scores 1-10 for each domain, then an independent overall score (1-10). Each author also rated tools using the System Usability Scale (1-100, normalized to letter grades A-F) and the Health IT Usability Evaluation Scale (1-5). Results: The median overall score for all tools was 3.5/10.0 (IQR 2.2), and correlated well with median domain scores. The median scores for each of Scientific/clinical basis, Functionality, Usability, Accountability, and Accessibility ranged from 3.8 to 5.0. Most tools did not address many domain components. Inter-rater agreement of overall scores was moderate (Krip-α 0.49). The mean SUS score was 62/100 (SD 25, letter grade D), and mean HITUES score was 3.4/5.0 (SD 0.6).
Conclusion(s): Using a structured rubric, authors found that the quality of autism screeners was poor overall, with shortcomings in each domain of scientific basis, functionality, usability, accountability, and accessibility. Parents should take caution using digital autism screening tools. Moderate score agreement between authors suggests reliability of these results and opportunity for rubric improvement. More work is needed to validate a comprehensive evaluation method for mobile and online screening tools that would permit systematic comparisons.
