PMID- 32338618 OWN - NLM STAT- MEDLINE DCOM- 20201020 LR - 20201020 IS - 1438-8871 (Electronic) IS - 1439-4456 (Print) IS - 1438-8871 (Linking) VI - 22 IP - 4 DP - 2020 Apr 27 TI - Massive Open Online Course Evaluation Methods: Systematic Review. PG - e13851 LID - 10.2196/13851 [doi] LID - e13851 AB - BACKGROUND: Massive open online courses (MOOCs) have the potential to make a broader educational impact because many learners undertake these courses. Despite their reach, there is a lack of knowledge about which methods are used for evaluating these courses. OBJECTIVE: The aim of this review was to identify current MOOC evaluation methods to inform future study designs. METHODS: We systematically searched the following databases for studies published from January 2008 to October 2018: (1) Scopus, (2) Education Resources Information Center, (3) IEEE (Institute of Electrical and Electronic Engineers) Xplore, (4) PubMed, (5) Web of Science, (6) British Education Index, and (7) Google Scholar search engine. Two reviewers independently screened the abstracts and titles of the studies. Published studies in the English language that evaluated MOOCs were included. The study design of the evaluations, the underlying motivation for the evaluation studies, data collection, and data analysis methods were quantitatively and qualitatively analyzed. The quality of the included studies was appraised using the Cochrane Collaboration Risk of Bias Tool for randomized controlled trials (RCTs) and the National Institutes of Health-National Heart, Lung, and Blood Institute quality assessment tool for cohort observational studies and for before-after (pre-post) studies with no control group. RESULTS: The initial search resulted in 3275 studies, and 33 eligible studies were included in this review. In total, 16 studies used a quantitative study design, 11 used a qualitative design, and 6 used a mixed methods study design. In all, 16 studies evaluated learner characteristics and behavior, and 20 studies evaluated learning outcomes and experiences. A total of 12 studies used 1 data source, 11 used 2 data sources, 7 used 3 data sources, 4 used 2 data sources, and 1 used 5 data sources. Overall, 3 studies used more than 3 data sources in their evaluation. In terms of the data analysis methods, quantitative methods were most prominent with descriptive and inferential statistics, which were the top 2 preferred methods. In all, 26 studies with a cross-sectional design had a low-quality assessment, whereas RCTs and quasi-experimental studies received a high-quality assessment. CONCLUSIONS: The MOOC evaluation data collection and data analysis methods should be determined carefully on the basis of the aim of the evaluation. The MOOC evaluations are subject to bias, which could be reduced using pre-MOOC measures for comparison or by controlling for confounding variables. Future MOOC evaluations should consider using more diverse data sources and data analysis methods. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/12087. CI - (c)Abrar Alturkistani, Ching Lam, Kimberley Foley, Terese Stenfors, Elizabeth R Blum, Michelle Helena Van Velthoven, Edward Meinert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 27.04.2020. FAU - Alturkistani, Abrar AU - Alturkistani A AUID- ORCID: 0000-0001-7935-8870 AD - Global Digital Health Unit, Imperial College London, London, United Kingdom. FAU - Lam, Ching AU - Lam C AUID- ORCID: 0000-0002-9137-749X AD - Digitally Enabled PrevenTative Health Research Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom. FAU - Foley, Kimberley AU - Foley K AUID- ORCID: 0000-0003-3664-8100 AD - Global Digital Health Unit, Imperial College London, London, United Kingdom. FAU - Stenfors, Terese AU - Stenfors T AUID- ORCID: 0000-0002-0854-8631 AD - Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Stockholm, Sweden. FAU - Blum, Elizabeth R AU - Blum ER AUID- ORCID: 0000-0003-3729-3946 AD - Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Stockholm, Sweden. FAU - Van Velthoven, Michelle Helena AU - Van Velthoven MH AUID- ORCID: 0000-0003-1245-8759 AD - Digitally Enabled PrevenTative Health Research Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom. FAU - Meinert, Edward AU - Meinert E AUID- ORCID: 0000-0003-2484-3347 AD - Global Digital Health Unit, Imperial College London, London, United Kingdom. AD - Digitally Enabled PrevenTative Health Research Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom. LA - eng PT - Journal Article PT - Research Support, Non-U.S. Gov't PT - Systematic Review DEP - 20200427 PL - Canada TA - J Med Internet Res JT - Journal of medical Internet research JID - 100959882 SB - IM MH - Cross-Sectional Studies MH - *Education, Distance MH - Humans MH - *Learning MH - Research Design PMC - PMC7215503 OTO - NOTNLM OT - computer-assisted instruction OT - learning OT - online learning COIS- Conflicts of Interest: None declared. EDAT- 2020/04/28 06:00 MHDA- 2020/10/21 06:00 PMCR- 2020/04/27 CRDT- 2020/04/28 06:00 PHST- 2019/02/27 00:00 [received] PHST- 2020/01/22 00:00 [accepted] PHST- 2019/11/20 00:00 [revised] PHST- 2020/04/28 06:00 [entrez] PHST- 2020/04/28 06:00 [pubmed] PHST- 2020/10/21 06:00 [medline] PHST- 2020/04/27 00:00 [pmc-release] AID - v22i4e13851 [pii] AID - 10.2196/13851 [doi] PST - epublish SO - J Med Internet Res. 2020 Apr 27;22(4):e13851. doi: 10.2196/13851.