MOOC Quality: the need for new measures
DOI:
https://doi.org/10.56059/jl4d.v3i3.165Keywords:
Massive open online courses, quality, quality assurance, indicatorsAbstract
MOOCs are re-operationalising traditional concepts in education. While they draw on elements of existing educational and learning models, they represent a new approach to instruction and learning. The challenges MOOCs present to traditional education models have important implications for approaching and assessing quality. This paper foregrounds some of the tensions surrounding notions of quality, as well as the need for new ways of thinking about and approaching quality in MOOCs.
References
Adamopoulos, A. (2013) What makes a great MOOC? An interdisciplinary analysis of student retention in online courses. Thirty Fourth International Conference on Information Systems, Milan 2013.
Admiraal., W., Huisman, B., Pilli, O. (2015). Assessment in massive open online courses. Journal of e-Learning, 13 (4), 207-216.
Agudo-Peregrina, Á., Iglesias-Pradas, S., Conde- González, M., & Hernández-García, Á. (2014).Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31, 542–550.
Alario-Hoyos, C., Perez-Sanagustin, M., Cormier, D., & Delgado-Kloos, C. (2014). Proposal for a conceptual framework for educators to describe and design MOOCs. Journal of universal computer science, 20(1), 6-23.
Amo, D. (2013). MOOCs: Experimental approaches for quality in pedagogical and design fundamentals. TEEM '13, November 14 - 15 2013, Salamanca, Spain.
Arbaugh, J. B. (2014). System, scholar, or students? Which most influences online MBA course effectiveness? Journal of Computer Assisted Learning. 30 (4).
Bellum, J. (2013). The Adult Learner and MOOCs. EDUCAUSE Review. Available from: http://www.educause.edu/ero/article/adult-learner-and-moocs
Biggs, J. (1993). From theory to practice: A cognitive systems approach. Higher Education Research & Development, 12(1), 73-85,
Bond, P. (2015) Information literacy in MOOCs. Current issues in emerging elearning, 2 (1), article 6.
Bruff, D. O., Fisher, D. H., McEwen, K. E., & Smith, B. E. (2013). Wrapping a MOOC: Student perceptions of an experiment in blended learning. Journal of Online Learning and Teaching, 9(2), 187-199.
Butcher, N., Hoosen, S., Uvalić-Trumbić, S., Daniel, J. (2013). Guide to quality in post-traditional online higher education. Available from: http://www.eadtu.eu/home/policy-areas/quality-assurance/publications/227-guide-to-quality-in-post-traditional-online-higher-education.
Caulfield, M., Collier, A., & Halawa, S. (2013, October 7). Rethinking online community in MOOCs used for blended learning. EDUCAUSE Review. Retrieved from http://www.educause.edu/ero/article/rethinking-online-community-moocs-used- blended-learning
Chandrasekaran, M., Ragupathi, K., Kan, M., & Tan, B. (2015). Towards feasible instructor intervention in MOOC discussion forums. Thirty Sixth International Conference on Information Systems, Fort Worth 2015.
Christensen, G., Steinmetz, A., Alcorn, B., Bennett, A., Woods, D., & Emanuel, E. J. (2013). The MOOC phenomenon: Who takes Massive Open Online Courses and why? Available from: http://ssrn.com/abstract=2350964.
Conole, G. (2008) New schemas for mapping pedagogies and technologies. Available from: http://www.ariadne.ac.uk/issue56/conole.
Conole, G. (2013). MOOCs as disruptive technologies: strategies for enhancing the learner experience and quality of MOOCs. RED - Revista de Educación a Distancia, 39. Retrieved from http://www.um.es/ead/red/39/conole.pdf.
Daniel, J. (2012). Making Sense of MOOCs: Musings in a Maze of Myth, Paradox and Possibility. Journal of Interactive Media in Education, 2012(3). Doi: http://doi.org/10.5334/2012-18.
Daradoumis, T., Bassi, R., Xhafa, F., & Caballe, S. (2013). A Review on Massive E¬Learning (MOOC) Design, Delivery and Assessment. In Proceedings - 2013 8th International Conference on P2P, Parallel, Grid, Cloud and Internet Computing, 3PGCIC 2013 (pp. 208-213). Doi: http://doi.org/tpk.
DeBoer, J., Ho, A., Stump, G. & B. (2014) Changing “Course”: Reconceptualizing Educational Variables for Massive Open Online Courses. Educational Researcher, 1-11. DOI: 10.3102/0013189X14523038.
Department for Business, Innovation & Skills (2013). The maturing of the MOOC. London: Department for Business, Innovation and Skills.
Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrolment physics class. Science, 332 (6031), 862-864.
Davis, H., Dickens, K., Leon, M., del Mar Sanchez Ver, M., & White, S. (2014). MOOCs for Universities and Learners: An analysis of motivating factors. In: 6th International Conference on Computer Supported Education, 01 - 03 Apr 2014.
Dillenbourg, P., Fox, A., Kirchner, C., Mitchell, J., & Wirsing, M. (2013). Massive Open Online Courses: Current State and Perspectives. Manifesto from Dagstuhl Perspectives Workshop. Doi: 10.4230/DagMan.4.1.1.
Dolan, V. (2014). Massive Online Obsessive Compulsion: What are They Saying Out There about the Latest Phenomenon in Higher Education? International Review of Research in Open and Distributed Learning, 15 (2).
Downes, S. (2008). Places to go: Connectivism & connective knowledge. Innovate: Journal of Online Education, 5(1). Retrieved from http://www.innovateonline.info/pdf/vol5_issue1/Places_to_Go-__Connectivism_&_Connective_Knowledge.pdf
Downes, S. (2009, February 24). Connectivist dynamics in communities [Web log post]. Retrieved from http://halfanhour.blogspot.co.uk/2009/02/connectivist-dynamics-in-communities.html
Downes, S. (2013). The Quality of Massive Open Online Courses. Available from: http://mooc.efquel.org/files/2013/05/week2-The-quality-of-massive-open-online-courses-StephenDownes.pdf.
Emanuel, E. (2013). Online education: MOOCs taken by educated few. Nature, 503. doi:10.1038/503342a.
Firmin, R., Schiorring, E., Whitmer, J., Willett, T., Collins, E. D., & Sujitparapitaya, S. (2014). Case study: Using MOOCs for conventional college coursework. Distance Education, 35(2), 178-201. doi: 10.1080/01587919.2014.917707
Gibbs, G. (2010). Dimensions of Quality. York: The Higher Education Academy.
Gillani, N. & Eynon, R. (2014). Communication patterns in massively open online courses. Internet and Higher Education, 23, 18-26.
Gillani, N., Yasserie, T., Eynon, R., & Hjorth, I. (2014). Structural limitations of learning in a crowd: communication vulnerability and information diffusion in MOOCs. Scientific Insights, 4: 6447.
Guardia, L., Maina, M. and Sangra, A. (2013). MOOC Design Principles.A Pedagogical Approach from the Learner’s Perspective. eLearning Papers, 33, 1-5.
Hattie, J. (2009). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. London and New York: Routledge.
Hew. K. (2014). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47 (2), 320-342. doi:10.1111/bjet.12235.
Ho, A. D., Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1). doi:10.2139/ssrn.2381263
Holotescu, C., Grosseck, G., Cretu, V., & Naaji, A. (2014). Integrating MOOCs in blended courses. Proceedings of the International Scientific Conference of eLearning and Software for Education, Bucharest, 243-250. doi: 10.12753/2066-026X-14-034
Hood, N., Littlejohn, A., & Milligan, C. (2015). Context counts: How learners' contexts influence learning in a MOOC. Computers & Education, 91, 83-91.
Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.
iNACOL (2011). National Standards for Quality Online Courses. Vienna, VA: INACOL. Available from: http://www.inacol.org/wp-content/uploads/2015/02/national-standards-for-quality-online-courses-v2.pdf.
Israel, M. (2015). Effectiveness of Integrating MOOCs in Traditional Classrooms for Undergraduate Students. International Review of Research in Open and Distributed Learning, 16 (5), 102-118.
Istrate, O., & A. Kestens (2015). Developing and monitoring a MOOC: The IFRC experience. The 11th International Scientific Conference eLearning and software for Education Bucharest, April 23-24, 2015. Doi 10.12753/2066-026X-15-179.
Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition. International Review of Research in Open and Distributed Learning, 16 (3), 341–358.
Kanwar, A. (2013). Quality vs Quantity: Can Technology Help? Opening Keynote, 25th ICDE World Conference, Tianjin, China, October 16, 2013.
Kizilcec, R., Piech, C., & Schneider, E. (2013). Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses. LAK ’13 Leuven, Belgium.
Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and Intention in Massive Open Online Courses: In Depth. EDUCAUSE Review Online. Available from: http://er.educause.edu/articles/2013/6/retention-and-intention-in-massive-open-online-courses-in-depth.
Kop, R., Fournier, H., Mak, J. (2011). A Pedagogy of Abundance or a Pedagogy to Support Human Beings? Participant Support on Massive Open Online Courses. International Review of Research in Open and Distributed Learning, 12, 74-93.
Lin, Y-L., Lin, H-W., & Hung, T-T. (2015). Value hierarchy for Massive Open Online Courses. Computers in Human Behaviour, 53, 408-418.
Liyanagunawardena, T., Adams, A., Williams, S. (2013). MOOCs: A Systematic Study of the Published Literature 2008-2012. International Review of Research in Open and Distributed Learning, 14 (3), 202-227.
Littlejohn, A., Hood, N., Milligan, C. & Mustain, P. (2016). Learning in MOOCs: Motivations and Self-Regulated Learning in MOOCs, Internet and Higher Education, 29, 40-48.
Littlejohn, A. & Milligan, C. (2015). Designing MOOCs for professional learners: Tools and patterns to encourage self-regulated learning, eLearning Papers, Special Issue on Design Patterns for Open Online Teaching and Learning, 42. Available from: http://www.openeducationeuropa.eu/en/node/170924
Milligan, C., Littlejohn, A., &, Margaryan, A. (2013). Patterns of Engagement in Connectivist MOOCs. MERLOT, 9 (2), 149-159.
Marks, R. B., Sibley, S. D., & Arbaugh, J. B. (2005). A Structural Equation Model of Predictors for Effective Online Learning. Journal of Management Education, 29(4), 531-563.
Mackness, J., Mak, S. F. J., & Williams, R. (2010). The ideals and reality of participating in a MOOC. In L. Dirckinck-Holmfeld, V. Hodgson, C. Jones, M. de Laat, D. McConnell, & T. Ryberg. (Eds.), Proceedings of the Seventh International Conference on Networked Learning (pp. 266-275). Lancaster, UK: University of Lancaster.
Mackness, J., Waite, M., Roberts, G., & Lovegrove, E. (2013). Learning in a small, task–oriented, connectivist MOOC: Pedagogical issues and implications for higher education. International Review of Research in Open and Distributed Learning, 14 (4), 140-159.
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80, 77-83.
Martinez, I. (2014). The Effects of Nudges on Students’ Effort and Performance: Lessons from a MOOC. Working Paper, EdPolicyWorks. Retreived from: http://curry.virginia.edu/uploads/resourceLibrary/19_Martinez_Lessons_from_a_MOOC.pdf.
Milligan, C., Littljohn, A., & Margaryan, A. (2013). Patterns of Engagement in Connectivist MOOCs. MERLOT, 9 (2), 149-159.
Munoz-Merino, P., Ruiperez-Valiente, J., Alario-Hoyos, C., Perez-Sanagustin, M., & Delgado Kloos,, C. (2015). Precise Effectiveness Strategy for analyzing the effectiveness of students with educational resources and activities in MOOCs . Computers in Human Behaviour, 47, 108-118.
Quality Matters (2013). Quality Matters Continuing and Professional Education Rubric Standards. Available from: https://www.qmprogram.org.
Rayyan, S., Seaton, D., Belcher, J., Pritchard, D., & Chuang, I. (2013). Participation And performance In 8.02x Electricity And Magnetism: The First Physics MOOC From MITx. arXiv preprint arXiv:1310.3173. Available from: http://arxiv.org/abs/1310.3173.
Reich, J. (2013). MOOC Completion and Retention in the Context of Student Intent. EDUCAUSE Review. Available from: http://er.educause.edu/articles/2014/12/mooc-completion-and-retention-in-the-context-of-student-intent.
Rientes, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UK. Journal of Interactive Media in Education. Doi: http://dx.doi.org/10.5334/jime.394
Rhoads, R.A., Berdan, J. & Toven-Lindsey, B. (2013). The Open Courseware Movement in Higher Education: Unmasking Power and Raising Questions about the Movement’s Democratic Potential. Educational Theory. 63 (1), pp.87–110.
Rosé, C., Goldman, P., Zoltners Sherer, J., & Resnick, L. (2015). Supportive technologies for group discussion in MOOCs. Current Issues in Emerging eLearning, 2 (1), Article 5.
Rosewell, J., & Jansen, D. (2014). The OpenupEd quality label: benchmarks for MOOCs. INNOQUAL: The International Journal for Innovation and Quality in Learning, 2 (3), 88–100.
Ross, J., Sinclair, C., Knox, J., & Macleod, H. (2014). Teacher experiences and academic identity: The missing components of MOOC pedagogy.Journal of Online Learning and Teaching, 10(1), 57.
Shah, D. (2015). MOOCs in 2015: Breaking Down the Numbers. EdSurge, 28 December 2015. Available from: https://www.edsurge.com/news/2015-12-28-moocs-in-2015-breaking-down-the-numbers.
Siemens, G. (2012). MOOCs are really a platform. ELearnSpace. Available at: http://www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform/.
Sinha, T., Li, N., Jermann, P., Dillenbourg, P. (2014). Capturing “attrition intensifying” structural traits from didactic interaction sequences of MOOC learners. Proceedings of the 2014 Empirical Methods in Natural Language Processing Workshop on Modeling Large Scale Social Interaction in Massively Open Online Courses.
Skrypnyk, O., de Vries, P., and Hennis, T. (2015). Reconsidering Retention in MOOCs: the Relevance of Formal Assessment and Pedagogy. EMOOCS conference 2015, Third European MOOCs stakeholders summit, Mons, Belgium.
Tabba Y., & Medouri, A. (2013). LASyM: A Learning Analytics System for MOOCs. International Journal of Advanced Computer Science and Applications, 4 (5), 113-119.
Tempelaar, Dirk T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: learning analytics in a data-rich context. Computers in Human Behavior, 47, 157–167.
Uvalic-Trumbic, S. (2013). MOOCs – Mistaking brand for quality? University World News. Retreived from: http://www.universityworldnews.com/article.php?story=20130206180425691.
De Waard, I., Abajian, S., Gallagher, M, Hogue, R., Keskin, N., Koutropoulos, A. & Rodriguez, O. (2011). Using mLearning and MOOCs to understand chaos, emergence, and complexity in education. International Review of Research in Open and Distance Learning. 12 (7), pp.94–115.
Wang Y., & Baker, R. (2015). Content or platform: Why do students complete MOOCs?. MERLOT, 11 (1), 17-30.
Warburton, S., & Mor, Y. (2015). Configuring Narratives, Patterns and Scenarios in the Design of Technology Enhanced Learning. In: M. Maina et al. (Eds.), The Art & Science of Learning Design, 93–104.
Wartell, M. (2012). A New Paradigm for Remediation: MOOCs in Secondary Schools. EDUCAUSE Review. Retrieved from: http://er.educause.edu/articles/2012/11/a-new-paradigm-for-remediation-moocs-in-secondary-schools.
Wen, M., Yang, D., & Rosé, C. P. (2014a). Sentiment Analysis in MOOC Discussion Forums: What does it tell us? Proceedings of Educational Data Mining. Available from: http://www.cs.cmu.edu/~mwen/papers/edm2014-camera-ready.pdf.
Wen, M., Yang, D., Rosé, D. (2014b). Linguistic Reflections of Student Engagement in Massive Open Online Courses, in Proceedings of the International Conference on Weblogs and Social Media. Available from: http://www.cs.cmu.edu/~mwen/papers/icwsm2014-camera-ready.pdf.
Whitelock, D., Gilbert, L. & Wills, G. (2013). Feedback generators: providing feedback in MOOCs. In CAA 2013 International Conference. University of Southampton. Retrieved from: http://caaconference.com.
Yang, D., Wen, M., Kumar, A., Xing, E., & Rosé, C. (2014). Towards an integration of text and graph clustering methods as a lens for studying social interaction in MOOCs. International Review of Research in Open and Distributed Learning, 15 (5).
Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D. & Emanuel, E. (2015). Who’s benefiting from MOOCs, and Why?. Harvard Business Review, September 2015.
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Accepted 2016-08-16
Published 2016-11-10
Contents of the 
