Week 7: MOOCs & Quality by Martin Weller

avatarMartin Weller is Professor of Educational Technology at the UK Open University. He chaired the OU’s first major elearning course in 1999 with 15,000 students, and has been Director of the VLE and SocialLearn projects. His research interests are in open education, the impact of new technologies, and digital scholarship. His recent book, The Digital Scholar, was published by Bloomsbury Academic under a Creative Commons licence. He blogs at edtechie.net

I’m tempted to suggest that above all MOOCs should hang a sign that declares “abandon all quality measures”, because most of the ones we have developed for formal education don’t apply in MOOCs. We have developed a set of quality measures based on a specific relationship between the education provider and the student. That relationship is fundamentally altered in a MOOC, and so those of existing measures are not applicable.

Let us consider why we measure quality. Largely it is to verify that aims and intentions have been met. The aims of the institution may be to have a sufficient number of students, for them to stay with and pass the course, and for the institution’s reputation to be upheld. The educator in charge of the course may have similar aims, along with those of personal interest. The student will have the aim of learning what they set out to, of passing the course, enjoying the experience and gaining useful skills.

We therefore develop quality measures and procedures that monitor these intentions. These could be student completion rates, student satisfaction scores, external assessment of course content, checks against external benchmarks, etc. In a MOOC many of these intentions are altered, either radically or subtly. At the moment it’s not entirely clear what the intentions of institutions are – is it to attract more formal students, to provide a public good, to make money? In the early experimental stage it might be a confused mixture of all of these, combined with a feeling of needing to do something about those bloody MOOCs. For educators the intention might be to experiment with curriculum or pedagogy, to gain personal reputation, or personal development.

The really interesting difference though is with the intentions of the learner. While some of the original aims may remain, for instance it may help in career development, others are exaggerated or absent. The need to pass the course for instance is drastically reduced, because progress on to subsequent courses is not dependent on it, and most importantly, because there is no financial commitment to passing. The personal interest in learning is heightened I would suggest. In conventional courses there will be a wide range of different types of learner, but in MOOCs, the presence of what we often term ‘leisure learners’ is much higher. They’re nearly all leisure learners – they don’t have to do this after all, it’s something that is competing with watching the TV or walking the dog. And a whole new class of learners exist in MOOCs that you rarely see in formal education. These are what we might term drive-by learners (after Jim Groom’s drive-by assignments). These are learners who are signing up because they can. It costs nothing to sign up, they can take a look, see if they like anything and move on. They may dip in and out over the course, taking bits they find engaging, or they may not even turn up at all. The financial and emotional commitment to formal education is much higher, making drive-by learners very rare. There is probably another, very small group who are also absent from formal education, and that is the antagonistic learner. These are learners who know they won’t like the course and take it for precisely that reason, so they can highlight its numerous faults.

If we consider these new types of learners and their intentions then the existing quality measures don’t map across well. For instance, very few of these learners have course completion as a major goal. And progression on to other courses is not yet a metric in a pick-and-choose world, although we will undoubtedly see increasing pressures to make MOOC learners stick with a particular brand of MOOC provider, just as we see this with computer or phone providers. With such a broad range of learners, MOOCs find themselves up against a tough comparison with formal education. In many ways higher education filters learners before they arrive, even with an Open University, the commitment to study performs a significant filtering function. To use Weinberger’s phrase, higher education filters on the way in, whereas MOOCs filter on the way out. The quality measures are therefore very different. Student satisfaction rates when you are open to enrolment and filter on the way out are unlikely to compare favourably with a system where there has been a filtering already. Which is not to say we should follow my advice and abandon all quality measures, but that the existing measures need to be recognised for what they are – tools for an entirely different purpose. Some measures may still remain, for example I ran a MOOC as part of an existing Masters programme and it was thus subject to the same quality procedure in production as any OU course, including two rounds of critical reading by an external audience, full editing and surveying. But when we filter on the way out, and operate in the open, then it opens us up to new types of quality measures. These could be altmetrics type measures (what kind of ‘buzz’ does it create, what is the public reaction of participants), or analytics (how many people come back, what is the dwell time, bounce rate, etc). But the comparisons here need to be with other MOOCs, not with formal education.

One last plea – MOOCs are still a new kid on the block. Let them make mistakes, let them be experimental, let people play and explore in this space without tying it down with the types of quality overhead we already have in formal education.

24 thoughts on “Week 7: MOOCs & Quality by Martin Weller

  1. Pingback: Week 7: MOOCs & Quality by Martin Weller | ...

  2. Pingback: Week 7: MOOCs & Quality by Martin Weller | ...

  3. Hi Martin,

    Thanks for your thought-provoking and honest account on MOOCs and Quality; there is so much hype around MOOCs and on the way they are going to ‘make or break’ education, and it is good to put things into perspective.

    You make a well-reasoned argument explaining why formal quality measures do not apply on MOOCs. Adopting your position, I would argue that the ‘C in MOOC’ is slightly misleading as they are not exactly ‘courses’ but something else.

    Every single letter of the MOOC acronym is debatable with the exception of the ‘O for Online’; not all of them are ‘massive’, their ‘openness’ is debatable and often a move backwards compared to the OER movement, but to me the ‘C for Courses’ is probably one of the main reasons there has been so much controversy around them, coupled with the efforts by some to present them as courses that would soon compete with formally accredited courses.

    If MOOCS were MOOLAs – L for Learning and A for Activities – or something like that, perhaps it would be easier for educators to follow your plea to ‘let them (MOOCs) make mistakes, let them be experimental, let people play and explore in this space without tying it down with the types of quality overhead we already have in formal education’. But then again, had this been the case, venture capitalists would have found it much harder to raise millions for their platforms wouldn’t they?

    Thanks,
    Timos

  4. Pingback: Week 7: MOOCs & Quality by Martin Weller | ...

  5. Pingback: Week 7: MOOCs & Quality by Martin Weller | ...

  6. Pingback: #MOOCs & Quality by Martin Weller | Digital...

  7. Pingback: #MOOCs & Quality by Martin Weller | MOOCs |...

  8. Pingback: #MOOCs & Quality by Martin Weller | Educati...

  9. Pingback: #MOOCs & Quality by Martin Weller | Utbildn...

  10. Pingback: #MOOCs & Quality by Martin Weller | technol...

  11. Pingback: Week 7: MOOCs & Quality by Martin Weller | ...

  12. Pingback: #MOOCs & Quality by Martin Weller | MOOC-SC...

  13. Pingback: #MOOCs & Quality by Martin Weller | Flippin...

  14. Pingback: #MOOCs & Quality by Martin Weller | Elearni...

  15. Pingback: #MOOCs & Quality by Martin Weller | Brainfr...

  16. Pingback: Week 7: MOOCs & Quality by Martin Weller | ...

  17. Pingback: Week 7: MOOCs & Quality by Martin Weller | ...

  18. Pingback: Week 7: MOOCs & Quality by Martin Weller | Flexibility Enables Learning

  19. Pingback: #MOOCs & Quality by Martin Weller | Learnin...

  20. Pingback: MOOCs & Quality | Martin Weller - MOOC Qual...

  21. Hi Martin,
    Yes, I agreed with you on the quality aspects, and the associated challenges that have arisen when MOOCs are evaluated from a “traditional” or formal quality perspective.
    Applying formal quality assurance system onto MOOCs (cMOOCs in particular) would be like fitting round pegs into square holes, where evaluation and assessment to be challenging. Quality is viewed, perceived and evaluated by different stakeholders and customers, and so it is difficult to come up with a consensus view or agreement on what constitutes quality in MOOCs. This is similar to the critique of Connectivism based on other Learning Theories and vice versa, as I have shared in my previous post http://suifaijohnmak.wordpress.com/2011/12/26/change11-what-are-those-learning-theories/ Thanks again for your sharing of great insights.

    John

  22. I just interpret MOOCs as 20 years old online courses provided by elite schools not for credit and free . Sure they are not massive. They should not be free. They should be for degrees .

    But nobody talks about the cost of MOOCs .
    Even the elite schools provide today MOOCs as an experiment to make Money .
    Cost of an online by elite university is less than $ 1 since they can attract 10,000-100,000 enrollment in 5 years or so .
    If they charge only $ 10 per course , they make billions dollars .
    Think globally . There are 200,000,000 university age people in the World .
    Just fremember only 15 % of the enrollments of MOOCs are Americans rest is foreigners.

    Coming to quality , sure the quality must be perfect. That is measured by brand name .
    Brand name is a measure of quality 100 % sure .
    Therefore elite schools will make billions from MOOCs even at a fee of $ 10 per course. They are really smart .

    Please see , Coursera is charging to 10 state universities $ 36 per course.
    That shows I am really right .

  23. Pingback: #MOOCs & Quality by Martin Weller | Ebooks ...

  24. Pingback: #MOOCs & Quality by Martin Weller | cftc10

Join the Discussion...