Factors affecting choice of access to content in a Learn Moodle MOOC and their effect on MOOC completion rates

A project for the Edinburgh Napier MBOE on:

Factors affecting choice of access to content in a “Learn Moodle” MOOC and their effect on MOOC completion rates

Discussion

First, we will consider the questions relating to completion and content release:

  • How does offering a choice of course content release affect the final overall completion rate of the Learn Moodle Basics MOOC? 
    • How does the choice of timed content release (all at once or step-by-step) affect a participant’s chance of completion of the Learn Moodle Basics MOOC? 
      •  Which other factors, if any, affect full, partial or non-completion of the course?

      Then, we will consider the question relating to the reasons why participants choose a particular path:

      • Which factors affect a participant’s choice of course content release (All at once or step by step)? 

      Completion rates and choice of content release:

       In order to ascertain whether offering a choice of course content release affects the final overall completion rates of the Learn Moodle MOOC, it would be necessary to compare the completion rates of the MOOCs where this choice is offered with the completion rates of the earlier MOOCs where the choice was not offered. Although these earlier figures are publicly available, caution must be exercised when making a comparison because a bug in Moodle prior to the January 2018 MOOC meant that enrolment figures were artificially high (Moodle Tracker report MDL-54106) and therefore completion figures were potentially lower than in reality. What can be stated, however, is that giving participants the option to choose when they access the materials has resulted in a small rise in completion rates during the last three MOOCs (when defining completion by the number who signed up) Future rates must be observed before a trend can be observed with any certainty.See postscript below

          • Jan 2018 - 14.1%
          • June 2018 - 16.0%
          • Jan 2019 - 17.3%

      Regarding completion rates and content release, the reports show the following:

      • Choosing to see the course materials "All at once" ( referred to as AAO)  rather than "Step by step" (referred to as SBS) is slightly more popular - 1191 v 1038 (Jan 2018); 157 v 854 (June 2018) and 1331 v 637 (Jan 2019)
      • However, the actual percentage of participants in each group who went on to complete is very similar and does not seem to suggest one way is consistently more successful than another:

          • 34.1% AAO v 35.4 % SBS in Jan 2018
          • 32.8% AAO v 32.3% SBS in June 2018
          • 31.4% AAO v 32.2% SBS in Jan 2019

      That there is no significant difference is a result which agrees with an earlier study (Mullaney and Reich, 2013) where it was stated "We find few differences in [...] completion between the two runs."

      In contrast to the earlier study, MOOC participants were given the choice before they accessed the contents. In the Mullaney and Reich study, two separate MOOCs were run on two separate occasions.

      The results from this study  suggest that a participant's chosen path to access materials does not affect his or her eventual  chances of completing the course.

      By filtering (in MS Excel™) the downloaded completion reports to display the number of participants in each cohort, AAO and SBS, who completed every activity except one peer assessment activity with a deadline, it was possible to show  the number of participants who 'partially completed' and thereby received a 'Certificate of achievement'.  Here the numbers are significantly lower than the full completion:

          • 3.0% AAO v 2.4% SSB (Jan 2018)
          • 4.1% AAO v 4.15 SSB (June 2018)
          • 5.0% AAO v 2.5% SSB (Jan 2019)
      While there are some individual differences per year, the main point is that these percentages are so low contrasted with the completion percentage (and non-completion percentage) 

      The option of a 'Certificate of achievement' for participants who did not make the peer assessment deadline but completed all other  activities was provided a few MOOCs ago based on participant feedback from those who felt disappointed they did almost everything and yet 'left with nothing'. 

      However the low results obtaining the certificate do imply that in fact, participants either aim to complete the MOOC fully or are not engaged or successful enough to do all activities and so the offer of a secondary 'Certificate of achievement' is not sufficient a motivator. 

      Registration to the course is not disabled until the final week, meaning that participants can sign up and join in but it is too late to meet the workshop deadline, but never too late to obtain the Certificate of achievement and partially complete. It could be speculated that, realising they have joined too late to complete, participants decide not to continue. A certain number of participants are also encouraged to join by their managers, using the MOOC as professional development. Possibly the option of a lesser award, not a completion badge and certificate, is not deemed worthwhile. Indeed, some participants who fail to complete one MOOC return again to try another time, as with this participant in the 3.7 June 2019 MOOC:

      ".... Because we´re using our own LMS (Moodle) at work I would like to improve my skills. Last time I missed the dates for the workshop task thus I´m here for the second time. ..."

      Reasons for choosing when to access course content

      According to the participant survey, a slightly larger percentage of the Step by step group were doing the MOOC for the first time. (85% SBS v 69.21 AAO) This matches the personal  experience of the two MOOC facilitators in that returning and experienced Moodlers tend to opt for All at once in order to move quickly through the materials and focus on assisting newcomers. Since the MOOC began to be run on a regular basis in 2015, a sizeable number of returning participants have joined each time with the express aim of helping out. (Indeed, a "MOOC helper" badge is awarded manually to those who make a particularly useful contribution in the forums and other activities.) This exchange of support is very much inline with the social constructionist referents outlined by Moodle's creator Martin Dougiamas (Docs.moodle.org (b), 2019). "All of us are potential teachers as well as learners; in a true collaborative environment we are both.....we learn particularly well from the act of creating or expressing something for others to see".

       Perhaps, as suggested by Hood, Littlejohn and Milligan (2015) these regular participants who choose AAO have a high SRL score and are "more informal and less structured in their learning approach". The higher percentage of SSB participants who are doing the MOOC for the first time would also tend to support Littlejohn and Hood (2018) who describe the "cautious student" taking the "predefined, linear course structure to scaffold his learning"

      It is worth noting too, that both groups show a high percentage of participants doing the MOOC for the first time. Thus, the MOOC is successfully attracting those new Moodlers it wishes to attract.

      It is interesting to note, however, that both groups state they feel confident they will complete the MOOC (95.8% AAO v 96.78% SSB). When considering these figures however, we must remember that the participants completed the survey after the first week when they were perhaps still keen and engaged. However, early engagement has been seen by Jiang et al (2014)  to be a good indicator of subsequent success, and in earlier runs of this MOOC (Cooch, Foster, Costello, 2014) and more recent ones, those participants who get involved in the first week, for example by obtaining a Participtant badge, are more likely to continue to completion. It is a limitation of the participant survey results that they did not show who went on to complete the MOOC. Identifying these will be a recommendation for further study.

      Likewise, although the percentages of both groups who stated they are confident in English was high (95.58% AAO v 92.76% SBS) the figures do not tell us if the remaining participants  who stated they were NOT confident in English (4.42% AAO v 7.24% SBS) eventually manged to complete. This would be useful knowledge to obtain.  The percentage of AAO participants who state they are confident in English is slightly higher than the SSB group. Is this perhaps because a number of these are returning or experienced Moodlers, who return because they feel at ease in our MOOC environment? Certainly the free text answers support this: 15% of the All at once group said they were 'experienced' while only 09.% of the Step by step group did. Only 0.7% of the All at once group considered themselves 'new Moodlers' whereas 12.10% from the Step by Step group did.

      Another factor affecting participants' choice of content release is, unsurprisingly, the desire to view the materials in a 'manageable' way. A higher percentage of the Step by step group said they had chosen the path to make their learning manageable - the word 'chunking' was even mentioned (18.20% AAO v 26.39% SBS) and in fact nobody from the All at once group used language implying feeling overwhelmed but 7.99% from the Step by step group did. The All at once group were keener to have 'flexiblity' of viewing course materials when they wanted to (16.80% AAO v 2.90% SSB). These result were not surprising to the facilitators when they were considered in the paired discussion, which follows in the next section.

      Finally, when  discussing the findings of this study, it is important to consider its limitations:

      Limitations

      The data for this study came from two types of reports readily available to course facilitators in Moodle: Completion reports and Feedback responses. While these are useful, the information they can offer is limited. 

      Sometimes, actions are too complex to be evaluated purely by the database server and presented to course facilitators in ready-made reports. In those cases, another "layer" of processing must be added. For Moodle,  typically, data can be pulled from the database via SQL and further processed in PHP before being presented to the user. Thus, other reports are available to those with server access or via contributed reporting plugins, but were not available for this particular project. Thus some questions remain unanswered. For example, according to the Participant survey at the end of Week 1, over 90% of members in both groups stated they were confident they would complete the MOOC. An SQL query could have revealed how many of those confident participants did actually go on to complete the MOOC. Likewise, a small percentage in both groups said their level of English was low. It was not possible with the standard reports available to discern how many, if any, went on to complete the MOOC despite their level of English. An SQL query such as the example in the section on Methodology could have shown how many of those who  accessed the course settings in  a language other than English actually completed it

      Guo and Reinecke (2011) discuss the different ways students from different countries navigate through MOOCs, stating that those who are ultimately more successful tend to navigate in a non-linear way.  According to their reseach of four edX MOOCs,  students from the US and Europe tend to navigate in a non-linear way.  Choosing the AAO path offers more opportunities to navigate in a non-linear way as the materials from all weeks can be accessed at any time.  An SQL query could quickly identify the countries of origin and chosen paths of MOOC participants to ascertain whether these participants are, in fact, from the same demographic areas as suggested by Guo and Reineke. 

      Postscript

      Since this study, a further Learn Moodle MOOC has been run and the results obtained. Thus the most up to date completion figures are:

      • Jan 2018 - 14.1%
      • June 2018 - 16.0%
      • Jan 2019 - 17.3%
      • June 2019 - 16.8%