Factors affecting choice of access to content in a Learn Moodle MOOC and their effect on MOOC completion rates

Abstract

This study considers the reasons for, and subsequent consequences of MOOC participants choosing to see content "All at once" or released weekly "Step by step". It aims to identify the types of MOOC participant most likely to choose selective release over seeing the materials all at once, and studies whether one path or another is more likely to lead to successful completion of the MOOC.

The MOOC in question is the twice-yearly Learn Moodle Basics MOOC which regularly attracts over 3000 participants and which in recent years has offered participants when signing up the choice of how to access the course content.

It is envisaged that the report will be useful to educators and researchers interested in MOOC motivation and completion rates in general and to instructional designers considering how best to present their materials.  As the MOOC focuses on how to teach with Moodle, the report should also be of interest to researchers exploring how the Moodle platform can be suited to MOOC delivery.

The study involved taking the completion data from three recent Learn Moodle Basics MOOCs and analysing the results to identify which cohort ("All at once" or "Step by step") had higher results in terms of completion, partial completion and non-completion - if indeed there was a  significant difference. An earlier study along similar lines (Mullaney, T and Reich, J 2013) suggested there is no significant difference. We test the findings against the earlier study and, additionally,  explore the reasons behind participants' choice of path through the course. The most recent MOOC also included a participant survey asking why one path was preferred over the other. The results of this survey also form part of the study.

The findings reveal that there is no appreciable difference between the completion and non-completion rates of participants in both groups. An alternative certificate,  a "Certificate of Achievement," is awarded for partipants who complete all but one specific activity, and while again, no significant difference was noted in how many from both cohorts, it was observed that far fewer participants in both cohorts obtained this. It could be implied that, realising they would not obtain a Certificate of completion, participants were far less motivated to continue with the remaining activities.

The results from the participant survey again unsurprisingly reveal that experienced Moodlers and those having previously done the MOOC are more likely to select the  All at Once path whereas those who wish to 'chunk' their learning into 'manageable' sections are more likely to select the Step by Step path. However, there is no appreciable difference in confidence or competence in English.

The study recommends  continuing to offer both options of content release in further MOOCs.  It also recommends working together with the authors of a recent study of Learn Moodle MOOCs (Monllau Olivé D et al, 2019) to explore in greater depth indicators of success and failure in both groups by enabling the Learning Analytics feature on the Learn Moodle site.

About this project

Here is the transcript in PDF format

This project considers three recent 'Learn Moodle Basics' MOOCs run by Moodle Pty Ltd, usually known as 'Moodle HQ',  and aims to answer the following questions:

• How does offering a choice of course content release affect the final overall completion rate of the Learn Moodle Basics MOOC? 
• Which factors affect a participant’s choice of course content release (All at once or step by step)? 
•  How does the choice of timed content release (all at once or step-by-step) affect a participant’s chance of completion of the Learn Moodle Basics MOOC? 
•  Which other factors, if any, affect full, partial or non-completion of the course?

The study involves secondary data in the form of completion data from three MOOCs, involving 7437, 4957 and 3592 participants and a participant survey from the third MOOC where participants are asked about their choice of content release. The survey was completed by 1127 participants. All participants who completed the MOOC completed the survey. However, not everyone who completed the survey completed the MOOC.  623 went on to complete the  MOOC. A limitation of the survey therefore is that it is not possible to be sure that those who, for example, claimed they were very confident they would complete actually went on to complete.

Following download and analysis of the three reports and one participant survey, a paired discussion was set up and recorded with the two co-facilitators - myself, Mary Cooch - and Moodle Community Manager Helen Foster, during which the results were considered and some observations and recommendations made.

Literature review

Introduction

Since the launch of the first Massive Open Online Course over ten years ago – arguably by Stephen Downes and George Siemens in 2008, MOOCs have become a popular method of delivering online learning to large numbers of people. However, their high attrition rates and low completion rates have brought criticism that MOOCs are not particularly effective in so-called mass teaching. Studies have been undertaken to identify what causes these high dropout rates and low completion rates in order to adapt courses and increase participants’ chances of success.

 In my role as Moodle Community Educator, I have facilitated a MOOC for teaching non-technical educators how to use the Moodle software. First run in 2013, the four-week MOOC has been running (regularly) since 2015 and changes slightly each time, based on participant feedback. The most recent change has been to offer participants the choice of seeing all teaching materials at once, ‘All at once’, allowing them to work (mainly) at their own pace, or of having them revealed one week at a time, ‘Step by step’, avoiding information overload.

Some MOOCS display all teaching content from the start, while many display it based on a weekly schedule or upon a participant’s completion of an earlier activity. The Learn Moodle MOOC is different in that participants have a choice. Recent research on this is scant: apart from one study six years ago (Mullaney, T and Reich, J 2013), previous studies of MOOCs have not really dealt with the motivations and results of offering such a choice of course content release in MOOCs. The literature does however provide some insights into how their personal situations affect participants’ navigation through a MOOC, what motivates them ultimately to complete a MOOC and how we can predict performance.

The purpose of this literature review is to identify studies which investigate the reasons behind MOOC participants’ navigation through course modules and possible predictors of eventual success. It is hoped that the review will inform and  provide a background to my own project, exploring why participants select a particular learning path and how it affects their chance of completion.

• Factors affecting navigation through a MOOC
• Predictors of retention and completion.

Factors affecting navigation

Existing research papers have identified the following as factors affecting navigation through MOOCs. As such, each will be considered in turn:

• Confidence:

• Age and location: 

• Scheduling:

We explore the consquences of self-paced versus fixed schedule MOOCs and content release

• Layout:

We consider adaptive learning and its potential effect on course completion

• Language:

We look at possible language constraints/challenges for participants when engaging in a MOOC in a language other than their own.

Confidence

Because of their open (and massive) nature, MOOCs attract a wide variety of participants, and these participants may have very different educational and professional backgrounds. For all of them, however, the onus is on the individual to do the work.

 Littlejohn, Hood, Milligan and Mustain (2016) discuss self-regulated learning (SRL),  first explained by Zimmerman (2000). It is not static, but dependent on a number of elements. Referring to Pintrich (2000) they state that:

  “The ability to self-regulate one's learning is shaped by both personal-psychological and contextual factors”

In an earlier paper, Hood, Littlejohn and Milligan, (2015) suggest that  “a learner’s current role and context influences their ability to self-regulate their learning in a MOOC”. A high SRL score means a participant is autonomous and aware of their own strengths and weaknesses. They noted that it tended to be the participants with a low SRL score who were actually more focused on getting through the course and obtaining a certificate – in other words - they were extrinsically motivated – whereas participants with a high SRL score demonstrated intrinsic motivation and “tended to be more informal and less structured in their learning approach, engaging with the content and activities that they determined would support their individual needs.”

 This would suggest those in the Learn Moodle MOOC who choose the ‘All at once’ path might have high SRL scores and be less concerned with ultimate success and more with improving their understanding. It might suggest that those who choose the ‘Step by step’ path might have low SRL scores and want this scaffolding support to get them through the hurdles towards a completion certificate. 

In their book on reconceptualising learning,  Littlejohn and Hood (2018) provide a number of learner stories, one of which is ‘the cautious student’, who “struggled to determine his own learning journey and consequently used the predefined, linear course structure to scaffold his learning”, again suggesting those lacking in confidence or experience tend to take what they consider the safest route with most support.

Age and Location

According to Guo and Reinecke (2011), there are both age and demographic differences between how students navigate through MOOCS:

 “Older students and those from countries with lower student-teacher ratios (e.g., the US and European countries) visit and repeat more lecture sequences, which indicates more non-linear navigation and learning strategies. Younger students and those from countries with higher student-teacher ratios (e.g., India, Kenya) visit and repeat fewer sequences, which indicates more linear navigation.”

 Those participants desirous of and subsequently successful in obtaining a completion certificate tend to use non-linear navigation, jumping between activities and skipping course content:

 “Certificate earners view only 78% of learning sequences, on average; they completely skip 22% of course content.”

 This study however, while presenting findings on participants’ navigational preferences, starts from the assumption that a MOOC is organised in a linear way (such as weekly videos and assessments) and that participants can move freely backwards and forwards. It does not make references to possible choice of content display, such as selective release or full visibility from the start, and as such, it is uncertain whether this would affect participants’ navigation. Its relevance to the Learn Moodle MOOC research proposal is therefore somewhat restricted.

 However, the authors themselves are aware of the limitations of their own research and suggest further studies may be welcome:

 “Finally, while we were only able to speculate about the reasons for students’ navigation strategies, it would be especially interesting to conduct qualitative follow-ups with various demographic groups.”

 A challenge with the above survey, and with other research in this literature review, is both the diversity of MOOC types and subject matter and the varying ranges of students studied.  Guo and Reinecke (2011) based their research on four edX MOOCs with 140,456 participants. Other studies, which will be considered shortly, draw conclusions from other figures.

Scheduling

Watson, Yu and Watson (2018) explored and compared participants’ learning gains (‘attitudinal learning’) in a self-paced version of a  MOOC (which could be taken at any time and had minimal facilitator support)  versus a fixed schedule version (a ‘live’ MOOC running on specific dates with greater interaction). Their results were that those in the self-paced version “were more likely to perceive higher levels of attitudinal learning gains and satisfaction than those in a fixed-scheduled MOOC”.  Additionally they remarked that “…those learners enrolled in the self-paced offering of the course reported much more diverse reasons for enrolling in the course.”

 While this study does not directly relate to the Learn Moodle MOOC, which only offers a fixed schedule of twice a year, the results are interesting in that they suggest a self-paced approach suits learner of a greater diversity who ultimately gain more satisfaction. Might these be the same learners who opt for the ‘All at once’ approach in the Learn Moodle MOOC?

There is a caveat when interpreting the results of the study, however: the authors point out that the two MOOCs did not have entirely the same instructional design, and that no demographic study was done of the participants, which might have revealed other possible learner differences.

In their 2013 study, "Staggered versus All at once", Mullaney and Reich of Harvard university reported on the results of their experiment testing a MOOC in two 'runs', one where all content was released immediately and the other where content was released a week at a time over the four month period. Their study focused on comparing 'on trackedness' (ie how learners engaged with the recommended pathway), rather than the reasons learners preferred one option over another. Additionally, their MOOC lasted four months rather than the four weeks of the Learn Moodle MOOC. However, their findings might prove interesting to compare and contrast with the results of this project.

Mullaney and Reich "discovered few differences in persistence, participation, and completion between the two runs." They also discovered that providing the content 'all at once' led (to a small degree) to students being slightly less on track then when the content was released in stages, and tutors had more control over the cohort. However they also accepted that learners appreciate having the flexibility to work at their own pace, and return to materials of earlier weeks even though they might not be following the syllabus in the strictest manner. They concluded that:

1. Releasing content all at once gives learners valued flexibility
2. 'Staggering content'  keeps learners more tightly controlled (and therefore might be preferred by some tutors)

The authors offer these conclusions as advice to future MOOC designers; however they do not suggest offering both tracks, only the benefits of one or the other. The Learn MOOC MOOC differs  in this respect , in that participants can choose how to access the course content. It remains to be seen from the data analysis whether this choice affects completion in a more significant way than observed by Mullaney and Reich.

Layout and Language

As well as self-paced versus fixed schedules, MOOCs can have differing layouts, although many tend to follow a pattern of short videos followed by quizzes, with occasional discussions and assignments.

 The papers in this review do not provide much detail on how effective the layout is to the success of a MOOC, although in their study of 205 learners and professors, Yousef et al, (2014) did survey MOOC participants about the layout. However, their question “User Interface: Does the MOOC layout have impact on the learning experience?” only in fact refers to video layout and and search facilities, not weekly layout or selective release. Again, this highlights the paucity of research on choice of course content display and the effect on retention and completion in MOOCs.

Adaptivity

Sonwalker (2013) reports on a MOOC on molecular dymamics which used Adaptive learning to present content to participants dependent on their performance in a pre-MOOC quiz and weekly assessments and their preferences in terms of how they liked to learn. According to the author, adaptive learning will: 

"..allow students to learn better by customizing their own learning strategies in a MOOC environment."

The Learn Moodle Basics MOOC is 'adaptive' only in that it displays content dependent on participants' responses to a Moodle Group Choice activity at the start of the MOOC. However, adaptive learning features (activity restriction and display based on previous activity completion) is a widely used feature of Moodle courses, albeit not a widely researched one.  The paper by Sonwalker (2013) is less useful in that it  focuses on the scalable Cloud architecture behind such an 'adaptive MOOC' rather than the effect differentiating content display might have on completion. 

Léris et al (2017) propose six indicators for adaptivity in a MOOC, with the aim of improving motivation and increasing completion rates. Four hundred and seventy five MOOC participants  were surveyed about which indicators they found the most important. (The authors warn  that the MOOC participants were mainly middle-aged Spanish speakers, which could potentially restrict the scope of the findings.) The first indicator refers to materials being made available according to participant choice (or previous performance) and this directly relates to the Learn Moodle MOOC choice. However, participants responded that for them, most important were the second indicator (adapting the MOOC to the pace of personal work) and the third indicator (offering various levels of difficulty and targets). 

Language:

With the exception of  Léris et al (2017), the MOOCs in the above studies were run in English. It is not mentioned what proportion of the participants did not have English as their first language and whether this made a difference to their motivations, navigation and completion. 

Moodle open source software offers participants the opportunity to display course instructions in their own language. The option was taken up in the first  ever Learn Moodle MOOC, according to Cooch, Foster, Costello (2014):

 “Although the MOOC was delivered in English, participants were encouraged to post in forums and enter data in other activities in their own language. Participants could select their own language in their profile; 53 different languages were selected.”

 In her position paper on four MOOCs, Bali (2014), commenting on the courses she explored, noted that non-native speakers preferred “to use transcripts or lecture slides rather than listen to lectures in English” and then when “reading the discussion forums, one could see that some had used some sort of translation software in order to contribute” Whether this extra hurdle affected their performance is not stated.

Predictors of retention and completion

Existing research papers identifty the following predictors of retention and completion. Each will be considered in turn, following a caveat on the definition of 'Completion'.

• Participation in automatically graded activities

This relates to (for example) short quizzes at the end of each unit of study.

• Positive ratings by co-participants

This relates to rated activities such as forum posts.

• Early engagement

This relates to the eventual success of participants who are active in the first week of a MOOC.

• Participant and facilitator interaction

This relates to facilitator presence and the ongoing relationship between participants and facilitators.

Completion: Definition and caveat

Many educators and researchers accept the definition by Jordan (2015a) that ‘completion’ means the number who passed a course and/or obtained a certificate. The number is usually taken from the number who initially enrolled, although this is arguably misleading since it is often the case that participants sign up and never return to engage with the content. Thus, figures can be skewed, and completion may be higher or lower than the mean of 12.5% (Jordan, 2015b) depending on the point where the calculation is made.

 Additionally, when considering in the reasons for low completion rates and how to improve them, it is worth remembering from pedagogical point of view, that Littlejohn and Hood (2018)  offer a cautionary note, pointing out that:

 “ …completion, or at least engaging with all of the content and participating in learning activities, is not necessarily indicative of learning or of the learner’s ability to participate in a MOOC.”

Participation in automatically graded activities

 A recent study (Monllau Olivé  D et al, 2019) exploring the creation of analytics frameworks for predicting student drop-out used anonymised datasets from eight Learn Moodle Basics MOOCs. The authors, including Moodle HQ's Lead Data Scientist, identified performance in the automatically graded quizzes as a predictor of future success. The MOOC has a quiz each week on the course materials with the  final quiz in the fourth week being a summative assessment with a pass mark of 70%. The authors discovered that:

"The indicators The student attempted at least one quiz and Percentage of quizzes attempted in the course {...] have a strong correlation to course drop outs."

The authors speculated that those participants interested in testing their knowledge each week are perhaps those most motivated to complete the course.

Another indicator "The student failed a quiz attempt" was less significant. From my experience running the Learn Moodle MOOC, I would suggest this is because the participants are allowed to retake quizzes as often as they wish, so  they are aware they can constantly improve.

The authors did not differentiate between 'All at once' participants and 'Step by step' participants in their research, so this is a question still to be answered.

Postive ratings by co-participants

Monllau Olivé D et al 2019 also identified "Positive ratings in database entries" and "Positive ratings in forums" as further indicators of the likelihood of drop out or completion. 

Adding a recipe to a Moodle database activity is one of the required tasks for completion. Rating the entries of others ("Delicious" " Not my thing") is not required, but as an optional task, some participants engage with it.

The MOOC includes several forums with ratings such as "Helpful". Again, adding ratings is not obligatory but the feature is well used.

Monllau Olivé and colleagues suggest two reasons for the link etween these positive ratings and eventual drop-out or completion. One possiblity is that participants, like all of us, feel more motivated when we are praised. Another possibility is that those particpants who receive positive ratings are those who are working harder in the course anyway, and therefore more likely to continue.

However, as with the previous indicator (Participation in automatically graded activities) the authors did not differentiate between 'All at once' and 'Step by step' group members so it is not possible to gauge how much this affects those who choose the one path or the other.

Early engagement

Completion (however you calculate it) continues to be the benchmark of both participant and MOOC success, and Littlejohn and Hood (2018)  refer to findings by Jiang et al (2014) that a learner’s behaviour during the first week of the MOOC is indicative as to whether they will complete the MOOC or not. Analysing a four week Biology course, they noted that a “significant decrease in participation usually takes place by the second week of the course” They also point to peer assessment and assignment performance in week 1 being a strong predictor of final success.

Mullaney and Reich (2013) observed, when exploring 'ontrackedness' that 

"for each week, a every week of the course, a substantial portion of visitors engage with the very first contents of the course"

In terms of early engagement in a MOOC, it does suggest that the start of each week is key to engaging and retaining learners. The authors suggest it might be pertinent to have the first element of a course provide a résumé of the main ideas of the whole course rather than simply providing the week one materials.

 A positive start in the first week has also been observed as a sign the participant will go on to complete the Learn Moodle MOOC. During the 2013 MOOC a ‘Participant badge’ was offered from the launch, as a incentive to those who engage in forums.

 “A third (34%) of the 9,522 users completed the Track A of the course earning a participant badge…. high compared with MOOC persistence rates (Jordan, 2014). We speculate that the high participatory nature of the course activities may have helped contribute to this by keeping the learners engaged” (Cooch, Foster, Costello, 2014)

 The practice has continued in subsequent MOOCs. The percentage of those obtaining the participant badge, typically in the first week and then going on to complete the MOOC is consistently higher than the widely agreed 12.5% median, as outlined in publicly visible post-MOOC reflection posts. (Foster, 2017)

Participant and facilitator interaction

Participant interaction cannot be ignored as a factor affecting the likelihood of completion. While this is only speculated by Cooch, Foster and Costello, (2014), statistic results are provided by Yousef et al (2014)  from a study of 205 learners and professors  where “Supporting the collaborative learning among Learners” scores highly amongst the criteria for “to empower learners in MOOCs.” Equally, De Barba, Kennedy and Ainley (2016) also conclude “the strongest predictor of performance was participation, followed by motivation.“ 

 Adamopoulos (2013) puts forward other predictors of course completion, the most significant one, in his opinion being Facilitator, or online professor presence:

“The findings of our analysis illustrate that Professor(s) is the most important factor in online course retention and has the largest positive effect on the probability of a student to successfully complete a course” 

However, with MOOCs being offered by universities as well as other, non-HE providers, it is uncertain whether these ‘professors’ are already known and appreciated by the MOOC participants or whether indeed, any supportive facilitator is vital to the final outcomes.

 Students’ gender or education did not, according to Adamopoulos, affect the chances of attrition or subsequent completion. This correlates with similar findings by Breslow and Colleagues (2013), quoted by Greene, Oswald and Pomerantz (2015) who, when studying student performance in an edX MOOC, concluded “there were no relations between age,gender, or prior degree attainment and achievement” but in fact disagrees with the authors’ own research, which suggests “Increases in age and level of schooling were associated with a decreased likelihood of dropout (i.e., greater likelihood of retention.)”

 The results of the above authors must be considered carefully because on their own admission they focused on only one MOOC, run a single time, with a sample of 3,875 participants who completed the MOOC and completed the authors’ survey.

Growth mindset

Another example of the need to show caution when digesting the research findings of Greene, Oswald and Pomerantz (2015) is in their discussion around ITT (implicit theories of intelligence).

 Referencing Dweck (2012),  the authors presumed  participants’ perceptions of intelligence might affect their continued engagement in a MOOC: if the participants believed their intelligence was not fixed, and could be improved, they should be more likely to persevere than participants who believed their intelligence is fixed at birth. However, in their final results they were “surprised to find that students’ ITT did not predict either dropout or achievement.” Again, this might be due to what they term their  “homogenous sample”.

Conclusion

When exploring the papers in this literature review, it has become apparent that the large variety of MOOCs, their difference in participant number, subjects, length and formats leads to a wariness about generalisations.

 While some common factors have been identified – it is likely, for example that the less confident will take a more structured, scaffolded path – other elements, such as language skills and the effect of selective release of materials have not been widely considered. One study (Mullaney and Reich, 2013) suggests that selective release or 'all at once' release does not make an appreciable difference to completion rates, and the results of this study will be borne in mind when analysing the MOOC data.

Useful in terms of numbers of participants surveyed is the recent study by Olivé et al (2019) where data from eight Learn Moodle Basics MOOCs was used, covering a total of 46, 995 students. Olivé et al propose that performance in automatically assessed tasks along with positive ratings by others are factors affecting eventual completion or drop-out. This study is particularly relevant not only because of the number of students involved but also because it directly relates to the MOOC in this project, the Learn Moodle Basics MOOC.

 Several of the researchers comment in their own conclusions that more research is needed, and for this reason it is felt that a project investigating the reasons for and results of choosing a particular learning path (course content display) through a MOOC (albeit a single MOOC of several thousand participants) may be a useful addition to MOOC studies.

Methodology

General overview of data and reporting available

Data and reporting available depends on the role and permissions in a course and in the Moodle site. It also depends on whether the site relies only on standard reports or has access to any non-standard, contributed report plugins or  reports available via direct database access.

Administrator reports:

From within the admin interface of a Moodle site, an administrator can access site and course logs for all users, showing when they logged in and out and what actions they took. Course reports are the same as those available to a teacher. (See the section below: 'Teacher (Facilitator) reports').

Additionally, if enabled, an administrator can access and manage Analytics Models, a feature of Moodle which generates predictions based on indicators, as outlined in the Moodle documentation on Analytics (Docs.moodle.org, 2019) Of these the most useful to MOOC facilitators would be the model "Students at risk of dropping out". Another model "Upcoming activities due" could also provide notifications to prompt learners to stay on track.

If the Moodle site host permits the addition of contributed plugins, there is a wide range of reporting and analytics tools, free and at a cost, which can assist in identifying trends in the course. Moodle itself is partnered with  Intelliboard (Intelliboard.net, 2019) which provides customisable dashboards for admins, teachers and learners with reports not available as standard in Moodle, such as a learner's progress in relation to that of others and to the course average. There is a popular, free plugin, Configurable reports which allows admins and regular course teachers to create custom reports without SQL knowledge.

If the Moodle administrator or the Moodle site host has access to the server, it is possible to devise SQL queries and create reports which can contain further information not available in standard course reports. This offers far more possibilities for reporting since, rather than being constricted by the available reports (such as Completion reports as below) the teacher or admin comes up with the desired output of the report and then the SQL is composed to produce those figures. Thus, for the Learn Moodle Basics MOOC, it would be possible via SQL to find out the selected language of all course participants by using an SQL query similar to the one below. This example queries the language selected by  participants throughout the whole Learn Moodle site:

The raw output of such a database query can then look like this example (which only gives some of the data)

Teacher (Facilitator) reports

Anyone with the editing teacher role in a Moodle course can access logs and reports from the course administration area. Reports available depend on course settings and site settings. For example, the course completion report is only available if course completion has been set in a course and the Statistics and Competency breakdown reports are only available if Statistics and Competencies are enabled on the site.

These reports provide useful information about participant engagement. For example, it is possible for a regular course facilitator to see how often (if at all) learners have contributed to a particular forum discussion, by selecting the relevant fields in the course participation report:

Methodology for this project:

This project uses a Mixed Methods methodology, quantitive and qualitative.

 A quantitive approach was taken because of the ease with which it is possible to obtain reports from Moodle courses and because I, a facilitator on the MOOCs in question, had access to these reports. I was not able to run SQL queries, and Learning analytics were not enabled on the three MOOCs in this study.

A qualitative approach was also taken in order to learn about the motivations behind participants' choice of access to course materials. As facilitator with editing rights to the most recent MOOC I was able to add specific questions to a participant survey, requiring free text answers. The quantitative and qualitative data were then reflected upon by the co-facilitators in a paired discussion.

The data for this study comes from three sources:

• the completion reports from each of the three MOOCs, Jan 2019, Jan and June 2018. The numbers involved are as follows: 

• a participant survey from the Jan 2019 MOOC including both mutiple choice answers and free text answers. 1127 participants completed the survey.
• a paired discussion with the co-facilitator in which she gives her impressions of the collated data.

Advantages and limitations of  the data:

Moodle provides easy to download reports for teachers such as the completion report and participant survey. Such reports are available as long as the courses are still accessible. The completion reports for previous MOOCS have been used to provide a short forum post summarising the results after each run. However, the reports are limited in how refined the data can be and ideally, SQL reports run by an administrator with server access would give even more detailed information. Google Analytics (which have been used in earlier MOOCS) could give insights into the engagement and 'on trackness' of the participants of both pathways. (Olivé et al, 2019) designed and tested a learning analytics framework to identify indicators of success and failure of the Learn Moodle MOOC but this feature was not available at the time of this project. For the sake of organisational simplicity it was decided therefore to only use the standard Moodle reports.

Completion reports

Each of the MOOCs has a report option which shows, via ticks, which activities all participants have completed and which not, with a final column showing who has completed the course by obtaining completion ticks in all required activities.

These reports were downloaded for the Step by step group and for the All at once group as Excel-compatible format (CSV) and added into 6 separate worksheets of an Excel spreadsheet which was kept in a password protected folder.

By adding a "completed" filter to each worksheet, it was possible to filter each column and each worksheet in order to specify how many participants in each group for each MOOC had

• completed the MOOC by completing all activities
• partially completed the MOOC by completing everything except the Workshop, thereby obtaining a Certificate of achievement
• not completing the MOOC and therefore receiving no certificate.

Participant survey

A Moodle Feedback activity was used for the survey and was set to anonymous mode. The MOOC contains a regular 'Feedback after the first week' survey for participants as a way of helping the facilitators gauge the mood after the first week, so questions specific to the research were added to the January 2019 MOOC. These were all mandatory.

1. Is this your first time doing the MOOC? (yes/no)
2. Which path did you choose? (Step by step/All at once)
3. Why did you choose this path? (free text)
4. How confident are you that you will complete the MOOC? (Very confident/Quite confident/Not at all confident)
5. How well do you understand English? (Very well/quite well/not very well)

As with the completion reports, Moodle offered the option to download the results in Excel-compatible format (csv) and once downloaded, filters were applied to divide the participants into the two groups and analyse the data.  

Limitations of the Participant survey:

While everyone who completed or partially completed the MOOC also completed the Participant survey, not everyone who completed the survey then went on to complete or partially complete the MOOC. Therefore, caution should be used when assigning the findings to the group as a whole, since some responses might have been given by those who did not progress further in the MOOC.

Analysing the free text responses

The free text responses to the question "Why did you choose this path?" were initially run through an online text analyser, Online Utility.org. However this was not particularly satisfactory  since although the tool could count up frequencies of words and phrases, it could not combine those which had similar meanings for example "I thought it would be easier" and "I chose the simple option"

A wordcloud generated by wordclouds.com using the free text responses of the Step by step group.

Because of this drawback, it was instead decided to read through every text comment and place it in a particular category. The categories were chosen based on the types of responses given, and were added to during the analysis. Filters again allowed for filtering by group and by category. The categories were:

• Easier
• Experienced
• Newbie
• Flexibility
• Full/Detailed
• Time issues
• More time
• Manageable/chunking
• Other
• Overwhelmed
• Own pace
• Want to!
• Non-English*

*Non-native English speakers

Participants were not asked specifically if they were native English speakers, as it was considered that the options in question 5 were adequate. However, for the free text responses in question 3, participants were informed they could answer in their own language. These responses were then filtered by "Non-English", counted up and then their content read and added to the other categories as with English language responses. The short text answers were very straightforward and where I did not myself understand the languages, I used Google translate, justifiable in these circumstances.

Paired discussion with the co-facilitator

When considering joint interviews, Kelly, N., Nesbit, S. and Oliver, C., 2012 quote Seymour et al 1995  that these can be "an effective means to uncover the different kinds of knowledge held by each participant and produce a more comprehensive picture".  Because the co-facilitator also has access to the MOOC reports it was decided that, rather than conducting a formal interview with pre-prepared questions, the meeting would be a paired discussion in which she would give her own impressions of the completion results and participant survey and together we would reflect upon their significance and potential usefulness for future MOOC improvements.

The discussion took place on line using Zoom software and was recorded and transcripted.

The results

Here is the transcript  in PDF format

Here is the slideshow of the results in PDF format

Click to play the video below, and click the square icon bottom right to view full screen:

Discussion

First, we will consider the questions relating to completion and content release:

• How does offering a choice of course content release affect the final overall completion rate of the Learn Moodle Basics MOOC? 
• How does the choice of timed content release (all at once or step-by-step) affect a participant’s chance of completion of the Learn Moodle Basics MOOC? 
  
•  Which other factors, if any, affect full, partial or non-completion of the course?

Then, we will consider the question relating to the reasons why participants choose a particular path:

• Which factors affect a participant’s choice of course content release (All at once or step by step)? 

Completion rates and choice of content release:

 In order to ascertain whether offering a choice of course content release affects the final overall completion rates of the Learn Moodle MOOC, it would be necessary to compare the completion rates of the MOOCs where this choice is offered with the completion rates of the earlier MOOCs where the choice was not offered. Although these earlier figures are publicly available, caution must be exercised when making a comparison because a bug in Moodle prior to the January 2018 MOOC meant that enrolment figures were artificially high (Moodle Tracker report MDL-54106) and therefore completion figures were potentially lower than in reality. What can be stated, however, is that giving participants the option to choose when they access the materials has resulted in a small rise in completion rates during the last three MOOCs (when defining completion by the number who signed up) Future rates must be observed before a trend can be observed with any certainty.^{See postscript below}

• Jan 2018 - 14.1%
• June 2018 - 16.0%
• Jan 2019 - 17.3%

• Choosing to see the course materials "All at once" ( referred to as AAO)  rather than "Step by step" (referred to as SBS) is slightly more popular - 1191 v 1038 (Jan 2018); 157 v 854 (June 2018) and 1331 v 637 (Jan 2019)
• However, the actual percentage of participants in each group who went on to complete is very similar and does not seem to suggest one way is consistently more successful than another:
  
• 
  
• 34.1% AAO v 35.4 % SBS in Jan 2018
• 32.8% AAO v 32.3% SBS in June 2018
• 31.4% AAO v 32.2% SBS in Jan 2019

That there is no significant difference is a result which agrees with an earlier study (Mullaney and Reich, 2013) where it was stated "We find few differences in [...] completion between the two runs."

In contrast to the earlier study, MOOC participants were given the choice before they accessed the contents. In the Mullaney and Reich study, two separate MOOCs were run on two separate occasions.

The results from this study  suggest that a participant's chosen path to access materials does not affect his or her eventual  chances of completing the course.

By filtering (in MS Excel™) the downloaded completion reports to display the number of participants in each cohort, AAO and SBS, who completed every activity except one peer assessment activity with a deadline, it was possible to show  the number of participants who 'partially completed' and thereby received a 'Certificate of achievement'.  Here the numbers are significantly lower than the full completion:

• 3.0% AAO v 2.4% SSB (Jan 2018)
• 4.1% AAO v 4.15 SSB (June 2018)
• 5.0% AAO v 2.5% SSB (Jan 2019)

The option of a 'Certificate of achievement' for participants who did not make the peer assessment deadline but completed all other  activities was provided a few MOOCs ago based on participant feedback from those who felt disappointed they did almost everything and yet 'left with nothing'. 

However the low results obtaining the certificate do imply that in fact, participants either aim to complete the MOOC fully or are not engaged or successful enough to do all activities and so the offer of a secondary 'Certificate of achievement' is not sufficient a motivator. 

Registration to the course is not disabled until the final week, meaning that participants can sign up and join in but it is too late to meet the workshop deadline, but never too late to obtain the Certificate of achievement and partially complete. It could be speculated that, realising they have joined too late to complete, participants decide not to continue. A certain number of participants are also encouraged to join by their managers, using the MOOC as professional development. Possibly the option of a lesser award, not a completion badge and certificate, is not deemed worthwhile. Indeed, some participants who fail to complete one MOOC return again to try another time, as with this participant in the 3.7 June 2019 MOOC:

".... Because we´re using our own LMS (Moodle) at work I would like to improve my skills. Last time I missed the dates for the workshop task thus I´m here for the second time. ..."

Reasons for choosing when to access course content

According to the participant survey, a slightly larger percentage of the Step by step group were doing the MOOC for the first time. (85% SBS v 69.21 AAO) This matches the personal  experience of the two MOOC facilitators in that returning and experienced Moodlers tend to opt for All at once in order to move quickly through the materials and focus on assisting newcomers. Since the MOOC began to be run on a regular basis in 2015, a sizeable number of returning participants have joined each time with the express aim of helping out. (Indeed, a "MOOC helper" badge is awarded manually to those who make a particularly useful contribution in the forums and other activities.) This exchange of support is very much inline with the social constructionist referents outlined by Moodle's creator Martin Dougiamas (Docs.moodle.org (b), 2019). "All of us are potential teachers as well as learners; in a true collaborative environment we are both.....we learn particularly well from the act of creating or expressing something for others to see".

 Perhaps, as suggested by Hood, Littlejohn and Milligan (2015) these regular participants who choose AAO have a high SRL score and are "more informal and less structured in their learning approach". The higher percentage of SSB participants who are doing the MOOC for the first time would also tend to support Littlejohn and Hood (2018) who describe the "cautious student" taking the "predefined, linear course structure to scaffold his learning"

It is worth noting too, that both groups show a high percentage of participants doing the MOOC for the first time. Thus, the MOOC is successfully attracting those new Moodlers it wishes to attract.

It is interesting to note, however, that both groups state they feel confident they will complete the MOOC (95.8% AAO v 96.78% SSB). When considering these figures however, we must remember that the participants completed the survey after the first week when they were perhaps still keen and engaged. However, early engagement has been seen by Jiang et al (2014)  to be a good indicator of subsequent success, and in earlier runs of this MOOC (Cooch, Foster, Costello, 2014) and more recent ones, those participants who get involved in the first week, for example by obtaining a Participtant badge, are more likely to continue to completion. It is a limitation of the participant survey results that they did not show who went on to complete the MOOC. Identifying these will be a recommendation for further study.

Likewise, although the percentages of both groups who stated they are confident in English was high (95.58% AAO v 92.76% SBS) the figures do not tell us if the remaining participants  who stated they were NOT confident in English (4.42% AAO v 7.24% SBS) eventually manged to complete. This would be useful knowledge to obtain.  The percentage of AAO participants who state they are confident in English is slightly higher than the SSB group. Is this perhaps because a number of these are returning or experienced Moodlers, who return because they feel at ease in our MOOC environment? Certainly the free text answers support this: 15% of the All at once group said they were 'experienced' while only 09.% of the Step by step group did. Only 0.7% of the All at once group considered themselves 'new Moodlers' whereas 12.10% from the Step by Step group did.

Another factor affecting participants' choice of content release is, unsurprisingly, the desire to view the materials in a 'manageable' way. A higher percentage of the Step by step group said they had chosen the path to make their learning manageable - the word 'chunking' was even mentioned (18.20% AAO v 26.39% SBS) and in fact nobody from the All at once group used language implying feeling overwhelmed but 7.99% from the Step by step group did. The All at once group were keener to have 'flexiblity' of viewing course materials when they wanted to (16.80% AAO v 2.90% SSB). These result were not surprising to the facilitators when they were considered in the paired discussion, which follows in the next section.

Finally, when  discussing the findings of this study, it is important to consider its limitations:

Limitations

The data for this study came from two types of reports readily available to course facilitators in Moodle: Completion reports and Feedback responses. While these are useful, the information they can offer is limited. 

Sometimes, actions are too complex to be evaluated purely by the database server and presented to course facilitators in ready-made reports. In those cases, another "layer" of processing must be added. For Moodle,  typically, data can be pulled from the database via SQL and further processed in PHP before being presented to the user. Thus, other reports are available to those with server access or via contributed reporting plugins, but were not available for this particular project. Thus some questions remain unanswered. For example, according to the Participant survey at the end of Week 1, over 90% of members in both groups stated they were confident they would complete the MOOC. An SQL query could have revealed how many of those confident participants did actually go on to complete the MOOC. Likewise, a small percentage in both groups said their level of English was low. It was not possible with the standard reports available to discern how many, if any, went on to complete the MOOC despite their level of English. An SQL query such as the example in the section on Methodology could have shown how many of those who  accessed the course settings in  a language other than English actually completed it

Guo and Reinecke (2011) discuss the different ways students from different countries navigate through MOOCs, stating that those who are ultimately more successful tend to navigate in a non-linear way.  According to their reseach of four edX MOOCs,  students from the US and Europe tend to navigate in a non-linear way.  Choosing the AAO path offers more opportunities to navigate in a non-linear way as the materials from all weeks can be accessed at any time.  An SQL query could quickly identify the countries of origin and chosen paths of MOOC participants to ascertain whether these participants are, in fact, from the same demographic areas as suggested by Guo and Reineke. 

Postscript

Since this study, a further Learn Moodle MOOC has been run and the results obtained. Thus the most up to date completion figures are:

• Jan 2018 - 14.1%
• June 2018 - 16.0%
• Jan 2019 - 17.3%
• June 2019 - 16.8%

Co-facilitator paired discussion

• Although more participants opt for the All at once path, the actual percentage who complete the MOOCs is not very different. This result surprised both facilitators who expected one pathway to have more success than another. Helen expected the Step by step group would have a higher completion rate because, working through the MOOC week by week, they would feel more supported. On the other hand, I had expected the All at once group to have a higher completion rate as they demonstrated more confidence and experience.
• Despite adding an additional certificate for partial completion, the Certificate of achievement' this has not motivated participants to aim to obtain it. It was speculated that only the Certificate of completion is considered to be worthwhile. Despite the apparent lack of interest in obtaining it, it will still remain an option for those who do not fully complete.
• The  high confidence levels in the participant survey were suggested by Helen to be because it was the first week and participants were motivated. She also believed it was because those participants who log in and do the first week activities actually hope to complete. This supports the early engagement indicator of Jiang et al (2014)
• Finding other ways to identify participants at risk of dropping out and ways to motivate others to complete prompted the suggestion to trial the Learning analytics feature available in core Moodle (Since the paired discussion was recorded, a report has been published (Olivé et al, 2019) demonstrating that such a learning analytics framework could be very useful in future MOOCS.)

Conclusion: What have we learned?

The study asked four questions, which will be considered individually:

How does offering a choice of course content release affect the final overall completion rate of the Learn Moodle Basics MOOC?

The overall completion rates of the three MOOCs studied have risen slightly each time (14.1%, 16.0% and 17.3% if defining completion according to those who signed up.) However it is not possible to state with certainty whether this is because these three MOOCs have offered participants a choice of course content release or whether there are other factors. The MOOC has run twice yearly since 2015 and slight changes are made after each MOOC based on participant feedback. It is conceivable that these improvements, rather than offering a choice of content release, are the reason for the slightly higher completion rate. To test whether the access to materials is a contributing factor, the facilitators would have to run several MOOCs without a choice, and this would not be popular.

Which factors affect a participant’s choice of  course content release (All at once or step by step)?

Analysis of reponses in a participant survey in the January 2019 MOOC provides a number of insights. It is important to bear in mind that the data is from a single MOOC; however, 1127 participants responded, which is a fairly large cohort. Additionally not all participants chose to give free text replies.

• The MOOC is for beginner, non technical educators wishing to teach with Moodle. It is not designed to be a regular experience, but some participants engage in it more than once either to refresh their skills or simply to join a learning community. Participants who are doing the MOOC for the first time are more likely to choose the Step by Step group than the All at once group (85% of SBS were doing the MOOC for the first time as opposed to 69% AAO)
• Participants were asked how confident they were of completing the MOOC. There was a high level of confidence expressed by both groups: 95.85%  of the All at once group and 96.78% from the Step by Step group. However, participants took the survey at the end of their first week in the MOOC. Confidence levels, along with motivation might still be high, and it is important to bear in mind that not all those who completed the survey went on to complete.
• The MOOC is in English, although participants can communicate in forums in their own language. The level of competency in English does not appear to be a significant factor in choice of path, since both groups expressed a high level of competency - 95% AAO and 92.76% SBS said they were "very! or "quite" confident. The AAO group consider themselves slightly more confident but only by a very small percentage.
• An analysis of the free text responses enabled us to see how often certain key terms were used by both groups. The responses were grouped into themes and their frequency counted. From this, it is possible to deduce that more participants in the All at Once group considered themselves "Experienced" (15.68 % AAO v 0.9% SBS) while more participants from the Step by Step group felt they thought the weekly release of content might be "Easier" (2.10% AAO) v 10/16% SBS. Unsurprisingly, more from the All at Once group wanted the "Flexiblity" of a full overview (16.8% AAO v 2.9% SBS) and to be able to study at their "Own pace" (12.46% AAO v 2.9% SBS) whereas more from the Step by Step group hoped for the course to be "Manageable" (18.3% AAO v 26.3% SSB)

How does the choice of timed content release (all at once or step-by-step) affect a participant’s chance of completion of the Learn Moodle Basics MOOC?

This is in line with an earlier similar study (Mullaney and Reich, 2013). Although the MOOCs in that study  are separate and longer, the authors also concluded that there is no appreciable difference in completion rates. Thus we can say that a participant is no more or less likely to complete based their choice of course content release, and that we must look elsewhere for factors likely to improve completion rates.

Which other factors, if any, affect full, partial or non-completion of the course?

The peer assessment 'My home country workshop' activity is the only activity with deadlines, the first of which must be met in order to complete the activity and complete the whole MOOC. This has always been considered to be the main factor why more participants do not complete the MOOC. However the results of this study suggest that this is not the case.

Originally the course offered just one completion certificate based on completing all activities including one with a deadline. However, feedback from participants encouraged the facilitators to offer an alternative certificate for those who completed every activity except the single one with the deadline.

A surprising fact revealed by the study is the low figures for participants who, unable to obtain a full Certificate of completion, succeeded instead to obtain a 'lesser' Certificate of achievement. Completing over 30 required activities is a worthwhile achievement in itself, and so it was expected that the figures for obtaining this Certificate of achivement would be much higher than the overall mean which is either 0.01% or 0.0.2% depending on whether the figure is based on the number who sign up or the number who log in at the start. The mean percentages of the All at Once group who partially completed the course is 4% while the mean percentage of the Step by Step group is 3.0%, only slightly higher from the All at Once group and perhaps related to the fact that more participants from the All at Once group were either experienced or returning Moodlers and therefore more comfortable with the course materials.

It would appear that the promise of an alternative certificate for those who do not meet the single deadline is not motivating enough to encourage either cohort to pursue the course if they are unlikely to complete everything. 

Points raised in the paired discussion lead to speculation that participants, realising they cannot obtain a full completion certificate, lose interest in completing other, non-time dependent activities. Additionally it must be remembered that some participants join simply to experience the latest version of Moodle, or to help others, and for them completion is not important.

The following section will discuss how these findings can inform and improve the design of future Learn Moodle MOOCs, with the hope of increasing the completion rates.

Recommendations: Where to now?

Although the data suggests it makes little difference to eventual completion (or partial completion) whether a participant opts to see all course content at once or released weekly, the choice is popular with participants and during the paired facilitator discussion it was agreed that there is a social benefit in offering the choice. Thus we shall continue to do so.

The one activity with a deadline may prevent some participants obtaining full completion, but the peer assessment 'Workshop' is a very powerful activity and merits being showcased. The partial completion report data demonstrates that not completing the workshop does not, on its own, prevent participants from completing the MOOC. If it were a more significant factor, then the figures for the Certificate of achievement would be much higher. Possible action could involve restricting sign up to the MOOC so that, once the first workshop deadline is passed, it is not possible to join. This would limit the number of people who could fail but would also limit the number of people who could both experience the course with no wish to complete, and limit the number of people who could obtain a Certificate of achievement by partially completing. This course of action is unlikely to be chosen.

This study only used secondary datasets to obtain information about participants. An improvement on this would be to select a certain number of participants - perhaps by asking for volunteers at the start of the MOOC - and interviewing them at various stages of the MOOC. A short interview in the first week, the third week and at the end of the MOOC might provide more insight into the motivations and ongoing pressures of participants than the basic survey (feedback activity) and completion reports have done.

In terms of overall completion, it is proposed to enable Analytics on the site, in accordance with the paired facilitator discussion.  Moodle core offers  powerful analytics features which can, for example, alert the facilitators to  students at risk of dropping out or at risk of not completing, allowing the facilitators to contact and encourage the students in question. It is felt that actively engaging with individuals to support them could be a valuable way forward, and would have the added advantage of providing a live, large-scale testing ground for Moodle's Analytics features. Thus it is recommended that for the Learn Moodle 3.8 Basics MOOC in January 2020, this be trialled. Recent research carried out by Moodle HQ's Lead Data Scientist (  Monllau Olivé D et al, 2019) used datasets from eight  Learn Moodle Basics MOOCs to develop an analytics framework. Using the  Analytics features in the January 2020 Learn Moodle Basics MOOC would provide useful collaboration between the researchers of the previous project and the MOOC facilitators' desire to identify and minimise factors predicting drop out. This would link in with the participant interviews proposed earlier.

A recommendation for further data analysis might give a greater insight into the motivations of those participants who actually completed the MOOC. The participant survey included responses from 1127 participants who did not all go on to complete. The method of collection did not allow for filtering those who had completed. A future useful exercise would  be to extract the data such that the responses of those who completed could be compared with those who did not and identify any possible trends in, for example, confidence, experience or other motivations.

Since it would appear from the results of this study that the path learners choose does not affect the completion results, then the reasons for low completion (and for this MOOC, partial completion) must be sought elsewhere. What is is that makes so many sign up in the months before the MOOC and then either fail to begin the MOOC,  or begin  the MOOC and fail to complete? 

Another recommendation is to explore the process for participants signing up and first joining the MOOC. Currently sign up is a couple of months before the MOOC is launched -for example, at the time of writing the next Learn Moodle Basics MOOC will begin on January 13 2020 and the course will be open for sign-up from mid-November. Is this too distant a time from the launch, so that participants forget or lose interest? Or is it too close to the launch and there should be a much longer period - for example -should registration be open as soon as the previous MOOC has ended? The promotion on Social media could also be investigated to identify whether it is adequate and targetting the appropriate channels. The reminder email to participants who signed up should be reconsidered and the possiblity of a futher reminder email (or other notification) sent out at the end of the first week to those participants who have not yet engaged with the course. This can link in with the Learning Analytics feature discussed earlier, where alerts ("insights") can be sent out to participants and to course facilitators to help ensure learners are on track.

It is felt, therefore, that the most positive way forward is both to focus on maintaining the early  interest of participants who sign up for the MOOC by reminding them and encouraging them in the days prior to and early on in the course. We should follow Jiang et al (2014), who  stress the importance of monitoring learners' behaviour during the first week, as they claim it can point to subsequent completion. Additionally, it is recommended to make use of Moodle's Learning Analytics features to monitor and support them during the subsequent weeks until final certification. 

 By being more proactive in our support of participants via this closer monitoring, we hope to increase the number of completers and partial completers, in line with Adamopoulos (2013) who considers the online presence of the facilitator a significant factor in MOOC completion rates.

References

Adamopoulos, P., 2013. What makes a great MOOC? An interdisciplinary analysis of student

 Bali, M., 2014. MOOC pedagogy: gleaning good practice from existing MOOCs. Journal of Online Learning and Teaching, 10(1), p.44. retention in online courses.

 Breslow, L., Pritchard, D.E., DeBoer, J., Stump, G.S., Ho, A.D. and Seaton, D.T., 2013. Studying learning in the worldwide classroom research into edX's first MOOC. Research & Practice in Assessment, 8, pp.13-25.

 Cooch, M., Foster, H. and Costello, E. (2014). Our MOOC with Moodle - Moodle Research Library. [online] Research.moodle.net. Available at: http://research.moodle.net/6/

 Davison, C. (2013). What Was the First MOOC?. [online] HASTAC. Available at: https://www.hastac.org/blogs/cathy-davidson/2013/09/27/what-was-first-mooc [Accessed 3 Nov. 2018].

 De Barba, P.G., Kennedy, G.E. and Ainley, M.D., 2016. The role of students' motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32(3), pp.218-231.

Docs.moodle.org (a). (2019). Analytics - MoodleDocs. [online] Available at: https://docs.moodle.org/en/Analytics [Accessed 6 Jul. 2019].

Docs.moodle.org (b) (2019). Pedagogy- MoodleDocs. [online] Available at: https://docs.moodle.org/en/Pedagogy[Accessed 13 Jul. 2019].

 Dweck, C. S. (2012). Implicit theories. In P. A. M. Van Lange, A. W. Kruglanski, &

E. T. Higgins (Eds.), Handbook of theories of social psychology (pp. 43–62). Thousand Oaks, CA: Sage Publications.

 Foster, H. (2017). Teaching with Moodle MOOC January 2017: Feedback and actions. [Blog] Moodle.org. Available at: https://moodle.org/mod/forum/discuss.php?d=347029 [Accessed 10 Nov. 2018].

 Greene, J.A., Oswald, C.A. and Pomerantz, J., 2015. Predictors of retention and achievement in a massive open online course. American Educational Research Journal, 52(5), pp.925-955.

 Guo, P.J. and Reinecke, K., 2014, March. Demographic differences in how students navigate through MOOCs. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 21-30). ACM.

 Hood, N., Littlejohn, A. and Milligan, C., 2015. Context counts: How learners' contexts influence learning in a MOOC. Computers & Education, 91, pp.83-91.

Intelliboard.net. (2019). IntelliBoard - Your Data. Right Here. Right Now.. [online] Available at: https://intelliboard.net/ [Accessed 6 Jul. 2019].

 Jiang, S., Williams, A., Schenke, K., Warschauer, M. and O'dowd, D., 2014, July. Predicting MOOC performance with week 1 behavior. In Educational Data Mining 2014.

 Jordan, K., 2015b. Massive open online course completion rates revisited: Assessment, length and attrition. The International Review of Research in Open and Distributed Learning, 16(3).

Katyjordan.com. (2015a). MOOC completion rates. [online] Available at: http://www.katyjordan.com/MOOCproject.html [Accessed 10 Nov. 2018].

Kelly, N., Nesbit, S. and Oliver, C., 2012. A difficult journey: Transitioning from STEM to SoTL. International Journal for the Scholarship of Teaching and Learning, 6(1), p.18.

Lerís, D., Sein-Echaluce, M., Hernández, M., & Bueno, C. (2017). Validation of indicators for implementing an adaptive platform for MOOCs. Computers in Human Behavior, 72, 783.

 Littlejohn, A., Hood, N., Milligan, C. and Mustain, P., 2016. Learning in MOOCs: Motivations and self-regulated learning in MOOCs. The Internet and Higher Education, 29, pp.40-48.

Littlejohn, A. and Hood, N., 2018. Reconceptualising Learning in the Digital Age: The [un] democratising potential of MOOCs. Springer Singapore.

Monllau Olivé, D.M., Huynh, D.Q., Reynolds, M., Dougiamas, M. and Wiese, D., 2019. A supervised learning framework: using assessment to identify students at risk of dropping out of a MOOC. Journal of Computing in Higher Education, pp.1-18.

Mullaney, T and Reich, J 2013 Staggered Versus All-At-Once Content Release in MassiveOpen Online Courses: Evaluating a Natural Experiment

 Pintrich, P.R., 2000. An achievement goal theory perspective on issues in motivation terminology, theory, and research. Contemporary educational psychology, 25(1), pp.92-104.

Sonwalkar, N., 2013, September. The first adaptive MOOC: A case study on pedagogy framework and scalable cloud Architecture—Part I. In MOOCs Forum (Vol. 1, No. P, pp. 22-29). 140 Huguenot Street, 3rd Floor New Rochelle, NY 10801 USA: Mary Ann Liebert, Inc..

Watson, W.R., Yu, J.H. and Watson, S.L., 2018. Perceived attitudinal learning in a self-paced versus fixed-schedule MOOC. Educational Media International, 55(2), pp.170-181.

Yousef, A.M.F., Chatti, M.A., Schroeder, U. and Wosnitza, M., 2014, July. What drives a successful MOOC? An empirical examination of criteria to assure design quality of MOOCs. In Advanced Learning Technologies (ICALT), 2014 IEEE 14th International Conference on (pp. 44-48). IEEE.

Gratitude

Thanks to the tutors of Edinburgh Napier university, in particular to  Dr Anne Tierney.

Thanks to  Moodle HQ Community Manager Helen Foster, Senior developer David Mudrák and Lead Data Scientist David Monllau for their assistance with the project and to MoodleNet Lead Dr Doug Belshaw for encouragement and support.