Recommendations on Formative Assessment and Feedback Practices for stronger engagement in MOOCs

Many publications and surveys refer to the high drop out rate in Massive Open Online Courses (MOOCs) which is around 90%, especially if we compare the number of students who register against those who finish. Working towards improving student engagement in MOOCs, we focus on providing specific research-based recommendations on formative assessment and feedback practices that can advance student activity. In this respect, we analysed some significant research papers on formative assessment and feedback methods applicable to face-to-face teaching environments that advance student engagement, and concluded with related requirements and conditions that can be applied also to MOOCs. We also analysed 4050 comments and reviews of the seven most active and highly rated MOOCs (6 Coursera ones and 1 from EdX) provided by the students who have mainly completed those courses via CourseTalk. Based on this content analysis, we have formulated fourteen recommendations that support also the requirements/conditions of our conceptual and theoretical framework analysis. The results obtained shed some light in a rather unexplored research area, which is the research on formative assessment and feedback practices specifically for stronger engagement in MOOCs.


Introduction
Massive Open Online Courses are the latest trend in the area of e-learning and remote education and have significant popularity in the higher education community.For instance, as of July 2014 more than 8.2 million students have enrolled in at least one course in Coursera and more than 678 courses have been offered by 110 Universities and other institutions (Anderson, 2014).However, the drop out rates are very high in the range of 90% or more (Clow, 2013;Lewin, 2013) and the research community should focus on addressing this issue by trying to understand the causes and suggest specific solutions so that open education achieves its high potential and does not fail.In this respect, there is already some research activity on identifying the factors that influence student engagement that can be categorized into two broad categories: a) the non-didactic ones (students' and instructors' profiles, their demographics, reputation of institutions and of the teaching staff involved, certification options, fee options, course popularity, etc.) and didactic ones such as course structure and content, self-paced or not, workload and duration, course topic, type of exams, type of assessments and feedback, and interaction with students and instructors, etc. (Adamopoulos, 2013).Nevertheless, little research has been carried out to establish whether different types of assessment (formative, non-formative, peer and self-assessment) or different ways to provide feedback to students may actually affect course quality and impact (negatively or positively) on the MOOC students' learning experience and consequently on their engagement with the MOOCs.In this respect, we have built a conceptual and theoretical framework of requirements related to Open Praxis, vol.7 issue 2, April-June 2015, pp.141-152 feedback and formative assessment and based on this framework, we have analysed the reviews provided by students on related feedback and formative assessment practices applied in popular and highly ranked MOOCs.
The objective of this study is twofold: • Objective 1: To formulate a framework of requirements and conditions for effective feedback and formative assessment practices in MOOCs based on literature.• Objective 2: To suggest specific recommendations on feedback and formative assessment practices applied in MOOCs for increasing student engagement.
Our first objective is addressed through the conceptual and theoretical framework analysis (following section) and the second one is achieved through the analysis of the selected samples and the related content (methodology and results).

Conceptual and Theoretical Framework Analysis
In this paper we focus specifically on formative assessment and feedback practices that could be applied in MOOCs for enhancing student engagement.
The proposed paper is based on the identification and the analysis of specific research and scientific papers that conclude with recommendations on formative assessment and feedback that can be applied especially in MOOCs to advance student engagement (i.e.increasing students' activity) and acquisition of knowledge.
Although there is no evidence yet on which specific assessment and formative models can be applied in MOOCs for advancing student engagement, the research carried out by Nicol and Macfarlane-Dick (2006) was analysed further to examine whether its recommendations, if applied also in fully online eLearning environments including MOOCs, can support student engagement.Nicol and Macfarlane-Dick (2006) argue that there are three conditions as pre-requisites for students to be familiar with in order to benefit from feedback in academic tasks, i.e. students should be familiar in advance with: 1. what good performance is (i.e. the student must possess a concept of the goal or standard being aimed for); 2. how current performance relates to good performance (the student must be able to compare current and good performance); 3. how to act to close the gap between current and good performance.
The above three pre-requisites imply that in order for students to be able to compare an actual performance (of their own or of their peers) with a standard good one and take action to close the gap, the whole training process should dedicate much more effort on strengthening the students' self-assessment skills for better learning experiences.In this context, Nicol and Macfarlane-Dick (2006) have identified seven principles for good formative assessment and feedback that should be applied in traditional teaching environments in order to strengthen the students' capacity to selfregulate their own performance.These seven principles, in brief, suggest that effective assessment and feedback practices should a) help clarify what good performance is, b) facilitate the development of self-assessment (reflection) in learning, c) deliver high quality information to students about their learning, d) encourage teacher and peer-dialogue around learning, e) encourage positive motivational beliefs and self-esteem, f) provide opportunities to close the gap between current and desired performance, and g) provide information to teachers that can be used to structure the teaching approach.
Similarly, other research (Gibbs & Simpson, 2004) argues that assessment has positive effect on students' learning and engagement and proposes a set of conditions for this to happen in traditional teaching environments.In order for student engagement to be enhanced, assessment tasks should a) be sufficient to require students to dedicate appropriate study time, b) orientate students to allocate appropriate amounts of time and effort to the most important aspects of the course, c) engage students in related productive learning activities.Furthermore, feedback should a) be provided in sufficient detail and often enough, b) focus on students' performance, on their learning and on actions under the students' control, rather than on the students themselves and on their characteristics, c) be delivered in time for students so that it still matters for them, d) be aligned with the purpose of the related assignment e) be well received by the student and f) advance future learning and use by the student.
Additionally, according to Hew (2015), student engagement in MOOCs is defined as the level of a student's engagement in a learning activity.The more the student is active within a course, the more engaged she/he is with this course.Furthermore, Hew reviews specific literature (Fredricks, Blumenfeld & Paris, 2004;Helme & Clarke, 1998) on student engagement and has identified its three main dimensions: 1. behavioral engagement referring to the learning activities that students are doing within a course such as completing an assignment, watching videos, participating in forums, etc, 2. affective engagement referring to the feelings that learning activities generate in students towards other colleagues, tutors, the course itself or the institution that runs the course 3. cognitive engagement referring to the emerging thoughts that learning activities provoke in students, e.g.cognition activity for asking and answering questions, for giving clarifications, for reasoning, etc.
Furthermore, Hew with the support of other literature (Reeve, 2012;Skinner, Kindermann, Connell & Wellborn, 2009) directly links student engagement with motivation and more specifically with Self-Determination Theory (STD) (Deci & Ryan, 1991;Deci & Ryan, 2000;Hardre and Reeve, 2003) and concludes that student engagement and more specifically 1. behavioral engagement is driven from the need of autonomy (the need students feel to sense they are non dependent on other peoples' actions), 2. affective engagement is driven from the need of relatedness (the need students feel to connect with other people) and 3. cognitive engagement is driven from the need of competence (the need students feel to master specific knowledge).
Based on this conceptual and theoretical analysis, we can argue that formative feedback and related assessment methods can support student engagement as long as they follow a specific framework of requirements and conditions: 1. Requirement 1: The course should fulfill the need of autonomy and consequently behavioral engagement that is for example addressed through self-assessment practices (i.e. a student to assess their own work and assignments), 2. Requirement 2: The course should fulfill the need of relatedness and consequently affective engagement that is for example achieved through peer assessment practices (i.e. a student to assess the work and assignment of other students/peers) 3. Requirement 3: The course should fulfill the need of competence and in this manner also cognitive engagement through for example formative assessment and feedback (that is,

Methodology Selection Process/Sample
In our approach, we have used a special online review platform for MOOCS, i.e.CourseTalk (www.coursetalk.com)and we have applied the following content analysis methodology.
1. We have collected all the responses from all the e-courses reviewed at Coursetalk that satisfy all of the following conditions based on the data available on Coursetalk on 24 th November 2014: • they are offered for free and • they are offered by Universities and • they are top-rated, i.e. 5/5 stars and • they have received more than 100 reviews (in order to have a significant content to analyse) 2. Based on the above, we have identified the following 7 MOOCs and around 4050 reviews for our content analysis The main formative assessment and feedback methods that have been applied in all those courses have focused on peer-assessments and feedback through discussion forums with the participation of the instructors in some cases or of their assistants as well.

Content Analysis Process Steps
The total 4050 reviews include all the comments/reviews received by all participants that have mainly completed any of the seven MOOCs.Therefore, in order to analyse these reviews and conclude with specific recommendations related to feedback and formative assessment practices that advance student engagement, we have applied five specific steps: • Step 1: filtering and selection of comments related to feedback and formative assessment • Step 2: further selection of those comments that specifically contribute or provide some input on the recommendations to be formulated • Step 3: Reading each one of the finally selected comments and formulation of specific recommendations based also on the requirements from the analysis of the conceptual and theoretical framework.Special attention is given so that total suggested recommendations are linked with the requirements that were identified in our conceptual and theoretical framework analysis in a previous section in this paper.• Step 4: Re-reading each one of the finally selected comments and identification of a) how many times each review/comment was used as source for each recommendation and b) whether there were any other recommendations raised that may have been missed from the previous step or whether there were any further revisions needed.Our content analysis took place mainly at step 3 and step 4 and in order to validate its findings, two additional judges that had not been initially involved in the development process of steps 3 and 4 were engaged.They applied steps 3 and 4 in a sample of reviews in order to ascertain agreement on the findings.• Step 5: Based on the number of reviews/comments that are used as sources for each of the formulated recommendations, we identify the significance of each recommendation and we rank them accordingly.

Step by step Approach -Results
Below, we outline each step in our approach and their results.
Step 1 Out of the 4050 reviews, we collected all responses that specifically commented on feedback and assessment methods by identifying them with the search terms "Feedback", "Assessment", "Peer" and "Assignment".In total, we have collected 418 reviews.
Open Praxis, vol.7 issue 2, April-June 2015, pp.141-152 Step 2 Out of the 418 reviews, we eliminated those that were general, either positive or negative, and were not contributing or providing any specific details in regards to feedback and formative assessment practices that have been applied in each MOOC and we were left with 237 comments/reviews in total.
Step 3 We carefully read each of the 237 comments/reviews and we used each one of them as a reference source to elaborate a specific recommendation on feedback and formative assessment for advancing student engagement.Each review was checked carefully on whether it was the source of a new recommendation or the source of an already raised recommendation from previous comments/ reviews.Also each of the formulated recommendations was checked concerning whether it was related to any of the sixteen requirements from our conceptual analysis.Based on this approach, we ended up with a total of fourteen recommendations/findings which are all linked to our main requirements on feedback and formative assessment practices for higher student engagement as shown in table 1. Recommendation/finding 1: Assessment models, such as self assessments (students judge their own work) and peer-assessments (students judge the work of their peers) can be relatively complex in MOOCs and a comprehensive rubric should be provided in all MOOCS that involve peer assessments.
Requirements 4, 5, 6, 9, 12 Recommendation/finding 2: Self and peer assessments can be obligatory or penalties can be applied and should have as priority that peers examine the work of others and provide constructive feedback (furthermore, if possible, students should be able to modify their own related assignment work even after its submission based on feedback received).
Requirements 4, 6, 9, 14, 15 Recommendation/finding 3: Although MOOCs can automatically assess quizzes, self and peer assessments must deal with more comprehensive assignments.However, self-and peer -assessment should be formative as well, i.e. specifically intended to generate feedback on performance and improve and accelerate learning rather than just providing a mark.
Requirements 3, 6, 8, 9, 11, 15, Recommendation/finding 4: Peer assessment in MOOCs does not always deliver accurate results and in many cases it creates frustrations or negative feelings since it doesn't provide any feedback on the work assessed or there are concerns on the competencies of the peer-assessor.
Requirements 2,3,4,5,6,8,9,14,15 Recommendation/finding 5: Peer assessment was more welcome in those MOOCs where there was no mark given or alternatively feedback was given on the quality of the related assignment.
Requirements 1, 2, 3, 4, 5, 6, 8, 9, 14, 15, Step 4 Then, we read again each of the 237 comments/reviews in order to a) identify how many times each review/comment was used as source for each recommendation and b) check whether there were any other recommendations raised that may have been missed from the previous step or whether there were any revisions needed on the 14 recommendations.Consequently, we haven't identified any other recommendations apart from the 14 initial ones.Also the text of three of them was slightly revised, which is addressed above.Based on this step, table 2 was prepared, showing for each course: the total number of reviews given (Column 3), the number of potentially useful reviews (Column 4), the number of identified useful reviews (Column 5).In the same manner, table Open Praxis, vol.7 issue 2, April-June 2015, pp.141-152 3 shows the number of reviews used as sources for each recommendation (Columns 2-Column 15).Also the overall totals are shown accordingly.In addition, in order to avoid having the courses with a large number of related reviews to influence the final ranking of the recommendations, we have applied the weighted average method for each recommendation as is shown in the next step.Step 5 We applied the weighted average method for each recommendation.For example, for Recommendation 1, the weighted average is (17x174+3x7+0x4+7x17+3x16+0x11+1x8)/237=13 We have calculated the weighted average for each of the 14 recommendations as is shown on table 3 above based on the number of reviews that were used as sources for each of the fourteen recommendations.In this way, we have avoided the influence in the ranking of one course with many comments.Based on the weighted average derived for each recommendation, we have ranked them accordingly as the following chart shows (see Chart 1).Therefore, our recommendations/ findings are ranked and provided below from the highest to the lowest significance by considering also their weighted average.a. Recommendation/finding 14: Assignments can be based on practical problems with clear expected output that give the sense of completeness to the students, and are not that easy to address but are sufficiently challenging and interesting (65 reviews supported this recommendation/finding with 42 as weighted average)  for students to receive support, sympathy, formative feedback and clarification as well as share ideas on their work as long as there is an effective mechanism on managing and accessing the discussion threads.(36 reviews supported this recommendation/finding with 18 as weighted average) e. Recommendation/finding 1: Assessment models, such as self assessments (students judge their own work) and peer-assessments (students judge the work of their peers) can be relatively complex in MOOCs and a comprehensive rubric should be provided in all MOOCS that involve peer assessments (29 reviews supported this recommendation/finding with 13 as weighted average) f.Recommendation/finding 5: Peer assessment was more welcome in those MOOCs where there was no mark given or alternatively feedback was given on the quality of the related assignment.(23 reviews supported this recommendation/finding with 10 as weighted average) g.Recommendation/finding 4: Peer assessment in MOOCs does not always deliver accurate results and in many cases it creates frustrations or negative feelings since it doesn't provide any feedback on the work assessed or there are concerns on the competencies of the peerassessor.(26 reviews supported this recommendation/finding with 9 as weighted average) h.Recommendation/finding 6: Immediate feedback mainly from automated simple but well structured quizzes is appreciated by students since it confirms their understanding about what they have learnt.The appreciation is even higher in cases that some further feedback on the given answer or on the correct answer is provided.(18 reviews supported this recommendation/finding with 8 as weighted average) i. Recommendation/finding 12: The posting of comments by human experts, for instance instructors' assistants that intervene and provide comments and views in discussion forums or even in students' work is appreciated.(11 reviews supported this recommendation/finding with 3 as weighted average) j.Recommendation/finding 9: Peer assessment can be supplemented with related forum discussions for interaction and the possibility to offer a communication channel for clarifications.(9 reviews supported this recommendation/finding with 3 as weighted average) k.Recommendation/finding 2: Self and peer assessments can be obligatory or penalties can be applied and should have as priority for the peers to examine the work of others and provide constructive feedback (furthermore, if possible, students should be able to modify their own related assignment work even after its submission based on feedback received).(4 reviews supported this recommendation with 3 as weighted average) l.Recommendations 11, 8 and 7 are not considered since they have weighted average close to 0.

Conclusions
We have analysed existing significant research papers that deal with formative assessment and feedback methods as well as their requirements and conditions for supporting student engagement mainly in traditional teaching environments and we have confirmed their validity and applicability also in MOOCs.This is the first time that such research validation takes place for eCourses and more specifically for MOOCs, which is a relatively new field.MOOCs have unique characteristics and our research was based on new data to confirm, prioritize and also emphasize some existing pedagogical concepts valid in traditional education.Consequently, we have confirmed that specific requirements and conditions on feedback and formative assessment for advancing student engagement that are applicable in traditional teaching and learning practices, are also applicable in MOOCs.Furthermore, we have formulated and ranked fourteen related recommendations as good practices based on review comments given by students that completed any of the seven most popular courses in a specific online review platform, namely Coursetalk.The ranking was based on the frequency of occurrence of the same or similar comments given by the students and we have assumed that the higher the number of comments that support one recommendation, the higher is the importance of this recommendation.
In this way, we have concluded some very interesting findings.For example, MOOCs can really benefit and encourage students' activity and consequently their engagement, when assignments are based on practical problems with clear expected output that give the sense of completeness to the students and are not that easy to address but are sufficiently challenging and interesting.
Also MOOC students really appreciate it when self and peer assessments deal with more comprehensive assignments and are formative as well, i.e. specifically intended to generate feedback on performance and improve and accelerate learning rather than just providing a mark.
On the other hand, some specific practices that are applied in MOOCs, do not seem to be that important for the students, for example, use of the same peers in all peer-assessment exercises, or the provision of feedback on assignments in time before the next assignment, or no restrictions on the number of quiz attempts.
The results of this research are a first attempt to shed some light on how student engagement in MOOCs can be improved via specific formative assessment and feedback practices.However, the dataset was rather limited and it was based only on the MOOC participants that provided feedback through the Coursetalk platform.Further research could focus on comments and reviews collected directly from the MOOC participants in regards to formative and feedback assessment practices and how they can affect their engagement in the course.
Ranking of each recommendation with respect to its significance/no. of linked reviews Students should know in advance what good performance is and based on that to be able to compare assessments 5. Requirement 5: Students should know in advance the necessary actions needed to reach good performance 6. Requirement 6: Formative assessment and feedback practices should help students clarify what good performance is and how much different this is with respect to their current performance 7. Requirement 7: Formative assessment and feedback practices should encourage dialogue around learning between peers and teachers 8. Requirement 8: Formative assessment and feedback practices should create positive motivational beliefs and self-esteem 9. Requirement 9: Formative assessment and feedback practices should encourage students to take actions in order to achieve good performance 10.Requirement 10: Formative assessment and feedback practices should provide information to teachers to improve their teaching approach 11.Requirement 11: Assessment tasks should be sufficient in order to require students to dedicate appropriate study time for addressing them 12. Requirement 12: Assessment tasks should orientate students to allocate appropriate amounts of time and effort to the most important aspects of the course 13.Requirement 13: Assessment tasks should engage students with related productive learning activities 14.Requirement 14: Feedback should be provided in sufficient detail and frequently enough 15.Requirement 15: Feedback should focus on students' performance regarding their learning 16.Requirement 16: Feedback should be delivered in time to students so that it still matters and it is well received 17.Requirement 17: Feedback should be aligned with the purpose of the related assignment and advance future learning and use by the studentThe above analysis shapes the necessary framework of requirements to analyse students' reviews on highly rated MOOCs and to formulate related recommendations on formative assessment and feedback methods that can advance student engagement.

•
An introduction to Interactive Programming in Python, offered by Coursera (1863 reviews) • A beginner's Guide to Irrational Behavior, offered by Coursera (375 Reviews) • Epidemics -the Dynamics of Infectious Diseases offered by Coursera (315 Reviews) • Design: Creation of Artifacts in Society, offered by Coursera (204 Reviews) • Modern and Contemporary American Poetry, offered by Coursera (170 Reviews) • An introduction to Operations Management, offered by Coursera (104 Reviews) • Think101x: The Science of Everyday Thinking, offered by EdX (1037 Reviews)

Table 2 : Number of reviews per specific category and per MOOCs
Although MOOCs can automatically assess quizzes, self and peer assessments must deal with more comprehensive assignments.However, self-and peer assessment should be formative as well, i.e. specifically intended to generate feedback on performance and improve and accelerate learning rather than just providing a mark.Discussion forums are appreciated and are a very good way Recommendations on Formative Assessment and Feedback Practices for stronger engagement in MOOCs Open Praxis, vol.7 issue 2, April-June 2015, pp.141-152