Reflections on assessment in Open Distance Learning ( ODL ) : the case of the University of South Africa ( UNISA )

The article explores the challenges of assessment in open distance learning (ODL). The authors argue that ultimately assessment should be about improving the quality of teaching and effective learning. It should be based on making expectations explicit and public, setting appropriate criteria and high standards for learning quality, systematically gathering, analyzing and interpreting evidence to determine how well performance matches expectations and standards, and using the resultant information to document, explain, and improve performance. However, getting all these variables to work in ODL presents mammoth challenges. How can ODL lecturers validate and authenticate students’ written work? How can they tell whether the students’ submitted work sufficiently reflects their knowledge and understanding? South Africa has inherited an unequal, racially skewed and inequitable educational provision from its apartheid past. This poses serious challenge for assessing quality. The article therefore seeks to understand these context-specific challenges of ODL assessment at UNISA.


Introduction
We teach in the Bachelor of Education (BEd undergraduate), Postgraduate Certificate in Education (PGCE) and Honours Bachelor of Education (Honours BEd) programmes at the University of South Africa (UNISA).Running through all these programmes there is a course-Theoretical Frameworks in Education-that introduces and consolidates the students' grasp of philosophical frameworks and how they shape educational practice.We will come back to this course in more detail later.UNISA (2012) is the biggest open distance learning (ODL) institution in Africa.Unlike campusbased, full-time contact institutions that cater mainly for young school leavers entering university for the first time, ODL institutions provide access to higher education to mature working students.Such students would not be able to obtain a university qualification were they to have campus-based full-time contact institutions as their only avenue for accessing higher education.
Most ODL institutions market themselves to prospective students as open, accessible, flexible, supportive and affordable.Presumed in this market discourse is the view that adult working students take responsibility for their learning; they learn alone or in small groups; they learn at their own pace and in their own time; they learn from a variety of learning materials, including the use of audio-visual media; they are active rather than passive learners; they need less frequent help from their teachers; they therefore learn from other people besides their teachers, and that they will do a lot of self-assessment (Rowntree, 1996).UNISA (2008, p. 2) commits to adhering to "responsible open admission policy."Its open learning approach "gives students flexibility and choice over what, when, where, at what pace and how they learn.Open learning is all encompassing and includes distance education, resource-based learning, correspondence learning, flexi-study and self-paced study."UNISA's commitment above resonates with access requirements of the Open University (OU) in the United Kingdom (UK), which are based on the principle of "no entry requirements."Our understanding is that nearly all OUUK courses have no entry requirements.
Open Praxis, vol. 5 issue 3, July-September 2013, pp. 197-206 On the surface these pronouncements sound noble.However, at a deeper level they pose profound challenges with respect to ways of assessing and assuring the quality of ODL teaching and learning.For instance, it might not be easy to validate or authenticate ODL students' written work and to ascertain whether the work they have submitted is theirs and that it constitutes a true reflection of their level of content knowledge and understanding of the subject matter.While this might be reasonably monitored at campus-based full-time institutions, the challenges are more pronounced in ODL institutions where students are not required to attend classes in person.In this article, which builds on our previous work on the challenges of teaching and learning in ODL (Letseka & Pitsoe, 2012, 2013), we explore the challenges of assessment in ODL, with a focus on UNISA.The article has six sections.First, we sketch various conceptions of assessment with a view to ascertaining how they pertain to the course we offer.Second, we outline formative and summative assessment.Third, we describe the course Theoretical Frameworks in Education, its purpose, and exit level learning outcomes.Fourth, we briefly sketch ways in which the course is assessed.Fifth, we reflect on the challenges of assessment with respect to assuring the quality of teaching and learning in ODL.In the final section we provide some concluding remarks.We now turn to conceptions of assessment.

Conceptions of assessment
There is agreement among advocates of assessment that effective assessment should be ongoing and focused on improving students' learning (Angelo, 1991(Angelo, , 1994(Angelo, , 1995(Angelo, , 1999;;Bell & Cowie, 2001;Cohen, 2008;Hunsaker, 2004;Shepard, 2000).For instance, Thomas Angelo (1999) is concerned that most assessment efforts have resulted in little learning improvement because they are implemented without a clear vision of what "higher" or "deeper" learning is and without an understanding of how assessment can promote such learning.He attributes this to "piecemeal" attempts that stem partly from a mechanistic, "additive" model of assessment.He suggests that a "transformative" assessment-as-culture-change model should replace this.Angelo (1999) argues that "transformative" assessment-as-culture-change rests on the following four pillars: • Building shared trust by lowering social and interpersonal barriers to change; • Building shared motivation by collectively identifying goals worth working toward and problems worth solving, and considering the likely costs and benefits; • Building a shared language by developing a collective understanding of new concepts (mental models) needed for transformation; • Building shared guidelines by developing a short list of research-based guidelines for using assessment to promote learning.
It follows that if lecturers were to plan their assessment as if learning matters most-and not just student learning, but theirs as well-the distance between means and ends would be reduced and chances of success increased.Angelo (1995) conceives of assessment as an ongoing process aimed at understanding and improving student learning.He argues that assessment should involve making expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analysing and interpreting evidence to determine how well performance matches expectations and standards; and using the resulting information to document, explain, and improve performance.Angelo's views above are echoed by Brown and Knight (2004), who argue that assessment techniques work better where learning outcomes have been articulated in advance, shared with the students, and assessment criteria agreed upon.Thus assessment helps create a shared academic culture dedicated to assuring and improving the quality of teaching and learning in higher education.Taylor (2005) advocates the following nine principles of good practice in assessment.For him assessment of student learning • Should begin with educational values; • Is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time; • Works best when its programs have clear, explicitly stated purposes; • Should pay attention to outcomes and equally, to the experiences that lead to those outcomes; • Works best when it is ongoing, not episodic; • Should foster wider improvement when representatives from across the educational community are involved; • Makes a difference when it begins with issues of use and illuminates questions that people really care about; • Is most likely to lead to improvement when it is part of a larger set of conditions that promote change; • Should enable educators to meet their responsibilities to students and to the public.That is, it should foster accountability.Hunsaker (2004) reiterates the view on assessment as an on-going process oriented around curricula change aimed at improving learning.He advocates scholarly assessment, which he defines as a tool for helping the lecturers understand the teaching and learning process, a means by which lecturers can learn what is working, and where they might do well to approach things differently.In this regard assessment should be all about teaching and learning.One of the courses which we teach at UNISA, and which we describe below, targets ODL undergraduate and entry-level postgraduate students.While its goal is to initiate the students in the knowledge of the theories and practice of education 1 , at the heart of its assessment plan is a concern with the best practices that ensure students' learning is improved regardless of the ODL nature of the institution.As Brown and Knight (2004) note, assessment should be used to shape and direct future learning.Angelo (1991) highlights this purpose more succinctly in his suggestion that assessment should provide faculty and students with information and insights needed to improve teaching effectiveness and learning quality.The focus should be more on finding out what and how much (or how little) students have learned between points A and Z, that is, to establish accountability and improve efficiency (Angelo, 1994).In support of this view Cohen (2008) argues that assessment must be interactive and proactive.It must be a form of participation.Cohen (2008) regards participation as a way of enhancing learning and a form of feedback on the nature of the learning that is, or is not taking place.The lecturer and the students should collaborate actively to produce the best performance (Yorke, 2003).Cohen (2008) contends that because there is the content that the lecturers want the students to understand, discussing readings in class should be central to teaching in that it gives the lecturers the opportunity to take a closer look at student learning and to assess how well they are learning.
Participation provides an indication of students' understanding of their readings.How lecturers respond to students' comments, the feedback they give on various assignments, and the way they use the results from class participation could significantly impact student learning.These views resonate with the South African Qualifications Authority's (SAQA) (2001) policy position on assessment.SAQA (2001) posits that assessment in education and training should involve gathering evidence of learners' work so that judgements can be made about their achievements or non-achievements.Thus university students should ideally be more involved in their assessment on the grounds that they are the relevant stakeholders.However, this is easier said than done.Most Open Praxis, vol. 5 issue 3, July-September 2013, pp. 197-206 forms of assessments are still top-down and largely driven by the educators.Below we briefly explore formative and summative assessment.

Formative assessment
Formative assessment is of critical importance to student learning and retention (Yorke, 2001).Its central purpose is to contribute to student learning through the provision of information about performance (Yorke, 2003).Formative assessment has the advantage in that it is dialogic.The students receive regular feedback on their performance from their teachers.The exchange between the teachers and the students is (ideally) mutually hermeneutic in that each is seeking to interpret and understand the communications of the other with the aim that the students will become better equipped to deal with the challenges of future study.Mentkowski (2000) argues that formative assessment should serve as supportive criticism, and an important support for learning and motivation.In this regard feedback procedures should assist students in forming accurate perceptions of their abilities and in establishing internal standards with which to evaluate their own work.For Bell and Cowie (2001), formative assessment must be ongoing, dynamic and progressive, informal, interactive, unplanned as well as planned.It must be reactive as well as proactive, with the class, group, or individual.Black and William (1998) are persuaded by formative assessment's effectiveness in promoting student learning across a wide range of educational settings (disciplinary areas, types of outcomes, levels).They argue that an important determinant of the effectiveness of formative assessment is the quality of the feedback received by learners.In framing the feedback to students, lecturers should keep in mind several important and delicate considerations that are neither widely known nor understood.Whether lecturers across the different education sub-disciplines recognise this critical role of quality feedback to students remains to be seen and is an area that still needs further research.

Summative assessment
In summative assessment, assessors look for evidence of achievement at exit level.This involves identifying some data as relevant to specified goals and the criteria that derive from them (Knight, 2002).Judgments are made about the match between evidence and criteria.But as Brown and Knight (2004) point out, summative assessment has always been a vexed business.First, because higher education institutions are generally expected to have learning goals that are far more extensive and complex than mastery of subject matter alone, and they are being held to account for student achievement in terms of those goals.Second, a greater range of assessment techniques has come into currency, which has introduced substantial practical and theoretical problems, with the comparability and aggregation of performances judged by different assessment methods.Third, public sector services are nowadays marked by low-trust management systems, when once there would have been a greater readiness to trust that good people engaged in worthwhile activities would learn the sorts of things that were intended.Thus assessment is supposed to supply evidence to bridge the trust gap.Fourth, the eternal concern with value for money has taken a rationalist turn, with the belief that it is prudent to specify objectives, measure inputs, assess performance in terms of those objectives, allocate the next round of resources to efficient providers and apply sanctions to the less efficient.How do these conceptions of assessment play themselves out at UNISA?We mentioned above that running through the Bachelor of Education (BEd undergraduate), Postgraduate Certificate in Education (PGCE) and Honours Bachelor of Education (Honours BEd) programs is the course, Theoretical Frameworks in Education.In the next section we briefly sketch the course, highlighting its internal structure, and the kind of competencies it seeks to develop.

The course: Theoretical Frameworks in Education
The above-mentioned course is one of the central components of the Bachelor of Education (BEd undergraduate), Postgraduate Certificate in Education (PGCE) and Honours Bachelor of Education (Honours BEd) programs at UNISA.The course explores different ways of thinking about education and undertaking research in education.It aims to equip pre-service and in-service teachers with the skills to identify, articulate and critically engage with their own underlying assumptions about the nature of education, their work in education, and the ideas of others (Venter, Higgs, Jeevanatham, Letseka & Mays, 2006).The course encourages the students to "return again and again to the course material, re-examine their assumptions, the questions they are asking, and the answers they are discovering."But by phrasing the challenges of the course as they do, the course designers seem to assume that all the students have been introduced to some prior learning in the elements of philosophic inquiry, which is often not the case.
Theoretical Frameworks in Education is conceptual, philosophical, and contemplative.It is structured around nine (9) theoretical frameworks or meta-theories in philosophy of education, namely, logical empiricism, hermeneutics, systems theory, feminism, phenomenology, critical theory, African philosophy, critical rationalism, and post-modernism.The structure of the course is consistent with the broader aim of the Honours BEd programme, which is the development of applied competence.In this regard, applied competence is understood to involve the integration of the following four other kinds of competences: • Ability to conduct independent inquiry in a specialised field of education, training or development, and to report findings in academically appropriate ways; • Demonstrate acquisition of a sound knowledge base and critical understanding of education in general and of the chosen area of specialisation in particular; • Ability to critically analyse and evaluate knowledge in the chosen area of specialisation, and to contribute to systematic and disciplined thinking about educational matters and issues; • Exhibit the potential to act as academic leader and expert in the field of education, training and development.
Thus understood applied competence can be briefly summed up as involving the integration of foundational competence, that is, the ability to demonstrate understanding of key concepts and issues; practical competence, that is, the ability to use what has been learned in some practical way; and reflexive competence, that is, the ability to evaluate one's work and that of others in order to identify areas that need to be improved (UNISA, 2010).
The course's reading material comprises a study guide, about which the students are informed "works like a teacher."It will "structure your learning, explain concepts and direct you to other parts of the module at appropriate times.It will facilitate your learning through the development of a dialogue involving guided activities and feedback" (Venter et al., 2006, p. vi).Then there is the prescribed textbook, which "is like a resource person to whom you go for advice.The textbook explores some of the main schools of thought influencing education decision making and research."Third are Tutorial Letters.These are a series of communiqués that provide students with useful Open Praxis, vol. 5 issue 3, July-September 2013, pp. 197-206 information on the course: "tutorial letters are the means by which we [lecturers] maintain an ongoing relationship with you [the student]," the study guide declares.Tutorial Letters provide information on the purpose, nature, structure and learning outcomes of the course.Finally, the students are encouraged to keep a workbook or file or computer folder.This is a record of the students' thinking and development throughout the programme.The students are encouraged to use a workbook, file or computer folder to record their responses to activities, write notes and summaries, and generally record their ideas as they work through the programme."Thoughts are a bit like fish," the students are warned."They can easily slip away," which is why it is important to make a habit of writing ideas down as they occur, and to revisit and reflect on them.
The students are encouraged to read through the prescribed book; develop a broad overview of the theoretical frameworks, be able to compare the main ideas of the various theoreticians, and to contrast similarities and distinguishing characteristics.In the end, they should be able to draw on one or more theoretical frameworks to resolve issues pertaining to specific educational practices.For instance, the students might be required to draw on a particular theoretical framework to identify curricular subjects in the teaching/learning context, and to develop appropriate instructional strategies to critically reflect on their practice as teachers.We now turn to the course's assessment.

How Theoretical Frameworks in Education is assessed
So far we have argued that assessment should be an ongoing process and a means for lecturers to ascertain what is working, and what needs to be done to improve student learning.We have established that the main purpose of assessment is to provide lecturers and students with information and insights in order to improve teaching effectiveness and learning quality.Moreover, we have highlighted that assessment should be a collaborative endeavour between the lecturers and the students.It follows from the above exposition that assessment is a powerful tool for use by lecturers to make timely and effective curricular interventions that will ensure effective students' learning.
We now briefly outline the assessment plan of the course, Theoretical Frameworks in Education.We should mention that the planning and preparation of assessment should involve four critical aspects: first, why is assessment being conducted?Second, what competences should be assessed?Third, who should be involved in assessment (Teachers, lecturers, students)?And fourth, how assessment should be carried out (methods, instruments, resources and procedures)?
It should be clear by now that Theoretical Frameworks in Education is not about rote learning and/or regurgitation of memorised information.Instead the course requires students to critically engage with what they have learned, and to "try things out and evaluate the outcomes" (UNISA, 2010).The course is assessed through a combination of essay assignments and an examination.The students are required to submit three to six essay-type conceptual, philosophical, and contemplative assignments and obtain an aggregate of at least 50% towards their continuous, formative aspect of the assessment of the programme.The students are cautioned that the following verbs will be used in the assignment and examination questions: "analyse," "compare," "define," "describe," "discuss," "evaluate," "explain," "illustrate," "interpret," "juxtapose," "outline," "synthesise," and "tabulate," to mention a few.This caution is intended to alert the students to the conceptual, philosophical, and contemplative nature of the course.How lecturers are able to instil among ODL students these conceptual, philosophical, and contemplative inclinations is a challenge to which we now turn and reflect upon.

The challenges of assessment in ODL
We mentioned above that ODL institutions market themselves as open, accessible, flexible, supportive and affordable.Indeed most ODL institutions operate "no entry requirements" and/or other use-friendly alternative admission criteria such as the Senate Discretionary Admission (SDA) requirements and/or recognition of prior learning (RPL).The reason for this is that most ODL institutions provide access to higher education to mature working students who would otherwise not be able to obtain a higher education qualification were they to have campus-based contact higher education institutions as their only option for accessing higher education.And yet noble as this provision of access might seem, in the end the principle of "no entry requirements" raises a number of red flags with regard to assuring the quality of teaching and learning in distance learning.
What we want to do in this penultimate section is to highlight some of the contestations around the notion of "quality" in education with a view to showing how such contestations have implications not only on how we assess, but also on how we assure the quality of academic work submitted by students in ODL.The notion of quality in education has always been a contested terrain (Ball, 1985;Frazer, 1994;Green, 1994;Massy, 2003).For instance, in the mid-1980s when Christopher Ball (1985) asked: "What the hell is quality?" he was grappling with the muddle that surrounds the meaning of the term.In her book-What is Quality in Higher Education-Diana Green (1994, pp.12-13) argues that quality, like "freedom" or "justice," is an elusive concept.In the same vein Malcolm Frazer (1994, p. 103), one of the contributing authors in Diana Green's book above, acknowledges that quality in higher education is a complex idea about which there is no agreement either between, or within countries about what is meant by quality.But There is agreement among commentators on quality that quality is a measurement of some kind.For instance, while Frazer (1994) suggests that quality should be about what students have learned, Green (1994, p. 5) contends that institutions should not only be more efficient, but should also be more responsive to the needs of their students and accountable to the taxpayer.To come back to Chris Ball (1985), he eventually conceded that "quality in education" is a subject extraordinarily difficult to come to grips with, and full of pitfalls.
The above analysis of quality is pertinent to our conception of assessment and how we go about assuring the quality of our students' written work.It is our view that the challenges of assessment are more pronounced in ODL institutions than in campus-based, full-time contact institutions.Elsewhere we have argued that "the concept of open and distance learning suggests an educational approach designed to reach learners in their homes/offices/shops etc, provide learning resources for them to qualify without attending formal classes in person, or create opportunities for lifelong learning, no matter where or when they want to study" (Letseka & Pitsoe, 2012, p. 222).
It seems such a challenge to sufficiently ascertain what students really know, what they can do, and what attitudes they might have acquired as a result of interacting with the learning material in an ODL environment.The reason for this is not hard to find.Even in a full-time contact education institution it is not easy to ascertain whether the students actually interact with the learning material in the first place, or whether they make time to interact with the learning material at all.The challenge becomes more profound in the learning environment where the students are not required to attend "formal classes in person." We may, on the surface be misconstrued as ODL sceptics.Our scepticism, if there is any, is borne out of the practical challenges at UNISA where students submit well-written, well-argued and coherent assignments, but perform dismally in the examination.On numerous occasions we have been left wondering whether a relative or a close friend, who has probably graduated from the programme, Open Praxis, vol. 5 issue 3, July-September 2013, pp. 197-206 could have assisted one of our students in writing the assignments given that during the examination some of the students perform poorly in the conceptual, philosophic and contemplative questions.Given the very nature of the ODL environment, and regardless of having watertight assessment rubrics and criteria, there is evidence of conspicuous inconsistencies in performance among our students between assignments and the examination.There are other challenges too, especially with respect to the majority of students who live in the rural areas and previously disadvantaged parts of South Africa where basic services such as the postal, electricity, Internet and online connectivity continue to be elusive.If there should be any doubt at all about students' interaction with their learning material, our view is that there is no way of knowing whether we are indeed assessing what we should be assessing in the first place.We do not regard ourselves as prophets of ODL doom.And while we recognise that there are pockets of excellence among some of our ODL students, we also acknowledge that there are pockets of our ODL students whose responses to assignment and examination questions leave much to be desired and suggest that their attitudes might not be changing as a result of enrolling in ODL programmes.On the positive side though, and consistent with the above exposition of the principles of quality and assessment, we are systematically gathering, analysing and interpreting the evidence; and using the resulting information to document, explain, and improve the performance of our students.

Conclusion
In this article we have sketched various conceptions of assessment.We have acknowledged that there is general consensus among advocates of assessment that the purpose of assessment should be mainly to improve the quality of teaching and learning.And given that teaching and learning involve both lecturers and students, we suggested that assessment should be a collaborative endeavour between lecturers and students in order to produce the best performance in teaching and learning.We argued that assessment should involve making lecturers and students' expectations explicit and public; it should lead to a common understanding on appropriate criteria and standards by which quality learning will be assessed; it should lead to gathering, analysis and interpretation of evidence to determine how well performance matches expectations and standards.In this regard assessors should draw on the information to document, explain, and improve teaching and learning performance.We raised concerns about the challenges of assessment in the ODL sector.We argued that while ODL is ideally suited for breaking the barriers of distance by providing access to higher education to students without the actual physical contact with their lecturers, there is the unintended consequence that quality assessment might be compromised.We argued often that lecturers have no way of knowing whether the assignments that the students submit are their own authored work or a product of assisted collaboration.This suspicion gains credence especially when students perform poorly in the final examination, which is their summative assessment.
Frazer offers what seems to us a useful understanding of the notion of quality.He suggests that quality should be about what students have learned-what they know, what they can do, and what their attitudes are, as a result of their interactions with their teachers, department and university.