Burton, K; McNamara, J --- "Assessing re?ection skills in law using criterion-referenced assessment" [2010] LegEdDig 37; (2010) 18(3) Legal Education Digest 20

---

Assessing reflection skills in law using criterion-referenced assessment

K Burton & J McNamara

Legal Education Review, Vol 19, No. 1, 2009, pp171–188.

Universities can assist students to become reflective practitioners by encouraging them to engage in reflective practice as part of their education.

The ability to engage in reflective practice is an important skill for legal practitioners who are faced with the constant stressors of legal practice. A recent Australian study found a high incidence of depression and substance use amongst members of the Australian legal profession. Encouraging students to become reflective practitioners is one way of addressing this concern.

Reflective practice has been identified as a desirable capability of law graduates; however, despite this recognition of the importance of reflective practice, reflective skills (along with other generic skills, attitudes and values) are often not assessed in a traditional legal education. This is in part because law schools tend to adopt a ‘technical-rational’ style of education that focuses on analytical skills and which is not conducive to the development of reflective skills. The reluctance to rely on reflective assessment can also be attributed to a perception that the assessment of reflective practice is challenging.

This paper focuses on the assessment of reflective practice, an issue that has not been fully explored within legal education literature.

Reflection becomes critical reflection when it extends to evaluating what is being reflected upon. At its highest level, reflective practice involves students or practitioners engaging in this type of critical self-reflection so that they can unpack and question the assumptions and values which underlie their judgements and actions.

The term ‘reflection’ is generally used to refer to the thinking process whereas ‘reflective practice’ is a formal process, the outputs of which are evidence of reflection. Reflective practice may include a range of activities such as learning journals, portfolios, diaries, logs, reflective reports and notebooks.

The key themes that arise from the literature relating to reflective practice are that reflection requires purposeful thinking and contextualising of what is already known, relating learning to existing knowledge, values and beliefs, considering a range of solutions or options and developing one’s previous knowledge, values and beliefs. These themes should be incorporated in criteria for assessment of reflection in legal education.

Experiential learning is propounded by David Kolb who defines learning as the process whereby knowledge is created through the transformation of experience. The cycle of learning is fundamental to lifelong learning because it enables learners to learn how to learn. The learner may enter the cycle at any stage, but must follow the cycle sequentially. For example, the learner might enter the cycle at the abstract conceptualisation stage (by considering theoretical principles) and then undertake an experience which relates to these principles before reflecting upon the experience. Experiential learning may differ from reflection because reflection may simply involve an internal experience, whereas experiential learning necessarily involves a concrete experience upon which the reflection is based.

Experiential learning courses in law can help students adjust to their roles as professionals; become better problem-solvers; develop interpersonal and professional skills; and learn how to learn from experience. In order to fully appreciate the workings of the legal system, law students need to have some practical experience of it. Experiential learning can also generate passion and provide context for law students. It has also been suggested that experiential learning is the most effective way to teach practical legal skills. In particular, experiential learning is vital to developing communications skills, which are essential for law students.

Experiential learning situations for law students may be formal or informal and include court visits, advocacy exercises, role plays, mooting, prison visits, work experience at legal offices and formal internships.

The benefits of reflective practice in the context of experiential learning are many. Embedding reflective practice in a subject also improves a student’s ability to recognise the need for and implement change.

Reflective practice not only enhances students’ learning experiences, it is also an attribute that will assist them in becoming lifelong learners. Students who are able to engage in reflective practice are more likely to be able to accept and make sense of critical feedback received from their tutors, mentors and supervisors. They are also more likely to become reflective practitioners who are more able to learn from experience; cope with uncertainty and anxiety; exercise sound judgement; assess their own skills and abilities; develop career goals and plans and implement strategies to achieve those goals; accept critical feedback; and continue to improve skills and aptitudes.

While recognising the benefits of reflection, some assessors dismiss the idea of embedding and/or assessing reflective practice because it is complex and difficult. Some acknowledge the high importance of reflection, but choose not to summatively assess reflective practice at all because of the complexity. Such an approach undervalues reflection because students are unlikely to pay close attention to reflection unless it is assessed. If reflective practice is to be valued highly, particularly in the context of legal education where students are highly competitive and motivated by grades, it must be summatively assessed. Assessment also motivates students to focus their learning, provides students with feedback on their understanding, and achieves quality assurance in the sense that it confirms the learning outcomes.

However, in addition to the difficulty in assessing reflective practice, there is an argument that summative assessment may interfere with the genuineness of students’ reflections; it can make students ‘reluctant to express their innermost thoughts and feelings, seeing this as an ‘invasion of privacy’, and ... inclined to write what they think the assessor wants to read’. Writing in such an assessment-driven manner defeats the value of reflection and may cause students to articulate beliefs that are inconsistent with their actual thoughts and feelings. Such a perception may cause students to merely write positive comments about their strengths; when, in fact, weaknesses deserve the same, if not more, attention. It is suggested that this issue need not deter law teachers from assessing reflective practice. It can be addressed by explaining to students the value of reflective practice in addressing personal weaknesses and by drafting appropriate criteria that encourage students to address personal weaknesses in their reflection.

Not only is it desirable to summatively assess reflection, it is also possible to assess it in a way which is both rigorous and objective.

The first step in assessing reflection is to determine what is being assessed. In some cases, reflection is used as a means of enabling student learning; however, what is being assessed may be merely the product of that learning rather than the reflective process itself.

Sumsion questions the value to be placed on the process of reflecting compared to the product, but does not provide any advice for assessors on how to weight the two factors. While there is probably no single solution to this, it is suggested that an answer lies in the designated learning outcomes for the reflective task. Where the ability to engage in reflective practice is a desired attribute (as in the case of law students) then it is important to assess the reflective process and not just the outcome.

While journals and portfolios are promoted as traditional vehicles for assessing reflection, legal educators should steer away from using written communication as the sole method for demonstrating reflection. The importance of engaging in the process of reflection (rather than on the written product) should be emphasised to students and, accordingly, other methods of reflective assessment should be considered. Such alternative methods of assessment include videos, blogs, oral presentations, discussion forums and mind maps, some of which are not written forms of assessment.

Assessing reflective practice is difficult because only the person who does the reflection knows how much they have learned, and because it is challenging to determine the extent of learning utilising an ‘independent’ or ‘external’ standard. Further, emotion is integral to reflection, and this makes assessing reflective practice taxing. However, assessing reflective practice is not impossible, and it should not be abandoned as an illegitimate form of assessment merely because it is difficult. The key is to design objective criteria. These criteria should be provided to students in advance, a recommendation consistent with the literature on criterion-referenced assessment.

Various commentators have suggested criteria for the assessment of reflective practice. In the context of a reflective diary, Jenny Routledge et al refer to ‘reflective account’ and suggest that the marker focus upon: (1) a reflection on the initial statement of intent in the light of experience;

(2) how personal learning, values and theories have developed and changed; (3) insights and understandings gained or altered; (4) implications for future practice; and (5) future learning needs and strategies to facilitate these needs.

There is some circularity in these factors because the first consideration specifically refers to ‘reflection’. The remaining considerations however unpack reflection further and raise two main themes. The first theme is how the experience informs (and may change) future practice, which accords with the view of developing understanding evident in the conceptions of reflections, discussed above. The second theme is to improve weaknesses.

The assessment of reflection skills based solely on the criterion of written communication may be inaccurate. The relationship between quality of writing and quality of reflection skills is often blurred in such assessments. A student may have excellent communication skills but may not be able to engage in reflection, while another student may be able to engage in reflection but not be able to write about it in an appropriate manner. Consequently, assessors need to go beyond assessing the quality of written communication, and should focus specifically on the key themes of reflection identified above, such as: purposeful thinking and contextualising of what is already known; relating learning to existing knowledge values and beliefs; considering a range of solutions or options; and developing one’s knowledge, values and beliefs.

Wong et al distinguish between non-reflectors, reflectors and critical reflectors. The non-reflectors discuss what happened, use a descriptive and impersonal style of writing, make assumptions without testing them, do not appreciate contextual considerations, and present views without supporting them with literature or experience. Analysis is integral to critical reflection and descriptive writing demonstrates ‘poor reflective learning’. Descriptive writing reports what happened, and descriptive reflection reports not only what happened, but why it happened. In contrast to non-reflectors, reflectors make connections between fresh and previous knowledge, adapt prior knowledge to new scenarios, and are insightful. Critical reflectors go a step further than the reflectors and are able to validate assumptions made, demonstrate an awareness of the contextual considerations, use literature and previous knowledge to pursue different options, address issues by using a diverse range of perspectives and do not choose to simply follow practices because of habit.

The characteristics of reflectors and critical reflectors correlate to the key themes of reflection including purposeful thinking and contextualising what is already known, considering a range of solutions or options, and developing understanding. All of these themes are at the heart of reflection and have been incorporated in the rubric below.

Rather than clustering reflective writing into the three categories identified by Wong et al, an alternative approach is to use four levels of reflective writing as performance indicators: descriptive writing, descriptive reflection, dialogic reflection and critical reflection. Each level incrementally develops reflective writing, with critical reflection being the highest level and descriptive writing being the lowest level in the hierarchy. Descriptive writing is characterised by a description of events without any discussion of alternative viewpoints or analysis. Descriptive reflection goes beyond descriptive writing by providing a discussion of alternative perspectives, but it generally considers the issues from only one perspective. Dialogic reflection shows a progression from descriptive writing to analytical writing and reveals alternative ways of judging or explaining events, but may also show inconsistent reflections. Dialogic reflectors mull over their reasons and options internally. Critical reflection transcends these levels by including not only an analytical discussion, but also a ‘contextual awareness’; that is, an appreciation of historical, social and political perspectives and

underpinning assumptions. Thus, a critical reflector goes beyond their emotional past and personal perspective, and considers the issues in context.

Another possible factor to take into account in developing performance criteria are the three aspects of critical reflection: dispositional, contextual and experiential. Dispositional reflection is based on the student’s values and characteristics, and how they prioritise them. The importance of context to reflection has already been noted; in particular, gender and race may impact on the experience. Experiential reflection involves recounting the experience and solving inconsistencies between the reality and expectations.

Reflective learners take a deep or active learning approach by asking questions of the experience to develop their own understanding, whereas unreflective learners take the experience at face value and thus engage in surface or passive learning.

Evidence (LWB432) is a compulsory final year undergraduate law subject at the Queensland University of Technology, which builds upon foundational knowledge acquired in criminal law and develops advocacy skills that may be extended in the elective skills subject, Advocacy (LWB356).

In 2008, a compulsory court report was embedded and assessed in Evidence. In semester two 2008 and semester one 2009, it was formatively assessed. Students were referred to the Queensland Courts Daily Law List website and required to physically attend a Queensland court on one occasion to observe a real world criminal trial. They were also required to complete a court report using a template provided to them. The purpose of the exercise was to enhance student understanding of the stages in the trial process – that is, examination-in-chief, cross-examination and re-examination – and to observe counsel asking questions of witnesses and objecting to an opponent’s questions during the stages of the trial process.

Overall, student and staff feedback about the court report was that it ‘promoted real world learning’, was ‘authentic’, ‘a worthwhile process’ and ‘a positive, practical step’. However, student feedback also suggested that some students had not actually attended the courts and had allegedly fabricated the content of their court reports, using court transcripts available online. Other students commented that the court report involved considerable time and effort, and that it should be summatively assessed to reflect its workload.

In semester two 2009, the court report was made a summative item of assessment worth 20 per cent. Students are now required to reflect more fully on their observations. The purpose of the court report in its new format is to make students more familiar with attending court, and to encourage students to make connections between what they learn in the classroom and what actually happens in practice. Whilst in court, the students are required to complete a one-page (double-sided) court report template, which directs their attention to the role of the judge and counsel, competence and compellability of witnesses, special measures put in place to make the process less stressful for certain classes of witnesses, the order of proceedings and different types of evidence.

Students begin the cycle at ‘abstract conceptualisation’ when they are provided with the abstract concepts of evidence and the trial process via lectures, tutorials and readings, as well as an explanation of the criterion-referenced assessment rubric. Students engage in ‘active experimentation’ when they complete a trial process exercise where they plan questions that they would ask their witness in examination-in-chief, their opponent’s witness in cross-examination and their witness in re-examination, as well as how they would object to their opponent’s questions and/or answers where appropriate. Attendance at a criminal trial is the ‘concrete experience’, and the ‘reflective observations’ take place when the students observe the trial process and reflect on this in the court report. If students encounter any inconsistencies between what they observe in practice and what they learned in the classroom, they are encouraged to review their lecture notes and other resources to resolve these disparities, returning to ‘abstract conceptualisation’ and thus closing the experiential learning cycle.

The court report requires students to reflect on what they observed in the court room on a one- off occasion as compared to what they have learned in the classroom during the semester.

Written communication is implicit in the reflection skills criterion, and is particularly noticeable when the performance descriptor for excellent reflection skills is compared to poor reflection skills; that is, a critical reflector’s writing style is analytical, personal and contextual, and a non-reflector has a very descriptive and impersonal writing style.

Even if a student identifies a weakness in their understanding in the trial process, if they can also put forward a strategy for resolving their weakness, they may receive an excellent mark for reflection skills.

While the rubric has been designed specifically for the assessment of a reflective court report, it is suggested that it is transferrable to the assessment of other types of reflective practice both in law and in other disciplines. The rubric provided in this paper provides a useful framework for assessing reflection skills and will be a useful springboard for academics in law and in other disciplines seeking to embrace the assessment of reflective practice.