Krieger, S H; Martinez, S A --- "Performance isn't everything: the importance of conceptual competence in outcome assessment of experiential learning" [2013] LegEdDig 3; (2013) 21(1) Legal Education Digest 8

• Performance isn’t everything: the importance of conceptual competence in outcome assessment of experiential learning

---

Performance isn’t everything: the importance of conceptual competence in outcome assessment of experiential learning

S H Krieger and S A Martinez

Clinical Law Review, Vol 19, 2012, pp 251-296

The American Bar Association (ABA) has recently signalled its intention to make a major change in its accreditation standards for law schools by moving away from the current focus on input measures – what students are taught in law school – to a focus on learning outcomes– what students learn in law school. In 2007, amidst a growing chorus calling for the incorporation of outcome measures into American legal education, the ABA’s Council of the Section of Legal Education and Admissions to the Bar (the ‘Council’) created a ‘Special Committee on Output Measures’ (the ‘Outcome Subcommittee’) and charged it with determining ‘whether and how [the Council] can use output measures, other than bar passage and job placement, in the accreditation process.’ After conducting a thorough review of legal education around the world, professions other than law, and American legal scholarship about outcomes,’ the Outcome Subcommittee’s report recommended a re-examination and reframing of the Standards for Approval of Law Schools (the ‘Standards’) ‘to reduce their reliance on input measures and instead adopt a greater and more overt reliance on outcome measures’.

Responding to the Outcome Subcommittee’s report, the ABA Standards Review Committee has proposed new approval criteria (the ‘Proposed Standards’) that heavily incorporate outcome measures. The latest version of the Proposed Standards would require each law school to articulate the outcomes it will seek to give its students. In conjunction with selecting outcomes, law schools would be required to design a curriculum to create those outcomes, assess student learning through a variety of methods, give students meaningful feedback, and regularly and continuously assess their success at achieving their student learning outcome goals.

Although the proposed changes to the accreditation requirements would generally afford law schools flexibility in determining their own learning outcomes, they also include several required outcome goals. One mandatory outcome measure that schools would be required to adopt is ‘competency as an entry-level practitioner’ in ‘a depth in and breadth of’ professional skills. This requirement would represent a significant change for experiential legal education: at present, law schools are required only to provide ‘substantial instruction’ in professional skills, with no competency requirement.

Much of the recent focus by the ABA – and legal scholarship on defining learning outcomes has been influenced by the Carnegie Foundation for the Advancement of Teaching’s 2007 monograph, Educating Lawyers: Preparation for the Practice of Law (the ‘Carnegie Report’ or the ‘Report’ or ‘Carnegie’), which argues that law schools should make a greater effort to prepare students for the professional practice of law. By focusing on performance as the measure of competence, Carnegie emphasises the importance to curriculum design and outcome assessment of protocols and techniques for performing particular lawyering tasks.

Expertise in practice involves thinking, reasoning, and decision-making processes. Therefore, this research indicates that to teach law students to handle their clients’ problems expertly, learning objectives should include the development of effective reasoning strategies to handle different and varied problems, not just the ability to perform in prescribed ways.

If Carnegie’s approach to learning outcomes prevails, all aspects of experiential legal education – from curriculum development to individual course design to outcome assessment – will be focused on training students for standard performance, not for reasoning in practice.

Carnegie asserts that traditional legal education has been narrowly focused on the cognitive apprenticeship, through the almost universal use of the ‘case-dialogue’ method of education, in which professors lead students through discussions of appellate cases to tease out legal analysis.

Carnegie argues that a narrow focus on legal analysis has a ‘corrosive effect’ on professional development because it marginalises practical and ethical aspects of lawyering. To remedy the current imbalance in legal education, the Carnegie Report argues that legal education should establish a new signature pedagogy, one that incorporates training in cognitive, practical, and ethical aspects of law through ‘educational experiences oriented toward preparation for practice’.

An important aspect of Carnegie’s model is its proposal for increased emphasis on the practical apprenticeship in legal education. In the traditional legal apprenticeship, as Carnegie describes it, a significant amount of learning came informally from observation and imitation of an expert. The expert modelled performance in such a way that the learner could eventually imitate the performance, while the expert provided feedback to guide the learner in making the activity his or her own.

Carnegie acknowledges that the traditional apprenticeship, consisting of an extended, close, individual relationship designed to offer practical training, is not realistic in legal education today. Nonetheless, Carnegie purports to reformulate the apprenticeship model to incorporate the benefits of such a relationship into a modern educational setting. In its reformulation, Carnegie urges that the central elements of expert performance should be passed on in a more formal fashion. They should be studied, distilled, simplified, and taught to novices ‘in the form of rules, procedures, protocols and organising metaphors for approaching situations or problems.’ Carnegie refers to these devices as ‘scaffolds.’ A scaffold could be, for example, a particular interviewing procedure, a protocol for problem solving, a technique for negotiating a deal, or a method for drafting a contract. With its focus on techniques and procedures, Carnegie’s model is based on studying the performance of experts and teaching students to imitate that performance.

As an example of the concept of a scaffold for law students, Carnegie points to the textbook Lawyers as Counselors, which provides a framework for legal interviewing and counselling developed by several law professors.

One of Lawyers as Counsellors’ techniques that is touted by Carnegie is ‘T-funnel’ interviewing. Such an interview ‘is a pattern of information-seeking’ with two elements: open questions – the upper part of the T-induce clients to think freely about the problem and elicit information from the clients’ perspective. Closed questions – the lower part of the T – are used to focus narrowly on gathering information related to specific legal aspects of the client’s problem. The premise underlying this technique is that ‘thorough information gathering rests on a combination of open and closed questions’ in a particular sequence: start with an open question and gradually narrow the questioning.

For a teacher, using a scaffold in Carnegie’s model takes on a distinct form: the scaffold is introduced to students, who are given multiple opportunities to practice using it, and to receive feedback from the instructor on how they use it. By practicing the application of the T-funnel technique to interviewing, a student can accumulate enough experience with this scaffold to allow her to master its application. The T-funnel technique will then become part of the ‘toolkit’ that, according to Carnegie, the student must assemble to achieve ‘skilled performance’ – the goal of the practical apprenticeship.

In light of the potential impact of the Carnegie Report on selection and definition of outcomes in the wake of the ABA’s proposed reformulation and application of its Standards, it is important to scrutinise Carnegie’s recommendations rigorously rather than accepting them uncritically.

The crucial problem underlying Carnegie’s focus on performance assessment is that it does not rest on a sound theoretical or empirical foundation. Cognitive science findings challenge the expertise theory that Carnegie employs and suggest a different approach to the issue.

This alternative approach highlights the cognitive attributes of expert performance rather than the performance itself.

Carnegie’s theory of expertise is based entirely on the work of two brothers, Hubert and Stuart Dreyfus, trained respectively as a philosopher and an engineer. The Dreyfus brothers posit that expertise is simply a matter of pattern recognition. They call this ability to use patterns without cognitive rules ‘holistic similarity recognition’ and argue, ‘Normally, experts don’t solve problems and don’t make decisions; they do what normally works’.

With this theoretical outlook, the Dreyfuses assert that acquisition of this kind of expert intuition requires the novice to learn protocols and strategies for identifying the facts and features of a particular situation and performing in response to these facts.

These stages of development, the Dreyfuses claim, reflect an evolution from the abstract toward the concrete, ‘from the analytic behavioUr of the detached subject, consciously decomposing his environment into recognizable elements, and following abstract rules, to involved skilled behavioUr based on accumulation of concrete experiences and the unconscious recognition of new situations similar to whole remembered ones’. In the Dreyfuses’ own words, as students become experts, they act ‘arationally’. In other words, expert performance is essentially mindless.

Consistent with Dreyfus, Carnegie envisions that students should first learn rules, strategies, methods, and protocols to enable them to recognise patterns and perform in particular situations.

In sum, Carnegie contends that the focus of experiential education should be primarily on performance. The desired outcome of the educational process is the student’s ‘ability to judge that when a situation shows a certain pattern of elements, it is time to draw a particular conclusion ... [and to] act in a certain way to achieve the selected goal.’ From this perspective, the student’s action, rather than her reasoning process, has paramount importance.

Most cognitive scientists recognise the role that pattern recognition plays in expert performance. Nonetheless, they reject the notion that intuitive pattern recognition alone is determinative of expert performance.

Unlike driving a car or riding a bike, handling a legal problem in practice requires more than intuition based on pattern recognition. Lawyers must juggle, for example, the substantive legal doctrine, the procedural context, the particular facts of the situation, the client’s needs, and the cultural and social context. The Dreyfus theory may account for simple procedurally-oriented skills like asking for an adjournment or conducting an inquest to obtain a default judgment, but it does not address the kinds of complex decision making required in most lawyering.

Relying on the Dreyfus model of expertise, the Carnegie Report eschews the importance of ‘formal modes of thinking’ and instead emphasises ‘skilled human performance’ as the focus of lawyering skills education.

One attribute of expertise identified by cognitive scientists is the use of schemas, frameworks of representations that help experts engage quickly in complex reasoning. Our working memory is only capable of holding seven elements of information at a time and can only deal simultaneously with two or three items when processing information. In contrast, long-term memory can store large amounts of information and engage in complex cognitive processes.

Knowledge is stored in long-term memory in the form of schemas. Schemas are ‘ordered patterns of mental representations that encapsulate all our knowledge regarding specific objects, concepts, or events’.

In regard to the acquisition of expertise in a profession, researchers theorise that as a result of greater experience in a particular domain, experts reflexively use their well-developed schemas to filter out irrelevant data and focus on relevant information to derive solutions to problems.

In the field of law, the anchors for schemas are basic legal doctrine (e.g., contract, tort, property, evidence, and agency law; the rules of procedure; professional responsibility). As novice lawyers delve into practice, the doctrinal framework helps them to organise their experiences, and they begin to construct schemas. As they accumulate experiences, these schemas assist them in handling new and unfamiliar cases more effectively and efficiently.

Schema theory differs in important respects from the Dreyfuses’ model of holistic similarity recognition. While perception plays an important role in both theories, for the Dreyfuses, it is the basis for all expert decision making. In their model, experts recognise patterns based on their perceptions of the surface characteristics of a particular situation and their abilities to relate those characteristics to the specific contexts of prior experiences. Under schema theory, however, the expert practitioner unconsciously delves beneath these perceived characteristics. Her mental representations are organised around concepts from her domain knowledge, which she uses to discover all the relevant facts she encounters in the situation. Such practice has a cognitive as well as a perceptual basis, requiring the filtering out of irrelevant information and the focusing in on relevant facts.

Cognitive scientists also assert that the application of knowledge by an expert in handling ill-structured problems requires the simultaneous consideration of multiple concepts that are individually complex, a process called cognitive flexibility.

Following the Dreyfuses, one might think that instructors should adopt methods that simplify concepts, compartmentalise knowledge, and teach rules and strategies for performing particular skills. But cognitive flexibility theory shows that reductivism is only helpful in training novices in well-structured domains in which the solution to problems is simple and straightforward.

Adaptive expertise entails the ability to use standard strategies to efficiently solve problems in routine situations, and the ability to develop innovative strategies to solve problems in novel situations. The most effective experts have the metacognition to distinguish between these two types of situations and to judge when to be efficient and when to be innovative.

To foster adaptive expertise, researchers recommend that novices be exposed to a variety of problems with differing complexity but sharing a similar theoretical base. With appropriate feedback during this process, learners can begin to distinguish between routine problems and those that require more deliberate consideration, consultation with an expert, or collaboration with others.

Since cognitive processes play only a secondary role in expert practice, when viewed through the Carnegie-Dreyfus lens, performance becomes the primary focus of outcome assessment in experiential courses. Alternatively, following the research described in the prior section, we suggest that cognitive competence is an essential element of expertise. In our view, then, assessment must consider not only a learner’s performance, but also the cognitive processes revealed during that performance.

A review of the empirical research on this subject raises serious doubt about Carnegie’s unsupported claims regarding the benefits of performance assessment of students’ learning of lawyering skills. Indeed, one major study in the medical field suggests that in assessing clinical ability, cognitive competence may be at least as important as, if not more important than, performance.

Contrary to the assumption implicit in the Carnegie Report’s recommendations, performance assessment may not be the best measure of a student’s long-term lawyering ability. Instead, scores on tests of clinical reasoning and cognitive processes appear to be better predictors of acquisition of expertise and ability in practice.

Simply stated, it is impossible to peer into someone’s head, as ‘thinking cannot be observed by other people’.

To address this problem, one of the experimental methods now employed by medical educators to assess cognitive competence is the use of the ‘think-aloud’ interviewing methods employed in cognitive science studies of the reasoning process.

Using the medical field’s ‘think aloud’ technique as inspiration, we have designed an experimental assessment method intended to identify the different kinds of cognitive processes used by students as they solve problems in practice. Specifically, we give students in a clinical program a hypothetical problem that is representative of work they have experienced in the program, and record them as we ask them to talk it through. Our hypothesis is that by prodding students to talk about a problem without a filter, we will understand, as much as we possibly can, what they are thinking ‘in practice.’

We invited several clinic students to participate in the project, emphasising that they would not be graded.

In the actual interviews, the interviewer simply gave a written hypothetical to the students and asked them to think aloud through their responses as they reviewed the problem. The interviewer followed up as needed with open-ended questions to help the students express their thoughts, to make sure they continued to think aloud, and to encourage them to explore as fully as possible their reasoning processes as they were happening. The entire interview lasted approximately 15 minutes for each student.

The think-aloud interview, like other types of assessment, ‘offers two insights: one into the students, and one into the instruction’. Insights about students can be used to give meaningful feedback. Insights into the effectiveness of instruction can inform course design.

Contrary to Carnegie’s suggestion, there is no standard ‘toolkit’ of protocols and procedures possessed by all experts that can simply be taught to novices. Rather, through their experiences in practice and the effective feedback they receive, novices construct their own schemas for solving problems. The assessment of think-aloud interviews can help instructors identify areas in which they have assisted students in developing cognitive frameworks and those in which they have not been effective.

Assessment is only useful for guiding students to competency, however, if it accurately measures their abilities.

For example, in experiential legal education, a typical subject for assessment is the initial interview with a client, in which the lawyer meets the client for the first time to begin the representation. An assessment based on performance would evaluate elements such as the student’s greeting of the client, whether the student made the client feel comfortable, how effectively the student applied the T-funnel technique of asking open-ended questions and then narrowing them down, how the interview was organised, the appropriateness of the ‘small talk’ phase of the meeting, the cultural awareness and sensitivity of the student, the discovery of any important deadlines, the degree to which client and lawyer know what happens next, and setting the time for the next meeting.

In contrast, an assessment of the same interview focused on the student’s reasoning would look at very different elements, such as: What were the student’s goals in the interview? Why did the student ask a particular question at a particular time? What was the student thinking when the client said, ‘I just want justice’? How did the student attend to multiple, competing, interdependent factors such as the client’s goals, the relevant law, and the interests and resources of the other party in the case? What legal theories was the student considering while talking to the client?

Performance-based assessments capture only the surface elements of practice, rather than the deeper reasoning processes that are central to expert practice.

By focusing on the reasoning process, instructors can see whether the student is developing and applying schemas, demonstrating cognitive flexibility, or exhibiting the ability to identify novel situations. These are more significant elements of competency in practice than simple performance skills.

We fear, however, that the ABA and schools moving toward a focus on learning outcomes will rely too heavily on performance based assessment, given its relative ease of administration and the growing number of voices that support this form of assessment.

Only by focusing on essential elements of expertise, such as reasoning in practice, can assessments reveal students’ progress toward becoming experts themselves, and not merely their ability to imitate experts.