Legal Education Review
|
|
Home
| Databases
| WorldLII
| Search
| Feedback
Legal Education Review |
|
FRED KELLER GOES TO LAW SCHOOL
SHIRLEY L RAWSON* AND ALAN L TYREE*
It seems axiomatic among some law teachers that small group “Socratic” teaching in one form or another is a superior teaching method. 1 Unfortunately, when taught to small groups it is also an expensive teaching method. While new and/or well funded schools have been able to embrace it, others find that they cannot afford the method.
The Law School at the University of Sydney is one of the oldest in Australia. For many years, it has also been one of the worst funded. Our exploration of other teaching methods was driven by our dissatisfaction with the traditional lecture method of teaching, fuelled by these economic restraints.
Our explorations led us to consider the Keller Plan teaching method. Educational research suggested that the Plan could be at least as effective as intensive small group teaching, but available at a price that we could afford.
The Keller Plan, also known as a Personalised System of Instruction (PSI), is a teaching method which combines “mastery learning” with principles of reinforcement learning theory.
Since we find that few law teachers are familiar with these concepts, the first part of this paper will be an introduction to mastery learning and its particular refinement in the Keller Plan. We will then describe some of the research which compares the Keller Plan with other teaching models, and finish with a description of a particular implementation of the Keller Plan which we used in 1990 at the University of Sydney, in a class of 130 students.
Can we describe what it is that we expect our students to learn? If so, are there any reasons why they cannot or do not achieve the defined goal(s)? Are we teaching at a level which is appropriate for our students?
Mastery learning assumes that our students can learn the material which we teach, subject to rare exceptions. Under that assumption, the basic concepts of mastery learning involve only two essential features. 2 First, we tell our students what it is that we wish them to learn. Secondly, we provide them with the means to evaluate their own performance, that is, to determine if they have in fact learned the material. Both of these features require some elaboration.
Mastery learning requires that the desired student performance be stated precisely. Objectives such as “students should learn to be creative” are unacceptable because there is no objective measure to determine whether the objective has been met.
The objectives should be “behavioural” in the sense that they provide a specific performance measure. This measure can be used by the student (and the teacher) to determine if the student has achieved “mastery”.
“Mastery” does not mean that the student is a “master” of the subject matter. It means only that the student has met the performance standards of the particular course. It is possible to achieve “mastery” in a beginner’s course.
Behavioural objectives sometimes arouse controversy. The argument against them is that the most important products of education are incapable of being “measured”. How can we state the broad aims of a liberal education in behavioural terms? The requirement for behavioural objectives, it is argued, is opposed to these aims, for the focus is on that which can be measured rather than on what cannot.
While such arguments have a superficial attraction, we do not believe that they stand careful scrutiny. Can we seriously expect students or anyone else to be satisfied with vague aims? If the aims are not vague, then why can we not state them in a reasonably precise way? It is surely unsatisfactory for teachers to adopt a “I don’t know what it is, but I recognise it when I see it!” approach to teaching.
Sometimes the argument against behavioural objectives is the result of a misunderstanding. There is no reason for the objective to be simple or elementary. “State three major criticisms of the Australian Constitution and suggest remedies” is hardly a simple objective, yet it is clearly objective.
Nor do behavioural objectives exclude personal judgements or opinions. In the objective of the last paragraph, there is obvious room for opinion in the word “major”.
The purpose of the objectives is not to limit the educational process, but to define it. Students are entitled to know what the teacher expects. Mastery learning assumes that the teacher knows what is expected, and is skilled enough at the job to be able to state it.
“Mastery” is determined by examination of the student. The student is examined to see if he or she has achieved the stated objectives. Thus, the subject matter of the examination is the stated objectives; there is no room for examining objectives which are not previously stated.
The examination procedure is somewhat different from the methods which law teachers usually use. In mastery learning, we attempt to examine using “criterion referenced” examination methods. Most law teachers probably use a mix of criterion referenced and “norm referenced” methods. 4
Norm referenced methods attempt to place a student by reference to the performance of other students. There is clearly a large component of norm referenced examining procedure in most law school exams, for many faculties require a specified distribution of marks. It would be a rare school which did not call for explanation when one marker differed consistently from the local norm.
Norm referenced questions must discriminate among students, for a question which does not discriminate adds nothing to the testing process beyond shifting the average.
Criterion referenced testing, on the other hand, sets specific performance standards. Questions are measured against the performance requirements rather than by the profile of answers. A criterion referenced question may be satisfactory even though there is a near perfect answer rate, ie, even though it contributes little or nothing to the discrimination process.
Of course, it would be unusual for any test to be completely norm or completely criterion referenced. A law teacher will naturally pay heed to performance criteria when drafting questions, but, as noted above, will also be disturbed if there is a 100% pass rate on questions. In fact, the marking criteria in most examining probably shift subtly during marking in order to ensure that the final marks fit into some norm.
In the context of mastery learning, the aim is to key the test questions to the stated behaviour objectives. The test questions should be criterion referenced against the stated objectives. If the objective is one which we should expect most students to learn easily, and there will be some, then there is no objection to a question which 100% of the students pass on the first attempt.
It is clear that the concepts of mastery learning can be introduced into existing courses. One way of doing so is to give the students a statement of the behavioural objectives at the beginning of each week, and to test them at the end of the week.
In the standard mastery learning implementation, the weekly tests do not count as part of the final mark. Their sole purpose is to provide the student with a measure of their understanding of the material.
It will perhaps come as a surprise to many teachers to learn that adding mastery learning techniques results in a measurable improvement in student performance. 5 The credit for this is probably equally divided between teacher and student. The formulation of behavioural objectives provides a focus for both, and the examination procedure can make the student aware that he or she may not know the material as well as thought.
In the 1960s Fred Keller devised a teaching plan which is now known as the Keller Plan or the Personalised System of Instruction (PSI). Keller, a psychologist, integrated the ideas of mastery learning with those of reinforcement learning theory. 6
The Plan is based on several fundamental observations. First, if students are all expected to achieve mastery, then they cannot all be expected to do it in the same time. Secondly, smaller amounts of material are more digestible than larger amounts. This leads to the adoption of a “modularized” course. Thirdly, students will learn better if they are given frequent and immediate rewards. These rewards are, in the Keller Plan, instant feedback on tests and credited marks for success in each unit.
The following account given by Keller himself of the main features of his Plan shows very clearly the differences between this teaching model and those which are more traditional: 7
Keller and his associates implemented a PSI course at the then new University of Brasilia in 1964. The method was refined over the next few years, and Keller’s landmark paper was published in 1968. 8 There have been thousands of successful Keller Plan courses run throughout the world since that time. 9 We can find no reference in the literature to a Keller Man course in any law school. Before describing the implementation of the Keller Plan in Technology Law, we will discuss in more detail some of the particular features which Keller thought important.
Since students must proceed at their own pace in a PSI course, the traditional scheduled lecture or class must be abandoned. Lectures could still be the primary method of communication, but either they must be frequently repeated or reduced to audio or video tape.
Most Keller Plan courses have relied on written materials as the primary means of student/teacher communication. In a law course, this has the additional advantage of requiring students to learn in a manner similar to that which they must use for the rest of their professional life.
Unit materials are somewhat different from those which are usually reprinted for student use. Remember that there will be little opportunity for guided discussion.
In addition to the ordinary materials, there is a teacher written “study guide”. This must provide a “map” to the materials, as well as provide any up-to-date information or opinions. The teacher is free to write his or her analysis of any of the written materials. Most importantly, the study guide must also contain the detailed behavioural objectives of the unit.
If proctors are used, the teacher should also prepare detailed proctor guides. These contain general instructions to proctors concerning their role (see below) as well as specific instructions on marking the unit tests.
In the typical Keller Plan implementation, students are free to test at specified times during the week. There is no penalty for failure of a unit test. The test is marked immediately by the proctor, and the proctor may seek clarification from the student concerning answers given.
Tests are generally short (20 minutes seems a popular figure) and consist of a variety of question types. The Plan has nothing to say about test form other than that it must test all of the objectives and nothing but.
Because there is no penalty for failure, the teacher must prepare several equivalent but different tests for each unit. 10 We found test preparation to be one of the most demanding aspects of the Keller Plan.
The examining procedure used by Keller was labour intensive. It was clearly impossible for the teacher to mark all of the examinations. Indeed, Keller and others thought that it would be a waste of highly skilled talent to employ teachers in that capacity. The examiners were known as “proctors”.
Proctors administer the tests and provide reinforcement to student performance. Instant feedback is possible, and a small amount of tutoring is inevitable although proctors are not engaged as traditional tutors. Indeed, Keller cautioned against proctors acting as tutors. They are not there as teachers, and are not intended to impose their views of the materials on students.
Proctors provide important human contact in an otherwise “teacherless” course. Proctors also provide valuable information to the instructor or course convener who is responsible for the course. In a typical implementation, a proctor would be “responsible” for 15 to 20 students. Implementations of the Plan show considerable variety in the use of proctors. In some, students in the same course who had tested earlier were used as proctors. This was thought to provide a more intimate learning environment. In others, postgraduate students were employed in some instances with certain credit points for other courses being allocated for their participation. Another popular scheme was to use students who have completed the course in previous years.
The Keller Man calls for non-compulsory lectures and extra demonstrations. The purpose of these lectures is motivational only. The content is not examinable. The lectures help to combat the sense of isolation that may develop by providing an occasion for teachers and students to get together.
We scheduled a series of five guest lectures. The lecturers were all prominent in their area, and the area was always related to the course although not directly overlapping with any of the units. Attendance at these lectures was very poor, usually less than 25% of the class.
The initial Keller Plan did not include a final examination. However, in subsequent courses one was introduced. The reason for this seems a little vague given Keller’s concern about student fear of examination often being a cause of poor performance. One reason given later was that his colleagues were more likely to employ the Plan if there was some traditional method of assessment available as well as the progressive unit testing.
The Keller Plan is one of the few teaching methods which has a theoretical basis. For that reason alone, it attracted a great deal of attention in the late 1960s following the publication of “Goodbye Teacher …”. There are many papers devoted to a study of one or another aspect of the Plan. 11
The literature grew so large that it was possible to do a form of scientific analysis of the published results. The study, known as a “meta-analysis”, showed conclusively that the Keller Plan was at least as good as very intensive small group “Socratic” methods. 12
In comparison with traditional lecture methods, Kulik, et al were able to show that an average student could be expected to improve his or her final examination mark by 8–9%. 13 The study verified that students not only learned more, but (and this may be the same thing) they retained the knowledge better. Low-aptitude students benefited to the same degree as high-aptitute students, and re-testing after six months showed that the advantage of the Keller students increased. 14
Students believed that they worked harder under Keller Plan courses, although the evidence was unclear as to whether they actually worked harder. Given this belief, it is perhaps surprising that students liked the Keller Plan course better than the alternatives offered. 15
If the student perception of working harder is correct, it is perhaps not surprising that they learn more. They may like the method better for a number of reasons. In comparison with the “Socratic” methods, there is less stress. Students like the “go at your own pace” feature which allows them to schedule their own time. The method involves the student more than the traditional lecture. And, of course, they may like the Keller Plan precisely because they believe (correctly) that they learn more.
The Keller Plan may also be better than the’ Socratic method for students who are less vocal or less aggressive. Feminist writings have attacked the Socratic method for precisely this reason. 16
We believe that one of the primary advantages of a Keller Plan for a law course is that students learn primarily by reading. We found that our students’ ability to learn efficiently from reading increased dramatically as the course progressed. Learning from reading is obviously an important skill for lawyers, but one that is often neglected in favour of developing oral skills.
The research findings of Keller Plan studies convinced us that it was an alternative worth exploring. Our original plan was to implement a “standard” Keller Plan course, together with a research program which would allow us to replace proctors with examination by computer.
As our plans developed, we decided to omit the first stage of the program and proceed directly to examination by computer. This may have been a mistake, but at the time we rationalised the decision on the basis that we could not afford to continue a “traditional” Plan beyond the one year experiment. It seemed better to bite the bullet and proceed directly to the second stage.
There is one matter of terminology: most writing on the Keller Plan refers to “units”. In our school, the .word is used to refer to the amount of credit attached to a course. We use the word “module” to describe the “chunks” of material.
Technology Law was first offered as an optional course for later year students in 1989. It was taught by lectures and tutorials with an extensive research project and final examination. The course was divided into a number of separate areas covering a range of topics which were capable of forming distinct units.
The course was a mixture of “policy” and “black letter” subjects. In some parts, eg, computer crime, there were statutes and students were expected to learn “the law” as well as to undertake an exploration of the underlying policies (or lack thereof). In other subject areas, eg, electronic data interchange or trans-border data flows, there is was no specialised local law, and the course considered the application of the general law to the special problems together with various international initiatives. 17
There seemed to be no reason to change the general plan of the course, and the Keller Plan course covers broadly the same topics.
The Keller course consists of 15 modules. Ten of these cover computer related topics such as Computers and Computer Crime, Electronic Funds Transfers, Artificial Intelligence, Privacy and Data Protection, Freedom of Information. Five modules deal with various problems of biotechnology including Plant Variety Rights, Artificial Conception, Genetic Manipulation and Genetic Fingerprinting.
Each module is divided into two parts — the “Study Guide” and “Materials”. The study guide provides a detailed explanation of what is expected from students with a “procedures” section suggesting a path of study to follow. Objectives are set for each module and suggested issues are presented. The objectives and suggested issues are based on the materials which have been specially chosen to facilitate individual study.
The amount of materials has been kept within reasonable limits. We expected that students would take approximately 10–13 hours to read and prepare for testing. This conforms to Faculty of Law guidelines for course structuring. Other readings have been suggested but this is merely for interest and these readings do not form part of the examinable materials.
When a student decides that he or she has achieved “mastery” of the module, he or she attends the Faculty computer laboratory for testing. Tests average 10 .items, are unsupervised and generally require 20–30 minutes. There is no penalty for failure. A student may sit the test as many times as required to pass it. 18
Students required an average of 2.5 attempts to pass the tests. Once the test is passed a set mark is awarded –5 marks for each module.
Although proctors were not used, we tried to design the tests to provide the immediate feedback that the Keller Plan requires. In addition, the teacher tried to be available freely for consultation concerning difficulties with testing, materials and other concerns. This decreased any perceived isolation and has encouraged many students to assist the researchers in the development of improvements for future courses.
The assessment for Technology Law, 1990 was divided into two parts — module tests worth 5% for each module passed (a possible 75%) and a final examination (to cover all module materials) worth 25%.
To put this into perspective, Faculty guidelines call for no more than 20% of students to receive a “Distinction” or higher. A “Distinction” is a mark of 75 or more. We had a unusually large number of “Distinctions”. 19
Our only major departure from the traditional Keller Plan was the use of computer testing. Our hope was that the branching facilities of computers could be used to provide the instant feedback required by the Keller Plan. This would open the possibility of using the Keller Plan in distance education as well as reducing the continuing costs of the Plan.
We found little assistance in the educational literature for this aspect of the project. It is obvious that computers may be used to administer multiple choice question exams, and there is no doubt that the resulting reduction in marking and manual bookkeeping is most welcome.
The deficiencies of this are obvious. The reasoning process of the student can only be examined implicitly, yet it is this reasoning process that is often of most interest in examining law students. Examining reasoning by multiple choice questions requires the careful construction of choices in the question. Construction is slow and difficult.
We hypothesized that by using “tree” questions, we could better explore the reasoning of the student and, at the same time, reduce the construction time of questions. An example might clarify this.
One of the units concerns plant variety rights, and one of the specialised issues to be tested concerned the identity of the “breeder” of a plant variety. The question was a typical law exam short story which described a person, John, who worked for a university who cultivated a special form of tomato. If the cultivation was in the course of employment, then the university was the breeder, if not in the course of employment, then the employee was the breeder.
We wished to test that the student understood that either John or the University might be the breeder, and that the resolution of this issue depended upon whether the cultivation was in the course of employment.
The “tree” question begins by asking baldly “who is the breeder”. On the basis of the answer to that question, the next part of the question is a multiple choice question which looks at the possible reasons why the student might give the response actually given.
There is no doubt that this question could be cast as a single multiple choice question, but it would be substantially more difficult to construct. With the “tree” question, the examiner is not required to think of so many possibilities at each stage of the construction. It is easier to provide alternatives that do not give away the answer, for much of the question is hidden from the student at each stage of the process.
At the end of the tree, the student is informed that the question has been passed or failed. In either case, we usually gave some additional feedback. In the case of a failed question, we would indicate either the reason for the failure or a reference to particular materials which should be studied further.
The language used both for construction of the questions and for the administration of the entire examining program is LES, a language developed by Andrew Mowbray as part of the DataLex Project. 20 LES is available to educational institutions for educational purposes free of charge.
The computer testing aspect of the experiment had mixed success. It was completely successful in the sense that it worked. We were able to administer almost 5000 examinations to 130 students in a 15 week period. The entire system contained approximately 400 questions and, subject to the comments below, functioned flawlessly.
We aimed for three alternatives of each question, that is, three separate questions which were intended to test the same objective. An exam consisted (generally) of 10 questions. A question was selected at random from the pool (usually two or three) of related questions. This system has the advantage that there are a large number of possible examinations. It is also convenient to add questions to the pool as and when time allows. Although construction of exams was still a very demanding process, it seems that it was less so than in a traditional Keller Plan course.
The examination system was not, however, entirely satisfactory. Students were often frustrated by what they saw as inflexible structures and the impossibility of explaining their answer further. On some of the material, there were frequent complaints that the questions merely imposed the teacher’s views in situations where the issues were far from clear.
Some of the student complaints were justified. In many cases this could be explained by our lack of experience with criterion referenced testing, and with the principles of constructing multiple choice questions. This led, in some circumstances, to ambiguous questions or to questions which failed to address the more important issues. These problems will be overcome by teacher experience. 21
Some complaints were not justified. Students often read the screen quickly, believing that they had seen the question before whereas it was, in fact, a variation. Some frustration was the result of bright, highly motivated students “failing” exams for the first time in their academic career. The fact that the “failure” cost them nothing except a further exam did not offset the emotional trauma that they experienced.
We do not wish to give the wrong impression here. The examination system was certainly successful enough that we will use it again. Most students handled most questions in the way expected. Many of the problem questions are amenable to simple modifications.
There is one improvement in the examining system which we believe would still most or all of the student frustration. It is best illustrated by the student reaction to the final examination in Technology Law.
That examination was a 60 item true/false exam which allowed the students a two line explanation of their answer. The explanation was optional. If the student gave the “wrong” answer, it could be partially or completely salvaged by giving an acceptable reason. Both the teacher and the students found this to be a valuable modification to the T/F format. Several questions had ambiguities that the examiner had not detected during exam construction. There were a significant number of students who gained marks for these questions on the basis of explanations which showed that they had given a reasonable answer.
This system may be incorporated into the computer examination system, although current technology does not allow the complete computerization. The existing system is easily modified to accept short reasons, even though it cannot assess their validity. The plan would be to have a single “proctor” in the computer lab whose task it would be to examine reasons given when the problem is marked incorrect by the computer.
This system would also have a valuable psychological advantage. The students would feel that they were receiving more attention. It also has the advantage (from the student point of view) that attention by a human proctor could only improve their results, never diminish them.
Ideally, of course, the computer marking system would be extended to mark short answer questions. This would reduce the question construction time required by the teacher, as well as being more satisfactory for the students. The SAGES (Short Answer General Examination System) Project is addressing this problem, but it is too soon to report progress. Information may be obtained directly from the authors. 22
Procrastination
Every teacher has experienced the “procrastinating” student. The problem is exacerbated in a Keller Plan course. We had several students fail Technology Law because of a late start. Our particular problem was acute because of a very poor system of enrolment at the University of Sydney. We had no way of knowing who should have been completing module exams until late in the semester.
Assuming that the teacher can determine who should be in the class, the problem of procrastination can be dealt with simply by using automatic notification procedures. When a student’s record shows that he or she is dangerously behind schedule, the teacher should be notified automatically by the computer system.
On the other hand, the Keller Plan is “self-pacing”. There are arguments that the entire responsibility should be shifted to the student. We do not subscribe to that philosophy, but we accept that students should be allowed a certain leeway.
The Keller Plan is dangerous. It is easy to be tempted to develop a “complete” course, putting in every article that is relevant to the subject. This must be resisted at all costs. Remember that the students are expected to “master” all of the material. It is poor teaching to misjudge the amount of material that students should be expected to learn.
Our Faculty guidelines call for a total student commitment of 196 hours for a two unit course such as Technology Law. Since we had 15 modules, this meant that students should spend approximately 13 hours per module. Guidelines also indicate that students can read 20 pages per hour, but we believe that this is a gross exaggeration when the written materials are the primary source of teaching. We used a rough guideline of 10 pages per hour. Most of the modules contained about 100 pages of reading material, accompanied by four to six pages of study guide.
We find that many students complain about the “work load” in their courses. Complaints are made concerning the number of hours they must spend at the teaching institution, the amount of reading, the vast numbers of different topics and so on. Some of these aspects form part of a “hidden agenda” in many courses and students’ complaints could be seen as warranted. In a carefully organized Keller Plan course students will know in advance of the ACTUAL work load and may then decide whether such a course is for them.
Another concern that students have about their work load is that they are often prevented from organizing their time because not all of the materials or course outlines are available. In other words, they are happy to take on any size workload as long as they know the actual quantity. Once more, a Keller Plan course would address this problem. The entire course is available for perusal at the beginning of the semester. Since it is complete with detailed outlines and study guides as well as materials, students may plan ahead.
Students in Technology Law generally believed that they were working far more than in comparable courses. We believe that this perception was correct, although it is obviously difficult to obtain accurate hard data.
However, based on the log sheets which each student was required to keep, the actual work was well within the guidelines set by the Faculty. This can be illustrated by anecdote. Early in the course one student came to us in anger. She was attempting to do two modules per week. She complained bitterly that this was requiring 25 hours work, that this was wholly unacceptable and that it was clear that we did not care what happened to students. When we pointed out that this was almost exactly what was required by the Faculty guidelines, she responded that she did not work anywhere near the same number of hours for any other course.
This story, in much less dramatic form, was repeated to us by a number of students. Of course, in a traditional course, students often spend many hours immediately preceding the final examination. This may raise the overall workload to near Faculty guidelines, but is much less educationally effective.
Very little has been written on the time taken by teachers for the preparation of materials, research for lectures, preparation of examinations, class exercises and so on. Law School folklore usually allows two to three hours preparation for each lecture if the course is one which has been taught before. Our Faculty allows six hours preparation for a one hour class if the course is a new one.
There would appear to be little difference in preparing a Keller Plan course. The only real difference is that the whole course should be in place before the students enrol. Time must be taken to set teaching and learning objectives but this has always been the position. There is, in fact, the possibility that less time will be taken in actual preparation in a course which has clear objectives and where materials and outlines must comply with those objectives.
The exams require a lot of time to prepare the first time a course is offered. If traditional examination systems are used, then the literature recommends that there be three or four complete exams for each module. Computer administered tests could reduce the load, even if traditional multiple choice questions are used, since it is easier to “mix and match” questions to build a new examination.
We cannot over-emphasise the importance of having the entire course prepared and ready to go on day one. In spite of reading similar warnings, we had only 10 of the 15 modules completed at the beginning, the only serious mistake that we made in the course. Very sensibly, some students made a calculated decision to spend a disproportionate amount of time on Technology Law in the early weeks so as to free time later in the term for other courses.
It is simply unacceptable to have students ready to continue to the next module when either the materials or the examination is not ready. The teacher is placed under strain to get materials and examinations in place, and the result cannot but suffer in quality. Further, it simply cannot be good teaching when the teacher begins to resent and dislike the best students in the class!
This need to have an entire course in place before the beginning of a term means that there are transitional problems when first offering a Keller Plan course. Ideally, the teaching commitment should be reduced substantially in the preceding term. Once the course is in place, the demands are similar to a conventional course, except that the work being done in this term is for the course to be offered to the students next term. The Keller Plan imposes a planning burden on the teacher as well as on the student.
The answer to the question is clearly “No”. The teaching of verbal skills, for example, seems to be outside the scope of Keller Plan courses.
In fact, there have been no attempts by Keller fans to introduce PSI to an entire course. There will be inevitably aspects of all courses where a different teaching plan would be appropriate.
On the other hand, “Kellerization” of parts of most courses is possible and even desirable. Most courses have material which, for one reason or another, students learn at dramatically different rates. We can give several examples from our own experience.
The fundamentals of negotiable instruments seem difficult for some students, simple for others. Such material could be taught by a Keller Plan, the requirement being that the students complete this partial course as a prerequisite to attending the remainder.
Teaching computer based research methods presents a similar problem but for a different reason. Students simply come with a very wide diversity of computer experience. Don Rothwell of our Faculty has successfully implemented a Keller Plan short course for students in the computer based research component of our Legal Research and Writing program.
One of the concerns which appears to many critics is that where there is a student body without supervision there will be large scale collusion or even worse. Again, it will be the responsibility of the course designers to address these problems. Within reason, certain guidelines and codes of conduct should be incorporated to answer such criticisms.
Many of the problems which surround the question of isolated students seem to be fanciful in the context of a Keller Plan course. Teachers using the Plan have stated that they have been far more involved with their students than while giving lectures and tutorials on a regular basis.
One user of a Keller Plan states, “[T]hrough repeated evaluations and modifications this course has successfully evolved over a four-year period, teaching me much about self-study and the Keller Plan and teaching me significantly more about students that I ever learned from lecturing or giving conventional tutorials”. 23
We attempted, without significant success, to encourage the formation of formal discussion groups. However, discussions with students lead us to believe that there were very active informal discussion groups. Some students told us that there was more discussion of Technology Law material than of any other.
The most costly element of the traditional Keller Plan is the employment of proctors. Labour costs will often cripple a project like this. This was one motive for exploring the “proctorless” method of testing by computer at the University of Sydney.
There is no need for large college or university computer facilities. The LES software used at the University of Sydney will run on a wide variety of machines. We used a minicomputer with terminals. 24 This has the advantages of a central record keeping facility, security and simple updating of exams.
However, we were prepared to implement the examination system on personal computers in the event of “meltdown” of our main computer facility. There is no software problem in this, but there are certain record keeping and other administrative problems that require solution. 25 The solution adopted will depend upon the particular hardware available.
Other cost saving approaches to the “proctor problem” have been used. Advanced students in a course can be used as proctors for less advanced. Some reports have indicated that the “prestige” associated with being a proctor has been adequate reward. 26 We considered this approach, but we doubted that the solution could fit into a student culture which has been traditionally very competitive. 27
The major “political” problem is the distribution of grades. Keller himself warned that the Plan “… is not recommended for the use of any teacher who believes that, because of genetic or environmental factors, only a handful of his pupils can ever be deserving of an A or its equivalent”. 28
We anticipated correctly that this would be a problem. The Dean and Teaching Committee approved our assessment scheme in advance and on the understanding that our mark distribution might very well be skewed. The actual results were much better than we anticipated (see the section on statistics), but this just makes the political problem worse! There are several solutions to this problem, none entirely satisfactory. It is possible to modify the “no penalty for failure” rule. This would discourage students from using the exam system as a tutorial method. It would also dramatically increase student dissatisfaction with the examination system, particularly if the computer examination system is used.
Another solution is to reduce the marks given for completion of modules, mark the final exam ruthlessly and scale the final marks if necessary. There is little educational justification for this, but it is the most direct response to the problem. We intend to argue that successful completion of all modules is at least worthy of a credit, so that the Keller Plan course would still be outside Faculty guidelines.
One strategy that might be a solution is worthy of exploration. Students would not receive full marks for a module, but instead a percentage of full marks based on their performance in the module exam. 29 The difference from the first solution is that students could re-examine until they reached a level which they found satisfactory. A class of very ambitious students could still yield a grade distribution which would leave the Faculty foaming.
Any dramatically different teaching device is bound to meet with some disapproval from students and colleagues. We tried to minimize and defuse the adverse reactions by providing early and complete information on the Keller Plan. We prepared a small information booklet which was circulated to interested faculty members. At the beginning of the course, we gave a faculty seminar at which we outlined the main features of the Keller Plan. Although these measures did not succeed in stifling all opposition, we have been pleased with the general level of support and interest shown by both Faculty and students.
Although Faculty members were generally interested and supportive, there remained a few problems. There seems little way to counter these problems except to be aware of them and to provide information when they occur. Several teachers advised students that the marks received in a Keller Plan course would be “worthless” because of the assessment method. This is puzzling, since the Keller Plan students were subjected to an average of about 15 hours examination over the life of the course! The real problem here is the political one of too many students receiving too high marks.
Some Faculty members have indicated that they do not believe that Keller Plan teaching is really teaching, that somehow we are looking for methods to abdicate our responsibilities as lecturers. This is not a view that could be held by anyone who has actually worked on a Keller Plan! Again, we see no solution to this other than providing information. It is an attitude which we believe will disappear as the Keller Plan becomes a familiar part of teaching life.
These were small problems. We again wish to emphasise that general Faculty support was generous and welcome.
Student attitudes to the course were more directly measurable. Students who were enroled in the course were required to fill out evaluation forms. Many provided additional advice on the way in which the course might be improved.
The overwhelming tone of the student evaluation was very positive. They liked the entire concept of the Keller Plan course. They generally thought that they learned more than they would have from a comparable lectured course. They believed that they worked harder.
They were critical of the computer examination system. This aspect of their evaluation is discussed above. Some found that they missed the classroom interaction. A variety of reasons were given. Some simply missed the social activity that regular lectures provide. Some felt that lectures would have clarified difficult areas. Some mentioned that they gained confidence from a lecture which verified that the teacher understood the material in the same way that they themselves did!
On the other hand, nearly all students liked the self-pacing feature of the Keller Plan. In conversations with students, we have found that those who say that they miss the classroom interaction also say that, if they must choose, they would choose the self-pacing. Which illustrates what we all know. No method of teaching is perfect in all respects.
Many students enroled in the course under the impression that it would be an easy mark. When it was made clear during the first and only lecture that this impression was mistaken, many changed their enrolment immediately.
The magnitude of this phenomenon is not easy to identify precisely, but some measure may be gained from a consideration of the computer records. We established 192 accounts on the machine, which means that, at one time or another, 192 students indicated that they were enroled in Technology Law. Only 132 students ever attempted an exam. We do not know how to account for the missing 60. Some dropped out as described above, but others never even attended the opening lecture.
Of the 132 who attempted at least a single exam, 2 discontinued after a short period of time. Two more failed to begin the course in time to complete enough modules to pass.
Of the remainder, 116 completed all 15 modules, 92 then went on to do sufficiently well in the final examination to achieve a “high distinction”, a mark of 85 or better. Faculty guidelines indicate that 3% (plus or minus 2%) of a class should receive high distinctions. Although we believe that the marks received by the Technology Law students were fair, it is clear that some solution must be found to the problem of integration.
We believe that the Keller Plan deserves a permanent place in the Law School. It is an effective and affordable system of teaching. We believe that it is a superior method for some courses and a superior method for parts of all courses.
It is particularly well suited when the material is such that students can be expected to learn at dramatically different rates. This might be because of the nature of the material or because of the varying background of the students.
The Keller Plan also deserves serious consideration when there are problems with scheduling student time. For example, busy practitioners might prefer a Keller Plan for Continuing Legal Education courses.
In-house education programs could be Kellerized for the same reason. A Keller Plan course obviates the need to find a time when all students will be available.
Keller Plan courses using the computerised examination system seem ideal for distance education. The sad truth is that students in such programs often receive very shabby instruction. Although it is unlikely that any method of distance education will ever be entirely satisfactory, a good Keller Plan course would be a significant improvement over most existing schemes.
We gratefully acknowledge the support of the Law Foundation of New South Wales. Without that support, we could not have developed our particular implementation of the Keller Plan.
Discussions with David Boud of the University of New South Wales were invaluable during the early days. He made us aware of both the possibilities and the pitfalls of PSI.
We received much psychological support from our colleagues at the Law School. It is fair to say that many were, and are, sceptical of the feasibility of PSI. To their credit, however, those doubts were not transferred to the kind of opposition that occasionally occurs in schools.
We would also like to express our deep appreciation for the support which we received from the staff of the general office of the Law School. Tight printing schedules combine with nervous academics to form an explosive mixture which requires patience and tolerance. We inevitably received both.
Finally, we would like to thank the students of Technology Law 1990. They supported our efforts with enthusiasm. They tolerated our mistakes with patient good humour. If that patience and understanding sometimes took a form usually reserved for the extreme aged, infirm or otherwise bewildered, it was nonetheless appreciated.
[*] Faculty of Law, University of Sydney.
© 1991. (199B91) 2 Legal Educ Review 253.
[1] The “axiom” is being questioned by some teachers. For example, J Morgan, The Socratic Method: Silencing Cooperation (1989) 1 Legal Educ Rev 151.
[2] “Mastery learning” is the term used by BS Bloom, Human Characteristics and School Learning (New York: McGraw-Hill, 1976).
[3] For a fuller discussion of behavioural objectives, see JS Vargas, Writing Worthwhile Behavioural Objectives (New York: Harper Row, 1972). See also J Heywood, Assessment in Higher Education (New York: John Wiley & Sons, 1989).
[4] For a discussion of criterion-referenced and non-referenced examining procedures, see J Heywood, Assessment in Higher Education, 2nd edn (New York: Wiley, 1989). See also RL Ebel and DA Frisbie, Essentials of Educational Measurement, 4th edn (Englewood Cliffs: Prentice-Hall, 1986).
[5] CR Clark, TR Guskey and JS Benninga, The Effectiveness of Mastery Learning Strategies in Undergraduate Education Courses (1983) 76 J Educ Research 210.
[6] For a thorough introduction to the Keller Plan, see FS Keller, and JG Sherman, The Keller Plan Handbook (Menlo Park, California: WA Benjamin Inc, 1974).
[7] See FS Keller, Good-Bye Teacher … (1968) J of Applied Behaviour Analysis 79–89, 83.
[8] Id.
[9] For some typical examples, see PS Allen, Developing a Remedial Keller- Man Course (1978) 3 Higher Educ 213; PL Schwartz, A Controlled Trial of Teaching Clinical Biochemistry by a Keller Plan (1980) 55 J of Medical Educ 1013; W Ferguson, Personalized Instruction in Business Education (1981) 50 J of Experimental Educ 9.
[10] Keller & Sherman, supra note 6, at 78 recommend four separate tests for each unit. We were able to reduce this load somewhat using the computer examining system described below.
[11] For example, in PS Men supra note 9, self-instruction was emphasised. In PL Schwartz, supra note 9, from a class of 196 medical students 30 were randomly chosen in a biochemistry course to study by the Keller Plan and the rest were used as a control group. Schwartz concentrated on the final examination results of these groups in his paper.
[12] Kulik, Kulik and Cohen, A Meta-Analysis of Outcome Studies of Keller’s Personalized System of Instruction (1979) 34 American Psychologist 307.
[13] Id at 311,317.
[14] Id at 313,317.
[15] Id at 317–318.
[16] See Morgan, supra note 1, and the articles cited there.
[17] It may seem strange that there is no intellectual property component in such a course. It is strange, but is explained by a desire to avoid “overlap” with longer-established courses.
[18] Actually a different but equivalent test. See the discussion on examination by computer infra at 260.
[19] See infra at 268–269 for a discussion of distribution of grades.
[20] The DataLex Project is a pint research project managed by Alan Tyree of the University of Sydney, Graham Greenleaf of the University of New South Wales and Andrew Mowbray of the University of Technology, Sydney. The major activities of the Project have been in legal artificial intelligence and legal information retrieval.
[21] These are not problems which are unique to the computer examination system. By using human proctors, however, the problems can be caught at marking time, and so may be less serious.
[22] We expect to have preliminary results of the project by the time that this appears in print. The SAGES system uses a combination of parsing algorithms and statistical methods to classify an answer.
[23] Allen, supra note 9, at 213.
[24] The computer lab at the Law School has two Honeywell Superteam computers. These are connected to 28 IBM-compatible machines each with their own 30mb hard disk. The lab is used for teaching legal information retrieval using the DataLex AIRS software, for student word processing and a variety of other projects
[25] We are developing a PC based examination system which will hold questions in an encrypted form. Contact the authors for further information.
[26] For example, see PJ Stoward, Self-Instruction Through Reading: The Keller Plan (1976) 10 Medical Educ 316 at 325.
[27] Although we were able to introduce self and peer assessment of written essays: see SL Rawson and AL Tyree, Self and Peer Assessment In Legal Education [1989] LegEdRev 11; (1989) 1 Legal Educ Rev 135.
[28] Keller & Sherman, supra note 6 at 75.
[29] See D Munro, Quality Distance Education = Computer Based Feedback + Electronic Mail in J Barrett & J Hedberg, (eds) Using Computers Intelligently in Tertiary Education, Papers presented to the Australian Society for Computers in Learning, (Sydney, 29 November-3 December 1987) at 147–152.
AustLII:
Copyright Policy
|
Disclaimers
|
Privacy Policy
|
Feedback
URL: http://www.austlii.edu.au/au/journals/LegEdRev/1991/12.html