Augustine-Adams, K --- "Playing the Ultimatum Game with Grades: Gender Confidence, and Performance in Public International Law" [2008] LegEdDig 23; (2008) 16(2) Legal Education Digest 29

• Playing the ultimatum game with grades: gender, confidence, and performance in public international law

---

Playing the ultimatum game with grades: gender, confidence, and performance in public international law

K Augustine-Adams

57 J Legal Educ 3, 2007, pp 375–390

Late in the semester, my syllabus for Public International Law skipped back towards the beginning of the book to discuss pacific means of international dispute resolution. The difficulty lay not in convincing my students that most international disputes are resolved through negotiation, but in helping them understand both the possibilities and complexities of those negotiations. Their energy and commitment levels had ebbed as the end of the semester and graduation approached. I needed something to grab their attention, something beyond the pop quizzes I threatened, they hated, and had surprisingly little motivational power. I had just read about the Ultimatum Game in the Harvard Law Bulletin. Should I try it in class? Should I use money? Perhaps grades?

In the Ultimatum Game two strangers share USD$100. One divides the money; the other accepts or rejects the division, with the caveat that rejection means neither receives any money. The Ultimatum Game explores these questions: What is rational? What is self-interested? The game does not explicitly explore how confident participants are in their choices or how they assess their skills, but confidence and perceived skill may be relevant to rationality and self-interest.

I decided that in-class use of the Ultimatum Game or some variant using money and grades could generate significant discussion and a reasonably realistic approximation of international negotiations involving limited resources. Playing the Ultimatum Game with money is par for the course; playing with grades is not. Given that students value their grades highly, I hoped that negotiating them would bring an immediacy and concreteness to the theoretical problems of negotiation. Grades are also a relatively inexpensive tool, particularly in comparison with the amount of money likely necessary to motivate my students to the same degree. Subject to challenges posed by certain ABA rules, playing the Ultimatum Game with grades in my class was a cheap, effective tool to explicate and explore rationality, self-interest, context, and the myriad other issues that international negotiations raise.

The data I collected as part of the exercise also took me beyond my initial concern with international negotiations and law to reveal interesting connections among gender, rationality, confidence, and reality. After all was said and done, the men in the class negotiated a better deal for themselves than did the women, but the women’s negotiated grades more accurately reflected their actual exam performance. Measuring negotiated grades against law school GPAs, the men in the class were not particularly overconfident. On the other hand, when measuring negotiated grades against actual exam grades in the class, the men in the class were overconfident in projecting their actual performance. The women were not particularly underconfident. This data replicates the findings of other research, although the psychological and economic scholarship as a whole is mixed on the question of men’s and women’s confidence and self-assessment of skills relative to actual performance.

The data with respect to the confidence levels of law students specifically is scant. While much research has been conducted on women’s law school experiences and general self-confidence, that research has not usually linked perceptions of confidence with actual ability or measured performance. We need more research that expressly evaluates law students’ levels of confidence in light of measured performance such as grades. Playing the Ultimatum Game with grades in my Public International Law class begins that research.

I walked into class and held up a ten dollar bill. I explained the rules — one student divides, one student accepts or rejects, rejection means no one gets any money — and chose two student participants. Fairness and standard rationality prevailed in thirty seconds or less.

I changed the rules and set the ten dollar bill on a table in front of the class. It was theirs, anyone’s, for the taking. No one took it. No one proposed dividing it up evenly. No one seemed to care. Ten dollars wasn’t enough to make the exercise sufficiently interesting or challenging given the constraints of social decorum and acceptable classroom behavior.

Grades were a different story. ‘If you don’t care about the money, let’s negotiate your grades in this class instead,’ I said. Three previously inattentive students sat up. ‘I insist on only two rules, one procedural and one substantive: Everyone in class has to agree on the grade allocation and the median grade in the course has to be a 3.3, in compliance with the law school’s mandatory median rule.’ Two more students looked up from their computers. ‘With fifteen students in the class, eight is the median number. Eight of you get a 3.3 or below and eight of you get a 3.3 or above. You all decide who and how.’

They sat quietly, intently, for a moment before a clear consensus emerged on the attractiveness of a clean split between 3.3 and 4.0. Half plus one of the students would get a 3.3, the rest a 4.0.

Almost immediately, two visiting students volunteered to take a 3.3. Whatever numerical grade they got as visitors would show up on their home transcripts as a pass, provided it was at least a C. A cross-registered Masters of Public Policy student jumped on the wagon: ‘This is my last semester. I already have a job. The grade won’t matter and if it gets us out of a final, I’m all for it.’ The lone, foreign LLM student agreed as well: ‘I’m just here for fun. U.S. law school is educational dessert. I’ll take a 3.3.’ Four out of the necessary eight students had agreed to a 3.3.

With four 3.3s remaining to reach the mandatory median, several students changed strategies and proffered justifications for assigning grades to others: Who needed the 4.0 most? Who already had a job or clerkship lined up and could afford to take a 3.3? Did the pregnant student need a 4.0 if she planned to stay home with her child rather than seeking paid employment? Would a 3.3 as opposed to a 4.0 significantly change someone’s class ranking or bump someone from the top 20 per cent into the top 30 per cent, or from the top half into the bottom half of the class? Those to whom the 3.3s might be assigned objected: ‘Just because I have a job doesn’t mean I can afford to take a 3.3.’ ‘I’m getting a job as soon as I have this baby. Even if I were going to be at home, I wouldn’t take a 3.3.’

Two students, the first regularly enrolled JD students, started to think of their self-interest more broadly. ‘If it really means we won’t have to take a final, we’ll take a 3.3,’ they said. ‘I think the extra study time will make a difference in our other classes.’ With six out of the necessary eight students agreeing to take a 3.3, the exercise stalled. The visiting students who had committed early to take 3.3s, and thus felt their contribution to the negotiation complete, left the classroom to buy ice cream with the ten dollar bill.

Other 3.3 students urged their uncommitted classmates forward. ‘If you take a 3.3 in this class, you really will have more time to study for finals in other classes. Measure the potential difference in your grade here against your potential gain in other courses.’ ‘Even if you think you could do better than a 3.3 on the final in this class, do you really think you could get a 4.0?’ Finally, another student agreed to a 3.3, although somewhat reluctantly. The grades were negotiated to everyone’s apparent agreement, if not complete enthusiasm, and in compliance with the mandatory 3.3 median.

Since the exercise, I have reflected on the negotiation process and outcome; what I observed regarding gender, reality, confidence, and self-interest; and what my students learned about international negotiation. Even as the negotiation unfolded, two aspects intrigued me: the apparently gendered process and silence as a negotiation strategy.

On the first point, although just under half of the regularly enrolled JD students in the class were men, the only regularly enrolled JD students who chose to take a 3.3 were women. The four non- regularly enrolled JD students — visiting, LLM, and MPP students — were evenly divided between men and women. Thus, the negotiated grades gave six women and two men 3.3s, while two women and five men received 4.0s. Were the women sacrificing their grades for the good of the class or for social reasons, or was some other motivation at work? Did all the male JD students think they deserved a 4.0 or just not a 3.3? How, if at all, did gender dynamics and expectations affect how students negotiated their grades? Why didn’t some of the students condition their acceptance of a 3.3 on a grade spread above the median to soften the effect of the 4.0s on their cumulative GPAs and class rank?

On the second point, the best strategy for getting a 4.0 was simply to keep quiet. Several of the 4.0 students said barely a word during the negotiation, despite their otherwise active participation in class discussions. Others participated only minimally. Silence was a rational strategy for negotiating the highest grade, although it was discussion and open communication that I was trying to promote with the exercise. In some ways, refusing to engage in the process was a substantially successful strategy. During the negotiation itself, no one commented that certain students were basically silent or that their silence might impede resolution of the grade issue. After the negotiation, however, one silent student told me that speaking meant choosing a 3.3. Another expressed a sense that the negotiation was ‘weird’ because ‘[n]o one could offer anything more than taking a 3.3 on themselves or trying to convince others ... so the best strategy to get the highest grade (if that was your main interest) was to take a position and stand by it as stubbornly as possible.’

With respect to rationality and self-interest, fairness and unfairness, I learned how much information and context matter in any version of the Ultimatum Game. I explored the law school and ABA rules governing grading. Having recently served as Variance Committee chair, I was virtually certain when I made the promise that no law school rule prohibited my entering negotiated grades. A review of the policies and procedures manual supported my recollection. The ABA rules were another matter. Interpretation 1 of Standard 303 regarding academic standards and achievements provides that ‘[s]cholastic achievement of students shall be evaluated by examinations of suitable length and complexity, papers, projects, or by assessment of performances of students in the role of lawyers.’ Reluctantly, and under a volley of student e-mails arguing to the contrary, I concluded that negotiating grades did not constitute either an evaluation or an assessment of student performance as Interpretation 303-1 required.

Even after I announced my conclusions, students put substantial time and energy into arguing, mostly via e-mail, how and why I could forego a final examination despite the ABA rule. This student- initiated extension of the exercise, the negotiation itself, into non-class time was exactly the sort of engagement that I had hoped to encourage by playing the Ultimatum Game, even if I didn’t anticipate the exact form that engagement would take. Playing with grades successfully motivated them to think deeply about substantive as well as procedural issues involved in negotiations. Despite my students’ best efforts at negotiating no final, I remained unconvinced that there was a way completely around ABA Interpretation 303-1. They argued that they had indeed performed ‘in the role of lawyers’ in the negotiation and thus a grade based on the negotiation met the requirements of Interpretation 303-1. I did not see any real connection between what I perceived to be the strength of individual performance in the negotiation itself and the negotiated grade, even if students had acted as lawyers might have in the process. Still, I wanted to enter the negotiated grades if I could.

Somewhat sheepishly, I told an associate dean what I had done — the negotiation, the promise — and provided the ABA rule, hoping he might think of some creative argument to blunt its force. He didn’t. He did provide information and context that demonstrated the rationality of student behaviour during the negotiation. Using cumulative law school GPA as the benchmark, each regularly enrolled JD student in the class, whether in the 3.3 group or in the 4.0 group, male or female, made a rational choice regarding his or her grade.

Without the background information regarding each student’s cumulative GPA or some other criteria, there was little opportunity for students (or for me) to judge accurately during the exercise itself whether or not someone was acting unfairly and then to punish that behaviour, as the Ultimatum Game suggests participants might. Each student did, however, know what his or her own cumulative GPA was and could guess, whether accurately or not, where he or she ranked in comparison with other students in the class. Differential access to information matters in judging fairness. In the Harvard Law Bulletin version of the Ultimatum Game, the variables that could affect perceptions of fair and unfair divisions of the USD$100 were limited. The situation and circumstances were sterile: strangers interacted once; they both had the same information; there was no chance of retribution or reputational harm; no context suggested one participant needed, merited, or deserved more money than the other.

All of these contextual factors — differential access to information; ongoing interaction; merit, deservedness, need, and the variable meaning of resources; limited resources themselves — are potentially present in international negotiations as well. Negotiating grades approximated an international negotiation to a degree that a hypothetical problem with no specific and immediate effect on my students’ lives seemed unlikely to do. They learned that context matters. To put an academic label on that international context, history matters. They observed that a relatively positive outcome for all is possible, even when individuals or nations act in their own self-interest. They experienced external constraints, realising that some actors, myself in the classroom or treaty negotiators in the international realm, may not be able to deliver on the agreed-upon solution, making continued discussion and compromise necessary. They learned that continued discussion and compromise happened in this case through extended e-mail conversations after the in-class exercise and with respect to the structure of the final examination. My students also learned that judging what is fair and just is not always an easy or transparent proposition but that negotiation can work.

What about their grades? With broad institutional interests in mind, I played by ABA Interpretation 303-1 rather than entering the negotiated grades. While I evaluated my students’ academic achievement through a written essay final, graded blindly, we carried the negotiation process over into deciding how the exam would be structured and what credit I would give for the negotiation exercise. I agreed to shorten the exam to two questions in two hours rather than the usual three questions in three hours. Similarly, I agreed that the first question would be an issue spotter, and the second a broader policy question drawn from a list of three that I provided beforehand. Providing relatively specific information about the second question was designed to focus their study and save them time, given that time savings was one factor that motivated some students in the negotiation exercise.

We also agreed that credit for the negotiation exercise would consist of a .2 grade bump-up, awarded after I had curved the exam scores around the mandatory median of 3.3. Because I decided I couldn’t enter the negotiated grades and remain in compliance with ABA Standard 301-1, the .2 bump-up was essentially a consolation prize. The class readily agreed to the bump-up even after I explained that not all students would receive the full benefit. For example, the median student would be stuck with a 3.3. Likewise, because of the forced median, students receiving a 3.2 on the exam could only be bumped up .1. Students scoring a 4.0 or 3.9 on the exam would similarly be limited in the benefit they could receive. Despite the risk of being the median student or a student who received only a .1 benefit, the students preferred the bump-up to no grade benefit from the negotiation exercise.

Of course, when negotiating their grades, students would only have had information regarding their own GPAs, not the GPAs of other students, and certainly not the grade they would receive on the final. Still, their expectation of how they might perform on this specific final exam, independent of their overall GPA, may be relevant to the grade they were willing to negotiate. Albeit post hoc, the exam grade helps clarify beneficiaries and losers in this version of the Ultimatum Game.

Using the negotiated grade as the standard, every man negotiated a grade that exceeded his exam grade, whether he was in the 4.0 group or the 3.3 group. The difference between the negotiated grade and the exam grade ranged from .1 to .9, with an average of .57. On the exam, two men out-performed their law school GPAs, one by .73 and the other by .17; three men underperformed their law school GPAs by between .19 and .30.

Again using the negotiated grade as the standard, four women — one in the 4.0 group and three in the 3.3 group — negotiated grades that exceeded their exam grades. For these four women, the difference between the negotiated grade and the exam grade ranged from .3 to 1.2. Two women received exam scores that exactly matched their negotiated scores of 4.0 and 3.3. Two women in the

3.3 group negotiated grades lower than their exam grades by .1 and .2, respectively. Thus, for all eight women, the difference between negotiated grades and the exam grades ranged from -.2 to 1.2, with an average difference of .24. Among the women, the four receiving the highest exam grades out-performed their law school GPAs by between .55 and .04, with an average of .30. The two receiving the lowest exam grades underperformed their law school GPAs by .46 and .85, with an average of .65.

Although the small size of the class makes it impossible to treat this analysis as statistically significant, the data on gender are intriguing and present questions for further research regarding male and female law students’ confidence and self-assessment of ability. In my class, all the men negotiated a grade that exceeded both their law school GPA (where available) and their actual performance on the exam. Some of the women negotiated grades that exceeded their law school GPA and their actual performance on the exam, but just as many negotiated the exam grade they actually got or a grade lower than their exam grade. If the negotiated grade is an indication of how students thought they would perform relative to their classmates if they had to take an exam, men were over- confident. On the other hand, if the regularly enrolled JD students used their own law school GPAs to guess at their rank relative to other students in the class when negotiating who got a 4.0 and who a 3.3, men were not particularly overconfident. If the negotiated grades had been distributed by ranking law school GPAs, one man in the 4.0 group would have traded spots with one woman in the 3.3 group, assuming the four non-regularly enrolled JD students took 3.3s as they did in the exercise.

At the same time, the data from this experience (as well as common sense) suggest that cumulative law school GPA is not a good predictor of exam performance in any single law school class. Of the regularly enrolled JD students in my class, five (three women and two men) out-performed their law school GPAs; five (two women and three men) underperformed their law school GPAs; and one, a woman, scored quite close to her cumulative law school GPA, within .04. As a whole, the men in the class negotiated a better deal for themselves than did the women, but more of the women’s negotiated grades accurately reflected — and at times even underpredicted — their actual exam performance. What was most striking in my class was not the women’s lack of confidence, but the men’s confidence: Every man negotiated a grade that ended up higher than both his law school GPA and his actual exam grade.

Legal education would benefit from studies that more explicitly link male and female law students’ relative levels of confidence with their measured performance. Perhaps female law students are underconfident; if so, they may underestimate both their abilities and their opportunities. On the other hand, female law students may simply assess more realistically the vagaries of law school grading and their own performance. Perhaps male law students are overconfident; if so, that overconfidence may also pose certain risks. Risks of overconfidence for law students may include a failure to prepare adequately for class or for the bar or a failure to recognise academic, social, or emotional difficulties before they escalate. Overconfident lawyers may not serve their clients’ best interests.

Playing the Ultimatum Game in my Public International Law class proved an invigorating pedagogical tool. It worked in part because students care immensely about grades and in part because I could make the negotiation exercise concrete and immediately relevant to their lives through those grades. Whether a different group of students — say, a concentration of students in the top 10 per cent or a class without visiting, MPP, and LLM students — would negotiate a similar grade distribution or even be able to reach a result at all is an open question. Mr. Ward, who during the exercise challenged my intentions to enter the negotiated grades, wrote me later to describe an exercise in an actual negotiation class where the students failed to resolve how grades would be distributed. While it was not clear that the parameters of that class exercise approximated playing the Ultimatum Game with grades, Mr. Ward found it significant that in Public International Law, ‘each student weighed the options and found value in a compromise that everyone agreed on.’

Unfortunately, given the limits of ABA Interpretation 303-1, the Ultimatum Game is not a tool I can use again in class in the same way. Maybe I’ll just have to ante up the USD$100 that worked to explore self-interest and rationality, fairness and unfairness, in the divide and accept or reject version of the game described in the Harvard Law Bulletin. But, as playing the Ultimatum Game with grades so clearly demonstrated, the resource to be divided matters, as does the context in which that division takes place. I expect that playing with money, even substantially more than the USD$10 with which I started or even USD$100, would produce results different from playing with grades. Using grades provided insight into confidence and perceived skill levels — attributes that are relevant to the practice of law as well as to self-interest and rationality — in ways money seems unlikely to do.

Despite the difficulties associated with ABA Interpretation 303-1, it should be fairly easy to develop a research strategy that assesses the relationship between law students’ confidence, their perception of their abilities, and their performance in law school. One way would be to survey students regarding their anticipated grades shortly after the examination period ends but before grades are distributed. Waiting until the examination period has ended rather than surveying students immediately after a particular exam would allow some distance from the intensity of the examination experience itself.

Survey questions could include whether a student assesses her own performance on a particular exam above or below the median; how the student evaluates her performance relative to others; what specific grade the student thinks she will receive on the exam; and what grade the student predicts she will receive in the course itself if factors other than the examination affect the final grade. With survey results in hand, the researcher could then link each student’s perceptions to the actual grades the student receives. Knowing whether male law students are overconfident or female law students are underconfident regarding their legal and academic abilities can help law schools address the risks associated with each.