Florio, C M S; Hoffman, S J --- "Student perspectives on legal education: a longitudinal empirical evaluation" [2013] LegEdDig 13; (2013) 21(1) Legal Education Digest 47

• Student perspectives on legal education: a longitudinal empirical evaluation

---

Student perspectives on legal education: a longitudinal empirical evaluation

C M S Florio and S J Hoffman

Journal of Legal Education, Vol 62, 2012, pp 162-187

There is currently a dearth of data on the law school experience. The vast majority of knowledge on the subject is drawn from anecdotal sources and the few empirical instruments currently in use, such as the Law School Survey of Student Engagement.

Empirical evaluation is distinguished from other types of research by its reliance on collected data and its use of the scientific method of inquiry. Evaluations include both quantitative and qualitative investigations, often on the perceived or actual impact of enacted or proposed policies.

One of the most interesting strands of research on student experiences of law school explores the role played by gender. An early study at Berkeley Law School included a comprehensive questionnaire, with both quantitative (four-point Likert-type responses) and qualitative elements. This study found lower participation rates, lower academic performance and greater dissatisfaction with academic performance and themselves among women. A majority of women reported mixed or negative feelings about their law school experience.

Further empirical studies of gender’s effect on student experiences were undertaken at the University of Pennsylvania Law School and Brooklyn Law School in the mid-1990s. The first study at Penn incorporated data on academic performance, self-reported survey data and written narratives. The study reported gender-based disparities in grades and honours, lower in-class participation rates and higher anxiety among female students. Differences were attributed in part to alienation of female students through the employment of the Socratic Method of instruction, and their relative discomfort in approaching male professors. Ultimately, the study called for exploration of alternatives to the Socratic Method for first-year instruction (including smaller classes), investigation of alternatives to the adversarial model of problem solving in legal instruction (such as negotiation) and further investigation of how students learn best in a law school environment.

This ground breaking study was later repeated at Brooklyn Law, where researchers observed that many of the reforms recommended by the Penn study already had been implemented. Several features at Brooklyn were posited that set it apart from peer institutions, including Penn. It had a large proportion of female faculty (45 per cent of all faculty and 37 per cent of tenured and tenure-track faculty). It had adopted a small-group approach to legal research and writing instruction. Finally, it used a range of non-adversarial teaching methods. Despite these institutional differences and the fact that female students at Brooklyn Law achieved similar grades, honours and levels of out-of-class faculty contact as their male peers, women there still exhibited significantly different attitudes toward classroom participation, in-class discomfort and psychological issues. Ultimately, the Brooklyn study concluded that while the Penn recommendations may be effective in narrowing the gender divide, they were not sufficient to fully address the gendered experience of law school.

A limited number of studies have assessed law students specifically, concluding that they experience significant declines in well-being over the course of their degree. These studies suggest strong stress issues associated with the first year of law school, and reinforce notions of a gender divide, particularly with respect to coping with stress.

With the exception of the psychology-based studies mentioned above, most articles addressing the first-year law school experience are anecdotal, discussing deep-seated issues of anxiety, control, alienation and declining satisfaction.

In addition to the academic literature, the Law School Survey of Student Engagement has provided a relatively new source of comprehensive information on the legal education experience in North America. It suggests that first-year students are intensely engaged in their education but that by third year they are relatively disengaged. Results also suggest professor availability is related to students’ overall experience and that male students receive more professor feedback than female students.

To help fill the empirical lacuna, a multi-year survey project was launched at the University of Toronto’s Faculty of Law by the student government in collaboration with the dean and administration to systematically capture and track students’ perceptions of their law school experience. A total of nine surveys were conducted with data collected, analysed and disseminated on a continuous basis. Results were put in context and grounded with other quantitative and qualitative data available from various sources, including the Law School Survey of Student Engagement, town halls, focus groups and student leadership meetings.

In addition to soliciting basic demographic information on gender, prior education, and mature student status, questions targeted students’ satisfaction with academic and non-academic aspects of their experience. Students’ priorities for law school were also solicited. A five-point Likert scale was used to capture students’ responses for most questions, with responses ranging from ‘strongly disagree,’ ‘disagree,’ ‘neither agree nor disagree,’ ‘agree’ to ‘strongly agree.’ Several open-ended questions solicited written responses.

The questionnaire was pilot tested during the 2007-2008 academic year on all 172 first-year law students. Surveys were conducted in October 2007 in the middle of first term (27 of 172 students participated for a 74 per cent response rate), January 2008 after first-term exams but before grades were released (93 of 172 participated for a 54 per cent response rate), and May 2008 after final exams but before grades were available (119 of 172 participated for a 69 per cent response rate). Responses were anonymous without any identifiable characteristics to protect student privacy and confidentiality.

Building on the successful pilot, empirical data collection was expanded and systematised the following year to capture the perceptions of all students. A new questionnaire was developed for upper-year students and the original questionnaire was revised for a new cohort of first-year students based on feedback from student leaders, faculty and external consultants.

Surveys were conducted in the fall and spring of the 2008-2009 academic year of both first-year students (149 of 192 students participated in the fall 2008 survey for a 78 per cent response rate; 107 of 192 participated in spring 2009 for a 56 per cent response rate) and upper-year students (322 of 427 participated in fall 2008 for a 76 per cent response rate; 145 of 427 participated in spring 2009 for a 34 per cent response rate). Data collection was repeated in fall 2009 for first-year students (144 of 196 participated for a 73 per cent response rate) and in winter 2010 for upper-year students (209 of 428 participated for a 49 per cent response rate).

All questionnaires after the pilot test were administered online with continued anonymity to ensure confidentiality and encourage participation. Invitations to participate were sent by e-mail to all law students who were not on academic exchange from other law schools.

All statistical tests were conducted using the full range of responses provided by the five-point Likert scale, whose intervals between values were assumed to be equal. Each survey iteration was treated as a separate cross-sectional study and was not aggregated into panel data.

Multiple linear regressions were conducted to identify the various factors that influenced students’ overall happiness with the law school, sense of community and transition into law school. A binary logistic regression was also conducted to uncover the factors that influenced whether students have confidence in their ability to secure careers of choice upon graduation.

Different variables were chosen for these tests by balancing their relevance and the availability of data for them across as many surveys as possible.

All statistical analyses were conducted using SPSS 16 for Windows.

Quantitative data was supplemented and put in context using the written feedback provided by open-ended questions on the surveys. Where issues of interest or concern were identified, student leaders partnered with the faculty in town halls, focus groups and consultative meetings to gather more information. Areas in which these additional data-gathering efforts were undertaken included curricular reform, pedagogy, assessment, mooting, advocacy training, financial aid and career services.

General results of the surveys were continuously shared with students via e-mail and the law student newspaper. Students were invited to provide their thoughts on the results and suggestions for policy improvements.

In all but one of the surveys, the majority of respondents were female (ranging from 48 per cent to 61 per cent across the nine surveys). Educational backgrounds varied, with a minority of respondents reporting undergraduate degrees in sciences (21 per cent to 23 per cent) or graduate degrees in any area (18 per cent to 28 per cent). A minority reported being ‘mature students,’ defined as students admitted to law school after five or more years of non-academic experience (6 per cent to 11 per cent). One-tenth of students in the upper-year surveys identified as having transferred from other law schools (9 per cent to 10 per cent).

Students in all years were generally satisfied with their academic experience at law school (ranging from 79 per cent to 91 per cent across the nine surveys), and upper-year students were particularly satisfied with the quality of teaching in their professor-led classes (85 per cent to 92 per cent). While only half of students reported feeling a sense of community (42 per cent to 59 per cent), the vast majority were still happy with their decision to go to law school in general (91 per cent to 95 per cent) and to the University of Toronto in particular (84 per cent to 96 per cent). A majority of respondents were also confident in their ability to enter their career of choice upon graduation (61 per cent to 74 per cent), despite the global financial crisis of 2008 (which would have affected results in four of the nine surveys), lower levels of upper-year satisfaction with workshops offered through the Career Development Office (42 per cent to 52 per cent) and low satisfaction among upper-year students with the availability of international internships (21 per cent to 34 per cent). A third of students were satisfied with the advice and guidance provided by the financial aid office (30 per cent to 44 per cent) and half reported satisfaction with opportunities to raise concerns with faculty (41 per cent to 53 per cent).

Among first-year students, there was widespread satisfaction with the support provided by the admissions office after their acceptance (ranging from 72 per cent to 80 per cent across the nine surveys) and with the division of the first-year class into two separate sections (70 per cent to 75 per cent). Satisfaction varied with the structure of the first-year curriculum (49 per cent to 80 per cent), the overall assessment framework in first year (32 per cent to 57 per cent), and level of engagement in large lectures (51 per cent to 81 per cent). A large majority of students were satisfied with the teaching methods used by their professors in first year (62 per cent to 81 per cent).

For the students of the graduating classes of 2010 and 2011, some statistically significant changes were observed over the course of the first year of studies (i.e., between the first fall survey and the final spring survey). For the class of 2010, one-way ANOVA tests revealed significant negative declines in students’ overall academic satisfaction and their satisfaction with the structure of the curriculum, the teaching methods employed by their professors and the level of engagement in lecture classes. The class of 2011 also experienced statistically significant declines in their satisfaction with the overall first-year evaluation framework, structure of the curriculum and the quality of student life.

The multiple linear regression addressing first-year law students’ impressions of how well their transition had gone from previous academic experiences explained approximately 27 per cent of the variance. Significant positive predictors of students’ satisfaction with their transition included gender and satisfaction with the structure of the first-year curriculum, the level of engagement in first-year lectures, and the support of the admissions office after they were admitted into the program.

Having an undergraduate degree in science was a statistically significant negative predictor and the most impactful of all factors assessed. A student’s prior graduate studies, mature student status, parental income and spousal income status were not statistically significant, though the presence of spousal income was close to being a statistically strong negative predictor. Other factors that were not found to be significant included a student’s opinion on whether the division of the class into two sections was beneficial as well as satisfaction with the quality of student life, social opportunities with peers and opportunities to raise concerns with faculty.

The results of the foregoing analysis support interesting inferences about the law student experience, not all of which fit neatly within the existing academic literature. While the analysis finds evidence to support a decline in general student satisfaction over the first year, gender does not seem to have the influential role it is currently accorded in legal education discourse.

These surveys found data to support the theory that first-year law students’ satisfaction decreases significantly over the course of their initial year of studies. While the graduating class of 2010 experienced this decline in the areas of academic satisfaction, teaching methods and engagement in lectures, the class of 2011 experienced similar statistically significant overall decreases in satisfaction with the evaluation framework and the quality of student life.

Given the academic literature, gender was expected to play a significant role in students’ transition to law school and this expectation was confirmed. Based on anecdotal evidence, another factor expected to play a major role in the academic transition was a student’s undergraduate preparation. This study found that students with science undergraduate degrees were less likely to report satisfaction with their transition. Law schools may want to consider offering targeted transition and preparatory support for students who studied technical subjects prior to matriculation.

Many other findings, while intuitively reasonable, were not expected. For example, mature student status had no effect on students’ satisfaction with their transition. Surprisingly, no significant determining role was found for a student’s satisfaction with either the quality of life at the University of Toronto or their social opportunities with peers, suggesting that the academic transition may well be independent of a student’s social adjustment to this new learning environment.

This study did not find the sort of gender-based experience of law school that has been the predominant focus of much of the empirical legal education literature. While gender had a role to play in students’ satisfaction with their transition, it did not affect their perceived sense of community, confidence in their ability to enter their career of choice upon graduation or happiness with their decision to attend law school at the University of Toronto.

A number of theories about why gender does not play a significant role in the context of this study may be suggested. Both the broader academic landscape and law schools in particular have undergone dramatic change in the last thirty years. While early studies exploring gender were conducted at a time when women made up only a minority of law students, the student body at the University of Toronto’s Faculty of Law is now mostly female (ranging from 50 per cent to 58 per cent across graduating classes). For many years, the school has been led by its first female dean. This sort of demographic and leadership change may account for a large part of the equalisation in student experiences along gender lines – even as female faculty are still in the minority and comprise similar proportions to those reported in the Brooklyn Law study. Also similar to the institutional setting at Brooklyn, Toronto does not employ the Socratic Method and students take one of their first-year classes in a small group, methods previously reported to help mitigate the gender divide.

Across the statistical tests run on the data, students’ satisfaction with quality of life at the law school emerged as a consistent predictor of satisfaction in other areas, including their decision to attend the University of Toronto’s Faculty of Law, their sense of community and confidence in their ability to enter careers of their choosing.

Developing an understanding of the determinants of law school satisfaction is crucial if legal educators and administrators are to be able to assess and adapt their own programs to better educate their students. Gathering data on the relative weight and influence of diverse factors that contribute to student satisfaction will also aid institutions in conducting the sort of cost-benefit analysis that has become essential to administrative decision-making.

Past data-gathering projects, though insightful in the context of their own particular institutional structures and the era in which they were conducted, are of limited value in attempting to inform current policy and pedagogical issues in legal education.

In this context, the process outlined in this study illustrates the potential role for recent, elaborated and continually updated data collection and analysis in identifying and addressing institution-specific issues and concerns. The study’s engagement of multiple stakeholder groups, including student government, administration, and the student body at large provided a strong base from which positive institutional changes could be suggested and implemented. The surveys have assisted in guiding the mandates of several committees at the University of Toronto’s Faculty of Law, all of which have incorporated survey feedback into their ongoing efforts to improve the overall law student experience.

Other law schools may benefit from collecting multi-year data, targeting questions to their own key institutional attributes, pedagogical methods and suspected determinants of student satisfaction. By implementing such measures, law schools will be better equipped to identify areas of concern and methods of improving the student experience.