Burns, K; Hutchinson, T --- "The impact of 'empirical facts' on legal scholarship and legal research training" [2010] LegEdDig 36; (2010) 18(3) Legal Education Digest 16

---

The impact of ‘empirical facts’ on legal scholarship and legal research training

K Burns & T Hutchinson

The Law Teacher, Vol 43 No. 2, 2009, pp153–178

Traditionally lawyers have been trained within a ‘doctrinal’ research methodology framework. There are existing rules of evidence in all jurisdictions allowing for a formal use of empirical data within the doctrinal framework. However these existing rules of evidence do not appear to adequately cater for the wide variety of ways in which empirical facts are utilised in judicial decisions. Increasingly, empirical fact assumptions and sometimes social science material are being subsumed within judgments. The way this material finds its way into judges’ decisions appears to rest primarily upon judicial discretion. Social science material relevant to empirical fact assumptions is not always (or even often) adequately acknowledged by judges. The recognition of the judicial use of empirical facts as part of judicial reasoning raises the need for new approaches to legal research and legal research training based in the social sciences. It suggests that lawyers need better training in non- doctrinal methodologies.

We define ‘empirical facts’ in this article as assertions of facts about society, the world and human behaviour which are hypothetically able to be proved by social science or empirical methodologies. They are assertions used as part of the judicial reasoning process. They may be used in a wide variety of ways by judges in their reasoning. They may be used to set background context, in a rhetorical way to support arguments of legal principle, to assist in the determination or interpretation of adjudicative facts, or as arguments of policy or consequence used in the development of law. Statements of empirical fact sometimes merge into statements of legal or social values, for example statements that refer to enduring community values such as the value of human life.

As Paddy Hillyard has pointed out, ‘Parliament, government, businesses and NGOs’ all appreciate the importance of ‘evidence-based research to inform the development of law, the administration of justice, and the practice of law’. It is therefore not surprising that a close textual examination of a variety of court judgments demonstrates that judges use empirical facts when they encounter gaps in knowledge.

The Burns study considered 11 negligence cases handled by the High Court of Australia in 2003. Burns found 325 statements of social facts in the relevant judgments. They were used to interpret adjudicative facts, as general context statements, as statements of consequence of liability and as mixed statements of social fact and value (for example the social value of human life). The vast majority of social fact statements made by judges were unsourced and only three social fact statements were sourced to a form of social science or empirical evidence.

The use of this form of empirical fact material in judgments in the United States has also been well documented. Many empirical fact statements are made by judges implicitly and without any empirical support. However, judges do sometimes explicitly reference empirical or social science material in their judgments. It is less common in the United Kingdom and Australia; nevertheless there are examples of the use of social science material in judgments.

What are the implications for legal research scholarship of this expanding body of social science literature? It is important that these methodologies and information are integrated into legal discourse. Doctrinal research has been the dominant influence in legal scholarship during the 19^th and 20^th centuries. However, limiting legal scholarship and research training to traditional doctrinal analysis has obvious limitations when lawyers (and judges) are being confronted with the need for and the relevance of results of empirical and interdisciplinary scholarship.

Traditionally law has been viewed as a closed system. What do we mean by this? In terms of legal research scholarship and research methodologies it has meant that lawyers have looked at the law in isolation. The sources of law have been the primary materials, the doctrine of the law – the case law and legislation. The research carried out has largely been confined to an analysis of legal doctrine. Thus doctrinal research is the established traditional territory of the lawyer-researcher. As a result, where legal research has been taught in the law schools the methodology taught has been doctrinal research. In some cases doctrinal legal research has not even been taught explicitly. We can define doctrinal research as: ‘Research which provides a systematic exposition of the rules governing a particular legal category, analyses the relationship between rules, explains areas of difficulty and, perhaps, predicts future developments’.

What is evident from this study on the use of empirical facts in the courts is that lawyers need to look at the law from a much broader angle than has been done previously. This is a quite concrete example of how the law does not work within a vacuum. Therefore, as researchers, lawyers need to be totally cognisant of the parameters of empirically based knowledge and research methodologies.

More extensive training needs to be offered in fundamental research. This is ‘Research designed to secure a deeper understanding of law as a social phenomenon, including research on the historical, philosophical, linguistic, economic, social or political implications of law’. This very important category was highlighted in the Canadian Arthurs Report on legal research in 1983 but totally overlooked by the Australian Pearce Committee review in 1987, and it is this category which is becoming more prevalent in current research agendas. Fundamental research, which can include empirical and social science models, needs to be part of the graduating lawyers’ research skills and attributes. Fundamental research encourages an interdisciplinary perspective and use of methodologies borrowed from the social sciences to study the law in operation.

There is a growing empirical law movement in the United States at present. Empirical work and the interface of law and social science is a continuing tradition in the United States, dating back to the Realist movement in the 1930s and 1940s. That movement was keen to highlight the differences between ‘law on the books’ and ‘law in action’.

Robert Ellickson, in a recent citation analysis of trends in US legal scholarship, found that ‘number crunching’ is also rising in law journals. Thomas Miles and Cass Sunstein are calling this a new intellectual movement: ‘The relevant studies have produced a New Legal Realism movement – an effort to understand the sources of judicial decisions on the basis of testable hypotheses and large data sets’.

The New Realists are aware that the movement has ‘jurisprudential implications’, but this is certainly not the focus of their work. Much of their research focuses on links between judicial behaviour, gender and politics.

The New Legal Realism is ‘Seeking to develop a rigorous, genuinely interdisciplinary approach to the empirical study of law’.

Apart from the established forums provided by the Association of Law and Society, recent examples include the recently formed Society for Empirical Legal Studies (SELS), and the Journal of Empirical Legal Studies (JELS) established in 2004. The first Annual Conference on Empirical Legal Studies (CELS) was held at the University of Texas in 2006, and there is a popular Empirical Legal Scholarship (ELS) blog. More recently, Elizabeth Chambliss reports on the establishment of Empirical Research Centres in several American law schools including the Center for Empirical Research in Law at Washington University, the Empirical Research Group at the University of California and the Empirical Legal Colloquium Series at Northwestern University School of Law. Law schools have been ranked on this basis.

ELS’s contributing disciplines include psychology, economics, sociology, anthropology, political science as well as law. The methodologies appear to be ‘more quantitative than qualitative and more contemporary than historical’. Certainly many of the scholars seem more intent on examining the US legal process rather than addressing the issue of infusion of the results of empirical work into the legal process itself, that is, into the determination of the law.

In Canada, Roderick Macdonald writing in 2003 states that ‘Published research by law teachers is still overwhelmingly doctrinal and oriented to the professional tasks of planning, dispute avoidance and dispute resolution’. However, Shanahan’s 2006 survey of legal academics demonstrates that legal academic researchers are using non-doctrinal methodologies to some extent.

This issue is being discussed in the United Kingdom. Anthony Bradney stated in 1998: ‘The academic doctrinal project which has dominated United Kingdom university law schools for most of their history ... is now entering its final death throes’.

The empirical studies movement is not as strong in Australia. The Australasian Law Teachers Association (ALTA) has no empirical legal studies interest group. There is a Law and Social Justice Interest Group and a Legal Research Communications Group which has a focus on research methodologies including the promotion of empirical approaches. However, the Law and Society Association of Australia and New Zealand which aims ‘to promote and foster scholarship broadly focusing on the interactions and intersections between law and society’ has a growing profile and maintains links with national and international socio-legal associations.

It is timely to consider how we as legal educators might inculcate these skills in our graduates

– and especially in our academic track higher degree research students. At present, greater use is being made by legal scholars of empirical methods. Academics need to ensure that the methods they are using result in ‘good’ empirical research. They need to ensure that the standards are high. There is an onus on legal academics to lead by example – to demonstrate academic leadership by joining

interdisciplinary groups and demonstrating an openness to learning and working with empirical methodologies.

Over the past decade there has been recognition that the law cannot be confined to a ‘black letter’ box. There has also been a move towards some relatively ‘safe’ research methods extensions including research into the philosophy underlying legal rules (theoretical research), research into the reform of legal rules (law reform research), and research into the policy behind legal rules (policy research). These extended research methods along with some more fashionable extensions such as comparative research, the use of case studies, and citation analysis are all helpful in arming lawyers with more extensive information on what is going on in court cases.

Internationalisation and the advent of transnational legal contexts (especially in the number of international students entering the Australian law faculties) has popularised the comparative law methodology. Citation analysis is being used to measure how many times a particular researcher is cited, and which journals tend to be most influential judged by the number of times articles published in the journals are cited, and to evaluate ‘the influence of other disciplines (such as economics) on legal scholarship, the sources which influence judges when they draft judgments and the influence of particular articles, scholars and legal journals’. Much of the empirical work has involved the ‘systematic and quantitative analysis of judicial decision making’. Content analysis has been used to reveal the role of empirical facts in judicial reasoning. Cases are read and particular features (for example categories of comment) are coded. The process involves a method of ensuring reliability and validity in the coding be established, before the data is analysed. The method combines a disciplined focus on legal subject matter with an assumption that other researchers should be able to replicate the results of the research.

Content analysis is used to evaluate ‘the influence of other disciplines (such as economics) on legal scholarship, the sources which influence judges when they draft judgments and the influence of particular articles, scholars and legal journals’.

When we look at the law more widely and when we look at how the law actually works, it is obvious that law teachers need to revise and widen their views on what they are teaching their students in terms of research methods.

Training in traditional doctrinal analysis methodologies does not equip students to deal well with empirical facts. In 1992, Twining noted that the ‘use of statistical arguments in court and in other contexts is developing fast in the United States and is likely to spread to other parts of the common law world well before the year 2000’. He termed this the ‘new evidence scholarship’. Twining also made the point that ‘in my experience most lawyers are innumerate and most law students are terrified of figures’. He noted that Oliver Wendell Holmes had argued a century ago that lawyers need to master economics and statistics. However, his predictions that ‘Holmes’ dictum will be incorporated in standard conceptions of competence by the year 2000’ has not eventuated. It is worthwhile noting too that Twining thought it ‘extremely unlikely’ that competence in empirical research could be developed by ‘quick fixes of CLE’.

Empirical research enhances lawyers’ ability to understand the implications and effects of the law on society. They are able to use statistics freely available and gathered by governmental organisations to enhance their views on the law’s operational aspects. This strategy has very definite advantages for unskilled lawyers as it saves time and ensures accuracy and that public verification of the data has already occurred.

It is easy to speculate on the reasons for reluctance to move beyond the familiar doctrinal methods. Legal researchers still need to know how to integrate the information effectively. There are often constraints in that the data collected may be too general and not necessarily that required to critique a legal or social issue effectively. Often lawyers do not have the skills to use publicly available datasets effectively. There is a lack of training in the undergraduate degree for non- doctrinal methods of research. Lawyers perceive they have insufficient expertise in order to judge empirical studies. It requires more time to undertake empirical work than doctrinal work. It costs

Empirical research is more inconvenient. The results are often uncertain and certainly not predetermined. Elementary errors can be fatal to the outcomes. Even a simple survey entails precision in sampling, wording of the questions, coding of the questionnaire for easy entry of returned data, conduct of speedy ethical consent processes, provision for privacy with returned forms and follow-up communication with those being surveyed. In addition, there is often a requirement to work as part of a group – and often an interdisciplinary team. This requires extra time and commitment. And once the research is completed and the reports written, there can be uncertainty in regard to where to publish – whether in a legal journal or an interdisciplinary one. The method and citation style for writing up the research will be different for each. The level and depth of analysis will be different. In all, therefore, using non-doctrinal methodologies equates to less control over the process and outcomes than doctrinal work.

It is now more than ever important to acknowledge that empirical research methodologies are relevant to the practice and research of law in the 21 ^t century.

Research training must include a broader non-doctrinal methodology component. There is a wealth of general social data that is used to some extent by the legislature but that also impinges on legal decision-making. There is a need to introduce students to the existence and nature of interdisciplinary research – the extensive work of anthropologists, sociologists, criminologists, economists and sociologists that impinge on the law. Law schools need to introduce a wider range of research methodologies into their research training particularly those based in the social sciences. Students must be aware of the basic principles of social investigation, where to source publicly available information, and how to critique empirical research from the perspective of validity and reliability. They must be able to distinguish valid empirical research from anecdotal evidence. This means that empirical methodologies must be introduced into the law curriculum so that law students can deal with empirical facts in a knowledgeable fashion.

In doing this, legal academics have a role in ensuring that students are aware that there are various components in the judicial reasoning process – including the evidence and legal principle, but also facts based in the judge’s views and information based in the social sciences.

Legal education has embraced skills in the last decade under the rubric of graduate attributes. However, each Australian law school curriculum must include the subject areas identified by the Priestley Committee in 1992. There are strong views from many legal educators that the Priestley 11, which is skewed towards substantive rather than skills-based instruction, is ‘a significant constraint on re-formulating Australian legal education in ways that are modern and relevant’.

Educational theory suggests two approaches that are relevant to any attempt to enhance non-doctrinal research training. In the first place, legal educators are advocating Cognitive Apprenticeship as espoused in the 2007 US Carnegie Report as a better educational framework than the Socratic Method or ‘case-dialogue teaching’. The Cognitive Apprenticeship approach to teaching advocates embedding ‘learning in activity’ and making ‘deliberate use of the social and physical context’. Secondly, current educational theory suggests the embedding of generic skills (which includes research methods) into the law curricula as a better framework than simply adding elective units at the end of the degree.

In a March 2008 survey of the curricula from 29 law schools’ websites in Australia, it was evident that very few courses explicitly included empirical training in their law degrees.

At the very least, existing research modules may need to be remodelled to integrate some coverage of empirical methodologies.

Legal research skills have consistently been regarded as basic requisites for both academic and practising lawyers, and have invariably been included in any listing of desired lawyer attributes.

Recently, the Centre for Learning and Professional Development at Adelaide University has developed a cross-discipline Research Skills Framework. The most efficient method would seem to be to expose students to the methodologies within compulsory undergraduate units, include further basic training within compulsory research units, and in addition to provide elective units for those seeking to augment the initial training.

At a very preliminary level, the challenge is to highlight empirical fact assumptions from first year. This can be achieved through the discussion of simple student surveys, and the thoughtful use of statistics and relevant empirical material in course content. Social science evidence can be highlighted within substantive areas, but enhanced treatment in evidence law units is of course warranted.

Lawyers need to be able to critique research that others have carried out in order to judge the reliability of empirical data whether it is discussed as evidence – or reported in the newspapers as fact.

There is more opportunity for students to achieve a depth of knowledge within the later year Honours units, undergraduate elective offerings and in the Masters, SJD and PhD research training units if the students come to higher studies armed with a basic understanding from their undergraduate courses.

Postgraduate students need exposure to the range of research methodologies possible for their projects. This requires an introduction to methodologies to augment the doctrinal work with which they are familiar. Postgraduates also need information on research ethics and the process of requesting ethics approval from the relevant university committees.

To effectively introduce empirical facts recognition and awareness the material has to be introduced as part of assessment in units. This is more difficult to accomplish. In Australian universities, there are often only 13 weeks in a semester. Even providing the students have their topic clearly defined at the beginning of the semester, there is still a lag-time required for the ethics approval procedure and a simple survey can take time to set up. For this reason students may be dissuaded from doing more than a doctrinal study within the timeframe. Small numbers of postgraduate law students are taking up the challenge of empirical non-doctrinal studies because of the obstacles being encountered. Where then will future researchers gain the training required to apply for large research grants and undertake meaningful research?

Central to this discussion is the cost of teaching research to large student bodies. Legal research requires academics with specific expertise. It is time consuming to teach. The levels of marking tend to be higher than a normal substantive unit.

However, given a commitment by the universities and government to the need for change, advances are possible. The UK Nuffield Report recommended a system of bursaries, grants and fellowships to encourage academic training in empirical research skills from undergraduate to post-doctoral level.

Traditional doctrinal models of legal research need to be supplemented by methodologies based on an awareness of the methods used in other disciplines, particularly social research methods.

Law is not a closed system. It is intrinsically embedded in its specific legal context and community. Despite what is said about the law being a closed system, the examples of the use of empirical facts in this article demonstrate that law is being pressured to recognise the existence of the work of other disciplines and its relevance to decision making in the courts, and therefore legal educators need to better equip the profession to deal with the contextual research that they ncounter.

At this point we need to better inform our profession – our judges, our law students, and academics on the wealth of data available to them and to encourage and to make provision for the proficient use of this data in the legal process.