Pasquale, Frank --- "The Automated State: Implications, Challenges and Opportunities for Public Law" [2022] SydLawRw 29; (2022) 44(4) Sydney Law Review 619

---

Book Review

The Automated State: Implications, Challenges and Opportunities for Public Law (2021)

by Janina Boughey and Katie Miller (eds)

Federation Press, 296 pp, ISBN 9781760022952

Frank Pasquale^[1]*

Please cite this book review as:

Frank Pasquale, ‘Book Review: The Automated State: Implications, Challenges and Opportunities for Public Law by Janina Boughey and Katie Miller (eds)’ (2022) 44(4) Sydney Law Review 619.

This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International Licence (CC BY-ND 4.0).

As an open access journal, unmodified content is free to use with proper attribution.

Please email [e-mail] for permission and/or queries.

© 2022 Sydney Law Review and author. ISSN: 1444–9528

At present, the commercial appeal of automated legal systems rests on three pillars: speed, scale, and preference satisfaction. The proto-smart contracts evident in algorithmic stock trading, for example, are deemed superior to text-based contracts directly agreed by persons because they can be executed more quickly, across a wider range of space, satisfying more traders’ preferences for trading than could ordinary contracts.^[1] There are, of course, larger purposes of the financial system, which should be weighed against these desiderata. But few would say there is a core essence of financial transactions at risk of being derailed by computation.

However, for many other parts of the legal system, there is a common sense that their translation into computation would be inappropriate. Terms of imprisonment meted out computationally, without mediation by a human, are unthinkable. Nor does civil judgment by robot seem wise, though ‘regulation by robot’ has been proposed for narrow and discrete arenas of state administration.^[2]

As these spheres begin to expand, there is a critical need for insightful commentary on the virtues and limits of automated decision-making (‘ADM’).

The Automated State helps address this pressing demand,^[3] as an important landmark for the study of predictive analytics, artificial intelligence (‘AI’), machine learning (‘ML’), and other statistical and computational methods for assisting administrators (and perhaps even taking on some of their work). The book includes much material of deep interest to judges, public servants, practitioners, and law and technology academics. By combining theoretical insights and practical references to national security, social security, immigration, health, and other contexts, the contributors both illuminate the present and suggest fruitful paths for further investment in (and control of) automation.

Following Justice Duncan Kerr’s thoughtful foreword,^[4] Justice Melissa Perry’s chapter is a perfect opening for the main text of the volume.^[5] It discusses the many virtues of ADM, including the possibility of reducing the burden of repetitive and obvious work on those who staff immigration and customs offices. Perry’s chapter then goes on to cover both legal and normative limits on the growth of automation. This survey nicely sets up the many reformist and critical interventions that come later in the volume.

Guzyal Hill continues this mapping project, though from a more legislative perspective.^[6] One core of the chapter is the exploration of the use of plagiarism detection software to identify similarities between sets of Uniform Acts, to then sort them via a similarity index. This type of software-driven comparison does suggest new ways of engaging in a distant reading of legal texts to discover opportunities for harmonisation. Expect future legal scholars to continue to dig deeply into unexpected points of convergence and divergence among legal systems exposed by the brute force of computational comparison of strings of words of varied lengths.

In a quite far-seeing chapter, Lyria Bennett Moses, Janina Boughey and Lisa Burton Crawford identify some ways in which AI and ML can assist statute drafters in navigating ‘the legislative labyrinth that is modern government’.^[7] They note that the Social Security Act 1991 (Cth) has been amended roughly 10 times per year between 2013 and 2017, and the Income Tax Assessment Act 1997 (Cth) was amended over 18 times per year in the same period.^[8] In such a context, automatic, triggered notifications based on the specific concerns of an advocate could be quite useful, particularly with respect to more obscure provisions unlikely to be covered by a reliable secondary source. Internal cross-referencing and technical provisions help create webs of statutory meaning where the alteration of the scope of one term may have unexpected second- and third-order effects. The authors are particularly adept at identifying and elaborating on the opportunities and challenges posed by efforts to move from machine-readable to machine-consumable law. The latter term covers digital renditions of law that can enable a computer to automatically perform a task. While such a move may expand access to justice by bringing scale efficiencies to law enforcement (as noted in the Administrative Review Council Report of 2004),^[9] it also raises concerns about due process, transparency, and human rights, which are expertly addressed in other chapters.

For example, Maria O’Sullivan forcefully observes that automation ‘means that legal errors that may ordinarily be limited to a small cohort of affected individuals tend to be amplified and become systemic in nature’.^[10] Her chapter’s exploration of effective remedies for such wrongdoing demonstrates the importance of a deontologically informed, rights-focused perspective. All too many considerations of ADM are dominated by utilitarian analysis, where efficiency gains are likely to outweigh error costs. However, when an error denies a fundamental right, deeper cautions are advisable. One practical response that O’Sullivan explores would be to grant the Administrative Appeals Tribunal the prerogative to complete ‘group-based review of claims which involve a common algorithm or data matching system’.^[11] Such a plenary review authority would save both the legal system and adversely affected parties a great deal of time and effort, while more quickly vindicating meritorious rights claims.

Marc Cheong and Kobi Leins complement O’Sullivan’s chapter well by further exploring the question of how to modernise regulation and review of Australian ADM, focusing on requirements for algorithmic explainability.^[12] Explainability here means that a decision ‘must be comprehensible not only to data scientists or controllers, but to the lay data subjects (or some proxy) affected by the decision’.^[13] Robust legal requirements for explainability may limit the types of ML and AI that are deployed in ADM. Thus, Cheong and Leins’s chapter is particularly notable for its careful review of the technical details of the varied types of ML and expert systems that may be at the core of distinct ADM systems, since addressing these ‘technicalities’ (to evoke Annelise Riles’s research)^[14] clarifies the limits of computation in legal proceedings. This is an important clarification in part because it helps vindicate Joe McIntyre’s and Anna Olijnyk’s claim (in ‘Public Law Limits on Automated Courts’) that AI’s ‘role should never extend to the core business of judicial determinations’.^[15]

To assure that ADM is properly limited and guided, transparency will be essential. This collection includes three strong chapters presenting the threats to transparency posed by outsourcing of ADM to the private sector,^[16] the ‘need for greater transparency’ to evaluate automation,^[17] and how parliamentary committees may play a more significant role in scrutinising actions of an increasingly automated executive branch.^[18] As O’Donovan shows, freedom of information law exceptions may be exploited by agencies seeking to shield their automated systems from public scrutiny. The problem is compounded, as Boughey demonstrates, when corporate actors with a deep interest in trade secrecy deflect requests for accountability by asserting their commercial interests in keeping their products’ methods of operation proprietary. She presents a compelling case for legislative reform to require more transparency and explainability from ADM systems, while also presenting pathways for agencies and courts to realise these values under current law.

A final set of chapters supplements proposed transparency requirements, by articulating the principles of fairness that should animate ADM policy going forward. Joel Townsend’s treatment of Robodebt thoughtfully demonstrates how merits review of agency action failed ‘to provide an appropriate check on this high volume, technology-assisted decision-making process’.^[19] In short, a poorly designed, multi-tiered merits review process effectively ‘insulated Robodebt from public scrutiny’,^[20] while many citizens suffered unfair accusations and debt claims. Perhaps the only ‘silver lining’ of the Robodebt fiasco is the motivation it should give to senior administrators to closely consult and follow work like Townsend’s, as well as Matthew Groves’s application of general principles of fairness in ADM to the Australian context.^[21] Sarah Crossman and Rachel Dixon present practical proposals for ensuring future government procurement and project management better reflects values of fairness, transparency, and equity.^[22] And Miller’s proposals for keeping ‘citizens in the loop’ of ADM procedures also provide a rich source of insight on how to humanise the use of computation in state administration.^[23]

Book review space limitations do not permit extended engagement with the conclusions of The Automated State, but a few key points may be developed here. First, this is a volume that should be of great interest both within Australia, and in many other jurisdictions (both common and civil law). The chapters demonstrate that Australia has long experimented with the automation of administration, with concomitant experience of its advantages and shortcomings. This sophisticated legal discourse on ADM has much in it to instruct, say, European and Californian authorities as they implement rules on the right to meaningful information about corporate profiling of data subjects. Even though the European Union’s General Data Protection Regulation^[24] and the California Privacy Rights Act of 2020^[25] are primarily focused on the private sector, the Australian public law discussions developed in this book feature a sophisticated understanding of computational methods and legal principles that are relevant to data protection and ADM generally.

Second, the chapters in this volume illuminate the need for more sociolegal research on the place of AI and big data in legal systems. While rationalising reformers may model the law as a code to be translated from natural to computer languages, the life of the law is experience. Careful attention to citizen-users of ADM systems (and the fate of those whom these systems profile) should be built into their implementation budgets. That is one important way to ensure that policymakers have the data necessary to continually improve systems, or, where improvement is impossible and serious errors persist, to limit or phase out their application.

Third, The Automated State suggests that there will be continuing opportunities for legal educators to collaborate with and learn from technical experts — and vice versa. We may reach the point soon where some understanding of computational thinking is part of lawyers’ duty of technological competence. Similarly, computation is now informing so many aspects of daily life that it may well become incumbent upon Departments of Computer Science to give their students a clear sense of when a consultation with a lawyer is necessary for the proper development and deployment of software. I expect more cross-disciplinary courses in both law and computing to address these issues. And I hope to see books like

The Automated State assigned in them, as exemplars of fair-minded, practical, and insightful inquiry into the opportunities and pitfalls of using AI in administration.

---

^&#6[1] Jeffrey D Forchelli Professor of Law, Brooklyn Law School, New York, USA.

Email: [e-mail]; ORCID iD: https://orcid.org/0000-0001-6104-0944.

^[1] On the critical distinction between text-driven and code-driven law, compare Mireille Hildebrandt, ‘The Adaptive Nature of Text-Driven Law’ (2022) 1(1) Journal of Cross-Disciplinary Research in Computational Law, and Mireille Hildebrandt, ‘Code-Driven Law: Freezing the Future and Scaling the Past’ in Simon Deakin and Christopher Markou (eds) Is Law Computable?: Critical Perspectives on Law and Artificial Intelligence (Hart Publishing, 2020) 67.

^[2] Cary Coglianese and David Lehr, ‘Regulating by Robot: Administrative Decision Making in the Machine-Learning Era’ (2017) 105(5) Georgetown Law Journal 1147.

^[3] Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) (‘The Automated State’).

^[4] Justice Duncan Kerr, ‘Foreword’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) v.

^[5] Justice Melissa Perry, ‘iDecide: Digital Pathways to Decision’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 1.

^[6] Guzyal Hill, ‘Untapped Opportunities for the Use of Artificial Intelligence in Comparing Legislation for National Reforms’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 215.

^[7] Lyria Bennett Moses, Janina Boughey and Lisa Burton Crawford, ‘Laws for Machines and Machine-Made Laws’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 232, 253.

^[8] Ibid 233.

^[9] Administrative Review Council (Cth), Automated Assistance in Administrative Decision-Making: Report to the Attorney General (Report No 46, November 2004) vii:

Expert systems can play a significant and beneficial role in administrative decision making, particularly in areas where high volumes of decisions are made. Their potential to offer cost savings and improve efficiency and accuracy means it can be expected that the systems will become increasingly important tools of government.

^[10] Maria O’Sullivan, ‘Automated Decision-Making and Human Rights: The Right to an Effective Remedy’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 70, 70 (emphasis in original).

^[11] Ibid.

^[12] Marc Cheong and Kobi Leins, ‘Who Oversees the Government’s Automated Decision-Making? Modernising Regulation and Review of Australian Automated Administrative Decision-Making’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 174.

^[13] Ibid 189, quoting Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter and

Luciano Floridi, ‘The Ethics of Algorithms: Mapping the Debate’ (2016) 3(2) Big Data & Society, 17 n 26.

^[14] Annelise Riles, ‘A New Agenda for the Cultural Study of Law: Taking on the Technicalities’ (2005) 53(3) Buffalo Law Review 973, 975:

[T]he technicalities of law are precisely where the questions that interest us actually are played out. Humanists should care about technical legal devices because the kind of politics that they purport to analyze is encapsulated there, along with the hopes, ambitions, fantasies and day-dreams of armies of legal engineers.

^[15] Joe McIntyre and Anna Olijnyk, ‘Public Law Limits on Automated Courts’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 89, 89.

^[16] Janina Boughey, ‘Outsourcing Automation: Locking the “Black Box” inside a Safe’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 136.

^[17] Darren O’Donovan, ‘Evaluating Automation: The Need for Greater Transparency’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 31.

^[18] Sarah Moulds, ‘Holding an Automated Government to Account? The Role of Parliamentary Committees’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 110.

^[19] Joel Townsend, ‘Better Decisions? Robodebt and the Failings of Merits Review’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 52, 52.

^[20] Ibid 69.

^[21] Matthew Groves, ‘Fairness in Automated Decision-Making’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 14.

^[22] Sarah Crossman and Rachel Dixon, ‘Government Procurement and Project Management for Automated Decision-Making Systems’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 154.

^[23] Katie Miller, ‘Retaining the Citizen in the Loop — The Role of the Citizen in Digital Government’ in Janina Boughey and Katie Miller (eds) The Automated State: Implications, Challenges and Opportunities for Public Law (Federation Press, 2021) 197.

^[24] Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC [2016] OJ L 119/1, corrected by [2018] OJ L 127/2 (‘General Data Protection Regulation’).

^[25] California Privacy Rights Act of 2020, Cal Civil Code §1798.81.5.