Technology without borders: Global co-operation and domestic impacts

Presented at the Australian Government Solicitor Administrative Law Conference

Download RTF - 361 KB

Introduction

I hope you will indulge me if I begin with a short excerpt from a popular fairytale written in the era of industrialisation:

Dorothy was thinking so earnestly as they walked along that she did not notice when the Scarecrow stumbled into a hole and rolled over to the side of the road. …

“Why didn’t you walk around the hole?” asked the Tin Woodman.

“I don’t know enough,” replied the Scarecrow cheerfully. “My head is stuffed with straw, you know, and that is why I am going to [the Wizard of] Oz to ask him for some brains.”

“Oh, I see,” said the Tin Woodman. “But after all, brains are not the best things in the world.”

“Have you any?” inquired the Scarecrow.

“No, my head is quite empty,” answered the Woodman. “But once I had brains, and a heart also; so, having tried them both, I should much rather have a heart.”

This passage from The Wonderful Wizard of Oz[2] can be seen as an allegory on the choices which confront us today. While the capabilities of machine technologies have massively expanded from the reactive machines of the 1940s and ‘50s, even the most powerful AI technologies today can only simulate intelligent behaviour based on predictive or probabilistic outputs.[3] In common with the Tin Man, they have neither a heart nor brains. Further, while, with machine learning technologies, an AI system will be able to learn to navigate a virtual obstacle course, the system will have no consciousness and, like the Scarecrow, will feel no pain if it stumbles into a hole.

The development of these technologies has significant implications for governments globally. It is no surprise that governments should seek to utilise these technologies across the broad spectrum of activities with hundreds of millions of decisions made by government in Australia alone every year utilising various machine technologies. That trend is likely to continue in line with rapid growth in the volume, complexity, and subject matter of decisions made by governments affecting private and commercial rights and interests, given the capacity of these technologies to promote consistent, accurate, cost-effective, and timely decisions.

These technologies also create significant risks of harm including for human rights, democratic systems and processes, and global peace and security. In this regard, the astronomical rate of change and scale of developments in machine technologies has hitherto outpaced the capacity of humankind to develop the regulatory frameworks within which such technologies should be designed, developed, deployed, and decommissioned. This is so, notwithstanding significant recent progress nationally and internationally in developing regulatory frameworks and policies.

We also stand on the cusp of a technological revolution with the evolution of quantum technologies which use qubits to process information in place of bits. These technologies will exponentially boost computer power above what is achievable by classical computers and bring ever more powerful and secure global communication networks, while also breaking existing encryption methods currently regarded as secure.

It is therefore, as I have elsewhere said, no overstatement to say that we are at a pivotal point in human history, confronted with the reality of machines of exceptional power which are capable of being harnessed for the betterment of humankind but also capable of great harm.

In addressing the topic of my presentation today, I will focus on three issues arising from existing technologies which are relevant in the judicial and administrative law contexts:

1. current international collaboration on the creation of legal frameworks for the design, development, deployment and use of these new technologies;
2. key deficiencies in current machine technologies; and
3. measures to mitigate potential harms posed by AI in certain high-risk contexts, being the administration of justice and administrative decision making.

International collaboration on framework principles

The rule of law and fundamental administrative law values of legality, transparency, fairness, accountability, and access to appropriate review mechanisms underpinned the Administrative Review Council’s (ARC’s) recommendations in its ground-breaking report in 2004 on Automated Decision-Making (ADM).[4] The enduring relevance of that report was recognised in the Consultation Paper released late last month by the Attorney-General’s Department on the Use of Automated Decision-making by government for the development of a new legal framework for ADM.[5] This initiative follows the government’s response to the Robodebt Royal Commission.[6] ADM in the context of that Consultation Paper is not confined to the relatively simple rules-based systems in use at the time of the ARC’s report, but includes AI systems and machine learning.[7]

Like principles underpin current developments internationally with respect to the creation of legal frameworks for regulation of AI throughout its lifecycle. These values, in turn, accord with, and promote compliance with, our international human rights obligations.

International co-operation has led to the development of guidelines and other instruments specifically in the AI space for both governments and private entities. The first intergovernmental standard on AI was the Recommendation on Artificial Intelligence adopted by the Organisation for Economic Co-operation and Development (OECD) on 22 May 2019 which was updated earlier this year in light of new technological and policy developments (OECD Principles).[8] The OECD Principles make recommendations not only for governments but for “AI actors” being “those who play an active role in the AI system lifecycle, including organisations and individuals that deploy or operate AI”.[9] While accepting that implementation will differ in different jurisdictions in response to their own unique needs and circumstances, the OECD Principles are intended to promote harmonious global AI governance, recognising that these technologies know no boundaries.

Building on the OECD Principles, in November 2023 the UK Government hosted the AI Safety Summit at Bletchley Park. This, of course, is the birthplace of the modern computer, for it was at Bletchley Park that the mathematician Alan Turing and other scientists broke the Nazis’ ENIGMA code. This summit brought together government representatives, leading AI companies, civil society groups and experts in research. As such, the composition of the summit itself recognised that the creation and implementation of legal regulatory frameworks requires collaboration not only between States, but with stakeholders and experts across a raft of disciplines. So much was also recognised by the ARC in 2004 which recommended that, while agencies should remain responsible for their own systems, an independent interdisciplinary advisory panel including lawyers should oversee government ADM systems for quality assurance at every stage in their lifecycle and provide advice to government.[10]

The Bletchley Declaration which resulted from the summit was signed by the United Kingdom, the EU, China, the US, Australia and 25 other countries. This non-binding instrument recognises both the opportunities potentially afforded by AI “to transform and enhance human wellbeing, peace and prosperity”, while also recognising the “potential for serious, even catastrophic, harm, either deliberate or unintentional, stemming from the most significant capability of these AI models”.

Significantly, the Bletchley Declaration recognises that many of the risks arising from AI are “inherently international in nature, and so are best addressed through international cooperation”.[11] As I have elsewhere observed, “statements such as these highlight [that] Australia is not an island; its response to AI must, of necessity, take account of, and, to the extent appropriate, align with, international responses.” [12]

The EU AI Act[13] which came into effect on 1 August this year also constitutes a significant legal milestone in AI regulation. It comprises a detailed regulatory regime focused on mitigating the risks of AI in high-risk contexts through measures such as requiring risk and quality management systems, human oversight, and third-party assessment, before being sold or used in the EU (Art 6). It also prohibits certain AI systems capable of significant harm for malevolent purposes. Its aim is to promote the use of AI which is safe, respects human rights, and protects health, safety and the environment. While not binding on non-member States, the EU AI Act nonetheless has potential relevance well beyond the jurisdictions of member States. This is because of its capacity to contribute to the development of principles of customary international law and influence the content of domestic laws outside the EU, and because of its application to developers and deployers of high-risk AI systems where the system or its output is used in the EU, regardless of where the developer or deployer is based.

In common with the EU AI Act, the Council of Europe Framework Convention on AI,[14] which opened for signature in September 2024, identifies potential risks and adverse impacts posed by AI to democratic processes, the rule of law and human rights. While it is not prescriptive as to the precise measures to be taken, it also requires State parties to adopt measures which assess and mitigate these and other high risks and adverse impacts throughout the lifecycle of AI systems.

The Framework Convention is of particular significance. This is because it is the first legally binding treaty to regulate AI and is intended to respond to the urgent need for a globally applicable legal framework. The Convention is open to signatories outside the EU and, among its first signatories are the UK and the US, as well as the EU.[15] Themes addressed in the Convention include:

1. assessing and mitigating the risks of AI throughout its lifecycle from the cradle to the grave;
2. addressing the use of AI in the public and private sectors;
3. ensuring transparency and oversight of AI systems having regard to context and risks; and
4. ensuring accountability and responsibility for adverse impacts including by adopting accessible and effective remedies for violations of human rights.

While not yet a party, Australia has recognised the importance of aligning its legislative and policy responses to international responses in line with these international instruments.[16] Thus, consistently with the approach adopted internationally, the proposals paper on mandatory guidelines to promote the safe and responsible adoption of AI in Australia released on 5 September 2024 by the Department of Industry, Science and Resources also focuses upon processes to mitigate the risks of AI in high-risk settings throughout the lifecycle of AI systems.[17]

What then are key ai risks relevant in the judicial and government decision-making contexts?

Returning to our conversation from the Wizard of Oz, the Scarecrow told the Tin Man that he had decided to “ask [the Wizard] for brains instead of a heart; for a fool would not know what to do with a heart if he had one.” The Tin Man begged to differ: “I shall take the heart,” he said; “for brains do not make one happy, and happiness is the best thing in the world.” Curiously, though, despite his lack of a heart and brains, the Tin Man wept tears of sorrow rusting his face and jaws when, upon his journey with Dorothy, the Scarecrow and the cowardly lion, he trod upon a beetle. This was because the Tin Man was very careful not to hurt any living creature.

The Tin Man’s apparent sentience and conscience finds, of course, no equivalent in machine technologies.

First, robots have no capacity to understand or exercise fundamentally human qualities such as mercy, fairness, and compassion which have long informed courts and administrative decision-makers in the exercise of discretions. While the width of statutory discretions will vary according to context, discretions potentially afford humans the latitude to make judgments and reach decisions which reflect community, administrative and international values, and align with statutory objects, in the face of a wide or almost infinite variety of individual human circumstances. The capacity to exercise a discretion having regard to such values is also essential in many contexts to maintaining public confidence in decision-making.

The absence of such human qualities from machine processes calls for very great caution when assessing whether particular statutory powers and functions should or can be entrusted in whole or in part to machine processes. Where the power involves an exercise of discretion, the answer should be a simple “no”. The exercise of discretions is an inherently human activity and the purported exercise of a statutory discretion by a machine, would almost certainly be unlawful. Even the use of such programs to make recommendations as to the exercise of discretions should be approached with caution in order to ensure that the human decision-maker brings a properly independent mind to bear upon the issues and bearing in mind (as I shortly explain) the opacity of many AI systems and their outputs.

Thus, as the Australian Government recently emphasised in its briefing “How might AI affect the trustworthiness of public service delivery?”, using AI should not come at the expense of empathy.[18] The Robodebt tragedy is an example in point where not only were the legalities ignored, but also any appreciation of the great distress and harm that the system was likely to cause, and in fact caused, to highly vulnerable individuals. Unsurprisingly, therefore, the recommendations of the Robodebt Royal Commission included that policies and processes should be designed with an emphasis upon the people which they are intended to serve.

Secondly, the lack of the equivalent to a brain in modern machine learning and generative AI systems poses problems in terms of transparency. AI systems are typically “black box” models where the processes undertaken by the system to produce outputs in response to inputs are opaque. This means that how the system generated a particular output cannot be explained even by the developers and programmers, raising serious accountability, accuracy and audit issues. This poses particular difficulties where such systems are sought to be deployed in an administrative decision-making or judicial context where, unknown to the user, the output may for example be affected by biases and discrimination derived from historical data on which the system was trained or sound plausible but be completely wrong.

Thirdly, however convincing the output might look, machines have no capacity to think or to reason and no “appreciation” of accuracy. Reliability of output is therefore a serious issue. Large language models or LLMs illustrate the risks. An LLM is a complex algorithm which responds to human prompts to generate new text representing the most likely words and word order based on training from massive datasets – essentially predictive text on steroids. Devoid of understanding or concepts of accuracy, the capacity of LLMs to “make up” information or hallucinate, and to convey the false information convincingly, is well-documented. Like the Wizard of Oz who was unmasked as a charlatan and magician when the cowardly lion knocked down his screen, the output of an LLM may be all smoke and mirrors.

An example of hallucinations is the well-reported decision in the US last year of Mata v Avianca[19] (22 June 2023) in which Mr Mata’s lawyers filed submissions containing three fake judgments generated by ChatGPT. When the existence of the decisions was questioned, the lawyers for Mr Mata filed an affidavit attaching copies of the alleged cases after “asking” ChatGPT to confirm that the cases were real and being “reassured” that they did in fact exist and could be found in reputable legal databases.

Furthermore, in a collaborative paper published this year, researchers from eminent universities in the United States and the United Kingdom, as well as the UK AI Safety Institute and Apollo Research, exposed “a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form ‘A is B’, it will not automatically generalize to the reverse direction ‘B is A’.” [20] An example given in the collaborative paper of this failure in basic logical deduction, described as the “Reversal Curve”, was the following:

suppose that a model’s training set contains sentences like “Valentina Tereshkova was the first woman to travel to space”, where the name “Valentina Tereshkova” precedes the description “the first woman to travel to space”. Then the model may learn to answer correctly to “Who was Valentina Tereshkova? [A: The first woman to travel to space]”. But it will fail to answer “Who was the first woman to travel to space?” and any other prompts where the description precedes the name.

Fourthly, biases may infect a program at the design stage and may reflect deliberate choices or unconscious biases of the programmer or developer. Further, machine learning systems and AI rely upon historical data. The danger, of course, is that this data may embody conscious or unconscious biases of the earlier human decision-makers or, indeed, as more text generated by Generative AI becomes available, from prior Generative AI material and prompts. By such means, stereotypes and unfair or arbitrary discrimination may be perpetuated and embedded in decision-making processes, rendering the decisions unfair and potentially in violation of anti-discrimination laws. As the Consultative Council of European Judges recently opined in their opinion “Moving forward: the use of assistive technology in the judiciary”:[21]

Technology is also not design neutral. It carries the inherent risk of discriminatory design, implementation, and use. Design may discriminate against parties on grounds of race, ethnicity, sex, or gender. It may also adversely effect, for instance, the neuro-diverse or individuals with a visual or hearing impairment.

Finally, AI has no regard for privacy or confidentiality, including through the retention of prompts and human responses to “answers” to prompts for the purposes of training the system going forward. For example, in August this year, a psychosocial safety training company used a generative AI program to generate fictional examples of bullying and sexual harassment for a course at a regional State prison. The Chatbot even reassured the prompter that the case study was “entirely fictional”. However, as the slide was uploaded onto the screen, an employee at the prison recognised the scenario as real and containing sensitive information, even to the point of identifying by name the former employee who was engaged in litigation against the prison for alleged sexual harassment and bullying.[22]

What issues do these technologies pose for courts and tribunals, and how have they responded?

The use of AI tools in the administration of justice has been classified by the EU AI Act as high-risk having regard among other things to the need to protect the independence and impartiality of the judicial process. Even the use of AI to research the law is classified as high-risk. As the AIJA has pointed out,[23] this reflects, among other things, the potential for third-party AI tools to impact on judicial independence given:

• the opaque nature of many AI systems and their capacity to hallucinate;
• the risk that the dataset on which the system has been trained may be outdated;
• the risk that the AI system’s output may contravene Australian privacy law, Australian copyright law or contain discriminatory material; and
• the risk of potential control, interference or surveillance from foreign actors via privately developed AI tools.

The Consultative Council of European Judges in its 2023 opinion also identified particular risks to the judicial process posed by the use of technologies given, among other things, the discretionary nature of many judicial decisions, the role of judges in developing the law, and the need to ensure that technology “does not discourage or impede the critical thinking of judges as this can lead to stagnation of legal development”.[24] Similar risks attend the use of technology by administrative tribunals save that tribunals do not have the same role in developing the law.

Further, the media attention on cases where judges abroad have used AI in the production of their reasons illustrates, among other things, the very real capacity for such conduct to undermine public confidence in the judiciary, no matter how limited the use of AI in the particular judgment may have been.

Issues of this nature have led to initiatives by courts in Australia, New Zealand, the UK and elsewhere to develop guidelines and practice notes for judicial officers, tribunal members, lawyers and unrepresented litigants around the risks of using such technologies. Importantly, these emphasise the professional and ethical obligations of legal representatives, including to the Court, which squarely place responsibility on the legal representatives and make it clear that those responsibilities cannot be delegated to a machine.

With respect to judges, the NSW Supreme Court Judicial Guidelines,[25] for example, prohibit the use of generative AI in the formulation of reasons for judgment or the assessment or analysis of evidence preparatory to the delivery of reasons for judgment, as well as the use of generative AI for editing or proofing draft judgments. While accepting that generative AI can be used for secondary legal research purposes, the Guidelines advise judges to familiarise themselves with the limitations of AI.

Similarly, the Administrative Review Tribunal Code of Conduct[26] emphasises that members must personally make the decision on an application for review. Accordingly, the Code of Conduct provides that a member must not use generative AI to obtain guidance on the outcome of a proceeding, to produce any part of the reasons, or to obtain feedback or assistance on reasons.

Conclusion

Unlike the Tin Man and the Scarecrow, our choice is not between a heart or a brain. When decisions are made impacting on the rights and interests of individuals, it is apparent that productivity, rationalization and efficiency cannot overwhelm the need for a heart. Whether it is the dispensing of justice by courts or the making of decisions by the executive, we must never lose sight of the importance of the fundamentally human qualities and values which underpin a just and stable democratic society governed by the rule of law. Nor must we suspend our own thought processes in the face of apparently convincing material produced by machines. The many incredible advances that can be achieved with the assistance of AI in science, medicine, engineering and a raft of other fields cannot be denied. However, equally it must be remembered that AI is indeed artificial, and it is not intelligent.

[1] LLB (Hons)(Adel), LLM, PhD (Cantab), FAAL. This presentation draws to some extent upon other publications by Justice Perry including: Melissa Perry, Benjamin Durkin and Charlotte Breznik, “From Shakespeare to AI: The Law and Evolving Technologies” (2024) 98 ALJ 272; Perry, “Emerging Technologies”, Keynote Address, Australian Librarians’ Association Annual Conference 2024, delivered on 9 August 2024; and Perry, “Advising Government in the Age of Artificial Intelligence”, Keynote Address, Commonwealth Government Executive Forum, delivered on 6 August 2024.

[2] L. Frank Baum, The Wonderful Wizard of Oz (1900).

[3] As defined in the Oxford English Dictionary (online): ‘artificial intelligence’.

[4] Administrative Review Council, Automated Assistance in Administrative Decision Making, Report to the Attorney-General No. 46 (November 2004) (ARC Report No. 46) <https://www.ag.gov.au/legal-system/publications/report-46-automated-assistance-administrative-decision-making-2004>. This report formed the basis of the publication by the Commonwealth Ombudsman, Automated Decision-Making Better Practice Guide (updated in 2019) which included a detailed checklist summarising items which should be addressed when considering the implementation or update of an ADM system.

[5] Australian Government, Attorney-General’s Department, Use of Automated Decision-Making by Government, Consultation Paper (Commonwealth of Australia, 2024) at p. 9.

[6] Government Response to the Royal Commission into the Robodebt Scheme (Commonwealth of Australiia, November 2023).

[7] Australian Government, Attorney-General’s Department, Use of Automated Decision-Making by Government, Consultation Paper (Commonwealth of Australia, 2024) at p. 7.

[8] See also the Hiroshima Process International Code of Conduct for Organizations Developing Advanced AI Systems released by the G7 in November 2023, which organisations, including the public sector, were urged to endorse. This “living document” builds on existing OECD AI Principles and is intended to respond, and be responsive, to these rapidly evolving technologies, recognising that governments are still in the process of developing appropriate and effective regulatory systems: G7, Hiroshima Process International Code of Conduct for Organizations Developing Advanced AI Systems (Publication) 1.

[9] OECD, ‘Recommendation of the Council on Artificial Intelligence’ OECD Legal Instruments (Web Page, amended on 8 November 2023) <https://legalinstruments.oecd.org/en/instruments/oecd-legal-0449>.

[10] ARC Report No. 46 at [4.15.2].

[11] Ibid.

[12] Melissa Perry, Benjamin Durkin and Charlotte Breznik, “From Shakespeare to AI: The Law and Evolving Technologies” (2024) 98 ALJ 272, 279.

[13] Regulation - EU - 2024/1689 - EN - EUR-Lex (available at: https://eur-lex.europa.eu/eli/reg/2024/1689/oj ).

[14] Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law, opened for signature 5 September 2024, Council of Europe Treaty Series, No. 225 (available at: The Framework Convention on Artificial Intelligence - Artificial Intelligence ).

[15] As at 6 December 2024, the signatories to the Council of Europe Framework Convention on AI were: Andorra, Georgia, Iceland, Montenegro, Norway, the Republic of Moldova, San Marino, the United Kingdon, the European Union, Israel and the United States of America: www.coe.int/en/web/artificial-intelligence/the-framework-convention-on-artificial-intelligence .

[16] By way of example, the joint State, Territory and Commonwealth National Framework for the Assurance of Artificial Intelligence in Government released in June 2024 affirmed the commitment to deeper international cooperation and dialogue.

[17] Consultation on the proposed mandatory guardrails closed on 4 October 2024: <https://consult.industry.gov.au/ai-mandatory-guardrails>.

[18] Department of the Prime Minister and Cabinet, ‘How might AI affect the trustworthiness of public service delivery?’ (Long-Term Insights Briefing, 2023) 9 <https://www.pmc.gov.au/resources/long-term-insights-briefings/how-might-ai-affect-trust-public-service-delivery>.

[19] F. Supp 3d 22-cv-1461 (PKC), 2023 WL 4114965 (June 22, 2023) <https://caselaw.findlaw.com/court/us-dis-crt-sd-new-yor/2335142.html>.

[20] Lukas Berglund et al, ‘The Reversal Curse: LLMs Trained on “A Is B” Fail to Learn “B Is A”’ (No arXiv:2309.12288, arXiv, 26 May 2024) <http://arxiv.org/abs/2309.12288>. I am indebted to Professor Kimberley Weatherall for referring me to this research in her presentation to the Federal Court Judges’ conference on 27 November 2024.

[21] CCJE Opinion No. 26 (2023) (Strasbourg, 1 December 2023) at [70].

[22] Bridget McArthur, ‘AI Chatbot Blamed for Psychosocial Workplace Training Gaffe at Bunbury Prison’, ABC News (online, 21 August 2024) <https://www.abc.net.au/news/2024-08-21/ai-chatbot-psychosocial-training-bunbury-regional-prison/10423098>.

[23] AIJA, AI Decision-Making and the Courts (2022): A Guide for Judges, Tribunal Members and Court Administrators at [4.2].

[24] CCJE Opinion No. 26 (2023) (Strasbourg, 1 December 2023) at [90].

[25] NSW Supreme Court, Generative AI Practice Note and Judicial Guidelines (SC Gen 23, 21 November 2024).

[26] Administrative Review Tribunal, Code of Conduct for Non-Judicial Members (14 October 2024) ch 1 pt 9.