Courtroom chronicles: ai and human rights in international legal precedents

Digital Technologies and Law (

секция

на

английском

языке

)

______________________________________________________________________________________________________________________

200

3. Kreditwesengesetzes [Electronic resource]. – URL:

https://www.gesetze-im-

internet.de/kredwg/

4. URL:

https://www.bafin.de/DE/Aufsicht/FinTech/Geschaeftsmodelle/DLT_

Blockchain_Krypto/DLT_Blockchain_Krypto_node.html

;

BaFin Guidance Notice

Crypto Tokens

;

URL:

https://www.bafin.de/SharedDocs/Downloads/DE/Merkblatt/

WA/dl_wa_merkblatt_ICOs.pdf?__blob=publicationFile&v=3

5. Gesetz über elektronische Wertpapiere. [Electronic resource]. – URL:

https://www.gesetze-im-internet.de/ewpg/

6. URL: https://www.bafin.de/SharedDocs/Veroeffentlichungen/DE/Fachartikel/

2021/fa_bj_2107_eWpG.html

7. Frankfurt School Blockchain Center. Industry Insights (CAC23A) – The Le-

gal Frameworks of Tokenized Debt in Europe. [Electronic resource]. – URL:

https://youtu.be/3l0ZemDI1qs?si=UeX6sWGmD

-gIRC5u

8. URL:

https://www.dentons.com/de/insights/alerts/2021/may/10/bundestag-

clears-the-way-for-electronic-securities

9. URL:

https://de.wikipedia.org/wiki/Namenspapier

10. URL:

https://www.fachanwalt-gesellschaftsrecht-hamburg.de/aktuelles-im-

bankrecht-versicherungsrecht-und-anlegerschutzrecht/bitcoin-und-kryptow%C3%A4

hrungen/

;

URL:

https://opendata.uni-halle.de/bitstream/1981185920/80367/1/

BeitraegeTWR_162.pdf

11. Concentrating on warnings to consumers. [Electronic resource]. – URL:

https://www.bafin.de/DE/Verbraucher/GeldanlageWertpapiere/verbraucher_kryptowert

e.html

12. URL: https://gesetze.berlin.de/bsbe/document/KORE223872018

13. Jens der Finanztester. Tether (USDT) kaufen & verkaufen mit Euro. Einzah-

len, Umtauschen, Versenden (bei Binance, Kucoin). [Electronic resource]. – URL:

https://www.youtube.com/watch?v=k7XhMkKpe_U

14. URL:

https://www.estatut.ru/download.php?trid=5087

15. URL:

https://www.rbc.ru/crypto/news/6697b1f79a794796605a1a81

16. URL:

http://publication.pravo.gov.ru/document/0001202403110010

Elena E. Gulyaeva

Diplomatic Academy of the Ministry of Foreign Affairs of Russia

COURTROOM CHRONICLES: AI AND HUMAN RIGHTS

IN INTERNATIONAL LEGAL PRECEDENTS

Abstract.

This article reviews the AI and Human Rights in International Legal

Precedents in Courtroom Chronicles. The object of the study is public relations regu-

lated by both international and national law, which include certain actions for the pro-

vision of artificial intelligence.

Keywords:

artificial intelligence, human rights, cybersecurity, international

law, legal precedents

Digital Technologies and Law (

секция

на

английском

языке

)

______________________________________________________________________________________________________________________

201

Introduction.

In international practice, an increasing number of court cases

each year involve AI-based systems violating certain human rights provisions. Most of

these cases focus on issues such as discrimination and privacy, but the scope of com-

plaints coming to courts is not limited to these areas.

European courts handle the largest number of cases related to protecting human

rights from the adverse effects of AI technologies. In 2014, the Court of Justice of the

European Union addressed the problem of unauthorized personal data collection and

confirmed the illegality of third-party systems interference in people’s private lives,

citing the European Charter of Fundamental Rights and Freedoms. This case marked

the first instance where the Court established a precedent for handling cases involving

AI-based systems.

In 2018, a significant case was presented to the European Court of Human

Rights (ECHR) concerning the UK government's extensive use of AI programs in

mass surveillance and communications interception systems. These systems accessed

private correspondence and other forms of communication of individual users, em-

ploying automated tools for their analysis. After a thorough investigation that included

consultations with subject matter experts, the Court delivered its ruling in 2021. The

ECHR found that the implementation of such AI systems violated Article 8 of the Eu-

ropean Convention on Human Rights, which safeguards the right to privacy, and Arti-

cle 10, which protects the right to freedom of expression.

The Court's decision underscored the lack of effective measures to prevent po-

tential human rights violations and criticized the absence of clear procedures to ensure

that the surveillance measures were necessary and proportionate to their intended pur-

poses [1]. This judgement emphasized the urgent need for specific regulations to gov-

ern the use of surveillance technologies by law enforcement in a manner that respects

fundamental human rights. In 2020, two human rights organizations, AlgorithmWatch

and European Digital Rights (EDRi), specializing in monitoring human rights risks on

the Internet, lodged a complaint against HireVue, a job-selection agency that utilizes

facial recognition software powered by AI algorithms [2]. The complaint alleged that

the use of such systems contravened the General Data Protection Regulation (GDPR)

and anti-discrimination laws. The facial recognition technology and other biometric

data were used to make hiring decisions about potential employees without adequately

informing the candidates involved [3].

The case against HireVue remains pending before the court, and a final decision

has yet to be reached. However, the court's acceptance of the complaint has instigated

a broad investigation into human resources practices by data protection authorities

across several EU countries. This case has highlighted concerns regarding employment

discrimination exacerbated by AI systems and the pressing need for regulatory over-

sight in this domain. In Germany, for instance, the use of predictive analytics by em-

ployment agencies to profile and categorize jobseekers has come under scrutiny [4].

Concerns have been raised that such profiling could lead to discriminatory practices,

particularly against vulnerable groups. As a result, legal measures and reviews of do-

mestic regulations are underway to ensure compliance with anti-discrimination laws

and the protection of individuals' rights to fair and equal treatment.

Digital Technologies and Law (

секция

на

английском

языке

)

______________________________________________________________________________________________________________________

202

An illustrative example of AI's application in Germany is the Hessendata sys-

tem, which employs AI to predict potential “propensity for criminal activity” based on

personal data analysis [5]. Civil liberties groups have expressed concerns about priva-

cy violations and racial profiling associated with the application of such tools. German

courts have examined the system and previously demanded reforms to enhance trans-

parency and accountability [6]. The ongoing legal scrutiny of the system’s functioning

has spurred calls for stricter oversight of AI technologies in tasks that directly impact

human rights.

In Canada, the use of AI in immigration and refugee decision-making has

sparked significant controversy. Agencies have been utilizing AI tools to review visa

applications and assist with asylum claims [7]. Public outcry and calls for greater

oversight and regulation of the use of AI in migration policy have prompted the Cana-

dian courts to address these concerns. While the issue is still under judicial review,

preliminary recommendations have called for increased transparency and specific

oversight mechanisms for AI programs in immigration decision-making to ensure fair-

ness and accuracy in this domain [8].

In 2020, a Dutch court ruled against the government's use of an AI-based risk

assessment system known as SyRI (Systemic Risk Indicator). The tool was designed to

detect welfare fraud by analyzing data from various government databases. The court

found that the system's checks were discriminatory, disproportionately targeting low-

income individuals and minorities, thereby violating their rights to privacy and non-

discrimination. This decision underscores the importance of ensuring that AI systems

do not perpetuate or exacerbate social inequalities, aligning with both national laws

and the European Convention on Human Rights [9]. In its judgment, the Dutch court

underscored the lack of transparency and accountability in the SyRI system, emphasiz-

ing the risks associated with automated decision-making systems that disproportion-

ately impact marginalized groups.

Legal scrutiny of AI-based predictive tools is also prominent in the United

States, where such systems are widely utilized. In particular, PredPol [10], an AI sys-

tem used to predict criminal activity and maintaining public order has faced significant

criticism for exhibiting racial bias in its results [11]. Numerous lawsuits and civil

rights complaints have been filed, with human rights activists arguing that the program

violates the 14th Amendment of the US Constitution by disproportionately targeting

minority communities, thus infringing on their right to equal protection under the law.

There is an ongoing effort to have the system declared unconstitutional, which could

result in its complete prohibition.

In Brazil, the use of facial recognition technology in public surveillance sys-

tems, such as those deployed during the Rio de Janeiro Carnival [12], has sparked con-

siderable controversy as with all the previously mentioned cases. Human rights organ-

izations have taken legal action, asserting that the pervasive use of these technologies

violates citizens’ rights to privacy and freedom of movement. The primary concern is

that without impartial regulatory oversight, facial recognition mechanisms could facili-

tate unlawful tracking of individuals, including political dissidents [13], thereby re-

stricting freedom of expression and assembly. The preliminary rulings resulting from

these lawsuits have called for stricter regulations governing the use of artificial intelli-

Digital Technologies and Law (

секция

на

английском

языке

)

______________________________________________________________________________________________________________________

203

gence in public surveillance and enhanced oversight to prevent the misuse of facial

recognition technology. A parallel issue emerged in South Korea, where civil rights

activists filed a lawsuit in 2020 against the installation of AI-enabled surveillance sys-

tems in Seongnam City. They argued that such systems violated privacy rights and

could lead to discrimination and abuse by law enforcement officials [14]. The court

ordered a halt to the expansion of the AI-based tracking program until a comprehen-

sive review of privacy and civil liberties concerns could be conducted.

On the other side of the world, in Australia, the government's use of an auto-

mated debt collection system known as Robo-Debt [15] has also garnered significant

attention. A legal challenge to the legitimacy of the program was initiated in response

to public concern. The review revealed flaws in the algorithm used to identify over-

payments of social security benefits, which led to erroneous notifications of large

debts to several thousand welfare recipients [16]. In 2019, the Federal Court of Aus-

tralia ruled that the use of the Robo-Debt system was unlawful [17]. It found that the

system not only made calculation errors but also placed the burden of proving inno-

cence on the, on the “debtors”, innocent recipients were required to prove that no non-

existent debts were actually due, thereby violating their rights to proper process and

fairness. A class action lawsuit against the Robo-Debt system culminated in the Aus-

tralian Government agreeing to pay substantial compensation and issue an apology to

the affected individuals.

In 2020, a significant controversy regarding the use of artificial intelligence

emerged in the United Kingdom. Due to the cancellation of final exams as a result of

the COVID-19 pandemic, a decision was made to assign final grades based on a ma-

chine-learning algorithm analyzing data accumulated over the students' educational

careers. The algorithm, which processed extensive student records, ended up dispro-

portionately downgrading students from low-income and disadvantaged backgrounds.

This decision triggered widespread media coverage and public outcry, with allegations

of bias and injustice levied against the use of AI in education [18]. The backlash led to

petitions and legal actions, compelling the government to abandon the use of the algo-

rithm.

The incident underscored the risk that AI systems, often anticipated to be impar-

tial, can exacerbate societal inequalities. It highlighted the urgent need for oversight in

the deployment of such technologies in day-to-day sensitive areas [19]. Similarly, the

integration of AI systems into public administration in France, particularly in sectors

like school placement and social benefits allocation, has faced legal challenges. The

French data protection authority, CNIL, has conducted investigations into several cas-

es where automated systems potentially infringed upon individual rights as outlined by

national and EU data protection regulations. These investigations primarily addressed

concerns regarding the transparency and accountability of AI systems, as well as po-

tential discriminatory impacts on certain demographic groups. As a result of these in-

quiries, CNIL mandated that Clearview AI, the company responsible for developing

and deploying AI tools within public systems, cease the processing of data related to

French citizens and delete all previously collected data [20]. This enforcement action

followed the identification of multiple privacy violations.

Digital Technologies and Law (

секция

на

английском

языке

)

______________________________________________________________________________________________________________________

204

In India, artificial intelligence is employed by the Aadhar biometric identification

system to evaluate citizens' eligibility for various social services. At the time of the pro-

gram’s implementation, critics contended that such mechanisms directly violated priva-

cy rights and posed a significant risk of excluding marginalized communities from ac-

cessing essential services. This concern is particularly pertinent in the context of Indian

society, which continues to grapple with the challenges of eradicating the caste system

[21]. The Supreme Court of India has determined that the Aadhar biometric identifica-

tion system does not violate the constitutional provisions of India. However, the Court

has imposed several restrictions to safeguard privacy and prevent potential misuse. Spe-

cifically, it prohibited the collection of biometric data through bank transactions and

mobile phone data. Additionally, the Court emphasized the necessity for stringent safe-

guards by system operators to mitigate risks of malicious use [22].

Conclusions.

The case studies presented by the courts on human rights viola-

tions and freedoms resulting from the introduction of artificial intelligence-based sys-

tems into everyday life highlight the widespread impact of such technologies on fun-

damental rights. These cases also illustrate the growing tension in protecting these

rights from the harmful impacts of technological innovations. The global recognition

by courts of the potential human rights risks associated with AI underscores the urgent

need to develop and enhance regulatory frameworks. As machine learning tools and

algorithms become increasingly sophisticated, it is imperative to bolster legal and

oversight mechanisms to ensure responsible and ethical AI deployment. Notably, most

legal actions concerning AI-related human rights violations have resulted in decisions

to either ban or restrict the use of technologies that pose significant risks.

Despite the innovative nature of artificial intelligence, courts have utilized a

range of established normative tools–both foundational and emerging–to address hu-

man rights violations attributed to AI systems.

References

1. Big Brother Watch and Others v. The United Kingdom [ECtHR Judgment

Review September 2018 No. 58170/13, 62322/14, 24960/15.]. – 2018. –

14 September. –

URL:

https://european-court-help.ru/obzor-reshenii-espch-za-

sentiabr-2018-goda/

(accessed on 01.08.2024). – Text: electronic.

2. Aszodi, N. Civil society statement: we call on members of the EU Parliament

to ensure the AI Act protects people and our rights / N. Aszodi // Algorithm Watch. –

2023. – 19 April. – URL:

https://algorithmwatch.org/en/civil-society-statement-ai-act-

protects-people-rights/

(application date: 01.08.2024). – Text: electronic.

3. Willner, K. M. Class Action Targeting Video Interview Technology Reminds

Employers of Testing Risks / K. M. Willner, C. S. Murphy // Paul Hastings [website]. –

2022. – 16 February. – URL:

https://www.paulhastings.com/insights/client-alerts/class-

action-targeting-video-interview-technology-reminds-employers-

of#:~:text=The%20complaint%20claims%20that%20the,class%20members'%20facial

%20biometric%20identifiers

(application date: 01.08.2024). – Text: electronic.

4. Lucht, M. Is your use of AI in the workplace compliant and guided by poli-

cies?

/ M. Lucht // Squire Patton Boggs [website]. – 2024. – 21 May. – URL:

https://www.employmentlawworldview.com/is-your-use-of-ai-in-the-workplace-

Digital Technologies and Law (

секция

на

английском

языке

)

______________________________________________________________________________________________________________________

205

compliant-and-guided-by-policies-germany/

(application date: 01.08.2024). – Text:

electronic.

5. Hessendata and its Impact on Personal Data Protection and Privacy. – Police

It [website]. – 2022. – 3 February. – URL:

https://police-it.net/hessendata-and-its-

impact-on-personal-data-protection-and-privacy

(application date: 17.04.2024). – Text:

electronic.

6. Legislation in Hesse and Hamburg regarding automated data analysis for the

prevention of criminal acts is unconstitutional [Judgment of 16 February

2023;

1 BvR

1547/19, 1 BvR 2634/20]. – Bundesverfassungsgericht [official website]. – 2023. –

16 February.

–

URL:

https://www.bundesverfassungsgericht.de/SharedDocs/

Pressemitteilungen/EN/2023/bvg23-018.html

(application date: 01.08.2024). – Text

electronic.

7. Canadian Council for Refugees v. Canada (Citizenship and Immigration)

[2023 SCC 17, 16.06.23]. – Can LII [website]. – 2023. – 16 July. – URL:

https://www.canlii.org/en/ca/scc/doc/2023/2023scc17/2023scc17.html

(application

date: 01.08.2024). – Text : electronic.

8. Canadian Council for Refugees, et al. v. Minister of Citizenship and Immi-

gration, et al. (2021, 2022). Case summary. – LEAF-FAEJ. – 2022. – URL:

https://www.leaf.ca/case_summary/canadian-council-for-refugees-v-canada-2021/

(application date: 01.08.2024). – Text: electronic.

9. Digital welfare fraud detection and the Dutch SyRI judgment. – IAPP [web-

site]. – 2021. – 23 September. – URL:

https://iapp.org/news/a/digital-welfare-fraud-

detection-and-the-dutch-syri-judgment

(application date: 01.08.2024). – Text electronic.

10. Crime and prediction. – Fundamental systems of analysis [website]. –

2020. – 24 April. – URL:

https://fsa3d.com/2020/04/24/crime-prediction-solutions-19/

(accessed 01.08.2024). – Text: electronic.

11. Erickson, D. PredPol: A Case of Mistaken Identity / D. Erickson // Santa

Cruz Works. – 2020. 2 July. – URL:

https://www.santacruzworks.org/news/the-

mistaken-identity-of-predpol

(application date: 01.08.2024). – Text: electronic.

12. Mesquita, C. E. Facial recognition and public security in the city of Rio de

Janeiro: a critical analysis in the perspective of federative competences and fundamen-

tal rights / C. E. Mesquita, C. S. de Teffe // IUS Publicum. – 2022.

13. URL:https://www.researchgate.net/publication/361228313_FACIAL_REC

OGNITION_AND_PUBLIC_SECURITY_IN_THE_CITY_OF_RIO_DE_JANEIRO

_A_CRITICAL_ANALYSIS_IN_THE_PERSPECTIVE_OF_FEDERATIVE_COMP

ETENCES_AND_FUNDAMENTAL_RIGHTS_Eleonora_MESQUITA_CEIA_-

Chiara_SPADACCINI_D (application date: 01.08.2024). – Text: electronic.

14. Coelho, L. Brazil turns facial recognition on rioters despite racism fears / L.

Coelho // Reuters. – 2023. – 12 January. URL:

https://www.reuters.com/

article/idUSL8N33311N/

(application date: 01.08.2024). – Text: electronic.

15.

Seongnam City, Korea's first Corona 19 response 'Artificial Intelligence

Care Call Service'. – Smart City Korea. – 2020. – 9 March. – URL:

https://smartcity.go.kr/en/2020/03/09/%EC%84%B1%EB%82%A8%EC%8B%9C-

%EA%B5%AD%EB%82%B4-%EC%B5%9C%EC%B4%88-%EC%BD%94%EB%

A1%9C%EB%82%9819-%EB%8C%80%EC%9D%91-%EC%9D%B8%EA%B3%

Digital Technologies and Law (

секция

на

английском

языке

)

______________________________________________________________________________________________________________________

206

B5%EC%A7%80%EB%8A%A5-%EC%BC%80%EC%96%B4%EC%BD%9C-

%EC%84%9C/

(application date: 01.08.2024).

– Text: electronic.

16. Burton, T. Robo-debt is a case study in bad government and public service

failure / T. Burton // Financial review. – 2023. – 9 March. URL:

https://www.afr.com/politics/federal/robo-debt-is-a-case-study-in-bad-government-

and-public-service-failure-20230309-p5cqob

(application date: 01.08.2024). – Text:

electronic.

17. Roughly 443,000 Australians received false Robodebt notices. – ABC News

Australia. – 2023. – 8 July. – URL:

https://www.abc.net.au/news/2023-07-

08/robodebt-case-studies/102577632

(application date: 01.08.2024). – Text: electronic.

18. Roughly 443,000 Australians received false Robodebt notices. – ABC News

Australia. – 2023. – 8 July. – URL:

https://www.abc.net.au/news/2023-07-

08/robodebt-case-studies/102577632

(application date: 01.08.2024). – Text: electronic.

19. Carney, T. Unraveling Robodebt: Legal Failures, Impact on Vulnerable

Communities and Future Reforms / T. Carney // The University of Sydney [official

website]. – 2023. – 13 December. – URL:

https://www.sydney.edu.au/law/news-and-

events/news/2023/12/13/unraveling-robodebt-legal-failures-impacts.html

(application

date: 01.08.2024). – Text: electronic.

20. Kolkman, D. What the world can learn from the UK’s A-level grading fias-

co / D. Kolkman // LSE. – 2020. – 26 August. – URL:

https://blogs.lse.ac.uk/impactofsocialsciences/2020/08/26/fk-the-algorithm-what-the-

world-can-learn-from-the-uks-a-level-grading-fiasco/

(application date: 01.08.2024). –

Text: electronic.

21. Walsh, B. How an AI grading system ignited a national controversy in the

U.K. / B. Walsh // Axios. – 2020. – 19 August. – URL:

https://www.axios.com/2020/08/19/england-exams-algorithm-grading

(application

date: 01.08.2024). – Text: electronic.

22. Facial recognition: 20 million euros penalty against CLEARVIEW AI. –

CNIL. – 2022. 20 October. – URL:

https://www.cnil.fr/en/facial-recognition-20-

million-euros-penalty-against-clearview-ai

(application date: 01.08.2024). – Text:

electronic.

23. Khera, R. The Different Ways in Which Aadhaar Infringes on Privacy /

R. Khera // The Wire. – 2017. – 19 July. – URL:

https://thewire.in/government/privacy-

aadhaar-supreme-court

(application date: 01.08.2024). – Text: electronic.

24. Suraksha, P. Aadhaar vetting by private entities against Supreme Court rul-

ing: Experts / P. Suraksha // The Economic Times. 2023. 26 April. – URL:

https://economictimes.indiatimes.com/tech/technology/aadhaar-vetting-by-private-

entities-against-supreme-court-ruling-

experts/articleshow/99769150.cms?from=mdr

(application date: 01.08.2024). Text: electronic.