Equality before the Law: Legal Judgment Consistency Analysis for Fairness

by   Yuzhong Wang, et al.

In a legal system, judgment consistency is regarded as one of the most important manifestations of fairness. However, due to the complexity of factual elements that impact sentencing in real-world scenarios, few works have been done on quantitatively measuring judgment consistency towards real-world data. In this paper, we propose an evaluation metric for judgment inconsistency, Legal Inconsistency Coefficient (LInCo), which aims to evaluate inconsistency between data groups divided by specific features (e.g., gender, region, race). We propose to simulate judges from different groups with legal judgment prediction (LJP) models and measure the judicial inconsistency with the disagreement of the judgment results given by LJP models trained on different groups. Experimental results on the synthetic data verify the effectiveness of LInCo. We further employ LInCo to explore the inconsistency in real cases and come to the following observations: (1) Both regional and gender inconsistency exist in the legal system, but gender inconsistency is much less than regional inconsistency; (2) The level of regional inconsistency varies little across different time periods; (3) In general, judicial inconsistency is negatively correlated with the severity of the criminal charges. Besides, we use LInCo to evaluate the performance of several de-bias methods, such as adversarial learning, and find that these mechanisms can effectively help LJP models to avoid suffering from data bias.


page 4

page 10


HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models

Fairness has become a trending topic in natural language processing (NLP...

Certifying and removing disparate impact

What does it mean for an algorithm to be biased? In U.S. law, unintentio...

Are Models Trained on Indian Legal Data Fair?

Recent advances and applications of language technology and artificial i...

Beyond Incompatibility: Trade-offs between Mutually Exclusive Fairness Criteria in Machine Learning and Law

Trustworthy AI is becoming ever more important in both machine learning ...

Towards causal benchmarking of bias in face analysis algorithms

Measuring algorithmic bias is crucial both to assess algorithmic fairnes...

Designing Equitable Algorithms

Predictive algorithms are now used to help distribute a large share of o...

A Fair, Traceable, Auditable and Participatory Randomization Tool for Legal Systems

Many real-world scenarios require the random selection of one or more in...

Please sign up or login with your details

Forgot password? Click here to reset