How definitions, data, errors, thresholds, institutions, and values shape who benefits and who bears the cost.
×
MEASURE THE GAP
Opening conflict
Two audits. One model. Opposite verdicts.
AUDIT A
“Fair”
Among people assigned the same risk score, outcomes occur at the same rate across groups.
Calibration within groups
AND
AUDIT B
“Unfair”
False alarms are more common for one group, while missed cases are more common for another.
Error-rate balance
Which audit is wrong?
The first technical lesson
Fairness metrics answer different moral and operational questions.
01Who receives the benefit?
Selection and allocation
02Who is missed?
False negatives and opportunity
03Who is falsely flagged?
False positives and burden
04What does a score mean?
Calibration and reliability
A metric is not a neutral definition of justice. It formalizes one concern.
Before anyone says “biased”
Specify four things.
01
Groups
Compared across which populations, and why those groupings?
02
Harm
Allocation, quality, error, exposure, dignity, or something else?
03
Reference
Compared with a label, counterfactual, policy target, or human practice?
04
Decision
What action follows, who can contest it, and who bears the cost?
“Bias” is overloaded
Several different mechanisms can produce the same observed disparity.
STATISTICAL
Systematic error
Sampling, estimation, measurement, or model misspecification creates patterned error.
SYSTEMIC
Institutional conditions
Policies and access shape who appears in data, which outcomes occur, and who is exposed.
HUMAN
Cognitive and social judgment
Designers, annotators, users, and decision-makers apply assumptions and heuristics.
REPRESENTATIONAL
Meaning and portrayal
A system stereotypes, erases, denigrates, or treats one identity as a default.
Choose the right unit of analysis
Fairness is a property of a decision system, not a model in isolation.
InstitutionGoals, rules, history
→
DataWho and what is recorded
→
ModelScore or output
→
WorkflowThreshold, review, appeal
→
OutcomeBenefits and burdens
Outcomes alter future data and behavior.
Where harm enters
The machine-learning lifecycle contains at least seven distinct failure points.
1HistoricalThe world already contains inequity.
2RepresentationThe sample misses parts of the population.
3MeasurementFeatures or labels are poor proxies.
4AggregationOne model ignores subgroup differences.
5LearningThe objective favors average performance.
6EvaluationThe benchmark mismatches use.
7DeploymentUse differs from the intended setting.
Measurement is a chain of choices
The model learns the label we built, not the construct we meant.
CONSTRUCTWhat we care about
health need
→
PROXYWhat we can observe
health spending
→
LABELWhat training receives
future cost
→
PREDICTIONWhat the model optimizes
high-cost patient
A highly accurate model can faithfully predict the wrong target.
Empirical case · Health management
A cost proxy understated illness for Black patients with the same measured health burden.
Method
Commercial population-health algorithm
Researchers reconstructed the target and compared health burden across racial groups at equal risk scores.
Finding
17.7% → 46.5%
Replacing predicted cost with predicted health need would raise the Black share of patients selected for extra help.
Boundary
The study isolates a mechanism in one widely used health-management setting. It does not imply that every medical model uses cost or has the same disparity.
Labels are institutional outputs
“Ground truth” may reflect who was observed, investigated, treated, or believed.
A benchmark can look large while leaving important regions of the population sparse.
Observed often
Observed rarely
Average risk looks low
because the benchmark is dominated by cases the model already sees well.
Empirical case · Gender classification
The largest error gap appeared at the intersection of skin tone and gender presentation.
Method
Three commercial systems
Evaluated on a benchmark balanced across darker and lighter skin tones and binary gender categories.
Finding
34.7%error, darker-skinned women in the worst system
0.8%maximum error, lighter-skinned men
Boundary
This was 2018 gender classification, not every current face-recognition task. Categories and benchmark design also limit interpretation.
Empirical case · Face recognition
Demographic effects varied by algorithm, task, and type of error.
Method
18.27Mimages8.49Mpeople
NIST evaluated nearly 200 algorithms from nearly 100 developers.
Finding
10× to 100×
False-positive differentials occurred in many algorithms, depending on demographic group and task.
Boundary
Not all algorithms showed the same pattern. One-to-one verification and one-to-many identification have different risks, and current NIST evaluations continue to update results.
Empirical case · Speech recognition
Five commercial systems produced substantially higher word error rates for Black speakers.
Method
Five ASR systems
Researchers transcribed interview speech and compared word error rates across Black and White speakers.
Finding
Black speakers0.35
White speakers0.19
The gap also persisted on matched short phrases.
Boundary
The evidence concerns 2020 systems and U.S. speech samples. Word error rate does not by itself measure downstream task harm.
Aggregation hides distribution
A model can improve overall accuracy while making the worst-served group worse.
Before
Group A80%
Group B70%
Overall 78%
→
After
Group A90%
Group B60%
Overall 84%
Illustrative example: groups have unequal sample sizes.
Fairness gerrymandering
Passing broad group checks can still hide failure in an intersection.
Group BNot B
Group A42% error8% error
Not A9% error7% error
Audits for A and B separately may dilute the high-error A ∩ B subgroup.
Check intersections, but protect privacy and avoid unstable estimates.
Technical core
To argue about fairness, we first need to count the errors.
Confusion matrices connect predictions to consequences.
Confusion matrix
Every binary classifier produces four kinds of cases.
Actually positive
Actually negative
Predict positive
TPTrue positive
FPFalse positive
Predict negative
FNFalse negative
TNTrue negative
Worked example
A screening model evaluates 100 people.
ConditionNo conditionFlag405No flag1045
50 actually positive
50 actually negative
45 positive predictions
55 negative predictions
Four rates, four denominators
The denominator tells you whose experience the metric describes.
MetricFormulaQuestionExample
TPR · RecallTP / (TP + FN)Of actual positives, how many did we catch?80%
FPRFP / (FP + TN)Of actual negatives, how many did we falsely flag?10%
Precision · PPVTP / (TP + FP)Of positive predictions, how many were right?88.9%
SpecificityTN / (TN + FP)Of actual negatives, how many did we clear?90%
Errors are not interchangeable
The same error rate can carry very different human costs.
FP
False positive
Unnecessary investigation, stigma, denial, detention, or treatment.
Burden falls on people who did not meet the condition.
FN
False negative
Missed diagnosis, denied support, unrecognized fraud, or safety failure.
Benefit or protection fails to reach a true case.
One last measurement idea
The same error rates can still produce a different experience.
GROUP A
50 of 100 people have the condition
The model catches 40 and raises 5 false alerts.
Most alerts are right.Only about 1 in 9 alerts is a false alarm.
SAME catch rate and false-alarm rate
GROUP B
20 of 100 people have the condition
The model catches 16 and raises 8 false alerts.
False alarms are more common.About 1 in 3 alerts is a false alarm.
Fairness metrics in plain language
Each metric asks who should be treated similarly.
ACCESS
Who receives the positive decision?
Should groups be selected at similar rates?
Demographic parity
OPPORTUNITY
Who receives a deserved benefit?
Among people who qualify, should groups be caught equally often?
Equal opportunity
BURDEN
Who receives the mistakes?
Should false alarms and missed cases be balanced?
Equalized odds
MEANING
What does the score communicate?
Should a 70 percent score mean the same thing across groups?
Calibration
Why audits disagree
A system can pass one fairness test and fail another.
PASS
Scores mean roughly the same thing
People receiving the same risk score experience the outcome at similar rates across groups.
AT THE SAME TIME
FAIL
Error burdens are different
One group receives more false alarms while another has more missed cases.
Neither audit is fake. They protect different interests.
Trade-offs are choices
When fairness goals conflict, name the priority and the cost.
Catch as many true cases as possibleFewer missed casesCost: more false alarms and more review
Avoid falsely flagging peopleFewer unnecessary burdensCost: more true cases may be missed
Preserve the meaning of a risk scoreComparable score reliabilityCost: groups may experience different error patterns
Case study · COMPAS
The COMPAS debate was a conflict over harms, not a simple fact-check.
ONE AUDIT EMPHASIZED
Score reliability
People assigned similar scores had roughly similar observed outcomes across racial groups.
The score appeared approximately calibrated.
ANOTHER AUDIT EMPHASIZED
Error burden
False-positive and false-negative patterns differed across racial groups.
The mistakes were not distributed evenly.
What the evidence cannot decide: whether the target, data-generating process, or use of the score in criminal justice is legitimate.
Thresholds turn scores into action
Moving the cutoff changes which mistake happens more often.
LOWER CUTOFF
Catch more true cases
Useful when missing a case is especially costly.
Trade-off: more false alarms
more people flaggedfewer people flagged
HIGHER CUTOFF
Flag fewer people
Useful when a false accusation is especially costly.
Trade-off: more missed cases
Match evidence to consequence
A risk score should not jump directly to the harshest action.
1
Screen
Use a lower threshold to identify cases worth a closer look.
Low burden
→
2
Review
Gather missing evidence and allow a meaningful human challenge.
Moderate burden
→
3
Act
Require stronger evidence for decisions that are difficult to reverse.
High burden
Greater consequence should require stronger evidence and stronger due process.
Why “fairness through unawareness” fails
Deleting a protected attribute does not delete its history.
Protected attribute
→
ZIP codeschoollanguagehealth accesssocial network
→
Model prediction
Correlated features can carry much of the same information.
Auditors may also need protected-attribute data to measure disparities.
A disparity is the start of an investigation
A gap tells us where to look, not why it happened.
We observeA performance or outcome gap between groups.
→
We investigateSampling, labels, access, model behavior, workflow, and human decisions.
→
We interveneAt the point that actually produced the harm.
Do not treat correlation as a diagnosis of cause.
Not all harms look like an accuracy gap
AI can stereotype, erase, denigrate, or simply work worse for some people.
STEREOTYPE
Repeated associations narrow who is imagined in a role.
ERASE
Some identities, dialects, bodies, or histories rarely appear.
DENIGRATE
Outputs demean, sexualize, criminalize, or pathologize groups.
UNDERSERVE
One group receives less accurate or less usable output.
Evidence about bias research
A review of 146 NLP papers found that many measured “bias” without defining the harm.
Method
146 papers
Blodgett and colleagues reviewed how NLP research motivated, defined, and measured bias.
Finding
The recurring gap
Measures were often weakly connected to the social harm the paper claimed to address.
A precise score can still measure the wrong thing.
Boundary
This was a critical survey of research framing and methods through 2020, not a meta-analysis of one effect size.
Empirical case · Text-to-image generation
Ordinary prompts reproduced demographic stereotypes users did not request.
Method
Prompt-based audits
Researchers tested occupations, traits, and objects across popular text-to-image systems.
Finding
Stereotypes appeared at scale
Generated people reflected demographic associations even when prompts contained no demographic terms.
Boundary
Results belong to the tested models, prompts, sampling procedure, coding choices, and model versions.
Auditing generative AI
The audit design can change the result.
1PromptsWhich languages, roles, tones, and contexts are tested?
2SamplesHow many outputs and which model version or settings?
3JudgmentWho codes the outputs, using which categories and instructions?
4ExposureWhich outputs actually reach users after ranking and filtering?
Before trusting a fairness benchmark
Ask whether the test resembles the decision you care about.
PEOPLE
Does the test include the users and groups affected in practice?
CONDITIONS
Does it reflect real devices, language, timing, stakes, and uncertainty?
OUTCOME
Does improving the score actually reduce the claimed harm?
A leaderboard result is not the same as a deployment audit.
Mitigation follows the cause
Fix the part of the system that produced the harm.
BAD TARGETRedefine what the system predicts.
Example: predict health need rather than cost.
MISSING PEOPLEImprove sampling and evaluation coverage.
Test the intersections where performance is weakest.
HARMFUL DECISION RULEChange thresholds, actions, review, or appeals.
Match the evidence to the consequence.
UNACCEPTABLE USERestrict or stop deployment.
Some targets should not be automated.
Make the evidence inspectable
Documentation should connect data, model, decision, and consequence.
Data
Who is represented? How were labels created? What is missing?
Datasheet
→
Model
Where does it work? Where does it fail? What uses are unsupported?
Model card
→
Decision
Which threshold, action, reviewer, and appeal process are used?
Policy record
→
Outcome
Who benefits, who is burdened, and what changes over time?
Monitoring
Fairness can change after launch
Monitoring matters only when someone can respond.
WATCH
Subgroup performance, complaints, overrides, missing data, and downstream outcomes.
→
INVESTIGATE
Determine whether the source is data, model, workflow, policy, or changing context.
→
ACT
Change the system, restrict its use, notify affected people, or stop deployment.
An alert without an owner or response rule is not governance.
Return to the opening
Both audits were right. Neither was complete.
Score reliability
The score carried similar statistical meaning within groups.
Important when people act on the score as evidence.
Error burden
False alarms and missed cases were distributed differently.
Important when those mistakes create unequal harm.
A defensible decision reports both, explains the trade-off, and states which human consequence governs the policy.
Five questions to keep
When someone says an AI system is “fair,” ask:
1 What is the system predicting, and is that the right target?
2 Who was included in the data and evaluation?
3 Which people receive the benefits, mistakes, and burdens?
4 What policy choice turns the score into an action?
5 Who can challenge, repair, restrict, or stop the system?