health statusfinancial stresswork routinepolitical interestrelationship changes
Low-sensitivity traces can become a high-sensitivity profile when combined.
Privacy as contextual integrity
Privacy depends on whether an information flow fits its context.
SENDERWho has the information?
→
ATTRIBUTEWhat is being shared?
→
RECIPIENTWho receives it?
TRANSMISSION PRINCIPLEUnder what condition, expectation, duty, or purpose does the flow occur?
The same datum can support very different flows
A sleep record changes meaning when the recipient changes.
SLEEP RECORD
Cliniciantreatment under professional dutiesexpected
Wellness apppersonal feedback under product termsconditional
Employerproductivity or scheduling decisioncontested
Advertisertargeting based on vulnerabilityunexpected
Purpose limitation
Data collected to provide a service should not quietly become fuel for another business goal.
What was examined
Online counseling service
The FTC examined BetterHelp’s promises and its disclosures of email addresses, IP addresses, and health-questionnaire information.
What was found
$7.8M
The final order prohibited sharing health data for advertising and required deletion, consent, retention limits, and a privacy program.
What it does not prove
The order resolved allegations and does not establish that every health app follows the same practices. Health-app coverage also depends on the applicable law and entity.
Function creep
A useful data stream can acquire new purposes one small step at a time.
YEAR 1Building access
Open doors securely.
→
YEAR 2Attendance
Estimate who is on site.
→
YEAR 3Productivity
Compare time and movement.
→
YEAR 4Discipline
Trigger employment action.
Each expansion may sound reasonable in isolation. The system becomes something different in aggregate.
Empirical case · Platform surveillance
The FTC found extensive collection, retention, sharing, and weak minimization across major platforms.
What was examined
9 services
FTC compulsory-process orders examined major social-media and video-streaming companies and their data practices.
What was found
Vast surveillance
The report described broad collection on and off platforms, extensive monetization, long or indefinite retention, and inadequate protections for children and teens.
What it does not prove
The report describes the practices of the examined companies during the covered period. It is not a measurement of every platform or current product version.
The economy of personal data
Data create private value while pushing some costs onto other people.
Value to the firm
Better prediction
Targeting and personalization
Fraud detection
Product development
Market leverage
DATA MARKET
Costs borne elsewhere
Loss of bargaining power
Manipulation or exclusion
Re-identification
Future use nobody anticipated
Risk to related people
Consent occurs under information asymmetry: people rarely know every collection path, future use, or consequence.
Current market evidence · Surveillance pricing
Personal data can shape not only which offer you see, but the price or promotion itself.
What was examined
FTC 6(b) market study
Initial staff analysis examined documents from pricing intermediaries using consumer characteristics and behavior.
What was found
250+ clients
Intermediaries worked with at least 250 clients. Inputs could include location, browsing, shopping history, cart behavior, and mouse movements.
What it does not prove
These were preliminary, aggregated findings. The FTC used hypothetical illustrations and did not determine that a particular company acted illegally.
A live 2026 development
California turned many broker opt-outs into one request.
JAN 1Consumers could submit one DROP request to active registered data brokers.
AUG 1Brokers were required to begin retrieving and processing deletion requests.
Deletion includes matched inferences.
Active brokers must check the platform at least every 45 days and report request status, subject to legal exemptions.
Start privacy analysis with a threat model
“Who might misuse the data?” has more than one answer.
PERSONAL DATA + INFERENCES
External attacker
Curious insider
Vendor or partner
Model user
State actor
Authorized institution
Privacy and security overlap, but they are not synonyms
Security protects the system. Privacy protects people from data processing.
SECURITY FAILURE
An unauthorized actor steals prompt logs.
A database is altered or disabled.
A credential exposes a model endpoint.
BOTH
PRIVACY FAILURE
An authorized team reuses prompts for an unrelated purpose.
A model infers a sensitive trait.
A correct profile is used to manipulate or exclude.
De-identification
Removing names reduces risk, but linkage can put identity back.
DIRECT IDENTIFIERS REMOVEDage · ZIP · dates · events · ratings
The dataset still contains patterns that may be rare.
+
SECOND DATA SOURCEpublic records · device data · purchases
An attacker or partner links overlapping attributes.
→
RE-IDENTIFICATION
A record becomes associated with a person again.
De-identification is a managed risk-reduction process, not a magic state.
Models create new privacy attack surfaces
An attacker may learn about training data without seeing the dataset.
MEMBERSHIP
Was this record used?
Infer whether a person or example was part of model training.
INVERSION
What did the model learn?
Use outputs or confidence to reconstruct sensitive features or prototypes.
EXTRACTION
Can the model repeat it?
Query the model until memorized training examples appear.
Empirical case · GPT-2 training-data extraction
Researchers recovered hundreds of verbatim training sequences by querying a language model.
What was examined
Black-box generation attack
Carlini and colleagues sampled GPT-2 outputs, ranked likely memorization, and verified matches against training data.
What was found
Hundreds
Recovered sequences included names, phone numbers, email addresses, code, chat logs, and UUIDs, even when a sequence appeared in one document.
What it does not prove
The attack targeted GPT-2 and a known training corpus. Extractability depends on model, data duplication, size, prompting, and access.
Empirical case · Production language models
Alignment reduced obvious leakage but did not eliminate memorization.
What was examined
Open, semi-open, and closed models
Nasr and colleagues scaled extraction attacks across Pythia, GPT-Neo, LLaMA, Falcon, and ChatGPT.
What was found
150×
A divergence attack made aligned ChatGPT emit training data at a rate 150 times higher than normal chatbot behavior in their experiment.
What it does not prove
This was an adversarial research setup from 2023. Providers, models, defenses, interfaces, and training data have changed.
Enterprise AI has several data stores
The model is only one place where sensitive information can persist.
USER INPUTprompts · files · images · audio
↓
APPLICATIONsession state · identity · tools · agents
↓
RETRIEVALdocuments · embeddings · access metadata
↓
MODEL SERVICErequests · outputs · safety classifiers
↓
OPERATIONSlogs · traces · feedback · analytics · support
Derived representations are still sensitive
Embeddings, summaries, and logs are not privacy-free substitutes for source data.
EMBEDDING
May preserve semantic attributes and enable similarity searches the user never anticipated.
SUMMARY
May turn uncertain or contextual text into a durable claim about a person.
LOG
May collect prompts, outputs, tool calls, identifiers, errors, and operational metadata.
FEEDBACK
May move production conversations into annotation or model-improvement workflows.
Three privacy engineering objectives
A privacy-preserving system should be understandable, manageable, and hard to link unnecessarily.
PREDICTABILITY
Can people and operators form reliable expectations?
Data practices behave as described across the lifecycle.
MANAGEABILITY
Can data and processing be changed?
Access, correct, delete, withdraw, restrict, and review.
DISASSOCIABILITY
Can processing avoid unnecessary identity linkage?
Use data without tying every operation to a person or device.
Control 1 · Reduce the data surface
The safest sensitive record is often the one the system never collected or already deleted.
Collect less
Remove fields, telemetry, precision, or history that the task does not need.
Keep it briefly
Set deletion schedules for prompts, raw files, logs, embeddings, and backups.
Process locally
Keep data on device or within a controlled boundary when central collection adds little value.
For each data element: what decision requires it, and for how long?
Control 2 · Constrain the flow
Separate data, identities, purposes, and authority.
IDENTITYKeep identity keys apart from analytic data where possible.
PURPOSEUse separate stores or pipelines for service, safety, analytics, and training.
ACCESSGrant the minimum role, record every access, and review unusual use.
OUTPUTLimit query detail, bulk export, confidence, and repeated probing.
Access control should follow the decision, not the organizational chart.
Control 3 · Privacy-enhancing technologies
Different technical tools protect against different disclosures.
DIFFERENTIAL PRIVACY
Add calibrated randomness so population statistics reveal less about any one person.
Best for aggregate analysis and some learning settings.
FEDERATED LEARNING
Move training toward devices or local data stores instead of centralizing raw records.
Updates can still leak without added protections.
SYNTHETIC DATA
Generate artificial records that resemble patterns in real data.
May preserve bias or memorize rare examples.
ENCRYPTED COMPUTATION
Compute while data are protected through specialized cryptographic techniques.
Can add cost, latency, and implementation complexity.
Technical controls have boundaries
Privacy risk is reduced, not wished away.
EncryptionProtects data from some observersbut not an authorized harmful use
De-identificationReduces direct linkagebut auxiliary data can restore identity
Federated learningLimits raw central collectionbut model updates can still disclose information
Differential privacyBounds individual contributionbut only under the stated implementation and privacy budget
Part II · Contestability
What happens when the data are wrong, the inference is disputed, or the decision should never have been automated?
An explanation is useful only if the institution can respond.
Explanation is not contestability
“Here is why” is different from “you can change what happens next.”
EXPLAIN
Describe the decision
Show the role of the system, relevant evidence, policy, uncertainty, or important factors.
Outcome may remain fixed.
CONTEST
Open the decision to dispute
Accept corrections and new evidence, route review, reconsider the decision, and provide remedy.
Institution must be responsive.
Five capabilities of a contestable system
People need more than a complaint box.
1NOTICEKnow that AI or data processing shaped the decision.
2ACCESSSee the relevant data, evidence, policy, and decision record.
3CORRECTFix inaccurate or mismatched information.
4CHALLENGEPresent reasons, context, and new evidence to a capable reviewer.
5REMEDYReceive a timely correction, reversal, explanation, or other repair.
Explanations have audiences
The right explanation depends on who must act.
DEVELOPER
Which component failed, shifted, leaked, or produced unstable behavior?
Needs diagnostics and traces.
OPERATOR
What evidence and uncertainty should guide this case?
Needs decision support and escalation.
AFFECTED PERSON
What mattered, what can I correct, and how do I challenge?
Needs reasons and a usable path.
AUDITOR
Does the system comply, discriminate, or produce unacceptable risk?
Needs records, tests, and authority.
Procedural requirement · Credit decisions
A lender cannot use model complexity as an excuse for vague denial reasons.
What was examined
CFPB guidance
The Bureau interpreted ECOA and Regulation B adverse-action requirements for creditors using complex algorithms and AI.
What was found
Specific principal reasons
Creditors must accurately identify the actual reasons for adverse action. A generic checklist or “the algorithm decided” is insufficient.
What it does not prove
The rule is sector-specific and does not require public disclosure of every model detail. It also does not guarantee that a consumer can easily reverse the result.
Empirical case · Face surveillance
Rite Aid’s system generated thousands of false matches, while customers often had no notice or meaningful challenge.
What was examined
Retail face-recognition deployment
The FTC complaint examined enrollment images, match alerts, employee actions, safeguards, and demographic patterns.
What was found
5-year ban
The proposed order barred face surveillance, required biometric deletion, and mandated stronger safeguards for future automated systems.
What it does not prove
The allegations and order concern Rite Aid’s deployment and period of use. They do not establish that every facial-recognition implementation produces the same harms.
Design the dispute pathway
A contestable workflow needs evidence, authority, and recovery.
1DECISION NOTICE
Action, system role, reasons, data sources
→
2CORRECTION
Fix data, identity mismatch, missing context
→
3REVIEW
Capable person with independent evidence
→
4REMEDY
Reverse, repair, notify, prevent recurrence
Case record · response deadline · accessible channel · no retaliation · appeal owner
A live 2026 rights landscape
European law connects data rights, human review, complaints, and explanations of significant AI decisions.
GDPR
Data-subject rights
Access and rectification
Information about processing
Safeguards around certain solely automated decisions
Human intervention and contestation in covered cases
EU AI ACT
Individual AI decision rights
Article 85: complaint to a market-surveillance authority
Article 86: clear and meaningful explanation of the AI system’s role and main decision elements in covered high-risk cases
General application began August 2, 2026, with phased exceptions
Rights remain fragmented across sectors and jurisdictions
The applicable right often depends on the decision, data, sector, and jurisdiction.
CREDITLenders must provide specific principal reasons for adverse actions
HEALTH APPSThe FTC Health Breach Notification Rule can cover apps outside HIPAA
DATA BROKERSCalifornia’s DROP turns deletion requests into operational infrastructure
EUROPEAN RIGHTSGDPR and AI Act protections are broader, but still depend on scope and conditions
Do not translate “legal somewhere” into “protected everywhere.”
Measure whether rights work in practice
A contestability dashboard should track people, not only tickets.
DISCOVERYCan affected people find the process?notice viewed · channel found
ACCESSCan they use it without specialist help?completion · abandonment · accessibility