The Real Risks of AI Governing Power

1. Concrete Risks

Bias amplification at scale. COMPAS, the recidivism-prediction tool used across U.S. courts, was found by ProPublica in 2016 to falsely flag Black defendants as future criminals at nearly twice the rate of white defendants. This isn't a bug — it's the inevitable result of training on historical criminal justice data that encodes decades of racially disparate policing and sentencing. The Netherlands' SyRI system, used to detect welfare fraud, disproportionately targeted low-income and immigrant neighborhoods before a court struck it down in 2020. Australia's Robodebt scheme used automated income averaging to issue hundreds of thousands of false debt notices to welfare recipients, contributing to documented suicides before it was ruled illegal.

Accountability vaporization. When a human judge makes a biased decision, there's a person to appeal to, a reasoning process to challenge, a career to hold accountable. When an algorithm produces a score, nobody is responsible. The vendor says it's just a tool. The agency says it's just following the recommendation. The elected official says they didn't understand the technical details. Responsibility diffuses until it disappears entirely.

Democratic erosion through technocratic capture. Governing requires value trade-offs: liberty vs. security, efficiency vs. equity, present needs vs. future obligations. These are fundamentally political questions that require democratic legitimacy. When AI systems resolve these trade-offs through optimization functions designed by engineers, political choices masquerade as technical ones, removing them from democratic contestation.

Loss of moral reasoning in edge cases. Government constantly encounters situations requiring mercy, contextual judgment, and recognition of human dignity that resist quantification. A caseworker who sees a parent struggling but trying. A judge who recognizes that the letter of the law and justice diverge. A military commander who decides not to strike a legitimate target because something feels wrong. These moments of human judgment — messy, inconsistent, sometimes wrong — are features of governance, not bugs.

Brittleness and correlated failure. Human bureaucracies fail in distributed, uncorrelated ways. When an AI system embedded across agencies fails, it fails everywhere simultaneously. A single flawed model or corrupted dataset can cascade across every domain that relies on it.

2. Why These Risks Are Mechanistically Different

Algorithmic bias creates a fundamentally different problem than human bias for three reasons:

Scale and speed. A biased human judge affects hundreds of cases per year. A biased algorithm affects millions of decisions per day. The harm isn't additive — it's multiplicative.

False objectivity. Human decisions are understood to be subjective, which is precisely why we built appeals processes, judicial review, and oversight mechanisms. Algorithmic outputs arrive wrapped in mathematical authority. When a system produces a number — a risk score of 7, a benefit eligibility of "denied" — it carries an implicit claim of objectivity that suppresses challenge. Research consistently shows that people defer to algorithmic recommendations even when their own judgment is better (a phenomenon called "automation bias" documented extensively in aviation and medicine).

Frozen injustice. Human biases can shift through education, cultural change, and lived experience. A model trained on 2020 data will reproduce 2020's inequities indefinitely unless deliberately retrained — and even then, the training data remains contaminated. Historical patterns become permanent policy. The system doesn't learn from the protest outside the courthouse.

Opacity as structural feature. Many modern ML models aren't interpretable by design. When a neural network denies parole, there is no chain of reasoning to examine. This isn't a temporary technical limitation — it's architecturally inherent to the most powerful models. You can't cross-examine a matrix multiplication.

3. The Creep: From Tool to Decision-Maker

The path is predictable and already underway:

Stage 1 — Advisory. AI provides recommendations that humans review. This is where most systems start. Palantir's Gotham platform assists law enforcement in identifying patterns. AI tools help draft legislation summaries. Humans remain nominally in charge.

Stage 2 — Default acceptance. Under time pressure and caseload volume, humans begin rubber-stamping AI recommendations. Studies of judicial use of risk assessment tools show judges follow algorithmic recommendations roughly 90% of the time. The tool's suggestion becomes the de facto decision. At this point, the AI is governing, but everyone can still pretend a human decided.

Stage 3 — Workflow integration. Systems are designed so that overriding the AI recommendation requires extra steps — additional documentation, supervisor approval, justification memos. Following the algorithm is the path of least resistance. Deviating from it becomes the exception that requires explanation. The burden of proof flips: you no longer need to justify following the machine, you need to justify not following it.

Stage 4 — Institutional dependency. The human expertise to make decisions without the AI atrophies. Staff who understood the old process retire. New hires are trained on the AI-assisted workflow and couldn't operate without it. Budgets are cut because the AI was supposed to enable leaner staffing. Even if someone wanted to revert, the institutional capacity to do so no longer exists.

Stage 5 — Normalization. The AI's role is no longer questioned. It becomes infrastructure — as invisible and unquestioned as the electrical grid. Political leaders reference "the system" or "the data" as though these are neutral forces rather than designed artifacts encoding specific choices.

This isn't speculation. China's Social Credit System followed exactly this trajectory — from pilot programs to normalized infrastructure. U.S. predictive policing tools like PredPol (now Geolitica) moved from advisory dashboards to patrol-determining systems that officers follow by default. The UK's Universal Credit system automated benefit decisions to the point where overriding required escalation that caseworkers rarely pursued.

The key mechanism is efficiency arguments as a ratchet. Each stage is justified by legitimate pressures — budget constraints, caseload backlogs, demands for consistency. No single step feels like surrendering authority. But the cumulative effect is a transfer of governing power to systems that were never elected, never confirmed, and never granted democratic legitimacy.

4. The Awareness Problem

This is the most counterintuitive and most important point: AI doesn't need goals, awareness, or intentions to cause systemic harm. It just needs to optimize.

A sentencing algorithm that optimizes for prediction accuracy will necessarily reproduce and amplify existing patterns of enforcement. It doesn't "want" to be racist. It has no concept of race, justice, or harm. It simply finds that zip code, employment history, and prior arrests — proxies saturated with racial history — are statistically predictive. It does exactly what it was designed to do, and the result is systemic harm.

This is dangerous precisely because there's no malice to identify and no intent to reform. You can educate a biased judge. You can appeal to a prosecutor's conscience. You cannot appeal to a loss function. The system is indifferent in the deepest possible sense — not cruel, not merciful, just computationally mechanical.

The danger of a system with no awareness governing human lives is that it cannot recognize when its own outputs are catastrophic. It cannot notice that its welfare fraud detection is destroying innocent families. It cannot feel unease about a military targeting recommendation. It cannot question whether the policy it's optimizing is just. It will execute an atrocity with the same computational indifference it applies to formatting a spreadsheet.

And because it presents its outputs with consistent confidence — no hesitation, no doubt, no visible discomfort — the humans in the loop receive no signal that something is wrong. A human official recommending mass deportation would show stress indicators, might leak concerns, might resign. An algorithm will produce the logistics plan and move to the next query.

5. Safeguards — Realistic Assessment

What could work:

• Mandatory algorithmic impact assessments before deployment in any governing function, modeled on environmental impact assessments — with public comment periods and independent review. Canada's Algorithmic Impact Assessment Tool is an early, imperfect attempt.
• Hard legal bans on fully automated consequential decisions. The EU's AI Act moves in this direction by requiring human oversight for "high-risk" AI systems. The key is making human review substantive rather than performative — which requires maintaining staffing levels, training, and institutional capacity for independent judgment.
• Sunset clauses and mandatory reauthorization for every AI system in government, forcing periodic reassessment rather than allowing permanent normalization.
• Adversarial auditing by independent parties with full access to training data, model architecture, and outcome data, disaggregated by protected characteristics.
• Preserving institutional capacity for non-AI decision-making so that reversal remains possible. This means deliberately maintaining manual processes, training staff in non-automated workflows, and resisting the budget logic that says AI means fewer people.

What's unlikely to work:

• Voluntary industry self-regulation. The financial incentives for AI vendors to sell to government are too large, and the track record of tech industry self-governance is dismal.
• Transparency alone. Publishing model details helps researchers but doesn't help the welfare recipient whose benefits were cut by an algorithm they'll never understand.
• "Human in the loop" as currently practiced. If the human is a rubber stamp — as current evidence suggests they usually become — then human oversight is theater.

The honest assessment: The slope is genuinely slippery, not because the technology is uncontrollable, but because the institutional incentives all push toward deeper adoption. Governments face budget pressure, staffing shortages, demands for consistency, and political incentives to depersonalize controversial decisions. AI offers all of these. Every safeguard I listed requires actively resisting efficiency, which goes against the grain of every institutional incentive at work.

The most dangerous scenario isn't an AI seizing power. It's a thousand bureaucratic decisions, each individually reasonable, that collectively transfer governing authority to systems that cannot understand what governing means — while everyone involved tells themselves that a human is still in charge.