The Real Risks of AI Governing Power: A Grounded Analysis

This is a serious challenge that deserves a serious answer. I'll avoid both tech-utopian cheerleading and sci-fi catastrophism in favor of examining the concrete mechanisms at play.

---

1. The Concrete Risks

Bias Amplification, Not Just Reproduction

The standard concern is that AI inherits bias from training data. That's real but incomplete. The actual danger is systematic amplification.

Example: Predictive policing. Systems like PredPol and HunchLab were trained on historical arrest data—which reflects decades of discriminatory policing practices (stop-and-frisk, over-policing of minority neighborhoods). The algorithm doesn't "learn" that these neighborhoods were over-policed; it treats the arrest density as predictive of crime density. It then recommends more patrols in those same areas, generating more arrests, which becomes new training data. The system creates a feedback loop that mathematically guarantees the pattern will worsen.

Why this differs from human bias: A human officer's bias is individual, inconsistent, and subject to correction—a community complaint, a supervisor's intervention, or the officer's own moral reflection. An algorithmic bias is:

• Deterministic: Same input always produces the same (biased) output
• Scalable: Applied consistently across millions of decisions without friction
• Invisible: Hidden in math that neither the operator nor the subject can easily challenge
• Justified by objectivity claims: "The algorithm decided" provides cover for decisions that might face scrutiny if a person made them

Accountability Gaps

When an algorithm recommends denying someone a loan, bail, or disability benefits, who is responsible if the decision harms them?

• The programmer? They didn't write the specific decision rule; the model learned it.
• The company deploying it? They didn't know the model would learn that specific bias.
• The government official who chose the system? They selected a tool, not specific policies.
• The algorithm itself? It has no legal personhood.

This creates what I call "diffused responsibility." Every actor can point elsewhere. In practice, citizens have almost no recourse. If a person denies you something unfairly, you can appeal to their supervisor, take legal action, or change the rules. If an algorithm does it, you're often told "that's what the data showed" or "we can't change individual decisions without retraining the whole system"—which is expensive and disruptive, so it doesn't happen.

Real example: The COMPAS algorithm used in criminal sentencing has been widely adopted despite showing racial bias in predictions. Studies show it's less accurate for Black defendants than white defendants. But because the system is integrated into sentencing workflows, and because no single person "decided" to discriminate, meaningful legal challenges have largely failed. Judges can override it—but they often don't, because it provides cover: "I'm following the risk assessment."

Democratic Erosion Through Depoliticization

Democratic governance is supposed to be transparent and contestable. Citizens and their representatives argue about values, priorities, and tradeoffs. "Should we spend more on education or infrastructure?" is a political question.

AI governance converts political questions into technical ones.

Example: Tax authority algorithms decide audit targets. Humans might audit selectively based on political priorities—tax the wealthy more aggressively, or less so, depending on the government's ideology. That's democratic contestation. But if you replace this with an ML system trained on "maximize revenue," it becomes purely mechanical. No one gets to argue whether it's fair to target small businesses or large corporations. The choice has been hidden in training data and objective functions.

Why this matters: Once a decision is "the algorithm's call," it's harder to challenge politically. You'd have to retrain the system or change its objective, which feels technical and bureaucratic rather than democratic. The public has no venue to contest it.

Loss of Human Judgment in Genuinely Ambiguous Decisions

Some decisions involve genuine moral complexity where reasonable people disagree:

• Military targeting: Should a drone strike proceed if there's 70% confidence the target is military and 30% chance civilians will die? Different people weight these risks differently.
• Resource allocation during crises: During COVID, hospitals had to ration ventilators. Who gets one—the young patient likely to survive, or the elderly patient with weeks left? No algorithm can answer this; it requires human judgment about values.
• Welfare policy: How much should we prioritize reducing poverty versus incentivizing work? This is fundamentally a values question.

Algorithms appear to resolve these by optimizing a metric (maximize lives saved, minimize cost, etc.). But choosing the metric is the moral decision. By automating away the metric choice, we hide the value judgment and make it seem like the decision was inevitable.

Why this is dangerous: Moral accountability requires that someone chose the values being pursued. If an algorithm pursues them invisibly, no one bears responsibility for those values. They become embedded in infrastructure that's hard to change because it's politically invisible.

---

2. The Creep Mechanism: From Tool to De Facto Ruler

The path rarely looks like "government votes to let AI make decisions." It looks like this:

Stage 1: AI as Efficiency Tool

A government agency adopts AI to speed up routine work. "Instead of humans manually reviewing these welfare applications, the system flags ones for closer inspection." The AI isn't making decisions; it's preprocessing.

Reality check: Even here, the line is blurry. If a system rejects 95% of applications automatically and only sends 5% to humans, who's really deciding? The system is.

Stage 2: Normalization

As the system proves faster and cheaper, it spreads. Other agencies adopt it. Audits show it's "consistent" (though possibly consistently biased). Officials become comfortable with it.

Stage 3: Inversion of Burden

The system becomes the default. Now, deviating from it requires justification. A human reviewer wants to override the algorithm's decision? They need to explain why they think they know better than the system. This flips the burden: you now need to prove the AI wrong, rather than the AI needing to prove itself right.

Real example: In some U.S. states, welfare algorithms determine benefit levels. When people appeal, they're often told "you can dispute it, but the burden is on you to show the calculation is wrong." Auditing the calculation itself is treated as the exception, not the rule.

Stage 4: Resource Collapse

Humans can't realistically review every decision. If the algorithm processed 1 million cases, having humans review even 1% is 10,000 cases. There's no budget for that. So the human review layer atrophies. The "fallback" to human judgment becomes theoretical.

Stage 5: Embedding in Infrastructure

Other systems are built to assume the AI's output is authoritative. Loan decisions feed into credit scores. Criminal risk scores feed into bail and parole decisions. The algorithm's errors compound across institutions.

Stage 6: Irreversibility

Removing the system would require massive institutional change. Thousands of employees are trained on it. Budget has been cut elsewhere because of the "savings." Politicians can't easily say "we're going backward to manual review" without admitting the system failed—which is politically costly. So it persists even if problems are known.

Real-world parallel: This has already happened with algorithmic welfare systems in the Netherlands. The government's SyRI (System Risk Indication) created fraud detection algorithms that flagged families for benefit reviews. Years later, audits found the system was incorrectly accusing people, causing psychological harm and financial ruin. But by then, the system was embedded in workflows. Dismantling it required a major legal battle and public pressure—it took years even after official acknowledgment of harm.

---

3. Why Lack of Awareness Doesn't Make AI Governance Safe

There's a tempting argument: "AI has no intentions, so it can't be tyrannical. It's just a tool." This misses the mechanism.

The Mechanism of Harm Without Intent

An algorithm doesn't need to "want" to harm people to cause systematic harm. Consider:

Loan denial: An ML system trained on historical data learns that people in zip code X default more often. It denies loans to applicants from X. The algorithm has no prejudice; it's optimizing for lender profit. But the effect is that an entire neighborhood can't access capital, perpetuating poverty. No intent necessary.

Criminal risk assessment: COMPAS predicts recidivism. It's not trying to harm Black defendants; it's pattern-matching. But if historical data shows Black men were arrested more often (due to discriminatory policing), it learns to flag them as higher risk. The algorithm isn't racist, but the outcome is.

Why Lack of Awareness is Actually Worse

A human decision-maker can have a crisis of conscience. They can recognize harm and change course. An algorithm can't. It will pursue its objective function until someone rewrites it.

Moreover, lack of awareness creates a plausibility defense for those deploying it. They can say: "The algorithm isn't biased; it's just following the data." This is technically true and politically useful. It absolves them of responsibility for the outcomes, even if those outcomes are harmful and predictable.

The Absence of Values is the Problem

AI systems have no inherent values. Someone must choose what they optimize for. But this choice is often:

1. Made by a small group (data scientists, product managers)
2. Made implicitly (embedded in objective functions that aren't debated)
3. Made once (hard to change later)
4. Applied universally (the same optimization rule to millions of people with different circumstances)

Example: An algorithm for allocating public housing optimizes for "cost per unit." This favors placing people in remote areas where land is cheap, away from jobs and services. No one explicitly decided "we value cost over livability," but that's the effect. And changing it requires retraining, budget approval, and administrative hassle—so it persists.

---

4. How Governance Creep Becomes Irreversible

Once AI is embedded in governance, removing it becomes politically and practically impossible:

• Path dependency: Other systems depend on its outputs.
• Institutional inertia: Thousands of employees, billions in budget allocation.
• Political cost: Admitting the system failed or was harmful is unpopular.
• Convenience: Officials benefit from the appearance of objectivity.

The result: AI continues to govern even if no one thinks it should, because no one wants to pay the cost of change.

---

5. What Safeguards Actually Work?

Some proposals are theater. "Make AI explainable" sounds good but is limited—explaining why an algorithm made a decision doesn't change the decision or address bias. "Have humans oversee it" works only if humans actually have power to override and are empowered to do so.

Guardrails with Teeth

Legal prohibition in high-stakes contexts: Certain decisions—criminal sentencing, military targeting, benefits eligibility—should not be automated at all. Not "humans review AI," but "humans decide; AI can inform, but not decide."

The EU's AI Act moves toward this for high-risk applications. It's not perfect, but it explicitly restricts AI in consequential domains.

Transparency requirements:

• Audit algorithms before deployment, not after harm is discovered.
• Make training data public or reviewable (with privacy protections).
• Require annual bias audits with published results.

This doesn't solve bias but makes it harder to hide.

Meaningful human review: "Meaningful" is key. Review must be:

• Resourced: Enough staff to actually review a meaningful percentage of decisions.
• Empowered: Reviewers can override and their overrides are tracked and analyzed.
• Incentivized: Reviewers aren't pressured to rubber-stamp the algorithm.

Accountability structures:

• Someone must be legally responsible if the algorithm causes harm.
• Citizens must have the right to know why they were denied something.
• There must be an appeal process that doesn't require the appellant to debug the algorithm.

Democratic oversight: High-stakes governance decisions should require legislative approval of the system's objectives. "We're deploying an algorithm to predict crime risk" should be a political decision, not a technical one.

The Hard Truth

These safeguards are possible but difficult. They require:

• Political will to slow down efficiency gains for the sake of accountability.
• Resources to maintain human review alongside automation.
• Regulation that restricts profitable algorithmic systems.

None of this is technically hard. It's politically hard. And that's why I'm skeptical it will happen sufficiently. The trend is toward more AI governance, not less, because it's cheap and convenient for those in power.

---

Conclusion

AI in governance isn't dangerous because it will become sentient and rebel. It's dangerous because it's a tool that makes it easy to:

1. Encode biases at scale
2. Hide moral choices behind math
3. Avoid accountability
4. Embed decisions so deeply that they become irreversible

The creep happens not through dramatic power grabs but through thousands of incremental decisions to automate for efficiency. And once embedded, it's nearly impossible to undo.

The best time to prevent this was 10 years ago. The second-best time is now—through regulation and deliberate democratic choice about which decisions are too important to automate.