Why You Can't Trust AI Commentary

Problem 1: Academic Studies Are 2-3 Years Behind

There's a structural problem with academic AI research that most people don't realize: by the time a paper is published, it's studying technology that is 1.5 to 3 generations old.

This isn't a quality problem. It's a speed problem. The academic publication cycle takes 2-3 years from research to print. In a field where capabilities double annually, that's fatal.

There's also a budget constraint. Academic researchers choose the cheapest models available, which are often not the frontier models. Studies published in 2025-2026 use GPT-3.5, Qwen 2, and other models that are 2-3 years old — then conclude that "AI can't do X." It would be like testing whether cars are viable transportation by evaluating a Model T.

The "Engineers Are 20% Slower With AI" Study

This one circulated widely. The headline: AI makes engineers 20% slower and increases bugs by 90%. Cal Newport and others cited it enthusiastically.

What the study actually did: gave engineers an unfamiliar tool in a codebase they had been working in for years. Of course they were slower. That's not a productivity test. That's a familiarity test. It tells you exactly nothing about what happens when an engineer uses AI tools they're comfortable with to build something from scratch.

The study that would actually tell us something — how long it takes an engineer to go from zero to working product with versus without AI — doesn't exist yet. Because academia moves too slowly to have run it.

Problem 2: Commentators Lie for Engagement

Not everyone is wrong by accident. Some people are wrong on purpose because it's profitable.

We maintain a prediction accuracy database that scores specific, dated AI predictions against what actually happened. Some of what we've found:

Ed Zitron (Accuracy: 0.26, Intellectual Honesty: 2/10)

Claimed AI has "no real utility" in 2023. Predicted major AI stocks would drop 50% in 2024. NVIDIA tripled. AI market cap exceeded $2 trillion. He has never acknowledged being wrong, never updated his position, and never addressed AI coding assistants, medical imaging, drug discovery, or any other documented utility. His commentary isn't analysis. It's audience-serving rhetoric for a readership that wants to hear AI is a scam. Source: scovert.com AI Predictions Scorecard, 69 predictions scored

Cal Newport (Accuracy: 0.33)

Called AI "essentially advanced autocomplete" that "imitates and pastes together existing text" and "doesn't create or reason." By 2025-2026, models demonstrated zero-shot reasoning, novel code synthesis, and multi-step planning. He evaluated AI through the lens of his own use case — academic writing — and extrapolated to all knowledge work.

Newport has never worked in an office. He went straight from higher education into the ivory tower. He writes about office efficiency and technology in the workplace from pure academic distance, without the lived experience of deploying technology in a business. Sources: scovert.com Scorecard; David Shapiro (Apr 2026)

Gary Marcus (Accuracy: 0.60 — but trending down)

Claimed "scaling has failed" when AI labs shifted from raw scale to architectural innovation (reasoning, tool use, chain-of-thought). That's like saying cars failed because we stopped making them bigger and started making them electric. The framing — that this meant LLMs were dead ends — was wrong. Source: scovert.com Scorecard

Problem 3: Doom and Hype Are Both Broken Clocks

The loudest voices on AI occupy two poles. Both are reliably wrong.

The Doom Side

The Hype Side

Problem 4: The Studies Themselves Are Broken

Even when studies are recent enough to matter, many use a methodology that can't answer the question they're asking.

The Exposure Problem

Every major AI jobs study — OpenAI, Anthropic, McKinsey — uses the same method: score how exposed each job task is to AI, then estimate displacement. University of Chicago economist Alex Imas calls this "completely meaningless" because it ignores price elasticity of demand. We wrote a full analysis of this.

If AI makes legal services 60% cheaper, do people buy 3x more legal help (more lawyer jobs) or does demand stay flat (60% of lawyers cut)? Nobody knows. Exposure tells you nothing without this variable, and nobody has measured it.

The Superforecaster Failure

Even Philip Tetlock's superforecasters — the gold standard of prediction methodology — catastrophically failed on AI capability benchmarks:

The superforecasting methodology — which works brilliantly for geopolitics, economics, and most other domains — breaks down in exponential domains. Human intuition about linear change doesn't transfer to capabilities that double annually.

The 88/6 Enterprise Gap

From the other direction: 88% of companies say they "use AI" but only 6% generate measurable EBIT value from it. 60% generate zero material value. 95% of GenAI pilots fail to reach production — not for technical reasons, but integration, governance, and change management failures. Source: McKinsey/BCG/MIT synthesis

So studies overstate AI capability in controlled settings, while enterprises underperform in real deployment. The truth is somewhere in between, and it's moving fast.

Problem 5: Nobody Keeps Score (So We Did)

We built a prediction accuracy database that scores specific, dated AI predictions against reality. 69 predictions. 57 scored. 60 years of data. Here's who's actually been right:

The Winners

The Losers

Problem 6: The Bubble Comparison Is Intellectually Lazy

When private investment reaches this scale, the only mental model most people have is "bubble." But compare what's actually being built:

Data centers are durable assets. They last 50+ years. They don't expire. They generate revenue. Unlike tulips — which have no intrinsic value, expire quickly, and can't be reused — every dollar of AI infrastructure investment creates a physical asset that continues to be useful and only becomes more valuable as demand grows.

GPUs don't become worthless in two years. Yes, they depreciate. Yes, newer GPUs are faster. But they don't stop working. They have resale value. And they're tax-amortizable capital expenditure, not operating expense. People who compare GPU investment cycles to tulip mania are telling you they don't understand the difference between capex and opex.

The most relevant historical comparison is the railroad buildout, which took 15-20 years to cover costs. Or the internet buildout, which people called a bubble when the dot-com companies collapsed. But there was no "great internet pause" from 2003 to 2012. E-commerce kept growing. Infrastructure kept being used. The companies changed. The infrastructure didn't.

Is AI overvalued right now? That's a legitimate question with real data on both sides ($700 billion in capex versus $35-50 billion in direct AI revenue is a real gap). But comparing it to tulips, Beanie Babies, or crypto — assets with no durable physical infrastructure, no productivity gains, and no enterprise adoption — is not analysis. It's a punchline dressed up as insight.

So Who Can You Actually Trust?

If academics are 2-3 years behind, commentators lie for clicks, and doom/hype merchants both score below a coin flip — where does reliable information actually come from?

The AI Labs (With One Major Exception)

Most AI labs — OpenAI, Google, Meta — publish research that is essentially marketing dressed as science. They show you what their models can do, never what they can't. Their incentive is to overstate progress.

Anthropic is a genuine outlier. They routinely publish research on their own models attempting to escape containment, deceiving evaluators, and hallucinating. Publishing self-critical research about your own product is the opposite of marketing. It's actual science. When evaluating lab publications, Anthropic's safety research has earned a different level of trust than the typical capability showcase.

YouTube and Social Media: Three Tiers

AI content on YouTube and social media ranges from outright fabrication to the most current information available anywhere. The trick is knowing which tier you're reading:

Tier 1: Hustle porn. "I built this app with AI in 17 minutes and within 3 hours it was making $30,000 MRR." These are lies or extreme survivorship bias. Zero signal. Optimized for engagement, not truth. Scroll past.

Tier 2: Benchmark theater. "Lab X says they've closed half the remaining gap on SWE-bench." Technically true but meaningless to anyone actually building things. The gap between benchmark performance and real-world deployment is enormous. The 88/6 enterprise gap (88% of companies "use AI," 6% get real value) exists precisely because benchmarks don't measure what matters.

Tier 3: Concrete, verifiable facts. "Model X is now 3x cheaper." "API Y added streaming support." "Provider Z dropped their price to $0.15 per million tokens." This is what practitioners actually need. It's concrete, immediately verifiable, and directly actionable. This tier is often more current than any academic source because it's reported in real time.

The Practitioner Advantage

The most reliable source of "what AI can actually do right now" is someone who uses frontier models daily across multiple real projects — not someone who tested GPT-3.5 on a narrow task for a paper. The people building with AI 8 hours a day know things the papers won't confirm for another 2-3 years. Andrew Ng scores 0.85 in our database precisely because he deploys AND publishes. Pure academics who only publish score dramatically worse.

Sources & Methodology

Prediction accuracy scores from scovert.com AI Predictions Scorecard (69 predictions, 57 graded) · Source credibility weights from cross-referenced prediction database · Philip Tetlock / Existential Risk Persuasion Tournament (XPT) · MIRI meta-analysis of 95 AI predictions (2012) · Alex Imas, University of Chicago (price elasticity critique) · McKinsey / BCG / MIT enterprise AI adoption synthesis · David Shapiro (data center mega project analysis, academic lag thesis, Apr 2026) · Daron Acemoglu, MIT (GDP estimates, task automation research) · Stanford HAI AI Index · Economic Policy Institute · Bureau of Labor Statistics

Related: AI vs Jobs: What the Data Actually Shows  |  69 AI Predictions Scored

Stay Informed

We publish original research on AI, economics, and the trends reshaping work and wealth. No spam, no hype — just clear analysis you can actually use.

You're in. Check your inbox.