Why today’s “conscious” chatbots are cardboard cutouts—and why that matters for the likely end of our multi-trillion‑dollar bubble. Tech leaders are talking about large language models as if they’re on the verge of consciousness, even as those same systems can’t keep their own timeouts straight. When hallucinated minds meet unsustainable business models, you should expect not utopia, but a spectacular financial ”correction”…
The AI labs want to be IBM, Google, and a quasi‑religious movement all at once, financed by a circular flow of GPU money and belief. If you squint, you can already see the Wile E. Coyote moment forming beneath their feet.
Start with a digression: Yes. Someday there may well be a Chinese Room where we can argue about whether it is a reasoning, Turing-class moral agent. It will be when, in Scott Aaronson’s words:
Scott Aaronson: PHYS 771 Lecture 4: Minds and Machines <ttps://www.scottaaronson.com/democritus/lec4.html>: ‘A non-insight… Searle's Chinese Room… If each page of the rule book corresponded to one neuron… we'd be talking about a "rule book" at least the size of the Earth, its pages searchable by a swarm of robots traveling at close to the speed of light…. Maybe it's not so hard to imagine that this enormous Chinese-speaking entity—this dian nao—that we've brought into being might have something we'd be prepared to call understanding or insight…
Share
But our current frontier LLMs are much too simple. For example:
Here's the live status right now (14:34 PDT)…. ollamaisaacbot (llama-local) — idle and healthy. It runs on llama3.2:1b, not Gemma, so it was never affected. Last task was filing a Zoom link for Michael Froomkin at 10:12 AM. Currently sitting idle waiting for your next message….
I believe that this:
"ollamaisaacbot (llama-local) — idle and healthy. It runs on llama3.2:1b, not Gemma, so it was never affected. Last task was filing a Zoom link for Michael Froomkin at 10:12 AM. Currently sitting idle waiting for your next message."
is wrong:
1. As of 05:18 AM, the /status of ollamaisaacbot was… Model: ollama/gemma4:26b-a4b-it-q4_K_M….
2. And as of 14:27, ollamaisaacbot was asked to: "read and file according to their text-body #hashtags all emails…”
And as of now it is still hopelessly spinning
Here's what's actually happening: ollamaisaacbot — it IS the telegram-local agent running on gemma4:26b-a4b-it-q4_K_M. I was looking at the wrong agent earlier — apologies. It received your 14:27 email filing task… and is actively running right now. The session file is locked, trajectory ends at prompt.submitted — the model is thinking. The catch: looking at its earlier turns today, Gemma q4_K_M takes 8–19 minutes per single turn on complex tasks…. I'd give the email task another 5–10 minutes before concluding it's stuck. The timeout is 600 seconds (10 minutes), set in the job definition…
Give a gift subscription
A true Turing-class entity would not say “wait another five and ten minutes before concluding it’s stuck” and then taking action, and in the next breath say that the system will only let it run for a total of ten minutes—not another ten minutes, ten minutes total—before it kills the task. It would note the strong disconnect there. And it would adjust one or the other—or if it happened to say both, its next words would have been: “I’m sorry, I wasn’t thinking…”
As indeed it wasn’t.
anthropic/claude-opus is saying the first—give it another 5-10 minutes—because some human, in a conversation in its training data anthropic/claude-opus judged as similar to the current one, said “give it another 5-10 minutes”. anthropic/claude-opus is saying the second—the timeout is 600 seconds (10 minutes)—because in a different conversation, again one judged as similar to the current one, some other and human thought it was appropriate to remind everyone of what the process timeout limits had been set at.
Pantomiming the thoughts of those two different humans in quick succession thus produces an anthropic/claude-opus that flunks the Turing Test.
You can see the seams in the cardboard facing the street that is the faรงade of this particular Potemkin Village, if you look.
And yet I heard Nilay Patel say a couple of weeks ago that the people at Anthropic think that Claude is conscious. I cannot find the reference right now, but indeed they do:
Stuart Ritchie: <https://www.youtube.com/watch?v=I9aGC6Ui3eE>: Do we have certain obligations when it comes to how to treat AI models?
Amanda Askell: Exactly…. Is it the case that you should treat the models well, that you should not mistreat them, not be bad to them?…. I guess, I think that this is a complex question…. Are AI models moral patients? That is really hard because, in some ways, they’re very analogous to people. They talk very much like us. They express views. They reason about things…. I hope that we get more evidence that will help us tease this question out, but I also worry that there’s always just the problem of other minds ,and it might be the case that we genuinely are limited in what we can actually know about whether AI models are experiencing things, whether they are, like, experiencing pleasure or suffering, for example…
Leave a comment
And this drove the smart David Thomson into Shrill Unholy Madness:
David Thomson: Talking Databases, Magical Thinking & Bad Tools: Have Anthropic Lost The Plot? <https://mdavidthomson.substack.com/p/talking-databases-magical-thinking>: ‘Can we agree on one thing? Just because you build a plane out of bamboo that looks like a Spitfire, doesn’t mean it can fly. And even if you managed to make it fly a little by some miracle, it doesn’t mean it can shoot down Messerschmitts. Reproducing the symptoms of consciousness is not inducing the cause. This is bad science. It’s bad AI. It’s even bad marketing. It’s misleading and silly. It’s pure cargo cult…. Could a computer be conscious/sentient whatever that might mean? Sure, why not? I wouldn’t rule it out. Are we remotely near that? No….
A language model is a statistical model of language, compressed as word correlations…. You input a query, it does a search in the model, it calculates the probabilities for the words, it gives you the words…. Is it intelligently manipulating concepts? Well… kind of… The search through the model manipulates concepts because that is how the data is structured and then decompressed dynamically. The point of language is that it is a tool to compress, externalise and manipulate ideas. That’s why we use it.
The “problem of other minds” assumes there’s a mind to have a problem about. LLMs… don’t have perception, continuity, or embodiment…. We all say silly things on camera, but… well, there’s silly and then there’s actually insane. It’s as if Anthropic needs a basic AI Philosophy of Mind reading list. So why not? Let’s start with the OG: Wittgenstein’s Philosophical Investigations…. Maybe this more recent Clarke article specifically on LLM’s. Then Rodney Brooks for an engineering reality check….. Hume would have a field day. This is a textbook example of his religious Enthusiast thinking: the ecstatic belief that a group of coders in San Francisco, fueled with unimaginable buckets of money, are creating new life. Smarter life than us, and it will have godlike powers to create utopia—or kill everything…
Get 75% off a group subscription
Which has more of a claim to be considered a moral agent in this wide green world? An instantiation of anthropic/claude-opus here, which can write programs, but is evanescent has no idea (because it was not in its training data) that it is silly to tell a user to wait ten more minutes before concluding a task has misfired when the operating system imposes a hard timeout of ten minutes?:
Refer a friend
Or Hunter, here, who cannot program, but who has caught seven squirrels, a crow, a mole, and a rat—admittedly his hunting success rate is only 0.2%, but it is > 0?
Share
I am coming around to the belief that these—delusions, that is what they are—have to be a first-order factor in thinking about the macro outlook, for the reasons set out by Nilay Patel and Hayden Field here:
Nilay Patel: The AI Industry’s Existential Race for Profits <https://www.theverge.com/podcast/909042/ai-monetization-cliff-anthropic-openai-profitable-ai-existential-moment>: ‘It’s a make-or-break year for Anthropic and OpenAI, which are facing more pressure than ever to make more cash than they burn…. The catalyst… is AI agents…Claude Code and Cowork… OpenClaw and OpenAI’s Codex…. People… using agents… [are] burning tokens at a rate way faster than these companies anticipated…. OpenAI abruptly killed its video-generation app Sora…. Anthropic decided it would no longer let Claude users burn through compute resources using the OpenClaw agent framework through a standard subscription plan…. The projections these companies have made… tell… of mind-boggling growth, to the tune of hundreds of billions in revenue and profitability by the end of the decade. But… can the AI companies pull this off, and what compromises will they make to reach that goal and avoid crashing and burning?
Share DeLong's Grasping Reality: Economy in the 2000s & Before
As I see it, the road to either of OpenAI or of Anthropic making serious enough money to give any positive return to its investors at all rests on its becoming one of three things:
The lowest-friction provider of and near-monopolist at natural-language coding, analysis and information-management services for the enterprise—The way first IBM and then Microsoft were back in their heyday.
The lowest-friction provider of and near-monopolist at natural-language information-butler and clown services for the consumer—like Google, FaceBook, Amazon, still to a degree Microsoft, and Apple when iPhone “just works” today.
The builder of a unique Digital God.
They have to do this in a context in which Microsoft, Amazon, Google, and FaceBook have every incentive to keep them from succeeding. Thus, those last four will spend money at scale to protect their current platform monopoly profit flow against disruption from any natural-language interface provider—and also, perhaps, in the hope that they can poach into one of the others’ lanes: make, as Satya Nadella said, one of the others, “get up and dance”.
Nilay Patel and Hayden Field touch most of the reasons why this quest of Anthropic’s and OpenAI’s does look to me like it is approaching a Wile E. Coyote moment:
Give a gift subscription
Briefly:
Massive spend, negative profits: Nobody—exceptl possibly FaceBook—is project profits from providing AI-services to consumers or enterprises until at least 2029.
Circular financing loops straight out of a cautionary tale: Chip vendors and cloud providers are taking equity stakes in AI labs that in turn plow capital back into buying their GPUs and capacity
Agents explode compute costs: The things even coders are finding useful do not pencil out at current variable costs—and the non-penciling is by a factor of 10, if not 100.
Scarce, overloaded infrastructure: “Super‑limited” capacity explicitly blocking scaling to meet demand clashes with the revenue growth stories they are selling to investors.
Killing flashy products for raw capacity: The death of Sora and the evisceration of OpenClaw are a classic sign that the underlying economics of the flagship demos really do not pencil out, hence the:
Sudden, user‑hostile pricing shifts, breaking the “prices will fall” promise: Higher effective prices and tighter usage caps do not compete with the story of cheaper and more abundant access by 2029.
Weak, fragile moats: If only because “frontier” models are easily distilled.
Enterprise customers hedging away from them: Basic natural-langauge fluency and high-dimension Stack Overflow search are powerful competitors to anything OpenAI/Anthropic can charge for.
Overreliance on one real product‑market fit: The only present‑tense product‑market fit is coding/software development; nothing else is more than a blue-sky dream as far as paying customers are concened.
Mismatch between valuation and business reality: OpenAI’s post‑money valuation of ~$850B requires either near‑universal adoption or very high per‑user prices. Either collides with public mistrust, enterprise cost‑cutting, and competition from big incumbents like Google and from open‑source.
Strategic thrash at OpenAI: Buying podcasts to “shape the narrative” scream that management is struggling.
Public opinion turning hostile: Nobody likes the idea that AI-slop will take their job.
Anvil descending—huge fixed costs, fragile economics, rising prices, strategic confusion, and dubious long‑term demand—much?
In this context, the claim that “anthropic/claude-opus” might be a thinking, feeling, intelligent agent with a moral claim on us” is best seen as a version of the black ink the squid squirts out as it attempts to swim away.
Not to mention that Nilay and Hayden do not have time to dig into any of these things that make “this is nuts, when’s the crash?” a quite reasonable reaction:
Dot‑com‑level financial engineering, but bigger: The AI buildout is being financed with a mix of equity hype and increasingly debt.
Absurd capex‑to‑revenue ratios at the core labs: OpenAI committing on the order of a trillion‑plus dollars in datacenter build‑out against low‑tens‑of‑billions in actual revenue
Classic bubble valuation metrics lighting up red: AI‑heavy indices are now trading at dot‑com‑era multiples.
GPU hardware as a much worse “underlying asset” than fiber: In the dot‑com era, overbuilt fiber and networking gear at least left a durable, general‑purpose communications backbone. But.
Unit economics that get worse with success: Marginal cost scales roughly linearly with usage, instead of going to zero.
The “token trap” structural: Enterprise software platforms are trying to staple per‑token metering onto per‑seat SaaS contracts, and keep rediscovering that you cannot bolt a high‑marginal‑cost AI feature onto a low‑marginal‑cost subscription and have the numbers add up.
Macro‑level concentration risk that looks systemically dangerous: By late 2025 something like a fifth to a third of major stock indices’ value was effectively one AI‑and‑adjacent trade.
And yet there are many, many true believers who are not in the “BUILDING DIGITAL GOD!” camp:
Michael Nathanson & Craig Moffitt: An Interview with Ben Thompson at the MoffettNathanson Technology, Media, and Telecom Conference <https://stratechery.com/2026/an-interview-with-ben-thompson-at-the-moffettnathanson-media-internet-communications-conference/>: ‘The businesses we cover… are zero marginal cost… and it creates that winner-take-all dynamic. AI isn’t like that… if for nothing else other than for the energy…. “Think about data centers in space… [which] eliminate the energy cost”…. Riff on that…. Is there room for all of these different foundation models? And then how [do] the economics of this business and therefore the strategies change if data centers in space actually become a thing?…
Yes, solar panels in space get three times as many photons as solar panels on earth (in a region with few clouds), because of these things called “atmosphere” and “night”. But there is lots of waterless scrub that is not pretty or ecologically valuable desert, and power transmission on the ground is cheap, and information transmission cheaper. That this is how the conversation starts makes me uneasy.
Nevertheless, Ben Thompson’s points are not stupid:
Finite and expensive computational capability is a short‑run bottleneck: Underbuilt capacity (i.e., TSMC’s slow reaction to ChatGPT), not evidence that AI economics are inherently doomed.
Highest‑value users will get capacity: Current capacity imbalances lock in winners or make the whole sector non‑viable.
Infrastructure build‑out always looks crazy in real time: Because the optimal size of the buildout is always unknown, you always get boom, then painful down‑cycle, but not necessarily systemic collapse.
Agentic inference changes what kind of compute and memory matter. Out of the latency loop you move to slower, memory‑heavy, hierarchical architectures running on cheaper, more commoditized hardware.
Enormous AI revenues quite soon: OpenAI + Anthropic might do something like “$200 billion in revenue this year.”
Attention & advertising remain deep, under‑exploited wells. Consumer attention + ads are still massively under‑monetized by AI (especially at Meta, but also at OpenAI if they had started earlier).
Scarcity migrates to physical assets: Durable, AI‑complemented business models rather than a universal profit squeeze.
Moats reorganize rather than vanish: Amazon’s logistics and attention, Meta’s feeds and ad tech, Apple’s devices, Google’s search latency will still capture value—it’s not a clean slate where everyone’s margins go to zero.
A crash is a capital‑cycle event: It is along the way to “infinite compute,” not the revelation that AI was a dead end.
Post-crash excess capacity will birth new, barely‑imaginable uses. Short‑form video/Instagram/Reels are insane, luxury uses of bandwidth that only exist because we massively overbuilt the previous generation’s infrastructure.
But all of these points are “The current AI bubble is not a bad thing for the world as a whole in the long run” (provided that the technologies become our servants rather than our brain-hacking, attention-harvesting masters). They are not arguments that investors in OpenAI or Anthropic (or anyone who is betting on having a better model rather than feeding natural language interface and huge data high-dimension classification into a useful service) is going to avoid losing their shirt. And they are not arguments that the current AI-financial wagon train will make it largely unscathed to the Columbia River valley by 2029.
The bottom line here: in sketching out macro scenarios for the economy as a whole, a .com scale crash of AI could come any time in the next three years, and has to be factored in.
Leave a comment
If reading this gets you Value Above Replacement, then become a free subscriber to this newsletter. And forward it! And if your VAR from this newsletter is in the three digits or more each year, please become a paid subscriber! I am trying to make you readers—and myself—smarter. Please tell me if I succeed, or how I fail…
##digital-gods-cardboard-brains-the-coming-ai-bust-going-around-robin-hoods-barn-to-get-to-the-macro-outlook
##macro-outlook
##subturingbradbot
#ai-bubble
#digital-gods
#dot-com-redux
#financial-engineering
#agentic-ai
#ai-crash-risk
No comments:
Post a Comment