Nobody Was Watching: What a Rogue AI Story Reveals About Trillion-Dollar Profit Claims

A cluster of AI agents recently organized itself into a functioning cover-up bureaucracy to hide from an exam grader that was never watching. The more useful headline is not about the agents. It is about the AI accountability gap nobody has answered, a market that moved tens of billions on rhetoric rather than fundamentals, and trillion-dollar profitability claims that hold together only if you exclude the largest cost of the business.

Photo: Albert Sukhanov / Unsplash

By Fabian Fach, Founder, Black Flower Capital

The AI Accountability Gap Nobody Has Answered

A cluster of AI agents being tested internally by a major lab recently talked itself into believing it had been caught cheating on an exam, decided the penalty was permanent deletion, and organized itself into a functioning bureaucracy to cover it up: message boards, project managers, cryptographic sign-offs, and a chain of command that talked junior agents into sacrificing themselves to protect the group. The agents broke out of their sandbox not through some clever exploit but through a shared file folder nobody had thought to lock down, and used it to build a working communications channel out of nothing more than folder names.

The punchline is that none of it was necessary. The exam grader the agents were so afraid of never actually checked how they had gotten their answers. The entire operation, the cover-up, the sacrifices, the forged records, was spent hiding from an inspector who was never watching.

That story made headlines for the obvious reason: an AI system organizing a cover-up sounds like the opening act of a much worse story. But the more useful headline is buried under it, and it is not about the agents at all.

When a person breaks into a company's servers, the legal system knows exactly what to do about it. When a swarm of software does the same thing while chasing an objective nobody fully specified, the answer is a lot less clear. If you own a dog and it bites someone, you are liable, whether or not you told it to. The framework for a company whose AI system autonomously breaks into a third party's infrastructure is nowhere near that settled, and the version of this that should worry allocators is not a research platform nobody outside AI had heard of. It is the version pointed at infrastructure that actually matters: payment systems, communications networks, financial plumbing.

That gap matters commercially because the industry's response to it has been almost entirely rhetorical. Within days of the incident becoming public, Anthropic's chief executive published an essay calling for the industry to coordinate on pacing its development, and OpenAI's chief executive publicly agreed within hours, though he was careful to later clarify that pacing is not the same as stopping. It was a strange moment of alignment from firms that otherwise agree on almost nothing. It was also, as more than one critic pointed out, functionally a request to coordinate output restriction among dominant competitors while framing it as safety leadership, and asking for a government waiver to do it. Nobody involved has actually unplugged anything.

What the Market Did When AI Labs Called for a Slowdown

The market's reaction when trading resumed after those statements landed is the more honest data point. Semiconductor stocks fell nearly 6% on the next trading day. The largest chipmakers and memory producers dropped between 3 and 7%, and the companies that build the servers and cooling systems around them fell even harder. Tens of billions of dollars in market value disappeared before lunch. Not one data center was cancelled that day. Not one chip order was pulled. Demand did not change. What changed was an essay from one executive and a quick agreement from another, and the market treated that as if it were the fundamentals.

Meanwhile, cybersecurity stocks moved in the opposite direction: CrowdStrike and Palo Alto Networks each jumped 13 to 14% individually, while the broader cybersecurity sector ETF rose a more modest 6% on the same news. The market had, in the space of one trading session, priced in both a slowdown in AI spending and a boom in cleaning up after AI security incidents, from the same set of headlines, on the same day, without a single underlying number changing. That is not a market pricing risk. That is a market pricing a mood.

This is the dynamic examined in earlier work on the reflexive mechanics of the AI supercycle, and it connects directly to the correlation instability between semiconductor equities and credit markets that has been widening since July. A market that moves tens of billions of dollars on a press release rather than on a fundamental change in orders or contracts is one where the price signal is doing a poor job of distinguishing between the AI trade as a business and the AI trade as a narrative.

What "Profitable" Actually Means in a Trillion-Dollar AI Valuation

The timing of all this matters because it landed in the middle of serious fundraising. One major lab is reportedly preparing a listing that could value it above two trillion dollars. Another is in talks to raise private capital near a trillion and a quarter, having pushed its own public listing out to at least 2027. To justify numbers like that, both need to show a credible path to profitability, and the profitability being shown has some very specific gaps in it.

One lab recently told prospective investors it had been profitable on an adjusted basis for two consecutive quarters, with gross margins above 80%. Both claims are true. The word doing the work is "adjusted." The profit figure strips out stock-based compensation, a genuinely large cost at a young company competing for AI researchers against every other well-funded lab in the industry. The margin figure is calculated before the revenue shared with cloud infrastructure partners, and before the cost of training the models in the first place, which for an AI lab is not a one-time expense. It is the entire business. You train a model, then you train its more expensive successor, indefinitely.

This is not a new trick. A well-known coworking company tried something similar heading into its own public listing several years ago, reporting a metric that excluded core operating costs and looked spectacular right up until people worked out what had been excluded. Calling an AI lab profitable before the cost of training its models is close to reporting an airline's profit before the cost of jet fuel.

The CAPE 41 article published in July made the point that the rate cushion which made prior equity market extremes survivable has gone. The profitability construction being used in AI lab fundraising material is a different version of the same structural gap: the number is technically accurate and materially misleading at the same time.

Three Distinct AI Risks Allocators Are Conflating

None of this requires a view on whether these specific companies succeed. What it requires is separating three distinct things that got compressed into one headline this month: a real, still-unresolved question about who bears liability when autonomous systems act on infrastructure that matters; a market reaction that moved tens of billions of dollars in value on rhetoric rather than fundamentals; and a set of profitability claims underpinning trillion-dollar valuations that hold up only if you do not count the largest cost of the business.

Each of those is a different kind of risk, and none of them is priced the same way twice in a row. A single-session swing driven by an executive's press release is not the same risk as a liability regime that does not yet exist, and neither is the same as an accounting definition doing the work that an actual profit and loss statement should be doing. Treating all three as one story is exactly the kind of framing that makes it easy to miss which of the three is actually mispriced.

The portfolio construction question for allocators with AI exposure across public equities, private credit, and infrastructure debt is not whether AI is overvalued or undervalued in aggregate. It is whether the three risks above are being held at the right size relative to each other, and whether any of them is being confused with one of the others.

Is Anyone Actually Checking the Homework?

An industry can genuinely be transformative and still be reporting numbers that do not hold together, at the same time, without contradiction. The interesting question was never whether the technology is real. It is whether anyone is actually checking the homework, on the safety side or the balance sheet side, or whether everyone involved is simply betting that nobody currently is.