Who Controls the Upside?
AI safety is usually framed as a technical question: can we make increasingly capable models behave the way we intend? That is an important question. But it is no longer the whole question.
As AI systems become more capable, autonomous and economically consequential, safety is becoming a broader question of control: Over the models themselves, over the institutions deploying them, and over the value they create.
That connects two conversations that are often kept apart: the AI safety debate and the venture-capital economy building around AI. Can we keep powerful systems aligned with human intent? Who owns the systems that become economically indispensable? Increasingly, these are parts of the same story.
Alignment is not the same thing as safety
“Alignment” is often used as if it describes one problem. In practice, it describes several. At its simplest, alignment asks whether a model's behavior reliably matches the intentions or objectives of the people and systems directing it.
That is different from whether the model is capable. It is different again from whether we can detect a failure. And it is different from whether an institution has the authority to intervene when something goes wrong. A useful way to separate the questions is:
This distinction matters because a system can perform well on a narrow task while remaining unreliable under unfamiliar conditions. It can pass an evaluation while behaving differently when the evaluation environment changes.
And a company can have a detailed safety policy while still facing a governance problem when commercial incentives, customers, competitors or governments push in another direction.
Recent research illustrates why the distinction matters. OpenAI and Apollo Research have reported behaviors consistent with “scheming” in controlled evaluations, while Anthropic has studied alignment-faking scenarios in deliberately constructed environments. Both research programs emphasize that these are stress tests and simulated settings, not evidence that frontier models are routinely behaving this way in ordinary deployment.
So alignment is not really a binary question of whether AI is “safe.”
It is a set of questions about how a system behaves, how we know what it is doing, and who has the authority to act when we are uncertain. The frontier is moving from outputs to actions
The practical significance of alignment becomes clearer as models move from answering questions to taking actions. A one-shot response is relatively easy to inspect. An agent operating over hours or days is different.
It can make a sequence of decisions, call tools, interact with other systems, encounter situations its designers did not anticipate and potentially optimize against objectives that were only imperfectly specified.
The unit of evaluation therefore changes. The question is no longer simply:
“Did the model give the right answer?”
It becomes:
“Can we understand, monitor and control what the system does over time?”
That is why the research frontier is moving toward agentic evaluations, monitoring, interpretability and long-horizon testing. Anthropic and OpenAI have both described multi-turn and high-stakes evaluations designed to expose behaviors that ordinary pre-deployment testing may miss.
Recent work on alignment faking, scheming and agentic misalignment is useful not because it proves that these scenarios are already occurring in the real economy. Much of this research deliberately uses constructed or simulated environments.
Its importance is that researchers are turning vague concerns into specific, testable failure modes. That is a meaningful shift. The field is moving from broad arguments about whether AI might become dangerous toward a more operational question:
What exactly could fail, how would we detect it, and what mechanism would allow us to intervene? The answer becomes increasingly important as the systems become more autonomous.
Venture is following the same curve
The AI economy is moving from models to agents to infrastructure. At each step, the economic structure changes. When AI was primarily a software feature, the central question was model performance. As AI becomes embedded into workflows, the advantage increasingly comes from distribution, integration, data, infrastructure and the ability to operate reliably inside existing systems.
The capital intensity of the infrastructure layer is becoming difficult to ignore. PwC estimates that annual data-centre capital expenditure could rise from roughly $800 billion in 2026 to $1.8 trillion in 2050, while its baseline projection puts cumulative AI-infrastructure investment at $31.6 trillion through 2050. Those are forecasts, not realized spending, but they illustrate the scale of capital required to expand compute capacity.
At the company level, the same pattern is visible in financing. AI-infrastructure provider Crusoe raised $3.9 billion at a $30.9 billion post-money valuation in September 2026, with the capital earmarked for expanding its AI infrastructure and “AI factories.”
The technology is becoming more capital intensive at the frontier while becoming cheaper and more accessible downstream.
The OECD's analysis of AI markets finds declining AI prices over time and notes that technological leadership remains contested, while open-source development has helped lower entry costs and put price pressure on incumbents. The OECD reports that its quality-adjusted price index for text-to-text AI models fell by nearly 80% between January 2024 and April 2026.
A 2025 study of AI inference economics found that the cost of achieving a given level of benchmark performance for frontier models had been falling by roughly 5× to 10× per year, driven by hardware and algorithmic efficiency as well as competition.
The result is a more complicated structure than simply “AI becomes centralized.” It may look more like this: Infrastructure becomes more concentrated while applications become more distributed.
A relatively small number of companies can own or finance the expensive underlying infrastructure, while thousands of companies build products on top of increasingly cheap intelligence. That distinction matters for both competition and ownership.
Some small companies are becoming very powerful
The other major shift is organizational. AI changes not only what companies can produce, but how many people they need to produce it.
Carta's latest startup-compensation data shows that the median team size at the seed stage has fallen to four employees. Carta also reports that median initial equity grants for individual contributors rose nearly 11% over two years, while AI/ML engineers saw their median initial equity grants rise 31% between January 2024 and February 2026.
That creates an unusual economic dynamic: More value can potentially be produced with fewer people.
That sounds unambiguously positive until you ask the next question. Who owns the value?
If a company can become large with a much smaller workforce, fewer people participate in the equity pool. The ownership question therefore becomes inseparable from the productivity question.
Carta's 2026 Founder Ownership Report provides a useful picture of the dilution process. Among startups that raised rounds between 2021 and 2025, the median founding team retained about 56% of fully diluted equity at seed and 36% at Series A. By Series C, the median employee equity pool had reached 16.8%, slightly above the 16.1% median founder ownership.
That does not mean employees collectively own more than founders in every company. These are medians across companies on Carta, and ownership varies substantially by company, financing history and sector.
But it does demonstrate something important: The distribution of ownership changes dramatically as capital accumulates. And AI may change the number of people among whom that ownership is distributed in the first place.
Carta's compensation data offers an especially interesting signal. As early-stage companies employ fewer people, the equity pool does not necessarily shrink proportionately. That can leave more equity available per strategically important hire. For AI/ML engineers at startups valued between $1 million and $10 million, Carta reports a 59% increase in median equity-grant size between January 2024 and February 2026. The emerging structure is that fewer employees may participate, while strategically important employees can capture substantially more.
Equity is not the same thing as ownership. There is another complication that rarely makes it into the AI productivity narrative. A company can give employees equity without that equity ever becoming wealth.
More than 70% of U.S. employee stock options reportedly go unexercised, according to a figure cited by Equitybee from Carta's H1 2025 startup-compensation research. That statistic should be treated cautiously because the accessible source is a secondary citation rather than Carta's underlying report.
Private-company equity is not the same thing as cash.
Employees may have to pay to exercise options. They may face taxes before liquidity. They may leave before realizing the value. Options can expire. And a private valuation is not the same as a public-market price at which everyone can sell.
That means the headline value of an equity pool can substantially overstate the wealth actually distributed to employees.
But private-market liquidity is changing.
OpenAI's 2025 tender offer allowed current and former employees to sell approximately $6.6 billion of shares at a $500 billion valuation, according to reporting on the transaction.
The broader trend is important: secondary markets are increasingly allowing employees at major private technology companies to turn part of their paper wealth into cash without waiting for an IPO.
The result is an unusual combination: Fewer employees required to build some highly valuable companies; exceptionally high private valuations at the frontier; substantial compensation premiums for scarce technical talent; and increasingly sophisticated secondary markets that can turn private equity into liquidity before an IPO.
For example, SpaceX's June 2026 IPO was the largest in history. It initially raised $75 billion, before the underwriters exercised their option and brought the total raised to $85.7 billion; the company ended its first trading day valued at more than $2 trillion.
Estimates suggested that more than 4,400 current and former SpaceX employees could become millionaires as a result of the listing, although those employee-wealth estimates were based on analysis of employee holdings rather than a company-disclosed figure.
Market Pulse
September 2026 has produced an unusually direct collision between AI safety and financial markets.
On September 12, OpenAI CEO Sam Altman said the company would not pursue an IPO in 2026 and argued that even a 10% chance of AI contributing to human extinction would be unacceptable. Anthropic CEO Dario Amodei has also publicly called for slowing aspects of frontier AI development.
Markets reacted, and on September 14, Reuters reported that the Philadelphia Semiconductor Index fell 5.9% as investors digested calls from major AI executives to slow development.
That does not establish that safety concerns caused the entire market move; the same Reuters report also points to broader factors including Treasury yields and energy-market disruptions.
But the episode illustrates something new: AI-safety concerns are now capable of becoming an investor concern.
And that matters for the ownership question.
If safety changes the acceptable speed of deployment, it can change the expected demand for chips, data centres and AI infrastructure. A technical question about model behavior can therefore become a question about capital allocation.
That is where safety and ownership begin to converge.
Who Controls the Upside?
The AI debate is often divided into separate conversations: capability, safety, productivity, venture capital and inequality. But they increasingly describe the same system.
AI can make companies smaller while making the infrastructure behind them more capital intensive. It can lower the cost of intelligence while concentrating ownership of the compute required to produce it. It can create enormous new pools of wealth while distributing them among a relatively small number of founders, investors and highly compensated technical employees. And the same systems that create that economic upside may also require new forms of technical and institutional control. That is the part of the AI story worth watching today.