The AI Safety Trade: Who Pays, Who Profits, and Where the Money Actually Flows

• Two numbers
• How to read this article
• The funding map
• Seven business models
• The issuer-pays problem
• The finance lens: safety as risk management
• Gaps and opportunities
• Where the money is: seven ways to actually make money
• Conclusion
• Sources

Two numbers

In July 2026, Alphabet, Microsoft, Amazon and Meta together guided to roughly USD 720 to 745 billion of capital expenditure for calendar 2026, most of it for AI data centers (datacentres.com, compiling company guidance, July 2026). By the third-quarter results, analysts put a slightly wider group at USD 803 to 828 billion (Hargreaves Lansdown, 2026).

In August 2026, METR, one of the best-known independent evaluators of frontier models, reported about USD 71 million in funding commitments raised over the previous six months. It takes no payment from AI companies for its evaluations, but it says those same companies provide "a significant amount of free tokens" for its work (METR funding update, 14 August 2026).

That gap in scale, and that dependency, sum up the field. The people checking whether frontier AI is safe are small, mostly philanthropically funded, and run on resources lent by the people being checked. Finance has seen this setup before. It was called issuer-pays, and in 2008 it did not end well.

This article maps who actually pays for AI safety, which business models can carry safety work without corrupting it, and where the gaps are. My thesis: safety funding is not mainly a volume problem. It is a payment-structure problem. More money routed through the wrong channel buys more safety-washing. Money routed through deployers, insurers and pooled mandatory mechanisms buys more actual risk reduction.

How to read this article

I use three labels throughout. Documented means a primary source: a filing, a government budget line, a regulation, a funder's own page. Reported means credible secondary reporting I could not trace to a primary document. Analysis means my own reasoning and estimates, which you should treat as a practitioner's view, not data.

Where a number could not be verified, I say so. Several figures that circulate widely in this field, including annual totals for "AI safety spending," rest on community estimates with wide error bars.

"Safety" here covers technical alignment and interpretability, dangerous-capability evaluations, security of model weights and infrastructure, assurance and audit, and the governance work that makes those enforceable. It excludes general "responsible AI" marketing and ordinary content moderation, which is part of why totals differ so much between sources.

The funding map

Philanthropy: the backbone, and concentrated

Philanthropy still funds most independent safety research. The largest single funder is Coefficient Giving, the new name since November 2025 for Open Philanthropy (LongtermWiki profile, reported). In early 2025 it ran a technical AI safety request for proposals and said it expected to spend "roughly $40M" over about five months across 21 research areas (Coefficient Giving RFP page, documented).

Annual totals are harder. Stephen McAleese's widely cited overview put identifiable philanthropic AI safety funding for 2024 at about USD 189 million, of which Open Philanthropy contributed USD 63.6 million (LessWrong, updated January 2025, reported). A later community funder bulletin, partly self-reported, lists Coefficient Giving at around USD 400 million donated in 2025 with about USD 1 billion expected in 2026, the Survival and Flourishing Fund at USD 34.3 million in 2025 (about USD 29 million to AI), and Longview at over USD 60 million (EA Forum AI safety funder bulletin, reported). The same bulletin is not explicit about how much of the Coefficient figure is AI safety specifically, so I treat it as an upper bound.

Two newer pools matter. Humanity AI, a coalition co-chaired by the MacArthur Foundation and Omidyar Network, committed USD 500 million over five years from October 2025, with "security" as one of five priorities alongside democracy, education, humanities and labor (MacArthur Foundation press release, documented). Most of it is not technical safety. And the OpenAI Foundation, which emerged from OpenAI's October 2025 recapitalization holding equity then valued at about USD 130 billion, committed USD 25 billion to health and "AI resilience" (OpenAI, "Built to benefit everyone", documented). No disbursement schedule for the resilience portion had been published that I could find.

Analysis: the structural point is concentration. A few donors, many of them with wealth tied to AI company equity, fund most of the independent field. One commenter on the funder bulletin expects philanthropic flow to rise toward USD 2 billion a year as Anthropic-linked equity becomes liquid (EA Forum, reported projection). That would be welcome, and it would also tie the size of the independent safety budget to the share price of a frontier lab. A safety field that gets richer when the industry booms and poorer when it busts is pro-cyclical in exactly the wrong way.

Frontier labs: the largest budgets, the least disclosure

The labs almost certainly spend more on safety in absolute terms than everyone else combined. They have the models, the compute and most of the senior researchers. But no frontier lab publishes a safety budget, and I could not find an audited figure for any of them.

The scale of what they could spend is documented. Anthropic raised USD 30 billion at a USD 380 billion post-money valuation in February 2026, reporting USD 14 billion of run-rate revenue (Built In, 12 February 2026, reported), then USD 65 billion at a reported USD 965 billion valuation around the start of June 2026, with run-rate revenue above USD 47 billion (Built In, 1 June 2026, reported). OpenAI closed a round of USD 122 billion at an USD 852 billion valuation in March 2026 (Yahoo Finance, reported).

What the labs have disclosed is commitments, and the track record on those is mixed. In 2023 OpenAI pledged 20% of its secured compute to its Superalignment team. Fortune later reported, citing six sources, that the team never received anything close to that allocation and that the pledge had no clear metric (Fortune, 21 May 2024, reported). The episode is older than my 2025-2026 window, but it is the clearest public example of why an undisclosed internal safety budget cannot be relied on: nobody outside can tell whether it was spent.

Labs also fund external work. The Frontier Model Forum's AI Safety Fund, launched in October 2023 by Anthropic, Google, Microsoft and OpenAI with philanthropic partners at over USD 10 million, announced a cohort of 11 grantees receiving more than USD 5 million in December 2025 (Frontier Model Forum, documented). Anthropic and OpenAI are also partners in the UK's Alignment Project (below). These sums are small relative to lab revenue.

Governments: real institutions, small budgets

United Kingdom. The UK has the best-funded dedicated public body. The DSIT research allocations show the AI Security Institute at GBP 66 million for 2025-26 (DSIT R&D allocations 2025/2026, documented). The 2025 Spending Review set aside GBP 240 million for the institute over the settlement period (Spending Review 2025, documented), and Budget 2026 added GBP 120 million over 2026-27 to 2029-30 (Budget 2026, documented). The government's January 2026 progress report says the institute has over 100 technical staff and has tested 30 frontier models (AI Opportunities Action Plan: One Year On, documented). It also leads the Alignment Project, launched in July 2025 with over GBP 15 million from public, philanthropic and industry partners (gov.uk press release, documented); later reporting puts the pool at GBP 27 million after OpenAI and others joined (LongtermWiki, reported, unverified against gov.uk).

United States. The former AI Safety Institute was renamed the Center for AI Standards and Innovation (CAISI) in 2025. Fiscal 2026 appropriations reportedly provided up to USD 10 million for it within a USD 55 million NIST AI line (Granted AI summary of the CJS bill, reported). NIST's June 2026 budget briefing to its advisory committee describes a plan to grow CAISI from USD 11 million to USD 28 million in fiscal 2027 (NIST VCAT, June 2026, documented). Bills to codify the center have advanced in committee without settled funding (Roll Call, 26 June 2026, reported).

The US also has state law. California's SB 53, signed on 29 September 2025, requires developers of models trained above 10^26 operations to publish safety frameworks and transparency reports and to report critical incidents, with penalties capped at USD 1 million per violation; it dropped mandatory third-party audits (Future of Privacy Forum, documented summary). New York's RAISE Act was signed on 19 December 2025 (Governor of New York, documented). Neither funds evaluation directly. They create disclosure, which is cheap for the state and shifts the cost of assurance onto developers.

European Union. The EU AI Act's obligations for general-purpose AI models apply from August 2025, and the Commission's power to fine providers up to 3% of worldwide turnover or EUR 15 million, whichever is higher, applies from 2 August 2026 (AI Act Article 101, Commission service desk, documented; CSA research note, August 2026, reported). The Digital Omnibus on AI, adopted in 2026, postponed high-risk obligations to 2 December 2027 for standalone Annex III systems such as credit scoring and hiring, and to 2 August 2028 for AI embedded in regulated products (Addleshaw Goddard, reported). I found no reliable official budget for the AI Office; headcount estimates range from roughly 150 to over 200, and analysts have called its resources small relative to its mandate (Lawfare, reported).

Switzerland. The Federal Council decided in February 2025 against a horizontal AI act, choosing to ratify the Council of Europe AI Convention and adjust sector law instead (swissinfo, reported). The consultation draft is now expected in spring 2027 (BAKOM, Artificial Intelligence, documented). For financial institutions, FINMA Guidance 08/2024 already expects AI inventories, risk classification, clear accountability and ongoing testing (Lenz & Staehelin summary, reported). On the research side, the Swiss AI Initiative started with a CHF 20 million ETH Domain grant plus GPU hours on the Alps supercomputer, and its large-grant calls list AI safety as a topic, with per-project personnel funding capped at CHF 200,000 in the third call (EPFL, Swiss AI Initiative large grants, documented). Switzerland has no dedicated AI safety institute budget I could identify.

Venture capital: money arrives where there is a buyer

Venture money has moved into safety-adjacent companies, but only where there is a commercial buyer. Goodfire, an interpretability lab, raised USD 150 million at a USD 1.25 billion valuation in February 2026, less than a year after its Series A (Pulse 2.0, reported). Irregular, formerly Pattern Labs, raised USD 80 million led by Sequoia and Redpoint in September 2025 to run security evaluations for frontier labs, at a reported USD 450 million valuation (TechCrunch, 17 September 2025, reported). The Artificial Intelligence Underwriting Company (AIUC), which pairs an agent-security standard with insurance, raised a USD 15 million seed in July 2025 (Insurance Journal, 25 July 2025, reported) and a reported USD 40 million Series A in 2026 (Dealroom, reported).

Analysis: venture capital funds safety as a product. That is useful, and it means the research agenda follows whoever writes the purchase order. For Irregular, that is largely the frontier labs.

Academia

Universities supply people and ideas but rarely large safety budgets. Most academic AI safety work I can identify is funded by the philanthropic and government programs above, plus compute grants. The Swiss AI Initiative's model, compute plus small personnel grants, is typical. Analysis: academia's main constraint is not salaries but frontier model access and compute, which again routes through the labs.

Summary: a rounding error next to capabilities

Analysis: add up the non-lab channels generously and you get low single-digit billions per year. Assume, generously and without any lab having confirmed it, that labs spend a few more billion internally. Total safety spending is then perhaps 0.5% to 1% of the four hyperscalers' 2026 capex alone. The comparison is imperfect: capex is data centers, not research staff, and some of it serves non-AI cloud demand. But it is the best public proxy for the resources going into scaling, and even if you double my safety estimate, the ratio stays below 1 to 50.

Seven business models

Each model below gets the same questions: where does revenue come from, who buys, how far does it scale, and what does the payment structure do to the work?

1. Evaluations and third-party assurance

Revenue logic. Paid assessments of models or deployed systems against dangerous-capability thresholds, reliability targets or standards. Buyers today are labs (pre-release testing), governments (contracts) and, increasingly, enterprises deploying agents. Scalability is moderate: evaluation is labor-intensive at the frontier and becomes more automatable for standard enterprise systems.

Incentive risk. This is the issuer-pays problem in its purest form. When the lab chooses the evaluator, sets the scope, controls access and pays the bill or supplies the compute, the evaluator's next contract depends on the lab's goodwill. METR's policy of refusing cash while accepting tokens is an honest attempt to manage this; a LessWrong critique argues it still does not create real independence (LessWrong, reported). Both views can be true. The tokens are necessary, and they are still leverage.

2. Red-teaming and security

Revenue logic. Offensive testing of models and their infrastructure: jailbreaks, cyber-misuse uplift, weight exfiltration. Buyers are labs, governments and large enterprises. Irregular reports "millions" in annual revenue and over USD 7 million in US government contracts (Calcalist via search summary, reported). Scalability is good because it plugs into existing security budgets, which are large and recurring.

Incentive risk. Lower than for evaluations, because security findings are self-interested for the buyer: nobody wants their weights stolen. The risk is scope. A lab will pay to find vulnerabilities that threaten it, and less willingly for tests of harms that fall on third parties.

3. Interpretability and monitoring tooling

Revenue logic. Tools that inspect model internals, detect anomalous behavior or monitor agent actions in production, sold as software or as research partnerships. Buyers are labs, model developers and, for monitoring, enterprises running agents. Goodfire's valuation shows that investors now see interpretability as a venture category. Scalability is high for monitoring software, uncertain for frontier interpretability research.

Incentive risk. Moderate. The danger is that commercially funded interpretability drifts toward what sells (debugging, steering, performance) rather than what matters most for safety (detecting deception in the most capable systems). Those overlap, but not completely.

4. Compliance tooling

Revenue logic. Software and services that turn regulation into checklists, inventories, documentation and evidence: EU AI Act technical files, FINMA 08/2024 inventories, model-risk validation in banking under frameworks like the Federal Reserve's SR 11-7 (documented), insurer governance expectations. Buyers are deployers in regulated industries, paying from compliance budgets. Scalability is high and the revenue is recurring, because regulation does not expire.

Incentive risk. Box-ticking. Compliance tooling optimizes for passing an audit, which is not the same as reducing harm. The Digital Omnibus delay is also a reminder that demand for this category is policy-timed: when deadlines move, pipelines move with them.

5. AI liability insurance and risk transfer

Revenue logic. Premiums for affirmative cover of AI failures. Armilla launched AI liability cover backed by Lloyd's underwriters including Chaucer in 2025, paying out when a model underperforms its assessed baseline, and in January 2026 said it had raised limits to up to USD 25 million per organization (Armilla, documented; EIN Presswire, company release). Munich Re's aiSure, offered since 2018, insures non-malicious failure of an AI system to perform as intended against agreed metrics (Munich Re, documented).

At the same time, incumbents are pulling back. In late 2025, AIG, Great American and W. R. Berkley were reported to have sought regulatory approval for exclusions of AI-related claims, with one Berkley wording covering "any actual or alleged use" of AI; AIG said it had no plans to implement them (TechCrunch, 23 November 2025, reported). Verisk is reported to be offering standard forms that can exclude generative AI exposures in 2026 (Roughnotes, reported).

How insurers could price AI risk (analysis). The mechanics are standard. Premium equals expected loss plus a load for the cost of holding capital against the tail, plus expenses. Expected loss is frequency times severity. The AI-specific problems are in each term:

• Frequency has no loss history. The substitute is measured performance: error rates under evaluation, drift monitoring in production. This is why aiSure and Armilla price off agreed metrics and pre-cover assessment rather than actuarial tables.
• Severity depends on what the system is allowed to do. An agent that can move money or sign contracts has a fatter tail than a summarizer. Underwriters will price permissions, not models.
• Accumulation is the real constraint. Thousands of insureds running the same foundation model share a common failure mode, much as thousands of firms share one cloud provider in cyber. Chubb is reported to cover some AI incidents while excluding events that could cause widespread simultaneous harm (Tom's Hardware, reported). That exclusion is the market telling you where the systemic risk sits.

Buyers are deployers and AI vendors backing performance guarantees. Scalability is potentially enormous, since insurance is a multi-trillion-dollar industry, but limited today by data and accumulation. Incentive risk is the lowest of any model here: the insurer pays when the system fails, so it wants accurate risk assessment. That is the property the rest of the field lacks.

6. Safety as a product feature inside labs

Revenue logic. Safety work is funded from product margin because it makes models more reliable, more steerable and easier to sell to enterprises and governments. Buyer is the lab's own customer. Scalability grows with revenue.

Incentive risk. Safety gets funded to the extent it is commercially useful, and work that only reduces third-party or catastrophic risk gets funded by conscience and reputation. The safety-washing problem is concrete here: a NeurIPS 2024 paper found that many safety benchmarks correlate strongly with general capabilities, so capability gains can be presented as safety progress (Ren et al., "Safetywashing", documented).

7. Non-profit, focused research organization and hybrid structures

Revenue logic. Grants, donations, government contracts and sometimes a commercial arm. Focused research organizations (time-bound, mission-specific non-profits) and public-benefit corporations sit here, as do governance structures like the OpenAI Foundation's control of OpenAI Group PBC. Buyers are donors and governments. Scalability is limited by philanthropy unless the hybrid owns equity.

Incentive risk. Donor concentration and, for hybrids, the gravitational pull of the commercial arm. The Future of Life Institute's December 2025 index concluded that none of the leading companies had adequate guardrails against catastrophic misuse or loss of control, and in its summer 2025 edition no company scored above D on existential safety (FLI AI Safety Index, Summer 2025, documented; Axios, 3 December 2025, reported). Mission-locked structures have not yet produced visibly different risk outcomes.

Comparison

The issuer-pays problem

The core incentive problem has four precedents in finance. None maps perfectly. All are instructive.

Credit rating agencies

Rating agencies moved to an issuer-pays model in the 1970s. Issuers could shop for the best rating, and agencies competed for mandates. The Financial Crisis Inquiry Commission concluded in 2011 that the agencies' failures were "essential cogs in the wheel of financial destruction" (FCIC Report, 2011, documented). Congress considered a board that would assign raters to issuers instead of letting issuers choose; the SEC studied the idea and did not adopt it (SEC, Report to Congress on Assigned Credit Ratings, 2012, documented).

The parallel to labs choosing their evaluators is direct. If frontier labs can pick among evaluators, set scope and decide what to publish, an evaluator that finds too much risks losing the next mandate.

Financial audit

Audit is also issuer-pays, and it survives because of what was built around it after Enron. Arthur Andersen earned large consulting fees from the client it audited. The Sarbanes-Oxley Act of 2002 created the PCAOB to oversee auditors, restricted non-audit services to audit clients and required partner rotation (SEC Historical Society, reported). The lesson is not that issuer-pays must fail. It can work with a professional body, mandatory standards, inspections of the auditors, liability for negligent opinions, and limits on cross-selling. AI evaluation has none of these yet.

Basel capital rules

Basel took a different route: it does not trust anyone's opinion of a bank's risk, including the bank's own, without a buffer. Minimum capital scales with risk-weighted exposure, supervisors review internal models, and disclosure rules let markets check the result (BIS, Basel Framework, documented). The insight is that the safety mechanism is not a report. It is loss-absorbing capital that the regulated firm must fund upfront.

Insurance

Insurance is the one structure where the party assessing risk pays for being wrong. That aligns incentives well, but only for risks that are diversifiable and quantifiable. For tail risks that are neither, governments have built layered structures: a private primary layer, an industry-funded pool, and a public backstop, as with US nuclear liability under the Price-Anderson Act (US NRC, nuclear insurance fact sheet, documented) and terrorism risk under TRIA (US Treasury, Terrorism Risk Insurance Program, documented).

The strongest opposing view

The best case against my thesis runs as follows. Labs are the only actors with the models, compute and talent to do frontier safety research. Outside evaluators without deep access can only test the surface. Much of the most important safety work, such as alignment training and interpretability at scale, is capabilities-adjacent and cannot be done elsewhere. Enterprise buyers already pay for reliability, so the market funds much of safety without coercion. And mandatory third-party assurance would entrench incumbents who can afford compliance, which is regulatory capture by another route.

Much of that is right. Labs should do most of the research, and access is the binding constraint for outsiders. But none of it answers the payment-structure problem. Research quality and verification are different functions. Banks build their own risk models, and that is fine, as long as someone with independent incentives checks them and capital stands behind the result. The capture argument is real, which is why the remedy should be pooled funding and assigned evaluators rather than a licensing regime that only incumbents can afford.

The finance lens: safety as risk management

What "capital requirements" for AI would actually create

Analysis. Translate Basel into AI terms and, rather than a policy wish list, you get three new vendor categories, each already forming:

• Loss-absorbing capacity. A developer holding a ring-fenced reserve, a surety bond or minimum liability cover needs someone to structure, price and distribute that cover. That is an MGA and specialty-insurance opportunity, not a government program, in the same way catastrophe bonds and surety markets grew once other industries were forced to carry tail risk on their own balance sheet.
• Supervisory review. Independent evaluation of a developer's safety case is currently done by underfunded public bodies (the UK AI Security Institute, CAISI) at a fraction of the scale the job needs. The gap is the opening: a private assurance industry charging developers and deployers for the same service, sitting alongside public oversight the way external audit firms sit alongside the PCAOB.
• Disclosure. Standardized reporting of safety frameworks, incidents and spending is a ratings business waiting to be built. SB 53 and the RAISE Act already require frameworks and incident reports; neither requires spending disclosure. Whoever builds the reporting format that regulators and procurement teams start referencing by default owns a recurring data and certification line, the way Moody's and MSCI own theirs in credit and ESG scoring.

The EU's fine cap of 3% of turnover already behaves like a capital charge. Because it is capped and only levied after the fact, it is a weak deterrent on its own, but as a commercial signal it is useful: it tells you the ceiling a large developer will pay to avoid the fine, and that ceiling is what prices a compliance or insurance product sold against it.

Who ends up holding the tail, and who gets paid to take it

Analysis. Split the loss distribution into layers. Each one already has, or is growing, a paid intermediary:

• Frequent, bounded failures (hallucinated advice, faulty agent actions, discrimination in a credit decision) are moving to the deployer and its insurer. This is the layer aiSure, Armilla and the MGA model in the previous section are built to price, and it is the nearest-term, most fundable opportunity in the whole field.
• Correlated failures across many deployers of one model sit, in theory, with the developer, through warranty, indemnity or mandatory cover, if anyone can get a developer to accept it. No one has yet. Insurers excluding accumulation events are telling you this layer is currently unpriced and uninsured, which also makes it the most interesting long-dated bet for a specialty reinsurer willing to build the loss data first.
• The catastrophic tail (serious misuse, loss of control) is not privately insurable today. Nuclear liability and terrorism risk show the template if a public backstop is ever built: mandatory private primary cover, an industry pool, a public layer above it. Until that exists, nobody is being paid to hold this risk, which also means nobody is being paid to measure it. That gap is the field's largest open advisory and research opportunity, not a reason to wait for government.

The externality nobody prices, which is also the opening nobody has taken

Analysis. Harms that fall on third parties, such as disinformation, cyber uplift to attackers or erosion of shared information quality, are classic externalities, and right now nobody pays for them directly. Read as policy, that is a gap. Read commercially, it is unclaimed ground. A levy proportional to training compute or deployment revenue, paid into an independent pool, is the textbook fix, and if something like it is ever adopted, the EU's fine regime is the closest live precedent, it would fund an evaluation and monitoring industry an order of magnitude larger than today's: an illustrative 0.1% of the hyperscalers' 2026 capex guidance alone would raise roughly USD 0.7 billion a year, more than the UK, US and EU dedicated safety bodies combined. Whether or not that levy ever arrives, the exposure is real and growing, and a firm selling evaluation, monitoring or insurance products does not need to wait for a government to price it before selling it to the companies that already know they are exposed.

Gaps and opportunities

What is underfunded

Analysis, based on the funding map above:

1. Independent evaluation capacity with real access. Public evaluators are funded in the tens of millions; frontier labs raise in the tens of billions. Access to weights and internals is scarce and currently granted as a favor, which means the firms that do manage to negotiate it earn a credibility premium with regulators and insurers that is hard for a later entrant to replicate.
2. Loss and incident data. Insurance cannot scale without loss history. There is no shared, standardized AI incident and loss database comparable to what insurers use for motor or property. SB 53 incident reporting goes to the state, not to underwriters.
3. An assurance profession. Audit has standards, qualifications, inspection and liability. AI assurance has a handful of firms and competing frameworks. Standards like AIUC-1, which AIUC describes as working like SOC 2 for AI agents (Insurance Journal, reported), are early steps.
4. Deployment-side safety in regulated industries. Most funding targets frontier models. Far less goes to the agents banks, insurers and pension funds are wiring into payment, underwriting and advice workflows, where the losses will first show up on a balance sheet.
5. Smaller jurisdictions. Switzerland, a major financial center, relies on general FINMA guidance and sector law, with no dedicated evaluation capacity.

Where capital goes next

No single instrument wins this market. Here is the opportunity set, roughly ordered by how investable it is today:

• Pooled and layered insurance. This is the largest pool of future premium in the sector. Mandatory or not, primary liability cover for frontier deployments, an industry pool for correlated losses, and a priced (rather than implicit) backstop above it are all coming. The MGAs and reinsurers that build the loss data first will set the pricing everyone else has to compete against.
• Assigned-evaluator assurance. If developers above a threshold end up paying into a pool with accredited evaluators assigned rather than chosen, as the SEC once studied and declined for credit ratings, whoever builds the accreditation and panel infrastructure in a given jurisdiction effectively owns the assurance market there. AI is a better candidate than ratings were, because the number of covered developers is small and identifiable.
• Advance market commitments. Governments and large deployers committing in advance to buy evaluation suites, monitoring tools or interpretability capability that meets a defined bar creates a revenue line that does not depend on the labs' goodwill, and locks in a buyer before competitors can underbid on price.
• Safety-spend disclosure as a ratings product. A standard, audited format for safety and security spending, headcount and compute would do for AI safety what credit ratings and ESG scores did elsewhere: turn a transparency requirement into a recurring data and licensing business for whoever defines the standard first. Pledges without metrics, as the 2023 compute commitment showed, generate headlines, not accountability, which is exactly the gap a credible rating fills.
• Prizes. Smaller as a direct revenue mechanism, but useful for a firm's positioning: a prize for a specific, verifiable milestone, such as detecting planted deceptive behavior in a frontier-scale model, buys talent access and credibility that convert into paid mandates later.

Where the money is: seven ways to actually make money

The survey above says which models exist. This section answers a narrower question: if you are a founder, a small firm or an independent practitioner, where can you build a business that pays its bills and does real safety work? Everything here is analysis. Prices are illustrative shapes, not market data.

The honesty test

One test sorts good safety businesses from bad ones: does your revenue go up when you find more problems? An insurer earns more by pricing risk correctly. A bank's model-validation function looks better when it catches a flawed model before the regulator does. A lab-paid evaluator, by contrast, risks its next mandate every time it reports something uncomfortable. Build where the person signing the check wants bad news early.

A second filter: is there a forcing function? A regulatory deadline, a procurement requirement, an insurance condition or a board mandate turns "nice to have" into a budget line. In 2026 the main ones are the EU AI Act high-risk deadline of 2 December 2027 for uses like credit scoring and hiring, FINMA's AI governance expectations for Swiss banks and insurers, model-risk rules in banking, and insurers asking for evidence before they write AI cover.

1. AI model-risk validation for banks and insurers

What you sell. Independent validation of specific AI use cases (a credit pre-screening model, a claims-triage agent, an advisory chatbot): test design, performance and bias testing, failure-mode analysis, and a documented evidence pack the second line of defense and the regulator can read.

Who signs. The Chief Risk Officer or head of model risk. In banks this function already exists and already buys external validation for traditional models.

Pricing shape. Fixed fee per use case for the first validation, then an annual revalidation retainer per use case. Revenue scales with the number of AI use cases in production, which is rising at every institution I talk to.

Why it works. It passes the honesty test, it slots into an existing budget, and the regulatory language (inventory, risk classification, ongoing testing) is already written. The skill set is rare: you need people who understand both models and the balance sheet the model touches.

2. Agent runtime controls and monitoring

What you sell. Software that sits between an agent and the systems it touches: permission scopes, spending and action limits, human-approval gates for high-impact steps, full action logs, anomaly alerts and a kill switch.

Who signs. The Chief Information Security Officer, often with the head of operations.

Pricing shape. Subscription per agent or per volume of monitored actions, with an enterprise tier for audit exports. Usage-based pricing grows automatically as the customer deploys more agents.

Why it works. It fits security budgets, which are large and recurring. The logs it produces do three jobs at once: operational control, compliance evidence and underwriting data. That makes the product hard to rip out.

3. Underwriting partner or managing general agent

What you sell. Either the pre-cover assessment and continuous monitoring that lets a carrier write AI liability cover, or, more ambitiously, the cover itself as a managing general agent (an intermediary that underwrites on a carrier's balance sheet). Armilla, backed by Lloyd's syndicates, and AIUC are the visible examples.

Who signs. The carrier pays for risk data; as an agent you earn a commission on premium written, and the end customer pays the premium.

Pricing shape. Assessment fees plus commission, recurring at every renewal.

Why it works. It has the best-aligned incentives in the field: everyone in the chain loses money if the risk assessment is wrong. It also has a data flywheel. Every policy written adds monitoring and claims data that improves pricing for the next one. The barrier is regulatory licensing and earning a carrier's trust, so it suits teams with insurance backgrounds.

4. Own a standard and certify against it

What you sell. A published control standard for AI systems, the certification audits against it, and accreditation of other auditors who want to certify. The template is SOC 2 in cloud security: a certificate that procurement teams started asking for until it became table stakes. AIUC-1 explicitly follows that pattern.

Who signs. The AI vendor pays for certification because its enterprise customers demand it.

Pricing shape. Annual certification fees, auditor accreditation and licensing fees, and training.

Why it works. Network effects: once buyers ask for the certificate in requests for proposals, vendors must buy it. The risk is the issuer-pays trap, since the vendor pays. The standard only stays credible if certification is withdrawn when controls fail and insurers or regulators recognize it. A standard that never fails anyone becomes a logo, not a safeguard.

5. Buyer-side evaluation for AI procurement

What you sell. Testing of an AI vendor's product on behalf of the company buying it: does the contract-review agent actually perform on our documents, in our languages, under our edge cases?

Who signs. The procurement, IT or business owner at the buying company.

Pricing shape. Fixed fee per vendor assessment, plus a subscription for continuous retesting as vendors ship new model versions, which they now do every few months.

Why it works. It flips issuer-pays into investor-pays: the party paying wants the truth because it bears the cost of a bad purchase. Every model update creates a new retest, so revenue recurs naturally.

6. Sector-specific red-teaming

What you sell. Adversarial testing for harms generalist labs do not understand well: fraud and social engineering through banking chatbots, market-abuse patterns in trading assistants, mis-selling in advisory agents, prompt-injection paths into payment workflows.

Who signs. Security, fraud and compliance heads in the regulated firm; occasionally the AI vendor before a large sale.

Pricing shape. Project fees per engagement, converting into quarterly retainers.

Why it works. Domain knowledge is the moat. A team that knows how payment fraud actually works will find attack paths a generalist red team misses, and buyers will pay a premium for that.

7. Government and public-interest contracts

What you sell. Evaluation suites, testing infrastructure and specialist assessments to public bodies such as the UK AI Security Institute, CAISI and the EU AI Office (METR, for instance, reports a small AI Office technical-assistance contract alongside its donations).

Who signs. Public procurement.

Pricing shape. Contracts and framework agreements: lumpy, slow to win, but credible and independent of the labs.

Why it works. Not as a whole business, but as a stabilizer. A minority share of public revenue diversifies away from both lab money and donor concentration.

Comparison

How to sequence it

The practical path for a small team is services first, software second. Start with validation or red-teaming engagements: they are cash-generating from the first contract, they teach you what buyers actually worry about, and they reveal which steps you repeat every time. Turn the repeated steps into software (test harnesses, monitoring, evidence-pack generation), then sell the software to customers you never meet. The services margin funds the product; the product lifts the margin of the services.

For an independent practitioner, the strongest position is the narrowest one: one sector, one regulatory regime, one type of system. "AI agent validation for Swiss insurers under FINMA guidance" is a business. "AI safety consulting" is a hobby with invoices.

Two models I would avoid as a primary revenue line: evaluation sold mainly to frontier labs, which fails the honesty test, and compliance tooling whose pitch is "pass the audit faster," which sells the appearance of control rather than control.

Conclusion

AI safety is not underfunded because nobody cares. It is underfunded and mis-structured because the parties who pay for it are mostly the parties it constrains, or donors whose wealth rises and falls with those parties. Finance has run this experiment. Issuer-pays without standards, liability and independent capital produced the rating failures of 2008. Issuer-pays with a professional body, inspection and liability produced an audit system that, for all its flaws, mostly works. Insurance works best of all where it can, because the assessor pays for being wrong.

The thesis, restated commercially: the payment structure decides which safety businesses are fundable, not just which ones are honest. A revenue line paid by a deployer, an insurer or a pooled mechanism is a more durable business than one paid by the developer being assessed, because the buyer's incentive and the business's incentive point the same way. That is not a moral preference. It is the difference between a client who renews and one who eventually stops calling.

Takeaways:

1. Operators: run AI systems through the same model-risk discipline you already apply elsewhere. Keep an inventory, validate independently, monitor in production, and read your insurance renewals closely: an AI exclusion is a repricing event, and the cost of the gap it leaves does not disappear just because nobody sent you an invoice for it.
2. Investors: price safety businesses on who pays, not on the mission statement. Deployer-paid, insurer-paid and regulator-mandated revenue compounds; developer-paid revenue is a single-customer business with a built-in conflict, however good the team.
3. Founders: the biggest unclaimed categories are the ones this article maps as investable: pooled insurance, assigned-evaluator assurance, and a safety-spend ratings product. Each rewards whoever builds the loss data or the standard first, and each gets harder to enter every year it stays unclaimed.
4. Policymakers: a developer-funded, assigned-evaluator pool is not only a safety measure, it seeds a domestic assurance and insurance industry that jurisdictions without one will have to import later. Treat it as industrial policy, not as a cost.
5. Insurers: the firm that builds the shared loss and incident database first gets to price this market for everyone who comes after. That first-mover advantage is worth more than the near-term cost of writing the early, underpriced policies.

Sources

All links accessed in November 2026. "Reported" sources are secondary; figures from them are labeled in the text.

Capabilities spending and lab finances

• datacentres.com. "Hyperscaler capex trends: over $700 billion for 2026." July 2026.
• Hargreaves Lansdown. "AI hyperscalers are expected to finish the year spending a whopping $828 billion on capex." 2026.
• Built In. "Anthropic bags $30B funding round at $380B valuation." 12 February 2026.
• Built In. "Anthropic raises $65B Series H, achieves $965B valuation." 1 June 2026.
• Yahoo Finance. "OpenAI closes $122bn funding round at $852bn valuation." March 2026.
• OpenAI. "Built to benefit everyone." October 2025.
• Fortune. "OpenAI promised 20% of its computing power to combat the most dangerous kind of AI, but never delivered." 21 May 2024.

Philanthropy and independent research

• Coefficient Giving. "Request for Proposals: Technical AI Safety Research." 2025.
• LongtermWiki. "Coefficient Giving."
• McAleese, S. "An Overview of the AI Safety Funding Situation." LessWrong, updated January 2025.
• EA Forum. "AI safety funder bulletin."
• MacArthur Foundation. "Humanity AI commits $500 million to build a people-centered future for AI." October 2025.
• METR. "Funding update." 14 August 2026.
• LessWrong. "Is METR a meaningful check on Anthropic?"
• Frontier Model Forum. "AI Safety Fund."
• Future of Life Institute. "AI Safety Index, Summer 2025."
• Axios. "AI firms flunk existential risk planning, new report finds." 3 December 2025.
• Ren, R. et al. "Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" NeurIPS 2024.

Governments and regulation

• UK DSIT. "DSIT R&D allocations for 2025/2026."
• HM Treasury. "Spending Review 2025."
• HM Treasury. "Budget 2026."
• UK Government. "AI Opportunities Action Plan: One Year On." January 2026.
• UK Government. "AI Security Institute launches international coalition to safeguard AI development." July 2025.
• LongtermWiki. Alignment Project funding update.
• NIST. "Budget updates and discussion," VCAT. June 2026.
• Granted AI. "NIST $55M for AI standards and safety research." 2026.
• Roll Call. "Federal AI security center measure advanced by House Science." 26 June 2026.
• Future of Privacy Forum. "California's SB 53: The First Frontier AI Law, Explained."
• Governor of New York. "Governor Hochul signs nation-leading legislation to require AI frameworks for AI frontier models." 19 December 2025.
• European Commission AI Act Service Desk. "Article 101: Fines for providers of general-purpose AI models."
• Cloud Security Alliance. "EU AI Act GPAI enforcement goes live." August 2026.
• Addleshaw Goddard. "EU AI Act: AI Omnibus formally adopted." 2026.
• Lawfare. "How much power does the EU AI Office actually have?"
• swissinfo.ch. "Swiss government wants to adopt Council of Europe AI conventions." February 2025.
• BAKOM. "Artificial intelligence."
• Lenz & Staehelin. "FINMA issues guidance on AI use in financial institutions."
• EPFL. "Swiss AI Initiative: Large Grants."
• Board of Governors of the Federal Reserve System. "SR 11-7: Guidance on Model Risk Management."

Venture capital and insurance

• Pulse 2.0. "Goodfire: $150 million Series B at $1.25 billion valuation." February 2026.
• TechCrunch. "Irregular raises $80 million to secure frontier AI models." 17 September 2025.
• Calcalist. Irregular coverage. 2025.
• Insurance Journal. "AIUC emerges from stealth with $15M seed." 25 July 2025.
• Dealroom. "AIUC lands $40M Series A to insure AI agents." 2026.
• Armilla. "Armilla launches affirmative AI liability insurance with Lloyd's underwriter Chaucer." 2025.
• EIN Presswire. "Armilla AI raises Lloyd's-backed coverage to $25M." January 2026.
• Munich Re. "Insure AI."
• TechCrunch. "AI is too risky to insure, say people whose job is insuring risk." 23 November 2025.
• Tom's Hardware. "Insurers move to limit AI liability as multi-billion dollar risks emerge." 2025.
• Rough Notes. "AI exclusions."

Financial-regulation precedents

• Financial Crisis Inquiry Commission. The Financial Crisis Inquiry Report. January 2011.
• US Securities and Exchange Commission. Report to Congress on Assigned Credit Ratings. December 2012.
• SEC Historical Society. "Auditing the Auditors: Creating the Public Company Accounting Oversight Board."
• Bank for International Settlements. The Basel Framework.
• US Nuclear Regulatory Commission. "Nuclear Insurance and Disaster Relief."
• US Department of the Treasury. "Terrorism Risk Insurance Program."