AI Hallucinations Have Become a Procurement Problem

A fake citation in a public report used to be a footnote problem. Now it is a procurement problem, because the error can arrive wrapped in a contract, an invoice, a consultant’s logo, and the authority of an official document. The uncomfortable signal in the latest government AI embarrassments is not that models fabricate sources. Everyone knows that by now. It is that public institutions are starting to buy work whose evidence chain has been quietly delegated to systems nobody named, verified, or governed.

The error is now contractual

The freshest pattern came through a Rest of World account of five government AI hallucination incidents, including South Africa’s withdrawal of a draft national AI policy, Deloitte reports in Australia and Canada, and flawed cybersecurity reports tied to ENISA. The examples differ in geography and subject matter, but they share the same institutional shape: a public body receives or publishes a document that appears authoritative until someone follows the citations and discovers that the evidentiary floor is cracked.

That matters because the document is not just content. It is a policy input, a budget justification, a reform blueprint, or a technical assessment. In South Africa, the draft AI policy was pulled after fake citations were found, a failure covered by the Mail & Guardian’s report on Solly Malatsi withdrawing the draft. The damage was not merely reputational. A country trying to establish AI governance lost credibility because the policy process itself appeared unable to authenticate its own sources.

The obvious reading is too small

The easy interpretation is that governments need better fact-checking. True, but insufficient. Fact-checking treats hallucination as an editorial defect at the end of production. Procurement failure treats it as a control defect at the beginning: who was allowed to use AI, what disclosure was required, what evidence standard applied, and who carried liability when synthetic references entered the official record.

That distinction is where the next governance fight sits. A department can require citations and still fail if the vendor’s workflow uses a model to generate the research trail. A committee can demand corrections and still miss whether the underlying contract permitted undisclosed automation. The question is not whether a human eventually apologized. The question is whether the institution had a chain of custody for evidence before the report became official.

AI is becoming the invisible subcontractor

Consulting work already moves through layers: prime vendor, specialist team, analyst, subcontractor, data provider, reviewer. Generative AI adds another layer, but often without a name on the paperwork. In Australia, AP reported that Deloitte agreed to refund part of a government report fee after AI-related citation and reference errors surfaced. The refund is important because it converts hallucination from a quality embarrassment into a contractual event.

Canada points in the same direction. CBC reported that Newfoundland and Labrador asked Deloitte to review incorrect citations in a major health-care report. Once a government starts revisiting request-for-proposal language and vendor disclosure duties, the policy lesson changes. AI is no longer a tool sitting beside public administration. It is becoming an invisible subcontractor inside public administration, and procurement rules were not built to audit a subcontractor that can fabricate the bibliography.

The new control layer is chain of custody

This is why the issue connects to earlier Oria Veach coverage on verification gates. Prelaunch Testing Is Becoming the New AI Checkpoint argued that evaluation is becoming a condition of access before deployment. The same logic now applies to public-sector knowledge work. Governments do not only need to test models before release; they need to test evidence before purchase, publication, and policy reliance.

The technical agencies are not exempt. Reporting on ENISA described AI-hallucinated references in cybersecurity publications, with Cybernews detailing the controversy around AI-assisted threat reports. That is the harder case, because cybersecurity bodies are supposed to embody verification discipline. If a technical authority can publish reports with fabricated references, the problem is not lack of AI awareness. It is the absence of a mandatory audit trail showing how claims moved from source material into official output.

What governments should test next

The practical standard should be boring and strict: disclose AI use in public deliverables, preserve source trails, require vendors to identify where generative systems touched research or drafting, reserve audit rights before and after contract award, and make citation verification part of acceptance testing. This is less glamorous than model policy, but it is closer to where trust breaks. A procurement officer does not need a philosophy of intelligence to ask whether every cited source exists.

For builders and vendors, this becomes a product surface. The firms that can show provenance, review checkpoints, version history, and evidence validation will have an advantage over firms that merely promise faster reports. That is the same turn described in Codex Makes Safety a Product Surface: controls stop being background assurances and become part of what customers are actually buying.

The unresolved pressure is that governments are adopting AI faster than they are rewriting the paperwork that governs outsourced expertise. Capability will keep improving, but procurement credibility will not improve automatically with it. The next scandal will not be that a model invented a source. It will be that a public institution paid for the invention, cited it, and only afterward discovered that nobody had been assigned to prove the evidence was real.