Intelligence AI needs to distinguish bad evidence from an attack on the model

An intelligence system already works with incomplete, misleading or deliberately deceptive material. Adding AI creates another possible target: the mechanism interpreting the material. A wrong detection can result from an unfamiliar scene, a flawed training example or an attempt to manipulate the model. Those causes require different evidence and different remedies.

NIST’s March 2025 adversarial machine-learning taxonomy distinguishes attack types and stages across an AI system’s lifecycle. MITRE’s May 2025 Secure AI update describes work expanding threat-informed resources. These frameworks organise threats and observed techniques. They do not certify a specific intelligence model as resilient or publish a universal likelihood that an operational adversary can defeat it.

Identify the access an attacker actually has

Adversarial machine learning studies attempts to make AI behave in ways the attacker wants. Poisoning concerns manipulation of training material; evasion concerns inputs intended to mislead a deployed model. Privacy and extraction attacks seek information from or about the system. Naming the type is useful only when the assessment states what the attacker can access and change.

For an intelligence classifier, a laboratory manipulation may assume direct access to image pixels. A deployed adversary may instead control a physical object, a public post or a supplied dataset. Those are different threat models. A result under one should not automatically become a claim about all others. Testing needs to connect the attacker’s assumed access to the real collection chain.

Ordinary error can resemble an attack. Poor lighting or unfamiliar equipment can confuse a model without deliberate interference. A useful investigation preserves the input, processing steps and configuration so those explanations can be compared. Labelling every unexpected result adversarial can be as unhelpful as assuming every failure is ordinary noise.

The converse matters too. A model can work well on representative examples while being vulnerable to a narrow manipulation those examples never contain. Average accuracy alone cannot establish resilience against a motivated actor. The evaluation should therefore separate normal task performance from the specific attack hypotheses it tests and state the boundaries of each result.

Security includes the surrounding workflow

An AI model is only one part of the service. Data ingestion, retrieval, software dependencies and permissions can create opportunities to influence outputs. Protecting the model while accepting untrusted instructions from retrieved material leaves another path into the workflow. The relevant security boundary is the complete system performing the intelligence task.

A useful defensive exercise would choose a bounded, authorised manipulation and examine what the system does with it. Does the suspicious material change a finding, survive into a report or trigger a protected action? Retain the evidence showing which boundary was crossed. This is a proposed evaluation approach, not operational instructions or a claim that a named deployed system is compromised.

The countercase is spending the entire evaluation budget on rare, elaborate attacks while ordinary failures remain common. Threat-informed testing should prioritise plausible access and consequential outcomes. Frameworks help structure that choice, but cannot set the organisation’s risk tolerance or decide which mission errors it can accept.

Mitigation also needs an operational measure. A restriction that blocks legitimate data can make the system less useful; a review step can add delay and workload. The customer should assess whether the defence reduces the relevant harmful behaviour while preserving needed capability. Counting security controls is no more sufficient than counting sensors.

The broader intelligence concern is preserving a trustworthy relationship between the collected evidence and the judgement. An attack taxonomy makes the ways that relationship can fail more explicit. It becomes useful when the organisation can reproduce a failure, identify the attacker’s required access and demonstrate that a repair addresses that pathway.

A threat model should connect to the mission

Consider an illustrative analyst assistant that retrieves public documents. A malicious passage embedded in a source might try to influence how the assistant describes the evidence. That differs from falsifying the document’s factual content: it targets how the software processes the material. The organisation needs to test whether source text is treated as evidence or allowed to become an instruction that changes the workflow.

Investigation should also preserve the legitimate task. An assistant that refuses to read all unfamiliar material may resist some manipulation while becoming ineffective at open-source research. The meaningful comparison asks whether the system can process untrusted evidence within its permitted role and still stop that evidence from acquiring authority over tools or outputs. This is an assessment principle, not a claim that every available defence achieves it.

The larger impact is that intelligence protection must cover both deception of the analyst and manipulation of the automated interpreter. Traditional source criticism remains necessary, while software permissions and processing boundaries add another layer. The frameworks matter when they help customers identify a plausible failure path and demonstrate a targeted improvement, rather than substitute a long list of attack names for a mission-specific evaluation.