The AI release security gap

The research question

How do organizations determine whether a model change introduced a security regression?

AI teams already evaluate quality, performance and product behavior when a model is retrained, fine-tuned or replaced. The security question is different. A candidate may perform its intended task better while becoming easier to prompt inject, more willing to disclose sensitive context or more vulnerable to jailbreaks.

The AI release security gap is the difference between the maturity of functional release evaluation and the maturity of adversarial security evaluation.

This is a research agenda. There is no RedMaw dataset supporting claims about how common the gap is or which industries manage it best.

The release lifecycle as a security framework

A useful study can examine the model lifecycle through a sequence of decision points.

Model change event

A retrain, fine-tune, model replacement or significant serving change creates a new candidate.

Functional evaluation

The team determines whether the candidate performs the product task acceptably.

Adversarial evaluation

The candidate is tested against defined security boundaries such as prompt injection, disclosure, jailbreak and control bypass.

Baseline comparison

The candidate's security behavior is compared with the last accepted version or another approved baseline.

Release decision

The organization decides whether the candidate can be promoted.

Post-release validation

The deployed system remains subject to adversarial testing as application context and surrounding controls change.

The research problem is not whether AI teams test models. It is whether security regression has a defined place in the release decision.

What would need to be measured

• Which model changes trigger security evaluation.
• Which attack classes are included.
• Whether testing occurs before or after promotion.
• Whether a security baseline exists.
• Which security regressions can block release.
• How evidence is preserved.
• How remediation is verified.
• Which teams own the final decision.

The research would also need to distinguish black-box endpoint testing from evaluations that have access to the model artifact or candidate release. Those modes can answer different questions.

Security regression is not poisoning detection

The scope needs a clear boundary. Testing a model after fine-tuning or retraining for changed security behavior does not prove that the underlying training data was poisoned.

RedMaw's AI testing covers security regression testing, unexpected-behavior testing and exploit-oriented red-teaming.

Research should preserve the same distinction. Otherwise a study about release controls can accidentally imply a much deeper supply-chain capability.

Agentic systems create a future extension

The consequence of a security regression rises when model output can trigger production actions. A jailbreak in a conversational system may expose information. The same control failure in an agentic system may influence a tool call.

A research program can study the governance problem today, independently of any one product.

Release policy is part of the security system

Adversarial evaluation has limited value if the organization has not defined what happens when a candidate fails. A future study should therefore examine whether teams have explicit security release criteria, who can override a failed gate and whether exceptions are recorded and revisited.

This is especially important for AI because product quality and security behavior can move in different directions. The organization may face a candidate that improves the intended task while regressing on a security boundary. The release policy determines whether that conflict becomes visible and governable.

What a useful dataset would look like

The most valuable dataset would follow candidate models through release. It would preserve the model-change event, functional result, adversarial result, baseline comparison, release decision and any post-release retest.

That would allow researchers to ask whether security regression changes release outcomes, whether certain classes of change create more retesting activity and how teams handle conflict between product improvement and security regression.

Evidence needs to survive the release decision

A release process also needs to preserve why a candidate was accepted or blocked. The adversarial input, resulting behavior, baseline comparison and approval decision should remain inspectable after the release event. Without that record, teams can repeat the test without understanding why the earlier result mattered.

Research should therefore examine evidence retention alongside testing coverage. A mature gate is not only a test that runs. It is a decision process with enough evidence to explain what failed, who accepted the outcome and how the same condition will be evaluated after remediation.

Why the topic matters

AI security will be easier to operate when it becomes part of release engineering rather than a separate ceremony.

The research opportunity is to understand how close organizations already are to that model, which controls they use and where evidence breaks down between model development, security and production approval.

A clear framework gives AI and security teams a shared language before any benchmark exists.