The validation gap

The research question

Security programs produce many kinds of evidence, but those outputs are often discussed as if they mean the same thing. A configuration observation, a known-vulnerability match, a suspected exploit path and a validated security outcome can all appear in a finding system even though the confidence behind them is very different.

The validation-gap research question is straightforward: how much of the security program is based on detected possibility, and how much is based on validated exploitability?

This is a research agenda rather than an empirical result. RedMaw does not currently have a dataset that can support market-wide claims, benchmarks or statistics. The goal is to define a framework that could be measured honestly.

A taxonomy of security evidence

A useful analysis needs categories that separate observation from proof.

Configuration observation

The system identifies a state that violates a rule or expected configuration. The observation may be important, but it does not necessarily establish that an attacker can exploit it.

Known-vulnerability match

The system identifies software, behavior or configuration that corresponds to a known weakness. This can be strong evidence that a vulnerability may exist, especially when deterministic detection is appropriate.

Suspected exploitability

The evidence suggests that a weakness could be exploitable in the current environment, but the security outcome has not been reproduced.

Validated exploitability

The relevant attack succeeds within authorized scope and the system preserves reproducible evidence of the result.

Validated attacker outcome

The validation goes beyond the weakness itself and establishes what access, identity, system or data became reachable.

Validated remediation

After the fix, the relevant attack no longer succeeds. The security state changes because of the retest result.

Detection and proof are not competing ideas. They are different confidence levels in the same security decision.

Why the distinction matters

A security team has limited engineering attention. If every observation is treated as equally proven, prioritization becomes noisy. Teams can end up spending significant time on findings that are theoretically severe while missing less dramatic weaknesses that create a validated route to sensitive data.

The opposite mistake is also possible. An organization can become so focused on exploit proof that it ignores broad deterministic coverage. Known-vulnerability scanning and posture checks remain useful because they provide scale and consistency.

The research question is therefore not whether proof should replace detection. It is where each type of evidence belongs in the security decision.

What would need to be measured

• The source of each finding.
• The evidence category assigned to the finding.
• Whether exploitability was tested.
• Whether the security outcome was reproduced.
• Whether the finding was remediated.
• Whether remediation was re-tested.
• Whether the same security condition recurred.
• Whether prioritization changed after validation.

The most useful dataset would preserve the original detection state and the final validation state. Without that history, the study cannot distinguish a false assumption from a condition that simply changed later.

The assumption-to-proof maturity model

The analytical frame can be represented as a progression from assumption toward proof.

At the assumption end, the program relies heavily on configuration, signatures, severity and inferred risk. At the proof end, the program preserves reproducible attacker outcomes and validates remediation.

Mature programs are unlikely to live entirely at either extreme. They will combine broad detection for coverage with adversarial validation where the decision warrants stronger evidence.

Why RedMaw cares about the gap

RedMaw's product thesis is proof over probability. The platform attempts to validate supported exposure, preserve evidence and close findings by re-attack.

That makes the validation gap strategically relevant, but it also creates a responsibility to research the topic carefully. A vendor should not turn its product philosophy into an invented market statistic.

The useful contribution is a taxonomy, a measurement method and a transparent research design that other teams can inspect.

Measurement cautions

Validation data is especially easy to misread because a failed exploit attempt does not always mean the original detection was wrong. The target may have changed, the test may lack required context, a compensating control may interrupt the path or the weakness may be real without being safely exploitable under the permitted test mode.

A rigorous study therefore needs to preserve uncertainty. It should distinguish "not validated" from "not exploitable," document the authorization boundary and record when the environment changed between detection and validation. Otherwise the research would turn absence of proof into proof of absence.

A possible research program

A future study could begin with anonymized finding-state data from organizations willing to participate. Each finding would be classified by evidence type, remediation state and retest outcome.

Qualitative interviews could then examine how security leaders make prioritization decisions when severity and validated impact disagree. Engineering interviews could explore whether reproducible evidence changes remediation speed or confidence.

The result would not be a score designed to make one product look good. It would be a clearer map of where security programs rely on assumption and where they require proof.