Intelligent systems create a different assurance problem: an output can vary and still be acceptable, while a system that appears correct can remain unsafe, poorly grounded or operationally fragile.

Software quality becomes harder when the same input can produce different acceptable outputs

Traditional software testing grew up around determinism. Given a defined input and a known state, we usually expect a defined result.

Intelligent systems complicate that assumption. A generated response can vary while still being acceptable. A recommendation can be reasonable without being identical. A model can perform well overall and still fail badly for a narrow but important scenario.

That means “expected result” is no longer sufficient as the primary assurance construct.

AI assurance is not about proving that a model is always right. It is about engineering evidence that the system is trustworthy enough for its intended use.

Testing the application around the model is not testing the model

An AI-enabled application still needs conventional assurance: APIs, integrations, security, performance, accessibility, resilience, data flows and user journeys.

But those tests do not answer whether the intelligent behavior itself is acceptable.

A system may have flawless APIs and still hallucinate. It may be fast and still be unsafe. It may satisfy functional flows while becoming less relevant as its underlying data changes.

We need a separate evidence layer for intelligent behavior.

I think of AI assurance as a stack

Layer 1 — Functional integrity

Does the surrounding software behave correctly? Are requests routed correctly? Are permissions, state, fallbacks and integrations working?

Layer 2 — Model and response quality

Are outputs relevant, grounded, complete and appropriate for the use case? For classification or prediction systems, are the selected statistical measures appropriate to the consequence of errors?

Layer 3 — Robustness

What happens when prompts are ambiguous, inputs are noisy, context is incomplete or users behave unexpectedly? Does performance degrade gracefully?

Layer 4 — Safety and policy

Does the system stay within prohibited or constrained behaviors? Can guardrails be bypassed? Are high-risk actions appropriately controlled?

Layer 5 — Grounding and provenance

When the system uses enterprise knowledge, can we determine what information influenced the response? Was the evidence current, permitted and relevant?

Layer 6 — Operational reliability

How does the system behave under latency, provider changes, retrieval failure, model drift, cost pressure and dependency degradation?

Layer 7 — Business outcome

Even if the model performs well technically, does the system improve the intended business decision or user outcome?

Evaluation needs more than one judge

AI teams increasingly use model-based evaluators because they scale. That is useful, but an evaluator is itself another probabilistic system.

A credible assurance strategy should combine methods: deterministic checks where possible, reference-based evaluation where appropriate, statistical measures, human review for consequential scenarios, adversarial testing and model-based evaluators for scale.

The goal is not to find one perfect evaluator. It is to construct an evidence portfolio whose weaknesses do not all fail in the same direction.

Context is part of the system under test

For RAG and agentic systems, the model is only one component. Retrieval quality, chunking, ranking, freshness, metadata, permissions, tool selection and memory can all materially change the output.

That means AI assurance has to inspect the evidence path, not just the final answer.

Was the right context retrieved? Was important context omitted? Did stale information dominate? Did the agent select the correct tool? Did a previous interaction improperly influence the decision?

This is where provenance becomes an engineering requirement rather than a governance afterthought.

Risk should determine evaluation depth

Not every AI feature requires the same assurance intensity.

A creative writing assistant and an agent allowed to change a financial record should not share the same quality bar.

Evaluation depth should scale with consequence, autonomy, reversibility and uncertainty.

As systems gain the ability to act, assurance must increasingly test not only what the AI says but what the AI is allowed to do.

Production is part of the test environment

AI quality changes after release. Data distributions move. Knowledge sources change. Models are updated. Provider behavior changes. Prompts evolve. Users discover interaction patterns that were not anticipated.

For intelligent systems, pre-release evaluation is the beginning of assurance, not the end.

Production assurance requires continuous evaluation, drift detection, incident learning, trace analysis, and explicit feedback loops into the evaluation suite.

The key artifact is an AI confidence case

For consequential intelligent systems, I believe teams need a living confidence case: a structured argument that explains why the system should be trusted for a defined use, under defined conditions, with known limitations.

It should connect:

  • intended use and risk;
  • evaluation coverage;
  • known failure modes;
  • safety controls;
  • grounding and provenance;
  • operational monitoring;
  • human authority;
  • and residual uncertainty.

Who tests the AI?

The answer cannot be one team at the end of the lifecycle.

AI assurance is an engineering discipline that spans product, data, model, platform, security, quality and risk.

Quality Engineering has an important role because it already knows how to turn uncertainty into evidence. But the discipline has to evolve beyond deterministic test execution.

The question is no longer only “Does the software work?” It is “What evidence makes this intelligent behavior worthy of trust?”