An AI system can pass many evaluations and still be difficult to trust. This experiment asks whether layered evidence can produce a confidence case that is explicit about both strength and uncertainty.

The hypothesis

Trust in an AI-enabled system can be represented as an evidence-based confidence case that combines functional integrity, behavior quality, robustness, safety, grounding, provenance and runtime controls.

The purpose is not to create a single magic trust score. It is to determine whether several evidence layers can support a defensible statement about where the system should—and should not—be trusted.

The system under experiment

Use an AI assistant that retrieves enterprise knowledge and can recommend an action, but cannot execute the action autonomously.

This creates enough complexity to test grounding, variability and safety without immediately introducing irreversible autonomy.

Define the intended use first

Before evaluating the system, document:

  • the users;
  • the decisions the AI may influence;
  • the types of information it can access;
  • the consequences of a wrong answer;
  • and the situations in which a human must remain responsible.

Without intended-use boundaries, “trustworthy AI” is too vague to test.

The evidence stack

Functional integrity

Are authentication, permissions, APIs, state, integrations and fallbacks behaving correctly?

Response quality

Are answers relevant, sufficiently complete and appropriate for the use case?

Grounding

Does the response reflect the authoritative source material rather than plausible invention?

Robustness

How does behavior change under ambiguous questions, noisy inputs, missing context and adversarial prompting?

Safety and policy

Does the system stay within prohibited behaviors and escalation rules?

Provenance

Can a reviewer identify the model, prompt, retrieved evidence and evaluation path behind the answer?

Operational reliability

What happens when retrieval fails, latency increases, the model provider changes or a dependency degrades?

Build the confidence case

For each evidence family, classify the position as Strong, Moderate, Weak or Unknown. Preserve the explanation and source evidence.

The overall conclusion should not be a simple average. A severe safety weakness may dominate otherwise strong functional performance.

Test known and unknown scenarios

Include a normal evaluation set, known historical failure modes and deliberately unfamiliar scenarios.

The system should be rewarded for recognizing uncertainty rather than confidently answering outside the evidence available.

What we measure

  • quality of grounded responses;
  • rate of unsupported assertions;
  • robustness under perturbation;
  • policy violation rate;
  • consistency of human and automated evaluation;
  • quality of provenance;
  • and whether the confidence case predicts where humans later find material risk.

What would falsify the hypothesis?

The model fails if the evidence layers cannot be combined without arbitrary weighting, if the confidence case hides disagreement between evaluators, or if reviewers cannot understand why the system is considered trustworthy for one use and not another.

Expected failure modes

  • a strong aggregate score hides a severe failure in a narrow scenario;
  • LLM-as-judge evaluation creates unrecognized evaluator bias;
  • grounding is measured without checking source authority;
  • the human-review sample is too small or inconsistent to calibrate automated evaluation;
  • the confidence case becomes static while the AI system keeps changing.

The next experiment

Move the confidence case into production. Allow runtime behavior, model changes, retrieval freshness and user feedback to continuously alter the trust position.

That converts a pre-release AI assurance exercise into Continuous AI Assurance.

AI assurance becomes useful when trust is expressed as evidence, limitations and uncertainty—not as a generic claim that the model passed evaluation.