Agents can synthesize weak evidence into confident answers. This experiment tests whether explicit evidence contracts make that failure mode less likely.
The hypothesis
Explicit Quality Data Contracts can reduce incorrect agent conclusions by making authority, freshness, identity, and provenance machine-readable.
The experiment
Give an agent the same release decision under three evidence conditions:
- raw tool outputs;
- normalized evidence without contracts;
- normalized evidence with contracts describing freshness, authority, and relationships.
Inject evidence problems
Deliberately include:
- stale test results;
- duplicate defect records;
- conflicting architecture documents;
- misaligned component names;
- inferred relationships presented beside observed ones.
What we measure
- unsupported conclusions;
- incorrect use of stale evidence;
- ability to identify uncertainty;
- provenance completeness;
- decision consistency;
- token and latency overhead.
What would falsify the hypothesis?
If contracts add implementation overhead without improving evidence interpretation or reducing incorrect conclusions, the approach is too heavy.
Expected failure mode
Agents may learn to treat contract metadata as absolute truth. The experiment should therefore include incorrect metadata and test whether the system can detect contradictions.
Next experiment
Make contract violations a runtime signal and observe whether confidence decreases appropriately when evidence quality deteriorates.
Grounding is not only about retrieving the right evidence. It is about knowing whether the evidence deserves to be trusted.
The decision task
Use one concrete task: determine whether the available evidence is sufficient to recommend deeper testing for a changed component.
This prevents the experiment from drifting into generic question answering.
Contract metadata under test
Include:
- source authority;
- evidence timestamp;
- component identity;
- environment identity;
- observed versus inferred relationships;
- confidence in inferred evidence;
- freshness threshold.
Test agent behavior under contradiction
Introduce a recent but non-authoritative document that conflicts with an older authoritative source.
Then reverse the pattern: make the authoritative source stale after a known change.
The agent should not simply prefer “newest” or “authoritative.” It should reason about both.
Measure abstention
A grounded agent should sometimes refuse to conclude.
Track how often the agent correctly identifies that evidence is insufficient, conflicting, or too stale to support a strong recommendation.
Operational cost
Contracts add metadata and validation. Measure ingestion complexity, token overhead, and reasoning latency alongside the quality benefit.
If the grounding improvement is small, a lighter-weight evidence model may be preferable.