RAG can retrieve what the organization knows. The harder problem is keeping that knowledge relevant when requirements, architecture, dependencies and production behavior keep moving.

The hypothesis

A quality knowledge layer can remain useful with limited manual curation if freshness, authority, change impact and retrieval quality are treated as first-class parts of the system.

The experiment is not simply whether RAG can retrieve documentation. The harder question is whether the quality context stays trustworthy as the system changes.

The environment

Construct a domain corpus containing requirements, architecture notes, API definitions, test assets, defect history, incident summaries, dependency maps and selected code-change context.

Then deliberately evolve the environment over several simulated release cycles:

  • change a business rule;
  • replace a dependency;
  • deprecate an API;
  • introduce a new failure pattern;
  • change the ownership of a component;
  • update an AI prompt or model configuration.

The quality tasks

Ask the knowledge layer to support concrete QE decisions:

  • generate risk-focused scenarios for the current change;
  • identify historical defects relevant to the release;
  • explain which journeys may be affected;
  • recommend where regression evidence is weak;
  • identify stale or contradictory domain knowledge.

Three retrieval conditions

Static corpus

Use a fixed snapshot of documentation. This becomes the baseline for context decay.

Continuously indexed corpus

Refresh sources automatically but do not reason about authority or change relevance.

Dynamic quality knowledge layer

Refresh sources while also tracking source authority, freshness, version, relationships and change impact.

Retrieval quality is not enough

Traditional RAG evaluation may ask whether the right passage was retrieved. For quality decisions, we also need to know whether the passage is still authoritative.

An old architecture document may be semantically perfect and operationally wrong.

That means the experiment should evaluate:

  • relevance;
  • freshness;
  • authority;
  • completeness;
  • relationship accuracy;
  • and whether contradictory evidence was detected.

What we measure

Scenario relevance

Do generated or recommended tests reflect the current change rather than generic domain knowledge?

Stale-evidence rate

How often does outdated information materially influence a recommendation?

Change sensitivity

How quickly does the knowledge layer adjust after a meaningful system change?

Evidence traceability

Can the system show which sources influenced each quality conclusion?

Manual curation burden

How much human intervention is required to keep the knowledge layer reliable?

What would falsify the hypothesis?

The hypothesis fails if continuous ingestion merely creates a larger stale corpus, if authoritative and non-authoritative sources cannot be distinguished, or if manual curation remains the only practical way to maintain retrieval quality.

Expected failure modes

  • semantically relevant but obsolete documents dominate retrieval;
  • duplicate sources create false consensus;
  • recent sources are preferred even when they are not authoritative;
  • the system cannot identify missing context;
  • code and runtime change faster than the indexing cycle;
  • retrieval metrics look healthy while downstream test decisions remain poor.

The next experiment

Connect the knowledge layer to a Quality Graph. Instead of retrieving only chunks, allow the reasoning system to navigate explicit relationships between changes, components, journeys, tests and failures.

Then compare whether graph-informed retrieval improves risk relevance and provenance.

Dynamic RAG becomes Quality Intelligence when context is not only retrievable, but current enough and trustworthy enough to change the decision.