Find the conflict before changing the wording
One member hears that a replay stays available indefinitely. Another is told it expires at the end of the month. The AI may be reading two documents that disagree, applying the wrong membership rule or missing an important exception.
Start by comparing the answers with the evidence behind them. Ask which source each answer used, whether that source is current and which member or product the rule applies to. Rewriting the assistant’s tone will not resolve contradictory policies.
Choose a small set of questions your team answers regularly. Use them as a baseline before making changes, so you can tell whether a cleaner knowledge base produces clearer results.
Label knowledge by program and audience
A replay policy for one cohort is not automatically valid for the next. Trading-education rules belong to the specific offer and current version. Product-onboarding guidance must match the service the person uses. Keep public explanations distinct from personal account records so a general answer does not become an unsupported account decision.
Give each important rule an owner
Maintain one approved reference for each important topic: membership access, replay availability, event schedules, cancellation procedures and support boundaries. Name someone who can confirm the rule and approve changes.
For every reference, make its scope explicit. A rule for an annual plan may not apply to a monthly plan. A historical promotion may not apply to a new member. A general product page should not override an account-specific entitlement.
- What question does this source answer?
- Which plan, product or situation does it cover?
- When does the rule apply?
- Who owns updates?
- Which previous version does it replace?
Retire or clearly mark old sources. Simply adding a new policy while leaving the previous one searchable can preserve the contradiction.
Separate general knowledge from private account facts
“The replay is included in the Plus plan” is reusable knowledge. “This member has an active Plus subscription” is a private account fact that must come from a permitted lookup.
Do not turn a billing screenshot or a past support conversation into the current truth for every member. When a request depends on an individual account, the response needs the correct verified identity and the relevant account data.
Illustrative question: “Can I watch the advanced workshop?”
Useful answer: identify the current access rule, verify the member’s entitlement when that lookup is available and provide the correct next step.
If the entitlement cannot be checked: explain that limitation and route the request to the appropriate team. Do not guess from a display name or a previous conversation.
Test uncertainty, not just easy questions
Your test set should contain questions with clear answers, questions with missing details and questions for which no approved answer exists. Add a deliberately conflicting source in a test environment and check whether the system recognises the problem.
A good response can ask a targeted question, point to the relevant rule or stop and request review. A high match score by itself should not override missing identity, contradictory evidence or an action that the system is not allowed to perform.
NIST’s voluntary AI Risk Management Framework organises risk work around governance, mapping, measurement and management. For a support team, a practical application is to name the owner, define the use case, test realistic failures and decide how corrections reach the live system.
Use a release checklist for knowledge changes
- The new source has an owner and a defined scope.
- Conflicting versions have been removed or excluded.
- Account-specific information stays outside shared answers.
- The changed rule passes your normal and exception test questions.
- The response explains a limit instead of inventing a missing fact.
- Your team can identify and reverse the change if a problem appears.
When an answer is wrong, fix the underlying source or decision rule and retest related questions. Keep a record of the correction so another update does not bring the same mistake back.