An internal knowledge assistant should help staff find the right approved source, explain what it supports and admit when the available material cannot answer a question. A fluent response is not enough evidence that the assistant is useful or safe to rely on.
This guide is for operations, IT and knowledge owners evaluating an assistant over company policies, procedures or product documentation. It focuses on source-grounded answers and evaluation. It does not promise a particular accuracy score or replace the people responsible for company policy.
Use the knowledge assistant evaluation worksheet to record expected evidence, answer or refusal, actual output and reviewer notes. Its example is synthetic; observed results and scores are blank. If your team uses the optional 0–2 labels, define 0 as unsupported or failed, 1 as partial or requiring correction, and 2 as meeting the agreed case criteria. These are local review labels, not a standardized benchmark or a launch threshold.
Start with a real work task and approved sources
Pick one group of users and a task they perform often, such as finding the current travel policy or locating a technical procedure. Name the source systems the assistant may search, the people who own those sources and the users who may see each one. Decide whether the assistant will only answer questions or can also take an action; keep those scopes separate.
Build a source inventory before indexing. Record each document's owner, status, last review date, access rules, update path and removal method. Exclude drafts, old copies and informal discussions unless an owner has approved them for this purpose. Agree which source wins when two current documents conflict.
Check how permissions reach retrieval. If two employees have different access, the assistant must preserve that distinction when it retrieves passages and when it cites or summarizes them. A filter in the interface alone does not prove that restricted text was excluded from the model context or from logs.
Create an evaluation set from work people actually do
Ask representative users and subject-matter owners for realistic questions. Remove personal or confidential details unless their use has been approved. For each question, record the expected source or answer, what evidence would count as a correct response and whether the assistant should abstain.
Include more than clean, easy questions. Add short queries, common synonyms, misspellings, questions that need two sources, conflicting or outdated material, restricted documents, and questions the corpus cannot answer. Label questions where a direct answer would be unsafe or where a person must decide.
For example, staff ask, “How many days can I work abroad?” The approved travel policy has a limit, while an older presentation gives a different number. The test should identify the current policy as the source of truth, require a link to the relevant section, and fail an answer that blends the two documents. A separate test asks for an exception the policy does not cover; the expected result is a clear limit and a route to the policy owner.
Microsoft's RAG guidance recommends evaluating retrieval with known relevant documents and testing both positive and negative examples. It describes precision at K, recall at K and mean reciprocal rank as ways to inspect how relevant results appear in the retrieved list. These measures help diagnose search behavior; they do not establish answer quality on their own. Microsoft: evaluate search results in a RAG solution
Evaluate search and answers as separate steps
First ask whether retrieval found the material needed for the question. Check if the expected source appears near the top, whether unrelated passages crowd it out and whether a restricted item appears for a user who lacks access. Review failed queries by source type, terminology and document layout.
Then assess the generated answer against the retrieved text. Can a reviewer trace each important claim to a source? Does the answer cover every part of the question without adding unsupported details? Does it flag ambiguity or disagreement? Does it decline when there is no suitable source? Keep source citation correctness separate from whether the final wording is clear.
Microsoft's evaluation guidance separates retrieval quality from answer measures such as groundedness, completeness and relevance. It also notes that the same prompt can produce different responses, so an evaluation should use an agreed acceptance range or failure policy rather than treating one run as conclusive. Microsoft: RAG end-to-end evaluation
Use a knowledgeable reviewer for policy-sensitive cases. Automated graders can help sort a large test set, but they can also miss a wrong citation or accept a plausible answer that contradicts the source. Keep the underlying question, retrieved passages, response, expected behavior and reviewer decision so a failed case can be understood and repeated.
Test boundaries and source changes
Include tests for a user without access to a document, a prompt that asks the assistant to ignore its rules, a stale source, an empty search result and an indexing or retrieval failure. Treat text inside retrieved documents as source content, not as permission to change the assistant's job. The application should respect access rules before presenting context to the model.
Agree what happens when an answer cannot be supported: say what is missing, offer the right owner or search path, and avoid making up an answer to appear helpful. If a source is withdrawn or permissions change, test that it disappears from future retrieval and citations within the agreed update window.
Keep a versioned baseline. When the team changes chunking, filters, ranking, prompt instructions or model settings, rerun the same evaluation cases and inspect important regressions. Add a newly discovered failure to the test set after the owner confirms the expected behavior. Do not interpret a single average score as proof that every department or document type is ready.
Use a practical acceptance checklist
Before an internal pilot, confirm that:
- every indexed source has an owner, access rule and update or removal path;
- test questions cover routine, ambiguous, unanswerable and restricted cases;
- retrieval can find the expected source without exposing inaccessible material;
- citations open the exact relevant source and support the nearby claim;
- the assistant identifies gaps and conflicting guidance instead of inventing policy;
- source changes and access revocations are reflected in a tested period;
- staff know how to report a bad answer and who reviews it.
Review a sample of actual pilot questions under the organization's approved privacy and retention rules. Track unanswered questions, source corrections, citation failures and access issues alongside user feedback. Ask source owners to review recurring gaps; a high answer rating cannot repair incomplete or outdated source material.
BrainGlow's AI knowledge base and RAG service covers approved sources, access-aware retrieval, source references and evaluation. AI application development may fit when the assistant belongs inside an existing product or internal workflow. Scope, integrations, access controls and data handling need review against the actual systems and provider terms.
Discuss an internal knowledge assistant with one user group, a short source inventory and sample questions that include cases the assistant should not answer. That gives the evaluation a clear starting point before anyone relies on generated responses.