Short answer
Before building an AI assistant on company data, inventory documents by source, owner, currency and access group; carry permissions along with each document and define the update and deletion flow. If the data is not ready, the assistant's quality will not be fixed by changing the model.
Inventory the sources
List documents by source, owner, freshness and access group. Identify duplicates and contradictory versions. Image-based PDFs and text documents may need different processing.
Carry permissions with content
Application sign-in does not solve document-level permissions. Search results must be restricted to sources the user can access, and index records must retain document identity and access information.
Updates and deletions
Define when source changes and deletions propagate. Answers based on old documents can misdirect workflows. Monitor refresh failures.
Evaluation questions
Include frequent answerable questions, questions that should remain unanswered and requests for restricted information. Test source relevance, errors and access behaviour together.
Data quality problems
The problems that affect answer quality most are often in the documents: conflicting versions that replaced one another, scanned and unreadable pages, ownerless documents of unknown currency, and information buried in tables or figures. Classifying these problems before indexing is cheaper than hunting for the cause of a wrong answer afterwards.
A practical evaluation exercise
When two documents cover the same subject, identify which version is authoritative. Test retrieval using different access roles and check that unauthorised users receive neither titles nor excerpts. Include questions the sources cannot answer. Define when the assistant must abstain as explicitly as examples of correct answers.
Frequently asked questions
Two questions we are often asked on this topic.
Do we need to load all documents?
No. First choose sources that are suitable, current and authoritative for the questions the assistant will answer. A few good sources give better results than many unclear ones.
How are sensitive documents handled?
Access rights should be applied at the retrieval stage, and sensitive content must not be given to an unauthorised user as context. Documents should not be shared until it is confirmed in writing which data goes to which service.
Sources and further reading
