Running internal-document Q&A in production: permissions, updates, evaluation and operations

Prototype vs. production

The document Q&A system was built for one person to try. Turn it into an internal system many people use, and problems beyond the model appear. For production, decide these requirements first.

This article does not recommend specific products. Product features and settings change, so it focuses on ideas and design points. The code is the smallest form that shows the mechanism.

Overall architecture

Think of a production setup as two paths: the path that answers questions and the path that ingests documents. The search index connects them.

• Question path: authentication establishes who is asking; the question service searches, filters by permission, has the model answer, verifies the answer, and returns it
• Ingestion path: detect changes in the document store, convert, split and embed, then update the index
• Logs and evaluation: both paths keep records, and an evaluation set measures quality regularly

Ingestion: keeping documents current

The prototype loaded every document at startup. In production, rebuild only the documents that changed.

• Store a hash of each document's content, and reprocess only what changed
• When a document disappears, remove it from the index too (otherwise answers use content you deleted)
• Keep with each document its source (path or URL), update date and the groups allowed to read it
• Convert PDF and Office files to text first. Tables and figures break easily, so check the result by eye
• Keep heading structure when splitting. Putting the heading at the start of each chunk makes it easier to find

import hashlib
import re
import sqlite3

from rag import split  # the split function from the previous article

db = sqlite3.connect("index.db")
db.executescript("""
CREATE TABLE IF NOT EXISTS docs (path TEXT PRIMARY KEY, hash TEXT, groups TEXT);
CREATE TABLE IF NOT EXISTS chunks (id INTEGER PRIMARY KEY, path TEXT, ord INTEGER, text TEXT);
""")

# Ingest: rebuild only changed documents, and remove vanished ones from the index
def sync(files):  # files: {path: (text, set of groups allowed to read)}
    changed = removed = 0
    for path, (text, groups) in files.items():
        digest = hashlib.sha256(text.encode()).hexdigest()
        row = db.execute("SELECT hash FROM docs WHERE path = ?", (path,)).fetchone()
        if row and row[0] == digest:
            continue  # unchanged
        db.execute("DELETE FROM chunks WHERE path = ?", (path,))
        for order, chunk in enumerate(split(text)):
            db.execute("INSERT INTO chunks (path, ord, text) VALUES (?, ?, ?)", (path, order, chunk))
        db.execute("INSERT OR REPLACE INTO docs VALUES (?, ?, ?)", (path, digest, ",".join(sorted(groups))))
        changed += 1  # recompute this document's chunk embeddings here too
    for (path,) in db.execute("SELECT path FROM docs").fetchall():
        if path not in files:
            db.execute("DELETE FROM chunks WHERE path = ?", (path,))
            db.execute("DELETE FROM docs WHERE path = ?", (path,))
            removed += 1
    db.commit()
    return changed, removed

If you swap the embedding model, the existing vectors become unusable, and you rebuild the vectors for every document. Record which model made the vectors in the index so the swap goes smoothly.

Permissions: filter before searching

Internal documents mix what everyone may read with what only certain departments may read. The key point: do not leave the permission check to the model. Writing “do not show this to this person” in the prompt gives no guarantee the model obeys.

Filter before the search (or together with it). Documents a person must not see are never candidates in the first place.

# Permissions: before searching, keep only the chunks this person may see
def visible_chunks(user_groups):
    rows = db.execute("SELECT chunks.id, chunks.path, chunks.text, docs.groups FROM chunks JOIN docs USING (path)").fetchall()
    return [(i, path, text) for i, path, text, groups in rows if set(groups.split(",")) & set(user_groups)]

• Leave authentication to your existing company system (SSO, etc.) and get the user's groups from it
• Build a path so that permission changes (leaving, transfers) take effect quickly. If you copy groups into the index, a sync delay becomes a hole
• If even the existence of a document must stay hidden, base answers like “no source found” on the permission-filtered results

Improving retrieval

Answer quality is mostly decided by retrieval. If the sources handed to the model are wrong, no model can answer correctly.

• Hybrid search: use both keyword search (strong on proper nouns and part numbers) and vector search (strong on paraphrases)
• Merging results: the two scores are not comparable, so merging by rank only (RRF) is common
• Reranking: reorder the top few dozen with a more precise method and keep a handful
• Metadata filters: narrow the search by department, document type, update date and so on

# Merging results: use ranks only (RRF). Keyword and vector scores are not comparable, so merge by rank
def rrf(rankings, k=60):
    score = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            score[chunk_id] = score.get(chunk_id, 0) + 1 / (k + rank)
    return sorted(score, key=score.get, reverse=True)

Generate the answer, then verify it

The prototype returned the model's answer as it was. In production, check it before returning.

• Citation check: confirm the source numbers in the answer are numbers of sources you actually passed. With no valid number, treat the answer as unsupported
• Choose not to answer: when search finds nothing, or only weak matches, return “do not know” without calling the model
• Show the source: show a link to the original document with the answer so a person can check the original
• Prepare for prompt injection: do not let instructions inside documents be followed. Be especially wary of documents outsiders can write (inquiries, wiki edits)

# Citation check: are the numbers in the answer really numbers of the sources we passed?
def check_citations(answer, found_count):
    cited = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
    return bool(cited) and cited <= set(range(1, found_count + 1))

Correct citation numbers do not guarantee correct content. The number check is the minimum; checking content is the job of the evaluation below.

Evaluation: measure after every change

Changing the chunk size, the search method, the model or the prompt can raise or lower quality. Do not judge by feel; measure with the same set of questions and compare.

1. Collect 50–100 real questions (ask people on the job, or use inquiry history)
2. Attach to each the part of the document that is the correct basis
3. After every change, measure retrieval (the share of questions with the correct part in the top k)
4. Check answers by sampling with people. You can have a model grade answers, but the model's grading is also wrong at times, so combine it with human checks

# Evaluation: the share of questions whose correct chunk is in the top k
def recall_at_k(cases, search, k=5):  # cases: [(question, set of correct chunk ids)]
    hits = sum(1 for question, expected in cases if expected & set(search(question)[:k]))
    return hits / len(cases)

• Include “questions the documents do not cover” in the set, to measure whether it correctly says “do not know”
• Grow the set from real use. Add questions from the “was not helpful” records
• Keep the ability to roll back the last change when the numbers drop

Operations: keeping it running

• Handling logs: questions and answers contain confidential or personal data. Protect logs as strongly as the documents
• Failures: if the model stops, still returning the search results (a list of related documents) keeps the service from going completely dark
• Updates: when changing the model, embedding model or search settings, measure on the evaluation set before switching
• Backups: the index can be rebuilt from documents, but it takes time. Know how long a rebuild takes

Checklist

• Have you decided the number of users, document volume, update frequency and unit of permission?
• Do document changes and deletions reach the index within the time you decided?
• Do you filter by permission before searching, rather than relying on the prompt?
• Have you tested that documents a person must not see never appear in answers?
• Is there an evaluation set that you run after every change, including questions the documents do not cover?
• Do you verify citations and show source links?
• Have you decided log retention and who can read the logs?
• Have you decided what happens when the model or search stops?
• Have you decided who runs it and who fixes problems?