The client wants a chatbot over their documents. Walk m...

First, I’d push back gently on the framing — “a chatbot over our documents” is a solution someone has already picked, not a problem statement. What decision does someone actually need to make that they can’t make efficiently today? What do they do right now without this tool? That answer usually reveals whether a chatbot is even the right shape.

Once there’s a real problem underneath, four questions shape the architecture more than any model choice:

• Is the corpus stable, or does it change regularly? That decides how much ingestion and re-indexing infrastructure this actually needs.
• Does the answer need to be citable back to a source? That’s the line between “nice to have” and “the system is unusable without it” in regulated contexts.
• Who’s liable if it’s wrong? That determines how much human-in-the-loop review the design needs.
• Can data leave the environment? That decides the deployment architecture, as above.

Only after those are answered would I build anything — starting with a thin vertical slice over a real subset of the documents, evaluated against questions written by actual domain experts, not synthetic questions I made up myself.