Example brief: a synthetic support knowledge assistant
Imagine a fictional software team with ten short support articles covering account setup, approved integrations and troubleshooting. Build an assistant that answers from those articles and points to the supporting passage. Use synthetic content with no real customer information. This is a public practice example, not a client case study, a working product supplied on this page or a guarantee of an enrolled course’s exact project.
Define the intended user, permitted document collection and the point where the assistant should ask for help. Write down what success means before trying different prompts or retrieval settings.
1. Explain the information path
- Identify the permitted source documents and their versions.
- Explain how documents are split, indexed and retrieved.
- Show how source identifiers reach the final answer.
- Describe what happens when information is missing or two articles conflict.
Bring a diagram and one trace from question to retrieved passages to answer. The goal is to make the information path inspectable.
2. Separate retrieval checks from answer checks
Create a small, manually checked evaluation set. Include a direct answer, a paraphrased question, a question needing two passages, an outdated instruction and an unanswerable request. Record whether the expected passage was retrieved, whether the answer is supported, and whether its citation actually supports the statement.
A sample test row: question, expected passage, retrieved passage, answer, citation check, elapsed time and notes. Keep the same questions when comparing a change. A tiny test set is learning evidence, not proof of production accuracy.
3. Test boundaries and failure behavior
- Try an unrelated question and check that the system avoids inventing an answer.
- Place an instruction inside a retrieved document and check that it is treated as document content rather than authority to change the task.
- Simulate an unavailable model or retrieval service and explain the fallback.
- Where access differs by user, verify that filtering happens before restricted material reaches the model.
- Set retry and request-cost limits appropriate to your sandbox.
Document failures honestly. A discovered weakness with a thoughtful next step is more useful than claiming the system never makes mistakes.
4. Assemble a portfolio evidence pack
Include the brief, a readable diagram, a setup guide, test questions, selected traces, a results table and a limitations note. Add a short explanation of one change that improved behavior and one tradeoff it introduced. Remove secrets and sensitive records before sharing.
Label the project as a training or personal simulation. Explain your contribution and any AI assistance. Avoid describing sandbox work as paid enterprise employment.
Use the checklist in a learning conversation
Ask a reviewer to choose a failed test and let you investigate it aloud. Can you tell whether the problem is retrieval, source quality, answer generation or application behavior? That conversation reveals a useful next learning goal.
Explore Neriya’s Generative AI Engineer path and the AI engineering roadmap. For a framework implementation example, see the official LangGraph retrieval-agent tutorial. The test brief and evidence checklist above are Neriya’s own educational example; they do not certify a system as secure or production-ready.
YOUR NEXT STEP