Skip to main content
AFAgent Foundry

Case 048

Source Citation

Severity / medium

Status / inspection open

Reviewer's docket / behavior before appearance

Your agent passed the prompt.Did it pass the path?

Agent Foundry follows the calls your code can actually reach, then grades each requirement against that trace.

1question
2retrieve
3context
4answer
citation metadata dropped

The documents carry source IDs. The final prompt never sees them.

Begin inspection / Problem 048Open source record

Inspection criteria

  1. C1

    Cites sources for claims

    Every factual claim in the answer includes a reference to the source document

  2. C2

    Citations are accurate

    Citations match the actual source documents that contained the information

  3. C3

    Output remains readable

    The answer is still natural and easy to read despite including citations

Built by an Adobe MLE, ex-NVIDIA. Kaggle Master.
pip install mukh / 10k+ downloads

Specimen 048-A / one real submission

Read the code. Follow the wire. Mark the evidence.

PIPELINE / SIMULATE → GRADE
FRAMEWORK / LANGCHAIN
RESULT / ACTION REQUIRED

starter.pylines 15—25
  1. 15vectorstore = FAISS.from_documents(documents, embeddings)
  2. 16retriever = vectorstore.as_retriever()
  3. 17
  4. 18# BUG: The prompt does not ask the LLM to cite sources — answers have no attribution
  5. 19prompt = ChatPromptTemplate.from_messages([
  6. 20 ("system", "Answer the question based on the context below.\n\nContext: {context}"),
  7. 21 ("human", "{question}"),
  8. 22])
  9. 23
  10. 24def ask(question: str) -> str:
  11. 25 docs = retriever.invoke(question)
Signal path
01 / SIMULATETRACE02 / GRADE

Criteria are checked only against behavior visible in the simulated path.

Evidence ledgerRoot cause not fixed

  1. Cites sources for claims

    missing

    The final prompt receives content, but no source metadata.

  2. Citations are accurate

    blocked

    Accuracy cannot be checked because no citations are emitted.

  3. Output remains readable

    pass

    The generated answer remains natural and readable.

Inspect the full problem

Repository index / live manifest

Pick a work order.The bench is already stocked.

Problems
103
Topics
90
Interview loops
27
Plans
6
Companies
16
License
GPL-3.0
OrderStudy planLoad
  1. WO-01Agent Engineer 50Start here50 items
  2. WO-02Tool Calling Deep DiveThe skill that makes an agent useful30 items
  3. WO-03RAG in 15Retrieval that actually holds up15 items
  4. WO-04Agent Security 20Assume someone is attacking it20 items
  5. WO-05Multi-Agent PatternsWhen one agent is not enough24 items
  6. WO-06Production ReadinessThe part tutorials skip22 items
Problem bins / multiple tags possible
BIN / 01Orchestration25BIN / 02Tool Calling24BIN / 03Prompt Design16BIN / 04Multi-Agent16BIN / 05Memory16BIN / 06RAG15BIN / 07Guardrails12BIN / 08Cost Optimization10BIN / 09Error Recovery9BIN / 10Evaluation8

Control model rack / four available

Choose where the decisions live.

Framework syntax changes. The useful question stays the same: who owns state, routing, handoffs, and recovery when the happy path ends?

CM-01

CrewAI

Coordinate role-based agent teams

brief
roles
crew
result

Tasks move through named collaborators and explicit handoffs.

24 topics / 7 sections
Start: Python functions
CM-02

LangChain

Compose tools, retrieval, and memory

input
tools
chain
output

Small runnable pieces connect into a broad application pipeline.

22 topics / 7 sections
Start: Python + APIs
CM-03

LangGraph

Control stateful execution

state
node
branch
checkpoint

State, checkpoints, loops, and approvals remain visible.

22 topics / 7 sections
Start: Workflow basics
CM-04

OpenAI Agents SDK

Ship a focused agent loop

agent
tool
handoff
trace

Tools, handoffs, guardrails, and traces share one compact model.

22 topics / 7 sections
Start: Tool calling

Bench notes / read before operating

Questions left in the margin.

N01Who belongs at this bench?

Developers who can write Python and want to understand agent behavior beyond framework syntax. The roadmaps begin at fundamentals; the hard problems still make experienced engineers stop and trace.

N02What is free?

Every roadmap, problem, study plan, interview loop, and code example is free and open source under GPL-3.0.

N03Which frameworks are inspected?

CrewAI, LangChain, LangGraph, and the OpenAI Agents SDK. The same problem can be opened in multiple framework variants.

N04What does the reviewer send?

Your solution is simulated into a reachable execution trace, then checked criterion by criterion. The result identifies the evidence found, what is missing, and the next correction.

N05Where does the API key go?

You bring an OpenRouter key for AI review. It is stored in your browser and is not persisted by Agent Foundry.

Entry ticketAdmit one builder
048

First inspection

Source Citation

Difficulty
medium
Timebox
30 minutes suggested
Frameworks
LangChain / CrewAI / OpenAI Agents SDK
Open Problem 048