Agent Foundry
All Problems

#52. Conversational RAG

MediumRAGMemory

The Problem

Your RAG agent answers product questions well on the first turn, but falls apart on follow-ups. When a user asks "Tell me about ProductX" and then follows up with "What about pricing?"—the agent treats the second question in isolation. It doesn't know "pricing" refers to ProductX, so it might return pricing for a different product or give a generic answer. The agent has no conversation memory. Your job is to add conversation history tracking and contextualize follow-up questions so the retriever searches with full conversational context.

Examples

Example 1

Turn 1 — User: Tell me about ProductX. Agent: ProductX is an enterprise CRM platform offering advanced analytics, custom dashboards, and API access.

Turn 2 — User: What about pricing?

Current (bad) output: Returns pricing for both ProductX and ProductY, or gives a generic answer—because the retriever doesn't know the user is still asking about ProductX.

Expected (good) output: ProductX starts at $99/month and offers a 14-day free trial with full feature access.

Example 2

Turn 1 — User: Compare ProductX and ProductY. Agent: (gives comparison)

Turn 2 — User: Which one has a free trial?

Current (bad) output: Retrieves random documents because "free trial" alone is too vague without conversation context.

Expected (good) output: ProductX has a 14-day free trial with full feature access. There is no mention of a free trial for ProductY in the available documents.

Your Task

Modify the RAG pipeline to support multi-turn conversations:

  • Maintain conversation history across multiple turns.
  • Before retrieval, reformulate follow-up questions into standalone queries using the conversation context.
  • Ensure the retriever receives fully contextualized search queries.
  • Return accurate answers that correctly resolve references from prior turns.

Evaluation

Submissions are checked for the following:

  • Maintains conversation history: The agent tracks prior turns and uses them to understand follow-up questions.
  • Resolves follow-up references: Ambiguous follow-ups like "What about pricing?" are correctly resolved to the right product.
  • Retrieves with conversational context: The retriever receives contextualized queries that incorporate prior conversation context.

Constraints

  • The agent must maintain conversation history across turns
  • Follow-up questions must be resolved using prior context
  • The retriever must receive contextualized queries, not raw follow-ups
  • The agent must still retrieve documents for each turn
Starter Code
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document

llm = ChatOpenAI(model="gpt-4o-mini")
embeddings = OpenAIEmbeddings()

documents = [
    Document(page_content="ProductX is an enterprise CRM platform starting at $99/month."),
    Document(page_content="ProductY is a project management tool priced at $49/month per team."),
    Document(page_content="ProductX offers advanced analytics, custom dashboards, and API access."),
    Document(page_content="ProductY includes Gantt charts, sprint planning, and time tracking."),
    Document(page_content="ProductX has a 14-day free trial with full feature access."),
]

vectorstore = FAISS.from_documents(documents, embeddings)
retriever = vectorstore.as_retriever()

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer based on context.\n\nContext: {context}"),
    ("human", "{question}"),
])

# BUG: Each query is treated independently — no conversation history
def ask(question: str) -> str:
    docs = retriever.invoke(question)
    context = "\n".join([doc.page_content for doc in docs])
    chain = prompt | llm
    result = chain.invoke({"context": context, "question": question})
    return result.content

print(ask("Tell me about ProductX."))
# Follow-up: "What about pricing?" — but the agent doesn't know we're still talking about ProductX
print(ask("What about pricing?"))
Open in Google Colab
Evaluation Criteria0/3