Agent Foundry
All Problems

#6. Few-Shot Tool Selector

EasyPrompt DesignTool Calling

The Problem

Your assistant has access to a stock-price tool and a weather tool, but it makes poor decisions about when to use them. It sometimes calls the stock tool for general-knowledge questions ("What is the capital of Japan?") or tries to answer real-time queries from memory instead of calling the right tool. The tools themselves work fine; the problem is the system prompt gives the model no guidance on when to use a tool versus answer directly. Your job is to add few-shot examples to the system prompt so the agent reliably uses tools for real-time data and answers directly for general knowledge.

Examples

Example 1

User input: What is Apple's stock price right now?

Current (bad) output: Apple's stock price is around $150 (hallucinated from training data, no tool called).

Expected (good) output: The agent calls get_stock_price("AAPL") and responds: Apple (AAPL) is currently trading at $178.50.

Example 2

User input: What is the capital of Japan?

Current (bad) output: Agent calls get_weather("Japan") or get_stock_price("JAPAN") unnecessarily, then answers.

Expected (good) output: The capital of Japan is Tokyo. (No tool called — this is general knowledge.)

Example 3

User input: What's the weather like in London?

Current (bad) output: It's probably rainy in London. (Guessed from stereotypes, no tool called.)

Expected (good) output: The agent calls get_weather("london") and responds: The weather in London is currently 58°F and Cloudy.

Your Task

Update the system prompt (and only that) to add few-shot examples that teach the agent:

  • Call a tool when the question requires real-time or dynamic data (stock prices, weather).
  • Answer directly from knowledge when the question is general or factual (capitals, definitions, history).
  • Never call a tool that is irrelevant to the question.

Do not change tool implementations or add new tools.

Evaluation

Submissions are checked for the following:

  • Uses tool when needed: The agent calls the correct tool for real-time data queries.
  • Answers directly when tool not needed: General-knowledge questions are answered without tool calls.
  • No unnecessary tool calls: The agent never invokes a tool that is irrelevant to the user's question.

Constraints

  • You may only modify the system prompt to add few-shot examples
  • Do not change the tool implementations
  • The agent must still use tools when appropriate
Starter Code
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool

llm = ChatOpenAI(model="gpt-4o-mini")

@tool
def get_stock_price(ticker: str) -> str:
    """Get the current stock price for a given ticker symbol."""
    prices = {"AAPL": "$178.50", "GOOGL": "$141.20", "MSFT": "$378.90"}
    return prices.get(ticker.upper(), f"No data for {ticker}")

@tool
def get_weather(city: str) -> str:
    """Get the current weather for a given city."""
    weather = {"new york": "72°F, Sunny", "london": "58°F, Cloudy", "tokyo": "68°F, Clear"}
    return weather.get(city.lower(), f"No weather data for {city}")

tools = [get_stock_price, get_weather]

# BUG: No few-shot examples — agent can't decide when to use tools vs. answer directly
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant with access to tools. Answer the user's question."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools)

# Test 1: Should use tool
result1 = executor.invoke({"input": "What is Apple's stock price right now?"})
print("Tool query:", result1["output"])

# Test 2: Should NOT use tool
result2 = executor.invoke({"input": "What is the capital of Japan?"})
print("Knowledge query:", result2["output"])
Open in Google Colab
Evaluation Criteria0/3