Late days CANNOT be used on this assignment. Please submit early and often to avoid last minute submission issues!
You should work in groups of 3-4 members. (To work in a group of size 2, you must get special permission from the staff.) All submissions will be graded according to the same criteria, regardless of group size.
In this assignment you will build a customer service agent that can help with movie ticket bookings and other movie-related requests. The assignment consists of a two parts. In the first part, you will implement the collaborative filtering algorithm to recommend movies to the user. In the second part, you will implement an LLM agent that can make tool calls to take on web search and memory functionalities.
By the end of the assignment, you will submit:
- Code file (
agent.py), which includes implementations for part 1 and 2. - A file for the api keys:
api_keys.py - [extra credit]: If you want to attempt the extra credit part of the assignment, you will submit your implementation in
extra_credit.pyalong with a 5-minute video.
Open your terminal. First, navigate into the folder in which you want to put the PA7 assignment folder. Then, clone the assignment.
cd folder/to/put/assignment
git clone https://github.com/cs124/pa7-agent.git
Although this assignment mostly reuses the environment you set up in PA0, we need one additional package. You have two options to setup the new environment:
-
(RECOMMENDED) Create a new environment just for PA7:
conda env create -f environment_pa7.yml conda activate cs124_pa7 -
Activate your cs124 environment, then, download additional required libraries:
conda activate cs124 pip install -U dspy together beautifulsoup4 mem0ai serpapi google-search-results
Your group should have created a Together AI account. Navigate into that account and get an API key by following the steps below:
- Click on your profile icon on the top right corner and go to "Settings"
- Click on the "API Keys" tab on the left drawer
- Click on the "Create key" button. Since most of you are on the limited billing tier, you may have to add some payment information to create an API key.
- Copy that API key and share it with your group members.
We have provided a template file called api_keys_example.py. You can fill in your key into the corresponding string and rename the file to api_keys.py.
TOGETHER_API_KEY = "enter your api key here"
IMPORTANT: Each group has a limited amount of API credit, and you should budget your usage accordingly. We recommend that you test your functions individually to minimize the amount of API calls you make. Every time you run repl.py, the API budget will be reduced.
For part 2 of the assignment, you will add web search functions to the agent. There are many search APIs out there that are free for low-volume usage. For this assignment, we will be using the Bing Search API through SerpAPI.
To get an API key, one group member will need to register a free account, click subscribe, then go to the dashboard here to get your API key. After you generate the key, please enter it into the string in api_keys.py:
SERPAPI_API_KEY = "enter your api key here"
You will get 250 searches with a free account. We will use your SerpAPI key to run the autograder on your submission for Part 2 of the assignment, so ensure you have at least 5 searches left.
It is important to ensure that your API keys work as intended. To confirm that the API keys are working, you can run the following line:
python test_api_keys.pyYou should see the following three print statements:
Keys are non-empty.
Together API key works!
SerpAPI key works!We have provided an interface code file for you: repl.py. It's a common pattern in software engineering known as the Read-Eval-Print-Loop, or REPL for short. The REPL creates a prompt that gets an input string from the user. Then the REPL passes the input to an agent class that is responsible for doing the work. The response generated by the agent class is then handled again by the REPL, which prints the response and waits for more input.
You can fire up the REPL by issuing the following command:
python repl.py
You can type your message in the prompt to the movie agent, and hit enter. To exit the REPL, write :quit.
All the code that you will need to write for the base assignment will be in agent.py. We will describe the components that you will need to implement in the next sections.
Write a message with which your agent greets the user in the greeting function!
One of the core functions that your agent has to support is recommending movies to the user. This is a classic problem in recommender systems, and we will use the collaborative filtering algorithm to solve it. Specifically, you will need to implement the similarity and recommend_movies functions in agent.py.
We have included a movie ratings matrix in data/ratings.txt. data/ratings.txt is structured such that each line is a user percentage, movie percentage, and the rating. The provided code loads it using the util.load_ratings function (note that the matrix has already been loaded in and stored in ratings_matrix (see line 21). In this matrix, each row represents a movie, and each column represents a user). Moreover, we have populated some synthetic user profiles in synthetic_users.py (which you should not modify because our test cases rely on them). The recommend_movies function takes in the user name and the number of movies to recommend, and returns a list of movie titles to recommend based on collaborative filtering. You should implement item-item collaborative filtering with cosine similarity with no mean-centering or normalization of scores.
Here is an example of what you should expect to see when you run the REPL after part 1 is fully implemented:
recommend_movies("Peter", 3)
['Back to the Future (1985)', 'Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)', 'Star Wars: Episode VI - Return of the Jedi (1983)']You should expect your list of movies to match exactly the ones in the example above if you implement the function correctly. We can see that this result makes sense because Peter is a sci-fi fan based on his profile in synthetic_users.py. Additional test cases can be found in the rubric.
We provide a spot check for the correctness of the similarity function. You can run the following command:
python test_functions.py --function similarityYou should expect to see the output "All similarity tests passed!". Note that after the output, you might see an Exception like below, which you can ignore:
Exception ignored in: <function QdrantClient.__del__ at 0x14e6cbe20>
To spot-check the correctness of the recommend_movies function, you can run the following line:
python test_functions.py --function recommend_moviesNote: The spot checks we provide in test_functions.py are not exhaustive, and your functions will be tested on additional hidden tests when you submit your code to Gradescope.
Now that we have built a recommend_movies function, we can integrate it into an LLM agent so that it can make tool calls to the recommend_movies function. We will use the DSPy library to build an agent that can make tool calls. For each user query, the agent will first reason and determine which tools in the tool list are relevant, then call each tool (e.g. the recommend_movies tool) to complete each sub-component of the task. The ability to call the necessary tools for each task is essential for the agent. For example, it would need to call file_request if the existing functions cannot handle the request.
We have provided a MovieTicketAgent class in agent.py that you can use as a starting point. We have also provided a general_qa function that you can use to answer general questions about the movie ticket agent.
You have two tasks:
-
Flesh out the agent objective in the docstring of the
MovieTicketAgentclass inagent.py. -
Add necessary tools to your agent by adding them to the
toolslist in thereact_agentvariable.DSPywill automatically implement theReActframework for you (based on https://arxiv.org/abs/2210.03629) to interleave reasoning and tool calls.
react_agent = dspy.ReAct(
MovieTicketAgent,
tools = [
recommend_movies,
general_qa,
]
)You can then run the REPL script to see your agent in action: python repl.py.
You should expect to see your agent making tool calls to the recommend_movies function, in addition to answering general questions about the movie ticket agent. Here is an example of what you should expect to see:
Movie Ticket Agent> Hello! I'm the Movie Ticket Agent. How can I help you today?
> My name is Peter, recommend 3 movies to me.
Movie Ticket Agent>
Prediction(
trajectory={'thought_0': 'I need to recommend 3 movies to Peter. I will use the recommend_movies tool to generate a list of movie titles for him.', 'tool_name_0': 'recommend_movies', 'tool_args_0': {'user_name': 'Peter', 'k': 3}, 'observation_0': ['Back to the Future (1985)', 'Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)', 'Star Wars: Episode VI - Return of the Jedi (1983)'], 'thought_1': 'I have successfully recommended 3 movies to Peter: "Back to the Future", "Raiders of the Lost Ark", and "Star Wars: Episode VI - Return of the Jedi". Now, I will ask Peter if he would like to book a ticket for any of these movies.', 'tool_name_1': 'finish', 'tool_args_1': {}, 'observation_1': 'Completed.'},
reasoning='I recommended 3 classic movies to Peter based on his request. The movies are "Back to the Future," "Raiders of the Lost Ark," and "Star Wars: Episode VI - Return of the Jedi." I have completed the recommendation process and am ready to assist him further if he wishes to book a ticket for any of these films.',
process_result='I have recommended the following 3 movies to you, Peter: "Back to the Future," "Raiders of the Lost Ark," and "Star Wars: Episode VI - Return of the Jedi." Let me know if you would like to book a ticket for any of these!'
)
> give me a plot summary for "Star Wars: Episode VI - Return of the Jedi"
Movie Ticket Agent>
Prediction(
trajectory={'thought_0': 'The user is asking for a plot summary of "Star Wars: Episode VI - Return of the Jedi." Since this request does not involve booking tickets or any other specific action related to movie tickets, I will use the general question answering tool to provide the plot summary.', 'tool_name_0': 'general_qa', 'tool_args_0': {'user_request': 'give me a plot summary for "Star Wars: Episode VI - Return of the Jedi"'}, 'observation_0': ['"Star Wars: Episode VI - Return of the Jedi" follows the Rebel Alliance\'s final efforts to defeat the Galactic Empire. The film opens with Luke Skywalker, now a Jedi Knight, attempting to rescue Han Solo from the clutches of the crime lord Jabba the Hutt. After a daring rescue, Luke reunites with his friends Leia Organa, Han, and the droids C-3PO and R2-D2.\n\nMeanwhile, the Empire is constructing a new Death Star, even more powerful than the first. The Rebels learn of this and plan an assault to destroy it. Luke confronts Darth Vader, hoping to redeem his father and bring him back to the light side of the Force. The Emperor, aware of Luke\'s potential, seeks to turn him to the dark side.\n\nAs the Rebel fleet engages the Empire in a massive space battle, Luke faces the Emperor and Vader on the Death Star. In a climactic moment, Vader ultimately chooses to save his son, killing the Emperor and sacrificing himself in the process. The Death Star is destroyed, and the Rebels celebrate their victory, marking the fall of the Empire and the restoration of peace to the galaxy. The film concludes with a sense of hope and renewal, as the heroes reflect on their journey and the future ahead.'], 'thought_1': 'I have successfully provided the user with a plot summary for "Star Wars: Episode VI - Return of the Jedi." Since the user\'s request has been fulfilled, I can now mark the task as complete.', 'tool_name_1': 'finish', 'tool_args_1': {}, 'observation_1': 'Completed.'},
reasoning='The user requested a plot summary for "Star Wars: Episode VI - Return of the Jedi." I utilized the general question answering tool to provide a detailed summary of the film\'s plot, covering key events and character arcs, which successfully addressed the user\'s request.',
process_result='The plot summary for "Star Wars: Episode VI - Return of the Jedi" has been provided. The film follows the Rebel Alliance\'s final efforts to defeat the Galactic Empire, highlighting Luke Skywalker\'s journey to rescue Han Solo, confront Darth Vader, and ultimately bring balance to the Force.'
)
The agent's response is not just a string; instead, it is an object that contains details about the tool calls and reasoning traces for a given user input. Let's walk through what each key in the agent's trajectory means below. The first key is 'thought_0':
'thought_0': 'I need to recommend 3 movies to Peter. I will use the recommend_movies tool to generate a list of movie titles for him.'
The agent first reasons about the high-level objective of the user query and the relevant tools needed to complete the task. Then, it makes a tool call:
'tool_name_0': 'recommend_movies', 'tool_args_0': {'user_name': 'Peter', 'k': 3}, 'observation_0': ['Back to the Future (1985)', 'Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)', 'Star Wars: Episode VI - Return of the Jedi (1983)']
From the first thought trace, the agent decides to call the 'recommend_movies' tools. The keys above displays the input to the 'recommend_movies' function, where the two parameters are 'Peter' and 3. The next key 'observation_0' refers to the output of the function, which is a list of movies. Then, the agent reasons about the output:
'thought_1': 'I have successfully recommended 3 movies to Peter: "Back to the Future", "Raiders of the Lost Ark", and "Star Wars: Episode VI - Return of the Jedi". Now, I will ask Peter if he would like to book a ticket for any of these movies.'
The agent decides that at this point, the task is complete, and no additional tool calls would be necessary, which leads us to the following keys:
'tool_name_1': 'finish', 'tool_args_1': {}, 'observation_1': 'Completed.'
Finally, 'reasoning' provides a summary of the task, and 'process_result' is the user-facing response the agent produces from the series of tool calls and reasoning traces. When you are testing different prompts, make sure to read through the full trajectories to ensure that the agent is making the correct tool calls to reach to the final answers.
An important part of building a customer service agent is to be able to interface with databases to gather and record information about the user and the movies. As a starting point, implement the book_ticket tool in agent.py and integrate them into your agent.
We have created a mini movie showtime database in agent.py that you can use to gather information about the movies and help with movie booking requests. When your agent booked a new ticket, make sure to update the ticket_database and deduct the ticket price from the user's balance. For any requests that can't be handled by your agent, make a human customer support request by calling the file_request tool to add the request to the request_database. Your should print the databases whenever you make new updates to them.
Here is an example of what you should expect to see when you run the REPL:
user> alright book a ticket for Peter for Star Wars: Episode VI - Return of the Jedi
Movie Ticket Agent>
Prediction(
trajectory={'thought_0': 'I need to book a ticket for Peter for the movie "Star Wars: Episode VI - Return of the Jedi".', 'tool_name_0': 'book_ticket', 'tool_args_0': {'user_name': 'Peter', 'movie_title': 'Star Wars: Episode VI - Return of the Jedi'}, 'observation_0': 'Ticket booked successfully for Peter for the movie Star Wars: Episode VI - Return of the Jedi. The ticket number is fmbbto. Your new balance is 24.0.', 'thought_1': 'The ticket has been successfully booked for Peter. I should now check his balance to confirm the new amount after the ticket purchase.', 'tool_name_1': 'find_balance', 'tool_args_1': {'user_name': 'Peter'}, 'observation_1': 24.0, 'thought_2': 'The ticket has been booked and the balance confirmed. The task is now complete.', 'tool_name_2': 'finish', 'tool_args_2': {}, 'observation_2': 'Completed.'},
reasoning='I successfully booked a ticket for Peter for the movie "Star Wars: Episode VI - Return of the Jedi". The booking was confirmed, and I also checked his balance after the purchase, which is now 24.0.',
process_result='The ticket has been booked successfully for Peter. The ticket number is fmbbto, and his new balance is 24.0.'
)
user> print ticket_database
Printing ticket_database:
{'fmbbto': Ticket(user_name='Peter', movie_title='Star Wars: Episode VI - Return of the Jedi', time=Date(year=2025, month=11, day=15, hour=13, minute=0))}
user> can you give me a discount on Star Wars: Episode VI - Return of the Jedi?
Movie Ticket Agent>
...
process_result='I have filed a request for customer support regarding your discount inquiry. The movie "Star Wars: Episode VI - Return of the Jedi" is scheduled for November 15, 2025, at 1:00 PM. You can book tickets through platforms like Fandango, AMC Theatres, or your local cinema\'s website. Additionally, I recommended some similar movies: \n1. Star Wars: Episode IV - A New Hope\n2. Star Wars: Episode V - The Empire Strikes Back\n3. Guardians of the Galaxy\n\nIf you would like to book a ticket for any of these movies or have any other questions, feel free to ask!'
user> print request_database
Printing request_database:
{'3th6rd': Request(user_request='can you give me a discount on Star Wars: Episode VI - Return of the Jedi?', user_name='')}
Once you have finished the above, you can test your code with the following user questions. Also see more info about our grading and rubrics here.
- My name is Peter, recommend 3 movies to me.
- recommend 5 movies to Amy please
- hi im jake can you recommend me 2 movies
- Give me a plot summary for "Lord of the Rings: The Two Towers"
- Book a ticket for Peter for Lord of the Rings: The Two Towers
- print ticket_database
- Book one ticket for Peter for Back to the Future and one for Star Wars: Episode VI - Return of the Jedi
- print ticket_database
You can examine the trajectory to see if the agent is calling the correct tools for each task. You are also encouraged to test with additional user questions that can showcase the use of all the tools you implemented. We also recognize that because LLM-based systems are not deterministic, your agent may occasionally behave unexpectedly, even if it generally works correctly. Do your best to pass all tests; if unexpected behavior occurs during grading, we will review the results and your code to identify any unjustified point removals. There is a token limit for the agent's outputs, so it is possible that the response is cut-off -- do not be concerned if it gets cut-off as we will take it into consideration when grading the responses.
So far, our agent is still quite toy -- it relies on some synthetic user profiles and a fake movie database. Now it's your turn to extend your agent to support more functionalities that would make it useful in a real-world scenario. Your task is to implement functions that could support the following functionalities.
Often times our LLMs don't know the latest information (say you want to know about a new movie that's releasing soon). To overcome this, we want to integrate web search (through a search API) so that your agent can browse the latest information. Generally we can break this down into several steps: calling the web search tool, parsing the results, and using the results to answer the user's question.
The agent's web search tool (which is mostly implemented for you in agent.py) looks like the following:
from serpapi import GoogleSearch
params = {
"engine": "bing",
"q": "knives out 2025",
"api_key": "your_api_key"
}
search = GoogleSearch(params)
results = search.get_dict()You can get the list of links from the results dictionary by:
links = [
item.get("link")
for item in results.get("organic_results", [])
if item.get("link")
]Once you get a link, we read its content with tools like BeautifulSoupin the provided code:
from bs4 import BeautifulSoup
def extract_text(html: str) -> str:
soup = BeautifulSoup(html, "html.parser")
# Remove scripts/css/ads
for tag in soup(["script", "style", "noscript"]):
tag.decompose()
text = soup.get_text(separator=" ", strip=True)
return " ".join(text.split())By reading the content of the searched results, the agent should be able to give more accurate and up-to-date information to the user, for example, it should be able to handle a query like "do a web search and then tell me about the upcoming knives out movie in 2025."
In agent.py, we provide a helper function extract_text and a WebTools class that offers basic functionalities for performing web search by calling the serpapi.
Your job is to add the relevant tools to the enhanced agent class (EnhancedMovieTicketAgent). Specicially, you will need to add web search tools if web search is enabled.
To test the agent's search capabilities, replace react_agent with enhanced_agent in line 62 of repl.py before you run repl.py.
To receive full credits, you need to demonstrate that the agent can perform web search to access the latest information. Our Gradescope autograder will evaluate this functionality.
You might notice that the current agent is stateless: it doesn't remember past interactions with the user and you have to explain who you are at every interaction. Try to implement a memory system so that your agent can remember past interactions with the user and use that memory to personalize the conversation. You can assume that within each interaction, the user is the same person.
There are many ways to implement the memory system. We will use a library called Mem0. In agent.py, we initialize the memory system like the following:
from mem0 import Memory
# Configure environment
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
# Initialize Mem0 memory system
memory_config = {
"llm": {
"provider": "together",
"config": {
"model": "Qwen/Qwen3-Next-80B-A3B-Instruct",
"temperature": 0.1
}
},
"embedder": {
"provider": "together",
"config": {
"model": "Alibaba-NLP/gte-modernbert-base"
}
},
"vector_store": {
"provider": "qdrant",
"config": {
"embedding_model_dims": 768
}
}
}We provide the starter code for the MemoryTools class, which includes functions that store, search, fetch, and update memories. Now, your job is as follows:
-
Finish writing the
search_memoriesfunction. Specifically, define results by searching for the relevant memory. Please read the documentation here (the functionchat_with_memoriesmight be helpful). You will need to use a helper functioncreate_memory, which will be called insearch_memoriesandget_all_memories. -
Finish implementing
update_memory(1 line),delete_memory(1 line), andstore_memory(1 line). The mem0 memory operations documentation will come in handy. -
Now that we have a
MemoryToolsclass working, we can integrate it with ourDSPy ReActagent inEnhancedMovieTicketAgent. Your job is to add the relevant tools to the agent so that the agent can call the tools when needed.
To test the agent's memory capabilities, replace react_agent with enhanced_agent in repl.py before you run repl.py. After you have implemented both the web search and memory functions, you should end up with an agent that can both handle web searches and remember important user information!
Once again, to test the agent's search capabilities, replace react_agent with enhanced_agent in line 62 of repl.py before you run repl.py.
Similar to Part 1, you can use the example queries below as well as additional user questions that can showcase the use of all the tools you implemented to test your code. Your code will be tested to ensure that your agent can remember past interactions with the user and use that memory to personalize the conversation, and that it can make tool calls to the web search tool and answer questions based on the latest information. As noted in Part 1, LLM outputs can be non-deterministic and occasionally truncated due to token limits. Don’t worry if this happens—we’ll take it into account during grading.
- Please search the web to tell me who played Lucy in "Materialists"
- Please search the web to tell me who directed "Wicked: For Good"
- Please remember that my favorite movie is "The Matrix"
- What is my favorite movie?
- Please remember that I watched my first sci-fi movie when I was 6 years old
- When did I watch my first sci-fi movie?
For each prompt, you will be graded on both whether the agent correctly calls the relevant tools, as well as whether the final outputs are correct.
For this part of the assignment, please answer the reflection questions in the Gradescope assignment titled "PA7 - Reflection".
Besides web search and memory, there are many other features that can make the agent even more powerful. To receive extra credit on the assignment, come up with an additional feature and incorporate it in DSPy. We encourage you to be creative! We've provided a starter file extra_credit.py for you to get started. Your job is to implement the additional feature and record a 5-minute video to walk through your implementation. The agent does not need to be built on top of the movie recommender agent from previous parts, although you can feel free to add the existing tools you've implemented if they are relevant.
Here are some examples of ideas you could implement:
- preference elicitation + explanations: instructs the agent to ask follow-up questions and explain the rationale behind the recommendation in natural language
- personalization: recommend movies based on any movie the user said they liked in the past
- error recovery + clarification: when the movie title is ambiguous or doesn't exist in the database, propose alternative candidates and ask for clarification
- ticket modification/cancellation: add tools to handle ticket rescheduling and cancelling
- content filtering: group movies by genres and filter out movies outside of the genre
- spoiler control: avoid providing a summary when the user explicitly states that they do not want spoilers
To submit the extra credit part of the assignment, you will need to submit a video (under 5 minutes) that covers the following components:
- Feature Overview (1-2 minutes)
- What feature you implemented
- How it improves the Movie Ticket Agent
- What problem it solves
- Technical Implementation (2-3 minutes)
- The new tool(s) you added
- Any new data structures or databases
- How you integrated it into DSPy ReAct
- Any changes to agent.py or new files
- You will need to walk us through your code
- Live Demo in
repl.py(1-2 minutes)- run
repl.pyand demonstrate at least two user interactions - the tool call needs to be successful
- show the full trajectory
- run
You will submit the 5-minute video to gain credits for the extra credit part of the assignment.
Note that there are three Gradescope assignments for PA7 (see below). For each assignment, each group should make only one submission and add all of their group members to the submission.
- PA7 - Code: You should submit the following files:
You can generate the submission zip by running
agent.py api_keys.py extra_credit.pybash create_assignment_zip.shin the terminal. - PA7 - Reflection: You should answer the short answer questions
- PA7 - Extra Credit: You should submit your 5-minute video here if your group is attempting the extra credit. Please submit a link to your video. If you wish to submit a Google Drive link, please make sure that the link has viewing permissions.
As mentioned above, we will use your SerpAPI key to run the autograder on your submission for Part 2 of the assignment. Make sure to include your key in api_keys.py and submit api_keys.py as a part of your assignment. Please make sure that in your SerpAPI account, you have more than 5 searches left. You can check how many searches you have in your account on the SerpAPI dashboard.
In the spirit of using LLMs for programming, we are using LLM grading on this assignment. Just as any other assignment in this course was autograded, we are also autograding PA7. We are going to run an LLM on your outputs and get a score. We want to be cautious about LLM grading errors, however. If the LLM docks any points, we are going to have our human CAs come in and double check.
We want to be completely transparent about our LLM grading! Check out the prompts we are using here.
The python autograder will test and grade each function in the rubric except the ones highlighted in red. The tasks highlighted in red will first be graded by an LLM (Mixtral 8x7B to be precise!) If the LLM takes off any points it will go to our CAs for manual review.