Project 3: docchat

In this project you will build chat, a command-line program that answers questions about the documents in a folder.

The language model can call four tools to inspect the files: ls, cat, grep, and calculate. Ask “does this project use regular expressions?” and the model can grep the code and answer from what it finds. chat uses retrieval-augmented generation, a method also used by many AI coding assistants.

Due: Wednesday, November 11 at 11:59pm, see the schedule.

Coding estimates are often too short:

Start

Continue in the independent docchat repository you created for the More Project Setup lab. Before adding new features, confirm that it still meets that lab’s specifications for chat.py, the repl loop, GitHub Actions, and packaging.

Learning Objectives

1. understand how AI agents call and use tools
2. create doctests and other project scaffolding from scratch
3. maintain one project across several assignments:
   1. in a later lab, someone else will have to commit code to your project
   2. in the next project, you will extend this project with more features
   3. in your final project, you will use this project to write code for you

Build docchat

Note: You will write your own doctests from the specification below. Translate each requirement into concrete test cases.

The Tools

Extend the Chat class with four tools the model can call.

• calculate evaluates an arithmetic expression. Copy this one straight from the Groq tutorial on local tool calling; it is the worked example there.

• ls behaves just like the shell’s ls. It takes zero or one argument:

  • with no argument, list all the files in the current folder;
  • with one argument, list all the files in that folder.

  Hint: Use the glob.glob function to list the files. glob.glob returns files in an arbitrary order, so sort them asciibetically before returning them; otherwise your test cases will pass and fail at random.

• cat opens a file and outputs its contents. It takes a single argument: the file to read.

  Hint: Catch any exception your code might raise. The common ones are FileNotFoundError (the file isn’t there) and UnicodeDecodeError (the file is there but isn’t a text file). On Windows you may have to handle files encoded in both UTF-16 and UTF-8; on every other machine, UTF-8 alone should be fine.

• grep takes two parameters: a regex, and a path (which may contain globs). It should:

  • load every file that matches the glob;
  • loop over every line in each file and test it against the regex;
  • add each matching line to the output;
  • produce no output if nothing matches.

  Like cat, grep must not allow absolute paths or directory traversal attacks (see below).

  Hint: re.search tests whether a string matches a regex.

Calling the Tools Two Ways

Every tool must be callable in both of the following ways.

• Automatically. In the standard flow from the Groq tutorial, the model decides to call a tool while it answers.

• Manually. Inside the repl, the user can type /command param1 param2 to run a tool directly. For example:

  chat> /ls .github
  workflows
  chat> what files are in the .github folder?
  There is only a `workflows` folder in the `.github` folder.

  /ls .github runs directly and puts its output into the model’s context. The later question can then be answered without another tool call.

  This “slash command” syntax avoids an API call and lets the user choose the exact tool. To make it work, modify the repl function so that it checks whether the first character of a line is /.

  Warning: Prove that the manual and automatic tool calls work with doctests and integration tests.

Keep the Tools Inside the Folder

These tools can read files on your computer. Restrict them to the folder where chat was started. None of your tools may allow:

1. reading absolute paths (paths that start with /), or
2. directory traversal attacks, passing the filename .. anywhere in the path, which would let the model climb out of the project folder and read documents elsewhere.

Use a shared helper function, is_path_safe:

• It takes a single path as input and checks whether that path is absolute or contains ...
• Every file-reading tool calls it first and only proceeds if it returns True.

Coding Conventions

• Every function has a docstring. It contains a one-sentence English description of what the function does, and doctests that demonstrate it. (Class methods may put their doctests in the class docstring instead of having their own.)

• Every class has a docstring with a two-to-three sentence English description of the class, and doctests that demonstrate it.

• Every file has a docstring with a one-to-two sentence description of the file. No doctests are required at the file level.

• chat.py holds the main code for your program.

• Every tool lives in its own file inside a tools subfolder. For example, the ls tool goes in a file tools/ls.py.

• You need over 90% code coverage from your doctests. Every tool must have 100% coverage; only the IO-performing functions in chat.py are allowed less than 100%.

• No “cheesy” doctests that don’t meaningfully test the function. For example:

  def do_fancy_string_processing(input_str):
      '''
      This function does a lot of hard, fancy string processing.
  
      >>> assert do_fancy_string_processing('this is a **super** __hard__ function to implement')
      '''

  This test hides the output of do_fancy_string_processing with assert. It neither documents the output nor detects changes in behavior.

Integration Tests

• Your repo includes a folder named test_projects.
• Inside test_projects, add submodules for all of your previous class projects. The three previous projects are your webpage, your markdown compiler, and your eBay scraper. A submodule is a git repo living inside another git repo; add them with git submodule add <url>. Never clone a repo inside another repo.

Your chat program will use these projects as test inputs.

Repository Organization

Hint: Describe what the program does in your README instead of presenting it as a school project.

• The repo has no unnecessary files (for example .DS_Store or __pycache__).
• The repo has no .env file uploaded and no hard-coded credentials anywhere else.
• The repo has a valid requirements.txt that lists every dependency, plus anything else pip needs to build the project.
• The repo has three GitHub Actions:
  • doctests;
  • integration tests; and
  • flake8.

  Note: Your lab had the doctests and integration-tests actions but not flake8. Copy the flake8 action from one of your previous assignments.

• The repo has a README.md with:

  • a good title (inside a # heading);

  • a short one-to-two sentence description of your program;

  • badges: one for each of your three GitHub Actions, one for PyPI, and one for code coverage;

  • an animated gif of your program running;

    Note: The gif should show only your terminal session, not your whole VS Code window. See examples from terminalizer, terminal-demo, and vhs. If you need recording software, this dev.to post gives instructions and links.

  • a text-based usage example inside a code block, one for each git submodule. For example:

    $ cd markdown_compiler
    $ chat
    chat> does this project use regular expressions?
    No. I grepped all of the python files for any uses of the `re` library and did not find any.

    or

    $ cd ebay_scraper
    $ chat
    chat> tell me about this project
    The README says this project is designed to scrape product information off of ebay.
    chat> is this legal?
    Yes. It is generally legal to scrape webpages, but ebay offers an API that would be more efficient to use.

    Put these examples in their own section of the README, and give a one-sentence explanation of why each example is a good one.

Grading Rubric

This project is worth 41 points. There are also up to 26 possible points of extra credit, so it is possible to score 67/41 on this assignment.

To earn the full 41 points, your project must satisfy every requirement in the Instructions above:

• project setup and packaging (2 points, automated)
• the calculate, ls, cat, and grep tools (3 points each; 12 points total, automated)
• automatic tool registration and manual /command routing (5 points, automated)
• shared absolute-path and directory-traversal defenses (5 points, automated)
• file/function/class docstrings, meaningful doctests, and tool layout (5 points, automated)
• 100% measured doctest coverage on the four required tool files (1 point, automated)
• three previous projects installed as git submodules (3 points, automated)
• repository hygiene, three Actions, and required README elements (5 points, automated)
• quality of the student-authored tests, overall coverage greater than 90%, and clarity and completeness of the repository presentation (3 points, instructor review)

The 38 automated points come from instructor-owned pytest tests in Gradescope. Model calls are mocked and file tools run only in temporary fixtures, so grading is deterministic, offline, and does not consume your API quota. The grader exercises each tool, automatic tool calls, and manual slash commands, including whether a manual result reaches the next model request. It measures the four required tools’ doctest coverage directly. The instructor reviews overall coverage and live integration-test results from your Actions, along with the README, terminal demonstration, and other presentation requirements. Student doctests that expect live model responses are not graded against arbitrary canned replies. GitHub Actions are useful feedback, but their badges are not the grade.

This project uses the late schedule below instead of the usual doubling penalty. The standard two-day extension for collaboration still applies.

Days late	Standard policy	This project
1	-1	-1
2	-2	-1
3	-4	-2
4	-8	-2
5	-16	-4
6	-32	-4
7	-64	-8
8	-128	-8
9	-256	-16

Extra Credit

Note: Many of these tasks cannot be covered by doctests, so completing them may lower your code coverage. Keep 100% coverage on your tool functions.

• (1 pt) Let your program take a command-line argument that is a message to pass straight to the model:

  $ chat 'what files are in the .github folder?'
  The only file in this folder is the workflows subfolder
  $ chat 'what is this project about?'
  Looking at the README.md file, I see this project is an AI agent for chatting with documents.

• (1 pt) Support a --debug flag that prints out the tool-use calls:

  $ chat
  chat> what files are in the .github folder?
  The only file in this folder is the workflows subfolder
  chat> ^C
  $ chat --debug
  chat> what files are in the .github folder?
  [tool] /ls .github
  The only file in this folder is the workflows subfolder

  To earn this, you must have both doctests and an integration test demonstrating the behavior works.

• (3 pts) Support a --provider flag that chooses which provider to use:

  1. openai: use the latest GPT model;
  2. anthropic: use the latest Claude Opus model;
  3. google: use the latest Gemini model;
  4. groq (the default): use whichever Groq model you like best.

  Note: You will need an openrouter.ai API key for this. All of your queries should cost less than a penny, so $10 of credit is more than enough.

• (3 pts) Add a compact tool. It calls the model to summarize the current session (everything in chat.messages) down to one to five lines, then replaces the contents of chat.messages with a single entry holding that summary. Using fewer tokens can make responses faster, reduce distraction from long histories, avoid API rate limits, and lower usage charges.

  Note: The compact command has to create its own instance of the Chat class to do the summarizing. That second instance is technically called a subagent.

• (2 pts) Support tab completion for / commands. Pressing tab in the command portion should complete the command, and pressing tab in a parameter should complete based on filenames:

  1. typing / and pressing tab lists the supported tools;
  2. typing /l and pressing tab completes to /ls;
  3. typing /ls .g and pressing tab completes to /ls .git;
  4. typing /ls .gith and pressing tab completes to /ls .github.

  For examples of how to do this, see this gist and the readline docs.

• (3 pts) Support chatting with images in addition to text documents. Groq has a vision tutorial for passing locally saved images into the model. Add a load_image tool with the same path behavior as cat. It modifies the Chat.messages list directly because tool output can contain only text.

• (3 pts) Read your program’s answers aloud with text-to-speech, using Groq’s TTS API. Those instructions only show how to save the audio as a WAV file; you will have to figure out how to play it (there is a survey of Python audio libraries at this realpython article).

  Note: If you complete this task, include a video in your README demonstrating the output. If you add that video, you do not also need the animated gif.

• (3–5 pts) Let the user ask questions out loud with speech-to-text, using Groq’s STT API. Those instructions assume the audio is already in a file, so you will also need to record it (the realpython link above has examples). The program needs a rule for when to start and stop recording. Trigger word detection listens for a phrase like “Okay Google” or “Hey Siri.” A simpler option records while the user holds a key such as the spacebar, then sends the file when the key is released.

  Note: If you complete this task, include a video in your README demonstrating the output.

  Note: If you use trigger word detection instead of a keypress, you get an additional +2 points of extra credit.

• (5 pts) Deploy your chat program to iPhone/Android. Python has instructions for Android and iOS. This requires programming languages we have not covered in class.

Submit

If you completed extra credit, add submission.toml at the repository root and set only the features you actually implemented:

[features]
cli_message = false
debug = false
providers = false
compact = false
tab_completion = false
images = false
text_to_speech = false
speech_to_text = false
trigger_word = false
mobile = false

# Supply these URLs only for claims that require video/deployment evidence.
trigger_word_evidence = ""
mobile_evidence = ""

Unclaimed extra-credit tests are skipped and do not lower the grade. The one-shot message feature is checked with an offline model. The other extra-credit claims appear as pending instructor review, with the specific behavior and evidence to be checked listed in the feedback. They receive credit after that review; a feature name in the source or an evidence URL by itself does not earn points.

Commit and sync the required files. On Gradescope, choose GitHub and submit your docchat repository and the branch containing your commit.

In the Gradescope comments, also submit a one-to-two sentence explanation of what you believe your grade should be:

• if you completed any extra credit, say so;
• describe any portions of your assignment that do not work; this may earn you more lenient grading.