AI Virtual Teaching Assistant
- problem
- Students need answers about a course that stay grounded in the course's own material.
- approach
- A modular pipeline where chunking, embeddings, retrieval (dense, BM25, hybrid), query transformation (HyDE, multi-query), RRF fusion, reranking and generation are each chosen by name in a YAML config; evaluation runs save their exact config and per-question results so a change can be measured.
- stack
- RAG · NLP · Embeddings · ML Systems
- links
- Code ↗
What it is#
A retrieval-augmented question answering system for the IIT Madras Tools in Data Science course. It answers student questions from the course's own notes and forum, with links to the sources. It is also built as a small research lab, to make it cheap to try a new RAG idea and measure whether it helped.
Why it exists#
Answers about a course need to stay grounded in that course's material, and it is hard to tell whether a change to a retrieval pipeline really improved anything. So every stage is swappable and every experiment leaves a record.
How it works#
Chunking, embeddings, retrieval (dense, BM25 or hybrid), query transformation (HyDE or multi-query), rank fusion, reranking and generation are each picked by name in a YAML config. Each evaluation run saves its exact config, per-question results and summary metrics, with a fingerprint so two runs can be compared fairly. Embeddings are cached by content hash, so changing the retriever or reranker costs nothing and a new chunker only embeds new chunks. The same pipeline runs behind a FastAPI endpoint in Docker.
Stack#
Python, FastAPI, Docker, YAML configs, BM25 and dense embeddings, an LLM judge for rubric scoring, pytest and CI.
Status#
It works end to end, and has an offline test suite and a retrieval-only mode that needs no API key. The evaluation set has 13 labelled questions, which the README itself says is enough to catch regressions but not to rank close approaches. Growing that set is the next job.