A Day With Claude

# CS 124 AI Chat Feature with RAG ## Overview Add an AI-powered chat assistant at `/chat` that helps students with CS 124 questions using RAG (Retrieval-Augmented Generation) to provide context-aware responses based on course materials. ## Key Decisions - Backend: `servers/api` (already has Azure OpenAI chat config) - Persistence: Store conversation history in MongoDB (7-day TTL) - Streaming: Non-streaming initially (simpler, can add SSE later) - Access: Current students and staff only (`ctx.hasRole` check) - Navigation: Direct URL access only (`/chat`) - no menu link yet ## Architecture Frontend (sites/cs124) Backend (servers/api) ┌──────────────────┐ ┌─────────────────────────────────┐ │ pages/chat.tsx │──POST────>│ server/chat.ts │ │ │ │ ├─ searchRAG() → Qdrant │ │ GPT-like UI │<─JSON─────│ ├─ chatCompletion() → Azure │ │ with sources │ │ └─ saveConversation() → Mongo │ └──────────────────┘ └─────────────────────────────────┘ ## Files to Create/Modify ### 0. `servers/shareable/server/rag.ts` (modify) Add sparse vector generation for hybrid search: // Add sparse vector generation function function generateSparseVector(text: string): { indices: number[], values: number[] } { // Tokenize, remove stopwords, compute TF weights // Use murmurhash for token → index mapping } // Update saveChunkToQdrant to store both vector types await qdrantClient.upsert(QDRANT_COLLECTION, { points: [{ id: formattedId, vector: { "dense": chunk.embedding, "sparse": generateSparseVector(chunk.text), }, payload: { ... } }] }) ### 1. `lib/collections/src/api.ts` (modify) Add collection constant: export const API_CHAT_CONVERSATIONS_COLLECTION = "api_chat_conversations" ### 2. `servers/api/client/chat.ts` (new) import { Array, Literal, Number, Optional, Record, Static, String } from "runtypes" export const ChatSource = Record({ contentId: String, title: String, timestamp: Number, author: String, type: Literal("walkthrough").Or(Literal("video")), url: String, }) export const ChatMessage = Record({ role: Literal("user").Or(Literal("assistant")), content: String, sources: Optional(Array(ChatSource)), timestamp: String, }) export const ChatRequest = Record({ message: String, conversationId: Optional(String), }) export const ChatResponse = Record({ response: String, sources: Array(ChatSource), conversationId: String, }) ### 3. `servers/api/client/index.ts` (modify) Add export: export * from "./chat" ### 4. `servers/api/server/chat.ts` (new) Key components: // Environment variables const AZURE_OPENAI_CHAT_ENDPOINT = String.check(process.env.AZURE_OPENAI_CHAT_ENDPOINT) const AZURE_OPENAI_CHAT_API_KEY = String.check(process.env.AZURE_OPENAI_CHAT_API_KEY) const QDRANT_URL = String.check(process.env.QDRANT_URL) const CHAT_RATE_LIMIT_PER_MINUTE = parseInt(process.env.CHAT_RATE_LIMIT_PER_MINUTE || "10") // System prompt const SYSTEM_PROMPT = `You are a helpful CS 124 teaching assistant... - Answer questions ONLY related to CS 124 (Java, Kotlin, introductory programming) - Direct students to relevant course materials with timestamps - Never provide complete homework/quiz solutions - Politely redirect off-topic questions` // Qdrant search function async function searchRAG(query: string, limit = 5): Promise<RAGResult[]> // Routes router.post("/", async (ctx) => { /* main chat endpoint */ }) router.get("/history/:conversationId", async (ctx) => { /* get history */ }) router.delete("/:conversationId", async (ctx) => { /* clear conversation */ }) ### 5. `servers/api/server/index.ts` (modify) Register chat router: import { router as chatRouter } from "./chat" .use("/chat", chatRouter.routes(), chatRouter.allowedMethods()) ### 6. `servers/api/package.json` (modify) Add dependencies: "@qdrant/js-client-rest": "^1.16.2", "openai": "^4.x.x" ### 7. `docker-compose.yml` (modify) Add env vars to api service: AZURE_OPENAI_CHAT_ENDPOINT: ${AZURE_OPENAI_CHAT_ENDPOINT} AZURE_OPENAI_CHAT_API_KEY: ${AZURE_OPENAI_CHAT_API_KEY} QDRANT_URL: ${QDRANT_URL} QDRANT_API_KEY: ${QDRANT_API_KEY:-} CHAT_RATE_LIMIT_PER_MINUTE: ${CHAT_RATE_LIMIT_PER_MINUTE:-10} ### 8. `sites/cs124/pages/chat.tsx` (new) GPT-like centered interface: - Message list with auto-scroll - User messages right-aligned (blue) - Assistant messages left-aligned with source citations - Input field at bottom with send button - Loading indicator during response - "New conversation" button to clear history Key patterns: const { headers, course } = usePersonable() // Require enrollment if (!course?.you) return <LoginRequired /> // Centered container <MainContainer maxWidth="md"> <Box sx={{ display: "flex", flexDirection: "column", height: "calc(100vh - 64px)" }}> {/* Scrollable message area */} <Box sx={{ flex: 1, overflow: "auto" }}> {messages.map(msg => <ChatBubble key={msg.id} message={msg} />)} </Box> {/* Sticky input */} <Paper sx={{ position: "sticky", bottom: 0 }}> <TextField ... /> <Button onClick={sendMessage}>Send</Button> </Paper> </Box> </MainContainer> ## API Contract ### POST /v1/chat Request: { "message": "What is a variable?", "conversationId": "uuid-optional" } Response: { "response": "A variable in Java is a named storage location...", "sources": [ { "contentId": "abc123", "title": "Variables and Types", "timestamp": 165, "author": "[email protected]", "type": "walkthrough", "url": "/play/abc123?t=165" } ], "conversationId": "uuid" } ### GET /v1/chat/history/:conversationId Returns conversation messages array. ### DELETE /v1/chat/:conversationId Clears conversation history. ## MongoDB Schema Collection: `api_chat_conversations` { conversationId: String, // UUID netID: String, email: String, semester: String, messages: [{ role: "user" | "assistant", content: String, sources: [ChatSource], timestamp: ISODate }], createdAt: ISODate, // TTL index: 7 days updatedAt: ISODate } ## Rate Limiting - 10 requests per minute per user (in-memory counter) - Returns 429 with retry-after header when exceeded - Use pattern from `servers/shareable/server/common.ts` ## System Prompt Strategy You are a CS 124 teaching assistant. Your role is to: 1. Answer questions ONLY about CS 124 content (Java, Kotlin, intro programming) 2. Reference course materials with timestamps when relevant 3. NEVER provide complete solutions to homework or quizzes 4. Guide students to understand concepts, don't just give answers 5. For off-topic questions, respond: "I can only help with CS 124 topics." When referencing content, cite: [Title by Author at MM:SS] ## RAG Retrieval Strategy Hybrid Search using Qdrant (Semantic + Keyword in one query): Qdrant natively supports hybrid search by storing both dense vectors (semantic) and sparse vectors (keyword/BM25) in the same collection. ### Setup (one-time, in shareable RAG indexing) // When indexing chunks, store both vector types await qdrantClient.upsert(QDRANT_COLLECTION, { points: [{ id: chunkId, vector: { "dense": embedding, // 3072-dim from text-embedding-3-large "sparse": sparseVector, // BM25-style sparse vector }, payload: { text, contentId, timestamp, author, ... } }] }) ### Query (in API server) async function searchRAG(query: string, topK = 5): Promise<RAGResult[]> { // Generate both dense and sparse vectors for query const [denseVector, sparseVector] = await Promise.all([ generateEmbedding(query), generateSparseVector(query), ]) // Qdrant hybrid search with RRF fusion built-in const results = await qdrantClient.query(QDRANT_COLLECTION, { prefetch: [ { query: denseVector, using: "dense", limit: topK * 2 }, { query: sparseVector, using: "sparse", limit: topK * 2 }, ], query: { fusion: "rrf" }, // Reciprocal Rank Fusion limit: topK, with_payload: true, }) return enrichWithMetadata(results) } Why Qdrant-native hybrid is better: - Single query instead of two separate systems - Built-in RRF fusion, no manual merging - Lower latency, simpler code Why hybrid search matters for CS 124: - "What is a variable?" → Semantic finds conceptually related content - "NullPointerException" → Keyword matches exact term - "public static void main" → Keyword matches code patterns ## RAG Context Format for LLM You have access to the following course materials to help answer the student's question. Reference these with [1], [2], etc. and I will include links for the student. [1] "Variables and Types" walkthrough by [email protected] [02:45] Variables in Java must be declared with a type. Unlike Python... [02:58] The primitive types include int, double, boolean, and char... [2] "Lesson 3: Data Types" video [05:30] When declaring a variable, think about what values it will hold... --- Student question: {question} ## Citations and Links The LLM response should include inline citations like [1], [2] which map to sources. Frontend renders citations as clickable links that open the video/walkthrough at the exact timestamp. ## Implementation Order ### Phase 1: Validate Hybrid Search Locally 1. Create migration script - `servers/shareable/bin/migrateQdrantHybrid.ts` - Reads all points from local Qdrant - Generates sparse vectors from payload text - Upserts back with both dense + sparse vectors 2. Test hybrid search - Write a simple test query to verify: - Semantic query: "how do variables work" - Keyword query: "NullPointerException" 3. Decision point: If hybrid works → continue. If not → fall back to full-text index. ### Phase 2: Build Chat Feature 4. Backend types - `lib/collections`, `servers/api/client/chat.ts` 5. Backend routes - `servers/api/server/chat.ts` with hybrid search 6. Docker config - Add env vars to docker-compose.yml 7. Frontend page - `sites/cs124/pages/chat.tsx` ### Phase 3: Update RAG Indexing 8. Update shareable RAG - Modify to generate sparse vectors 9. Update build script - Ensure lesson/MDX content also gets sparse vectors 10. Run checks - `rushx check` in all affected projects ### Fallback: Full-Text Index If sparse vectors don't work well, use Qdrant's payload text index instead: await qdrantClient.createPayloadIndex(QDRANT_COLLECTION, { field_name: "text", field_schema: "text", }) This requires no migration, just a one-time index creation. ## Verification 1. Start backend: `rushx backend` 2. Start frontend: `cd sites/cs124 && rushx start` 3. Navigate to `http://localhost:3000/chat` 4. Test scenarios: - Ask "What is a variable in Java?" → Should get response with sources - Ask "What's the weather?" → Should redirect to CS 124 topics - Send 11 messages quickly → Should get rate limited - Refresh page → Should see conversation history - Click "New conversation" → History should clear ## Future Enhancements (not in this PR) - SSE streaming for real-time response display - Navigation link in TopBar - Usage analytics dashboard - Feedback mechanism (thumbs up/down) - Context-aware follow-ups based on student's current lesson