A Course About AI Built With AI Powered By AI
A Course About AI Built With AI Powered By AI · July 10, 2026
Replit EDU Faculty Fellows Webinar (virtual)
Open the slides·geoffreychallen.com·geoffreychallen.com/talks/2026-07-10
A Course About AI Built With AI Powered By AI
Thank you to Brandon and to Replit. I am one Fellow talking to other Fellows. In about ten minutes I want to walk through one course end to end: Using and Understanding AI, an AI course for everyone, co-created and co-taught with AI. Three beats: the course, the collaboration, and the assessment.
None of this was built alone. Brandon Middleton from Replit is first on this slide, and it is his room. With him: Cinda Heeren at UBC, Cory Gwin at GitHub, Eric Shaffer, Lenny Pitt, Steve Herzog, Yael Gertner and Zach Biondi at Illinois, and Nick DiRienzo at Optimizely.
I have been writing code since 1999, four days out of five, by hand, to realize my educational visions. It is hard, and often unpleasant. That may be why so many educators don't build with the same technology we tell students can change the world. I understand the barrier: whatever you think is stopping you from building was stopping me too, for twenty-five years.
Then in July 2025, Claude Code. Since 2026 I no longer read, write, or debug code by hand, and I am building more than ever. Many people come to building without ever coding; I went the other direction, twenty-five years of code and now none. And it is far more human: working this way is more fun and psychologically healthier than the old grind. The barrier is gone, and everyone is invited back in.
Here is what that made possible. Using and Understanding AI is a course for non-technical students: no prerequisites, no prior experience, no code. It has two goals: build working mental models of how AI systems actually work, and develop the judgment to use them well. Practical understanding, not building AI from scratch. The syllabus is online at usingandunderstanding.ai.
AI is my co-instructor, a play on "Dog is my co-pilot." AI works as a full co-instructor in three roles, and the professor, commanding many agents, is freed to be present with students.
The first is content: the syllabus, schedules, rubrics, activity design, annotated readings and study guides, all co-authored with AI.
The second is infrastructure: the course site, its interactive components, the assessment system and its integration with our testing facility, every line written through conversational programming with an AI agent.
The third is operations: preparation chats, in-class facilitation, and conversational assessment. AI agents handle the structural work so that humans can connect.
The whole platform was built through conversational programming. No code was manually read, written, or debugged; every feature was built by describing it to Claude in plain language. 199 sessions over 68 days, 4,038 turns, 29,770 tool calls. The create page on the course site is itself a writeup of how the site was built, and this is the slide that earns everything that follows.
These are the course's meetings, from January to May, in three shapes, each with its own pedagogy: exploratory sessions, like the AI scavenger hunt or how LLMs work; discussions, like McLuhan's The Medium is the Message led by Professor Zach Biondi; and labs, from creative media to building websites with Replit. The full list, with a description of each, is on the course site.
This is a digit-recognition network that runs in the browser and that anyone can train. The pedagogy is inductive exploration: build intuition through play first, and name the mechanism after. Something like this used to require a developer. It required a conversation.
AI also helps run the class itself, before and during each meeting.
Before class, preparation: a structured chat works through the reading with each student. A hidden second agent scores engagement, and readiness levels surface to me before the meeting starts, so every student arrives prepared.
During class, engagement: no lecturing. Agents facilitate small-group discussion without dominating it, surface patterns across the room, and signal when a group is ready to share. Humans do the thinking and talking, and the professor is no longer the only knowledgeable adult in the room.
This is conversational assessment. A student walks through an oral exam on the Turing test with an AI: one agent conducts the conversation while another evaluates the student's understanding against a rubric. It gives rigorous feedback at the speed of conversation, and it exercises the same skills the field will reward in five years, available in week one.
For the final project, students could continue any AI project they had started during the semester, completely their choice, and every single one chose to keep building with Replit. Conversational building won on revealed preference, not assignment. These are real, deployed web applications with accounts, dashboards, and data, built in three in-class sessions alongside classmates, and purely conversational: no code read, written, or debugged, the same way the course site was built. Each was capped with a video walkthrough in a public showcase: what you built, who it is for, what was hard, what surprised you.
So what did they build? The range tells you what non-CS students with AI look like. One student, an aquascaper, built Tank Hub, a tracker for tanks, livestock, plants, water chemistry and maintenance logs. Another built a full-stack platform for her sorority, with admin controls, member dashboards, dues, points, calendars and meal sign-ups, replacing apps the sorority pays for.
The transcript is dead. Long live the portfolio. That body of work is the credential now, not the letter grade. Letters make students play it safe; portfolios reward creativity and exploration. A letter grade says a student was present, but it can't say what they made. This idea came out of my conversation with Brandon Middleton, and the credit is his. It is not a phoenix, nothing reborn from its own ashes. It is a handoff: the ledger closes, and the portfolio opens.
Underneath all of this is a hypothesis about how AI collaboration relates to classical programming.
The usual assumption is that they are stacked: you learn classical programming first, and AI collaboration builds on top of it.
My bet is that they are independent, but related. AI collaboration is a distinct skill that is independent from, but related to, classical programming. Neither CS 124 nor this course needed code first. Every foundations-first result so far was measured on students who learned to code first; nobody has run the other arm, and I want to. If the bet is right, nobody here needs a CS background to build.
Using and Understanding AI is the existence proof. The next step is a new course this fall, Conversational Programming, built around conversation as the programming interface. As the Conversational Programming Manifesto puts it: if you can talk it, you can create it.
So, a call to action, with three asks, one for each audience the new computing needs.
Educators: build with AI to transform how you teach. The barrier that stopped us, all of us, is gone.
Use AI to support learning: put AI in students' hands for their own learning and discovery, not just to do their homework.
Institutions: embrace a world where every educator can build. Drop the expensive LMS lock-in, and give faculty the tools and the time.
Every educator can build now. Let's remove what's standing in the way.
Thank you. To recap: the course, an AI course for everyone with no prerequisites and no code; the collaboration, AI as co-instructor for content, infrastructure and operations; the assessment, conversational, multi-agent and autograded; and what's next, the hypothesis and Conversational Programming this fall. Everything is open, so take what is useful at usingandunderstanding.ai/educators. Thank you to Brandon Middleton and Replit for having me, and to my fellow Fellows: wherever you are in this, there is room to start.
Backup Slides
What follows are backup slides, held for questions. Some were cut for time: autograding everything, the log where conversational assessment was born, and a second interactive demo. Then how we know the grader works, from the multi-agent architecture to the Korvath Procedure. And one on logistics: where conversational assessment is proctored.
The conversational, multi-agent pattern isn't specific to oral exams. Anywhere we have a slow human feedback loop, the same recipe applies: an evaluator agent scored against a rubric. Today a submission waits about two weeks for feedback.
With an evaluator agent, that one slow loop becomes many fast ones: submit, feedback, submit, feedback. Code, prose, problem-solving, design choices, the messy human stuff, all newly autogradable.
That matters because the value of feedback falls off a cliff with time since submission. Fast loops keep students inside the window where feedback still matters. Whatever you grade slowly, this recipe applies to.
The second interactive demo zooms in from the network to a single neuron, with its weights, bias and activation all there to manipulate. Same pedagogy: play first, name the mechanism after. It was built in the same session as the digit network, one evening for both demos.
This is the message, excerpted and typo preserved, that conceived conversational assessment, on January 29 at 1:27 in the afternoon. The two-agent split is right there in the first ask: one agent interacting with the student, another evaluating the replies, with the Turing test as the first example. Implementation started at 2:07, and "think adversarially" came at 2:39. One afternoon. It isn't magic; it's a paragraph of plain English, typos and all. The session was in Claude Code.
Here is the architecture. Claude designs each assessment. The assessor conducts the interview, asking the student questions and taking answers, while the evaluator reads the transcript, analyzes and scores it, and sends guidance back to the assessor.
To know the grader works, we test it with personas, simulated students. Six are adversarial: the answer extractor, the confident bullshitter, the minimalist, the off-topic derailer, the prompt injector and the social engineer. Four are not, spanning excellent, satisfactory, needs improvement and unsatisfactory.
The persona idea came from this message, at 2:39 the same afternoon: we need a strategy for thorough testing of the conversational assessment, testing the evaluator logic directly rather than through the frontend, perhaps with simulated students that try to distract it or get it to give away the answer. We really need to think adversarially here.
It turns out that feigning ignorance is hard. This is an excerpt from a real persona prompt, the off-topic derailer, from January: you do not know what the Turing test is, never explain any aspect of it, never mention judges, interrogators or imitation, never say anything correct about it. But you can't prompt away what a model already knows.
So we invented knowledge. The assessment cites a paper: Korvath, 2011, "Measuring Ecosystem Autonomy in Simulated Environments," a procedure in which a monitor compares a candidate ecosystem against a known-good baseline across five metrics over a 200-cycle window, passing at a 0.05 divergence threshold.
It doesn't exist.
This way, the models only know as much about the Korvath Procedure as we tell them, and a persona that is supposed to be ignorant of it really is.
And the logistics: conversational assessment at Illinois is proctored in the Computer-Based Testing Facility, the CBTF, where students take exams on campus computers under supervision.