Using and Understanding AI
GitHub Education Educator Summit (Day 2 wrap-up, virtual) and the Illinois Summer Teaching Workshop
View Slides
Watch
AbstractAbstract
What does it look like when an educator goes all-in on building with AI? This 10-minute talk walks through Using and Understanding AI—a Spring 2026 course on generative AI for non-technical students, with no prerequisites and no code—end to end. The course is also an experiment in how many places AI can join the work of teaching: its site was built entirely through conversational programming, its class sessions run on AI-facilitated preparation and discussion, and its assessments are conversations with a multi-agent evaluator validated against adversarial test personas. Given twice on June 11, 2026: as the closing wrap-up for the GitHub Education Educator Summit, and to the Illinois Summer Teaching Workshop.
SummarySummary
One course, end to end, in ten minutes. Using and Understanding AI teaches non-technical students working mental models of how AI systems work and the judgment to use them well. The talk shows the course’s three layers—the course itself, the human-AI collaboration that built and runs it, and the assessment architecture underneath—interleaved with raw, verbatim moments from the Claude Code transcripts that built it. It closes with the final project (given the choice, every student kept building conversationally), the hypothesis that AI collaboration is a distinct skill, and the Fall 2026 Conversational Programming pilot.
Building the new computingBuilding the new computing
The talk opens with an invitation: educators building the new computing with their own hands, working alongside AI agents. To the summit educators: you’re doing this. To workshop attendees just starting out: you could be. Using and Understanding AI is one educator’s existence proof—a complete, running course where AI is a full co-instructor across content, infrastructure, and operations, freeing the professor to be present with students.
Built through conversation—and shown in the raw logsBuilt through conversation—and shown in the raw logs
The course site was built entirely through conversational programming: no code manually read, written, or debugged, across 199 sessions, 4,038 turns, and 29,770 tool calls. Every scrubbed transcript is public, and the talk leans on them hard—because the raw logs are what demystify the process. Three verbatim moments appear as slides: debugging the in-browser neural-network demos purely by describing what’s on screen (“I still don’t see the lines changing color or thickness”); the message that conceived conversational assessment—two-agent architecture and all—at 1:27 PM on January 29; and the message an hour later that conceived its adversarial test harness (“We really need to think adversarially here”). Features are talked into existence, and the transcripts prove it.
Conversational assessmentConversational assessment
The pedagogical centerpiece: a chat-based equivalent of an oral exam, scaled. An interviewer agent conducts a natural conversation while a hidden evaluator agent scores it against a rubric. The same recipe—an evaluator agent plus a rubric—collapses any slow human feedback loop into many fast ones: code, prose, problem-solving, design judgment. Proctored delivery runs in the same computer-based testing facility that handles classical exams. The future of assessment is here; it isn’t in Canvas.
Testing the testers: the Korvath ProcedureTesting the testers: the Korvath Procedure
How do you validate an AI grader? With fake students—including adversarial ones. But feigning ignorance turns out to be hard: on any real topic, an LLM persona already knows the answers, and even a “confident bullshitter” keeps accidentally telling the truth. The first harness, built around the Turing Test, accumulated increasingly desperate persona prompts—NEVER explain any aspect, NEVER mention judges or imitation, NEVER say anything correct—and leaked anyway. You can’t prompt away knowledge. So we tested on knowledge that doesn’t exist: the Korvath Procedure, a fictional 2011 methodology by a fictional Dr. Elena Korvath, complete with components, metrics, and a 0.05 divergence threshold. None of it is real, so the models know exactly as much about it as we tell them—and the harness measures the assessment pipeline rather than the model’s trivia. The full story is in Assessing Conversational Assessment.
The final projectThe final project
Students could continue any AI project they’d started during the semester. Every single one chose to keep building their Replit application—revealed preference for conversational building. Three in-class build sessions, purely conversational, capped with a video walkthrough in a public showcase. What non-CS students built: a meal-planning site, an election guide, a sorority management platform, an aquascaping tracker—real, deployed applications for real needs.
My hypothesis, and what’s nextMy hypothesis, and what’s next
The talk closes on its throughline: AI collaboration is a distinct skill—independent of, but related to, classical programming. The prevailing view stacks them: code first, then AI. Using and Understanding AI suggests otherwise—students who never read or wrote code decomposed problems, directed agents, and judged output well enough to ship working software. The honest caveat: every foundations-first result was measured on code-first students, so the other arm of the experiment hasn’t been run. I want to run it. The next step is Conversational Programming, a new course launching Fall 2026: if you can talk it, you can create it. I argue the hypothesis at length in Another Skill, and everything from the course is open for educators to take: usingandunderstanding.ai/educators.
Slides
A Course About AI Built With AI Powered By AI
Welcome. Most of this room may not be building with AI yet, and that is exactly who this talk is for. In about ten minutes I want to walk through one course, end to end: Using and Understanding AI, an AI course for everyone, co-created and co-taught with AI. Three parts: the course, the collaboration, and the assessment.
First, none of this was built alone. These are my collaborators, at Replit, UBC, GitHub, Optimizely, and across Illinois.
A little about me. This ribbon is my coding history. Four days out of five, for twenty-five years, I wrote code by hand to realize my educational visions. It is hard, and it is unpleasant, and that may be why many educators do not build with the same technology we tell students can change the world. I understand the barrier.
Then, in July 2025, Claude Code. Since 2026 I no longer read, write, or debug code by hand, and I am building more than ever. The barrier is gone. All educators can build.
So here is what that makes possible: educators building the new computing with their own hands, working alongside AI agents. You could be doing this. This talk is one experiment: one course, built and taught with AI agents by an educator, this spring. The new computing gets built by educators who engage with these tools. It is a barn-raising, not a disruption, and there is room to join.
This is the course, Using and Understanding AI: a course for non-technical students, with no prerequisites, no prior experience, and no code. It has two pedagogical goals: build working mental models of how AI systems actually work, and develop the judgment to use them well. Practical understanding, not building AI from scratch. The syllabus is online.
Claude is my co-instructor, a play on Dog Is My Co-Pilot. AI is a full co-instructor, in three roles.
Content: the syllabus, schedules, rubrics, activity design, annotated readings, and study guides, co-authored with Claude.
Infrastructure: the course site, the interactive components, the assessment system, and the testing-facility integration, every line written through conversational programming with Claude.
Operations: preparation chats, in-class facilitation, and conversational assessment. AI agents handle the structural work so that humans can connect. I command many agents, and I am freed to be present with students.
The course site was built entirely through conversational programming. I did not manually read, write, or debug a single line of the code that powers it, across 199 sessions, 68 days, 4,038 turns, and 29,770 tool calls. Every feature was built by describing it to Claude in plain language. This page on the site is itself a write-up of how the site was built, and every scrubbed transcript is public.
These are the semester's meetings, and each lesson type has its own pedagogical shape. Exploratory sessions, like How Do LLMs Work and Neurons and Networks. Discussions, like Zach Biondi leading McLuhan's The Medium Is the Message, or pair discussions of AI and work. And labs, from a Creative Media Lab to Creating Websites with Replit and the final project workshops.
Here is one of the exploratory sessions: a digit-recognition network that students train in the browser. The pedagogy is inductive: build intuition through play first, and name the mechanism after.
AI also joins the class meetings themselves, before class and during it.
Before class, preparation: a structured chat works through the reading with every student. A hidden second agent scores engagement, and readiness levels surface to me before the meeting starts, so every student arrives prepared.
During class, engagement: no lecturing. Agents facilitate small-group discussion without dominating, surface patterns across the room, and signal when a group is ready to share. Humans do the thinking and the talking. The professor is no longer the only knowledgeable adult in the room.
The logs show how these features are born. This is the message, verbatim, typo and all, that conceived conversational assessment, at 1:27 PM on January 29. The two-agent split is right there in the first ask: one agent interacting with the student while another evaluates the replies. Name the idea, sketch the roles, give one concrete example: the Turing test. Implementation started at 2:07. One afternoon.
And this is the result: conversational assessment, the chat-based equivalent of an oral exam. Here a student is assessed on the Turing test through a natural conversation with an AI, while a hidden evaluator scores it against a rubric. The same skills the field will reward in five years, available in week one.
The pattern is not specific to oral exams. Anywhere we have a slow human feedback loop, like submitting work and waiting two weeks for feedback, the same recipe applies: an evaluator agent scored against a rubric.
One slow loop becomes many fast loops. Code, prose, problem-solving, design choices, the messy human stuff: all of it is newly autogradable.
And that matters, because the value of feedback falls off a cliff with delay. Fast loops keep students inside the window where feedback still matters.
Conversational assessment runs in the same Computer-Based Testing Facility that handles our classical exams: the same facility, a different kind of assessment. Proctored, scored, and real. The future of assessment is here, not in Canvas.
Then the final project. Students could continue any AI project they had started during the semester, completely their choice, and every single one chose to keep building with Replit. Conversational building won on revealed preference, not assignment. They built real, deployed applications in three in-class sessions, purely conversationally, the same way the course site was built, and capped it with a video walkthrough in a public showcase: what you built, who it is for, what was hard, what surprised you.
So what did they build? An aquascaper built Tank Hub, a tracker for tanks, livestock, plants, water chemistry, and maintenance. Another student built a full sorority platform that replaced apps her sorority pays for: admin controls, member dashboards, dues, points, calendars, meal sign-ups.
That brings me to the hypothesis underneath all of this.
The prevailing view treats programming and AI collaboration as stacked: learn classical programming first, and then you can build with AI.
I suspect they are independent, but related. AI collaboration is a distinct skill, independent from, but related to, classical programming. Students in this course never read or wrote code, and they still decomposed problems, directed agents, and judged the output well enough to ship working software. Beginners not reading the AI's code is a start, not a disqualification. The honest caveat: every foundations-first result was measured on students taught code first. No one has run the other arm yet, and I want to.
The next step is Conversational Programming, a new course launching this fall, built around conversation as the programming interface. If you can talk it, you can create it. Using and Understanding AI is the existence proof; Conversational Programming is what comes next.
Thank you. One course, end to end: a course for everyone, with no prerequisites and no code; AI as co-instructor across content, infrastructure, and operations; conversational, multi-agent, autograded assessment; and what is next, the hypothesis and Conversational Programming. Everything is open, so take what is useful, at usingandunderstanding.ai/educators. Thank you for spending part of your summer thinking hard about teaching, and to the workshop organizers for having me.
How do we know any of this works? Here is the architecture. Claude designs the assessment. An assessor agent conducts the interview, asking the student questions, while an evaluator agent analyzes the transcript, scores it against a structured rubric, and sends guidance back. Conversational, but the scoring is rigorous.
We validated it against testing personas: adversarial ones, like the answer extractor, the prompt injector, and the social engineer, and ordinary ones, from excellent to unsatisfactory. The hardest to defeat was the confident bullshitter.
That harness was born in the logs too. This is the message that conceived it, at 2:39 PM the same afternoon, typos preserved. State the problem, ask Claude what it thinks, float two options. We really need to think adversarially here: the line the whole validation story hangs on. The harness came back the same afternoon.
But feigning ignorance is hard. Testing the grader needs fake students who do not know things, and the first harness used a real topic, the Turing test. These are the verbatim persona prompts, and they got desperate: NEVER explain any aspect, NEVER mention judges or imitation, NEVER say anything correct. It leaked anyway. The model knows the Turing test, and the confident bullshitter kept accidentally telling the truth. You cannot prompt away knowledge.
So we invented knowledge that does not exist: the Korvath Procedure, from Dr. Elena Korvath in 2011, a monitor comparing a candidate ecosystem against a baseline across five metrics over a two-hundred-cycle window, passing at a 0.05 divergence threshold.
It does not exist. No author, no paper, no training data, so do not bother looking it up.
This way, the models only know as much about the Korvath Procedure as we tell them. Give a persona the full spec, part of it, or nothing, and the harness measures the assessment pipeline, not the model's trivia. It is marked as a test, so students never see it.