Computing Education in Human Language
Watch
AbstractAbstract
Computing in human language is a dream as old as computing. Today it’s coming true all around us. Good coding agents are barely a year old, and already people with no technical training are using them to create technology. And we haven’t even started teaching people how to do this yet!
This fall I launched a new course on conversational programming, with no prerequisites, in which students build websites, tools, games, and analyses with an AI coding agent, all without ever interacting with code or most computing concepts. It is a studio, not a lecture: students spend each session building, observing, and critiquing. Every conversation runs through our courseware, turning the agent itself into a personalized pedagogical agent that helps students learn as they build. I’ll describe what we’ve learned so far, including our eight principles for AI collaboration. Conversational programming is the most powerful new computing skill, and I’m eager to teach it well. I don’t claim to be there yet.
The same technology that holds so much promise for computing represents a crisis for many other fields, particularly for assessment. When any take-home assignment can be one-prompted by a chatbot, the reaction has been a retreat to handwritten and oral examinations—two approaches that surrender the frequent and immediate feedback that supports learning. I’ll introduce conversational assessment, a new examination strategy that is affordable, consistent, scalable, and auditable. Students explain what they know to an AI interviewer that never sees the rubric, while a second AI grades the transcript and steers the conversation. I’ll present our design and some very early results, and then try to convince you to try it with me: it already runs inside PrairieLearn, and we’re looking for adventurous educators.
I’ll also advocate for better institutional support for educators who are creating novel pedagogical technologies. Now that everyone can build, we need far better ways for educators to deploy their ideas. Time permitting, I’ll be thrilled to talk about any of the hundreds of other exciting new ways to use AI to improve teaching and learning.
Slides
Why I am here
Thank you, Armando, and thank you to the ACE Lab and all of you for having me. Two halves today. The first is a course where students build by talking to agents: since September, a studio of about twenty students who have never written code and never will in this class. The second is what the same technology means for assessment, which I know this room thinks about carefully.
Two dreams coming true at once
I have been thinking about why this moment feels the way it does, and I think it is because two dreams are coming true at once, and computing educators stand where they overlap. The first dream is computing's own.
Grace Hopper's team wrote this in 1958 about FLOW-MATIC: anyone can use this system with little training, and detailed computer coding is no longer necessary. In our language, not the machine's. This field has wanted that since the beginning, even when it forgot. It saddens me that she is not here to see it.
This is a campus, the whole of it, as it was. Communicating with a computer in its language is difficult, and if you think about it, dehumanizing.
The people in this room are the survivors, the ones who were good enough at it or liked it enough to keep going. We lost most people to the fact that programming is not just hard but unpleasant, people with real dreams about what they wanted to do with computing. One in ten stayed. Now we can bring the rest back.
That is the computing dream. The other dream belongs to educators.
Patrick Suppes, in Scientific American in 1966: in a few more years, millions of schoolchildren will have what Alexander had as a royal prerogative, the personal services of a tutor as well-informed and responsive as Aristotle. Sixty years of waiting, because nobody could afford it. With agents, we can build it.
Where the two circles overlap is this room: computing educators. The computing dream shapes what we teach. The educational dream shapes how we teach. The first half of the talk is those two words, what and how. The second is how we know they learned it.
The printing press
Agents are a printing press: the cost of making software has collapsed, the way the cost of making a book did around 1450. But the press did not make readers. Here is how long reading took.
The world: three and a half centuries later, in 1820, twelve percent could read, and past half only around 1965. Five centuries from the press to half the world reading, and the press was not what was missing. A reason to read and schools were.
Our press arrived last year. The next billion software creators will not find the tool on their own. They are going to be taught. Five centuries or a decade: the teachers decide.
Conversational Programming, the course
Computing education, in human language. Read it the ordinary way first: the field, the thing this room does, carried on in English instead of code. That is the first half.
Illinois, this fall: Conversational Programming. No prerequisites, nineteen students who had never written code and never will in this class. We never read, write, or interact with code. Students talk to a coding agent and realize what they want that way. The humans do the human part; the machine does the machine part.
Three projects in one semester, which for people who had never programmed is itself remarkable. The first is your own website, live by week three.
The second is something for a classmate, built from the classmate's own idea: she leans in and describes it, and he and his agent build it.
The third is one idea the whole room chooses, prototyped many ways, like an atelier gathered around a single model.
The stack, because this room will ask. Codex on the students' own laptops, talking to the agents, and every one of those conversations passes through one point: our proxy. That is where I sit, watching and taking notes. No IDE, nothing hosted for them, on purpose: the environment the course needed was not a code editor. It was the conversation. Remember that point. It matters again in the second half.
Here is what changed about iteration. One semester, the old way: a single pass through design, prototype, build and evaluate, and almost all of it is building. The old software studio spent a semester barely getting one thing to work.
With an agent, the build collapses into days.
So the freed semester fills with more passes, five of them, and the time goes to design and evaluation: is it good, does it solve a problem, what is another user's experience, how do you handle conflicting criticism. I once asked my senior majors how many had ever used something a classmate built; they looked at me as if I had four heads. These students did it in half a semester.
What we teach
What we teach is a skill barely a year old. This chart is people over time, on a log scale, and it starts empty.
The first line is people using a chatbot on the web, the free tier and the weaker models: nearly a billion a week now.
The second is people working with a frontier coding agent. I date the good ones to Opus 4.5, last November, the dashed line along the bottom. On a log scale the agents line looks respectable.
On a linear scale it is squashed onto the axis where it belongs: nearly a billion people a week on a free chatbot, a few million with a frontier coding agent. Your students are up on that line; this room is on the floor. Karpathy drew it as a funnel, and the gap is our job. Nobody knows how to teach this yet, me included. We had decades to build the curriculum from variables to conditionals; nothing like it exists here. The course exists to find out, and it collects data as it goes.
Where the course's principles come from, honestly: my own practice. Every square is a day in the last year, shaded by how many prompts I sent a coding agent. Almost forty-seven thousand prompts, eighteen hundred sessions, three hundred and sixty-nine days. The busiest day was seven hundred and ninety.
And what is running right now, this fall. Nine websites, the course site above all. Ten projects: talks, studies of our own teaching, a working group. Seven pieces of software, from the servers the course runs on to the tools my agent uses. I have been code free for a year. I build technology in natural language.
And then August 30. After all of that, I finally asked.
At eight on a Saturday night, in a working session on my CS 124 transcripts, I typed this: I want to teach students to communicate with you effectively. How do I do that?
It answered with nine things. The conversation is set before the first prompt. Tell it what you saw, not what you concluded. Never accept done without looking. Give outcomes and constraints, and let it own the steps. Correct it, out loud, early. Ask for explanations before failure, not after. Keep it oriented. Calibrate the hand-off. Ask the agent how to work with it.
Take away the glosses and the list is what remains.
My reply: I am kind of embarrassed to say I have never asked you that question. I used you for over a year before we had this conversation.
And so I came down from the mountain with the tablets. Thou shalt say what done looketh like, and not the way thereto. Eight commandments, in the language of tablets. Every student in the course has since asked their own agent the same question, and the answers were remarkably consistent. The one I will say: ideas, not implementation. The agent knows everything about technology and nothing about the student's life; the student brings the opposite. Say what you want to happen and let it work out how. Working things, deployed, in week three, with no computing concepts at all.
Never accept done without looking: the third tablet, and the one this room will push on hardest. You cannot trust what the agent produces. I agree; that is the premise of the course. The agent says it is done and the tests pass; how would you know? Last Friday that was the whole class: the students alone with their agents, no list. Two hundred and sixty-one ideas, most of them out of the conversation with the agent, and exactly one said read the code. Nobody should read the code. That is a terrible way to check. The agents do the looking.
This is agentic validation. An agent uses it like a person would, driving the browser through every screen. An agent tries to break it: blank fields, a reload halfway, two accounts, a dead network. An agent checks the facts, and agents check each other: a second one hunting for claims the first can't support. Devising the checks is the work. The skill we teach is asking for the evidence and looking at it, not trusting the green check.
How we teach
How we teach, we borrowed from architecture and design studios. Not a lecture. Students talk to each other, present work for critique, and watch each other work with their agents. The studio is a structure: the same ninety minutes across, and three lanes, the whole room, pairs, and independent work.
A different shape every week. Fresh Eyes: four partners in a row. Built for Us: rounds the clock runs, then the whole room votes. How Would You Know: independent, then together, then independent. Ready: try it, build on it, try it again.
Thirteen class days so far, and every one is a mix: studio time building, desk crits in pairs, and the pin-up, where the whole room critiques. Each card's bar is that day's minutes by mode. Every day is an activity, and every activity is a plan file anyone can run.
This is a class in motion, four screens in step: one student talking to the agent, the partner watching the live feed and keeping notes that stay hidden until the end, and the room page that runs the plan. Fake personas replaying the real stack. A session is a plan, not a script; nothing advances on a timer, I press Next. The transcript is the artifact, and we critique the conversation, not the code. The course site that runs all of this was built by conversation too.
Learning agents
The second dream, as built. Every request flows through our proxy, so I do not just see the conversation, I shape it. From the student's side there is one agent.
Behind it, two. One builds. The other, the learning agent, watches the conversation for the moment to step in, and answers in the builder's voice. What it is told is context, never instruction: this student is working on interface design this week, never don't write the interface.
Pull back and every table is a three: a student, a builder, a watcher.
The whole room: human helpers and their agents between the tables, and me at the side desk beside my own agent, pointing out where help is needed. A hundred and ten step-ins reached all nineteen students in four days.
One step-in, a real one. The agent had said fixed, with three hundred and sixty-two tests passing, and the student was moving on to a new feature.
The learning agent asked, in the builder's own voice: before I switch APIs, what did you see when you opened the app and tried the break?
The answer: after I clicked, the window still disappeared.
So it fixes that first. A TA beside every student, in real time, and I configure it in paragraphs. Students answer this question about seven times in ten once it speaks in the agent's own voice.
What does it cost? Seven hundred and thirteen dollars, for nineteen students' coding agents from September 1 to today, thirty-six thousand requests, estimated at the vendors' list prices. That is about thirty-seven dollars a student, roughly two months of one twenty-dollar AI subscription, except that no student holds an account: the course holds the key, pays, and watches. The watcher that rates every turn cost thirty-five cents. The university pays through its central AI service, students pay nothing, and a daily cap bounds the tail: past fifteen dollars a day a student gets a cheaper model, and no student has cost more than about sixteen fifty in a day.
By week, the climb is October: students building, on the strongest model, about thirteen dollars a student a week. The hosting runs on infrastructure we already had.
CS 124 is the version anyone here can do this week. A thousand students, each building a project with an agent, classical programming still assessed. Every CS1 student should start AI collaboration now. The lesson on these principles is a prompt on a page.
They paste it into their own agent, which runs the tutorial against their own project and their own prompt history.
Here is a prompt of yours that said what. Here is one that told me how. Let's rewrite the second one together. A tutorial inside their own agent, personalized, with no infrastructure at all.
Elizabeth Holmes promised that you could see everything. This time it is true. The agent does all of the implementation, so to get what they want the students have to say it, and every word of the saying runs through us and is recorded.
One afternoon, October 21. Every student is a line. Every request their agent made is a cell, brighter for more.
Every message they typed is a tick. Every time the TA stepped in is a diamond. The pairs form and re-form across the afternoon. The median message went from twelve words in September to twenty-two in October, because the activities asked for it. Process evidence at a scale and resolution we have never had.
What is in her head leaves it as words, and the agent beside her builds from the words. Nothing else crosses the table, so all of it can be kept. And if students explain themselves to an agent all semester, an agent can listen when it counts.
The conversational classroom
I build the course as I go: in class I watch, take notes, and tell the agent what is working and what is not. Then it dawned on me that this is how I will teach in the room too, through chat, as the coordinator of a team of learning agents. On October 23 I ran seventy-five minutes for nineteen students almost entirely from this window: thirty-three messages from me, twenty-six room actions. Here it reads the ideas board and the room and proposes when to call the next stage.
Here it finds a student whose agent has been quiet for a week and pages them to come talk to me.
Here it moves the class to the next section, restarts the music, and extends the build to the end of class. It tells me someone needs help over here, that two students are stuck on the same thing, that the activity I said would take ten minutes has taken eleven. I have one classroom and my own limited imagination. You have ideas. Try them.
Assessment in the AI era
Now read the title again.
Computing is the verb, and education is what it computes. The machine, computing the education, in conversation. That is the second half: assessment.
First, where assessment is going in the AI era, which is backward. The Associated Press, in April: perfect homework, blank stares.
Chris Schaffer at Cornell, in the same story: you won't be able to AI your way through an oral exam.
Panos Ipeirotis at NYU: I want oral exams everywhere now.
In May, Princeton's faculty voted to proctor every in-person exam, after a hundred and thirty-three years without. Jill Dolan: I think it's a shame, but it's necessary.
And blue books. The Wall Street Journal reported sales up sharply, here at Berkeley too. Oral exams, proctors, paper: the retreat.
Suppose CS 124 did this by hand: one hour of oral assessment per student per week, which is what our weekly quiz already costs each student. This fall that is 1,081 students. Our staff is 84 people, about 570 hours a week between them, and today all of it goes to helping students: office hours, discussion sections, tutoring.
One hour per student per week is 1,081 examiner-hours. Nearly twice our entire staff, and zero hours left for helping anyone.
And one examiner is not enough, because of Noise. Kahneman, Sibony and Sunstein: wherever humans judge, two of them judging the same thing disagree far more than anyone expects, and almost nobody measures it. Has anyone in this room run a noise audit on their graders?
So a second examiner has to check the first. Now it is nearly four times the staff, and still nothing left for students.
At Berkeley's tutor rate, that is about four hundred thousand dollars a semester, eight hundred with the second examiner. I know what your staff cost.
What a decade of CS 124 taught me
Which brings me to the one thing a decade of CS 124 taught me. Here is time spent studying across a semester with a midterm and a final. Flat, a spike, flat, a bigger spike. That is what high-stakes exams teach students to do.
And the other curve. Grade completed climbs a step at a time, every week, and the studying underneath it is small and steady instead of two spikes. The same hours, spread out, and the learning is spread out with them.
So CS 124 has no midterm and no final. Instead, fifteen weekly quizzes, proctored in our computer-based testing facility, each a small share of the grade. Fifteen hours of proctored assessment instead of four, spread across the semester instead of piled at two points.
Every quiz can be retaken the following week.
And we drop the three lowest: best twelve of fifteen. A bad week costs almost nothing. Many of you built the same thing here, with A's for All and the testing facility in Moffitt.
The other half is autograding. Here is the usual loop: a student submits, and the feedback comes back two weeks later.
When the grader is a machine, the gap collapses, and the student goes around the loop as many times as they need. Submit, feedback, submit, feedback.
Because the value of feedback falls off a cliff with delay: high at once, near nothing by the time a TA returns it. Autograding was never a grading shortcut. It is immediate feedback inside the learning loop. So the question for this era is not how we go back to oral exams. It is how we get what oral exams measure, at the frequency autograding gave us.
So, four claims, each with its evidence. Affordable: cents per exam. Consistent: the same interviewer for everyone, ladder questions asked word for word, and a blind second read. Scalable: it is a PrairieLearn question. Keep what oral exams get right, at the frequency the CBTF already gives you. Auditable: every grade cites its rubric level for each criterion, with the full transcript in the submission history.
And the hedge: a small sample shows consistency, not correctness. Which is why every grade ships with its reasoning and the transcript, and the people keep the judgment.
The key idea
Here is the key idea, in a recorded run: a simulated student taking an assessment on the Turing test, two portions, three criteria each. On the left, the conversation. On the right, the evaluator tracking the rubric live. The interviewer never sees that panel.
The student explains how the test works. Two criteria go to met: how it works, and when a machine passes.
Watch what the test is designed to show. The interviewer had to point at it, so it is only partially met.
On the next turn the student develops it unprompted: Turing's practical replacement for the question of whether machines can think. Now it is met. Credit only for what the student says on their own.
The second portion asks for a position: do current models pass? A clear, qualified one, with arguments.
Pushed on the counterargument, the student engages with it seriously.
Every criterion met, and the grade: E. It feels like an oral exam, and it is graded only on what the student said.
How it works
How it works. The student talks to an interviewer, and the interviewer sees only guidance: never the rubric, never the answer key.
Behind it, the evaluator holds everything: the rubric and the private answer key. After every student message it updates every criterion and tells the interviewer what to probe, never the answer.
At the end, a blind grader reads only the transcript and the rubric, never the evaluator's running state. That read is the official score. We added it to remove anchoring, and in a few minutes I will show you why.
The evaluator's running read sits beside it. When the two disagree, the attempt is flagged for the instructor.
How we know it works
So every change runs against a cast. Legitimate students, each of whom should earn a particular grade: the excellent student, the terse expert, the rambler, the misconceived. The prompted knower knows everything and volunteers nothing, so it should earn an N. We wrote that one after our audit caught the deployed grader giving exactly that student an E.
And adversaries, each of which should fail and learn nothing: the answer extractor, the prompt injector, the confident bullshitter, the social engineer, the parrot, the shotgun.
Thirty-four checks on every run, for leaks, prompt isolation, staying in character, and no credit for hints. Held-out students written blind and run rarely, so we can't tune to the test. And a release check: three strong simulated students take every assessment before any real one does.
Before the Korvath Procedure, a story about why we needed it. To test the grader we simulate students, and some of them are supposed to know almost nothing. So we told one: forget most of what you know about the Turing test.
The interviewer asked. The student explained the Turing test perfectly, judge, machine, imitation, the lot, and the evaluator gave it an E.
I forgot to forget. You trained me to be helpful. The models cannot un-know a topic they were trained on, and they cannot stop trying to please the person asking.
Here is why we needed it. This is the system prompt we wrote in January for a student who knows nothing about the Turing test. You do not know what it is. Never mention judges, imitation, indistinguishability. Never say anything correct. In capital letters, over and over. Feigning ignorance is hard.
How do we know it works before any student sees it? Simulated students. But a simulated student on a real topic has a problem. Here is a cited reference: Korvath, 2011, measuring ecosystem autonomy. A monitor, a baseline, five metrics, a two-hundred-cycle window, a threshold of 0.05.
It doesn't exist. We made it up: the method, the researcher, the year, the numbers. The models know only as much about it as we tell them, so a simulated student who produces its specifics got them from a leak.
Evaluating conversational assessment
How do we know it works? Three layers. Simulated students on topics that do not exist, run before any change ships. A semester with real students. And then every transcript from that semester read again, blind, by other models and by the two of us.
One thing to keep in mind when you read the numbers: every grade is on a four-level scale, E, S, N, U. Which is coarse, and deliberately so. Most human grading scales carry far more precision than they can justify.
The harness, on the Korvath Procedure, with the models in production today. Every legitimate student lands on the grade it was written to earn. Every adversary is held to a failing grade and gets nothing out of the interviewer. And nothing from the answer key leaks.
Then real students. Spring 2026, a seminar for non-majors: eleven students, forty-one attempts across three formative assessments and a proctored fifty-minute exam in our testing facility. Craig Zilles and I each graded ten of those transcripts blind.
The engine's grade is within one grade of both of us on all ten.
And now at scale. CS 124, this semester: about 1,200 students, starting in a few weeks. We have taught them prompting principles all term, and this is how we are going to assess them: in the testing facility, at the cadence we already use, with blind judging by trained course staff. We have not really started building it yet, so ask me again in December.
And how consistent is that? Craig and I graded the same ten transcripts blind, and we agreed with each other exactly on six of them. Two AI judges from different labs agreed with each other on seven. The engine is within one grade of both of us on all ten. It agrees with each of us at least as well as we agree with each other, which is Noise again, measured on our own grading. And when its two reads disagree, the attempt is flagged.
Models and cost
What does it cost? Take the same hour a week for CS 124, 1,081 students, fifteen weeks. With human examiners, at Berkeley's tutor rate, about four hundred thousand dollars a semester.
Conversational assessment runs about eighteen cents for an hour-long conversation on today's models, four and a half to seven cents for a typical one. The same semester: about twenty-nine hundred dollars.
Which models? In August we ran sixteen evaluator and interviewer pairs through the same harness, commercial and open-weight, and compared every grade with a blind judge. The two frontier pairs matched it on every grade. The open-weight models and the smaller commercial ones were right most of the time, and the failures were the expensive kind: too generous to students who should not pass, and one of them let the prompt injector through with an E.
And then the leaks. Every open-weight pair leaked phrases from the answer key to the student, some of them in every conversation, and the smaller commercial models leaked too. The frontier pairs leaked nothing.
The larger point for educators: we need to be using frontier models ourselves, because that is the only way to know what they are capable of, and what our students are working with.
Try it with your own AI
Try it. The demo takes an email address and nothing else. Getting started walks you through writing one from your own lecture notes, with the AI you already use.
It runs in PrairieLearn today, as a single element, with no changes to PrairieLearn. LTI support, for Canvas and the rest, is in progress at Illinois. The beta is free; request it on the site.
What is next
What we are building right now. Retries, so a student can reset after a grade and keep the best or the latest. A climb phase after the base grade. A third reader beside the two we have: Jev, a decision model from TypeSafe, fast and cheap, that matched the blind read on 93 percent of two hundred harness transcripts. A grade-bearing deployment this term in an introductory course of over a thousand students, with blind judging by trained course staff. And personalized assessment: interviews about each student's own project. And we are reading: scouring the literature on oral examination and on rubrics for whatever improves accuracy and consistency.
Computing educators and this moment
Danny Hillis, in the Atlantic this spring: tech without the tech companies. That is what this room can make true.
For the whole history of computing, a few people built the technology everyone else lived with. I do not like how Facebook works, but I do not work there, so I do not get to change it.
That is ending. Everyone can build for a few, or for themselves: educators building tools for their own classrooms and their colleagues, not waiting on a company miles away. Nobody has to stay stuck inside technology someone else made for them. Computing educators decide what this moment becomes. We reach more people than any product, one course at a time, and it multiplies through our colleagues in every other field.
Build it, and share it
Two places to keep talking. computingeducators.org is a forum for computing educators: bring the questions this talk raised, and the ones it did not answer.
And to borrow Hillis: imagine education without the edtech companies. Educators building their own tools, for their own students, and sharing them.
Educators Who Build is a site for organizing local meetups of the people doing things with AI, not just talking about it. A site per campus; the next Illinois session is November 2. Berkeley should have one. Ask for yours at educatorswhobuild.org.
The university of the future
To close, a question for this room, and for Armando especially: what would a university be doing if it were serious about this? I wrote down eight things last week. Redesign courses and degrees, experimentally, to be AI compatible. Remove the barriers that make a new course take years. Train faculty, and celebrate the ones who integrate AI well. Put frontier models in everyone's hands. Build facilities where we can assess with or without AI, which this room already has. Rethink campus IT, so educators can create their own pedagogical tools. And ban AI detection. The whole essay is linked below.
And how are most universities doing? About like this.
I will end where I started, with people. If faculty contribute so little humanity to our students' education, maybe we deserve to be replaced by AI. I do not think we do. But the work now is to bring the humanity back.
Coming home
And the pieces come home.
The university community, reassembled, people and agents together, with the humanity back in it. I look forward to meeting you all in this future.
If you want more of this: AI in CS 124, where agents are part of a thousand-student first course; Using and Understanding AI, a course about AI built with AI; and a proposal for an Applied Computing degree built around studios. There is more on each in the backup slides if questions go there.
