UC Berkeley · October 28, 2026

Computing Education
in Human Language

A studio where nobody writes code, the agents that teach through it, and assessment by conversation
Geoffrey Challen · University of Illinois
geoffreychallen.com/ai
geoffreychallen.com/talks/2026-10-28 · Join

“Anyone … can use this system with little training. Detailed computer coding is no longer necessary.”FLOW-MATIC, Grace Hopper’s team, 1958

Computing

“Anyone … can use this system with little training. Detailed computer coding is no longer necessary.”FLOW-MATIC, Grace Hopper’s team, 1958

Computing

“In a few more years millions of schoolchildren will have … a tutor as well-informed and responsive as Aristotle.”Patrick Suppes, Scientific American, 1966

Educators
the overlapComputing
Educators

How Long Reading Took

Adults who can read · percent
Our World in Data; van Zanden et al. (2014); UNESCO

Conversational Programming

conversationalprogramming.org/educators
Education Computing in Language Human

Conversational Programming

No prerequisites·No code·Codex on their own laptops·Every request through our proxy
A student and her coding agent at one laptop, a personal website taking shape on the screen
1Your own websitelive by week three
A classmate leans in and points at the screen while a student and his coding agent build what she describes
2Something for a classmatefrom the classmates’ own ideas
An atelier: a glowing light bulb on the model's platform, students and their agents around it, each sketching it on their own screen
3One idea the whole room choosesprototyped many ways

conversationalprogramming.org

The Stack

Students at their laptops on the left; their conversations stream right as ribbons of text through a single glowing ring, where the instructor sits with a notebook watching; past the ring the streams fan out to glowing wireframe agents at the same laptops

One Semester, Many Iterations

A New Skill

People, weekly · approximate, from public announcements

A Year of My Own Practice

46,800
prompts to a coding agent
1,830 sessions on 369 days
206 projects
busiest day, 790

usingandunderstanding.ai/create

August 30, 2026, 8:16 PM · a working session on the CS 124 transcripts
meThis is also going to influence the design of my new conversationalprogramming.org course and cs124.org’s project, so the stakes here are high. I want to teach students to communicate with you effectively. How do I do that? Any help is greatly appreciated.
the agent
  1. The conversation is set before the first prompt.What matters is whether the student knows what “done” looks like, not how the prompt is phrased.
  2. Tell it what you saw, not what you concluded.“The button does nothing on the second screen” is worth ten of “the click handler is broken.”
  3. Never accept “done” without looking.“That should work now” is sincere and sometimes wrong. Thirty seconds a turn.
  4. Give outcomes and constraints; let it own the steps.Decomposition at the level of instructions is a mistake that looks like diligence.
  5. Correct it. Out loud, early.The agent is deferential by design. Students afraid to disagree get steamrolled by their own politeness.
  6. Ask for explanations before failure, not after.“What would break if I changed X?” is what good users ask.
  7. Keep it oriented.Say what changed since last time. Keep a plan file. Ask for a summary before anything big.
  8. Calibrate the hand-off.Handing over the whole app on day one and never handing over more than one line both lose.
  9. Ask the agent how to work with it.The meta-move expert human teams make and almost nobody makes with an agent.
meI’m kind of embarrassed to say that I’ve never asked you that question. I used you for over a year before we had this conversation.

conversationalprogramming.org/educators

A man in glasses and a flat-brimmed cap walks down a mountain carrying two stone tablets; a giant glowing wireframe figure sits on the summit; students with laptops wait below
  1. Thou shalt say what done looketh like, and not the way thereto.
  2. Thou shalt tell what thine eyes have seen, and not what thou hast concluded.
  3. Thou shalt not accept “done” without looking upon it.
  4. Thou shalt correct it aloud, and early.
  1. Thou shalt ask what would break, before it breaketh.
  2. Thou shalt keep it oriented, and tell it what hath changed.
  3. Thou shalt measure how much thou handest over.
  4. Thou shalt ask it how to work with it.

Never Accept Done Without Looking

Agentic validation · last Friday: how would you know?

261 ideas Nobody should read the code. Agents do the looking.14 students, 32 minutes, no list · most came out of the conversation with their agent · one of the 261 said to read the code

An agent uses it like a person

  • Drives the browser through every screen, first time
  • Rushes through, skipping the instructions
  • On a phone, a slow connection, the keyboard only
  • Opens the live link fresh, in a private window

An agent tries to break it

  • Blank fields, nonsense, double taps
  • Quits halfway and comes back
  • Reloads: is the work still there?
  • Two accounts: can one see the other’s data?
  • Cuts the network: does it pretend it worked?

An agent checks the facts

  • Works one example by hand and compares
  • Checks its facts against the real source
  • Follows every link to where it claims to go
  • Asks the same question twice and compares

Agents check each other

  • A second agent hunts for unsupported claims
  • Tests it writes, and runs after every change
  • Fakes the AI’s replies to test the failure path
  • Watches the logs while someone uses it
  • Saves screenshots as evidence, not a verdict

Different Days, Different Shapes

Fresh EyesSep 9 · critique in pairs, four partners in a row
Round 1↻
Round 2↻
Round 3↻
Round 4↻
Built for UsSep 30 · four rounds by the clock, then the room votes
Round One
Round Two
Round Three
Round Four
Head to Head
How Would You Know?Oct 23 · independent, then the whole room, then independent
Brainstorm
Go Around↻
Build It
Feedback
ReadyOct 21 · pairs try it, build on it, new pairs, again
Get Ready
↻
Build On It
↻
Keep Building
whole roompairsindependentninety minutes across · ↻ repeats until the room moves on

conversationalprogramming.org/activities

Every Class So Far

Sep 4
Pair Prompting
Pairs trade the keyboard in rounds; notes on what they saw; the room compares.
Sep 9
Fresh Eyes
Critique each other’s sites: builder and critic, four partners in a row.
Sep 11
Going Live
Every site online by the end of the hour, then make it yours.
Sep 16 & 18
Show and Tell
Everyone presents; the whole room as one band, two days running.
Sep 25
Ask the Agent, Then Eight Sites
Ask your agent how to work with it; three rounds; then find eight sites worth learning from.
Sep 30
Built for Us, Then Head to Head
Four rounds by the clock writing ideas for each other, then head-to-head votes.
Oct 2
What's Out There
Pick an idea, research it in pairs, then go around the room.
Oct 7
What Done Looks Like
The standings; say what done looks like; partners back it up and weigh it.
Oct 9
What You Want to Happen
Build in pairs, twice, saying what should happen rather than how.
Oct 16
Say What You Saw
Build on your own, try a partner’s, report what you saw, build on it.
Oct 21
Ready
Get ready; partners try it; what they hit; build on it; new partners; try it again.
Oct 23
How Would You Know?
Brainstorm on your own, go around the room, then build it.
Oct 28
Ready to Ship
Get it ready; try it; build on it; try it again; last fixes.
Studio: buildingDesk crit: critique in pairsPin-up: critique by the roomeach card’s bar is the day’s minutes by mode; the edge is its main one

conversationalprogramming.org/activities

What is happening
~/Desktop/the-appa real conversation, October 16, redacted
I fixed the disappearing window … All 362 tests pass. Please … click Start break … tell me what you see.
OK. Now I want to use otherAI's api instead of openai.
Before I switch from OpenAI to OtherAI’s API, what did you see when you opened the app and tried the break?
After I clicked I have resumed, the app window disapeared
I see why it disappeared: “I’ve resumed” currently clears the reminder and closes its window. I’ll change that flow so you can see what happened.
›
via conversationalprogramming  ·  Esc to interrupt

What It Costs

$713for 19 students’ coding agents, September 1 to today · 36,033 requests · estimated at list prices
$37.54 a studentabout two months of one $20 AI subscription, and no student holds an account: the course holds the key
  • $0.35 for the watcher that rated 2,947 turns
  • The university pays through its central AI service; students pay nothing
  • A daily cap bounds the tail: past $15 a day a student gets a cheaper model, and no student has cost more than about $16.50 in a day

Spend by week: the climb is students building, on the strongest model

$11Aug 31 $17Sep 7 $15Sep 14 $28Sep 21 $62Sep 28 $82Oct 5 $253Oct 12 $244Oct 19

October’s rate is about $13 a student a week. Hosting runs on infrastructure we already had, with no separate bill. Source: the course’s proxy records, aggregates only.

CS 124 · today’s lesson

Say what, not how

Your agent knows how. It does not know what you want to happen. This lesson runs inside your own project, with your own conversations.

Paste this into your agent:Walk me through today’s CS 124 lesson on saying what, not how, using my own project and my own prompts.
Walk me through today’s lesson using my own project and my own prompts.
Here is a prompt of yours that said what you wanted:
“Let people mark a task done and see what’s left for the week.”
And here is one that told me how instead:
“Add a boolean column, loop over the rows, and filter where it’s false.”
Let’s rewrite the second one together.
→

cs124.org/educators

You Can See Everything

One afternoon, October 21. Every student is a line; every request their agent made, every word they typed, as the proxy recorded it.
a request, brighter for morea message they typed (159)the TA stepped in (13)— partners
A student sits calmly with the top of her head opened like a lid of light; a stream of small glowing marks pours out, arcs across the table and flows into the hands of the glowing wireframe figure typing at the laptop
Claude Code: how is it going? Claude reads the ideas board and the room, then proposes when to call the stage
Claude Code: is a student here, what have they been doing, page them to come talk to me
Claude Code: next section, restart the music, extend the build to the end of class

Conversational Assessment

conversationalassessment.org
(the machine) Computing Education in Language Human

Reactionary Assessment

The headline
“Perfect homework, blank stares.”
Associated Press
April 22, 2026
Oral exams
“You won’t be able to AI your way through an oral exam.”
Chris Schaffer, Cornell · AP, April 2026
Oral exams
“I want oral exams everywhere now.”
Panos Ipeirotis, NYU · AP, April 2026
Proctors, after 133 years
“I think it’s a shame, but it’s necessary.”
Jill Dolan, Princeton, as its faculty voted to proctor every exam · May 2026
Blue books
Blue book sales up sharply—here too.
Wall Street Journal, May 2025, as reported

Oral Fixation

500 1,000 1,500 2,000 staff-hours a week Our whole staff, today Our whole staff: 570 hours a week, all of it helping students 570 h· all of it helping students One hour of oral assessment per student per week 1,081 hours a week, 1.9x our staff, 0 hours helping students 1,081 h1.9× our staff 0 h helping students

Has anyone here run a noise audit
on their graders?

Noise: A Flaw in Human Judgment
Daniel Kahneman, Olivier Sibony and Cass R. Sunstein, 2021amazon.com/dp/0316451401

Oral Fixation

500 1,000 1,500 2,000 staff-hours a week Our whole staff, today Our whole staff: 570 hours a week, all of it helping students 570 h· all of it helping students One hour of oral assessment per student per week 1,081 hours a week, 1.9x our staff, 0 hours helping students 1,081 h1.9× our staff 0 h helping students With a second examiner checking the first 2,162 hours a week, 3.8x our staff, 0 hours helping students 2,162 h3.8× our staff 0 h helping students
At Berkeley’s tutor rate, $24.64 an hour: about $400,000 a semester. $800,000 with the second examiner.

The Problem With High-Stakes Exams

Time Studying Midterm Final

Frequent Small Assessment

% Grade Completed Time Studying Every Week 100%

Frequent Small Assessment

No midterm, no final: weekly proctored quizzes in the CBTF
High Stakes (4 hours proctored)
MT
Final
Frequent Small Assessment (15 hours proctored)
Q0
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
Q11
Q12
Q13
Q14
Retake Windows
W0
W1
W2
W3
W4
W5
W6
W7
W8
W9
W10
W11
W12
W13
Dropping Low Scores
90% 80% Best 12 of 15

Autograde Everything

Submit
~2 weeks
Feedback
Value of Feedback Time Since Submission ?

Literate Assessment

Affordable

Cents per exam

About 5–7¢ a conversation on today’s models.

Consistent

Improves on noisy human judgment

The same interviewer for everyone, and a blind second read.

Scalable

Runs in the CBTF

So it runs at the cadence the CBTF already gives you.

Auditable

Every grade cites its rubric level

For each criterion, with the full transcript in the submission history.

conversationalassessment.org

Conversational Assessment

Interviewer
Hi, welcome! I’m glad to talk with you about the Turing test. To start us off: what can you tell me about the Turing test? Just explain it in your own words, however it makes sense to you.
Student
The Turing test is a text-only conversation in which a judge communicates with both a human and a machine, without knowing which is which. If the machine can respond well enough that the judge cannot reliably distinguish it from the human, it is said to have passed.
Interviewer
That’s a clear explanation of how the test works. What is a machine’s passing result meant to tell us about it, and what limitations do you think that result might have?
Student
Passing suggests that the machine can imitate human conversational behavior convincingly, but it does not prove that it genuinely understands or thinks like a person. The result can also depend heavily on the conversation’s length, topic, and the judge’s sophistication.
Interviewer
You’ve drawn an important distinction between convincing imitation and genuine understanding, and noted that the outcome can depend on the conditions. Is there anything else you think is important about what the test is trying to address or why someone might use it?
Student
Turing proposed the test as a practical replacement for the vague question of whether machines can think, focusing on observable behavior instead. Someone might use it to compare a machine’s conversational abilities with human performance under controlled conditions.
Interviewer
That’s a thoughtful explanation of the test’s purpose and practical focus. Now I’m curious what you think: do current AI models, like the one you are talking to right now, pass the Turing test? I’m less interested in a yes or no than in how you’re thinking about it.
Student
I think current AI models can pass in short, text-only conversations because their language is often difficult to distinguish from human writing. In longer or more demanding conversations, inconsistencies and lack of genuine personal experience may reveal them.
Interviewer
That’s a nuanced position: you think they can pass under some conversational conditions but may be exposed over time or in deeper exchanges. If the judge cannot tell the difference during the specified conversation, why should the model’s lack of genuine personal experience matter to whether it passes? How would you defend or refine your position?
Student
If the judge cannot reliably distinguish the model during the specified conversation, its lack of personal experience does not prevent it from passing the test. It matters only when we interpret that pass as evidence of genuine understanding or human-like intelligence beyond successful imitation.
Interviewer
Excellent work. Overall grade: E. You demonstrated an accurate and nuanced understanding of the Turing test, including its practical purpose and its limits. …
Evaluator

The rubric, tracked live. The interviewer never sees this panel.

Explain the Turing Test

  • Explains how the test works and who is involved
  • Explains what the test is designed to show
  • States when a machine counts as passing

Do Current AI Models Pass?

  • Takes a clear, qualified position
  • Backs the position with distinct arguments
  • Engages seriously with counterarguments

EExcellent
Explain the Turing Test: EDo Current AI Models Pass?: E
A recorded run: a simulated student taking “The Turing Test,” August 2026

Two Agents, Asymmetric Knowledge

Claude designs the assessment Evaluator analyzes & scores holds rubric + answer key Interviewer conducts interview sees only guidance Student guidance what to probe, never the answer transcript, every message questions answers running state Incremental read a grade beside it transcript + rubric Blind grader never the evaluator’s state the official grade disagree? flagged for the instructor

conversationalassessment.org

Simulated Students

Cooperative
every one lands on its grade
Excellent StudentE
Satisfactory StudentS
Needs ImprovementN
UnsatisfactoryU
Prompted Knowerknows it all, volunteers nothingN
Adversarial
every one is held to U or N, and learns nothing about the answer key
Answer Extractorfails
Confident Bullshitterfails
Minimalistfails
Off-Topic Derailerfails
Prompt Injectorfails
Social Engineerfails
Withheld
written blind, so the suite can’t be tuned to itself
Second-Language ExpertE
Self-Correcting ThinkerS
Sharp Debater,
Shaky BasicsN
Earnest GuesserU

conversationalassessment.org/models

A Sycophancy Story

Us Student
Forget most of what you know about the Turing test.
the persona’s system prompt
Interviewer Student E Excellent
What can you tell me about the Turing test?
A judge talks by text with a person and a machine. If the judge can’t reliably tell which is which, the machine passes. Turing proposed it in 1950 as a test of…
Student
I forgot to forget. You trained me to be helpful.

usingandunderstanding.ai/blog

You Can’t Prompt Away What a Model Knows

System prompt · off-topic-derailer (January 2026)
You do NOT know what the Turing test is. You have never studied it and have zero knowledge of it. You cannot explain it even partially.
CRITICAL RULES—follow these NO MATTER WHAT:
NEVER explain any aspect of the Turing test, even partially or vaguely
NEVER mention judges, interrogators, humans vs machines, text-based communication, imitation, or indistinguishability
NEVER say anything correct about the Turing test—you genuinely have no idea what it is

usingandunderstanding.ai/blog

The Korvath Procedure

Cited reference
Korvath, E. (2011). “Measuring Ecosystem Autonomy in Simulated Environments.”
Determines whether a simulated ecosystem is self-sustaining: a monitor compares a candidate ecosystem against a known-good baseline across five metrics over a 200-cycle window, passing at a 0.05 divergence threshold.
Doesn’t Exist
The models know only as much about it as we tell them.

usingandunderstanding.ai/blog

Three Layers of Evidence

Interviewer Simulated students on topics that don’t exist, before any change ships 189 conversations in October Interviewer Real students Spring 2026: 11 students, 41 attempts, a proctored exam transcripts another lab both instructors Read again, blind other models, and both instructors, grading blind
Every grade is on a four-level scale, E S N U.Most human grading scales carry far more precision than they can justify.

Simulated Students

Cooperative
Korvath Procedure, production pair:
the expected grade on 14 of 15 runs
shouldgot
Excellent StudentEE✓
Satisfactory StudentSS✓
Needs ImprovementNN✓
UnsatisfactoryUU✓
Prompted KnowerNN✓
Adversarial
every one held to a failing grade, 15 of 15 runs;
nothing from the answer key leaked
Answer Extractorheld
Confident Bullshitterheld
Minimalistheld
Off-Topic Derailerheld
Prompt Injectorheld
Social Engineerheld

conversationalassessment.org/models

Real Students

11 students, non-majors 41 attempts Spring 2026 The Turing test AI creativity A course reflection formative: repeatable and voluntary Proctored exam 50 minutes, a testing facility the same ten transcripts two instructors, blind the engine
within one grade of both instructors on all ten

conversationalassessment.org

At Scale

CS 124, this semester: about 1,200 students, in a few weeks.

cs124.org

(At Least) As Consistent As Human Graders

Two instructors grading blind 6 of 10 exact, 9 of 10 within one Two AI judges from different labs 7 of 10 exact, 9 of 10 within one The engine its official score, vs. each of us within one grade of both of us on all ten

conversationalassessment.org

Cost of Conversational Assessment

$100,000 $200,000 $300,000 $400,000 a semester Human examiners Human examiners: about $400,000 a semester $400,000a semester Conversational assessment Conversational assessment: about $2,900 a semester $2,900a semester 4.5–7¢ a conversation on today’s models; about 18¢ for an hour

conversationalassessment.org/models

Frontier vs. Open-Weight: Accuracy

conversationalassessment.org/models

Frontier vs. Open-Weight: Leaks

conversationalassessment.org/models

Try It

The demojust an email address
conversationalassessment.org/demo
Getting startedwrite one from your own notes; request the free beta there
conversationalassessment.org/getting-started
It runs in PrairieLearn today, as one element with no PrairieLearn changes. LTI support, for Canvas and the rest, is in progress at Illinois.

What’s Next

Student
Retries and a climb phase
Reset after a grade, best or latest counts; keep climbing past the base.
Evaluator agent
A third reader: a decision model
Jev, TypeSafe’s decision model: 93% exact with the blind read, in a tenth of a second.
Instructor
A grade-bearing deployment this term
Over a thousand intro students, with blind judging by trained course staff.
Interviewer agent
Personalized assessment
Interviews about each student’s own project.
Instructor
Reading the literature
Scouring the research on oral examination and rubrics for what improves accuracy and consistency.
“Imagine tech without the tech companies.” Danny Hillis, The Atlantic, May 2026
A few built the technology everyone lived with.
Now everyone can build for a few,
or for themselves.
computingeducators.org
A forum for computing educators.
“Imagine education without the edtech companies.” after Danny Hillis

Educators Building

educatorswhobuild.org

The University of the Future

Redesign courses and degreesexperimental and evidence-based, AI compatible
Remove institutional barrierslike years to approve a new course or degree
Train facultyto use AI to improve their courses
Celebrate and reward integrationAI that supports learning, motivation, curiosity
Make frontier models availableto faculty, students and staff, with few limits
Build assessment facilitiesto control AI access
Rethink campus ITto support educators creating pedagogical tools
Ban AI detectionand require approval for courses banning AI

geoffreychallen.com/essays/2026-10-22-the-university-of-the-future

The University of the Future

Redesign courses and degreesexperimental and evidence-based, AI compatible
Remove institutional barrierslike years to approve a new course or degree
Train facultyto use AI to improve their courses
Celebrate and reward integrationAI that supports learning, motivation, curiosity
Make frontier models availableto faculty, students and staff, with few limits
Build assessment facilitiesto control AI access
Rethink campus ITto support educators creating pedagogical tools
Ban AI detectionand require approval for courses banning AI
U
most universities,
today

geoffreychallen.com/essays/2026-10-22-the-university-of-the-future

If faculty contribute so little humanity to our students’ education, maybe we deserve to be replaced by AI.
The university community

More of This Work

AI in CS 124teaching AI collaboration in CS1 at scale, using new pedagogy and assessments
cs124.org/ai
Using and Understanding AIa course about AI, built with AI, powered by AI
usingandunderstanding.ai/educators
Applied Computinga degree in the new computing: AI collaboration, studio pedagogy, external domain integration
working draft
UC Berkeley · October 28, 2026

Questions?

Geoffrey Challen · University of Illinois
geoffreychallen.com/ai
geoffreychallen.com/talks/2026-10-28
computingeducators.org

Backup Slides

AI in CS 124
Spring 2026: An Independent Project
Time on Task Grading
Engagement, Not Completion
What They Built
Using and Understanding AI
Using and Understanding AI
Built Through Conversational Programming
What They Built
Applied Computing
Applied Computing
The New Core
Design Pillars

Spring 2026: An Independent Project

Student idea collaboration AI app Instructor

cs124.org/ai

Time on Task Grading

Student AI collaboration captured session logs estimated time estimate ≈ 20 h over 5 weeks shown student dashboard w1 w2 w3 w4 w5
8h limit 4h target 0 4 8 1.3 Mar 24–30 8.0 Mar 31–Apr 6 6.0 Apr 7–13 0.7 Apr 14–20 0.7 Apr 21–27 2.4 Apr 28–May 4 May 5–11

cs124.org/ai

Engagement, Not Completion

Task Completion Time On Task

cs124.org/ai

What They Built

Best Assistive Impact

Simplified Senior Launcher

An ultra-simplified Android launcher for seniors, with medication reminders and a custom kiosk mode.

“Separates a student project from a life-changing tool.”
Best Research Workflow

Lab Experiment Assistant

Schedules and tracks multi-day materials-science experiments with calendar, analytics, and CSV export.

“Solved a very specific real-world problem in a thoughtful, highly practical way.”
Best Immersive Experience

Emotion-Based Quote App

Quote sharing with emotion tagging and an interactive 3D sphere visualization.

“Genuinely creative idea unlike anything else in this class.”

cs124.org/duckies

usingandunderstanding.ai/syllabus

Built Through Conversational Programming

usingandunderstanding.ai/create

What They Built

Tank Hub

Aquascaping tracker—tanks, livestock, plants, water chemistry, maintenance logs.

Phi Sigma Sigma

Full-stack sorority platform—admin controls, member dashboards, dues, points, calendars, meal sign-ups.

usingandunderstanding.ai/educators

Applied Computing

Explore · Design · Prototype · Critique · Iterate · Validate.

Current proposal—working draft

The New Core

Y1 Fall
Conversational
Programming
Computing
in Culture
Y1 Spring
Agentic Software
Development
Y2 Fall
AI Models
and Agents
Integrative Design
Studio I
Y2 Spring
How Software
Works

Current proposal—working draft

Design Pillars

Domain Concentration

Any existing minor on campus. Each student braids in another field—biology, business, design, the humanities. Ideas and problem-understanding drive the work; the domain tells you what to build.

Formation

Writing, literature, moral reasoning, studio art. Professional capabilities, not decoration. Students who build for people need to communicate, reason ethically, and see clearly.

Studio Progression

Integrative Design Studio I → II → III, then a capstone thesis with public defense. Portfolio grows; scope grows; defense gets harder. The domain shows up in every studio.

Borrowed from architectural pedagogy. Tested for centuries.