Mind the Gap
16 min read
A reading of this essay using a voice clone.
Andrej Karpathy has been describing a gap.
Not a gap in what AI can do, but in what people understand it can do.
In October he wrote that the gap is widening, into what he called a sharp funnel:
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
I think this gap runs through computing education. And I think I can show where our community sits in it.
AI in a StraitjacketAI in a Straitjacket
Why does the gap exist? Part of it is which model you use. More of it is what you let the model do.
Consider writing code with a web-based chatbot. You paste code in, read the reply, and paste code back out. The chatbot can’t run a test. It also can’t control what it knows. It can’t look around the project for an example, or read the source to answer its own question. Everything it learns, you hand it. Everything it decides, you carry out.
An agent can do things. Sometimes that means taking an action: running a test, examining a deployment. But it also means reading the code and running searches. Acting and gathering information turn out to be the same capability. And the more of it we give these tools, the better positioned we are to see what they’re capable of. Otherwise we’re the ones constraining them.
Picture AI in a straitjacket. Tie it up until all it can do is talk, and it still knows things. But now you’re in the way. You can do everything it asks: run the command, paste back the output, make the edit. You still constrain it, because it sees only what you think to show it. And you slow it down, because every step waits on you.
This is how I’ve come to see what these tools can do: I use agents, and I’ve given them access to a lot. One pattern that is now routine for my agents goes like this. A student reports a bug on the course forum. The agent reads the report. It reads the code, because it works inside all of the code that powers my courses. It runs the code, finds the bug, writes tests, and runs them. Then it deploys the fix, because it has access to my infrastructure. When a bug crosses repositories, it works with the agents in the other ones.
A web-based tool can do none of that. So people who mainly use web-based tools can’t fully comprehend what these tools do once you give them agency. That is the gap that concerns me.
Looking at SIGCSELooking at SIGCSE
The SIGCSE Technical Symposium is the largest venue where computing educators talk to each other. So I looked at what it publishes. Working with a coding agent, I collected every item in its programs from 2015 through 2027: papers, panels, posters, workshops, all of it. The 2027 symposium meets in February, so its items come from the lists of accepted work. That’s 5,516 items. Then I had a language model read each abstract. How I did it, including the prompts, is at the end.
AI is now half of the program.
Through 2022, most of that work took no position on AI. It didn’t need to. AI was something you taught, like operating systems, or a method for analyzing data. Then the tools arrived, and the community had to decide what it thought.
I sorted each AI item by the stance it takes. Pro work builds or advocates an AI tool, or reports mainly positive results. Cautious work sees value but gives real weight to risks and guardrails. Critical work argues against use. Neutral work describes without taking a side.
Through 2026 enthusiasm and caution were roughly balanced. When exactly caution took the lead depends on which model does the reading: two put it in 2026, and a third has 2026 as a tie. In 2027 there is no question: cautious work leads by between two and four to one on every reading, and enthusiastic work fell in number while AI work as a whole grew by half.
Almost nobody says no. Through 2026 no reading finds more than three critical items in any year. In 2027 there are somewhere between four and eleven, which is new, and still a sliver. The position of the conference is to use AI, with guardrails.
But guardrails for what? So I asked a second question of the items about generative AI. How capable does each one take AI to be?
In 2027 the largest group, 118 of 214, treats AI as a helper for bounded tasks: answer a question, explain some code, give a hint, grade a submission. A second model reading the same abstracts put that group at 95, so call it about half. Twenty-seven acknowledge that AI can do the coursework. Most of those are about assessment and integrity: how to catch it, or block it.
And 23 discuss AI that works on its own. The year before there were three, and two of those described the authors’ own pipelines.
But who is using the agent? I asked that as a third question. About seven of the 565 items are about teaching students to build software by directing coding agents. Four are papers: an agentic coding course, a course where students may not write code by hand, a one-week sprint inside a project course, and a degree program that is designed but not yet taught. Three are posters about K–12. None is aimed at university beginners. Four more are about instructors using coding agents on their own courses. The rest teach about agents as a subject, treat them as a threat to assessment, or describe tools the authors built.
Few abstracts say which model they used: 27 of the 214. The most common are small open models like Llama and Qwen. Nine say “ChatGPT” with no version. Five mention an agentic tool or practice, such as Claude Code, Cursor, or vibe coding. Abstracts undercount this, since papers name their models in the methods section. I do worry about studies that use weak models. They can be appropriate in some circumstances, but it’s generally impossible to determine what a frontier model can do by using a weaker model.
SIGCSE Is Only Now Finding the FrontierSIGCSE Is Only Now Finding the Frontier
I’m not a neutral messenger here. I’m bullish about these tools, and I’m concerned that our community isn’t acknowledging what they can do or teaching students to use them well. I’m also in this program. I’m a coauthor of a 2027 tutorial on responding to students’ overuse of these tools, one of the items the models labeled critical. My role in that group is to advocate for frequent small assessment. I’ll be arguing in favor of teaching AI collaboration and against oral examination. Read what follows with that in mind.
Here is my reading. We are becoming more cautious about tools that we still largely understand as chatbots. I suspect the two are linked. If AI is a helper that answers questions, then the main risk is that it answers too many, and the guardrails are obvious. The abstracts can’t prove that link. But they do show a community living somewhere in the middle of Karpathy’s funnel, and closer to the chatbot end than the frontier. The frontier is arriving at SIGCSE, but just barely. About seven items of 565 are about teaching coding agents. A tiny fraction of work covering the most powerful new computing skill, and the one we know the least about how to teach.
Even the work that concedes AI can do our assignments mostly asks how to stop it. That’s the wrong question. This is a new capability. Once these tools can do these things, humans won’t be. So what should students be doing when they collaborate with ever more capable agents? For the first time, a few papers are approaching that question.
It’s true that these tools are somewhat new. I date the arrival of good coding agents to Opus 4.5, about a year ago. SIGCSE 2027 papers were due in summer 2026, so only one full semester after that date. Perhaps we’ll see a flood of new work on agentic education in 2028. One can hope! Clearly we’re racing to catch a fast-moving frontier, and falling behind harms our students.
And many computing educators have yet to arrive at the frontier. About half of our work on generative AI still treats it as a helper for small tasks. It seems very possible to me that much of what we teach large numbers of students is no longer relevant. It is information the agents know well. As people find and use these tools, they won’t need to learn some of the same things anymore. That is a big and important conversation for our community to have, and one I’ll write more about. But to have it intelligently, people need to use and understand frontier coding agents and their capabilities. You can’t do that from web-based chat.
How This Was DoneHow This Was Done
I did this analysis with an AI agent over four days in November 2026. I’ll include the details so you can decide how much to believe.
The data. For 2015 through 2026 the list of items comes from DBLP: every entry in the Technical Symposium proceedings, in every track, with front matter removed. That’s 4,951 items. Abstracts come from OpenAlex and, where it had none, Semantic Scholar. We found abstracts for 4,905 of them. The other 46 were labeled from their titles alone.
For 2027 the items and their abstracts come from the lists of accepted work on the conference site, as they stood on November 16, 2026. That’s 565 items, the full accepted program in every track, so 2027 is a complete year like the others. The two sources line up: for 2026, where both exist, 526 of the 531 proceedings items match an item on that year’s site by title. The symposium meets in February and its proceedings aren’t out yet, so titles and abstracts can still change.
The labels.
There were two rounds.
In the first, one of OpenAI’s models, gpt-6-sol, read every item and answered the questions in the rubric below: is this about AI, is it about generative AI, what role does AI play, and what stance does the item take.
Its labels are the counts in the text, the first chart and the third.
A second OpenAI model, gpt-6-luna, labeled everything again as a check.
In the second round I had one of Anthropic’s models, claude-opus-5-5, do all of it over with the same rubric, since two models from one company can share the same blind spots.
Each pair agreed on whether an item is about AI 97% to 99% of the time, and on its stance 77% to 87% of the time.
The second chart shows the range across all three.
Most of the disagreement is over one boundary: tools that report benefits and also discuss limits, which one model tends to call cautious and another pro.
The stance counts leave out 37 AI items that apply machine learning to problems outside education.
What the models saw. Only each item’s year, title and abstract. No paper’s full text was downloaded or sent to a model. Some authors have concerns about their work being processed by AI or ending up in training sets, so the endpoint matters too. The OpenAI models ran on my university’s Azure OpenAI service, where prompts are not available to OpenAI and not used to train models. The Anthropic model ran on Anthropic’s API, which does not train on API inputs.
The capability reading.
A separate pass by gpt-6-sol over the 503 generative AI items recorded which models each names and how capable it takes AI to be, using the second prompt below.
claude-opus-5-5 repeated it in the second round, and the two gave the same answer for 83% of items.
They agree that agent items went from three or four to 23.
They disagree about how many 2027 items treat AI as a bounded helper: 118 or 95 of 214.
Model names were counted by keyword.
Who uses the agent.
After reading some of the 23 agent items myself, I asked for a third pass.
gpt-6-sol and claude-opus-5-5 read 577 items from 2023 through 2027: every item any labeler had marked as generative AI, plus any that matched a keyword like “agent” or “vibe coding”.
Each was sorted by the role agents play: students taught to build with them, students using them toward some other goal, instructors using them, agents as a subject, agents as a threat, or the authors’ own tooling.
The counts in the text are the items both models agree on.
The categories came from my reading of 16 of the abstracts, so the models matching my reading of those is a consistency check and not an independent test.
What to be careful about. Nobody hand-checked the stance labels. They rest on three models agreeing with each other, and all three read the same rubric, which I did not write. So their agreement doesn’t show that the rubric means what I would mean by each stance. Work that teaches AI as a subject was called neutral by two of the models and often pro by the third. I counted teaching AI as neutral: offering a machine learning course isn’t advocacy. One of the five critical items before 2027 is a 2021 hint generator that simply didn’t work, which is a negative result and not an argument. Abstracts are short, and they say what a paper wants to be known for, not everything it did. Papers were due on July 3 of the year before each symposium, so “2027” here means work finished by the middle of 2026. And stance here measures what the community publishes, not what the people in the room believe.
The prompts. This is my side of the conversation with the agent that did the analysis: everything I typed, in order, with a count of what the agent did before my next prompt. A few things are cut, each marked […]: a path on my machine, where my API keys live, and a remark about other work. Two prompts are left out: a duplicate, and one moving the project to a different machine after the plan was made. Nothing is reworded, typos included. Two requests are missing because I made them while drafting this essay and they were passed along: adding 2027, and the question of who uses the agent. Most of the prompts are short, which is the point: the agent made most of the choices.
Friday, November 13
Agent: 5 replies, 13 tool calls
- How many years of SIGCSE TS proceedings should the analysis cover? 2015–2026
- Which parts of the proceedings should count? Everything, tagged by track
- Simple positive/negative sentiment scoring tends to fail on abstracts (they are almost all upbeat in tone). I'd instead have an LLM classify each AI-related abstract's stance toward AI against a written rubric. How should that classification be run? Claude API script
- What stance categories should each AI-related paper be sorted into? Pro / cautious / critical / neutral
Agent: 1 reply, 3 tool calls
Agent: 0 replies, 7 tool calls
Agent: 2 replies, 2 tool calls
Agent: 1 reply, 1 tool call
Agent: 3 replies, 15 tool calls
Saturday, November 14
Agent: 1 reply, 4 tool calls
Agent: 1 reply, 3 tool calls
Agent: 1 reply, 1 tool call
Agent: 5 replies, 30 tool calls
Monday, November 16
Agent: 3 replies, 16 tool calls
Agent: 1 reply, 3 tool calls
Agent: 1 reply, 5 tool calls
Agent: 1 reply, 14 tool calls
Agent: 2 replies, 10 tool calls
Agent: 2 replies, 11 tool calls
Agent: 1 reply, 1 tool call
The agent wrote the rubric and the capability prompt, and chose the two OpenAI models. I did not review either prompt before it ran. Both are reproduced in full. The first is the rubric each model was given for every item. The whole analysis cost about $39: about $12 for the OpenAI models, $23 for the round with Anthropic’s, and $4 for the pass on who uses the agent.
You are labeling items from the proceedings of the SIGCSE Technical Symposium, the main computing education conference, for a study of how the community's attitude toward artificial intelligence has changed from 2015 to 2026. Each item is a paper, panel, special session, poster, workshop, birds-of-a-feather session, lightning talk, demo or keynote. You see its year, its title and its abstract. Some items have no abstract; label those from the title and use low confidence.
Label what the item itself says. Do not use outside knowledge of the authors or the work, and do not let the year change your answer.
# Step 1: Is the Item AI-Related?
`ai_related` is true when artificial intelligence is a substantive part of what the item is about. AI here means machine learning, deep learning, large language models and generative AI (ChatGPT, Copilot, Codex, Claude, Gemini and similar), natural language processing, computer vision, robotics presented as AI, intelligent tutoring systems, and data science courses whose content is machine learning.
It is false when:
- AI appears only in passing: one application in a list, a future-work remark, or one example assignment among several unrelated ones.
- The item is about data science, statistics, robotics or autonomous vehicles with no machine learning or AI content mentioned.
- The item describes an automated tool that is not AI: test-based autograders, static analysis, rule-based feedback, plagiarism detection by code similarity, program visualization.
- The word is a coincidence ("AI" as an abbreviation for something else, "intelligent" used loosely).
When `ai_related` is false, set `generative` to false and `role`, `stance` and `topic` to "none".
# Step 2: Generative AI
`generative` is true when the item concerns generative AI or large language models specifically: chatbots built on them, AI code generation or completion, prompt writing, AI-written text or code, or "AI" in a context that plainly means these tools (for example "students' use of AI on homework" in 2023 or later). It is false for classical machine learning, pre-LLM intelligent tutors, and AI courses that do not center on generative models.
# Step 3: Role
What part does AI play in the item? Pick the one that fits best.
- `teaching_tool`: AI is used, or proposed for use, to teach, learn, assess, give feedback, write code, or run a course. Includes AI tutors, AI teaching assistants, students using ChatGPT or Copilot, and ML-based graders or hint generators.
- `subject_taught`: AI is the content being taught: an AI or ML course, module, assignment, curriculum or outreach program, AI literacy, AI ethics education, or teaching students to work with AI as a skill.
- `research_method`: machine learning is the analysis technique in an education study (predicting grades or dropout, clustering submissions, classifying forum posts) and the item is not presenting a tool for teachers or students.
- `field_impact`: the item is about what AI means for computing education as a whole: curriculum and program change, academic integrity and policy, the job market, equity, the future of programming instruction, surveys of instructor or student attitudes and usage.
- `outside_education`: AI or ML applied to a problem that is not about education, with no teaching angle: typically a student research competition entry on, say, medical imaging, security or astronomy. Use stance `neutral` and topic `other`.
If an item fits two roles, prefer `teaching_tool` over `research_method`, and prefer the role that the item's main contribution is about.
# Step 4: Stance
Stance is the position the item takes on AI, mainly on the use of AI in education and by students, or on AI technology itself when the item is about teaching it. Judge the position, not the tone: nearly every abstract sounds upbeat about its own work. Findings count as a position: a study that finds AI helped is evidence for, and one that finds harm is evidence against.
- `pro`: builds, deploys, recommends or advocates AI use, or reports mainly positive results from it. Risks are absent or mentioned briefly. An item that builds an AI tool and evaluates it favorably is `pro` even if it lists limitations.
- `cautious`: sees value in AI but gives real weight to risks, limits or conditions. Includes tools whose stated purpose is to curb a risk of AI (a tutor designed to withhold answers so that students do not over-rely on it), studies with clearly mixed results, guidance on using AI responsibly, and items framed as "opportunities and challenges" where the challenges are developed and not just named.
- `critical`: argues against AI use, calls for restriction, or mainly documents harms: learning loss, over-reliance, cheating, bias, inequity, incorrect output, deskilling, environmental or labor costs. Includes studies whose main finding is that AI performed badly or hurt students.
- `neutral`: describes or measures without taking a side, or takes no position on AI at all. Includes most items that teach AI or ML as ordinary technical content, most `research_method` items, usage and perception surveys reported without evaluation, capability measurements reported flatly, and sessions that only convene a discussion.
Boundary rules:
- Teaching AI as a subject is not itself `pro`. A course or assignment on machine learning is `neutral` unless the item also argues about AI's benefits or dangers. AI ethics or critical AI literacy education that foregrounds harms is `cautious`, or `critical` if it argues against the technology's use.
- An item that reports a model can solve course assignments or exams: `neutral` if it only reports performance, `cautious` or `critical` if it frames this as a threat to assessment or integrity, `pro` if it frames this as an opportunity.
- Detecting AI-written work or redesigning assessments to resist AI treats AI as a risk to manage: `cautious`, or `critical` if AI use is treated as plainly harmful.
- When torn between `pro` and `cautious`, ask whether the risks shape the item's design, findings or recommendations. If yes, `cautious`. If they are a sentence of acknowledgment, `pro`.
- When torn between `cautious` and `critical`, ask whether the item still recommends or accepts AI use under some conditions. If yes, `cautious`.
# Step 5: Topic
Pick the single best topic.
- `tutors_and_assistants`: AI tutors, teaching assistants, chatbots and help-seeking tools for students.
- `code_generation`: AI that writes, completes, explains or repairs code, and how students program with it.
- `feedback_and_grading`: AI for grading, feedback, hints, or generating exercises, questions and course materials.
- `academic_integrity`: cheating, detection of AI-written work, misconduct policy.
- `assessment_design`: how exams and assignments should change given AI, and how well AI performs on them.
- `teaching_ai_ml`: courses, assignments, tools and outreach that teach AI or ML as technical content.
- `ai_literacy_and_ethics`: teaching about AI's social effects, ethics, fairness, or AI literacy for non-specialists and K-12.
- `curriculum_and_policy`: program-level and field-level change, institutional policy, workforce and careers.
- `perceptions_and_usage`: surveys, interviews and observations of how students or instructors use and view AI.
- `learning_analytics`: ML used to predict, cluster or analyze student data.
- `other`: AI-related but none of the above.
# Output
Return `rationale` first: one sentence, at most 30 words, naming the evidence for the stance. Then the labels. `confidence` is `high` when the abstract states its position plainly, `medium` when you inferred it, and `low` when the abstract is missing, very short, or genuinely ambiguous.
The second is the prompt for the capability reading.
You are reading abstracts from the SIGCSE Technical Symposium (computing education) that concern generative AI. For each one, record what it says about the AI systems involved. Use only the title and abstract. `models_named`: every specific model or product the text names, written as the text writes it (for example "GPT-4o", "ChatGPT", "GitHub Copilot", "Llama 3.1 8B", "Claude Code"). Empty if it only says "LLMs", "generative AI" or similar. `capability_view`: how capable the item takes current generative AI to be at computing coursework and programming. - `limited`: treats AI output as unreliable, error-prone or weak, or reports that it failed at the task. - `assistant`: treats AI as a competent helper for bounded tasks: answering questions, explaining code, giving hints or feedback, generating exercises, completing or writing short programs. - `solves_coursework`: states or assumes that AI can complete typical course assignments or exams largely on its own, so that courses, assessment or skills must change. - `agentic`: discusses AI that works autonomously over many steps: coding agents, tools that build or modify whole projects, tool use, "vibe coding", or students delegating entire tasks. - `unspecified`: the abstract gives no basis to tell. Pick the highest level the text actually supports, in the order limited < assistant < solves_coursework < agentic. `evidence`: a short quotation or paraphrase, at most 25 words, supporting `capability_view`.
New Essays by Email
Get an email when I publish a new essay: a short excerpt and a link, and nothing else. Unsubscribe anytime.