---
title: "Data AIght"
author: "Geoffrey Challen"
date: "2026-11-19"
description: "The ability to analyze data using AI agents is poised to transform society."
canonical: "https://www.geoffreychallen.com/essays/2026-11-19-data-aight"
---

# Data AIght

_The title is an homage to [Data 8](https://data8.org/), Berkeley’s Foundations of Data Science course._

There’s been a lot of attention paid recently to the ability of AI agents to generate code and create technology.
That’s incredibly exciting.
I think it’s going to change how people interact with software and completely transform computing education and the software industry, and [I’ve written about that before](https://www.geoffreychallen.com/ai).

But even that impact might be dwarfed by another capability of these tools that I don’t see discussed as often: their ability to work with and analyze data.

I’ve been using AI agents this way in my own work, and it is remarkable how good they are at it.
They gather data, process it, analyze it, and help you ask questions and draw conclusions from data that’s complicated, that comes from multiple sources, and that’s difficult to make sense of.

## Dumped Data

What makes this matter is that there’s a ton of data out in the world that isn’t useful.
Dumped might be the right word.
Maybe for compliance reasons, maybe for legal reasons, it exists, in some form.
But in the past, if you wanted to ask questions about it, it took a lot of manual work, even if you were using a computer.

It makes me think of the movie _Spotlight_.
Victims had told the reporters that a priest accused of abuse tended to be listed as on “sick leave” or “awaiting assignment” before turning up in a new parish.
And the archdiocese published every priest’s assignment, every year, in a printed directory.
So the reporters [went through nearly twenty years of those directories](https://ire.org/wp-content/uploads/2025/04/IRE-Journal_MayJune2003_PRIEST-SCANDAL-SACHA-PFEIFFER.pdf) and built a spreadsheet by hand, one priest and one assignment at a time.

The pattern had been sitting in public the whole time.
The records just weren’t useful, and certainly not to somebody with a quick question, idle curiosity, a suspicion, or a general concern.
A lot of data like that is still around today, and digitizing it didn’t fix the problem.

A more modern example is the Illinois [Gray Book](https://www.bot.uillinois.edu/gray-book).
For decades the University of Illinois has published the salaries of its employees: not everybody, but essentially all of the faculty, people in management positions, and athletic staff, so a lot of the university’s high earners.

The editions online go back to 1961, and the oldest are scans of something printed on paper, which presumably sat somewhere on campus where you could stop by and find out what the rest of your department was earning.
Today it’s a Word document and a set of web pages.
But it’s still built for browsing, not for questioning.
It can’t quickly tell you, for example, how teaching faculty are paid compared to research-focused faculty.

Now, using an AI agent, you can ask those questions.

It was never really that you couldn’t answer questions like these.
It’s that you wouldn’t.
Nobody cared enough about any one question to spend the amount of time it took to get to an answer.
Now you can get to a preliminary answer much, much faster, and that lets you decide whether to continue, and just to ask more questions.

## Four Examples

Here are four examples from my own work, ordered from the briefest to the most substantial.
Each started because I had a few minutes and got curious about something.

- [**Checking a claim on Reddit**](https://www.geoffreychallen.com/essays/2026-11-19-data-aight#checking-a-claim-on-reddit): does international enrollment matter to the university’s finances? 10–23 minutes.
- [**Sixty-five years of the Gray Book**](https://www.geoffreychallen.com/essays/2026-11-19-data-aight#sixty-five-years-of-the-gray-book): has spending on people who aren’t faculty grown? About an hour.
- [**Teaching-faculty pay across many campuses**](https://www.geoffreychallen.com/essays/2026-11-19-data-aight#teaching-faculty-pay-across-many-campuses): how underpaid are teaching faculty? 1–1.5 hours.
- [**What each unit spends on instruction**](https://www.geoffreychallen.com/essays/2026-11-19-data-aight#what-each-unit-spends-on-instruction): salary, schedule, and enrollment data, joined. 5–10 hours.

None of these analyses are finished.
Each would need more validation before I’d stand behind the numbers.
But they give you a sense of whether a question is interesting enough to keep going, and that’s helpful to know without spending weeks finding out.
These are also only a subset.
Some of the more interesting things I’ve done this way, and the more sensitive ones, aren’t here.

## Checking a Claim on Reddit

A conversation came up on Reddit about international student enrollment and the university’s finances.
International enrollment is starting to decline, and the United States has taken an increasingly hostile posture toward international students.
Someone claimed that the impact on the university wouldn’t be that big.
I suspected that was wrong.
So I pasted the claim to my agent and asked.

> **Me** 10:08 AM: Someone on Reddit claimed this: “UIUC only gets around 20% of its funding from tuition and only about 30% (smaller enrollment but higher cost) of that comes from international students. \[…] Majority of money comes from the state which is paid by Illinois taxpayers.” Can that be true? Can you pull the latest budget and take a look? \[…]
>
> **Agent** 10:12 AM: The claim is mostly wrong: its international share is in the right range, but tuition is a much bigger part of the budget than 20%, and the state is nowhere near a majority. \[…]
>
> **Agent** 10:12 AM: The budget does not split tuition by residency, so this part is my estimate, not a published figure. \[…]

It pulled the latest university budget.
That’s a document built not just to share but also, a little, to obfuscate.
There’s a ton of data in there, and it’s very confusing.
It makes sense to the person who wrote it, maybe.
It isn’t designed for public use, but now you can use it to answer a question like this one in a few minutes.

Here’s what I posted back:

> Those numbers don’t match the university’s own budget. In the FY2027 budget for the Urbana campus, tuition is \$1.1B of \$3.9B (28%), not 20%. The direct state appropriation is \$290M, about 7%. Even if you count the pension and health benefits the state pays on the university’s behalf, state support is about 27%—less than tuition, and nowhere near a majority.
>
> The total budget is also the wrong denominator, since it includes research grants, housing, dining and athletics, none of which pay for teaching. Instruction is funded almost entirely from the pool of tuition and state money, and that pool is 78% tuition and 20% state.
>
> Your 30% international share is about right: my estimate from the published enrollment and tuition rates is 26–33%. But that means international students contribute roughly as much as the entire state appropriation, or more. An international undergraduate pays about \$38,600 a year against \$15,200 for an Illinois resident. Replacing them all with residents would cost about \$127M a year, or around 25% more tuition per resident to make up.
>
> Source: [University of Illinois Budget Summary for Operations FY2027](https://www.pb.uillinois.edu/documents/budget-summary/UI-Budget-Summary-for-Operations-FY2027.pdf) and the trustees’ [FY27 tuition background documents](https://www.pb.uillinois.edu/what-we-do/budget-planning-and-analysis/tuition/FY27/1-Student-Costs_2026-02-10.pdf).

I said none of these analyses are finished.
This one was a quick estimate, and the international numbers are estimates on top of the published budget.
But it was done enough for Reddit.

## Sixty-Five Years of the Gray Book

I had tried to use the Gray Book before.
That project was so tedious that I got a few pages parsed and never figured out how to make it useful.
The format has changed a bunch of times over the years.

But I’d read in various places that universities spend more than they used to on administrators: people who aren’t faculty and don’t teach courses.
And we have all of these years of the Gray Book.
Could we use them to approach that question?

> **Me** 2:26 PM: Don’t base this on `~/code/graybooker`, unless you want to grab those PDFs. That tool never worked well. I’d rather have you take another stab at it.
>
> **Agent** 5:07 PM: \[…] **1961–2003** comes from scans. Each page was read by four OCR engines \[…], and each cell is voted on. \[…] Anything still unresolved is flagged, never guessed.

> **Me** 7:14 AM: OK! Now, can we also grab tuition data over that time period? \[…]
>
> **Agent** 7:32 AM: I found Urbana tuition back to 1989–90. \[…] **Correction to what I told you last time.** I said Urbana’s non-faculty headcount grew about fivefold from 1976. That overstates it. \[…]

The agent grabbed all of the Gray Books and used a variety of strategies to extract the data from them, including four different OCR engines, some of which I didn’t know existed, checked against each other.
A few years it couldn’t recover, and I decided to ignore those, since I was looking at long trends.
When I wanted enrollments to set beside the salaries, it went to DMI, where Illinois keeps its institutional metrics and which is very, very difficult to use.
The agent doesn’t have that problem.
It searches around, finds what it needs, and figures out the format it’s in.

The preliminary conclusion: yes, and then it stopped.
Across the university, faculty were about three quarters of the academic salary spending in the Gray Book in the late 1970s.
By the mid-2000s they were under half, and the share has held at 43–47% since.
Over the same period undergraduate enrollment at Urbana grew by half, while the number of tenure-system faculty didn’t grow at all.

## Teaching-Faculty Pay Across Many Campuses

This one is near and dear to my heart.
I more or less knew the answer, which is that teaching faculty are underpaid pretty much everywhere.
I was curious by how much.

It started in a conversation with a colleague about UT Austin, which is hiring quite a few computer science faculty.
I wrote to their faculty search and asked:

> Are instructional faculty positions paid equivalently to research-focused positions, particularly given the importance of tuition revenue in supporting faculty salaries?

Then I realized I didn’t have to wait for a reply.
Texas publishes its salaries.

In the past I would have tried to answer just that limited question.
With these tools I thought: why not turn this into a broader study?

> **Me** 4:14 PM: I’d start by trying to get as much public salary data for top public programs, not just CS, nationwide. I’d start by just seeing what’s out there. Maybe take the Top 50 or Top 100 public institutions?
>
> **Agent** 4:16 PM: The source survey is underway: six research agents are each covering one region of the country, about 110 public universities across all 50 states. \[…]
>
> **Agent** 7:04 PM: \[…] Some individual salary data is reachable online for most of these schools, but only about a quarter have a source good enough for a teaching-faculty comparison without extra work. \[…]

> **Me** 9:33 AM: \[…] And is there some way to quantify the level of unfairness or underpay, something like a Gini coefficient or something?
>
> **Agent** 9:34 AM: \[…] A Gini coefficient measures inequality within one group, so it doesn’t directly capture a gap between two groups. Two measures do the job together: **Pay ratio** \[…] **Separation index** \[…]

Many states have laws that require disclosing this information.
Of course, each one does it differently.
With the Gray Book the data changed over time; here every school is its own private universe of data.
Different formats, hosted in different places, with different amounts of information.
Some lack important pieces, which forces you to guess things like how many months a person was appointed.

I’m not done with this, and I’ll write more about it later, with the numbers.
The preliminary conclusion is the one I expected, which makes it interesting but sort of depressing: at the same rank, teaching faculty are not fairly paid.
One thing I didn’t expect is how few schools publish useful records: about a quarter of the ones we surveyed.
But it’s another example of what you can get started on now in a few hours.

## What Each Unit Spends on Instruction

The last one was fun because it brought together multiple data sources.
I got curious about how much different units at Illinois spend on instruction per student.
A coarse approximation: take what a unit spends on faculty salaries, which the Gray Book has, and divide by the students it teaches.

There were a bunch of twists.
We went back to the data several times, looked for outliers, and noticed things that didn’t look right.

> **Me** 2:25 PM: How can Microbiology be spending that much per course? Are they using multiple instructors?
>
> **Agent** 2:29 PM: Mystery solved. \[…] **MICR 599**, Thesis Research (15 faculty, each supervising their own section) \[…] This is a real methodological issue: thesis/independent study courses inflate matched faculty counts (and thus teaching spend) without representing actual classroom instruction.

> **Me** 3:09 PM: \[…] Actually there’s two models: in one you look at total spend and divide by total students. In the second, you look at the per course spend and _multiply_ by the number of students in the course, or at least don’t divide. \[…]
>
> **Agent** 3:10 PM: \[…] To do the “multiplies” model correctly, you’d need per-course spend allocation (which faculty teach which courses \[…]). \[…]
>
> **Me** 3:13 PM: But wait, we do know which faculty teach which courses…?
>
> **Agent** 3:14 PM: Right \[…] So we _can_ compute per-course spend. \[…]

That exchange is one of the twists.
If a department spends a certain amount on the faculty member teaching a large course, is that amount divided among the students, or does every student get the benefit of all of it?
[The report](https://gchallen.github.io/illinois-instructional-spend/) lets you slide between the two, along with other parameters.
In theory, for example, research-focused faculty spend 40% of their time teaching.
In practice it’s less.
So that’s another slider.

This pulled together salary data, the course schedule, and enrollment data, all of them sources that are hard to work with.

The preliminary conclusion: what a unit spends on instructional salary per student varies a lot, and it falls as units get bigger.
Across 87 units the median is about \$1,060 per student for the semester.
In the smallest units the median is about \$2,900, and in the largest about \$760.
Units of similar size can differ by a factor of seven.
And by the model’s default assumptions, about a third of faculty salary pays for teaching.

## So Much Easier

My main takeaway isn’t what I learned about my institution.
It’s how much easier this is now.
None of this work involved me touching a line of code or installing a software package.

| Project              | Time (approx.) | Prompts           | Tool calls | Data handled                                               |
| -------------------- | -------------- | ----------------- | ---------- | ---------------------------------------------------------- |
| Reddit claim         | 10–23 minutes  | 6 (187 words)     | 34         | 3 PDFs, 12 revenue lines, 357 rates and counts transcribed |
| Gray Book            | 1 hour         | 10 (423 words)    | 358        | 64 books, 22,896 pages, about 1 million rows               |
| Teaching-faculty pay | 1–1.5 hours    | 14 (559 words)    | about 770  | 110 universities surveyed, 18 sources, about 738,000 rows  |
| Instructional spend  | 5–10 hours     | 137 (3,772 words) | 1,261+     | 4,509 course files, 11,300 section rosters, the Gray Book  |

I suspect there’s a ton of really interesting data out there, and a ton of people with meaningful, high-impact questions they’ve never been able to ask of it, because the work required was too hard.
Those people are going to realize that they can be bored during a meeting, start digging into their school’s financial projections, and find mistakes.
I have prepared internal documents this way.
I have found significant errors in important university reports.

The four above are a sample.
This year I’ve also had agents build twenty-two years of Illinois admissions test scores out of the university’s Common Data Set spreadsheets, work out how my department fares in CSrankings using its own public data, and read [thirteen years of SIGCSE abstracts](https://www.geoffreychallen.com/essays/2026-11-16-mind-the-gap) to see how computing educators talk about AI.
I’ve had them compare computer science and mathematics at Illinois on enrollment, majors, and faculty salaries, look at salary parity in my own department and at grade disparities in the campus’s public grade data, and measure how much one of our core courses has changed across nineteen semesters using only its public websites.
I’ve used them to see how our teaching faculty’s time is spread across course levels, and to analyze a proposal to extend sabbaticals to specialized faculty.
Closer to home, the same approach turned my e-reader’s logs into [a page about what I read](https://www.geoffreychallen.com/reading), and tracked down what was running up a cloud bill.

This also opens up a huge opportunity for data science.
At Illinois and many other institutions, early data science courses have always had an unfortunate component: teaching students Python[^1].
Now agents will just do that on their own, and they’re really good at it.

I’m excited to be teaching my new [conversational programming](https://conversationalprogramming.org/) course, and I think that capability is going to change the world once people realize they have it.
But conversational data science might have more impact.
Maybe I’ll teach that course at some point.

What I’m really looking forward to is what we’re going to find out about the world, and the transparency and accountability that follow, when people begin questioning the data that’s been around them the whole time.

[^1]: Or whatever language you needed to process the data and generate the graphs.
