Less Peer, More Review

Before I became a teaching professor, I reviewed for eleven mobile systems conferences, including MobiSys. Here’s an invite I received for MobiSys 2017:

As Program Committee Co-chairs for ACM MobiSys 2017, we would be delighted if you would accept our invitation to serve on the Technical Program Committee. Your technical expertise and wisdom would be instrumental in helping us build a strong and diverse program. […] Based on last year’s numbers, we anticipate a review load of approximately 22 papers for TPC members, 15 in the first round and 7 in the second round. […] The TPC meeting is a single day […]. Attendance is required for all TPC members. Please note that agreeing to serve means that you attend the TPC meeting in person—there will be no remote call-in offered. […] You are expected to read and write reviews for papers yourself. Delegation of reviews to others is not permitted.

Now I review and submit work on computing education to SIGCSE⁽¹⁾For the record, I both reviewed for SIGCSE 2027 and had papers rejected by it. But my thoughts on the reviewing process predate the latest round of decisions.⁽²⁾Strictly, the SIGCSE Technical Symposium, billed as SIGCSE’s “flagship conference”. The organization also runs ICER, ITiCSE, and other events. But “SIGCSE” is shorter, and it’s still what everyone calls the symposium.. Here’s an invite I received for SIGCSE 2026:

Tl;dr: If you haven’t already, please fill out the SIGCSE TS reviewer form. We need over 700 volunteers and currently we have 550!

In all my time reviewing in mobile systems I never received a message like this—a mass appeal to lists reaching thousands of members. This is a sign that it’s time to revisit how SIGCSE reviews papers. The good news is that the fix is within reach.

Quality Is Job Number One

Quality Is Job Number One

Program committees have one job: produce high-quality reviews. Reviews determine what a community sees and learns from, they start up or shut down lines of inquiry, and they affect careers.

High-quality reviewing requires people who can and will produce high-quality reviews. Process matters, but no process can fully compensate for poor reviews, so the process should be designed to attract, motivate, and support good reviewers.

The need to do better is becoming more urgent. SIGCSE accepted 32% of papers in 2019, 32% in 2026, and 25.8% in 2027, when submissions grew by 40%⁽³⁾From the program chairs’ decision notification on September 14, 2026: “there were approximately 3004 reviews and 763 metareviews completed for the 763 paper submissions” and “the acceptance was approximately 25.8%.” The SIGCSE Bulletin reports that 2026 accepted 174 of 547.. When NeurIPS had two independent committees review a tenth of its 2014 submissions, they disagreed on about a quarter of the papers, against the 37% you’d expect from coin flips, and that was an invited committee of experts with discussion and area chairs. No reviewing process is perfectly accurate, particularly given that different community members would disagree on what accurate means! But as a conference gets more selective, more decisions land in the band where even good committees disagree, and process becomes more important.

Here’s how the process is organized for a systems conference like MobiSys. A steering committee chooses several program committee chairs, people with an exemplary track record of research and community visibility. The program committee chairs then invite a certain number of people to join the program committee, from established highly-visible researchers to promising newcomers⁽⁴⁾MobiSys now also accepts a limited number of self-nominations, but applicants need a PhD and “two full-length papers or more in MobiCom or MobiSys in the past 3 years.”. Participating in a systems program committee is an honor and a responsibility. Nobody needs to be begged to join.

Compare how the process is organized for SIGCSE. It starts out the same: with a steering committee and program committee chairs. But from there SIGCSE recruits both its reviewers and its senior reviewers through a web form that anyone can complete⁽⁵⁾The 2027 form invites you to “share your interest in reviewing or serving as a senior reviewer (formerly called Associate Program Chair APC).” The reviewer guidelines say that “the Program Chairs invite and appoint the Reviewers and Senior Reviewers” from that pool.. SIGCSE calls this a “community reviewing process”, despite the fact that peer reviewing is always done by community members. And at least for the past few years, people clearly need to be begged to join.

Too Big to Not Fail

Too Big to Not Fail

The size of any program committee is a balance. If the committee is too small, members are expected to do too much work, breadth is limited, and quality suffers. As it grows it becomes more diverse, but it becomes harder for the group to organize and communicate. For systems program committees, the goal seems to be just as large as necessary. SIGCSE’s goal seems to be as large as possible.

MobiSys 2025 had 85 committee members for about 2,800 submitted pages⁽⁶⁾233 submissions of up to 12 pages each, not counting appendices and references.: 33 pages per committee member. The committee is split into a 25-member heavy PC that follows papers through every round and attends the committee meeting, and a 60-member light PC that reviews in the first round only. Each paper gets roughly three reviews in the first round, and those that survive an early-reject cut get more in a second round, followed by a rebuttal and shepherding. Spread evenly, that is about 100 pages for a light-PC member, and roughly 200 for a heavy-PC member who follows papers into the second round. The committee is under no illusions about this: the 2025 self-nomination page warns that “joining the MobiSys PC is an extremely time-consuming commitment with the expectation of investing 200 or more hours.”

SIGCSE 2027 had 724 committee members⁽⁷⁾The 2027 site lists 618 reviewers and 106 senior reviewers for the whole conference; SIGCSE publishes one pool rather than per-track committees, and senior reviewers serve papers and posters. For 2026, SIGCSE thanked 867 reviewers and 121 senior reviewers. for about 4,600 submitted pages⁽⁸⁾763 submissions of up to six pages each, not counting references.: six pages per committee member. The committee is split into 618 reviewers and 106 senior reviewers, who lead the discussion and write the meta-review. Each paper gets three or four reviews in a single round; 2027 averaged 3.9. Spread evenly, that is about 30 pages for a reviewer⁽⁹⁾The chairs reported 3004 paper reviews, which spread over 618 reviewers is 4.9 each, or about 30 pages. I was assigned four.. So a MobiSys light-PC member reads more than three times as many pages as a SIGCSE reviewer, and a heavy-PC member nearly seven times as many, over two rounds rather than one. I estimate that I spent three to four hours on my reviews this cycle.

SIGCSE Board policy asks for at least four reviews and a meta-review per paper and suggests no more than three papers per reviewer, down from a historical four: 18 pages, down from 24. For 2027’s 4,600 pages, that works out to over a thousand reviewers. The 2027 chairs found 618, asked each of them for about 30 pages, and still delivered 3.9 reviews per paper, below the target.

A program committee takes on work on behalf of its community. They read everything, including papers that aren’t there yet, so that everyone else can devote their attention to significant results. I’d argue that this matters even more for SIGCSE. The community is larger, so it produces more work, and I suspect that it reads fewer papers than research-focused academics do. For many SIGCSE members, it would be better for the community for them to read more accepted papers rather than more submitted papers.

Under Your Own Name

Under Your Own Name

More reviewers means less work for each reviewer. But more reviewers also means less-experienced reviewers and potentially more lower-quality reviews. The rest of the process has to be robust to weaker inputs.

SIGCSE ends in six days of online discussion, led by a senior reviewer who writes the meta-review. Reviewers are anonymous to authors and to each other. Only the chairs can connect a review to a name. There is no meeting. As far as I can tell, senior reviewers are also anonymous to each other, although that part of the process is invisible to me. Neither a poor review nor a poor decision carries any reputational cost, which makes poor-quality reviews and decisions more likely.

Fewer, Better Reviewers

Fewer, Better Reviewers

As described above, SIGCSE already has a two-tier committee. When submissions grew, the senior tier grew with them, from 36 in 2017 to 121 in 2026. What’s missing on the papers track is the second round, the meeting, and the names.

So borrow MobiSys’s structure and keep SIGCSE’s people. Round one stays as it is: three reviews per paper from the full pool. Then make an early-reject cut, gated by a senior reviewer who has read the paper, so that three volunteers who misunderstood a submission can’t end it on their own. The rest, perhaps half, go to a second round in which the senior reviewers read them—not just the reviews, the papers—and add reviews of their own. That’s about 45 additional pages per senior reviewer⁽¹⁰⁾Half of 2027’s 4,600 submitted pages is 2,300. Two senior reads each, spread over 106 senior reviewers, is about 45 pages. A MobiSys heavy-PC member reads about 200.. Then hold a meeting: online, by track, a few hours, for the papers still in the middle after the second round. Reviewers stay anonymous to authors, but are identified to each other.

Fewer people reading more pages need more time. In 2027 papers were due July 3 and decisions went out September 14: ten and a half weeks, of which reviewing and discussion took about three and a half. A second round and meeting fit in that window if the notification is split: round one decisions first and round two following. And the smaller group has to be assembled for breadth, with representation from two-year colleges, K–12, and international educators. Fewer, better does not mean fewer, R1.

SIGCSE also has tracks, which invite a divide-and-conquer approach. Papers are already split into computing education research, experience reports and tools, and position and curricula initiatives, with reviewer instructions for each, and it would be natural to split them further: a CS1 track, a K–12 track, a tools track of its own. A track lets you assemble a committee of CS1 experts to review CS1 papers, and the smaller committee makes communication and coordination easier. The people who teach the same course are the sharpest readers of each other’s work, so if CS1 needs its own room, give it one.

This doesn’t need to happen all at once. The existing tracks also allow experimentation. Set up one this way for one cycle. Keep its reviewer pool for round one, add a senior committee sized so that nobody reads more than 50 pages in the second round, hold the meeting, and identify the committee to each other. SIGCSE already collects feedback on reviews, providing a way to evaluate the results. If it works, consider expanding the model to other tracks.

None of this is radical. It’s what other conferences already do, adapted in a way that’s sensitive to SIGCSE’s inclusivity goals and history. SIGCSE has the people, the tracks, and the senior reviewers. It just needs a better process.