The Adjudicator Problem: Why Designers Perform Certainty

Someone sends me the Schwartz essay about once a year. It is a single page, it appeared in the Journal of Cell Science in 2008, and it has been passed between graduate students ever since. The argument is that research feels like being stupid all the time, that this feeling is not a symptom of being bad at it, and that learning to sit inside the feeling is most of what a scientific training actually teaches. Schwartz describes research as "immersion in the unknown" and suggests the discomfort is the sensation of doing it correctly.

I like the essay. It earned its readers. And every time I read it I have the same two reactions in the same order: immediate recognition, then a small resistance I could never name.

I finally sat down with the resistance, and it turned out not to be a disagreement. Schwartz is right about science. The problem is that his argument rests on something design cannot supply, and once you see what that something is, a set of professional behaviours I have always found slightly embarrassing stop looking like personality and start looking like structure.

What productive stupidity quietly assumes

Read the essay closely and there is a mechanism running underneath it. You are ignorant. You design a probe. Nature answers. Your ignorance shrinks in one specific place, and the boundary moves. The discomfort is real, but it is temporary and it points somewhere.

The load-bearing part is the answer. An experiment is an external adjudicator, and its great virtue is indifference. It does not care how sure you sounded in lab meeting. It does not care whether your supervisor liked the hypothesis. You can be completely wrong, and the apparatus will tell you so, and the telling is not up to you or to anyone whose good opinion you need.

That indifference is what makes admitting ignorance cheap for a scientist. Saying "I do not know" costs a researcher very little, because the resolution was never going to come from their own confidence in the first place. Nobody's authority is on the line. The admission describes where a process currently stands, and the process has its own way of finishing.

Take the adjudicator away and the same sentence becomes a very different professional act.

The exam and the presentation

Schwartz makes a point about qualifying exams that I have not been able to stop thinking about. The committee keeps pushing until the student gets things wrong or says they do not know, and he argues this is the exam working rather than failing. The whole ritual is constructed to locate the edge of a person's knowledge and then look just past it.

Design has an equivalent ritual at the same career moment. It is the presentation. A young designer stands up and shows work to people who will decide whether it is good.

And it is built to do the opposite. Everything about the format rewards the appearance of resolution: the work is shown finished, the rationale is delivered as a narrative in which each decision led inevitably to the next, and the questions afterwards are answered rather than opened. Nobody pushes until you say you do not know. Saying it unprompted reads as not having done the work.

Two professions, two initiation rituals for the same kind of person at the same stage. One is engineered to find the boundary of what you understand. The other is engineered to conceal it. I do not think anyone designed it that way on purpose, and I think the reason it happened is entirely structural.

Design has no experiment

Horst Rittel and Melvin Webber described this in 1973, in a paper about planning that design theory has been quietly living off ever since. Among the properties they give to what they called wicked problems, three matter here. Solutions to such problems are good or bad rather than true or false. There is no immediate and no ultimate test of a solution. And every attempt is a one-shot operation, because you cannot learn by trial and error when each trial changes the situation it was meant to measure.

That is the adjudicator problem stated fifty years early and about a different profession.

Consider what it means in practice. You develop an identity for a company. You show three routes and one gets chosen. That route ships, and the other two cease to exist in any form that can ever be compared to it. There is no control group for a brand. There is no parallel world running the second route, and no procedure that could construct one, because the thing you would want to measure only becomes real by being built and lived in.

Even the results that do arrive are not evidence in the sense a scientist would accept. Sales moved, and so did the pricing, and there was a new hire in sales, and a competitor stumbled, and the season changed. The identity is one variable inside a system where nothing was held constant and nothing could have been.

There is a deeper version of the problem underneath the practical one. In an experiment, the fact exists before you measure it, sitting there waiting to be found. The rightness of a design does not pre-exist the design. It is partly created by making the thing, releasing it, and letting people build habits around it. You are not failing to detect a fact. You are in a situation where the fact is downstream of your own decision.

The feedback that does arrive is the wrong shape

Robin Hogarth and colleagues have a distinction I find more useful here than anything written inside design. They separate kind from wicked learning environments. A kind environment gives you feedback that is quick, plentiful, accurate and unambiguous, and in that setting experience reliably turns into skill. A wicked environment gives you feedback that is delayed, noisy, incomplete or systematically biased, and in that setting experience does something much less flattering. It produces confidence without producing accuracy.

Design is a textbook wicked environment, and it is wicked in several ways at once.

The feedback is slow. Whatever the market has to say arrives months after the decisions that caused it, by which point you have made two hundred other decisions.

It is confounded, for the reasons above.

It is selectively sampled, which I think is the most under-discussed part. You only receive outcome information from projects that launched, for clients who hired you. The pitch you lost teaches you almost nothing, because the feedback is a polite email and the real reason is usually unavailable and sometimes unknown even to the person who wrote it. Your sample is filtered by exactly the variable you would want to study.

And then there is the timing problem, which is the one that actually shapes careers. Two signals reach a designer about the same piece of work. One is the outcome, which is slow, confounded and often absent. The other is the client's reaction in the room, which is immediate, vivid and unambiguous. In any learning environment, the fast clear signal is the one that trains you.

So the thing that compounds across a design career is skill at producing good reactions in rooms. That is a real skill and it is not nothing. It overlaps with the skill of making good work, sometimes substantially. But it is a different skill, and here is the part I find genuinely unsettling: there is no mechanism anywhere in normal practice that would ever tell you how much of your success is one and how much is the other. A designer twenty years in has enormous evidence that their judgment is good, and almost all of that evidence is drawn from the fast channel.

So confidence does the adjudicating

Put the client in this position. Two routes, both defensible, no test available, a decision required this week and real money behind it. What is left to go on?

The judgment literature has a fairly blunt answer. Paul Price and Eric Stone ran a set of studies on how people choose between advisors and found what they called a confidence heuristic: people preferred the advisor who expressed more extreme confidence, including over an advisor who was better calibrated. Confidence gets read as a proxy for competence, and it is a proxy people reach for readily.

Cameron Anderson and colleagues found the effect running in the other direction too. In their studies of overconfidence and status, overconfident individuals attained higher standing in groups, and a lens analysis showed the mechanism: overconfidence produces a behavioural signature that observers read as competence. The desire for status also promoted overconfidence, which makes the whole thing a loop rather than a one-way error.

Neither set of studies was run on designers. I am importing them, and I will come back to that.

But the shape of the argument is hard to avoid. When no external adjudicator is available, something still has to adjudicate, because the decision does not go away. And in the absence of a truth test, the only remaining signal is the person presenting. Not the work, which the client is by definition not qualified to evaluate against an outcome that has not happened yet, but the apparent conviction of the specialist standing next to it.

This is why I have stopped reading the confident-designer routine as vanity. Clients are not being foolish when they weight conviction. Given a decision that must be made and no way to test either option, "which of these does the person who made them actually believe in" is a reasonable question. It is a bad instrument. It is also the only instrument in the room.

The field selects for the performance because the performance is the only thing available to select on.

Where I do it

I should be concrete, because otherwise this is a critique of other people.

In pitches I have said versions of "this is the right direction for you" with more certainty than I had. The honest sentence would have been that I had three routes, that I had reasons for preferring one, that my reasons were experienced judgment rather than evidence, and that here is what we could do in two weeks to find out whether I was right.

That second sentence is more accurate and less persuasive. It is more useful to the client and it makes me sound less worth hiring. I have watched myself choose the first version knowing all of this, which is roughly the same discovery I made when I measured my own prose against a machine's and found I could not tell them apart. The instrument is pointed at me too.

I do not think the answer is to start delivering the honest version verbatim in a pitch. I think the answer is to notice that the two sentences differ in accuracy and to stop confusing my fluency with my knowledge.

What the substitution costs

Four things, in rough order of how much they bother me.

Junior designers conclude that everyone else is certain. They see finished work presented as inevitable and never see the two weeks of not knowing that preceded it, so the normal condition of the job registers as personal deficiency. This is precisely how Schwartz's essay opens, with a friend who left research because it made her feel stupid, and the tragedy in that anecdote is that she was experiencing the job working correctly and had no way to know it.

It changes what a designer does with work under question. In a field with an adjudicator, the response to a challenge is to test. In a field without one, the response is to defend, because defending is the only move that exists. Over time the rationale stops being an account of your thinking and becomes an instrument of persuasion, and those two things drift apart without anyone deciding they should.

It corrupts the crit, which is the one ritual that could have functioned as a partial adjudicator. A room full of people trained to present resolution will critique for resolution, and the questions that would be most useful are the ones nobody has an incentive to ask.

And it trains the market. Clients who are repeatedly sold certainty learn to expect it, and then a designer offering calibrated uncertainty is not being refreshingly honest, they are underperforming against the category norm. Every practitioner who wants to be honest is competing against everyone who is not.

What actually helps

You cannot manufacture an adjudicator. Anyone who tells you a metric solves this is selling the metric, because the moment your measure is confounded and delayed it has the same problems the thing it replaced had, plus false authority. What you can do is make being wrong cheap and early, and make your own judgment auditable to yourself.

Move the discovery forward. Showing direction while it is still sketches is not an experiment, but it converts an expensive late correction into a cheap early one. This is the entire argument for working in checkpoints, and it is the closest thing the field has to running a trial: not because it settles anything, but because it changes the cost of finding out.

Write the prediction down before the presentation. Before I show work, what do I expect to land, what do I expect to be objected to, what am I actually unsure about. Sealed before, read after. This is a direct application of Hogarth's suggestion that you can deliberately make a wicked environment kinder, and it works because the failure mode of a wicked environment is that memory quietly rewrites what you thought you knew. A written prediction is the only defence I have found against being retroactively certain.

Separate the kinds of not-knowing, and add the one Schwartz did not need. He distinguishes relative stupidity, where somebody else knows the answer, from absolute stupidity, where nobody does. Design needs a third: permanently unknowable, where nobody will ever know, because the question has no procedure that could settle it. The three call for completely different responses. The first is a lookup and it is cheap, so go and do it. The second is the research and it is the work. The third is a decision, and it should be made explicitly and quickly rather than agonised over as though more thinking would resolve it.

Say the uncertainty with a plan attached. "I do not know" on its own reads as incompetence and, in a field with no adjudicator, that reading is not entirely unfair. "I do not know, and here is the cheapest way for us to find out by Thursday" reads as command of the problem. Same admission, and it keeps honesty and competence in the same sentence instead of making the client choose between them.

None of this is an experiment. It is harm reduction, and I want to be accurate about the size of what it achieves.

Where this could be wrong

Three tiers, because the argument does not have one kind of support throughout.

Documented, and cited above: Rittel and Webber's characterisation of problems without a test, Hogarth's distinction between kind and wicked learning environments, the confidence heuristic in advisor choice, and the status returns to overconfidence. These are published, and I have read them rather than repeated a summary of them.

Inference, and mine: that the absence of an adjudicator in design causes the professional performance of certainty. This is the spine of the essay and it is not a measured result. The confidence studies were run on financial advisors and laboratory tasks. Importing a laboratory effect into an unmeasured professional setting is exactly the move I would flag if someone else made it, so I am flagging it in my own.

Untested: whether designers actually perform certainty more than comparable professions. I have no measurement. It is entirely possible that every consulting field does this, that architects and lawyers and management consultants sit in the same position, and that design is unremarkable rather than special. I would not be surprised.

What would settle it is a study comparing designers' stated confidence at the point of recommendation against some real outcome measure, across enough projects to see whether confidence tracks accuracy. And the obstacle is immediate: the outcome measure is the thing I have spent this essay arguing does not exist. A claim that resists testing for exactly the reason it predicts is uncomfortably close to unfalsifiable, and I would rather say that plainly than let it pass as a clever ending.

The part I am confident about is smaller than the essay and survives all of the above. Schwartz can tell his students to get comfortable being stupid because he can promise them an answer eventually. Nobody can honestly make a designer that promise. Whatever we do about the not-knowing, we are not waiting it out.

References

• Anderson, C., Brion, S., Moore, D. A., & Kennedy, J. A. (2012). A status-enhancement account of overconfidence. Journal of Personality and Social Psychology, 103(4), 718–735. https://doi.org/10.1037/a0029395
• Hogarth, R. M., Lejarraga, T., & Soyer, E. (2015). The two settings of kind and wicked learning environments. Current Directions in Psychological Science, 24(5), 379–385. https://doi.org/10.1177/0963721415591878
• Price, P. C., & Stone, E. R. (2004). Intuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17(1), 39–57. https://doi.org/10.1002/bdm.460
• Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. https://doi.org/10.1007/BF01405730
• Schön, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action. Basic Books.
• Schwartz, M. A. (2008). The importance of stupidity in scientific research. Journal of Cell Science, 121(11), 1771. https://doi.org/10.1242/jcs.033340

Source note: all journal sources above were checked against the publisher record on 2026-08-09. The Schwartz essay is open access. Schön is included as the standard reference for reflective practice, which sits behind the section on what helps, though the argument here does not depend on it.