Mike Fox
← Writing

Judging Time: The Planning Fallacy, Frustration, and the Nature of Estimation

essayspsychologyestimation

Intro

There is a screw that has been sitting on my coffee table for well over a month. It belongs to our fancy-schmancy mid-century couch. I had recently reupholstered the thing and found it on the floor just after assembling the rest of the couch.

This morning, prior to hopping online for work, I decided that I’d just wrap things up and screw it in. It’s a flathead (objectively the worst screw type), so I figured I’d simply tighten it with my hand. That way it’d at least stay in place and out of sight.

Naturally, this was something I would have expected to take maybe like 15-20 seconds. So of course, it took well over 25 minutes.

mid-way through I had the couch tilted on its back. sections had come unmoored from their screws, the cloth I had put great effort into fitting was pulled out of shape, and the screw still wouldn’t fit. (strange, because it certainly did before). By this point my fingers hurt, and the living room was in complete disarray.

Feeling my patience giving way to frustration, I began dedicating an increasing amount of attention towards process:

  1. Is there a screwdriver nearby? Probably yes, but not anywhere I can see or locate unless I dedicate a minute-plus to finding one (vastly exceeding my original estimate for the whole job, on a single subtask).
  2. Can I find something else to use instead, any flat stiff object? Yes! A guitar pick. It’ll be soft, but as a proof of concept, and for a screw this size, it should be fine.
  3. It was not. The slot on the screw is unusually sharp (another frustration, and one I should have taken into account, as it was hurting my fingers) and it almost immediately cut into the guitar pick.

By this point, sweaty, exhausted, and fully preoccupied with my own irritation, I relented. Logging on to work in a horrible mood and thinking “This is total bullshit, fuck this couch and this screw.”1

What’s the Deal Here?

Managing Frustration

Keeping in mind my own embarrassment here, I believe there is a low-level stigma around talking about frustration/anger. At least within the communities I’ve grown up in, frustration feels embarrassing. If you can’t keep your cool, people don’t know what to do with you, and it’s regarded as outright childish. We’ve done a poor job providing language for why and how frustration arises, and a worse one at teaching each other to direct it to positive ends.

“Frustration Intolerance” / “This Shouldn’t Be This Hard”

It’s a natural impulse to ask: why me? Irritation and discomfort due to circumstance is an important aspect of human experience, and coping with the difference between our wishes and reality is a friction that occupies the heart of philosophy and art.

I can’t put it better than author Neil Harrington does here:

There is a scene in the film Zulu, in which a small group of British soldiers, surrounded at Rorke’s Drift, face almost certain annihilation. A young private voices his fear and disbelief: “Why is it us, why us?” A sergeant looks over, and replies, as if this were self-evident: “Because we’re here lad.”

We have all entertained similar thoughts, although probably in less dramatic circumstances: the idea that we should be spared the frustrations and discomforts of life, and disbelief at having been unfairly singled out. The sergeant’s reply also encapsulates a philosophy of how we might approach such difficulties in life: it suggests the key is the acceptance of harsh reality. Yet this is not a plea for passive resignation, but an encouragement to make the most of a situation, and take whatever action we can.

  • Harrington (2011)

As a kid who threw a lot of fits, and grew up feeling varying degrees of shame at my failure to articulate and express difficulty: this hits home. I’m a little surprised to have encountered so little literature on it during my education, and truthfully, it never occurred to me to look to see what kind of theories and therapies were out there.

Rational Emotive Behavior Therapy was one of the first frameworks to attempt to fully describe and define frustration, and to elaborate on when that frustration becomes intolerable: “frustration intolerance arises, not just from the wish that reality was different, but from the collision of a demand with reality.” (Harrington 2011)

So in this particular case, the demand was “I want this screw back in this couch in the next 10 minutes,” but the reality was “lol no.”

Mr. Harrington has provided us with a very handy “Frustration Discomfort Scale,” which conceptually breaks it into subtypes:

  • Entitlement
  • Emotional intolerance
  • Discomfort intolerance
  • Achievement frustration

Achievement frustration is what hits hardest for me, because it comes from distress when work is obstructed. The calling card for achievement frustration is the thought: this shouldn’t be this hard. (Which feels VERY familiar.) The literature’s move is not to talk yourself out of the frustration, but to drop the demand while keeping the preference. Easy enough, except when it isn’t.

“Give Me Strength to Change What I Can and Accept the Things I Cannot”

We often expect ourselves and others to behave as fully rational actors, with the idea that rationality is somehow divorced from emotion instead of being an extension of it. We have needs, preferences, and desires that might require our attention or action, and too often we subconsciously push them to the side in pursuit of other goals.

In my case with the screw: I didn’t want to waste time. I was upset at the prospect of losing relaxation time before work, and subsequently felt a decline in self-worth because I felt like I couldn’t fix a “simple” issue. Leaving me with aching hands and a sore back because I was on the floor holding a burred screw.

Trying to push through for an extended period netted me nothing. Operating with the expectation that “I can do this, I just need to keep pushing” - all because I didn’t want to get up, go get additional tools, and lose “extra” time - ended up dragging the process out longer and distracting me from fixing the real issues.

The Planning Fallacy (and Its Limits)

Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

Time and Expectation

I very regularly fear running out of time, panicked that my time doesn’t belong to me. Often when I’m getting frustrated or irritated, it’s less about the specific situation and more about “losing relaxation time.” I would like to believe that if I was given enough advance notice, and if I knew exactly the amount of effort that any given task would require, I wouldn’t lose my cool. If this isn’t a delusion, the clearest answer for me, then, is to do a better job planning and estimating before I sit down to do something. The problem, though, is that people (writ large, not just me!) are bad at forecasting when we’ll finish tasks.

The difference between reality and “this should take like five minutes” is referred to in contemporary psychological literature as the planning fallacy. While the term itself originated in the work of (much-hyped) psychologist Daniel Kahneman and oft-undercited partner Amos Tversky,2 researchers Buehler, Griffin, and Ross were the ones to introduce empirical evidence, in their 1994 paper Exploring the “Planning Fallacy”: Why People Underestimate Their Task Completion Times.

In their studies they found:

  1. People routinely (and consistently) underestimate timelines for tasks for themselves.
  2. Confidence in a timeframe doesn’t make a person more likely to finish on time.
  3. Applying memory of past experience directly to the task at hand might help generate more accurate timeframes.
  4. Outside observers overestimate how long it will take a person to complete a task.

The term “planning fallacy” itself seems a bit disingenuous if what we’re actually talking about is prediction.

The Limits of Study

It might be worth noting that I have a general skepticism about this field as a whole. Not because of the work of any particular academic, but more because of the real-world limitations in controlling for variables and getting ahold of a broad sample of the population for an extended time. (Academics, please feel free to tell me about how proper controls are or are not put into place for this.) Mostly, Buehler, Griffin, and Ross are looking exclusively at undergrads, in groups ranging from 33 to 123 people. That’s not a great representation of the populace as a whole.

But what’s truly irritating about this kind of “economic”-centered psychology is more that it’s quickly and blindly adopted by “business luminaries” and LinkedIn ghouls without context, and then applied broadly. It becomes quick and easy soundbites for folks to slip into their consultant slides (which they can then charge thousands for). Which indeed they do.

Buehler, Griffin, and Ross introduce some interesting thoughts and terminology for rhetoric, even if I think their conclusions can’t stand solely on the data they provide. As a paper, this has pretty broad acceptance, or is at the very least pretty consistently cited, so I think it’s fair to use it as a jumping-off point.

The Important Difference Between “How Long It’ll Take” and “When It’ll Be Done”

In 2011, researchers Halkjelsvik and Jørgensen published a paper that took this a step further. They found that while people routinely underestimate when something will be done (referred to as completion time), that’s different from when you ask them how many hours, minutes, or seconds it will take (performance time).

Every completion-time estimate was underestimated, but the median for performance-time was near zero. So if you ask a bunch of people when something is gonna be finished, almost everybody will forecast poorly. But if you ask them how long it’ll take, individual people will still be off, and altogether the median will work out to be closer to the truth than any given raw answer.

So if we’re forecasting our own deadlines, we’re almost always gonna be wrong. But if we’re estimating the amount of time we’re gonna need, we’re just inaccurate by about a third of the total time in either direction.

This would be like if, instead of just thinking “well hell, it’ll probably take 15-20 seconds,” I had thought about the cumulative amount of time it would take to complete each subtask, on average, of putting that screw into my couch after sitting down and observing the state of all objects and tools required.

On average is important here too. We’re talking about weighting out best case, worst case, and most likely estimates, PERT style:

(Best Case + Most Likely + Most Likely + Most Likely + Most Likely + Worst Case) ÷ 6

This is a prescribed solution to what Halkjelsvik and Jørgensen refer to as the summation fallacy.

Now, for performance-time estimates, we haven’t found anything that can get us closer to a correct estimation. Training, alternate scenarios, asking an observer; none of these can make someone consistently right about how much time something will take. Outside observers aren’t better at performance time than the individual actors either, unlike completion time.

Deadlines: Consequences and Externalities

Buehler et al. explored externally imposed deadlines and found that while 39% of students met their own deadlines, 81% met teacher-imposed deadlines about a day before an assignment was due. So there’s something to be said about how we define “finish” conditions, and the power of external forces to bound “getting it done.”

Maybe it’s a result of true perceived consequences. When we set our own deadlines, we can let ourselves roll past them, but there’s genuine effective pressure that comes from externalities: a failed grade that affects your prospects, disappointment from a loved one, or in more serious circumstances, health repercussions and hunger. The consequences have to feel meaningful in order to have an impact, and even then, there’s a fundamental reality that has to be respected. No human being can write a ten-page paper in 15 minutes.

As someone who is prone to imposing consequential emotional deadlines on themselves, I think great care and understanding should be taken with how and when you use deadlines as a strategy. They can create a ton of needless stress, and aren’t practical for a lot of small, self-motivated work. I don’t think folks should apply the same intensity of consequence to needing to do the laundry as they would to “storing food for the winter.” It’s easy to imagine that I would have felt even greater levels of stress and frustration if, when trying to fix the couch, I’d had a hard deadline for when it could be done.

But Not Me…

Maybe you’re not like this. All the time, anyway. The research I could find on this indicates that it isn’t tied to any particular individual attribute: gender, rated optimism, procrastination, and self-rated time management don’t seem to matter. Even experts who’ve done a specific task over and over again seem to be bad at setting their own deadlines, and not any better at estimating how long they’re gonna need to work on something. (Worth noting that repetition alone leads to greater optimism, not higher accuracy, but getting feedback on your estimates does seem to help things out.)

This genuinely appears to be something everyone is pretty shit at. Which I find strangely encouraging. How nice is it to know that we’re all walking around fooling ourselves into believing that “we can do that, no problem” and “it’ll only take a sec.”

Okay, But What About My Friend Who’s Really Good at Managing Themselves?

I don’t know your friend, but chances are they’ve leaned into overestimation and are breaking subtasks out from a whole. Maybe they’re really good at making detailed lists and generating if-then planning when they’re in the middle of a problem. Maybe they’re using externalities to their advantage.

We’ve all heard of a bunch of these strategies, but we usually don’t want to have to bother with them. We want reality to match our expectations rather than have to modify our process. But they’re not particularly negotiable. They’re what we know helps in a practical sense.

Effective Mitigation Strategies

Planning Solutions

Outside Solutions: Reference Class Forecasting

Kahneman and Tversky suggest that instead of imagining the steps, you use hard data to create a reference class of how long each task has actually taken you in the past, and sum those together. This does get us close, but maybe the biggest problem is keeping an accurate timetable reference for everything you want to estimate. It’s not exactly the most realistic course of action if we’re trying to apply their research to everyday life (though time-tracking apps may try).

So we must turn inwards to look for a solution.

Inside Solutions: Decomposition and Unpacking

A fairly common approach presupposes that unpacking a task into subtasks will make estimates longer, and therefore counteract the bias imparted by optimism and confidence, making them more accurate. This is kinda bumpkis, though; just making longer estimates doesn’t necessarily make the estimates more accurate. What about overshoot? Granted, depending on the project, an overestimate is gonna be better than an underestimate, but at that point, just say everything is going to take you the rest of your life and be done with it.

A long line of papers points this out. Researchers Connolly and Dean (1997) were the first academic source to confirm that decomposed estimates may be less biased while whole-task estimates were more accurate. Forsyth and Burt (2008) found the same, and also noted that segmentation produced even greater error for short tasks. Halkjelsvik and Jørgensen (2021) put another nail in decomposition as an easy out, by building out the summation fallacy and highlighting where it falls short.

As mentioned above, Halkjelsvik and Jørgensen suggest averaging best case, most likely, and worst case estimates for subtasks, and then summing those for the best result. Which still has serious limitations in terms of accuracy and scope. (It’s also not so practical a use case for something like, oh say, putting that loose screw back.)

An Aside on Complexity

Regardless of how you approach planning, you should take the time to identify as many real-world limitations and difficulties as you can. Having an understanding of what makes a task complex, and the kinds of complexity present, can help you develop a more thorough plan and manage your expectations. There are two key kinds of complexity to acknowledge (courtesy of software management god-king Fred Brooks):

  1. Accidental complexity - the manageable bullshit that we create and can fix ourselves.
  2. Essential complexity - the stuff you can’t get around. It’s caused by the very problem you’re trying to solve.

If I need to paint a portrait, accidental complexity would be me trying to do it in the dark despite having a window, or not taking any lessons or classes on how to paint. Essential complexity would be needing a method of putting paint to canvas, and needing different values of paint.

So it might be better to ask: which of the contributing problems is the most important to deal with out the gate? How can I build within myself the impulse to approach specific problems strategically and analytically?

Emotional Solutions

When it comes to the planning fallacy, there’s a lot to explore for mitigating its effects, but at the end of the day, shit’s still gonna happen. Predictions are predictions. So learning to handle yourself is almost always gonna be the most practical first step to improving the process.

Catching Yourself

We’re all just out here trying to find our way through, and even if (like me) you’re generally skeptical of psych and sociological research, it’s still the best we have in terms of rigorous understanding. What science and research is here to do is provide us with the best answers it can, with the best evidence we can muster. If you can keep track of yourself while you’re going into a project and remember these, maybe it can help make things easier for you as you press on.

Proactive efforts:

  1. Match coping to control. Like a cheesy motivational poster: it helps to focus on what you can control. Put problem-solving skills toward what’s controllable, and emotion-focused skills toward the rest. Putting energy into trying to “problem-solve” the uncontrollable is a great way to exhaust yourself.

    (Man, I REALLY wish I had been taught more about the difference between abstract processing (“why does this keep happening to me”) and concrete processing (“what specifically is blocking step 3”). If they aren’t already, I feel like schools could do society a great benefit in making that a consistent highlight across everyone’s education )

  2. Make if-then plans for obstacles. This is particularly well supported in modern literature: if you have time and can swing it, pre-specifying “if X blocks me, then I do Y” removes the in-the-moment decision point where frustration develops.

  3. Goal disengagement and reengagement (Wrosch, Scheier, and Carver). People who can disengage from their unattainable goals and then redirect to new ones show better wellbeing and lower cortisol and inflammatory markers. Worth noting, though, that disengagement alone won’t help. The ability to reengage is what provides the benefits.

  4. Detachment and recovery (Sonnentag). Remove yourself from the problem during breaks and downtime. Don’t keep letting your mind run over the problems.

  5. Self-compassion (Neff). Outperforms self-esteem interventions specifically under failure conditions - which is when self-criticism tends to spike.

Change Your Mind

Abramson, Seligman, and Teasdale’s reformulated helplessness model calls out global (“everything”) and stable (“always”) attributions as the ones that move a simple setback into an emotional state. Maier and Seligman’s 2016 revision is more interesting: that kind of generalized passivity under stress is the brainstem default. We’re ready to accept stress and fall prey to it; what’s learned is that we can control portions of the situation.

The intervention is behavioral: establish control somewhere, by doing anything at all. Cognitive load from unresolved problems reduces the very capacity we need to resolve them. It starts us tunneling, where a small obstacle occupies an enormously disproportionate amount of mental space. Small closable loops aren’t some kind of productivity hack - they’re a key part of how we find the strength to press on.

Closing

I started writing this with the expectation that it’d be done in an afternoon, and ironically it’s been a week. There’s something important to be said for what an optimistic, over-confident, biased timeframe provides. Maybe we would be crushed under the weight of our tasks if we didn’t consistently lie to ourselves about how long they were gonna take. The idea of a reward being so bright that we lie to ourselves to start down the path of reaching it, is good to me.

Where we fall short is in the moment that lie runs against reality, and how we choose to react to it. Where do we want to put our expectations once they’ve aged out of their roles? Let’s embrace them as a part of the project, and let ourselves enjoy being tricked as much as we can.

Addendum: “Productivity Hacks”

Now You Can Finally OPTIMIZE! EXCEED! Right??? GO GO GO!

That’s not really the framing I’m looking to communicate here. I want to better estimate time because often the thing holding me back from doing a thing is the anxiety over how long it’ll take. I know I fool myself, and there’s a part of me that would prefer to stay not doing anything ever.

But I have things I know will make me happy or proud, or bring joy to me by bringing joy to others. This isn’t about optimizing labor for labor’s sake. This is about understanding what makes this shit hard so I can do right by myself and others, not be a machine that can lock in and grind.

We need to consider what we’re optimizing for. It would be a great irony if I went to all this trouble to fix a couch and then never took the time to lay down on it, and enjoy the moment.

Addendum: The Magic of Suffering

We learn through (perceived) failure, and judging when to start or stop doing something that is making you miserable is one of the great tightrope walks of life.

However, increased tolerance of frustration and discomfort enables us to be better placed to make this decision. Indeed, it can be argued that only through our experience of frustration and discomfort do we truly understand the world and ourselves. Whilst we may read about love and loss, it remains an intellectual notion until experienced first hand; and without healthy negative emotions, arising from frustration, our choices would be bloodless, empty, and lacking motivation. As Sherman (2005) has noted, our feelings of healthy outrage at abuse and injustice are what define our humanity, and not something to be treated and removed.

  • Harrington (2011)

I don’t believe in enforcing suffering to build “character,” but doing difficult things - tasks that push us to do what we otherwise wouldn’t want to - provides particular rewards that cannot be earned through other means.

Online Resources

Footnotes

  1. I think screws and threads and the delicate balance of tolerances are very interesting, but this is not the time. ↩

  2. Kahneman’s popular work has taken real damage over the last decade and a half: the priming chapter of Thinking, Fast and Slow rested on findings that failed to replicate, and he said so himself, noting he’d “placed too much faith in underpowered studies.” Which is, in my opinion, made even worse by the fact that he never removed that section of his book. It is worth knowing that his own heuristics research has held up considerably better, but it’s a real mark against his public credibility. ↩