De-Rumsfelding, or Managing unknown unknowns

I recently spoke with a startup that is looking to shake up the world of short-term renting by filling existing but currently empty properties at the last minute by lowering prices. Their premise is that a low price is better than no price. Their business model assumes that X% of short-term rental properties remained empty at any one time, a figure easily determined by looking at the statistics coming from their short-term property rental portal of choice. Their intention was to increase utilisation of these empty properties by Y%, representing an opportunity valued at £Zm.

This is not an unusual approach in the world of business, especially startups. In fact, it’s one I see all the time:

The worldwide market for widgets is estimated to be eleventy gazillion pounds. We’re aiming to capture 13.57% of that market with our patented widget wrangler technology which represents an annual revenue of two squillion pounds. Please invest in us

The problem with this kind of thinking comes when it is assumed that the identification of an opportunity implies successful exploitation of that opportunity. Large numbers of properties lie empty for a proportion of the time so we should be able to fill them, right?

When I asked the short-term rental startup how they calculated that they could increase utilisation of empty properties by Y% their answer was that they looked at the number of days booked vs number of days not booked and assumed they could access a proportion of the available unbooked market. I then asked them whether they had accounted for variations in demand - for example, had they considered that the reason a property remains empty on a wet Tuesday in January is because no-one wants to book a property at that time? They had not (they have now!)

Cognitive Bias

Bias in decision making impacts our lives every day. We all have our own biases informed by our life experiences, education, culture and social interactions. We most often associate bias with negative impulses such as prejudice, but there is a category of biases linked by the umberella term “Cognitive Bias” which have come about through evolution and which have helped the human race survive. We often refer to this kind of bias as gut feel or intuition.

There are a startling number of cognitive biases that have been identified and what they mostly have in common is that they are great for making decisions when it comes to surviving and evolving as a species but are generally unsound when attempting to make informed business decisions.

It doesn’t matter how we categorise the bias that is preventing deeper investigation of the problem at hand. What matters is that a gut feel on the part of the founders combined with a particular (and incomplete) set of data has led to a particular assumption.

We all suffer from cognitive bias whether we are aware of it or not. Startups operate in a world of extreme uncertainty as it is, without the complication of cognitive bias making things more difficult. This is just one of the reasons why experimentation is so important to startups. How do we decide what experiments to do? We start by examining our assumptions.

Making an Ass of U and Me

The significance of an assumption depends largely upon the following factors:

1. The negative impact if it were to be wrong
2. The proximity of it’s occurrence
3. The likelihood of it’s being wrong

There is a saying:

Never assume - it makes an ass of you and me

What it really means is never assume and then stop. By all means assume, but then try to check whether your assumption is correct. Once we know what our assumptions are we can focus on those assumptions which have the greatest negative impact on our business if they were to be wrong. Beliefs about what must be true in order for a startup to succeed are called leap-of-faith assumptions.

My contention is that you have a greater chance of succeeding if you find the fastest possible path to realising your vision. I’m suggesting that the the first, best step on this path is to identify and test your assumptions as quickly and cheaply as possible by creating hypotheses and experiments to test these hypotheses and learn from the results. Start with your leap-of-faith assumptions.

Converting assumptions into hypotheses begins with breaking down your high-level assumptions into the lowest-level elements of which they are comprised (see my blog post Why? comes after What! for some suggestions about how to do this). You’ll know when you’ve gone far enough if your assumptions and their associated hypotheses are:

• Testable They can be shown to be true of false based on evidence
• Discrete They each describe only one distinct, testable thing to investigate
• Precise You know what success looks like (and you define this before you run the experiment to test the hypothesis)

Assumptions can be both explicit (you are aware of them and recognise them as being assumptions) and implicit (they are so much a part of your thought processes and understanding of a situation that you don’t see them as assumptions at all). One of the defining characteristics of cognitive bias is that the assumptions that result from it are overwhelmingly implicit. This is one of the benefits of running workshops with an independant third party: because this third party doesn’t share your way of thinking or familiarity with a situation they are better able to recognise and identify your implicit assumptions than you are.

Now don’t get me wrong. I’m not saying that the short-term rental startup I described above is wrong when they predict a return of Y%. I’m simply saying that I don’t know whether this return is realistic and neither do they. This is such a basic leap-of-faith assumption that it needs to be tested inside out because the consequences of it’s being wrong are so disasterous for them with their current model. The great thing about experiments is that if they test their thinking and it proves to be true then they can continue with confidence. If, however, it turns out to be false then the sooner they know this the sooner they can pivot and succeed in different ways.