During a busy travel season, digital travel platform Booking.com wanted to try an experiment that would mean changing the site’s landing page. The CEO at the time, Gillian Tans, made the decision to lean into the company’s “test everything” culture—even if it meant failure.
In this episode from 2019, Harvard Business School Professor Stefan Thomke and host Brian Kenny discuss how past experience and intuition can be misleading when attempting to launch an innovative new product, service, business model, or process. Instead, Booking.com and other innovative firms embrace a culture where testing, experimentation, and even failure are at the heart of what they do.
As you plan your summer travels, enjoy this episode about cultivating an experimental mindset with insights from the case Booking.com and Professor Thomke’s book Experimentation Works.
ROBIN PASSIAS: Hi Cold Call listeners. It’s producer Robin Passias welcoming you to a special edition of the podcast with an episode from the archives about Booking.com. Guessing many of you are planning your summer vacations right now. Did you know that according to Booking.com two of the top global destinations trending for 2025 are in Egypt? As you get your laptop or phone ready to do some summer travel planning, take a listen to Professor Stefan Thomke discuss how past experience and intuition can be misleading when attempting to launch an innovative new product, service, business model, or process in this episode from 2019 called: “At Booking.com, Innovation Means Constant Failure.”
Happy summer travels and enjoy!
BRIAN KENNY: The scientific method. It’s been around since the Renaissance. Historians credit Sir Francis Bacon, back in 1621, for having articulated a common platform and vocabulary that is still used today. We begin learning about it in grade school and, unless you work in a lab, you probably forgot about it a minute after high school. Here’s a refresher. The scientific method has six steps, beginning with a question, followed by research that leads to a hypothesis, that’s tested with an experiment, the results of which are analyzed, leading to a conclusion. Simple, yet this approach has led to astonishing advances in science for centuries. It’s been a bit slow to enter the world of business, where intuition, also known as going with your gut, too often drives big important decisions. In the dawn of the age of big data, the scientific method may be experiencing a renaissance of its own in the world of business. Today, we’ll hear from Professor Stefan Thomke about his case entitled, Booking.com. I’m your host, Brian Kenny, and you’re listening to Cold Call, recorded live in Klarman Hall Studio at Harvard Business School. Stefan Thomke is an expert on managing innovation. His research focuses on the process, economics and management of business experimentation in innovation. That’s a perfect topic for today’s conversation. Stefan, thanks for joining me.
STEFAN THOMKE: Thanks for having me.
BRIAN KENNY: Let me ask you to start by setting the case for us. How does it begin, who is the protagonist, and what’s on their mind?
STEFAN THOMKE: Booking.com is a remarkable company in the travel space. Many of your listeners here have probably used Booking.com. Here are just some numbers, about 1.5 million room nights are booked per day. Somewhere between, and these are external estimates, somewhere between 400 to 500 million people visit their website every single month. The case begins as follows. The protagonist, who is a director of testing, walks into the CEO’s office, Gillian Tans, and tells her that he wants to run a radical experiment. He wants to change the Booking.com landing page. This is a landing page that has been optimized for years, through tens of thousands of experiments, and very refined. All the people who use Booking there frequently, know what it looks like, know how to navigate it, and so forth. Here, you have this director of testing walking into the office, and he wants to change it all. He wants to run this experiment to make it look like a Google search page. The question to the class is, as the CEO, and she’s listening to all this, what should she do? Should she just encourage him to go ahead? Should she get involved, and try to encourage him to modify it? Just to add to that, he also wants to run it during the Christmas season, which is a very busy travel season. He wants to also run it on a fairly significant percentage of the regular users of Booking.com, so you have to imagine being a user of Booking.com, and you’re waking up in the morning, you’re getting ready to make your hotel bookings for the Christmas season, and you’re looking at this webpage, and it looks nothing like anything that you’ve seen before. It looks more like a Google search page. They want to test what the reactions are from their customers.
BRIAN KENNY: As the saying goes, what could go wrong?
STEFAN THOMKE: Yes.
BRIAN KENNY: Seems like a lot could go wrong with this. That’s a pretty big decision.
STEFAN THOMKE: A lot could go wrong. There’s a number of things that could happen. First of all, they could just look at it and leave.
BRIAN KENNY: Yeah, thinking they’re in the wrong place maybe, I don’t know.
STEFAN THOMKE: Go to Expedia, or somewhere else. Could be that they would play around with it a little bit, they get confused, and then give up, could be that they get very frustrated. A lot of things could happen. Or, they could actually like it. The problem is, in this kind of approach, the experimental approach, you just don’t know. The data says, actually, in the cases, that Booking has learned, over the years, that they’re wrong about nine out of ten times, that if you have a hypothesis, and the hypothesis seems very reasonable, and you go out and test it, and then something really surprising happens. The default is really that you’re much more likely to be wrong than to be right. He’s basically proposing this because he wants to find out what’s going to happen when they fundamentally change this page. Of course, there is also some background going on here as well, that is Google is getting into the travel business. These are potential competitive forces that are at play here. It’s more than just finding out what users would like to see, or how they react to certain pages, it’s also maybe to find out what their response to a Google-like experience would look like. Perhaps even offering different kinds of services than just the regular accommodations that they always have on their website.
BRIAN KENNY: What led you to write about Booking.com? You have a Harvard Business Review article on this topic that relates back to the case as well. Why is Booking.com of particular interest?
STEFAN THOMKE: Brian, there’s a general phenomenon that’s going on right now, especially in the digital space. As many companies learning, is as the digital economy is expanding, turns out that the way we interact with customers is fundamentally changing. There is an explosion of touch points that we have, through the various communication channels that we have with customers, and all these touch points have to be optimized. There are just a huge number of decisions to be made, and how do we make decisions involving novelty? The way we usually make decisions is we work from intuition. Something that may have worked in the past. We look around at what competitors do, and we may copy competitors, and so forth. Then we have to make all these decisions, but the data says that we’re wrong most of the time. There’s a great example from Microsoft that I’ve written about with my co-author Ronny Kohavi, from Microsoft, in this article. A few years ago, an employee at Bing has this really interesting suggestion. The AdWords that are placed here, he has this idea, “Why don’t we actually take some of the text that’s below the headlines, and just move it up to the headline?” Bing gets lots of suggestions like this. People have lots of ideas. Bing runs more than 15,000 tests a year. People looked at this, and shrugged, and didn’t really pay much attention to this. The idea was essentially lingering for more than six months, until finally, the employee took matters into his own hands, and just decided to make a few changes in the code. Took only a couple days, and launched this thing live on-
BRIAN KENNY: That’s a pretty risky move.
STEFAN THOMKE: Here’s what happens. As soon as he launches this, Microsoft gets this too-good-to-be-true alarm. They have KPIs set up, and they have various systems set up. When something looks really unusual, this alarm goes off. Usually when the alarm goes off, it’s because there’s a bug. This thing goes off, and they all scratch their heads. They check what’s going on, and they run it again. It’s a robust result, and what’s even more amazing is the change is amazingly high. That change alone resulted in more than $100 million of additional revenue in the United States alone. Ended up being, actually, the largest and most successful experiment that Bing has ever run. Kind of get a sense of what’s going on here, right?
BRIAN KENNY: I hope that guy got a promotion.
STEFAN THOMKE: I hope so too. I hope so too. We don’t have to know what works and what doesn’t work. The only way to really find out is to build these testing cultures, where you can run a lot of experiments, on a very, very large scale, and where the game is changing from not getting it perfectly right. Let’s say your hit rate is only about 10%, but if you’re running 10,000 experiments, you’re still getting 1,000 right. That then can actually have a big impact on your business. That’s kind of the backdrop for this, and of course in my own research, I heard about Booking, and that they have this really unique experimentation culture. I just heard it through my own network, and I approached them, and asked them whether they would be interested in letting me look behind the curtain.
BRIAN KENNY: I described, in the introduction, and in your background, that you look a lot at how organizations and firms manage innovation. I think we hear that term a lot, and it’s present in the business press all the time. Is this a facet of managing innovation?
STEFAN THOMKE: Absolutely, yes. It’s in fact a big part of managing innovation. The problem in innovation is that it involves novelty, and again, trying to predict novelty is very difficult, and again, we get it wrong most the time. Novelty comes in many different forms. Could be new products, new services, new business models, new processes, and all sorts of things. When you look at how decisions and these innovations are made, what happens? People go back, and they rely on their experience. Something may have worked well in the past. They may look at big data, and they look at correlations, trying to analyze the data, and they quickly find that correlations are not causality. We have a lot of examples where things correlate very highly, and there’s no causal relationship between them. Maybe the listener will enjoy some funny examples. There’s a big correlation, for example, between hand size and life expectancy, and so before you start looking at your hands and get depressed, there’s actually no causal relationship. The underlying causal variable is gender. Women tend to have smaller hands, and women tend to live longer as well. There are lots of things like that, and so yeah, big data gives us correlations, but it doesn’t give us causality. Then of course we look at context as well. We may look at what has worked in other contexts, but that can be also misleading. Some great examples, which you may be aware of. When Ron Johnson left Apple, he was clearly a retail god, having co-created the Apple store, goes to J.C. Penney, and they want him to do what he did for Apple at J.C. Penney, and things didn’t really work out that well. Then at the end, he had to go, and J.C. Penney was fighting for its survival, because they didn’t test.
BRIAN KENNY: We actually have a podcast on Cold Call, about Ron Johnson and his experience, so users can look for that. Let’s talk about Booking.com a little more. What is unique about the way that they approach this experimentation process?
STEFAN THOMKE: One thing that’s interesting is to look at the history. When you go back, even from the very early days, Booking was founded, essentially, in 1996, about a year after I joined the faculty here.
BRIAN KENNY: Wow, they’ve been around for a while.
STEFAN THOMKE: They’ve been around for a while. They started very small, jus