I'm Bullish on Experimentation
From evangelist to skeptic and back, thanks to ZIRP?
I had a conversion last week with an early stage investor about experimentation. The conversation really hinged on the TAM (Total Addressable Market) for A/B experimentation platforms. I nervously laughed and told him I had weighed in on this question ~3 years ago with investors from another firm. At the time insufficient TAM was a reason for my extreme bearishness… but I’d recently reversed course and was ready to tell him about my journey.1
For some context: the previous time I was asked to consider an investment in an experimentation platform, I was fresh-ish off ~5 years enmeshed in A/B experimentation at Airbnb. I joined not long after our experimentation platform launched, and was an enthusiastic evangelist for, and practitioner of, it. Alongside analyzing hundreds of experiments, I designed and ran courses on experimentation taken by hundreds of employees, created templates on experiment design and analysis viewed by thousands of engineers and data scientists, and led weekly experiment reviews for the Growth team, whose experiments contributed many millions of incremental nights booked to the company.
I was proud of the work, but also burnt out. When I left Airbnb in 2020, part of the reason was that it felt like we were testing too much; getting stuck in local maxima and limiting our impact to smaller (but testable!) bets.2 At the same time, I truly believed that what we created was so exceptional that it couldn’t be re-created through a 3rd party vendor.
Fast forward to today. I invest in early stage companies and lead a Data team and have found that the gains from experimentation are true differentiators between good and great companies. Primary among these gains is that it avoids what I like to call the “curse of execution”.
Motion vs. progress: The curse of execution
Nearly every Founder I’ve ever worked with loves changelogs. It’s easy to understand why: There’s an ethos in tech that if you just get enough shots on goal (really, “out-execute”), you’ll eventually score. Changelogs in many ways serve as a public declaration of your execution. The challenge here is that while changelogs can easily convey motion, they don’t always equal progress.3 The difference is subtle but meaningful: Motion is just (waves hands) “doing things”, while progress is advancement towards market-based goals, like customers or revenue. Progress is always something you can measure. This is the curse of execution often embodied by changelogs: there is often a focus purely on execution that confuses motion with progress.
One way to overcome the curse of execution is to, rather than count your executions, measure them! “A/B experimentation” is the gold standard for measuring progress. Back in 2014 we could argue that A/B experimentation was hard, but if you can now build a working experimentation platform in a day4 or buy one from a vendor, there’s really no excuse for not measuring your work towards market-based goals and avoiding the curse of execution.
The TAM will create itself
So, we’re back to where we started: Is the universe of companies that want to measure progress really that big? When I was asked in 2021, I wasn’t hopeful. ZIRP made the costs of motion over progress exceptionally low. The general state of data orchestration, transformation and the metrics layer; really, the primitives that experimentation requires, were not as mature. But things look so much different today.
I’m feeling much more bullish and am intrigued to see how things look in the next 3 years. :)
At the time of writing this I am neither an investor in, nor adviser to, any experimentation platform.
Experiments should never substitute for a strong product strategy/ POV.
Sean was once my coach. He’s a good dude!
Dan is a former colleague and current friend. Much of how I build Data Teams is based on finding people like him.88