Being data-driven is not a product strategy
If you stand for nothing, what'll you fall for?
Saying you’re data-driven is an excuse for having no point of view.
Your strategy shouldn’t be driven by data. Your product direction shouldn’t be driven by data. Your data definitely shouldn’t be driven by data.
To be clear, this does not mean that data doesn’t matter. Data matters for all sorts of applications and problems. But if you’re looking for data to tell you what to build, you’re doing it wrong. The answer isn’t in the data. In fact, the only thing that’s often “in the data” is an analyst’s wasted time. (Katie Bauer wrote an excellent post on this at
.)But I thought you loved experimentation?
I like experimentation because it a) forces rigor around hypothesis formation and b) helps test (measure) those hypotheses. It is not a solution in itself!1
If your business is already doing great, experimentation is a fantastic tool to measure incremental gains. But you cannot experiment your way to greatness. And if you're stagnant, experimentation is very much not the answer.
I completely agree with Karri above: if you're Frankenstein-ing experiments together, what you're demonstrating is an absence of product leadership. And if you use Spotify this stagnancy has probably been clear to you for a long time.
A poorly designed and analyzed experiment can do more damage than no experimentation.
I think "data informed" is a better mindset/approach than "data driven."
The article doesn't seem to talk about "Frankenstein-ing experiments together" though, rather it's about combining multiple metrics in a single A/B test to automatically produce a single decision heuristic.
I do have my doubts as to whether it's a good idea to apply a programmatic, blanket solution like this, but the tweet thread itself sounds a bit like someone who has grown disdainful of online experimentation due to the sclerotic political battles between the "design folks" and "data folks" at Airbnb..?