I’ve been to CRAP Talks, seen him on stage a couple of times, and often used a quote or two in my speeches. Of course, I want to hear what Bhav has to say about the past, present, and future of experimentation.
Top three highlights
We should build our own AB testing tool." When people say we should build our testing tool, it really makes me angry. It's like, okay, why don't we just go build the electricity power center to power the building here? I think a lot of people think that's the future, and I don't think it is.
I had this question a couple of times before in GrowthMentor: Should we build our testing tool? In my experience, 90% of the time it’s not the best return on capital invested, it’s a buy, not a build.
Something I'm working on right now, we often talk about prioritization frameworks like pie, rice, ice, cakes, you know, other food groups that we use. Those frameworks are there to help you prioritize solutions, right? But it's really interesting to take a step back and say is this the right problem we're solving? Something I'm working on right now is building a framework for identifying the right problem and prioritizing the problems.
It reminds me of Teresa Torres’ opportunity solution tree system, foundational to our product teams’ roadmap exercises. Product strategy exists in the opportunity space, where we account for the size of the opportunity, market, company, and customer factors. RICE becomes usable once we know what’s meaningful to work on now and break down solutions, potential builds, and viable experiments.
The role of AB testing platforms in the future of experimentation, if I use product analytics as an example, typically it has lived in platforms like Google Analytics, Heat, Mixpanel, Amplitude, etc. But there is a need to have product analytics data in your warehouse, which means you can take all that rich information on user behavior, and stitch it together with CRM and customer service data so you can start to understand behavior at a much deeper level. I think the future experimentation, probably heading down the same route, where you still have your AB testing platforms being used to build and serve the test, but the analysis becomes warehouse-centric.
Strong argument. Who else is navigating the reality where siloed metrics in the testing tool become insufficient as stakeholders’ expectations and maturity of analytical questions increase?
Book recommendation
One of the highlights from Andre’s chat in the series was his book recommendation to the community. I missed that from the episode.
We asked Bhav, and he gave us a recommendation in the comments: Superforecasting: The Art and Science of Prediction (4.08 stars on Goodreads).
Marylin Montoya another great session.
Original text adapted for briefness.
If you have five more minutes, continue reading AB Tasty's post.
