Experimentation ambassador programs
Flywheel Initiative to increase interest in testing and decrease costs.
Experimentation ambassador programs
I've been designing an Experimentation Ambassador Program to expand the impact and influence of Experimentation in Product-led organizations.
Here are some of the research triangulating thoughts from Experimentation giants.
Feel free to use it for your organization.
Start with the why? What problem are we solving?
For SIGKDD participants, "big tech" representatives, the problem seems to start with the lack of scalability of a centralized model (Center of Excellence).
While the concept of an A/B test is simple, there can be complex practical issues in designing an experiment to test a particular feature and analyzing the results of the experiment.
Product teams need custom support when running experiments, because they often have very specific questions that cannot be answered with a simple set of frequently answered questions.
A centralized support function does not scale very well.
Central teams end up spending too much time on support and not enough on other things. Additionally, specific product domain knowledge is often required to provide support.
A centralized support function requires deep knowledge of all supported products, which is often not feasible. Conversely, anyone providing support needs fundamental experimentation knowledge, which might be easier to scale.
Such democratization of knowledge and expertise enables a better experimentation culture.
For Vista, expanding the influence and impact of experimentation while maintaining a leaner specialists headcount seems to be part of their Hub strategy.
To enable teams to truly operate autonomously, domains (Business units) are responsible for the budget required to assign or recruit Experimentation Ambassadors within their own area. In some cases, depending on needs, this Experimentation Ambassador role might be filled by an existing embedded Analyst, in others, the domain might choose to recruit their own dedicated Ambassador(s).
To facilitate the recruitment of Experimentation Ambassadors, the Experimentation Hub provides interview scripts, technical challenges, and support for hiring managers who wish to recruit an Experimentation Ambassador for their domain as part of the Experimentation Ambassador Program.
This is FWF applied at the staffing level: the central team defines the framework within which teams are free to hire as much support as they think they will need.
At its core, the ambassador program is community building and this is not the type of work most experiments are used to.
Having this clarification is vital as the Direct Responsible Individual (DRI) for this initiative requires a different skillset than the expected applied statistics that are common sense in the industry.
Does it make the boat go faster?
Back to the experimentation flywheel, the Ambassador Program must increase interest in experimentation while decreasing the cost of running experiments.
How do we measure success?
Using Bhavik's metric tree that we explored during May's edition as our starting point.
Assuming the initiative kicks off with a training program for a selected few that is actively developing people. Eventually, those trained ambassadors will generate testing ideas.
By expanding the experimentation community, support availability to product teams increases.
We hope the appetite for testing also grows as ambassadors spread the testing philosophy.
Consider tracking:
Total tests ran with end-to-end support from an Ambassador
Number of certified ambassadors
Proportion of product teams in testing maturity stage with ambassador support
Impact of tests suggested by ambassadors
Standing on the should of giants
From all examples provided on Top Challenges from the first Practical Online Controlled Experiments Summit, I'm building on top of what Yandex and Booking.com have done.
No need to reinvent the wheel.
At Yandex, a program called “Experts on Experiment” exists to scale support.
These Experts are handpicked from product teams by the central experimentation group.
Any experiments must be approved by an Expert before they are allowed to ship.
Experts are motivated because their product needs approval before shipping, so they voluntarily sign up to be an Expert.
Their application is then reviewed by the central experimentation group.
Experts are motivated by the status provided by being an Expert.
They get a digital badge in internal staff systems, so their status is visible to others.
At Booking.com, the “Experimentation Ambassadors” program started about six months ago. The central experimentation organization handpicked people (~15) with experimentation experience and interest in providing support in product organizations that seemed to need the most support.
Ambassadors form the first line of support with a clear escalation path and priority support from the central organization.
Ambassadors are hooked into the central support ticketing system so that they are aware of other open support questions and can pick up tickets as they see fit.
They are included in the experimentation organization’s internal communications, to keep them aware of current developments or issues. There is a monthly meeting to discuss product needs and concerns.
Incentives for Ambassadors include feeling responsible for the product, getting priority support from the central organization, and acknowledgement on their performance review.
Is the juice worth the squeeze? When to prioritize?
Using the EXP model of maturity assessment, I would prioritize the initiative for programs between Walk and Run.
Anecdotes that might trigger the initiative
The type of thing you might hear from leadership or fellow experimenters that hopefully will remind you of Ambassador Programs:
We must increase our support to product teams
Some of our engineers and product leaders are keen to become more involved in the experiment lifecycle
The interest in experimentation is growing across the org, how do we capitalize on that?
What exactly needs to happen?
Getting down to the project management and delivery level, here are some actions and decisions to be made before the rubber meets the road:
Assign a DRI for the initiative, an experimenter with community-building skills (unicorn)
Define the first group of ambassadors to participate in the program
Design training material
Establish ambassador’s incentives, what's in it for them?
Clarify Central Experimentation Organization's expectations for ambassadors
Define rituals, communication structure, and cadence between parts
Clarify how ambassadors are incorporated within the experiment lifecycle, are they gatekeepers? How to legitimize their position within the business?
Pitch leadership for endorsement of the program during all-hands or similar
How do we envision mission complete?
Envisioning mission complete is a practical exercise of starting with the end in mind.
My tactical advice for crafting the vision starts by circling back to the Experimentation mission.
For Product-led organizations, where Experimentation is a Department or Chapter, think Spotify org-chart, my experience suggests that the mission revolves around helping product teams to make data-informed decisions.
How does the ambassador program support the vision?
Increasing support availability while maintaining it decentralized, as closer to product teams as possible, the Experimentation Department maximizes "help."
That's one interpretation.
The expectation is that over time, as more experiments are run, the product team will become more self-sufficient in running trustworthy experiments, and the person from the central experimentation platform team helps with a smaller and smaller percentage of experiments — those that are unique or have issues.
— Microsoft Analytics and Experimentation CoE expectations
Lukas answers specific questions
Q: Should the Center of Excellence (CoE) hire a person to lead the program?
We [Vista] deliberately hired someone who has some academic or at least a statistical background because I believe they have to be able to build authority. And it's very difficult to do that if you don't have a strong analytical background.
The most important part of this role is actually the soft skills of being able to get these ambassadors to feel that they're part of a group.
You want someone who has gone through the motions and ran experiments for a few years at least because otherwise, it's going to be very difficult to connect with these ambassadors.
The ambassador lead enjoys training others, not a statistician who wants to build fancy models or run experiments. Someone who is a community builder.
Some people are interested in stepping up a level but not necessarily becoming a direct manager. This is the ideal role for that.
Q: What is the CoE's expectation from ambassadors?
Follow up: What type of objectives do you give to this person for the next six to nine months?
Monitor how many people are participating in the program.
In our case, participation has doubled in the last few months, so that's starting to grow in the right direction.
For the past quarter, we started looking at how many experiments are executed by non-ambassadors supported purely by ambassadors.
For the future, I expect the new hire to think about how new ambassadors can self-onboard, maybe with a self-paced curriculum. That's how we scale.
Q: What are the incentives behind the program? What's in it for them?
This is about influence.
These are people who do not report to me, and I want to make them into a team.
I'm using all of the basics of team building:
give them a badge that tells them they're an ambassador;
put them on a Confluence page that says these are the ambassadors;
announce to the rest of the organization;
send them a book as a gift as a form of reciprocity.
All that is to build a group of people who has enough confidence so they can break a leg and start testing.
Q: What are the ceremonies with ambassadors? How's the ambassador community engaged with CoE?
Skip level meetings
I personally have one on ones with every ambassador every month. I tell them, "this is just a checkup; ask me anything.”
That is a way to gauge how the program is going.
Quarterly catch-up with sponsors
To make sure that the sponsor (Ambassador's Manager) still feels like they're getting their money's worth.
Content/training sessions
We share content a week before those sessions and get to ask questions.
That dynamic generates an FAQ that we recycle for new cohorts.
Your trusted advisor for scaling experimentation.
Watch the wise, book Lukas’ time.
If you’ve made it this far, I believe you will get value from subscribing.
I dedicate 10 hours so you can read it in about 10 minutes.
Handpicked from other brilliant minds
Kevin Anderson’s curated experimentation podcasts
Podcasts are by far my favorite media format.
Kevin gifted us all with a Spotify playlist that aggregates the gold dust of Experimentation discussions happening across multiple shows.
I’m eager to share more summaries building on top of his initiative.
Elena Verna expands on the five laws of growth
Growth is the maximum distribution of a product via the scientific method.
You gotta love the concise and accurate definitions.
Dublin, Ireland 🇮🇪
The high-minded person must care more for the truth than for what people think.
― Aristotle
From the magnificent Trinity College old library where you find the Book of Keels.
