How to Run Experiments for Product Market Fit
When you have validated your idea and validated Problem Solution Fit, the next important milestone for your idea is Product Market Fit.
There are many different conflicting definitions of Product Market Fit. Most are quite ambiguous, and make it difficult to understand when you have achieved it.
While this definition by Marc Andreessen makes perfect sense, it is impossible to measure. What is a ‘good market’? What does it mean that the product can ‘satisfy’ a market?
The above statement by Eric Ries is profound, but it is no help to you if you are among those that ‘had to ask’…
So first of all, we need a definition that allows us to objectively measure if Product Market Fit is achieved.
40% of users seems like something that is measurable. There are for instance several surveys startups can use to get an idea from their users how they are doing. The question that gets asked is usually a variation of this one:
“How would you feel if you could no longer use product X?”
Asking this to your users will definitely give you an indication of how they think they feel about not being able to use the product. But it is still a question about a hypothetical situation, and as Rob Fitzpatrick teaches us in the Mom Test, people can’t really answer those questions truthfully.
In the end, feedback, surveys, and comments are great to have, but the only real test is behaviour shown while using the product or service.
It would be great to have experiments that you can run besides surveys, right?
To test if you have achieved Product Market Fit, you can run several experiments.
The indication that 40% of customers indicate that they would be very disappointed if they no longer have access to a product can be surveyed, but it can also be measured in terms of retention. After all, if people want to continue to use something, they should be seen to continually use it.
This means, that Product Market Fit is for a big part depending on how well you can satisfy your user’s needs, and how happy they are with your solution. This satisfaction is the driver.
To enable measuring retention, it helps to use the AARRR — Pirate Metrics model. The letters AARRR stand for:
Acquisition. You acquire the user. This usually means a sign up.
Activation. The user uses your product. This means a first visit or login.
Retention. The user continues to use your product. This indicates they like your product.
Referral. The user likes your product so much he refers other new users.
Revenue. The user pays you.
For Product Market Fit, especially the Retention and Referral stages of the model are important. It’s not per se about revenue yet. It’s about how easy it is for you to keep users and to have people refer others to your product or service.
Retention and Referral are the signals that indicate the satisfaction level mentioned earlier.
To prepare for your experiments, first map out your AARRR funnel. Try to figure out what you know or don’t know about conversion rates. Where do new users come from before they enter the ‘acquisition’ bucket? How many of them become ‘activated’? How many refer the product or service to others?
To prove you have achieved Product Market Fit, you should be able to retain at least 40% of your users as active users. (The period of activity may vary depending on the nature of your product or service, i.e. some things you use every day, others once a month — it’s not the frequency that counts, but being the preferred solution).
Coming from Problem Solution Fit, the first step towards Product Market Fit is to achieve some (very) early traction. Can you go from your ~100 initial ambassadors to ~1000 enthusiastic users? And can you figure out how happy they are to use the product? This is your baseline.
Next, you need to learn how to influence and hopefully increase Retention. Retention is key in terms of Product Market Fit. You need users to be advocates of your product or service, and you must learn how to keep them happy and make them even more enthusiastic.
Finally, you need to learn how to increase Referral rates. Being able to influence this is key to lower your CAC (Cost of Acquiring a Customer) and is a good indication of how happy people are with your service. After all, people hardly tell their friends to use apps they don’t really like — even when they might still use those apps if they have no choice.
Where in the Idea Validation it was enough to have a rough sketch, and you could get away with a prototype for Problem Solution Fit Validation, now you really need to deliver on key features. That means, building a first functional product: an MVP.
There are many confusing definitions of what an MVP is, but I like to think of it as the minimum product that you can build to prove Product Market Fit.
Right after Problem Solution Fit, the first versions of that MVP will be rough (and won’t achieve the levels of enthusiasm in your users that you’re looking for) — working towards Product Market Fit you’ll get a better understanding of what users really look for in the product or service, and your MVP will develop.
Every feature you add or tweak needs to increase user happiness in terms of retention and referral rates. Most new features will start their lives in the form of small experiments, trying to gauge the effect of a change before actually implementing it.
Keep the ‘minimal’ aspect of MVP in mind! There will be a lot of varied and confusing feedback coming from users, and simply building everything they ask for will not help you. The feedback needs to be deconstructed and you need to look for the biggest common denominators, the real reasons behind the comments. These will show up as risky assumptions when defining your experiments.
Especially in this stage, it is vital to have a good experiment setup and use the experiment canvas as a guide. There are more variables to be aware of than in the earlier stages, and it can be tempting to ‘just try’ a large number of different things — which can lead to going in circles, not getting clear signals to guide your development, which is the whole point of doing experiments.
(Note: This doesn’t always mean that an experiment should lead to an improvement! It could be that you run an experiment to figure out if a certain feature really is important, and test it by removing it. If the output changes dramatically, now you know that feature is important, and you know it’s important to keep supporting it.)
To get clear signals, you need to test the most important hypotheses, and to do that, you need to look at the riskiest assumptions.
It is vital to keep asking what users are unhappy about, and why they are unhappy about that. Come up with assumptions of the reasons behind what users tell you, behind the behaviours you observe. Use the Riskiest Assumption canvas to sort through your assumptions for the best one to test.
Examples of risky assumptions for product market fit:
Example hypotheses for product market fit:
Example methods for product market fit
Keep in mind that the main reason to run experiments in this stage is still not to grow (although it would be nice!) but to learn how you can grow and to increase satisfaction.
The tools to use are a good AARRR model, built in a spreadsheet for instance, that has baselines for retention and referral; the customer journey and personas made earlier (as reference); surveys; user tests; and of course, analytics.
A good analytics setup is vital. You need it to be able to quantify if you are working towards Product Market Fit.
So, if you are working on a digital product or service, it pays to spend time and effort in setting up your analytics across your Pirate Metrics funnel, and to be able to get a grasp of where your users come from, what channels are the best, and what your most happy and loyal users actually do when they use your system. If you don’t know that, you are basically flying blind.
Remember that getting from a freshly validated Problem Solution Fit to a solid Product Market Fit will in most cases take quite some time. This is where you find out if your idea and the solution you built on it actually have a shot at becoming the next big thing. If you crack the code of making users fall in love with your product or service you’re in business!
Keep experimenting!
How to Run Experiments for Product Market Fit
Research & References of How to Run Experiments for Product Market Fit|A&C Accounting And Tax Services
Source