Stop Guessing, Start Testing
How Experimentation Helps Us Build Better Solutions
Experiments help us test ideas in a controlled and measurable way so we can make the right decisions, take the right actions, and build the right things. However, many organizations still see running experiments as a waste of time. Instead, they pressure product teams to move fast and deliver something, which often leads to solutions no one needs, wants, or uses. Speed without direction is meaningless.
What teams need is a reliable method of separating good ideas from bad ones. Experimentation helps them systematically understand where and how to channel their efforts and deploy their resources. It allows them to gather evidence before committing serious resources. When product teams act on validated insights, they can maximize their odds of creating successful outcomes.
Why Run Experiments
Experimentation improves decision-making quality by aligning actions with evidence.
In every organization, different people have different ideas about what to build. While these perspectives can be useful, it’s easy to get stuck in endless debates about what needs to be done and how to do it. Every product idea involves some assumptions. However, untested assumptions increase the risk of failure. Many products and features have failed to achieve commercial success because teams built something they believed to be useful without sufficient validation. An unused product represents a waste of effort and resources. While conviction is important, so is evidence.
If solutions don’t reflect what users care about, product teams risk building things that are not valuable to either the customers or the business. Experiments help teams verify that they are building the right things. They help teams verify if their solution will work as expected — Will it solve the problem? Will people use it? Will it produce the intended outcome for the customer and/or the business? Experimenting is the best way to get real answers and reduce guesswork. It helps teams objectively verify what does and doesn’t work. This creates consensus around evidence instead of opinion, allowing teams to collaborate more effectively.
The best companies (and employees) are willing to use data to back up their decision-making. They don’t believe that standards or aesthetics are rules written in stone. They don’t believe that only product people or founders know the solutions to problems. They believe that their users can help them understand what to build and how to build it if they are willing to implement the analytics, listen, and test.
— Michael Seibel, Group Partner at Y Combinator
How To Run Product Experiments
Identity The Assumptions
Teams build the wrong things when they operate on incorrect assumptions. For example, if the team is building a new referral feature to increase growth, the underlying assumption is that users want to refer others. However, if they don't, then the feature is doomed to fail. Therefore, product teams must identify the critical assumptions that need to be true for their solution to work. This helps them understand where they lack supporting evidence. When teams know where they need more clarity, they can design better experiments.
Set The Goal
Every experiment needs to have a clear goal (increasing revenue, decreasing churn, etc.), which must be aligned with the overall product and business strategy. The value of experimentation is not just learning new information but acting on it. You cannot run tests unless you have a clear action plan — “If the results indicate X, we do Y.” Therefore, the experiment results must meaningfully inform decision-making. They must help the team definitively decide what to build, improve, or abandon. Product teams cannot afford to waste resources on experiments that yield little to no benefit. Therefore, teams must prioritize high-impact experiments that address the most pressing issues, riskiest assumptions, and biggest opportunities.
Define A Hypothesis
Experiments must have a testable hypothesis that connects a proposed action to a measurable outcome — “IF [we do this action], THEN [we expect this outcome], BECAUSE [of this reasoning].” The product team believes they can solve a problem for their customers or the business. The experiment confirms this belief. However, to ensure their experiment will be useful, they need to clarify their logic and motivations. The product team must establish what measurable results they expect and why. This ensures that there is a strong rationale supporting the experiment and establishes a baseline for the experiment’s success or failure.
Design The Experiment
While experiments help product teams validate their hypothesis, not all experiments are created equal. Each type of experiment yields a different insight. Therefore, they need to identify the best way to gather the information needed. Depending on their goals, some formats may be more useful than others. It’s also important to choose an appropriate sample size and duration for the experiment. This ensures there are enough data points for the results to be statistically significant.
Run The Experiment
It’s important to run the experiment in a controlled environment. This means limiting participation to a chosen subset of users. This minimizes damage in case things go wrong. Regardless of whether the team runs experiments directly on their live app or website or shares prototypes with a small group of users, the experiment should not disrupt the normal user experience for everyone else. It’s also important to monitor the impact on core business metrics. For example, if a new feature on an e-commerce app is hurting revenue, it’s a clear signal to pause the experiment.
Analyze The Results
Experiment results help product teams determine if they have proved or disproved their hypothesis. If their hypothesis is valid, they know their solution works, and they can proceed with further development. If not, then at least they stop any further resources from being wasted. Both scenarios are valuable because they have gained a better understanding of what does or does not work, which will inform future attempts to solve customer or business problems. Also, when teams understand the why behind their results, they can accurately diagnose the cause of success or failure.
Common Experiment Methods
Different goals require different types of experiments.
A/B Testing
A/B testing helps product teams compare two or more versions of a product element (a page, a feature, a user action, etc.). For example, teams might test two onboarding pages with identical content but different calls to action (CTAs) — “Sign Up Now” vs “Join The Community.” Users are randomly assigned to each version (A or B). This allows the product teams to measure and evaluate the impact of changing a single variable. The goal is to collect data to determine which specific change produces the best performance (more clicks, more traffic, more sign-ups, etc.). This provides an objective, data-driven way to optimize product decisions.
Multivariate Testing
Multivariate testing (MVT) is similar to A/B testing, but instead of making a single change, product teams change multiple things simultaneously. For example, teams might test two onboarding pages with different buttons, different messaging, different calls to action (CTAs), etc. This allows the product teams to measure and evaluate the impact of changing multiple variables. The goal is to collect data to determine which combination of changes produces the best performance (more clicks, more traffic, more sign-ups, etc.).
Fake Door Testing
Fake door testing helps product teams gauge interest in a potential feature. For example, teams might include a button to perform an action that is not currently supported. When a user clicks on it, they might see something like a demo video, a landing page, a pop-up, etc., that notifies them that the feature is “coming soon.” The goal is to measure interest in the feature. If there is sufficient interest, the team can explore actually building it. If not, the team simply abandons the feature. This is a useful and cost-effective way to assess if new product ideas resonate with people before committing to developing them.
Click Tracking
Click tracking (also called clickstream analysis) helps product teams evaluate user clicks on their website or within their product. For example, teams might track individual clicks (For example, clicking “Pay” to complete a purchase) or a series of clicks (For example, the clicks made from the initial search to the final purchase). The analysis of user clicks provides insight into user behavior — how they take specific actions. The goal is to identify opportunities to reduce friction and make improvements so they can optimize the user experience.
Funnel Testing
Funnel testing (also called conversion rate optimization (CRO) helps product teams analyze how effectively they enable users to complete certain target actions (making a purchase, subscribing to a service, etc.). For example, teams might assess how effective an onboarding flow is by recording how many users successfully make it past each step of the onboarding process. Each step needs to incentivize the user to complete the next one to maximize the number of sign-ups. Funnel testing gives teams insight into how users behave at each step. The goal is to identify where improvements are needed to optimize business outcomes (user sign-ups, purchases, etc.).
Usability Testing
Usability testing helps product teams assess how easily people can use their solution (either the product as a whole or a specific feature). For example, the team might share a prototype of a new feature with users to test it. They observe how users interact with it and subsequently ask clarifying questions. This helps them understand where, how, and why users are struggling so they can make improvements and fixes before releasing the final version. The goal is to determine how to make using the product as intuitive and easy as possible.
Discovery Experiments
Discovery experiments help product teams validate assumptions and uncover user needs before they start development. For example, teams might run interviews, surveys, or focus groups to explore whether users actually face a certain problem or find a proposed solution valuable. These allow the team to discover what matters to users so they can narrow down specific problems to solve. The goal is to reduce uncertainty early in the product development process and ensure the team is solving the right problem, not just building a solution in search of a use case.
Conclusion
Experiments help teams systematically discover how to build great solutions.
Ambiguity and risk are unavoidable when dealing with real-world problems. The most valuable opportunities often involve the highest risk. That’s why product teams need a reliable way to navigate uncertainty without relying on guesswork. Experiments help product teams make smarter bets. They are a practical way to test hypotheses, validate assumptions, and assess the likelihood of success. They help product teams make informed decisions and intelligently manage risk, even when they don’t have all the answers. Ultimately, the goal of a product team is to make people’s lives simpler and easier in some way. Experiments are a great way to explore how to do that.