Avoiding the Waste of Test Churn: Three Tips to Prep Before You Test

Thanks to Beth Sordi for contributing this piece. She is Director of Product Management at BabyCenter where she introduced Lean Startup best practices, shifting the organizational approach to product development and innovation.

Early in my career, I’ll confess that I was a feature-hungry product manager. I wanted as much functionality as my team could build. Wasn’t more better? Schedule and budget were the only limit for how much my team could build, and it was my job to prioritize the long list of outputs.

One example: I spent the bulk of 2010 building a website for a regional hospital. I had enough good sense to be user-centered by conducting usability studies and card sorting exercises, which guided our work. By the time site was about to launch, it was beautiful and consumer-friendly (a novel concept at the time). But my team had spent a year on this project, and we hadn’t validated our assumptions, much less written them down. We hadn’t even really incremented the work to make sure we were on the right track as we went. The site was usable, but was anyone going to use it? Furthermore, was our work going to help drive business outcomes? Did we need to build everything that we built? The end product was successful, but we waited far too long to learn the truth.

After I read The Lean Startup for the first time, it fundamentally changed my approach to building products. From then on, I was inspired to build  in a way that maximized business value or outcome, and minimized waste. I’ve discovered this to be a common desire among my engineering friends too; we want to create things of value, that matter to our customers and to our business. It’s fulfilling and rewarding. When I provide this context for how the work matters, and as an opportunity to explore value, my team members thrive.

Several years ago, a small group of us were interested in learning how we could apply Lean Startup principles to our work so I brought the Lean Startup Co. education program into our organization to train our product development group. At first, the effects of the training were subtle. Discussions within our product teams included new vocabulary such as MVP (minimal viable product), experiments, and assumptions. Then we did more training and worked with a Lean Startup Co. coach to help us search for a viable new business. He helped us turn the corner with our project and influenced how we experiment with new ideas. Applying Lean Startup principles has helped my teams deliver worthwhile contributions to our customers and the organization.  

One particular area where it’s important to get the process right is test preparation or experiment design. Through trial and error, I’ve learned a few lessons about what to do before running any test or experiment. Whether you are exploring a new business model or optimizing a mature product, the following tips can help you get to a decision, eliminate wasted time, and build team cohesion.

Define and prioritize your assumptions

When a team member comes forward with an idea (usually a solution looking for a problem), we talk about what problem she is trying to solve and what opportunity there could be to provide value to our customer. We put together a business model to define how we think the idea will generate revenue for the organization. We discuss her assumptions about how she thinks an idea is going to drive the business model, and we write those assumptions down and put them some place public — if possible. This is important because it’s too easy to forget or revise our assumptions once we start testing. Sometimes this exercise alone can help us refine the idea. For a mature business, we may have a good sense of how the idea will improve various conversion rates all along the customer funnel, but we still need to confirm or test our assumptions.

Not all assumptions are equal. Usually there are only one or two key assumptions that absolutely have to be true in order for the idea to work. These are often the riskiest assumptions, so we build a plan to test these first.

Let’s walk through an example. We have an idea for a new service where new parents can get access to a sleep coach by paying a fee. We assume 10% of our customers will sign up and pay for this program. We will get them to sign up by promoting the service through a new module on our website. We think 20% of all our visitors will click the call to action button in the new module and of those 50% will complete the sign up for the new program. We also assume our website traffic rates stay the same. The most important assumption we want to test first is that 20% of our visitors will click the new module. Next, we will want to test that 50% actually sign up (and pay) for the program. All of this is written down and put somewhere public so later when we discover we can only get half of the 20% we assumed would click on our module, we know we have not met our threshold required for the idea to work. We need to make a decision to pivot, persevere, or pass on this idea rather than wandering through trying more tests and wasting time.

Make a test plan

The next step is to build a test plan where I deliberately write down what I am going to test, what I am going to do when it fails or passes, and for how long I will iterate. Another lesson I learned the hard way. Last year, I was leading a team where we had a policy that we’d continue to test an element until we all agreed we could proceed to the next element. This was intentional to create a collaborative team environment. But it would have been better to build a test plan upfront with decision deadlines and get agreement on the plan. We would have moved faster.

We all know the human tendency to keep testing when a test fails. We deliberate on all the reasons it failed, rework the test, fix it, try it another way, etc. What a waste of time! I see it in test plans when a team vaguely states they’ll just keep redesigning or reworking if the test fails. While I don’t need to and shouldn’t necessarily accept the results of one test, I write down how many cycles I am going to test an element for before proceeding, pivoting, or passing. I also get the team to agree to it before we conduct any test. When the test fails (which is the majority of the time), I can go back to the team with the previously agreed plan so we know what we need to do next.

Confirm sample size for each test

Another key area where testing leads to trouble is sample size with respect to the test goal. I want to run a test with enough users that I am confident I can expect to see those same results later when I build out the component. There are many good sample size calculators out there. But what if the minimal detectable effect or test goal is just too small to measure properly? Let’s say I have an idea that I think can improve our sign-up rate, which has a very low baseline rate in terms of overall traffic, by 5%. Because the baseIine rate is low (less than 10%), I may need to run millions of visitors through the experiment in order to get statistically significant or valid results—that could take forever! I need to rethink my plan. Could I increase the baseline rate by considering only visitors who took some action to reach the sign-up step (assuming that the test results generalize to all visitors?) By thinking this through ahead of time, I figure out how I can learn what I need to know in order to make the right decision without wasting time running erroneous tests.

Good test design is just one component of Lean Startup that has helped my team evolve from building what we could, to building what we should. Planning ahead of testing may seem like a hassle, but it’s not compared to the potential agony of debating the merits of a test later — or worse, building features on a pile of hollow assumptions. Plus, it’s rewarding to know, rather than guess, that I am delivering value for my customers and business, and I know my team members feel the same way.

If you are seeking to bring the entrepreneurial spirit to your large, complex organization, Lean Startup Company can help. We offer live and virtual training, coaching and consulting to empower people and companies to solve their own problems using entrepreneurial management, no matter their industry, size company, or sector of the economy. Email us.

Want more insights? Sign up for our newsletter