THE RIGHT QUESTIONS TO ASK BEFORE YOU BUILD SOFTWARE
People building new software products tend to have a lot of similar questions. To get useful answers, we talked with Lauren Gilchrist, product manager at Pivotal Labs, a leading software development consultancy.
An edited, condensed version of our interview is below, and you can also catch Lauren in person at this year’s Lean Startup Conference, December 8 – 12 in San Francisco, where she’s leading a session on getting confident shipping products quickly so you can learn from users faster.
Ellen Sturm Niz: What do clients ask you most often?
Lauren Gilchrist: They often ask about growth problems before they’ve built anything. They have this huge vision in their head, and they just want to know how to get to that vision faster. So they ask a lot of questions like, “Should I build this feature first and then release that? Or that first and then release this?” Or, “Which of these seven different marketing strategies should I try?”
As humans, we tend to start with solutions. When we go to build a product, we don’t generally say, “What are the kind of problems that, for example, new moms in New York have?” Or: “I wonder which of the problems new moms in New York have is the most painful, and what’s the smallest thing that I can build to solve that problem?” Instead, people say, “What we need is a service that delivers diapers right to your door.” They start with that, which is the solution to a problem. They don’t start with the problem.
So a lot of questions that people come to me with look like this: They have a solution in their head, and they’re like “I’ve already got the truck lined up, and I’ve got the diaper provider, and what I really don’t know how to do is how to expand to 35 cities.” And we’re like, “But you haven’t launched anything yet.”
ESN: So what do you do?
LG: I tell them, “Before we deal with world domination, let’s back up.” I help people walk back up the ladder to get to: Who’s the user? What problem are you solving for the user? Does your proposed solution actually solve that problem—and how can you answer that? Then, how can you answer that faster?
ESN: So, if somebody came to you and they said, “Oh, I’m going to deliver diapers to moms, and I want to do it in 35 cities,” and you’re like, “Wow, you haven’t done it in one city yet.” What would you bring them in to do, and suggest that they do instead?
LG: Where I start first is this: “What’s the need? Help me understand why this is the problem that you’re devoting all of your time to solving.” That usually brings up a couple of different scenarios. One, people see a really big opportunity, whether it’s a marketplace or a need that’s not met. Or two, they’re dealing with a problem that they, themselves, are really familiar with.
There’s a woman named Allyson Downey who’s the founder of the company called WeeSpring. She has this great story about how when she was pregnant, she started asking her friends what kind of bottles they purchased, what kind of cribs they bought, what they needed, what they didn’t need, and it wound up turning into this crazy email chain of spreadsheets. She wound up launching a company a couple years later that was about getting recommendations for baby products from your friends. So, for her, that’s something where she really understood the need, and she wanted to build it. And she started very small.
Contrast that with someone who doesn’t have kids and says, “I’m going to solve diapering for every major city. I’m going to solve the problem that no one ever has enough diapers in their house.” They start with the classic business school case: here’s the market size, here’s how many people are going to have babies and diapers, and they work their way down to the three-year business plan. And nowhere along that journey have they even validated whether or not someone wants to receive a shipment of diapers within six hours. They haven’t talked to any moms, they haven’t potentially tried to make a quick MVP of like, “Hey, here’s a list of five moms, and we’ll just rent a car, go to the grocery store, pick them up, and drive over with the diapers.”
There’s this great story about Zappos when they first started, their MVP was actually that they took photos of shoes in retail stores and posted them. Then, if someone actually bought something, they would go buy the shoes in the retail store and ship them to the purchaser. That’s a great MVP. [This may be obvious, but it bears emphasis: It was a great MVP because it quickly and cheaply answered the question, “Will people buy shoes online?” Once they proved that people would, it became worth it to start building a more thorough system to do that. -Eds]
ESN: I love the story of Zappos. When I heard it, I thought, “That’s genius.”
LG: It’s so smart. It’s amazing. I worked for a startup called Yipit, and they did something similar. This was back when daily deals like Groupon and Living Social were really, really hot. They thought, “Oh hey, I bet it would be cool to see all these in one place, rather than subscribing to six different emails.” So, in three days, they built a website where you could sign up, and they kind of aggregated every single daily deal into their database. So, they would get up at like 3:00 in the morning, and put all of these things into their database, and then they’d send out the email. That was their MVP. And a year later, they actually still didn’t have a way to automate it. It was still all manual—there was a team that manually copied and pasted these daily deals into their database.
Obviously, they automated it finally. But for a year while they were still growing, and still getting their user base, they were building from the ground up and it was a cheap way of making it work without starting with, “Oh, what we really need is a web scraper that goes into a back end and can handle all these different cases.” Because they didn’t know if they actually needed to invest that money at first.
ESN: What are one or two questions you wish clients would ask you, but they never do?
LG: I wish they would ask, “How do I know I’m right?” As entrepreneurs, we start with solutions, and sometimes you don’t learn until much, much later that nobody wants it. It’s this double-edged sword, because to be an entrepreneur, you have to face down a lot of adversity, and you have to deal with people saying, “You’re wrong, and that’s crazy, and you’re never going to do it.” You have to power through that.
At the same time, left unchecked, the assumptions that you’re making that your solution is actually adding value to your users can grow and grow and grow, and it can make a failure much more expensive.
ESN: In other words: The belief you need in your product to overcome the adversities can cloud your judgement. So you don’t see when you need to pivot and do something differently?
LG: Right. It’s a tough balance. Those are the moments when it’s really helpful to come back and say, “How do I actually know that I’m adding value? How do I know that this product that I’ve built is solving a problem?” Without hard evidence, it’s really challenging. That ties back into, “How do I know that I’m right? How do I know that I’m adding value—and how I can actually validate that?” That’s something I wish people would ask more of.
What I teach people over the course of working together, usually, is that you can actually know whether something is adding value before it’s a fully baked, fully featured product. Which is really exciting. That learning and that validation that you get is so satisfying.
ESN: At The Lean Startup Conference, you’re going to talk about how releasing a product that doesn’t solve all of your customers needs at once is okay. Is that what you mean here? That you can get the information and the validation that you need with a less-than-perfect MVP?
LG: Absolutely. An easy analogy is when you release a product. The time period right before you release is kind of like going grocery shopping when you’re really hungry. You’re like, “Oh my god, maybe I really need the rack of lamb with goat cheese and lavender and honey. Or maybe I need some chips. Oh my god, I need a frozen pizza too.” And you’re sort of just crazily throwing things into your cart, because you are so ravenous that you don’t necessarily know what you need. You just need something.
Right before you release a product, you’re like, “Oh my god, what if we forgot the share on Tumblr button? We forgot the “Share on Tumblr” button. We totally need that.” Or like, “Oh my god, we can’t ship to London. What if we get a user in London?” You have this moment of absolute panic, because you’re about to push this thing live, and it may not be perfect. You may have overlooked something. It’s terrifying. It’s absolutely terrifying.
But what I’ve learned over time is that you can mitigate that panic. Just like you’d say in this same analogy, “Eat an apple before you go to the grocery store.” Same analogy you’re carrying out to the product side. You can release smaller versions of your product, or pieces of functionality. It doesn’t even have to be a functional product. You can put wire frames, or sketches, or something that’s totally fake in front of people to get feedback before you have to release this product into the world. And that helps you answer some of these questions about like, “Is this actually adding value, and how, right now?”
ESN: Right, is this “Share on Tumblr” button really going to make or break the product?
LG: Right. Not only is it going to make or break it, but do I even need it? So rather than saying, “Let’s put the button in just in case and delay the launch for a week,” you say, “We’ve already figured out that none of our users use Tumblr.” Or, “Well, we put a ‘Share on Facebook’ button in, and no one used that, so we’re just going to assume that we don’t need that.” Or, “We put a ‘Share on Tumblr’ button in a prototype and no one even asked what it did, or no one tried to click on it, so we can probably do without that.”
ESN: How do you know when you should delay a product because you overlooked something really important, versus when you’re just having that, “I’m hungry at the supermarket” moment?
LG: Ideally, that overlook moment isn’t coming at the point when you’re going to ship something. It’s coming before that, because you’ve been putting your product in front of people before your ship date. You’ve been interviewing users. You’ve been doing actual user testing sessions. That tends to come up earlier and some of the red flags are that no one’s using this feature, or maybe it’s a product that’s supposed to be inherently viral and no one is inviting their friends.
To go back to the diaper example, maybe everybody signs up but no one actually orders diapers. Or, nobody signs up. Or maybe you show this product to someone, and then you ask them how they would explain it to a friend—then they explain it like, “This is a new baby registry,” when you think it is a product where you could get recommendations from your friends about baby products.
When you’re like, “Huh. Whoa. People perceive this differently than I do,” that’s where you might decide, “Huh, maybe this isn’t ready to go, because people aren’t using it and talking about it in the same way that we think they should be, in the way that communicates that they understand the value in the same way that we do.”
ESN: I would think that entrepreneurs would fear that releasing something imperfect would affect the reaction of users. Do they think, I know people won’t like this because it’s not perfect so why release it yet? I’m just setting myself up to get negative feedback?
LG: Yeah, there’s a lot of fear. When you’re an entrepreneur, you have an enormous vision. You’re trying to disrupt an industry, or disrupt patterns. You’re trying to cause people to change their habits and change their behaviors. That’s a very big vision. If what gets released doesn’t match what’s in your head, it can be really hard to accept that users aren’t using the product for a fundamental reason, not because it doesn’t contain everything in your vision.
Another thing we get a lot of, particularly for projects that are for internal customers, is, “Oh my god, it didn’t solve everything.” You get this fear that everything is going to be judged on this one big release and reveal. That’s not actually true. There’s so much work that goes into a release and so much work that happens after it. But you get caught up in the big reveal moment, and that’s another thing that trips people up.
ESN: Does releasing an imperfect MVP ever kill a project? Does it ever happen where you say, “I knew I should have put that ‘Share on Tumblr’ button on there, and that’s why they hate it, and now everything is ruined!”
LG: But there’s so many ways to fix that. You can say—and this is so much about communication—you can always say, “Thanks so much for your feedback. We incorporated that. We listened, and guess what, now there’s a ‘Share on Tumblr’ button, and we put it out three days later.” The other thing that’s important to think about is that products are never done, they’re always evolving. They’re living, breathing, evolving beings, and it’s not as if you just push it out there, and then it sits there for two years, and you don’t change anything.
ESN: You can’t just sit back and say, “Hey, look what we did! I’m done and now we can just make money.”
LG: Right, exactly. You don’t put your feet on the desk and your hands behind your head and have a beer. You’re like, “It turns out we have all these users in the Philippines, how did that happen? I don’t know how we wound up there. Huh, that’s interesting. They can’t check out. We need to fix that.” And all of a sudden you’re like, “Are we a company that caters to a Filipino audience? Do we need two different sign-up flows?” You notice these trends, and all of a sudden you’re like, “Huh, that’s not what it was a couple of months ago. That’s not what is was three days ago. What is this evolving to?”
Going back to the launch scenarios, you learn from releasing something that’s imperfect, because you learn what problems you’ve already solved for people—and you learn what problems you still have yet to solve. You also learn more about what your users think. No matter what stage you release something, whether it’s literally sitting down and putting a napkin sketch in front of someone and saying, “What do you expect to happen when you click here?” Or whether it’s functional software that you release to millions of people, you’re still going to learn. The earlier that you can get something in front of people, the faster you can launch.
ESN: Can you give an example of a company that did release something that was imperfect in the early stages, that didn’t quite solve all the problems, but was good enough, and it worked out?
LG: I want to go back to that Yipit example, where they launched with like four daily deals rolled into one email, and they launched in one city. And then a couple of months later they launched in another city, because they were getting emails saying, “Hey, can you launch in Boston?” And you couple of weeks later, they launched in 10 more cities. And a couple weeks after that, they decided to add in categories, because people were like, “I don’t want all the daily deals, I only want spa deals.” It evolved slowly but surely. Now it’s a way to search for any discount for anything. But that’s not where it started.
ESN: What’s another question that you wish entrepreneurs or enterprises would ask you, but they never do?
LG: “Should I even build this?” I wish they would ask that, because validating the idea in the first place would save time and money and frustration on their part. They’re convinced that they should build it, there’s no question in their mind. But when you ask someone, “What’s the evidence that you should build this?”, they’re very taken aback. It’s like, “Wait, why are you questioning my vision?”
ESN: Because to them it’s so obvious. They know.
LG: Right. It’s so obvious to them. But a lot of times, there’s not a lot of evidence. I’ll give you a concrete example. I was working on a project where we were building a content management system, and we interviewed all the users, and we tried to understand what they needed to accomplish. It was a live video system, and often, the users needed to cut or reorder the segments for breaking news. They also needed to be able to see what everybody else is working on, and they needed to be able to edit things quickly.
You listen to that list of requirements, and you think, “Huh, well, clearly we need drag-and-drop to be able to reorder things quickly.” Building drag-and-drop is actually pretty time-intensive: It can be a week or two of development to build that kind of interface.
What we started with was this: Let’s put a box next to each segment, and that box will have a number in it, and that number will determine the order on the page. If they want to change the order, they can change the number on the box, and then refresh the page. We built that, we put it in front of them and we said, “Build a show on this tool.” Guess what? There wasn’t a single person who said anything about it being necessary to reorder segments faster.
So we validated that we’d solved their problem, A, which was needing to reorder segments quickly, and, B, we saved two weeks of not needing to build drag-and-drop, because we solved the problem in an easier way. In other words, we built the smallest feature that we could to solve that problem, and then we put it in front of them. That let us learn whether that solution solved their problem or not.
Had we learned that it wasn’t fast enough, we would have progressed to a drag-and-drop, or maybe another solution.
ESN: One of the other things you’re going to be talking about at the conference is about how too much empathy for customers slows product managers down. It seems counterintuitive in the startup world to be less empathetic to users. Can you talk a little bit about that?
LG: Absolutely. In my job, the process of finding out who the users are and what their problems are involves interviewing people and asking them a lot of questions and asking why a lot. The project I’m on, we just interviewed eight users at this company. We asked them about everything from their day-to-day habits, to how they use certain tools, what their frustration with those tools were, how they knew when they did a good job. You’re doing this psychological digging of, “What is it that’s really bothering you about this thing?”
You then have all that information in your brain as you start building solutions, so you’ve become incredibly empathetic with these people. All of a sudden, you’re like, “Oh my god, we can’t ship that, because it doesn’t have Bob’s drag-and-drop, and Bob really wants drag-and-drop!” It becomes second nature to want to solve all of their problems and want to help them and want to be this interesting savior figure, whose software solves everyone’s problems.
That’s where I really empathize with entrepreneurs who say, “I just want to fix it all.” But you can’t. You have to realize that people have some large problems and some small problems and some things that aren’t really problems at all. If you can fix one and make people really, really happy, then they’ll start asking for other things.
ESN: So, users will forgive you if you haven’t anticipated everything, but if you solved one of their problems really well, they will be happy with that. And they’ll give you feedback about other problems you could solve that you add later.
LG: That’s exactly it. There’s a Nobel Prize-winning psychologist, Daniel Kahneman, and he has this saying, “Nothing is as important as you think it is while you’re thinking it.” Which basically translates to “Yes, this is the thing that’s keeping you up at night, but your user hasn’t logged into your tool in two weeks and doesn’t remember it.”
ESN: You think Bob is secretly stewing over the fact that there’s no drag-and-drop, but really, Bob is on vacation.
LG: Bob is really worried about the fact that he has a flat tire, or his kid is sick, or he is hungry. There are so many factors in our lives, and the way that people use software is just one of them. And interestingly enough, if you don’t solicit feedback on your product in a way that is encouraging to get both positive and negative feedback, a lot of times you only hear the negative. Because what do people post online? Angry rants.
A lot of the positive reviews will get lost, and people will only talk about their frustrations. What you miss was that experience of watching them use it for the first time. Seeing them say, “This is great.” So, a lot of what we encourage people to do is be there at the point when 10 or 15 people are interacting with your software for the first time, and observe them, and ask them questions. You can watch their faces, you can watch their movements to understand whether or this is actually solving a problem for them.
ESN: So you should get really hands-on user feedback?
LG: Yes, and testing. There are two different phases: user interviews and user testing. User interviews are how you understand who your users are, what’s their demographic information, what are their behaviors, what are their needs and goals, and what is their motivation. And you sort of do that in a sprint, five or 10 people, and you’re like, “Okay, I’ve got all the information about these people and now I understand how my product might possibly add value.” Then you put your head down and you build for a little bit, and then you pick your head back up, and you go back to those 10 people for user testing.
You say, “Hey, here’s this thing. I’d like to watch you use it for the first time. Can you try to do this, and can you try to do that?” And like, “Oh, I saw you got stuck there, why did you get stuck?” In that moment, you get these amazing reactions. You either get, “This is the greatest thing ever.” Or you get, “I don’t get it.” Or, “What goes here? I’m confused.” Or like, “This says name, does it mean customer name or my name?” Like, Bob didn’t realize that he could drag-and-drop.
That’s ideally how you learn whether or not this adds value, and, also, what the other problems are. You’re doing it first hand, so when you launch this thing, you’re confident that you actually have solved somebody’s problem, and you also have a decent understanding of what the complaints are going to be. Maybe you’ll get some surprises, but you’re much more confident.
ESN: Is there any other advice you would give to product managers who want to get their services and products to users more quickly and effectively?
LG: First, before putting your own product in front of people, practice user testing on someone else’s product. Go to a friend and say, “I’m going to show you this product called Wikipedia. I want you to use Wikipedia in front of me to do these specific tasks.” User interviewing and user testing are really sort of awkward things, because you have to be able to sit back and watch someone call your baby ugly—and then just ask, “Okay, why is my baby ugly? Why do you think that?”
If you practice doing that on things that you are not emotionally invested in, it makes the habit and the practice a lot easier when you work on something that you do care about. Even if you do it among your friends or some coworkers and pretend to demo something that isn’t yours, you get that fear out of the way. It becomes more natural for you to ask those questions and to demonstrate something that is new to someone else.
Second, whatever fidelity you have, put it in front of someone. I have people that come to me and they’re like, “Well, I can’t show it to anyone because it’s a sketch, or it’s a wire frame, or it doesn’t actually work.” What I tell them is that you can learn something from whatever level of fidelity you put in front of someone.
For the content management project, for example, we started with just wire frames, and we put those wire frames in front of people, and we said, “Does this have everything that you need to do your job? Is this laid out in the same manner?” They said, “We need to be able to enter data.” So, we built a tool to let them enter data. We actually built the front end of the website and hooked it all up so that it almost was working. We let them type into it, and that’s was where we realized people were missing the save button, which is really bad.
ESN: That’s super frustrating to the user.
LG: It’s super frustrating, and we were able to catch all of that before we actually had the functional software. So, on the one hand, someone could possibly say, “It’s not ready yet. It’s not actually built.” But on the other hand, we were able to learn that faster, which means that by the time that we actually released the software to these users, we were able to validate in a day whether this was a tool sufficient for their needs.
So no matter what you have, and no matter how polished/unpolished, whatever, just put it in front of somebody, because you’re going to learn something, and that’s going to help give you feedback, and it’s going to help you answer some questions, which always helps refine your product.
Third, at the lab, we have a regular user testing schedule, so we actually bring in a couple of people every Thursday from Craigslist. Staff can sign up for slots and say, “Hey, I need to test this new flow that I’m working on,” or, “I need to figure out if someone is going to understand what this app even is, and can they explain it to a friend?” They have the opportunity to have someone just in the office. If you make it a habit where you don’t even think about, you don’t have to have this existential crisis of getting all these users, and, ”Who do we interview and when do we bring them in?” It’s just routine, and it’s there, and you can do it once a week. It becomes less scary. It’s all about reducing this fear of getting things in front of people faster.
ESN: Just put it in front of a couple of people and see if they do it.
LG: Right. For a product that I’m working on right now, we have a date range. Except rather than its having two date fields, one is a date and the other shows up with a button that says, “Add end date.” I was panicking, like, “I don’t know if this is the right way to do it. Are they going to understand that to add a second date in a range, you click the button and then you get the thing?” Everybody did it without a problem.
So my advice is always whatever is keeping you up at night, you can probably figure out by putting something in front of users.
By the way, on this project, the date range worked, but we have a button to add the customer, and the users didn’t know what that did. So now I have a new problem to work on.
ESN: Are there any other articles or information on user testing that you think would be helpful for people?
LG: There is a guy named Rob Fitzpatrick, who wrote a book called The Mom Test. He also has a bunch of videos. He’s the one that taught me how to phrase your user interviews and your user testing questions in a way that is not leading, that actually gets the information out of the people that you need to get it out of. That one was really useful to me.
Ellen Sturm Niz is a freelance content strategist and writer for The Lean Startup Conference.