Why Production Is The Best Place To Do User Research

Not doing research in Production is like only testing a car with crash test dummies in a laboratory

Getting quality user feedback about your product is hard — it takes time to recruit users, prepare a study, ask the right questions, get people to answer honestly, and then summarize the results into actionable next steps.

But there’s no place as good as Production to get feedback. Talk to your users while they’re using their own devices (computer, smart phone, tablet, etc.) and most importantly, while they’re looking at their own account with their own data. Because there’s nothing like real user data to change feedback.

For example, with HelloWallet, we do a lot of testing before we release changes, but that requires us to either make up test data for user research purposes, or prepopulate prototypes / clickable experiences with fake data that inevitably is wrong for the person who’s looking at it. And they often get tripped up by that, throwing the study off on a tangent.

That’s not my 401(k) balance — I wish I had that much.

Well, I don’t spend that much each month on my credit card but am carrying more debt on it.

I’m not saying you shouldn’t test before getting to Production — you definitely should as it’s cheaper and faster than testing with your live app, but once you do release something, get back out there and ask people what they think when they’re using it in the wild. There’s nothing like that feedback.

5 Tips For Building Product Roadmaps


Managing a product without a roadmap is like driving off road.

April 2017 update: my thoughts on roadmaps have changed – if you’re working in Agile, be sure to read Why Agile and roadmaps don’t mix.

Not every product needs a roadmap. If the product is really young and you’re still trying to find product/market fit, don’t waste your time planning a roadmap that’s probably wrong. Instead, spend your time building what you think will sell and iterating based on feedback. (yes, I’m assuming you’re building your product using Agile because it’s 2016)

But if you have a lot of stakeholders (clients, sales, marketing, support, operations, etc.) who need to have an idea of what’s coming down the line, you’ll need a roadmap. Here are 5 tips:

1. Group changes into epics.

Ideally you’re grouping together changes based on the problem you’re trying to solve for a user. For example, with HelloWallet, we recently spent a few sprints focused on helping users decide how to juggle their savings and debt paydown goals simultaneously, since we knew from our user research that it was one of the toughest choices our users were facing.

The reason to group changes together is for efficiency sake. If every sprint is a hodge podge of user stories that cover a bunch of parts of your product, there’s a lot of context switching for everyone on the squad — design has to understand the existing UX and suggest changes, engineering has to touch different parts of the code base (some of which might have cobwebs on it), and QA won’t be able to prioritize regression testing scenarios since you’re changing things all over the place.

2. Prioritize epics based on KPIs.

A roadmap is nothing more than a listing of the things that you think have the highest likelihood of moving your KPIs (in particular, the $$$ KPI I discussed yesterday).

Prioritization is a science and an art. You can get scientific by creating a score card to evaluate epics on their likelihood to move different KPIs, or based on other factors. You can edit the weights of the different factors easily so that you’re scoring epics consistently.

example score card

The art comes in with a gut check of whether you’d actually prioritize epic 3 at the top of your list. Maybe you socialize the score card with your squad and get some qualitative feedback (“ugh, that?” from your favorite engineer or “oh, shit, that’d be killer” from your favorite sales lady). And you adjust accordingly.

3. T-Shirt size epics.

As a Product Manager, you should never make up timelines. Mainly because you’re not the one doing the work. You’ll need buy in on timelines. No one on your squad wants to feel like they are handed an unrealistic timeline and have to execute against it. Hence getting them involved in the process.

At HelloWallet and Morningstar, we use t-shirt sizes to get us in the right ballpark for an epic (it’ll never be very accurate). We start by writing the epic as a user story and getting a group of people together to estimate it — tech, design, product is usually a good starting point.

During estimation, a million questions come up. That’s why we write down assumptions to make sure we’re all on the same page in terms of scope. (“oh, what about this scenario?” — make an assumption about how to handle that scenario).

T-shirt sizes are as follows:

  • XS = < 1 sprint
  • S = 1–2 sprints
  • M = 3–5 sprints
  • L = 6–8 sprints
  • XL = 9–13 sprints

Once you have an estimate, get conservative and use the high end of the range. Assume you forgot about some part of the scope or that something you all thought was going to be easy will be hard. No one will ever complain if you execute on your roadmap faster than you said you would (well, I suppose the would if you ALWAYS executed faster). But they’ll really give you the stink eye if you’re late in delivering something they planned for.

4. Build a 12-month roadmap.

Don’t try to plan the next 3 years — it’s gonna be wrong. Just plan the next 4 quarters.

At Opower, we ran some stats and found that our roadmaps were really accurate for the next quarter, pretty accurate for the next two quarters, and almost always wrong after 6 months. That’s not bad — a lot changes in 6 months (especially at startups).

5. Get feedback.

Socialize your roadmap, especially with sales. If they shake their head, find out why. Is there something else they think would be more meaningful? Talk to other people in your organization, even ones who don’t work on your product or in your business unit. And in true Agile form, iterate the roadmap (once or twice). But don’t update your roadmap more often than quarterly. As you can see from above, there’s a lot that goes into the process. If you’re updating it monthly, you’ll never have time to execute on the roadmap.

Picking The Right Image To Spur Action


Yesterday I saw this ad on the train ride home. Now it’s been a while since I’ve dated, but I don’t remember many dates where I was tickling her neck while drinking tea. Nor am I sure that’s a dream date for girls.

My sad dating history aside, the real issue I see with this ad is that it seems to have nothing to do with how GrubHub can help me dazzle my date. It might have if it was a picture of two people eating some delicious food on a sofa while watching TV. Or maybe in a park eating…

And so ultimately GH failed to get me to take action by ordering dinner through their app. Instead I spent 2 minutes wondering what the design team (or agency) for this ad was thinking and then 5 minutes writing this post.

I’m all for real world imagery in products and ads since they get the audience to envision a scene (aspirational or realistic). But they need to inspire action, like this other ad I saw on the train.

The Only Product KPI That Matters

Managing a product without KPIs is like driving a car without navigation — you’ll never know if you’re closer to where you want to be, and neither will your passengers. (“Are we there yet?”, for your road-tripping parents out there)


Key Performance Indicators are simply metrics you use to measure the effectiveness of your product. They may sound simple and obvious but getting stakeholders to agree on what actions are most important to track isn’t easy. But doing so ensures alignment, which is really important, as I’ll explain in another post about using KPIs to generate roadmaps.

The Only KPI That Matters

The $$$ KPI: profitability, which can come in the form of “scalable revenue” or cost savings (or maybe both). If you’re not prioritizing your roadmap according to this KPI, don’t expect to be managing your product for long (even at a startup). But wait, what about helping users solve problems or reaching your company’s mission? Yes, those are definitely important but if you don’t have any users through sales, does it matter how great the user experience is or how much you want to change the world?

Other KPIs

I’ve found it best to have only 2–3 KPIs, including the $$$ KPI, to keep things simple (and to avoid prioritization conflicts when using the KPIs to generate a roadmap). The other KPIs should be related to the $$$ KPI, though — most likely leading indicators of whether you’ll generate more profit. These KPIs may change over time as you learn more about your business, customers and/or users. Maybe you think logins or email opens are the most important metric, until you realize that there’s a particular feature that’s really moving the $$$ KPI.

As an example, with HelloWallet, we use the Financial Wellness Score as a KPI — if we can measure whether we’re improving our users’ personal finances, we’re more likely to keep existing customers and win new ones.

Tracking KPIs

Monthly (maybe quarterly) is perhaps the cadence to track KPIs. Ideally you have a dashboard that’s updated automatically every night. But you probably don’t because it’s hard and there are a million other things that are more important (or are they?). Even if you have to manually update a dashboard, publish it somewhere so all your stakeholders can see it, and discuss the status of the KPIs regularly (including what actions can be taken by different parts of the organization to move the KPI). Otherwise you’ll never know if you reached your destination…