Not doing research in Production is like only testing a car with crash test dummies in a laboratory
Getting quality user feedback about your product is hard — it takes time to recruit users, prepare a study, ask the right questions, get people to answer honestly, and then summarize the results into actionable next steps.
But there’s no place as good as Production to get feedback. Talk to your users while they’re using their own devices (computer, smart phone, tablet, etc.) and most importantly, while they’re looking at their own account with their own data. Because there’s nothing like real user data to change feedback.
For example, with HelloWallet, we do a lot of testing before we release changes, but that requires us to either make up test data for user research purposes, or prepopulate prototypes / clickable experiences with fake data that inevitably is wrong for the person who’s looking at it. And they often get tripped up by that, throwing the study off on a tangent.
That’s not my 401(k) balance — I wish I had that much.
Well, I don’t spend that much each month on my credit card but am carrying more debt on it.
I’m not saying you shouldn’t test before getting to Production — you definitely should as it’s cheaper and faster than testing with your live app, but once you do release something, get back out there and ask people what they think when they’re using it in the wild. There’s nothing like that feedback.
Product Elevator 