Have a Well-Rounded UX Research Toolkit to Maximize Insights

Dan Berlin discusses the pitfalls of relying too much on unmoderated usability studies and missing out on deeper insights.

Sponsored by Watch City Research
Watch City Research is your trusted UX research partner

Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.

Transcript

Dan Berlin: 0:14

Hi, everyone, and welcome to another episode of the 97 UX Things podcast. Dan Berlin here, your host and book editor. As mentioned in the first episode of this third season, there'll be a few episodes where I go solo and don't interview someone. And this is one of those episodes. Today I'll be chatting about how teams should have a well rounded UX research toolkit to maximize insights. As you may be aware, I'm an independent UX research consultant. And this means I spend a lot of time drumming up business by chatting with others who work in user experience. Often enough, I know the people I'm chatting with pretty well, so there's no sense in launching into a sales pitch. Instead, we geek out about what they're up to professionally, what problems they're solving, and how they go about research operations. These chats are with UX practitioners working at enterprises and agencies that span an array of industries, so I get to hear a lot of different viewpoints. A topic that comes up a lot these days is the use of unmoderated usability tools. For those not familiar and unmoderated usability study is conducted online. Using a tool where UX practitioners can put an onscreen interface in front of study participants, along with a task to perform using that interface. Participants click around as they complete the task, then press a button to indicate they're done. They'll answer a few follow up questions then move on to the next task to repeat the process. Vendors who provide these tools often have a panel of potential participants you can screen to see if they meet your criteria, or you can use your own participants. An unmoderated usability study can tell you where participants made errors, how long it takes them to do these tasks, and other valuable quantitative metrics to uncover UX opportunities. The major player in the space is UserTesting, who recently merged with UserZoom and additional vendors include UXTweak, Maze and Usability Hub. There are plenty of competitors in the market to choose from these days, which is a good thing for smaller companies and consultants as the unmoderated tools have historically been unaffordable to anyone but large enterprises. The people I've spoken to say that these are great tools, and that stakeholders liked the quantitative data. But some lamented about not having the budget to follow up with moderated usability studies, and uncover more of the why behind the user's actions. Using unmoderated studies as your only research method is a strategic mistake. While they do provide insight into what's wrong with an interface, you can't probe on the spot to get at the root causes of participants behaviors and answers. Now, before digging into why unmoderated tools shouldn't be the only thing in your research toolbox. I'll start by saying that I'm not against using them, they can be very useful, especially for large organizations. When an enterprise has many different interfaces out in the world, it's important to learn where to focus UX efforts. Unmoderated studies allow companies to quickly survey a large number of digital properties, see which have the most friction points and then allocate budget to fix these. Additionally, and potentially even more impactful, they allow design teams of all sizes to quickly and quantitatively show that one proposed design performs better than another. It's also important to keep in mind that UserZoom and UserTesting have been key supporters of the UX community. Over the past 15 years, it's been rare to go to a UX conference and not see one or both of these companies as a sponsor, potentially as the primary sponsor. This level of support isn't cheap, and we should be grateful for them and other companies who keep our UX conferences and nonprofit professional organizations going. But back to the topic at hand, how companies are missing out on key insights by relying too much on unmoderated usability studies. This over reliance starts with the price of the most popular and long standing tools. Yearly subscriptions are in the 10s of 1000s of dollars, which greatly eats into a team's research budget. Once the company has spent this money teams often need to use the tool as much as possible to justify the cost. If they should then ask for money for qualitative interviews. They are told that they already spent 1000s of dollars for the online tool. Why should anything more be spent on research? Stakeholders may fixate on the quantitative data and view spending money on additional qualitative research is wasteful. This is a story I heard from at least three people in my recent chats, often followed by a wish for the deeper insights that qualitative research allows. The problem here is that during unmoderated research, you can ask questions of participants. But since these are all pre programmed, you can't ask any follow up questions that are based on the participants answers, which is the key to getting at the root causes of pain points and desires. When we ask only a single question of a participant without a follow up question that probes on their response, we only scratch the surface. As discussed on this podcast in the first episode of the season, we get root causes of participants utterances by asking "why" multiple times and in different ways. Tell me more about that. How does that make you feel? What prompted you to say that? By riffing off what the participant just said, and asking these probing questions, we can get at the deep qualitative insights that meaningfully informed designs. UserTesting does have functionality that allows you to And as an aside, it's worth mentioning that teams who are struggling to find budget for qualitative research can sometimes partner with marketing or consumer insights, who often have larger budgets than design teams. Qualitative interviews can answer questions from a variety of stakeholders and the budget burden can be spread around. go back to specific participants from the unmoderated study and Another trend related to unmoderated research that I noticed during my chats is that teams seem to view usability studies purely as tactical studies. Now, in the traditional sense, a usability study is all about getting tactical insights. Does this design work as the user expect? What are the friction points in the design? Does the interface have the needed information? But this approach doesn't maximize the time we have with participants. Instead, when we when we ask generative questions while participants have the context of doing the tasks, we have an opportunity to uncover deep relevant consumer insights during a usability study. This is why I've often said that usability studies are the Swiss Army knife of UX research - we can make them anything we want ask follow up questions. If you have access to this tool, it'll them to be. Do you need user input on categorization? Add in a card sorting exercise. Do you want to customer input on potential visual design directions? Add in desirability methods such as Product Reaction Cards. Want to know if the product idea is even useful to participants to begin with? Start with open ended questions before showing them the design so they don't get too anchored to it. Then show the prototype after those initial questions to see if it aligns with participants expectations and what they've described. To wrap things up. I think a key takeaway here is that a well rounded toolkit using mixed methods allows UX teams to learn where to focus design efforts, but also get the deep insights definitely benefit your research to take advantage of this needed for discovering the root causes of users behaviors and desires. Unmoderated studies can do a great job of uncovering friction points and other tactical insights. But if these live in a vacuum, it does a disservice to your design teams. Instead, pairing these with qualitative interviews to uncover the why behind the user and behaviors and desires is the key to creating great designs for your end users. If you're only using one tool, and you're a Swiss army knife, why do you have a Swiss army knife at all? Finally, it's worth mentioning that I'm a UX research consultant who's available to help with your research projects. I also provide training and coaching for UX ability to do follow ups and uncover those deep insights. teams, please check out watchcityresearch.com for information about my services. Thanks for listening, everyone. I hope you enjoyed this episode of the 97 UX Things podcast. You've been listening to the 97 UX Things podcast, a companion to the book 97 Things Every UX Practitioner Should Know published by O'Reilly and available at your local bookshop. All book royalties go to UX nonprofits as well any funds raised by this podcast. The theme music is Moisturize the Situation by Consider the Source and I'm your host and book editor Dan Berlin. Please remember to find the needs in your community and fill them with your best work. Thanks for listening.