97 UX Things
97 UX Things
Bring Rapid User Research Methods to Agile Teams (feat. Bob Thomas)
Bob Thomas discusses his experience implementing rapid research at a large company.
Sponsored by Watch City ResearchWatch City Research is your trusted UX research partner
Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.
Hi everyone and welcome to another edition of the 97 UX Things podcast. Dan Berlin here, your host and book editor. This week, I'm joined by Bob Thomas, who wrote the chapter 'Bring Rapid User Research Methods to Agile Teams'. Welcome, Bob!
Bob Thomas:Thank you, Dan. I'm glad to be here.
Dan Berlin:Glad to have you aboard here. Can you please introduce yourself?
Bob Thomas:Certainly, I'm happy to do that. I'm a user researcher. I've been running my own user research consultancy for two years now. Before that, I was director of User Research at Liberty Mutual Insurance in Boston, I worked for twelve years building a user research practice in the company. I started as a team of one and when I left, I was managing a team of ten user researchers. I also started an internship program at the company and we were able to convert a number of interns into full time staff. I'm also president of UXPA Boston, and I'm an advisory board member for the Master of Science program and User Centered Design at Brandeis University. As I'm easing my way into retirement, I'm finding a lot of satisfaction in volunteer work.
Dan Berlin:Nice. Can you fill us in on your career trajectory, how you discovered UX and how you wound up where you are today?
Bob Thomas:I've had a long and varied career, I used to teach college English, English composition, and Intro to American Literature at various colleges in Western Massachusetts. I also taught at the maximum and minimum security prison in Enfield, Connecticut. This is back in the 1980s. An untenured instructor doesn't get paid very well and I still think that they don't get paid very well. I moved from teaching into technical writing with my English background and that's where I was introduced to the wonderful world of user experience. I used to document word processing systems that weren't easy to use, I used to document fonts and typeface designs, and I documented early speech recognition systems. In all these companies, the prevailing wisdom at the time was that users were too stupid to figure out how the software worked. So product managers, and engineers would say to me, 'Hey Bob, document how to use our software, but keep the manual to under 500 pages.' At the time I thought, there's got to be a better way than this. That's when my boss, at one of my companies, our documentation manager said, I just read this great book and she gave it to me as a gift. It was Don Norman's'The Design of Everyday Things', which is all about making utilitarian things like tea pots and toasters useful and delightful. The lesson for me was, people are often ready to blame themselves when they can't figure out how to use something- we hear all the time in our usability studies. It's not really the fault of the user, but the lack of guidance that should be present in the design that's the cause of the problem. I have kind of a funny story about my first usability test.
Dan Berlin:Tell us what do you got!
Bob Thomas:This is a true story. So the first usability test I ran was for people to complete what we thought was a simple task- find this typeface font and install it on this computer. Now remember, this is the early 1990s. So I started this study by asking my participant to turn on the computer and this is what she did. She picked up her mouse. She pointed it at the computer monitor, and she clicked.
Dan Berlin:Good start!
Bob Thomas:Yes! After that pilot session, we ran this study with a computer turned on. The lesson for me was- always know at what point you should start your research study.
Dan Berlin:Yes, indeed. It's a good lesson.
Bob Thomas:But it was really Don Norman's book 'The Design of Everyday Things' that turned things around for me and made me think, why am I documenting these systems with these large manuals? Shouldn't we be designing software that's doesn't need large manuals? My trajectory went into product management, then to Bentley College, and then to Liberty Mutual Insurance.
Dan Berlin:Gotcha. Thanks for that. Teaching in the very beginning there- is there anything applicable to UX that you bring from your teaching experience?
Bob Thomas:I had very different audiences so I would have to be very flexible and customize my classes for the audience. For example, at Westfield State College, instead of teaching essays for people giving people homework, having people report the class, I ran it like a writing workshop. I would give people assignments and give them feedback during the session, but basically, let them go and write, and then critique them afterwards in one-on-one sessions. I think that's a good way to think about running user research sessions when we run our usability studies.
Dan Berlin:Absolutely. The flexibility.
Bob Thomas:Right, exactly. Flexibility for one thing, but let people go- don't interrupt them. I think a lot of people when they start, especially this was true for me- I would always want to start and say too much. I find that these days, I don't really talk a lot.
Dan Berlin:Yep. Always tell folks when teaching UX research- just shut up. Let the participant talk.
Bob Thomas:Exactly. Don't give them any clues. For example, I remember one usability study I ran in Liberty Mutual, when I sort of mentioned I like the color blue. Then all I heard during that session was how much this person loved the color blue. You can't give anything away in a session. You have to sort of recede and hide yourself and your biases, and your likes and dislikes. You don't want to bias your participants.
Dan Berlin:Yeah. And the smallest thing can easily bias a participant. Thanks for that. Let's turn to your chapter here, bring 'Bring Rapid User Research Methods to Agile Teams'. Can you tell us about that, please?
Bob Thomas:I sure can. I chose this topic because collaboration is so important in our field. It's important that everyone- engineers, product managers, marketing people, executives, presidents, watch users fail or succeed. You have to be involved to know what's happening. You have to observe behaviors. You have to watch your customers using your products. Otherwise, you're relegating yourself to building customer experiences based on subjective opinion. I've noticed through the years how much people build things just based on subjective opinion. You can't do that, you need that data. When I started at Liberty Mutual I had a couple of goals that included increasing the visibility of UX research and establishing a research strategy. I would say philosophically, that strategy was simply understanding customer needs and practically I would describe it as- test early and often. In my first half of my career at Liberty Mutual, we did plenty of How did you get stakeholders to dedicate that time? qualitative usability research. We used Think aloud protocols. We recruited participants, we moderated sessions using task-based scenarios. Now remember, Liberty Mutual Insurance does property and casualty insurance, so car insurance, and home condo renter's insurance. So, I might do a task-based scenario like'Imagine you bought a new car and want to insure it, please show me what you would do'. Participants told us what worked for them and where their pain points were. Typically, these studies took 45 to 60 minutes, and we involved 10 to 12 participants. We, as a user research team, watched and listened to users, but our stakeholders never did. Liberty Mutual Insurance, like a lot of companies today, are meeting-driven cultures. Everybody has to be at meetings all the time and I would just hear all the time that we don't have time to sit for an hour and listen to users. This created problems for us. At that time, Liberty Mutual was waterfall style, which is, you hand everything off. Business systems analyst write the requirements, product managers vet this out with marketing, developers write something, they hand it off to QA. At the end, user research gets involved. The attitude was- user research and UX designers are there to pretty things up, which is totally wrong. It took a total of- in this waterfall environment- five weeks for us to run traditional usability studies. We had to spend a couple of weeks recruiting participants, preparing for the sessions like- writing the mod guide, etc. We would run the We had buy-in from Management. We had buy in from everyone. sessions over a week and then we take a couple of weeks to prepare a readout for our stakeholders who didn't watch any sessions, but would come to the readouts- the presentations that we gave. These were what Steve Krug would call 'big honking reports'- like 50 slides, people would bring their laptops in and they'd be multitasking. But when we showed video clips of users struggling, heads popped up, people paid attention, people listened. That was really great for us. But if you're working in Agile, where you're turning things around in two weeks sprints, five weeks to run a user research test was basically not going to work. So at Liberty Mutual, like a lot of companies, we moved from Waterfall to Agile, sort of a combination of Waterfall and Agile, and then finally Agile. About six years into my tenure at Liberty Mutual, our User Liberty Mutual is a big company. It's like 50 to 60,000 Experience Research team tried to incorporate new methods to employees. And our SBU had about 8000 people. Our SBU, which was meet the needs of our Agile teams. Our goal, as I mentioned earlier, was to collaborate. We wanted to help our Agile teams test design ideas rapidly, validate-invalidate them with really customer-focused as opposed to business-focused, ran real users sharing UX UX insights earlier in the process. So instead of a five week process, we could do a one week process or in the case of design thinking workshops, three weeks process. What was different was that we required our stakeholders to be involved from the get go. We would require our stakeholders to watch sessions. In the case of our design thinking workshops, we would bring in anywhere from 15 to 25 people depending on the project, they would be there for anywhere from three to five days. They wouldn't just be sitting there, we required them to be active users. So they would be observing the sessions, we gave them notes pages, we gave them a presentation on how to observe sessions, what to look for, how to be terse and practical- this is where my English background came into play. They would write their observations finally on a whole year long series of three day sessions on being post-it notes, usually just something brief, terse, one sentence, and put them up on these big white pads that we had for each participant. Nowadays, you know, during the pandemic, and I think going forward, we ran remote sessions, and we would use a whiteboard like Mural. People would write sessions, and we would run these sessions. So we would take these post-it notes, observations, after they were done, we would group them into categories across all the participants like- needs feedback, doesn't want to call or wants to talk to a human, whatever that might be. We would label each of those categories, we would vote for which categories were the most important and then we'd identify the highest priority category problems. During design thinking workshop, this might take a couple of weeks. But if we're doing a user research study, using standard usability study, we could do this in three days, we could recruit participants from panels, or previous participants, we would bring them in over one to two days, our stakeholders would be there in a separate room observing the sessions. And then they would be writing their observations on post- its putting them up on white pads. And then, on the third day, we would categorize all those observations, label them, figure out what the highest priority items were. On the fourth day, we would do a follow-up, find the solutions collaboration session, in which we identified and worked out fixes for the problems found in the usability sessions. So the key aspect of all this was involving our internal stakeholders, and we would get VPs in there, we would get the president of our SBU in there, we would get product managers from other projects in there. It was so successful that when people wanted to go to the old methods of saying- 'Here, Bob and team go in and fix these things and come back to us in five weeks- we had the authority to say no, we don't do the things that way anymore. Yeah, we have to do these things the new way'. Like I say, it's important to observe users to watch user's behaviors so that you know what the problems are right there. You can see the light bulbs go off. So it worked out. customer-centric. That involved listening to tech-support calls, actually going around and talking to people in our call centers- we actually went down to our Center in Florida, and did some work down there. We actually listened to our people quoting for customers, people doing tech support for current customers, etc. Getting that buy-in from management meant everything to us. They were on our side.
Dan Berlin:To dig deeper there, how did you get that buy-in? Were there steps that you took to achieve that?
Bob Thomas:We actually started at the lower level and worked ourselves up. We worked with the product managers that we typically worked with on all of our projects, who in turn, worked upwards, getting their bosses, there'd be VPs involved, and all the way up. The other thing that we did was, we did a lot of Lunch and Learns and a lot of quarterly sessions. At the end of each quarter, or the beginning of each quarter depending on how we were working, we would be invited to present and talk about what we had done and the processes that we had gone through. Again, I could see all the light bulbs going off, and people saying that this is working. We would also do monthly baseball card presentation. This was like a one-pager that would cite the top three projects that we had done, the stats on how we moved people. We increased the first page of our online quote, by I think it was 12% one quarter, and it kept going up. We showed them quantitative data along with our qualitative data. We might show them a short video clip. Just getting in front of executives and showing them what we had accomplished, along with that quantitative data where we move the needle, that really made an impact with our executives.
Dan Berlin:The quantitative, that'll always get the business buy-in, that's for sure.
Bob Thomas:I remember, the president of our SBU, I was introduced to him on the second month on the job, a product manager introduced me to him. The product manager told the President I did Usability. The President said to me, we need to be doing more of that and talking to our customers. At that point I knew this was the right company for me. This is back in 2008, I had never heard that from the leader of a company before. Never. It's important to have that executive buy-in. If you don't have that executive buy-in, then my advice to you is to work at trying to get it and if you still don't get it, look for another job. Because, you're never gonna make it in that company. You're just going to be hitting your head against the wall.
Dan Berlin:Fair enough. How did you go about that transition from Waterfall to Agile and enabling the team to do that efficiently?
Bob Thomas:That's a great question. Essentially, as we were working in Agile, we started meeting with the Agile teams. In waterfall, there are a lot of handoffs, as I mentioned earlier, with Agile we started meeting more with the Agile teams. In fact, by the end of my tenure at Liberty Mutual, we had members of our user research team embedded in different Agile teams. So instead of being more centrally-focused and just hiring all of ourselves out to one project, we would be involved in daily standups, we would be involved in grooming sessions and demo sessions, we would be involved in putting together user stories. That's how it started out. Typically in Agile you work in two week sprints, right? So how can you do a long sort of discovery research or a foundational research project in two weeks sprints? Well you can do it in terms of user stories, and basically you assign points to people, to your UX team members who are working on something. So a small project might get you one point and be half a day or a one day, medium, so be small, medium, large, extra large, extra extra large. When you get to extra extra extra large, it might be the whole length of a sprint, or the whole length of a PI, which is essentially three months. By breaking things down for people, we could get that foundational research done. In terms of our user stories, what we would do is work ahead of those Agile teams, ahead by one sprint. Doing the research ahead of it, and then being able to come back to the team and saying- this is what we found, this is what we need to do. Or, working at several sprints, where we do the research in one sprint, then work with the designers in the second sprint. And then the third sprint, having our developers implement that. It involves really working more closely. Two things, the executive buy-in is number one, and number two is embedding ourselves in the Agile teams, so decentralizing ourselves.
Dan Berlin:Great. Anything else about implementing user research in Agile teams that you were hoping to convey here today?
Bob Thomas:Not that I can think of off the top of my head, I think that the most important part is collaborating, working with people, and probably choosing your battles. Choosing your battles is really important. I've had members of my team who fought with product managers over everything. I remember talking to one- they are very passionate people, who want to be involved in everything. I sat this person down and I said, 'Look, you have to prioritize what's the most important thing'. Getting into a fight over the color of a button is not really a big thing to work on. What's more important is, for example, making sure that in our core process, we're not having people ask who's interviewing who. You've been talking, for example, on asking for personal information later in the process instead of earlier in the process. That's where our drop-off is. That's the fight you want to fight, not the button you press at the end. I think in working with the teams and working with Agile teams, choosing the battles, choosing the highest priority projects to work on, which I think is true of anyone, whether you're in UX or not.
Dan Berlin:Absolutely. Especially in UX, we tend to be passionate about advocating for the users. That can come at very granular points, like choosing the color of a button. It's a great point of picking your battles in terms of what UX opportunities to focus on. In our final moments here, we'd love getting a career tip. Either for folks either breaking into the into UX, or who have a lot of experience in UX, do you have a career tip for folks?
Bob Thomas:Well, I have a career tip for folks breaking into UX, and for people in UX, you have time for that. My first tip for people breaking into UX is make connections. There are two ways to do that. One is networking. In fact, I think it's so important I'll say it three times- network, network, network. Update your LinkedIn profile, make connections on LinkedIn. For anyone listening to this podcast, I am on LinkedIn as Bob Thomas, you can reach out to me, happy to connect with you. I connect with a lot of people on LinkedIn. I would also say attend a virtual meetups and join local and national UX organizations. Here in Boston, we have AIGA Boston, we have BostonCHI, we have IxDA Boston. We have of course, my favorite UXPA Boston. We have New Hampshire UXPA- joining and becoming involved in your local and even national UX organizations helps, even volunteer with your local UX organizations. Help out! You can also do pro-bono work with nonprofits and not-for-profits, include those in your portfolio, and attend conferences, meet people. Whether it's virtually or in-person, you know, at UXPA Boston, just a little plug, we run the UX fair every year. This is for new UX professionals, or career changers, primarily. We keep the price low, we run it every year, and it's very popular. The other thing I would say in terms of make connections is- seek out a mentor. I think the thing I'm most proud of in terms of my work with UXPA Boston is establishing with Jen McGinn, another 97 Things author is our mentoring sessions at the UXPA Boston's annual conference in the UX fair. I realized the other day that I've been doing that now for 10 years. We have mentoring sessions at these conferences, both our annual conference, which we just celebrated our 20th anniversary, and the UX fair. Making connections and seeking out a mentor can help you, and I do a lot of mentoring. If people want to reach out to me on LinkedIn, like I say, feel free, I'll be happy to mentor. For people who are in UX, I would say probably the most important thing is to have a point of view. It's important not to be quiet, but to be heard as a user experience professional. It's one way to build a reputation within an organization. We can only have that impact on an organization or a company by being honest and candid and having an opinion. I don't believe there's any such thing as being objective or impartial when it comes to telling a story. You want to guide your stakeholders, your executives to come to the same conclusions you do. So, have a perspective and don't be shy about expressing yourself. At the end of any project you're on, especially when it comes to presenting your point of view to executives, make sure they agree with you at the end. You need to guide them to actually actionable recommendations. You have the experience and you know how to advise them on what to do. That is my advice to people in the field- express yourself, have that point of view.
Dan Berlin:Great. Bob, thank you so much for the great conversation here today in terms of bringing research to Agile and your career tips. Thanks for joining me here today.
Bob Thomas:It's been my pleasure, Dan. Thank you.
Dan Berlin:My guest today has been Bob Thomas, the author of the chapter, 'Bring Rapid User Research Methods to Agile Teams'. You've been listening to the 97 UX Things podcast. Thanks for listening. You've been listening to the 97 UX Things podcast, companion to the book '97 Things Every UX Practitioner Should Know', published by O'Reilly and available at your local bookshop. All book royalties go to UX nonprofits as well any funds raised by this podcast. The theme music is 'Moisturize the situation' by 'Consider the source' and I'm your host and book editor Dan Berlin. Please remember to find the needs in your community and fill them with your best work. Thanks for listening.
