Observed Behavior is the Gold Standard (feat. Kaaren Hanson)

Kaaren Hanson, PhD discusses her book chapter "Observed Behavior is the Gold Standard" and offers insight on effectively building UX teams.

Sponsored by Watch City Research
Watch City Research is your trusted UX research partner

Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.

Transcript

Dan Berlin: 0:14

Hi everyone, and welcome to another episode of the 97 UX Things podcast. Dan Berlin here, your host and book editor. I'm joined this week by Kaaren Hanson, who wrote the chapter Observed Behavior is the Gold Standard. Welcome, Kaaren.

Kaaren Hanson: 0:27

Thank you. I'm so happy to be here. Dan.

Dan Berlin: 0:30

Thanks for joining the podcast. Can you please tell us a little bit about yourself.

Kaaren Hanson: 0:34

So I'm the Chief Design Officer for Consumer and Community Banking, also known as Chase, which is a part of JPMorgan Chase. And in that role, I get to lead a wonderful team of individuals who are trying to make sure that customers are able to get tremendous value in a very easy to use manner. What I would say is professionally I grew up at Intuit, where I spent about 12 years, and one of the things I'm most proud of is really helping change that company to become basically design driven or design lead. It took a lot of time and a lot of people. After that I spent some time at a startup. And I went to Meta, where I learned a lot about how the power of metrics in terms of aligning teams and helping people to truly achieve more than perhaps they had expected. And then I moved back into the personal finance space where I went to Wells Fargo before coming here to Chase.

Dan Berlin: 1:25

Nice. Thanks for telling us your journey. Can you tell us about the beginning of that journey? How did you discover UX?

Kaaren Hanson: 1:32

Yeah, so, one of my first jobs, I was actually working at a company that was a national jury project where we really worked on certain sides of cases. So I worked for victims of sexual discrimination and victims of sexual harassment, or racial harassment, or sexual orientation, harassment, or discrimination or whatnot. And when I was in that role, I really was doing some exhibit design, I was really understanding people and how they thought, and I realized the power of helping people to understand the bigger story and seeing the forest for the trees, and then also what it took to really drive that change. That wasn't actually the right industry for me, I actually found it both very stressful, but also very boring. And so at the time, there were a lot of startups in San Francisco that were hiring essentially anybody who had some knowledge of design and research. And so I hopped onto one of those startups, and I learned a ton from my peers. And then from then, I just kept moving to company to company taking on different roles, learning more, and eventually realized that one of the areas I am particularly effective at is really helping to champion design within an organization. And so that has been what I've really gravitated towards is how do I create conditions in which teams can thrive and do their best work? Not just design, but the whole community with whom they work those engineers, data scientists and product managers?

Dan Berlin: 2:54

Yep.Tell me a little bit more about that. What's your philosophy or strategy when you come in to build a great culture for a design team?

Kaaren Hanson: 3:02

Yeah, I think a lot about do we have the right people in the right places? Some people perform really well in one job, but not necessarily in another? Right? So within a company, there are certain roles that where you need to do different types of work. So do we have the right people in the right places? Is the broader team that quad is what we refer to at Chase? Is that quad setup around the customer's purpose, the customer problem they're trying to solve? Is there a clear metric for them to know if they've succeeded? Not from a business perspective, but from a customer's perspective that then ladders to that business outcome. And are they aligned on the quality of the experience they want to provide? And how are they going to measure that? And once you've got a team that's aligned on this is the problem we're solving, here's how we're going to measure success, and this is the quality of the experience we expect to create, you're in pretty darn good shape. But it's a matter of getting there and then making sure that the processes around that company are reinforcing that customer centricity, and reinforcing empowering the teams as opposed to getting in the way. Once you start to build that team, when you start to build the relationships, the alignment, then is about really tackling the broader company operating mechanisms or structures to ensure that they're reinforcing conditions in which teams can thrive.

Dan Berlin: 4:20

That's wonderful. I love how you describe the quad as being based from a customer perspective, because often enough, in enterprise, we see teams or however we structure our teams, based on business goals and the business way of looking at it. And it's heartening to hear you describe it as a customer focused way of forming your teams.

Kaaren Hanson: 4:42

And I do have a very deep belief that if you do the right thing by the customer, in most cases, you will do the right thing by the business. And part of our job as designers and product managers and data and technologists is to really make sure that we're solving the customer problem in a way that works for the business, because we do operate in a business. And there are many ways to solve that customer problem. So our job is solving well for the customer. And in a way that works for the business.

Dan Berlin: 5:11

I love it. That's what we do in UX. It's that trifecta of business goals, user needs, and the technological ability to support these two, the way you just described that was wonderful, thank you for that.

Kaaren Hanson: 5:22

And what I think is really important is sometimes where I see people and UX getting stuck is because they think of it as customer or business. It's not, it's never customer or business. It's customer and business.

Dan Berlin: 5:33

100%. Thanks for all of that. Very interesting, it's always nice to hear how folks are looking at forming their teams from a high level. Let's turn to your chapter, Observe Behavior is the Gold Standard. Tell us about that, please.

Kaaren Hanson: 5:48

Yeah. So, I think that it can be very seductive to listen to what people say. And we're very used to having conversations with people. And we tend to believe what they tell us because most people are fairly honest. Now, the downside is, we also tend to believe what they tell us about how they will act in the future. And so it can be very easy for us to take what people say, whether verbally or in writing, and assume that's true. And when we do that, we are often wrong. We're not always wrong, but we're often wrong. And the problem is, it's hard to predict when are you going to be wrong? And when are you not going to be wrong when you simply listen to what people say, right? And I'm going to give you a couple of examples from Intuit. And then I also have a couple examples from Chase. But in one case, we were working on a payroll product. And there was a question of, hey, when people come and they're just trying to pay a new individual on their payroll, do they want to pay a person or do they want to actually set up the payroll, and then pay a person. And when we ask people, some enormous majority, like maybe 90% of people said, I would want to set up my payroll right away. Now, fortunately, the team didn't necessarily believe what they said, because often what we say and do are different. So they set up a painted doors test where they had people on the site, you could either go pay someone right away, or you could save your payroll, what we found was the majority of customers went to pay an employee right away, because it was a time of need. Because of that, the team was able to get the funding they needed in order to build a solution that solved this customer problem of needing to pay my employee. And they were able to get that funding because they had that behavioral evidence. And they didn't just listen to what people say. Very similarly, when I was at into Intuit, another product, we were looking at the experience of it, and we were talking to our customers, after they did some tasks about what their experience was, they were saying things like, I love it, it's so easy, it's amazing. And then we looked at what they had actually done. And what we realized is they had made so darn many errors, there are going to be in hell with their accountant in about, you know, maybe three to four months, right. So again, if we had simply listened to what they had said, we would have thought this company we bought had a great product. But because we also observed what they did and had very clear metrics for success, we knew that it wasn't actually that easy for them to be accurate in what they were trying to do.

Dan Berlin: 8:18

Right. And accuracy in the finance world is critical.

Kaaren Hanson: 8:23

Yeah.

Dan Berlin: 8:23

Tell me about the painted door. You mentioned doing a painted door research to figure out what they were actually doing. Can you expand on that, please?

Kaaren Hanson: 8:31

Yeah, well, essentially, what was done at the time is there was simply two options on this particular page that you would land on, right. And this was actually on the web is before the app was strong as it is now. And you could either set up my payroll, or you could pay an employee. And it turns out, majority of people want to pay an employee, because what had happened is maybe they had just hired somebody, maybe they had forgotten that they even needed to set up a payroll maybe was their first employee, they didn't know what was happening. And all of a sudden, it was Friday, and they had to pay somebody, and they didn't have time to find their EIN, and you know, other documentation, they needed to figure out all of the information that had to be had in order set this up correctly, they just needed to get Joe a paycheck, right. And so they did that. And then later on, they would come back and set it up. So let me give you another example of the difference between what people say and what they do at Chase. So we recently conducted some research on behaviors and needs of consumers. And we were really looking at what people said initially, and we asked them to rate the importance of like personalized shopping versus other drivers of customer value like price or return policy, etc, etc, etc. And so when we had this conversation, it turns out when people talk, personalization is ranked very low, likely because it's a fairly abstract concept, and it's hard to imagine what personalization might mean, right? Or everybody might be imagining it means something different in their head. And so to get around that, then what we really looked at was how to create examples of personalization live, that would be meaningful to individuals. And by doing that, we start to have very different conversation. Now, the reason I bring this up as an example is because it goes to show that there are limitations sometimes, and how people understand words and understand questions, which is another weakness of simply listening to what people say, as opposed to really going further, and helping them more deeply understand whether by showing them something different, or having them go after doing a task themselves.

Dan Berlin: 10:44

Yep, getting more of that reaction than just their their words.

Kaaren Hanson: 10:48

Yeah, then just like what they're imagining your thinking and what they're imagining their future self is going to do, right, that's a lot of imagination, which may be accurate, or may not be accurate.

Dan Berlin: 11:00

Right. So how can we get at these behaviors? What are some of the ways that you UXers should be trying to get at these

Kaaren Hanson: 11:08

Yeah, I mean, I think instrumentation and behaviors? knowing what are the core flows, the core behaviors are looking for, and doing the instrumentation for something that is live, I think a lot of it is the anthropology and going into people's houses and observing what they do today. Because what they do today is probably what they're going to do tomorrow. Again, if you go back to the Intuit days, and TurboTax, we would go into people's houses, and we would see all the different places, they kept their tax forms, right, and you know, all the different materials they had, and then somebody on the frigerator. And so would be like, you know, practically in the garbage can pull it all together. And they would procrastinate then and now that everything's electronic, well, let's not assume that everybody has their act together. It can be just as complicated and messy and as much procrastination has ever said the best day and actually, there's a great quote, I can't remember who made it. But the best predictor of future behavior is past behavior, even if the modalities and the technology is different.

Dan Berlin: 12:06

Yeah. And what about doing it in an artificial setting versus in someone's home? Right, bringing someone to someone into a lab or doing a remote usability study versus going to their place? You know, what are some advantages or disadvantages there?

Kaaren Hanson: 12:23

Yeah, I mean, I'm a fan of all of that, I think about it as a portfolio view of research methods, every research method has a benefit and a drawback, just like your stock has a benefit, your bond has a benefit, and they both have drawbacks you want both in your portfolio. And so what I would say is true about the usability studies, whether in person or remote is you are able to look at behavior, if you set it up to look at behavior, as long as you're making sure is, what they gave you accurate was the payroll they made did have the right taxes taken out, so they won't be caught in the end, right? It's very easy to do these studies in a more rigorous manner, which is probably going to be more valuable to you or a less rigorous manner. Now, if the goal is simply to have everybody have some experience of seeing what the customer goes through, as they're trying to attempt something, fine, don't worry about being rigorous, because then the goal is really to make sure everybody has a customer voice in their head or has seen the customer and is aligned around that problem, and aligned around the quality of the experience. But if you want to get deeper, be more rigorous, and be very clear on what success looks like and does not.

Dan Berlin: 13:29

Tell me a little bit more about that please. How did you define rigor in the way you're describing it here? What's the difference between non rigorous and rigorous.

Kaaren Hanson: 13:37

So I'll give you an example of a consulting company we worked with at one point, and they were they were fine. We just coached them to what we needed. But initially, we were going to do some benchmarking on a particular set of products and their competitors. And this was at a previous company. And so initially, what they had set up was, you know, having people do things and then asking questions, like were you successful and how easy was it and you have the usual questions. They didn't actually have any behavioral evidence. And so it was really pushing them back and saying, Wait, now we're in this flow, are we going to say that they're successful? What page do they need to get to? What is the screenshot that they have to have and send in or what is the screenshot we have to see, for us to know that this was a win, what needs to be on that screenshot? And so getting really rigorous about what it is gave us very different data results. But that wasn't necessarily how that particular set of practitioners was thinking to begin with. Again, going back to what are we going to do tomorrow is probably what we did yesterday. They were coming at it with the same protocol they'd used in many other places.

Dan Berlin: 14:43

Gotcha. So was there anything else you were hoping to convey here today?

Kaaren Hanson: 14:48

I guess I would just say that there's nothing wrong with talking to people. I don't want anyone to get the wrong idea, right. Talking to people is incredibly useful and can help you understand the context they're in and it can help you understand what motivates them. And some struggles they might have had or some dreams and wishes they have. But don't assume that talking is enough.

Dan Berlin: 15:10

Well put, don't assume that talking is enough. And this comes back to be mixed methods and always triangulating on your users using mixed methods, whether it's a survey and interviews or usability and interviews or however you're doing your triangulation. It's so important to get that that quant and qual almost.

Kaaren Hanson: 15:30

It is and it's also okay if your goal is simply to make sure the quad like the product managers and the technologists and the data scientists and the designers and whatnot. If your goal is to make sure everybody understands who this person is, talking may be fine. But if you're trying to make decisions around what products to go after, is their product market fit. You know, is this a good enough experience that they're able to be successful and confident? Probably talking will not be enough in many cases.

Dan Berlin: 16:00

Well Kaaren, you gave us a lot of great information here today. Thank you so much for joining the podcast.

Kaaren Hanson: 16:05

Oh, thank you so much. It was a pleasure.

Dan Berlin: 16:07

My guest today has been Kaaren Hanson, who wrote the chapter Observed Behavior is the Gold Standard. Thanks for listening, everyone. You've been listening to the 97 UX Things podcast, a companion to the book 97 Things Every UX Practitioner Should Know published by O'Reilly and available at your local bookshop. All book royalties go to UX nonprofits as well any funds raised by this podcast. The theme music is Moisturize the Situation by Consider the Source and I'm your host and book editor Dan Berlin. Please remember to find the needs in your community and fill them with your best work. Thanks for listening