Caitria: 00:00

We're really at the heart of our work. We're we're storytellers. Mhmm. It's our job to educate people. And if you think of any set of tricks that, like, a high school teacher or a storyteller would use, we have those same ones available to us as UX researchers.

Erin: 00:16

This is Erin May.

John-Henry: 00:18

I'm John Henry Forster, and this is awkward silence. Silences.

Erin: 00:32

Hello, everybody, and welcome back to Awkward Silences. We are here today with Ketra O'Neill. She's a UX researcher, formerly of Airbnb and heading to Google in just a couple weeks here. So we caught her caught her in between gigs, and we're here to talk about a really important topic. Could seem boring, but it's not.

Erin: 00:56

It's critical to being successful, and Cater's gonna tell us how to take something that could be really boring and make it not boring so that, you can be really impactful. And that thing is research reports and research findings and, you know, how to put them together and to share them with your teams to be very successful and impactful in your research work. So thanks so much for joining us to school us on all things research reporting, Kedra.

Caitria: 01:25

Thanks so much for having me.

Erin: 01:27

JH is here too.

John-Henry: 01:28

I am. I feel like the pressure's on to make this episode not boring given the topic, so we'll do our best.

Erin: 01:33

It's gonna be really, really exciting. So let's get started. Let's talk about what makes a good research report.

Caitria: 01:45

All right, so I first want to kick this off with what makes a bad research report, to give us a little bit of perspective. I bet everybody has experienced finding some amazing, you know, nugget of information in a report that was made, you know, years ago after you joined the organization or finding that, you know, something that you just discovered isn't that new. And the problem is often that the good information is buried in a bad report. And it might not even be that it was poorly framed or wasn't explanatory enough. It might just be that it didn't, it wasn't exciting enough for people to read or the way that the person brought it to the people, their stakeholders didn't encourage them to interact with it.

Caitria: 02:27

So the things that make a good research report are making it big, the big three for me are enjoyable, informative and actionable. And I can double tap on each of those if you guys are interested.

Erin: 02:40

Yeah, yeah, please.

Caitria: 02:42

Okay, so with enjoyable, I put that first, it's not the most important thing, but this is one of the biggest ways to make impact. It's to get people to interact with your research and making enjoyable means making it the kind of thing that people want to read. If you write a report that people don't want to read and don't read, it doesn't have any impact, even if it had the best possible insights in it. So make it enjoyable for me means using things like themes, memes on occasion, jokes, emojis. Basically you're trying to come up with some kind of narrative tool that will help the reader, first of all, engaged in the topic and then pull all the way through all of those possibly dry details.

Caitria: 03:27

So for some things that I find on in companies like Facebook, they were always coming up with crazy code names for things. And one of those names internally was Balto. And that turned into a wolves in space theme at some point on the I just went with it. I, every single report that I put out there had a wolf somewhere in space. I had to create some of the images on my own at some point, they looked terrible, but it got people pretty excited and you'd get people reading the reports and referring to them as, oh, know, the one with the wolf and the sunglasses.

Caitria: 04:01

I had another set of reports where I was working on date flexibility at Airbnb. And this wasn't even an image. I just titled the emails to all of my stakeholders about the results as weird flex, but okay. Mean, it summed up the whole thing. Was like, people were like, it was usable, but they were like, it kind of messed with their mental model.

Caitria: 04:22

And people in meetings that I wasn't even attending were referring it to it as the weird flex. Nice. So it's helping, it's using just any kind of memory trick. We're really at the heart of our work. We're storytellers.

Caitria: 04:34

It's our job to educate people. And if you think of any set of tricks that like a high school teacher or a storyteller would use, we have those same ones available to us as UX researchers. And it's a big part of our work because again, I've seen amazing findings crumble into dust in the wikis just because no one got excited or no one kind of knew what thread to pull to find them.

John-Henry: 04:57

Does the delivery channel matter at all? Like, does it matter if it's like a slide deck or a written doc or a Wiki or like, or can you do all these things in any of those formats?

Caitria: 05:07

That's an excellent question. I think it's a little bit easier to use memes and kind of images and stuff like that when you're using a deck. And that's a great thing for both presentation and then also to kind of like keep the findings in and share with people in an evergreen way. But for a written report, it might need to be cleaned up a little bit. There's not as much space in the borders and the margins for emojis and stuff.

Caitria: 05:31

So, do find written reports are a little bit drier. And then it's really, when you're thinking of the storytelling tool, then it's the narrative arc that's really important. It's those bullet points that frame the whole thing. What are the top three sections? That's the storytelling tool.

Caitria: 05:47

So like, if you're doing a report on, you know, the quality of the app you could make it fat, slow and big or something like that. And those, you know, that's still a storytelling device. You're really emotionally affecting people with those three section titles. I wouldn't suggest using them that bluntly unless you have a really good relationship with your team but you can still use those same kind of like interesting provocative tools no matter the type of tool you're using. The other parts that make a really good research report, so the informative and actionable side.

Caitria: 06:23

Informative means that it needs to teach the team valuable information. So that means a lot of times I see reports come out especially from researchers who are maybe dabbling for the first time in product research. And the goal was figure out, does this button work? Or does this work for the user? And the report comes back and the report says no, but that doesn't give anyone more information on what to do about it.

Caitria: 06:50

What didn't work? What about it didn't work? If it was changed in X, Y, or Z way, what would that mean for this whole design altogether? So making it a report informative means not just saying the what, so that's like, you know, four of X users had trouble finding this, but the Y. And so that digs into things like their reactions.

Caitria: 07:14

They just didn't see it. Their mental models, it looks like this other thing that so that's why all of these people thought it behaved in a different way. You can really start teaching someone about what about the design was not sufficient. And you can also start when you look into mental models, you can start doing mini audits like showing people, oh, if everyone thought it did work this way or should work a different way. Here's how it works on these other apps.

Caitria: 07:42

And that's what informs them. So basically your goal is to have the designer come away, not saying like I did it or I didn't do it. Now I feel bad but rather being a better designer because of all the information you gave them. And then the last bit is actionable. So, you're really targeting this stuff.

Caitria: 07:59

You can write an amazing report about like all the ways that the button was invisible to people. But again, you need people to act on it in the short term. So, you're really trying to inform your stakeholders who should do what and also what you will do next. So, on the actionable side, you need to put in clear near term recommendations. And that's sometimes hard for a researcher.

Caitria: 08:22

We're supposed to be very objective but that does not mean that you can't suggest the next steps in the process. So if the button was invisible, no one could find it. I don't have to tell the designer or the product manager how to make it visible. I can give some suggestions if I'm feeling very confident or if I did an audit, but I do get to say things like, I'm going to test the next version of this and the button needs to be findable. Or I can say things like, these are the top four things in order that make this design good or bad.

Caitria: 08:55

Button has to be findable, information has to be usable, whatever the product is I'm working on. But once it has to be a very clear recommendation and that push to make sure that people actually come back and act on it, even if they enjoy the whole deck and your memes doesn't mean anything if they don't walk away with clear next steps to work with it.

Erin: 09:18

Right, so which of these is most important? And you can't say all three.

John-Henry: 09:26

It depends.

Erin: 09:27

Yeah, depends is the other popular answer, yeah.

Caitria: 09:30

Yeah, I think, well, if I'm not allowed to say it depends. Would be it because I think it depends largely on the actions that you need people to take. So for example, if you're just like having people understand the value of UX research and your value as a researcher and understand the user and some of that stuff, enjoyable is a big deal. You're getting like an engineer who's never touched research before excited. On the other hand, like actionable, if it's like you're up against conversion requirements from your company, and it's like the end of the year and someone's gonna get fired, you know, it's much more important to go into actionable details.

John-Henry: 10:10

Yeah.

Caitria: 10:11

But I think overall to answer the question, informative is the most important thing you can do as a researcher. So teaching valuable information. And I think that there, I love to put out there like an important distinction in user research that is helpful to bring into your decks and to the other types of insights that you share. It's cutting things into first that short term product focused research, like did it work? Did it not?

Caitria: 10:39

Here's what worked, here's what didn't. And here are the next steps for the products that we're building. Separating that kind of information from evergreen learnings. So in any kind of research session, for example, if I'm doing a one hour product focused, usability focused, task based session at, let's say Facebook, and I start off the session with some questions, I usually ask things like what types of things are important for people to find on Facebook? How do they search for things?

Caitria: 11:11

These are warmup questions that I asked across hundreds of participants over the years. So you're getting over time a lot of foundational information even from that small usability focused study that you can eventually start to share. That kind of information, if the product succeeds or if the product fails, those are still important findings. Whatever can be stripped away from that individual effort, that individual guests that the designers and the product team have come up with and teach, that's really important. So usually when I'm doing a deck, most decks that I do, I'll separate out the information that's evergreen.

Caitria: 11:47

So anything about the user's behaviors, motivations, barriers to changing, current method of solving a problem. And then also I'll dig into their mental models in there. They expect, why, what are their current app uses, the barriers and differentiation between those scenarios where you might want to use one or another, that kind of stuff. I wanna be able to strip off the end of any deck and combine those together at the end and say, this is what you need to know about the user to design a blue sky version. And then, the rest of it is very much focused sharing screenshots, showing the exact state and time that that product was at and then the information about how the user reacted.

John-Henry: 12:32

So I wanna come back to the evergreen thing, but just a quick one. Since you mentioned informative being one of the most important or the most important. As I was thinking about this and how I might like deploy it myself, I was gonna write a report, informative to me also seems like kind of the hardest or the most at odds with the enjoyable aspect. Because like the first thing that comes to mind for me is informative. It's like, okay, just like throw everything in.

John-Henry: 12:53

And then you get this really dense, hard to navigate report, which is probably boring. So like how do you do informative in a way that doesn't hurt the enjoyability?

Caitria: 13:02

So I think that as a UX researcher, can take the position of being a product leader by helping define and redefine and share in your decks, the brief of what the product ideally should do. So that means, you know, if I'm looking at the product and it does X, Y, and Z, in my brief, can say, or sorry, in the deck, I can say that this is the most important thing that this needs to do. The user needs to be able to find the booking button. This is the second most important thing. The user has to feel engaged.

Caitria: 13:36

But by ordering those two, I can make take a position and say that it's more important that this is usable than that it is engaging. So I can basically, by taking a firm point of view on what the product should do, I can start to winnow down the amount of information that I should be sharing. And it's really, I don't think I've worded that very well, but it's a

Erin: 14:01

No, no, have. Yeah, please. No, I think it's really interesting because you said something interesting also before, right? Which was that researchers are supposed to be objective, right? But that doesn't mean that you don't come out of a study without actionable stuff, whatever you want to call that, opinions or whatever, right?

Erin: 14:22

And so you're talking about that a little bit again in the context of part of how you make your very informative deck approachable and not just gobs of information overload is taking a little bit of a stand and a point of view. How do you thread that needle, right? That between the coming to a point of view into an opinion and shaping even the product vision, but also this objectivity. How does that all come together?

Caitria: 14:53

It's a back flip for sure. But I think it's similar to the ones that we execute when trying to give really objective information without hurting the designer or the product team's feelings, right? So, there was a POV and we're saying it did or didn't match up to it. So, it's almost like working with your product team to understand their POV and why they think this is the most important thing for the product or that. And then also to educate them about what the user's needs are compared with those other product needs, for example.

Caitria: 15:26

So, part one is having a dialogue with your team. So, you're not just coming in at the end of the process saying, here's what we actually are trying to do. These are words they should have heard in meetings and stuff like that or your other briefs. Two, you can start building it into other parts of the product process before your deck comes out, before your insights come out. So that might look like writing in a set of user needs or user goals into or just even user statements into an experiment plan or a product description, working with the team for that.

Caitria: 15:58

And then you can point back at it and say like, hey, these are things that it did or didn't do. And then at the end, it's a little bit more about framing. So I'll use an example from Airbnb. Airbnb has several tiers of different types of merchandise now. So that means they have the luxe version, which is really expensive, like $5,000 a night, there's a butler included.

Caitria: 16:16

And then they have, you know, a one bedroom with a cat lady. And, you know, you're sleeping on her couch in a spare room that her child left many years before. There's very different levels. And what they were, what I was trying to do with the team when I just joined was figure out what does a luxury accommodation page look like? You know, it can't look like the one that we already have.

Caitria: 16:39

Can't look like the cat lady's home, you know, but what is luxury? And it's extremely hard for people to, for us to find people in the over $800,000 per year income range to talk to. So this is, you can't just do easy research on this either. So the team had kind of created a perspective that they had a point of view, a very strong point of view that showing things like the amenities in photo form would be luxurious feeling. So showing a picture of a Keurig instead of listing like coffee service or something like that in text.

Caitria: 17:15

And by working with them beforehand, I got to realize the goal was not to put photos on the page, even though that's how the product was like framed. They were like, we need these photos. I was like, why? To feel luxurious. Okay, that I can work with.

Caitria: 17:29

So we took three different versions of two different photo design directions and a list version out to very affluent guests and have them just free form describe and react and compare in a variety of different ways these three approaches. And we had people end up saying things like, oh, this looks like clip art or basic, or just the exact opposite of the team's stated goal. And we found out the list was much more effective for feeling, especially with using header titles in a list. You could do things like, oh, outdoor amenities, and that felt very fancy. So it was like, it was not what we expected, but we, and it because it was mixing a hundred, it was like a $200,000 photo shoot or something like that.

Erin: 18:19

At the

Caitria: 18:19

last minute, this was not well timed. And I was doing it on a new team. I was trying to be sensitive. So I was really trying to speak the language of what is our goal. So a whole slide about this is what we're trying to do.

Caitria: 18:33

And then a slide about what our hypothesis was that photos might do this. And then the slide that actually had the user's voice, basic clip art, word art, there were a lot of great choice quotes in there. And then it doesn't feel so much like you as a new person, as an untrusted product visionary or whatever. You're not coming in there with your own opinion. You're just saying we achieved or did not achieve this team goal on this particular thing.

Caitria: 18:58

And then, suggestions on things to do moving forward. So in this case, the photos were not good for showing amenities, but they were very helpful for teaching people about services that they didn't know. So, for example, if you're gonna book like a once in a lifetime villa, you might never have experienced a butler or something like that or transportation services. So, having photos that described those services was actually helpful. And it was on goal because we needed people to understand these to value them.

Caitria: 19:29

So, it about the photos, it was about how they were applied and it's about teaching the team where to use that tool.

Erin: 19:40

Right, yeah, the details matter there and Airbnb is famous for the photos being so important, right? In early part of the experience. So, the surprising insights are always the most satisfying. So thanks for sharing that one.

John-Henry: 19:54

Alright. A quick awkward interruption here. It's fun to talk about user research, but you know what's really fun is doing user research, and we wanna help you with that.

Erin: 20:02

We wanna help you so much that we have created a special place. It's called userinterviews.com/awkward for you to get your first three participants free.

John-Henry: 20:14

We all know we should be talking to users more, so we've went ahead and removed as many barriers as possible. It's gonna be easy. It's gonna be quick. You're gonna love it. So get over there and check it out.

Erin: 20:23

And then when you're done with that, go on over to your favorite podcasting app and leave us a review, please.

Caitria: 20:34

Yep.

John-Henry: 20:34

Could we do a quick tangent back to the evergreen insights you mentioned? Like, do you do you actually have or do you tend to build out reports for those? And do you employ the same sort of tricks of making them, I don't wanna call them tricks, but like same tactics, I guess that's not even a better word. The same approach of making them enjoyable and informative and stuff. And is it just like grow forever or like what does that actually look like?

John-Henry: 20:55

Because it seems really interesting.

Caitria: 20:57

I think that's my method of making my decks separating them into evergreen and then more actionable product insights helps me do something else that is in my toolbox which is research reviews. Whenever I join a team or whenever I start working on a new product, I have to learn a lot immediately. You know, I have to read through all of the research that's ever happened and, you know, look at the dashboards that exist for this and interview people about the product attempts that have happened in the past, look at experiment results, things like that. So I'm already doing that human computer thing and crunching a whole bunch of information. And for me, I also realized that teams are not, teams are not Teams change frequently in tech.

Caitria: 21:45

So you're gonna have people who are joining and having that same experience of needing to get up to speed and know all of this information and really understand that evergreen stuff as well as you. So I find it's a great opportunity whenever I'm getting started with a topic or a team to create a research review that briefly summarizes what we know about a topic. So I'll use an example from, let's say, yeah, let's say Airbnb. So for example, the team had just put in an auto complete system for Airbnb. They hadn't had an auto complete or an auto suggest system for a while.

Caitria: 22:26

I knew that we were gonna be starting to do some experimentation moving forward in the future. And we didn't have a lot of information about, collected about what auto complete should do ideally or query understanding, how the search system works. So as I was learning about it and starting to work with that team, I just started collecting everything into two big research reports around how people search on Airbnb, mental models, the types of terms, their navigation through the site, where they land, the types of things they're looking for, the ways that they make decisions when they land on search results, all those sorts of things. And then another one around auto complete and the types of queries people enter, you know, the types of things that they click on when they're shown a set of results, all of that kind of information. There was some research already done on it.

Caitria: 23:20

So, you can pull out and then cite basic principles. And then you can also identify when you're doing this process, what information we still don't know. So, if you're approaching a topic and you're like, okay, well, don't know any of these things. So, I have to find the reports that tell me this, or if they don't exist, I have to do that. It's a good way to go through and kind of like identify what the next big need might be for research as well.

Caitria: 23:45

And then when I'm creating these evergreen reports, I find that I just chop those sections off and then add them on. And so, with a research review, you shouldn't have all of the information or like every slide, you should have the point of all of that research as one slide and then have a double click into everything else. Basically, if you can give people a way to self-service this stuff, you do a lot less time in one on ones and explaining stuff to people.

Erin: 24:13

So it does sound like the evergreen can, know, maybe this is I don't what do they call it evergreen? Because they're ever okay. Get it. It grows you around, but like it's also like a it's a forest, right? It's massive.

Erin: 24:23

I mean, like, you know, you're like there for a year and you, you know, the you put the slides together and now we have a hundred slides, but we just got to the main point. So we're back down to 10 slides. But how do you manage this over time? And obviously this is a whole other topic in terms of insight management, but how do you right. Yeah.

Erin: 24:42

Like how does it scale and does the the validity of the quote unquote evergreen information have an expiration date? Are you pruning the forest over time? Let's go with this metaphor as much as possible. But yeah, right. Like how do you keep it?

Erin: 25:00

And this is related, I think to the topic of just sort of good reports in general. How do you keep this growing ecosystem forest of information manageable as it expands?

Caitria: 25:15

Wonderful question. So I'm gonna respond kind of with an example from a company at scale. So at Facebook, for example, they've been doing research on one of their goals is getting to the next billion users. And so therefore they've been doing research on India as a market forever. And every team has been doing research on the Indian market.

Caitria: 25:40

So for example, the teams that are doing app performance and app size, was a huge concern a couple of years ago that the app was so heavy. They had to create a light app. People are also looking into the cultural implications of Facebook or WhatsApp in India, or looking into the safety issues. For example, like many people might use one email for accounts. So there was just an incredible amount of information pouring out of every team at Facebook.

Caitria: 26:10

And when I was there, they had finally reached the point where the research reviews just, it wasn't effective anymore. But the best next step that they had gotten to before I left was having rolling research reviews committed to and created by a set of teams. So it was almost, I think it was like either quarterly or by half, they were looking, they were going through that research review that kind of document that should teach you all of the stuff you need to onboard about this topic. They were going through at least every half year and adding new information and changing it out. That's the same kind of process I've seen at smaller scales as well.

Caitria: 26:49

So for example, when I joined the search team at Airbnb, there was a document with like the biggest issues in search, the most common issues and some stuff about mental models, but we had already solved some of those issues. So it was time for another update that came up with the biggest issues, re ranked them and then still kept those important look at mental models as well. So I think it's a great question because I don't think the work is ever done. That's where another kind of communication step becomes really effective, which is the creation of self-service places to put this information where you can get into versioning and stuff like that. So, creating wikis, websites, indexes, where you list all the types of research done on one topic.

Caitria: 27:36

That kind of stuff is super effective because you can send people there instead

Erin: 27:39

of Yes please.

Caitria: 27:40

Slacking with them or sending the message individually to answer their questions.

John-Henry: 27:45

Yeah, there's like a, if a research report falls in a evergreen forest and nobody reads it, does it make an impact or something, Like, cool. The question I had actually is kind of a continuation of that a little bit. Like when you're putting together a set of slides to walk people through what was learned and everything, Are you doing it from the perspective of like making the slides in a way that you're planning to present these like live in front of people and be there to annotate it like with a voiceover? Or are you trying to like write them in a way that is standalone and people can digest on their own time and take it in independently?

Caitria: 28:19

I try to make sure that the assets that I'm creating with insights and findings are usable both for presentation and for sharing out. So I do a couple of things. One, I try to make sure that each of the slides are self explanatory. So linking out to other stuff, making sure that there's arrows, current screenshots, all of that. But two, I try to make sure that they are readable from the back of the room if I am going to present it in person.

Caitria: 28:43

So I do try to make sure that each of my slides are constrained to two or three different font sizes, a big one and then nothing smaller than about 30, if I can help it in Keynote. And then I also try and make sure that the screenshots that I'm showing, I basically, when you're presenting to people, you really need to ground them in the visible. When you're working with UI, you have to ground them in the visible issue that you're talking about to make it really hit. So I space my information really widely across these slides, show a big close-up of what I'm talking about, an arrow pointing right at it, some audits, the next slide might be a whole bunch of ways that other sites do it, actual screenshots. I try to stay super visual.

Erin: 29:29

To include a video or images of your participants at all or?

Caitria: 29:34

Yes, but only where it actually will make a difference. I find that people don't click on those play button symbols in decks very often. And so if I'm gonna give a presentation and I think it will be a nice effect, might include it, but rarely otherwise. The notable difference is when those videos make are like extremely important to the conclusion. So for example, I was testing different search input interfaces.

Caitria: 29:59

So those are the place where you'd either type or input your filters for Airbnb. And I noticed that because of the way the site was loading, we were having people pause, have another moment of cognitive load, then think, then take an action. And because cognitive load was so important to the way that this particular design stacked up against another one, it was really helpful to have the person kind of like go, with their finger hovering over it and then eventually take the action. And then I could show that side by side with a video of the alternative where the person was very quickly able to identify what they need and move on. But I think for presentations, especially I love to have that kind of stuff.

Caitria: 30:40

It kind of depends on personal preference, but when you're sharing it like not the team level. So I'll probably do everyone up to like, you know, director at the company level with like really fun stuff, really, you know, irreverent language. But once you get up to sharing at like maybe a C level of team meeting or something like that or a full company event, I do recommend polishing up and kind of formalizing some of the tone and really focusing even less on the data, except for where it supports that overall narrative. The narrative, the story that someone takes away, the urgency behind it is the big thing at that kind of meta level. And then as you drill in and then you're sitting down one on one with the engineer, that's where all the detail can come out.

Erin: 31:26

See, that's, I mean, that's what counterintuitive to me. You think like the C suite, right? Give me all the charts, give me the data. You're saying the takeaway is the most important thing. How do you convince them the takeaway is right?

Erin: 31:42

Or do they need to be convinced at that level? What's the goal?

Caitria: 31:45

I think that'll really depend on the circumstances. The example I'm thinking of right now, I was engaged at Airbnb in a 2020 planning efforts where we went through a small group of researchers went through a lot of the information and data from across the whole company and tried to figure out what are the big blind spots for 2020. What does the company need to focus on or other people will eat their lunch in 2020, that kind of thing. And for that, there's so much information. I think we came up with a set of seven different meta themes and then combined that into a group of five that we dug deeper into for several months with more research.

Caitria: 32:26

And there's so much data, there's so many charts at that point that it's overwhelming, people shut down. So, we eventually, through working with different really talented storytellers across the company, which I definitely recommend as a researcher. We are not the best storytellers, even though it's important to us. But it's an important tool rather. By working with really talented storytellers across the company, we were able to kind of break it all down into a two part message that really spoke to people.

Caitria: 32:54

It was instead of saying like, my first draft might've been the app is fat, it is slow, and it's too big to download. According to like everyone and all of our competitors are doing better. So, you know, I'm already getting into some of the data and stuff. But you could wrap that up in a much more positive way and say like, we could be a better, you know, host or something. Something.

Caitria: 33:14

Like, we could be better to the user. And that kind of message, you don't have to have every single bullet point of what was wrong to take that on as like the mindset of what we need to do better. So, the whole company doesn't need to know the exact app size. The whole company doesn't need to know the exact rate of bugs or a big issue between competitors, but they do need to know that the big focus is treating the user in this particular way moving forward.

John-Henry: 33:44

Cool. I have a quick two parter. One is how do you make the time for it? I feel like at least for me, whenever I finish research you're always just like ready to move on. And so like wrapping up the report sometimes can be hard to actually spend time on.

John-Henry: 33:59

And then the second one is like, how much time do you make for it? Like between like maybe like three categories of upfront alignment with the team, actually like scheduling and facilitating all the sessions and then like wrapping up a report. Like what's like the breakdown of your time spent across those buckets?

Caitria: 34:14

Awesome. I think I move pretty fast on this stuff but it's only because I use a ton of tricks and just speed methods to make sure that I don't have to spend a ton of time on each. I think if I was sitting down and like really giving it my all, any of these stages could take a ton of time. So, in terms of upfront, I try to make sure that I am engaged before any research is done with product planning. So it does take a little bit of time, but that's, it means for example, having people sending me their one pagers for product description, the experiment plan and timeline, experiment results and their hypotheses, those kinds of things.

Caitria: 34:58

It's like an open door policy where just want the team to know that I should be engaged in interpreting and hypothesizing what the user is experiencing in the product. And then that means that I kind of have an eye on what people think the answers and the questions are so that I can frame the report in a way that answers those questions or validates or invalidates those answers. So I'd say like upfront, I don't know how to give a good time estimate, but I will describe like for a typical study, let's say a usability study where the product is in flight. We're trying to figure out what to build, it's iterative. For one of those sessions, I will generally throw a meeting where I have the product manager, the designers, data scientists, and maybe one or two of the front end folks come and sit down and then go through where the product's at.

Caitria: 35:53

So have the designer walk through and then take down everyone's questions, what they're worried about, what they hypothesize. And then I add those to my own questions that I have about the product at that stage. So that I have some insight into what this study ideally needs to answer. I frame my questions around that but I also know kind of from that point on, I know what the report needs to say. So, I'm already able to have like a template for my notes before I start my sessions.

Caitria: 36:21

So I know for each session, need to answer these different five categories. I needed details on this. I can almost spreadsheet it out at that point. So the upfront time is one meeting and then framing the questions and dealing with the prototypes and all of that. I'd say like maybe half a day to a day of upfront time to prepare for the study.

Caitria: 36:44

The study themselves, I generally take a day to two days depending on the number of people that I'm actually going to interview. I'll generally do like, I'm doing more in the eight range to the 10 range recently because of the types of questions I've been asking, but generally eight to 10. And then one of the speed up methods I use at that time is I have those five topics or however many topics I know I need to get to per interview. I, first of all, take notes almost real time jotting down time signatures when something touches any of those points. And then I schedule about fifteen to thirty minutes in between sessions where I fill out a rubric that answers for each participant.

Caitria: 37:27

Did they, you know, what relevant information on each of these relevant points did we come up with? And then I share that generally in a Slack channel, my Airbnb channel. I had a Slack channel called WTF just happened. And anyone who was involved in one of my studies could join and see kind of like the short notes in each of the five or so relevant topics from each of the sessions. And then they could see, jump into the next recording or something like that or ask questions for follow-up.

Caitria: 37:56

So I'm trying to engage people and kind of share those early findings as we go. If I'm doing field work at the end of day one or the end of day two, I'll generally send a notes from the field or kind of that same kind of what happened quick notes. Then it's pretty easy. I have a lot of slide templates that I've built up over the years, but I generally start off by listing, make section dividers for the slides first. You already know what the slide deck needs to show.

Caitria: 38:25

So you can start filling out these are gonna be the five sections. What did you learn out of that? You go back to your sets of notes and you can really easily look across them and say like, these are what seems significant. I also, I try to make sure that when I'm doing a deck, it's always weighted from most important, most impactful thing down to least impactful. So, and I try and make sure that people know that it is impactful or is it because sometimes I've noticed people get excited about one detail at the expense of something more important.

Caitria: 38:55

And then I use a whole bunch of, you know, use a whole bunch of clip art or I go on Dribbble and I steal art. Don't tell anyone, sorry,

Erin: 39:04

this is a podcast.

Caitria: 39:05

You can cite people's, you can use and then cite people's work from Dribbble or any of those places. And just, yeah, just it's not our job to make it beautiful, but we do have a lot of tools that are at our disposal and templates that make it fast and easy to make something that's pretty attractive.

Erin: 39:27

You used the word that I've been wondering about, which is templates. Do you have like templates that you share with folks who are maybe newer at this, or certainly you have templates you're sharing with yourself? You know, how did you get to that template? And like, look, I can object to my own question. I don't know if you would, right?

Erin: 39:45

Like you kind of have to be, you know, enjoyable and informative and actionable, and there is no like template per se for that. It's based on, you know, the specific kind of research you're trying to do. But do you have any kind of shorthands or outline? I mean, we've talked about a lot of it, but I'm wondering if we can help folks out who, yeah.

Caitria: 40:06

I do have a shorthand. So I mean, the short answer to it is I prefer to make my decks most of the time without templates because it lets me have a lot of freedom and I've done so many at this point that it's feels like a template, you know, you just throw stuff up there. But I do, I have a set of old decks that maybe I'll go back and copy a ton of shit out of it if I'm really, you know, pressed for time. And then I share, I share very widely with people. I even have like medium articles where I've shared some like just more general templates for people.

Caitria: 40:35

Because you know, if right aligning or left aligning everything isn't your skillset, like you should be doing data analysis. Why are we, you know, people who don't wanna be messing with pixels should not be. In terms of like things, ways to help people out. I have a couple of kind of rules of thumb. The first is that cover slides.

Caitria: 40:55

So important, rarely done. People need to put the dates, their names and contact information and what the study concerned. Because usually it's like usability study trial number 17 or something like that. And that doesn't help people find that information. So yeah, and then it's something really memorable.

Caitria: 41:12

Some kind of really memorable either a color set that helps them remember it or an image that helps them know this is that particular product or even a screenshot of the screen in the particular thing that they're testing. Next, it's really helpful to highlight the method. So that's the sample size, what types of people it was and then also a note especially if your team hasn't worked with user research that much, a note on what that means. So that might mean either telling people like, for example, you can use this to understand what is usable or maybe not usable for the average person but not preferences and desires or something like that or attitudes. You can help them know how to interpret this particular method that you've used.

Caitria: 41:59

The other is a TLDR. So that upfront list of, if you don't have any other time, these are the things I really need you to know. And then bolding some of it, help those parsers and skimmers bold the most important stuff. And then the same thing with a list of recommendations. Put those upfront too because I get paid or don't get paid based on my impact.

Caitria: 42:19

So I want it right up there at the front of the deck. You should do this. Then I think another really helpful thing is to break something down into thematic sections. It feels overwhelming to get all of the information at once. You wanna help people.

Caitria: 42:34

So let's say they're only interested in the date picker or the button or the thing, a particular service, break it down into the sections that your team members need and then make clear section dividers that help them jump between. Then if you're using something like Google Slides or Keynote online, you might even be able to link those so that they can jump and have an upfront kind of glossary of what's gonna be in the deck. You get into the you should have very clear screenshots of what was tested. The product changes so often. So if you're like, it was hard to find which version it's changed again already before the sessions were even done.

Caitria: 43:14

So having that what we tested upfront and links to the prototype, links to the videos of the sessions or your notes from the sessions and the script are really helpful. And then having recommendations both in line. So when you're teaching them the thing, the button's not findable, give the recommendation right there, make it purple or move it to the center of the screen. But then also have that section at the end that recaps the recommendations. And then the last thing I like to put in place at the end of decks and the next steps, both for research and for the product team, just to get it out there and kind of give people that kick in the butt to get moving.

Caitria: 43:52

I'm like, okay, the next session needs to test these open questions. We still don't know X, Y, and Z. And then I'll say like the next step for the product team is to give me mocks that help us text X, Y, and Z. And you've kind of almost set up the product process at that point using research as the yardstick. You're saying, Hey, we're not done yet.

Caitria: 44:12

We haven't done this yet and we need to. So your job is to help us figure out what we need to do to do so.

Erin: 44:18

Love it. Cliffhanger.

John-Henry: 44:21

That was a great lift. Amazing.

Erin: 44:23

All right. Well we are at four zero four and I think we

John-Henry: 44:29

Hang on I a good question to close on. Favorite emoji?

Caitria: 44:34

Oh, I'm gonna say the scream emoji.

John-Henry: 44:37

That's a good one.

Caitria: 44:39

Do I need to give a reason?

Erin: 44:41

I think you do. This is a user research podcast. We need to know why.

Caitria: 44:46

Right.

John-Henry: 44:47

When's the last time you used it?

Erin: 44:48

Will you use it again?

Caitria: 44:49

Oh, definitely in a deck. Yeah. I think it was definitely in a deck. I think it was reacting to like a particular user's, it's just some quote that just seemed kind of crazy. This hasn't happened recently.

Caitria: 45:03

I haven't worked at Facebook in years, but I worked there during the election. You would just get like, you'd send a survey out about something that had nothing to do with anything. It's just like about messenger and then you just get crazy, crazy responses back. So I like sharing like just something that you don't, shouldn't have learned. It came out of the person's mouth, share that with the scream emoji.

Caitria: 45:25

And then it kind of like, it helps bring humor out in some of the responses.

Erin: 45:33

Well, JH, now you have to say what your favorite emoji Yeah,

Caitria: 45:35

that's true.

John-Henry: 45:36

What did I say? We did this as an onboarding thing the other day. What did I say? I said I like the eyes, like the they're kinda glancing to the side. I think you can use that in a lot of different ways.

John-Henry: 45:45

I'm big fan of the ghost emoji, just kinda playful.

Caitria: 45:48

Ghost is good. Yeah.

John-Henry: 45:49

Yeah. Erin, what was yours?

Erin: 45:51

Well, now I'm trying to think. I I should just let Slack tell me what it is because it knows which one I use the most often, doesn't it? Yeah.

John-Henry: 45:58

But that doesn't mean it's your favorite. You know mean?

Erin: 46:00

Well Like

John-Henry: 46:01

the plus like the plus symbol Favorite. Makes a lot of use, but that's always favorite.

Erin: 46:04

I do use the heavy plus sign quite a bit.

Caitria: 46:07

Balloon's another great one. And last pro tip, when I'm sharing findings, I put emojis in the email title and it gets people to click.

John-Henry: 46:16

Balloon. I don't think I've ever used balloon.

Erin: 46:19

Bro, you know my favorite? I like the shrug and the face palm. Those are my two favorite.

John-Henry: 46:23

Yeah. Yeah. Shrug ones are good.

Erin: 46:31

Thanks for listening to awkward silences brought to you by user interviews.

John-Henry: 46:36

Theme music by fragile gang.

Erin: 46:39

Editing and sound production by Carrie Boyd.