Daniel Gottlieb [00:00:00]:
If somebody is running a customer interview that is less than perfect, they're not doing the best job. I still think they are going to learn more from that than me telling them the results of an interview I ran. So I am all for supporting them doing it and that's how they're going to learn to do more of it. But I also believe that if you put too much pressure on doing it perfect, put too much pressure on. You have to meet these standards. You have to achieve this. You have to be ready to say this. You can't do X, Y or Z.

Daniel Gottlieb [00:00:28]:
Suddenly an interview is really intimid.

Erin May [00:00:43]:
Hello everybody, and welcome back to Awkward Silences. Today we're here with Daniel Gottlieb. Daniel is the head of research operations Developer division at Microsoft. And we're going to talk about a very research operations friendly topic today, which is how do you really enable and empower people who do research, which we call colloquially powder here, a lot through research education programs and how you've come to build those and find success with those. So, Daniel, thank you so much for being with us today.

Daniel Gottlieb [00:01:12]:
Absolutely. Thank you for having me. Excited to talk about research operations.

Erin May [00:01:15]:
Fantastic. We've got Carol here too.

Carol Guest [00:01:16]:
Hey everyone. Glad to be here and yeah, excited to dig into a fun topic in an area that Daniel has a ton of experience. So I think there'll be a lot to learn here.

Erin May [00:01:24]:
Awesome. So, Daniel, I think always an interesting question really with anyone in research, but research ops maybe in particular is how'd you get there? Most people didn't go to college to study research ops. How'd you find your way to research ops?

Daniel Gottlieb [00:01:37]:
Yeah, I would say I have a non traditional background in this, but like you said, I think most people do. You don't grow up saying, when I grow up, I want to do research operations for a big tech company. I just don't know that exists. And I certainly didn't know it existed for a while. So my background, I actually have a PhD in animal behavior. So I got that from UC Davis. And I specialized in animal welfare and in primate behavior. So that means when I say animal welfare, it's like quality of life.

Daniel Gottlieb [00:02:07]:
So doing research on what makes for a good quality of life for animals, what makes them happy, what makes them stressed, how do you know that? How can you evaluate their needs and how can you provide for them an optimal type of environment? And I specialized in primate, so monkey behavior in particular. And I did that for about 15 years. And I did research at a couple different primate facilities doing behavioral research. I ended up at the Oregon National Primate Research Center. It's a research facility with about 5,000 monkeys. So big facility, lots of monkeys. And I eventually transitioned from doing behavioral research myself to doing more oversight of a lot of our research programs. So we called it Research and Logistics.

Daniel Gottlieb [00:02:56]:
And it was basically about, like, how can we help everybody do all of the different research programs that they want to do, given a lot of the different constraints? And when you're working with animals, man, there's a lot of different, you know, things you want to consider, right? Like they have different needs, they're difficult to manage, they're a limiting resource. So how can we get all the different research going on with, you know, different researchers have different things they want. Only certain animals are going to be appropriate for them to work with. Certain animals aren't going to do well in certain types of studies. Certain ones are going to do great. How do you balance all of the needs and limited resources, limited time, doing that all at once? So I. I had kind of this big puzzle that I worked with and working with the different teams, the researchers, the animal care staff, the animal welfare experts, all kind of working together. So I did that for.

Daniel Gottlieb [00:03:46]:
For a long time. And how long.

Erin May [00:03:49]:
How long are we talking?

Daniel Gottlieb [00:03:51]:
So, yeah, I was with the monkeys for 15 years, and I think I did that part of it for maybe four or five years when I was doing less research myself and more oversight and logistics of research. Didn't call it operations, called it logistics. And I hit a point when I was ready for a career change, my family was moving up to Washington. And this was kind of a good point for me where I figured, okay, I'm ready to try something el out. And I. I had no idea where to begin, right? I knew I was going to Washington. There's a lot of tech there. So I was looking at tech.

Daniel Gottlieb [00:04:26]:
I mean, I was looking all. All over different types of jobs. And one thing that jumped out was this position at Microsoft. It was called, you know, Research Operations. And looking at the job description, it's a lot about balancing research, helping people do research. And that spoke to me. So I applied to it. I don't want to say on a whim, but it wasn't like, all right, this is a, you know, slam dunk.

Daniel Gottlieb [00:04:48]:
I've got this. So I met with the team, talked to the team, had a great time. I obviously jumped to the end. I got hired. And one of the funny moments for me during all of this is I was excited. They were willing to take a chance on me, right? Like, okay, they got this weird behaviorist who works with monkeys and they want him to come to help their UX research team. Fantastic. And on I think the first day I started, my boss said, all right, we need to write a little blurb.

Daniel Gottlieb [00:05:13]:
We're going to send out to the division about the new hire and about you. So can you write up something about your experience and make sure to highlight all of your research ops background. That's why we hired you. So really let everyone know all of your research ops background. I had this moment of just like, you think I have research ops background? I mean, I just like absolute panic of like, did they hire the wrong guy? Did I misrepresent myself? Did I say the wrong word? And I was like, really? Just like, okay, I don't really have ops background. What are they thinking? So I took a little time to like think it through and kind of look at the what I did and had to take a step back and go, okay, what did I do? That was actually research operations in my old job and how can that translate? But definitely, man, you talk about having that imposter syndrome. When I started off the bat and they felt that I had this experience and I didn't, that was a big moment of like, okay, am I supposed to be here? Type of moment for me, Daniel, I.

Erin May [00:06:05]:
Have this assumption and please disabuse me of this if it's wrong, which is that once you've spent 15 years doing logistics with 5,000 monkeys, that's kind of peak. Was everything easier from there or not so much. Just different contexts, different kinds of operations.

Daniel Gottlieb [00:06:22]:
Oh, I don't know if I want to say this out loud because I think people think I have a very stressful job. But the truth is it is so much less stressful when you're dealing with this type of environment. And like this type of the scale is still huge and we're actually doing more and more research in tech than we were. You know, when you're doing academic research and research with animals, like, it's a slow process and there's a lot of checks and balances. So things are going faster here, but at the same time, like the unpredictability is not there where it was with what a monkey is going to do on a day to day. I mean, I remember coming to work on days and be like, we had a cage war, which literally means it's a Game of Thrones style overthrow where one family decides to collectively overthrow the other family and it just throws everything into chaos for the next week to months. So not having to come to Microsoft and say that there was a cage war. I mean, like, yes, it feels.

Daniel Gottlieb [00:07:16]:
The stress levels for me are. That is far less than it was back at working with monkeys.

Carol Guest [00:07:22]:
This is amazing. I'm picturing, like a user. I'm picturing a panel that you're managing of a bunch of different users, but the attributes on them are like alpha male 5A or B, whatever it is that you say. Very interesting. We have to ask, what are some of the biggest similarities between research ops with monkeys and research ops with humans?

Daniel Gottlieb [00:07:42]:
I mean, when it comes to, like, the actual ops itself, there's a lot that's really similar. I mean, what it. What it all comes down to is you're trying to balance the needs of multiple different disparate groups. Right? You know, you got the researchers, you got the. Would you say the program managers or the colony managers, you got the customers. What do we want them to get out of it? You got the monkeys. What's going to be the best experience? And you want to find a way to balance everyone's needs. And you want to find a way to be able to get the research to happen with less friction, so people have to worry about less to be able to go further and do more research.

Daniel Gottlieb [00:08:17]:
You know, the researchers and the primate center researchers at a tech company, like, extremely intelligent, and the product manager is extremely intelligent. Like, they know what to do. They. They are very capable people. So it's a matter of how can I make it easier for them to do what they want to do? You know, how can I make it so that they can spend less time on the things that are harder, have an easy path that they can go forward with their research? So making research easier, making, you know, reducing the friction to get there, that's an. That's an obvious one. But what I also want to call out and I was trying to get at with the different groups all working together, one of the hardest things, but one of the most important things in both jobs is getting all groups to realize that they're working towards the same goal. Right.

Daniel Gottlieb [00:08:59]:
You know, with the monkeys, you know, people are doing this. They all want to get, you know, learn about the animals, get the best, have the best quality of life for them, get this awesome research out. Everyone wants that same goal. And so when people seem like they're conflicting, you kind of work with them to realize, no, no, you're working towards the same goal. And then people are more likely to try and work together and they can go faster together. It's the Same thing when you're working in research in tech company or I guess any company where you're doing UX research. The UX researchers, the product managers, the developers, the designers, we're all here to try and make the best products we can and to learn more about our customers so that we can make products that are going to solve problem for them and be exciting for them and, and sell for the company. So once you realize that we're all kind of doing that and show that each person together we're doing that and what research is going to help you get there together, then they can go faster as well.

Daniel Gottlieb [00:09:48]:
So a lot of research ops I think is both like the technical how can we remove friction and make the programs that are going to help people do research, get the tools, they're going to make them go faster. The documentation so that they can understand what to do next, that's obviously a big part of it. Those kind of hard skills. But the soft skills in both working with monkey research and working with UX research, those are equally important to get people to want to work together, want to use these tools and want to come together to be doing research, all for the same goal. So those are two big things that are common in both. Both sides of research and research ops or logistics.

Erin May [00:10:27]:
Yeah, and that's a great point. Cause I, you, I do associate operations with a lot of process which can sometimes maybe have a little bit of a pejorative connotation, but if you bring it back to this is helping you move faster and what are, what's the goal at the end of the day? And the shared purpose probably makes the process seem a lot happier and more followed to everybody.

Daniel Gottlieb [00:10:46]:
Yeah, yeah, yeah, yeah, yeah. I mean somebody who's happy and excited to be doing their job and working together, they're going to go so much faster and further and be willing to push through some of that friction as well.

Erin May [00:10:56]:
Maybe you could tell us a little bit about your team now. So you're at developer division at Microsoft. The blurb was a success.

Daniel Gottlieb [00:11:03]:
Yeah, yeah, yeah.

Erin May [00:11:05]:
You made it through the first day. So what are you doing now? What's research ops like there?

Daniel Gottlieb [00:11:10]:
So I work for the UX research team in developer division and in Microsoft we have multiple different divisions. Microsoft is a huge company, so every division itself is practically like its own company. We are a team of 2022 researchers and we have a research ops team of two, me and one other person who's doing research ops to support the team and in Microsoft and well, in developer division in Particular, we are all about learning together with our product teams. So we really involve our product teams in the learning experience and we empower our product teams to be learning together with our researchers. Or if they are able to pick up the ball and do some of that on their own, they're also involved in some of our product teams doing research on their own at the same time. So what my day to day is changes on any given day. But me and I'll, I'll include my other research ops person together because we do everything really in tandem. It's about finding ways to reduce the friction for our UX research team, reduce the friction for our product teams.

Daniel Gottlieb [00:12:12]:
Finding ways to empower our UX researchers so that they can do more research, but then finding ways to empower our UX researchers so that they can empower their product teams so that they can be learning more from their customers. We use the term force multiplier a lot when we're describing our job because we really feel like in research Ops I can multiply the efforts of the UX researchers because like I said, they are extremely smart, extremely capable people. If you just remove some of those roadblocks and make it so they only have to think about a, they only think about their research. They don't have to also think about what's the right path or what's the legal thing here. They can go really far. So I can force multiply their efforts. But because they're also helping product teams learn and they're empowering some product teams to learn on their own, they can also be force multipliers for those product teams. So it's this really cool thing you can kind of see of like if we can get them to do more, they can get the product teams to do more and you can make these huge, huge changes.

Daniel Gottlieb [00:13:04]:
So that. I know that was pretty broad. When we're talking about specifics, you know, we're researching the tools that should be done. We are helping to find the right recruitment platforms and recruitment different ways to find customers, which changes on any given day. Because people are constantly changing what they're looking for. We're documenting, you know, like document, document, document, constantly updating that so that people can easily find what they need to do and feel confident that they're doing the right thing. But another big part of our, of what we do is education. I mentioned we, we empower our product teams to learn.

Daniel Gottlieb [00:13:43]:
And if we're going to do that, we really need to set them up for success to understand like how do you learn from Our customers. Why is it important to learn from our customers? So we run workshops, and I get to run these workshops, head up these workshops for our new to Dev Div employees. And in these workshops, we are educating everyone in the division about how to do customer research, how to come up with a hypothesis, how to come up with a discussion guide, how to talk to customers, how to synthesize that, how to come up with a presentation and how to drive change by demonstrating the needs of the customers and how to integrate that into your products. So we run workshops like that and then we also do different types of activities to try and keep research and the value of research and the methods of research in the front of people's minds. So different ways that we can kind of celebrate research and make it fun. And so it's not like you do that workshop, then you're off and it's done and you're gone forever. It's more of a like, okay, well how do we level up? How do we keep on training and keep on making this something that people want to be engaged in and involved in?

Carol Guest [00:14:52]:
So just to go back a little bit and give us a sense of scale. So you mentioned it's two research ops, 20 plus researchers. How many, you know, people who do research or product teams are you, are you working with?

Daniel Gottlieb [00:15:04]:
I mean, I think there's, in our product org, there's probably like a thousand. And that's not including the developers. You're getting into thousands when you're talking about developers. And you know, we say, and this is absolutely true, that like everyone in the division is responsible for the customer experience and everybody is empowered to go out and find the customer needs and learn about the customers. That said, that is a very nice, high level thing to say. Is everybody doing that? No. And is every developer out there going and doing that? Definitely not. But everyone has the ability to.

Daniel Gottlieb [00:15:36]:
And I also, I think there's many different ways that people can be involved in the research process or appreciate the research process. You use the term powder, right? People who do research. And I don't know if you have a definition of like you gotta do X, Y or z to get into that. Actually, before I even go, I'm curious, how do you define powder? How do you define people who do research?

Erin May [00:15:58]:
This is a very good question. I would say someone who's not a full time researcher, they do not have research in their job title and yet they are doing some research. So then the question is, what does it mean to do research? No one's ever asked me this before, but I would say that they are not passively merely consuming research. Right. Although that is more than not consuming research. But they could be interviewing customers, could be affinity mapping together. Right. Being part of that learning and discovery process in some way would be a pattern.

Erin May [00:16:30]:
Perhaps they've done it, you know, more than once. Yeah, I don't know. What do you, what do you guys think?

Daniel Gottlieb [00:16:34]:
I mean, I think that that absolutely makes sense to me, what I would say and maybe, and I like that you gave that definition because where I'd expand and maybe, maybe even going beyond powder is people who are customer driven. Right? So there's a degree of appreciating the value of research, even beyond just passively consuming it and understanding why it's necessary, asking for it and being customer driven. That might not be a true research project. So when I think about like powder, people who do research, there's going to be people who are running their own study on their own, right? Just completely. They're not a researcher, but they are soup to nuts. Finding customers, coming up with a discussion guide, their own hypotheses, writing up reports, all of that. Then you're going to have people who are involved because they are doing research with the researchers. That's probably what we see a lot of.

Daniel Gottlieb [00:17:21]:
They're coming up with hypotheses together, building the discussion guides. Maybe they're running an interview or two, maybe they're not. They're taking notes, doing the sense making, involved in the research, but not necessarily running it all on their own. Or people who are kind of along for the ride, not comfortable doing that themselves, but they want to be a part of the interview, they want to see it, watch it, help with the sense making. Sometimes we'll pull people into the sense making even if they weren't a part of the research beforehand. We'll have people who are ready to do most of it on their own, but they need a little bit of guidance, a little bit of help. But then I'd also say it's equally important to have people who, they're not doing like a traditional UX study, but they're going out there and talking to customers, or they're looking at social media and getting some feedback, or they're looking at the telemetry and looking at the quantitative data and looking at the patterns and they're appreciating that even if they're not forming it into a traditional UX study, but they're customer driven because they appreciate the value of understanding the customer's behavior and letting that Guide us.

Erin May [00:18:16]:
Right?

Daniel Gottlieb [00:18:17]:
We'll have developers who maybe they aren't doing the study themselves, but they have been so ingrained in the value of the study that when a PM asks, hey, we need to make this change, they'll say, well, what's the reason behind this? Did you test this? Do we have customer evidence about making this change first? Do we have. Did you have a hypothesis that you tested? And what was that hypothesis? I want to know because it's going to drive the way I make this change. So I was curious why you call people do research because, like, I don't know if that's doing research, but that's definitely being customer driven beyond just passively consuming the research.

Erin May [00:18:50]:
And it's part of the group of stakeholders I imagine you're trying to educate and provide tools for in some way. And it's a really interesting distinction because it's not that you're putting a value judgment on it, but I will. Right. It's using user insight. And being customer centric is almost more important than like doing research wherever those insights came from.

Daniel Gottlieb [00:19:09]:
So, yeah, and I have to say it was a learning moment for me because at first when I started doing these workshops and we have, I'd say, like half the people going through our workshops are developers. And I was very much like, everyone, go do research and you can go do this and you're going to be a part of study. And why should I be doing this interview? Because you'll be doing it in your job and that's not always happening. And then I'd get feedback being like, yeah, that, that was fun, but then it didn't mean anything to me and I realized that, like, not everyone's going to. And that's okay, but they are going to be involved being customer driven. And it's important to demonstrate like, hey, you're going to learn the way the rest of the division works so that you can better be a part of the system and work with that and take advantage of what other people are doing around you. And that shift made a big difference for people. Could be like, okay, I can be on board with this.

Daniel Gottlieb [00:19:53]:
I can learn and be a part of this today, knowing how I will consume it later.

Carol Guest [00:19:58]:
So, so let's get. I'd love to get into some of these workshops. So imagine, imagine you're a research ops person and you're thinking about standing up some type of educational program. Where did you get started? And are there some places that you think are maybe highest impact for us, this workshop?

Daniel Gottlieb [00:20:13]:
And let me give credit where credit is due. So I jumped into this team when the workshops were rolling and I got to be a part of it. And I'm running them myself, but they were established before I came and they were all built on. And I'm going to show this book like ten times over the course of this interview, if that's okay. There's a video component to this too, right?

Erin May [00:20:31]:
Yes.

Daniel Gottlieb [00:20:32]:
Oh, how funny. You can see the green screen behind it.

Erin May [00:20:34]:
Yeah. So you can read it. Okay, good. That's a good. Still there. All right.

Daniel Gottlieb [00:20:38]:
It's called the Customer Driven Playbook. It was written by Travis Loudermilk and Jessica Rich. They are on our team. And this is a go to guide on how to be customer driven and how to do UX research. And it's written in a way that if you know how to do UX research, it really can help you come up with a game plan and understand the value of it and give you tips. If you know nothing about UX research, it's a really good way to just understand the basics of it and understand why it's important and how to do it. This is how I trained myself when we, when I started because I mentioned I came from monkey UX research, which is different from human UX research. And so the workshop we do is all based on this book and so it's based on this.

Daniel Gottlieb [00:21:19]:
So like we've come up with activities based around the Playbook and I gosh, I don't want to over promise. I do think we have some of that activities on Medium and maybe if I'll see if I can find them. If we can, we can include this. If not, you can cut that. I don't want to promise something that's not there. So we structured our workshop around how can we give people the basics about what it means to be customer driven? How do you develop a hypothesis? How do you come up with a discussion guide, talk to customers and synthesize it and then share it out and share it out to leadership. So I would say when, if people are interested in doing their own kind of training and educating, it's definitely about what are the needs of your division or your team or your company. Whatever it is for our team.

Daniel Gottlieb [00:22:08]:
The reason it was developed this way and the reason we structured our workshops this way is there was at first a big push to get people to talk to customers. We wanted our product and just talking to them like understanding who the customers are, just not just making decisions on their own. Step one, being customer driven, that was successful. But what ended up happening is the conversations were not guided in a specific way, so they go in any different direction. Hard to get insights. Are you really testing hypothesis? They developed this book to help people understand how do I structure my conversations and structure my research in a relatively low effort way to be more focused and driven and actually answering specific questions. So that said, these workshops are structured in a way that we're solving that problem where somebody can come into the workshop and understand. How do I come up with a question? How do I answer that? By talking to customers.

Daniel Gottlieb [00:23:00]:
So that was our goal. So that's how we structured it. If your goal is more about, we want people to understand how to get quantitative data and do telemetry analysis, of course that's going to be how you structure your workshops. When it comes to getting this type of thing off the ground, I don't know if that's what you're kind of getting at.

Carol Guest [00:23:20]:
Yeah, I'd love to get into that.

Daniel Gottlieb [00:23:21]:
Yeah.

Carol Guest [00:23:22]:
Who are you targeting for the first set of workshops? How do you think about things often? Does everyone do it? All that good stuff.

Daniel Gottlieb [00:23:27]:
Oh, and that's top of mind for me because what often happens is we get asked this question from other divisions. So I said, I jumped in. When this is already rolling, I got to take advantage of it. I didn't have to sell it. But then we'll get these other divisions saying, we want to do something similar and they do need to kind of sell it. So when we're working with other people about how do you get your own type of program like this rolling, one of the biggest things we recommend is find a small group that is excited about doing this. Right. Like you might have leadership, the top coo, the top boss, saying, I want everyone to do this.

Daniel Gottlieb [00:24:02]:
It's important you should go do this. Even if that person is saying it. If you go and force all of the different teams to go through this education program, it's painful. They don't want to do it. You're taking them away from their job. They're going through this required forced training. And then when it's done, they're like, all right, now I got some work I had to catch up on. And they're going to start doing that and it's not going to continue.

Daniel Gottlieb [00:24:23]:
But if you can find that one group that's like, yes, we've been trying to do this. We want to talk to customers, but we don't know how. We want to be doing telemetry analysis, but we don't know how. We say, great, I can Solve this problem for you. You work with that small group, they can then become the evangelist for your program after. And you highlight them and you celebrate them. You share them with everybody. Look at this group.

Daniel Gottlieb [00:24:44]:
Look what they did. Look how great it was. They can sell it, they promote it, and it can kind of grow in that way. I am a big believer in, unless there's, like, a legal reason, never forcing anyone to go through any certain training, any type of specific tool, use, any pattern for how they do their job. I never want to force it. I want to provide something people want to go through and are excited about. Now, to get people excited about it, sometimes you have to do a couple different things, like, smart, starting with a small group and letting them evangelize it. But it's.

Daniel Gottlieb [00:25:14]:
I really try and avoid these situations where you're forcing it down in people's throats or making them take this training that they don't want to do. So I rambled a bit. But I would say, in summary, it's kind of identifying the specific problem that you're trying to solve for your division and finding a way to educate that, and then finding a group that really wants you to solve that problem for them and is excited about the way you're going to do it, that can partner with you and then evangelize it.

Erin May [00:25:39]:
When you run these workshops, you talked a little bit before about, people are starting from a different starting point, right? In terms of their knowledge of research, what they're trying to get out of research, how they want to engage with customers. Are you running lots of different, like, sort of bespoke workshops, or is they sort of, like, fit regardless of where someone's coming from? Are people coming to you and to your point, asking for them because they've this, you know, enthusiasm is contagious. And like, hey, I hear you run a workshop. I want in on that. Like, how does. How's it spreading and how are you sort of scaling it across these thousands.

Daniel Gottlieb [00:26:11]:
Of folks at this point? We're doing it where we are. Anytime anyone new comes to the company on a regular basis, depending on how fast we're hiring, you know, whether it's like quarterly or annually, we'll run a big workshop that everyone is invited to and to invite to participate in. Um, and we don't hard require it, but we have leadership support and we have manager support. So people generally join into this, and there's kind of this knowledge of, like, oh, this is what we do. This is something that's going to help me with my job. I want to be a part of that. So for the most part we're getting people coming through as they come into the organization. But then we do have other groups that are like outside excited about it.

Daniel Gottlieb [00:26:50]:
We'll invite them to join us or people who missed out or maybe even want to re up. But it's a very, it is a baseline education, right? This act this week long, It's a week long workshop, kind of like a hackathon style, like shut everything else down. You're in here with us for a week. It's baseline. It's, it's understanding how to do research. It's, it's not going to be giving you brand new skills if you're someone who's an expert. Actually, if someone comes in who's an expert, we instead say, hey, why don't you come and coach with us? That's a great way for you to be able to be a part of this. You can be a coach for this workshop.

Daniel Gottlieb [00:27:22]:
And you know, that's also a way to get people to understand our methodology if they're an expert. But not doing it the way that we're trying to demonstrate they can, we can get them on our side, helping them to coach, to coach. So that's how we get people going through. I will say I had one of our partner teams in developer division in Shanghai. They, they recently started this program themselves and they had to start kind of where at the beginning. And they, similar to what I was saying before, started with smaller groups that were really excited about it and grew and grew to the point where suddenly they had a bunch of different divisions who were not a part of their division asking to join their group at the end because word of mouth and excitement over it had kept it going. Did I fully answer your question? I feel like I might have ventured off.

Erin May [00:28:06]:
Yeah, no, you got it.

Daniel Gottlieb [00:28:07]:
Okay.

Carol Guest [00:28:07]:
As we were chatting earlier, you were mentioning some of the things that you do during the workshops to keep engaging, keep people involved. You mentioned it's a week long, so it's a fair amount of time. Will you share some of what you do yet during the workshop to keep people involved?

Daniel Gottlieb [00:28:22]:
Yeah. So, you know, it's a week long and it's remote, which like I can only pay attention remote to a presentation for a certain amount of time before I glaze off, even if it's something I'm super excited about and super interested in. So we work really hard to make these workshops fun, engaging and we always think about how do you break the script and by that, like do something that's outside of the norm. So people are playing like, oh, I'm going to pay attention to this now. So we have a couple little tricks within there that we do. We. They seem small and like, I don't think people would be able to, like, even identify this if you're in the workshop, but you might. But it's something that we do to help keep it fresh.

Daniel Gottlieb [00:29:00]:
We do things like music cues. Like, I start every morning where I have music playing. It's a little different. You just come in, you're like, oh, this is different. We got some music playing. If you're watching this on YouTube, I have like a green screen behind me. And this is gonna seem kind of silly where it's my own office behind me, so it's not really showing anything different. But the nice thing about having this green screen here is that then I can pop up what we're presenting behind me, have it happen.

Daniel Gottlieb [00:29:23]:
It's a little bit different. And I know that doesn't seem like, wow, groundbreaking, exciting, but having that little bit gets people to pay attention again. When I'm lecturing, I am never going too long without interacting with the people. So whether it's asking questions, doing a book giveaway based on what we talked about gets people kind of on their seats. But then it is this rapid fire back and forth between lecture and then working together in a group. We do use, like, remote whiteboarding to do all of our activities. And the majority of the time is in these small groups that they're working together. And so that back and forth also helps keep it engaging.

Daniel Gottlieb [00:29:59]:
I think I mentioned this before, a little fun, like audio things we can do just to once again, like, it wakes you up, it gets you paying attention. But then as far as the involvement goes, like, that's all fun, that's all good. But then as far as the involvement goes, you also want the people to feel like they're getting something out of this week. So what we do is they're not just doing fluff. We make sure people are doing real work. So we spend a lot of time beforehand working with our leadership to understand what are some of the biggest problems you'd like a team like this working on. Like, we got a team working for a week. We got multiple teams.

Daniel Gottlieb [00:30:35]:
We might have like eight different. Ten different teams working on ten different problems. And we'll come back with ten real business goals. That's important to our leadership. And we'll have leadership record videos talking about what the goal is, why it's important, why this matters, and Then we'll show those to the teams beforehand. Suddenly, instead of just like a week of training where you're learning the basics of whatever, it's like, oh, I'm new to the company and I just saw my leadership talk about this important problem and I'm tasked with solving it in the next week. Right. So suddenly there's weight to it as well, and they're motivated to actually get through it.

Daniel Gottlieb [00:31:06]:
We try and make sure the week is filled with these big moments that are going to carry you through. So it's not like this drain over the whole week. Halfway through, we bring in real customers. And that's a moment that is exciting for people. It's different, it breaks the script. But also a little bit of a forcing fact, forcing function of like, you got to get your work done. So at the end of day one, if you don't have your discussion guide, you're going to go in to talk to a real customer the next day and you're not ready. So people have to kind of put in that work.

Daniel Gottlieb [00:31:36]:
I know it's a little bit of the balancing of stress and feeling comfort. We want people to feel comfort, but we also need a little bit of that pressure to actually sometimes get that work done. So they interview real customers on the next day and then they'll last. Big moment we have is we have our leadership come in and they come and watch final presentations. So remember I said at the beginning that I talked to leadership, get videos about, like, why this is important. That makes leadership invested too. They want to come in, they come to the final presentations and then suddenly you're a new employee and a weekend you're giving a presentation to your leadership about this new concept you came up with about how you talk to customers. So it's a big moment for them.

Daniel Gottlieb [00:32:14]:
It's an exciting moment. It's also a, like, once again, a little kick in the pants, you got to get your work done moment. And then the great, great thing at the end of all that is you're getting rewarded for doing this work. You know, you're getting a chance to speak to leadership and you're seeing them say, oh, that's great. I love that you talked to this customer. I love that you came up with this idea. I love that you realized you were wrong and that you found out that the customer surprised you and they'll. Our leadership is fantastic about saying that, demonstrating that.

Daniel Gottlieb [00:32:44]:
And suddenly you're like, oh, this is how I impress them. So that's a big part of how people leave the workshop. Feeling like this is important work, I want to keep this going. So I hope I kind of answered in there both like, how do we keep it exciting but then how do we keep it so that they want to keep going during the week and after it's all done?

Carol Guest [00:33:02]:
So it sounds like just to play some of that back to you, this is not a passive, I'm listening to a lecture for a week workshop. This is sort of more like a research sprint, right, where you're doing some education and then they're applying that to a real problem identify at the company and then going through the full step from question guide to synthesis presentation, all within a week. Is that right?

Daniel Gottlieb [00:33:23]:
Oh, yeah, absolutely. Yeah, yeah. This is real work. I mean they are talking to real customers and they're running those interviews themselves. They're coming up with real concepts, they're presenting them. So it's absolutely like a working week, which I, I probably should have started with because that's part of the way that we make it really engaging. Because like, gosh, it is just hard to sit and listen to lectures. And we want everyone to feel like they are empowered to do this, right? Like every role gets to do this.

Daniel Gottlieb [00:33:49]:
Developers are coming to this. PMs are coming to this, Design's coming to this. We have admin coming to this, right? Like, hey, understand how we work. Understand what your team's going to be doing. Everyone's empowered to be doing this.

Erin May [00:33:59]:
How often do you run these workshops?

Daniel Gottlieb [00:34:01]:
Uh, it, it totally depends on the hiring rate. Cause it's mostly for new. So we've done it every three months. We've done it every 12 months. It kind of depends on where you're at.

Erin May [00:34:10]:
So they probably got quite a few folks in there then.

Daniel Gottlieb [00:34:13]:
Oh, yeah, they, I mean, they can be, you know, up to. We've had like up to a hundred people coming at once. Um, it's so much easier to do remote. To be able to manage that remote has so many benefits of. Also, all those different people can be involved and active at the same time. Something I found is we see people can be involved in ways they wouldn't be involved in person by doing it like this. Because it's a lot easier to be in your home space. You can be involved by participating, by typing and putting your thoughts into the chat when you're not as comfortable speaking up.

Daniel Gottlieb [00:34:44]:
There's a lot of different ways. It's nice that it's more inclusion when we're doing it like this. And one thing we did, I should say, to make it successful and we're doing this many people. I brought up coaches before. We have, each group gets their own coach that's either a UX researcher or we have a number of PMs who like, they are just like those kick ass superstars who are doing customer driven research. And we ask them, can you help us out? And then they're coaching their teams as well. So everyone has someone they're working with. No one is kind of off on their own doing this.

Daniel Gottlieb [00:35:16]:
So it's the safety net and the guidance and support mixed with the stress of. But you're still on the hook for this at the end of the day.

Erin May [00:35:23]:
You ever check in with your graduates, see how they're doing?

Daniel Gottlieb [00:35:26]:
Oh yeah, all the time. And we try and we try and bring it back in like I said to, to help with coaching where we can.

Carol Guest [00:35:32]:
You also mentioned actually one clarifying thing and then we'll jump in. You said there are groups. So you take, imagine you have 100 people, you break it into smaller groups that are each working on one research area. Is that how it works?

Daniel Gottlieb [00:35:43]:
Yeah. And so I mentioned we get like these different business goals from leadership. What we'll have is each group will have a different business goal that's important to the, to leadership. And we try and structure it around. You're going to be working in the area that you're going to be continuing in your product space. Right. So if we have a business goal that's related to a specific product, those people are going to try and be in that group and we'll base the business goals based on who signed up because it changes. That said, we've gone back and forth a bit on this because there's a part of me that wants to give everyone a goal that has completely unrelated to what they do.

Daniel Gottlieb [00:36:18]:
Now, of course, when you do that, it doesn't have the same value of I'm solving a business goal that my boss cares about. But at the same time it puts everyone on a level playing field. Right. Like everyone's coming in. You brought this up earlier. Some people more experienced than others, some people total experts in the product. And they come in like I got my ideas, but this comes in and everyone's starting fresh. Everyone has a fresh set of eyes and that's.

Daniel Gottlieb [00:36:38]:
We've had some groups like that unintentionally and they do really well. So there's a part of me that wants to play around with that a little bit. We definitely find sometimes in our groups where you'll have a couple seasoned veterans who know the product space really well and a couple people who just happen to be in the group, but they don't know the product space because there was no group for their product space. They're the wild cards in the group and sometimes they'll come up with the crazy ideas or the new way of looking at things that actually changes the direction that you're going in a really wonderful way. When you're really in the product space and you see the same things over and over again, you start to anticipate what your customers are going to say, even if that's not actually what they're going to say. So those people who are new to the space are often a good outside perspective on what's happening.

Carol Guest [00:37:20]:
You also mentioned earlier, you mentioned activities you do to keep research in front of mind after the workshops are done. You share some highlights there.

Daniel Gottlieb [00:37:27]:
Yeah, so like one thing we recently did, we called it a Power up series. And this was a series that me and Travis Loudermilk hosted where we were sharing highlights of different ways to answer big hairy problems that the product teams have been doing. So we, the part of this was about highlighting the different research methodologies. Like we wanted to be able to remind people it's not all about interviews. Right. And we want to level up your, your, your method of doing research. You know, there's also certain types of, you'd be doing concept value tests, you could be doing telemetry analysis, you could be doing dog fooding, all these different types of ways you could be learning about the product and learning about our customers. But instead of us just lecturing it, which is once again just kind of boring to hear people lecture, what we did is we highlighted the product makers who are doing this.

Daniel Gottlieb [00:38:25]:
So if we had a method that we really want to demonstrate, what we do is we'd find a PM or developer who did a study on their own about this. And we brought them, we interviewed them, we had them share their experience and then we did this on like a bi weekly schedule. And the nice thing about this is it was a good way to show different research methods, but also celebrate the PMs and the developers doing this show that this isn't something that's just like the UX research team high above can do this research. Come to us if you'd like to. It's like, no, you're empowered to do this. Come to us and we can help you do it. And also our goal is to have a little bit of FOMO there, right? Like we want people watching, being like, I want to be up there. Why are they getting to talk about this? I should be able to do that.

Daniel Gottlieb [00:39:08]:
I can do that. So that's like one example. We've done things called like a Developer Day with something our team had done years back, which is where, like in math, we got a bunch of different developers all together and we partnered PMs with developers. Our developers could be confusing. When I say developer, developer, customers, I should say we brought a bunch of customers in and we paired RPMs with our developers. And together that pair would have their own research question, they'd come up with their own research discussion guide and they would interview a customer for short bursts and then rotate to the next customer. So what was great about this is we could do a whole bunch of research at scale, doing a lot of research at once. PMs and developers are together, asking questions from customers, quickly moving on to another customer, but also as a big event.

Daniel Gottlieb [00:40:03]:
And it's not just about that day, it's about the preparation for that day, reminding everyone, why is this important? How do we come up with a discussion guide together and celebrating and making that day a big celebration? So sometimes it's. It's not just about the experience itself, it's also about. It's not just about the learnings in the moment, it's about what did you learn to prepare for that? And then how do you feel leaving it? Do you feel empowered? Do you feel this is a good experience? Do you want to do this again and do you feel like this helped you do better at your job?

Carol Guest [00:40:31]:
I imagine one of the pieces of pushback you would get if you have less experienced interviewers talking to customers is, is it going to be a bad experience for customers? Are we going to be wasting their time and not learning anything? I think this applies to both Developer Day and the workshop. I wonder if there are anything that you recommend to sort of handle those objections.

Daniel Gottlieb [00:40:50]:
There should be some kind of training beforehand, right? To get people to understand how to talk to customers in an appropriate way, how to talk to customers in a way that's going to make them feel valued and respected. That's absolutely true. You can also do some kind of safeguards of having the UX researcher there. Just because the PM or developer is running the study doesn't mean the researcher shouldn't be involved or be there. One of the great things about doing things remote is you can have somebody running an interview. Let's say you have a PM running an interview and the researcher's on the call, they can be chatting on the side and they can be chatting on the side to the PM who's running that interview, giving them guidance and it doesn't have to be critical. It's not like, no, you missed this, or don't do this, don't say that. It's wow, did you notice that they said this? Why don't you ask more? Oh, that's really surprising.

Daniel Gottlieb [00:41:35]:
Go back and ask him this. So it's a way to help them do the right thing and they feel like they're ready to do it, but they're still running it themselves. I would also say if somebody is running a customer interview that is less than perfect, they're not doing the best job. I still think they are going to learn more from that than me telling them the results of an interview I ran. So I am all for supporting them, doing it and that's how they're going to learn to do more of it. But I also believe that if you put too much pressure on doing it perfect, put too much pressure on, you have to meet these standards, you have to achieve this. You have to be ready to say this. You can't do X, Y or Z.

Daniel Gottlieb [00:42:19]:
Suddenly an interview is really intimidating. Nobody wants to do it. I'm not good enough to do it, I shouldn't do it. I'm just going to let you do it. So even though it seems like, oh no, I'm helping you out, I gave you A checklist of 60 things to say in an interview. Look, just read this and it's great. Suddenly it's like I can't capture all that. I'm not going to do all that.

Daniel Gottlieb [00:42:35]:
This isn't for me. So I much prefer to let people try, maybe fail a little, try again and get better than to try and get it perfect the right time because it's really going to scare people off with that.

Erin May [00:42:49]:
Maybe a good place to kind of wrap before we go to our rapid fire is what is your kind of one tip? If you had just one to give folks to to do a good early interview, if they're not experts, but not, you know, feel overwhelmed by a 35 part checklist at the same time, write.

Daniel Gottlieb [00:43:04]:
Down the what you want to learn, like whether it's a hypothesis or even if you haven't structured hypothesis, if you have just like what is your one, you know, one or two things you want to learn and what are a couple questions that somebody could answer to get there? Write those down and be able to so you can go to them. If you have those in front of you, I know it's a discussion guide, groundbreaking idea. But having a discussion guide in front of you that you know, you can go to those and you can get off and you can otherwise go off topic, that's the key thing I think, is that you have that in there and recognize that you can try and fail and try again. Now, I one thing I will say, this is not going to apply to everybody, but for those of you who are lucky enough to recruit customers and pay them gratuity. They're getting paid gratuity. They want to. They're getting, they're doing a job, they're getting something. And I'm not saying that in, in terms of a, so do whatever you want.

Daniel Gottlieb [00:43:56]:
I'm doing saying that in terms of they're getting something out of this no matter what. And they also want to make you want to do good at their job. Everyone wants to do good at their job. So make them feel comfortable. If you make them feel happy, if they're going to feel good as well. So I guess what I'm trying to say here is as long as you ask kind of your core questions and can make your person feel comfortable, then you're doing your job. If you failed and didn't get your questions answered, it's okay. You can try again.

Erin May [00:44:24]:
Yep. Love that. Love that. And even if you aren't an expert, say, at making people feel comfortable, to your point, customers are quite forgiving in my experience. To your point, they have signed up to be there. They're getting paid to be there and.

Daniel Gottlieb [00:44:37]:
Actually sign up to be a customer. Go to a recruiting platform and like sign up as a customer. Go through that experience and you'll suddenly know what it's like on their side and then you'll be 10 times better because you'll have empathy for what they're going through and understand that like, oh, this is all I need to do and this is what I can do to make them feel good. So, yeah.

Erin May [00:44:56]:
All right, rapid fire, a couple quick questions. What are some resources that you recommend? I know, we'll. Let's flash your book again. Perfect time for that. The Customer Driven Playbook. Of course. Anything else that you lean on or recommend to folks?

Daniel Gottlieb [00:45:11]:
Yeah. So Customer Driven Playbook is a big one about how to do UX research. There is a sequel book, the Customer Driven Culture, which I'd also recommend. That's about how do you change an organization to be more customer driven and kind of drive change even if you're not in UX research. I actually, when I read that book, I sent it to my team at the primate center, I was like, this is what we need to be doing. This is what we should have been doing, or this is what we were doing to some degree. Because it's all about how do you change culture in a positive way without kind of forcing it. Right.

Daniel Gottlieb [00:45:38]:
It's about how do you make the improvements you want. So I highly recommend that if you're interested in Research Ops, anything, Kate Towsey is like gold. I'm sure you've heard that before on this podcast. I'm, I'm reading her book right now. Research that Scales is so far, it is fantastic and highly recommend that. Yeah, if you want to understand about Research Ops, the last thing I'd recommend, and this is going to be an odd one, but I always recommend this. From the animal behavior background is a book called Don't Shoot the Dog and it's a book on animal training. So it's a book on clicker training.

Daniel Gottlieb [00:46:15]:
I don't know if you're familiar with that, but it's basically a form of positive reinforcement training. How do you change animals behavior by giving him positive reward? I love this book because it's not just about how do you change animal behavior, but it comes back into, well, how do you use these practices to also change people's behavior and change the behavior of those around you? And I, I mean, like, if you think about, if I talk about what I learned from my old job and working with monkeys, that's probably one of the biggest things. Changing behavior through positive reinforcement is huge. You can get people to do the best work. You can get them to enjoy their work, and you can really shape behaviors and cultures through that. So if you have a dog and you want to learn how to train them, it's a great book. But also if you just want a fun read about how to influence behavior, Don't Shoot the Dog by Karen Pryor.

Erin May [00:46:58]:
Love it.

Carol Guest [00:46:58]:
Thank you. I see how it's coming out in your approach to education as well. Last question. Where can folks find you? Follow you online.

Daniel Gottlieb [00:47:07]:
You can find me on LinkedIn. I can't say that I post very much, but if you find me on LinkedIn, I will accept your invitation. And every once in a while I'll drop a post here or there.

Erin May [00:47:16]:
Perfect. Awesome. Daniel, thank you so much. Really appreciate you being here. This is a lot of fun. I love that we started and ended with Harry problems. That was fun. That's right.

Erin May [00:47:26]:
Yeah. Thanks so much and have a great one.

Daniel Gottlieb [00:47:29]:
Thank you very much.

Carol Guest [00:47:30]:
Bye.