Serena [0:00:00]: You're part of this ecosystem within your organization and and these people within it make up the culture and how it happens.

Serena [0:00:06]: And if you're not putting your work within that, I think there's a high chance of failure because you know, everyone else has other objectives and other things going on.

Serena [0:00:15]: And if you want your stuff to fit and work within that, you'd better, like, understand that space and include those people.

Erin [0:00:24]: Hey.

Erin [0:00:24]: This is Erin May,

Carol Guest [0:00:26]: and this is Carol Guest.

Erin [0:00:27]: And this is Awkward Silences.

Erin [0:00:30]: Awkward zones this is brought to you by User Interviews the fastest way to recruit targeted high quality participants for any kind of research Today, we have a real treat we have, not one not two, but free guests, all from Microsoft Cloud.

Erin [0:00:51]: And they're gonna talk about how they've built a variety of research programs and been able to really see them be successful over a period of a couple of years, and they're gonna have a ton of advice for you all on how you might do similar in your organization.

Erin [0:01:05]: So so happy to have you all here.

Erin [0:01:07]: We've got Jackie Ianni, Samira Jain and doctor Serena Hillman, who are user researchers, principal Ux research manager.

Erin [0:01:15]: At Microsoft data.

Erin [0:01:18]: So thank you all so much for being here.

Serena [0:01:20]: Yeah.

Serena [0:01:20]: That to be here.

Erin [0:01:22]: Alright.

Erin [0:01:22]: Fantastic.

Erin [0:01:22]: So to kick things off.

Erin [0:01:24]: I know you started these programs a couple of years ago, and you branded them as Swift, which is an acronym that you you can talk about.

Erin [0:01:32]: The Swift research lab is kind of the umbrella for all of these programs.

Erin [0:01:36]: So I'm curious what came first?

Erin [0:01:38]: Was it the name and the sort of acronym for this program?

Erin [0:01:41]: Was it the philosophy of the research programs?

Erin [0:01:44]: How'd did you get started?

Serena [0:01:45]: Yeah.

Serena [0:01:45]: I would say, as much as we love the branding and the fun we have with that.

Serena [0:01:49]: We certainly we certainly started with the philosophy, and we looked at clear goals that we wanted to accomplish.

Serena [0:01:56]: We knew, you know, this so so important.

Serena [0:01:58]: To understand our goals and the ecosystem we were in and, make sure we really grounded it in in research and, in our space first.

Serena [0:02:09]: So, you know, that meant supporting not just researchers, but also designers and content and project managers.

Serena [0:02:15]: Really, we wanted to create something that was collaborative and fully embedded in our team's work.

Serena [0:02:22]: So we certainly started with the philosophy and our objectives first.

Serena [0:02:26]: Yeah.

Erin [0:02:27]: Yeah.

Erin [0:02:27]: Great.

Erin [0:02:27]: And then how did you get to the name?

Erin [0:02:29]: When did that come to be?

Serena [0:02:31]: So after we had gone through the the steps, I think we're gonna go through later.

Serena [0:02:34]: Yeah.

Serena [0:02:35]: We saw these objectives, and we wanted to make it memorable and and we wanted to brand it.

Serena [0:02:40]: And we we had looked at other programs that had names like fast, and, I think quick quick pulse and things like this.

Serena [0:02:48]: So we needed something that was similar to convey.

Serena [0:02:51]: Mh.

Serena [0:02:51]: The speed that we were going for.

Serena [0:02:53]: You know, and Swift is Taylor Swift pretty big at the moments.

Serena [0:02:57]: We thought it was, like, great a great opportunity to, marry the two.

Serena [0:03:01]: Yeah.

Erin [0:03:02]: Yeah.

Erin [0:03:02]: Fantastic.

Erin [0:03:02]: And I think that's a great point because I know researchers is overcoming, some I don't know about Microsoft, but in other organizations.

Erin [0:03:09]: Maybe a perception of slowing things down, and I think there's been a real emphasis on, no.

Erin [0:03:14]: Actually done well, it can really help speed things up, which isn't to say all research is fast, but that it it's ultimately gonna help, move move things forward quickly.

Erin [0:03:23]: So I I love that branding.

Erin [0:03:25]: You have three different at this stage research programs at Microsoft, and I know that you have a five step method that you've used across each of these programs to help ensure their success.

Erin [0:03:38]: So maybe if we could start by just a brief overview of what are those three programs that you have an existence now?

Samira [0:03:45]: Yeah.

Samira [0:03:45]: And I can take this one.

Samira [0:03:46]: So within the swift search lab, we've developed three core.

Samira [0:03:50]: Programs right now, and we're always evolving.

Samira [0:03:53]: So there might be more programs in the future.

Samira [0:03:56]: But right now, we've got three.

Samira [0:03:57]: So the first one is the citizen content program and it focuses on terminology studies to ensure that the language that we use in our products.

Samira [0:04:06]: Resonates with our user.

Samira [0:04:08]: So it can be feature naming.

Samira [0:04:10]: It can be any copy that is there to describe something we're looking at that content.

Samira [0:04:16]: The second program is called the research engine, which is it's designed to streamline and accelerate our research processes and making it easier to gather and analyze user insights, so we do a lot of usability testing on features and on our product and we look at insights across products.

Samira [0:04:35]: So that's what the research engine does with a fixed cadence.

Samira [0:04:39]: And then the third program that we have is the heuristic review program, and that aims to identify usability issues early on, in the design process.

Samira [0:04:49]: So, you know...

Samira [0:04:51]: It's not a fully baked idea yet, you're thinking of something as a designer.

Samira [0:04:54]: But do you wanna ensure consistency with existing Ux principles, and you wanna evaluate that against, you know, what we already have in our products as well.

Samira [0:05:04]: Through a heuristic lens, and that's what this program does.

Erin [0:05:08]: Yeah.

Erin [0:05:08]: Wonderful.

Erin [0:05:08]: So three really discrete programs.

Erin [0:05:10]: One sort oriented on Ux copy and in the content and that side of things, another on...

Erin [0:05:16]: Streamlining processes around usability studies and another on the heuristic side.

Erin [0:05:22]: So that's great to hear about the the variety of different programs that you've been able to establish.

Erin [0:05:27]: So you have three very different programs, but I know that you have that common method that you have developed to create each of them and potentially, you know, future programs as well.

Erin [0:05:37]: So I'd love to hear about what are those steps, you know, and maybe how you've applied them to some of these programs, and even how you came up with these steps, whoever want to jump in.

Jackie [0:05:47]: Yeah.

Jackie [0:05:47]: I can start and then Nursery feel free to comment at any point.

Jackie [0:05:51]: And I think we'd all wanna say to these steps were anything but linear as we were going through an action creating this process.

Jackie [0:05:58]: Right?

Jackie [0:05:59]: But as we reflect back on, you know, what would be the best practice, what did we ultimately do, This is the very cleaned up way that someone could apply these steps to create a research program like the ones that we have?

Jackie [0:06:10]: So the first one we like to call research the research.

Jackie [0:06:14]: And this is once Serena gave a dot to write of looking at what other programs exist.

Jackie [0:06:19]: Right?

Jackie [0:06:20]: So we looked internally at Microsoft and we wanted to see what has already been done.

Jackie [0:06:23]: We don't need to reinvent the wheel and what made those pop research programs either sixteen or maybe not fail, but maybe Peter out.

Jackie [0:06:32]: Right?

Jackie [0:06:32]: So things can change over time.

Jackie [0:06:33]: So we did some internal research, and then we also looked externally it through Ux our specific literature I think this is where we earlier were a little bit surprised that there wasn't more that was formally documented about creating research programs.

Jackie [0:06:47]: Right?

Jackie [0:06:48]: And so that's what we're hoping to do with this effort is what can we maybe pull?

Jackie [0:06:52]: From other disciplines and what we done we've done and others have done to kinda summarize, hey, this is a really effective way to create a structured research program.

Erin [0:07:02]: Mh.

Erin [0:07:02]: Great.

Erin [0:07:03]: So research the research

Jackie [0:07:05]: from Well, they you might have a program too, so we did where with that heuristic program that Samira mentioned, where you do an extra deep dive.

Jackie [0:07:12]: And so that's where somewhere we've had a lot of different discussions, a lot of different heuristic exist out there?

Jackie [0:07:17]: So beyond just researching the program itself, there might be something more that you wanna looking into.

Erin [0:07:22]: Yeah.

Erin [0:07:22]: I was gonna ask you to go into this with an idea of what program or programs you might want to build or you, you know, starting broad when you're researching the research And then the programs kinda follow, you know, from that process.

Jackie [0:07:35]: Yeah.

Jackie [0:07:35]: That's a great question.

Jackie [0:07:36]: And I can start.

Jackie [0:07:37]: I I'd love to hear what Others say.

Jackie [0:07:38]: But they think we're starting with a little bit of research with our stakeholders.

Jackie [0:07:41]: Right?

Jackie [0:07:42]: And carry what are their needs and trying to see, like, what studies have they brought to the embedded researchers that might fit the prioritization matrix for us swift team could take over.

Erin [0:07:53]: Mh.

Jackie [0:07:53]: What would anybody else say on that point?

Serena [0:07:56]: Like also where we already have strengths employed that, you know, where where the gaps and where can we have the most impact.

Serena [0:08:03]: So certainly not from zero, but pretty kind of high level to start and then narrowing it down.

Serena [0:08:09]: Yeah.

Serena [0:08:09]: So I describe it.

Serena [0:08:10]: Yeah.

Samira [0:08:11]: Okay.

Samira [0:08:11]: And also, I would add to that is the way, for example, the terminology program started was because they were coming to us already.

Samira [0:08:18]: Content folks were coming, and they were asking for these...

Samira [0:08:21]: They were requesting for certain studies, and then we were doing it.

Samira [0:08:25]: And I was like, okay.

Samira [0:08:26]: Why don't we just make this into a program that is more formal, and there's a process.

Samira [0:08:30]: And but we just didn't know how that would work.

Samira [0:08:34]: So that's why we did all the work to put that together.

Erin [0:08:38]: Yeah.

Erin [0:08:38]: Great.

Erin [0:08:38]: And it feels like all of these programs, they have a different cadence a different frequency.

Erin [0:08:42]: There's lots of differences, but maybe a common thread is they're ongoing.

Erin [0:08:47]: Right?

Erin [0:08:47]: There's not a flash in the pan one off.

Erin [0:08:50]: And so when you're seeing a need for something on the repeat that might be a good signal that there could be a a program.

Erin [0:08:55]: That could benefit everybody here?

Erin [0:08:58]: Alright.

Erin [0:08:59]: So first, we're gonna research the research.

Erin [0:09:01]: What's our second step?

Jackie [0:09:02]: Yeah.

Jackie [0:09:02]: So then building relationships to really, make sure others are aware of these programs right?

Jackie [0:09:08]: So they start taking us itself on these offerings, so our key stakeholders were designers, Pms, and then you've also got those silent influencers.

Jackie [0:09:15]: Right, which could be other teams.

Jackie [0:09:17]: It might be marketing for the study Samira mentioned where we're naming something.

Jackie [0:09:21]: You wanna make sure they're in the loop.

Jackie [0:09:23]: And working with the other researchers on our team as well.

Jackie [0:09:26]: So that can be really important to get those champions who are saying, look, this program exists.

Jackie [0:09:32]: It's super valuable, and they're actually going out and proactively showcasing it at their Pm meetings can be really effective.

Erin [0:09:39]: Is there a particular time in the life cycle of building these programs where it's most or especially important to foster those relationships in the idea radiation phase later on throughout always.

Jackie [0:09:53]: Yeah.

Jackie [0:09:53]: Always.

Jackie [0:09:53]: But I'm curious for for serena and and we are who were there at the very beginning depending on what that looked like any thoughts on that.

Serena [0:10:01]: Yeah.

Serena [0:10:01]: Absolutely.

Serena [0:10:01]: I mean, you're part of this ecosystem within your organization and and these people within it, make up the culture and how it happens.

Serena [0:10:09]: And if you're not putting your work within that, I think there's a high chance of failure because, you know, everyone else has other objectives and other things going on, And if you want your stuff to fit and work within that, you better, like, understand that space and include those people.

Serena [0:10:23]: I think it's such a under underrepresented goal that you need to pay attention To you know, that's socialization, that understanding of the world you're in.

Serena [0:10:32]: Right?

Erin [0:10:33]: Yeah.

Erin [0:10:33]: Hundred percent.

Erin [0:10:34]: That's been been one of my huge learnings doing this podcast is that internal stakeholder research is as if not more important than the external in terms of making sure the research gets used.

Erin [0:10:46]: Right?

Erin [0:10:47]: And that it...

Erin [0:10:47]: That it's effective.

Samira [0:10:49]: Yeah.

Samira [0:10:49]: And also, like, if you're trying to build a program and make sure that it's successful.

Samira [0:10:52]: You want to build that trust so that people keep come back to you.

Samira [0:10:56]: And so it's it's not just...

Samira [0:10:59]: Okay.

Samira [0:10:59]: You know, We did this study, it was awesome.

Samira [0:11:01]: Thank you for your engagement But it's like, okay.

Samira [0:11:04]: We want you to come back and how is that gonna happen unless we don't foster these relationships And and a program is basically dependent on repeat business.

Erin [0:11:14]: Yeah.

Erin [0:11:14]: Yeah.

Erin [0:11:15]: Absolutely.

Erin [0:11:15]: Awesome.

Erin [0:11:16]: Okay.

Erin [0:11:17]: So two steps so far.

Erin [0:11:18]: Research through research, foster relationships, Jackie with step number three.

Jackie [0:11:23]: Create infrastructure and temp ties, and this is really a key aspect of making us move so swiftly.

Jackie [0:11:29]: Is having a brief that, you know, our stakeholders can fill out and we'll collaborate along with them of having prepared screener for our different personas, so we already have questions that we can tailor to whatever the study is same thing for guides and same things for reports.

Jackie [0:11:46]: And I wanna to emphasize too we're reviewing every step to make sure it fits the context of the specific study, but because we have us all set up, it enables us to move pretty quickly and continue delivering at quality.

Jackie [0:11:57]: And then on the back end to that infrastructure of where we save our reports and we save all of our insights allows us to know query that to kinda summarize and s that research.

Erin [0:12:09]: Yeah.

Erin [0:12:09]: Great.

Erin [0:12:09]: And you mentioned before the...

Erin [0:12:11]: These...

Erin [0:12:11]: As you were doing at these steps were not necessarily linear, but now that you have and looking back on it, set up these steps.

Erin [0:12:19]: Are they...

Erin [0:12:19]: Is this the order in which you kind of go through them or not so much?

Jackie [0:12:24]: Yeah.

Jackie [0:12:24]: Great question.

Jackie [0:12:25]: I think this she might not be so linear in terms of an order, and it might just be evolving throughout these conversations.

Jackie [0:12:32]: Right?

Jackie [0:12:32]: So say, as you're researching the research, you might find out what metrics other programs have used and already start having that discussion of she that's gonna be our next step, Spoiler alert measure success is, is this how we wanna be measuring success too, or you might get a template from another program or see how are they fostering relationships with stakeholders?

Jackie [0:12:51]: Do they have?

Jackie [0:12:52]: Like, a sharepoint site that they direct people to you know, what does that look like for us.

Jackie [0:12:56]: So I I guess, I would say all of that can kinda happen at once, but does anybody else see priority to these steps?

Serena [0:13:03]: No.

Serena [0:13:03]: I think the the world world's messy.

Serena [0:13:05]: And you try to go through the steps, but certainly, some of them can go...

Serena [0:13:10]: Depending on your situation what's happening can go on for a long time.

Serena [0:13:13]: Some can be quicker and and smoother.

Serena [0:13:15]: I think it just depends like, like, every researcher will tell you.

Erin [0:13:21]: Yep.

Erin [0:13:21]: So create infrastructure templates empathize, and this is, of course, only possible because you are working with a program.

Erin [0:13:27]: Right?

Erin [0:13:27]: Like, to your point Jackie, you could have templates that span all of research, but then you would have to really reconfigure them.

Erin [0:13:35]: For each thing.

Erin [0:13:36]: So as you're saying, you have a nice base to build off of, and then you're customizing them for each individual study.

Erin [0:13:42]: Exactly.

Erin [0:13:43]: Alright.

Erin [0:13:45]: So we've been through three steps, research to research, faster relationships, create infrastructure and temp ties Number four.

Jackie [0:13:52]: Number four, measure success, and this is really critical and where things can get spicy if you're not thoughtful.

Jackie [0:13:57]: Right?

Jackie [0:13:57]: Is you don't wanna tie yourself to metrics that are just sample and don't make sense for your actual program, So thoughtfully, the metrics are tied to those swift principles.

Jackie [0:14:08]: So thinking about Swift is that acronym supportive is one of the ways we think about are we being successful, so we're are we getting high engagement from our stakeholders, wide is the w?

Jackie [0:14:19]: Are we getting broad, domain participation across all of our products.

Jackie [0:14:23]: Impactful, we do have post study survey.

Jackie [0:14:26]: So we wanna make sure we're getting satisfaction and that in that survey Our partners are saying they'd like to use the program again and the results will be impactful.

Jackie [0:14:35]: Fast.

Jackie [0:14:36]: We have set cadences.

Jackie [0:14:37]: We make sure those studies run on time.

Jackie [0:14:39]: Same is wonderful about setting those timelines and making sure we stick to them.

Jackie [0:14:43]: And then timely, we're also evaluating all of these projects that come in is are they aligned to the highest priority needs of the organization at that point?

Jackie [0:14:52]: And then there's a piece of socializing all these things.

Jackie [0:14:54]: Right?

Jackie [0:14:55]: Is making sure that we're sharing these results in meetings in a bi or newsletter.

Erin [0:15:00]: Great.

Erin [0:15:00]: And are those measures of success also something that are you kind of c creating those with your stakeholders Or, I know it can be, you you mentioned can get spicy, but, you know, landing on what those measures of success should be?

Serena [0:15:13]: Yeah.

Serena [0:15:13]: I think it's it's all of the above.

Serena [0:15:14]: Right?

Serena [0:15:14]: We're also socializing those metrics and getting feedback.

Serena [0:15:17]: It's also based on the research that we've done and what we we think it should be.

Serena [0:15:22]: So a combination so a bit of c creation as well as, you know, basing it on the data we have and where we think we should go.

Erin [0:15:30]: Yeah.

Erin [0:15:30]: Alright.

Erin [0:15:31]: So research the research, foster relationships, create infrastructure and, temp ties, measure success, and then take us home with step five.

Jackie [0:15:39]: We'll round it off with continuously great And so, like Serena is was saying, we're constantly checking in with stakeholders throughout studies and each of these steps we've gone through, we've continuously improved our templates, the way we build relationships who we see as our stakeholders when we loop them into your question, Erin, and also looking, of course, hot top topic right now, Ai, looking at how Ai tools can help us make these processes even more efficient while maintaining rigor.

Jackie [0:16:06]: Right.

Jackie [0:16:06]: So all these kind of pieces we're playing with all the time and making sure that we're staying relevant as organization is evolving too.

Erin [0:16:14]: Yep.

Erin [0:16:14]: Alright.

Erin [0:16:15]: Fantastic.

Erin [0:16:15]: So a great five step framework that I'm sure many people listening can apply right away to either existing programs and improving those or to building new ones as well.

Erin [0:16:26]: I'd love to dive into a little more detail on some of the programs that you have launched over these last couple of years.

Erin [0:16:34]: What has been some of the impact that you have been able to observe from those programs?

Samira [0:16:40]: So I would say the impact has been quite significant in just two years, the swift research lab has been turning out at least one research activity or deliverable every one and a half weeks.

Samira [0:16:51]: And what's more?

Samira [0:16:53]: It's been hitting a hundred percent satisfaction, create with our designers, content designers and product managers.

Samira [0:16:59]: And so I would say that, you know, we're pretty happy with with yeah that impact.

Samira [0:17:04]: And what's also really exciting about these programs is that it's helped not only prioritize design improvements, but also simplify workflows and create user experiences that just feel more and effective.

Samira [0:17:20]: Right?

Samira [0:17:20]: Like, that's the overall goal.

Samira [0:17:21]: Let's build better products.

Samira [0:17:23]: And these...

Samira [0:17:24]: Our programs just helping us get their a bit faster in a more streamlined way, helping us take us go through that process in a little more painless way.

Samira [0:17:33]: And another big win, I would say would be the ability to catch those unplanned high priority research

Erin [0:17:41]: request.

Erin [0:17:41]: Mh.

Samira [0:17:43]: So, you know, the ones that pop up last minute, and our other researchers can't take on because they're already committed to other plan work, and so Swift gives that flexibility to the stakeholders.

Samira [0:17:54]: And I would add another one to add and few free jack so you had to add this as well, other Since it's a centralized program that run studies across multiple product areas, we're actually able to spot cross cutting insights, and those patterns or issues that you probably miss if you were only looking at one product at a time.

Samira [0:18:16]: So it give gives us this broad more strategic view of product improvement, which we wouldn't be able to do if we're looking at, you know, if we're, existing in silos.

Samira [0:18:27]: So the impact has also been breaking down those silos a little bit more and bringing us closer to our users and building a more cohesive product.

Erin [0:18:37]: Awkward interruption.

Erin [0:18:37]: This episode of Awkward Silences like every episode of Awkward Silences is brought to you by User Interviews

Carol Guest [0:18:44]: We know that finding participants for research is hard.

Carol Guest [0:18:46]: User Interviews is the fastest way to recruit targeted, high quality participants for any kind of research.

Carol Guest [0:18:52]: We're not a testing platform.

Carol Guest [0:18:53]: Instead, we're fully focused on making sure you can get just in time insights for your development, business strategy, marketing and more.

Erin [0:19:02]: Go to user interviews dot com slash awkward to get your first three participants free.

Erin [0:19:07]: Those are some great wins.

Erin [0:19:09]: That that's awesome to here.

Erin [0:19:11]: And so you kind of started with First of all, your internal stakeholders are happy, and they want to keep coming back to you and so I imagine that this feels really good for everybody involved.

Erin [0:19:22]: And then secondly, you're making the user experience, better, which is, of course, the the purpose of of research at the end of the day.

Erin [0:19:30]: Right didn't.

Erin [0:19:31]: And then third, you're bringing those insights across, right, across products across kinds of research which kinda of brings us back to that, researching the research.

Erin [0:19:41]: Right?

Erin [0:19:41]: Like, having that sort of meta view of of the research that's happening.

Erin [0:19:46]: And so, all sorts of wonderful things.

Erin [0:19:48]: Wonderful to hear.

Erin [0:19:50]: Anything to add Serena or Jackie.

Serena [0:19:53]: I think also the the branding component, it's interesting to see how people latch onto to that and data and have, like, ways to describe our team and what we're doing in our our lab, and what's going on.

Serena [0:20:05]: I think that's been a huge win.

Serena [0:20:07]: It's, you know, a lot...

Serena [0:20:08]: It's an exciting, like, visual element that makes it more memorable, and I think that's that's also been a big win, and we've seen the impact of that just people knowing our team and what we're doing.

Serena [0:20:18]: Yeah.

Erin [0:20:20]: Right.

Erin [0:20:20]: So I think any any team in any company right you want to, feel seen and, it people know what we do around here, Right, particularly when, again, the work is dependent on others to carry out and to do something with, and so it's...

Erin [0:20:35]: That's wonderful wonderful to hear.

Serena [0:20:38]: It's also a bit of our purple purple cow.

Serena [0:20:40]: I think it's the same?

Erin [0:20:42]: Yes.

Erin [0:20:42]: From there.

Serena [0:20:43]: Is it...

Serena [0:20:43]: I think it's Seth God, please.

Serena [0:20:45]: Change the

Samira [0:20:46]: the audio with Rebecca.

Serena [0:20:48]: Yeah.

Serena [0:20:48]: Yeah.

Serena [0:20:48]: Yeah.

Serena [0:20:49]: And so, know, something memorable and something different, and I think those.

Serena [0:20:52]: Cool.

Erin [0:20:54]: Yeah.

Erin [0:20:54]: Absolutely.

Erin [0:20:54]: Absolutely.

Erin [0:20:55]: Yeah.

Erin [0:20:55]: Of that.

Erin [0:20:56]: Alright.

Erin [0:20:57]: Biggest piece of advice for someone looking to build and impact full research program of their own.

Serena [0:21:05]: Yeah.

Serena [0:21:05]: I think the biggest piece of advice kinda goes back to what I think we started talking about, which is goals, getting your goals nailed down and making sure those goals are within context.

Serena [0:21:15]: To your organization.

Serena [0:21:16]: I think that's that's the key thing.

Serena [0:21:18]: And and I think the cool part about this is, you know, researchers are uniquely gifted to figure out to understand their ecosystem in their org.

Serena [0:21:27]: And to research and figure out what their goals should be.

Serena [0:21:31]: Like, This is what we do in our in our day job, so we can take those same skills and apply it to a program in the same way, and I think that...

Serena [0:21:39]: That's the most important element.

Serena [0:21:41]: So it it lends really well, I think.

Serena [0:21:42]: To our space and and the skills that we naturally bring as researchers.

Erin [0:21:47]: Yeah.

Erin [0:21:47]: I guess reflecting back, Ray, and you have talked about how these steps have been, again, iterative.

Erin [0:21:53]: One of the steps is.

Erin [0:21:55]: Right?

Erin [0:21:55]: To continuously improve.

Erin [0:21:56]: Right?

Erin [0:21:57]: So they've meta been a point what happened been some of your surprises, biggest learnings as you've gone through this process.

Samira [0:22:05]: For me, it's like, how big we've become.

Samira [0:22:06]: I did...

Samira [0:22:07]: Didn't when we plus studied, it it was just sitting and I, mean the trenches just doing the studies one after the other.

Samira [0:22:15]: And now, you know, we have we've grown...

Samira [0:22:18]: We have more researchers.

Samira [0:22:19]: We have more programs.

Samira [0:22:21]: We have so many more stakeholders.

Samira [0:22:23]: And it keeps growing.

Samira [0:22:25]: Right?

Samira [0:22:25]: It's not static.

Samira [0:22:26]: And that's been both exciting and challenging in in a good way.

Samira [0:22:32]: And so I'm just surprised by how big we've become in such a small

Erin [0:22:37]: amount of time.

Erin [0:22:37]: Yeah.

Erin [0:22:38]: That's great.

Serena [0:22:39]: I was surprised at...

Serena [0:22:40]: Like, I don't know.

Serena [0:22:41]: I pleasantly surprised at the the positive feedback we got.

Serena [0:22:45]: I think, you know, we we built it in a way that we're supporting.

Serena [0:22:48]: That's the s.

Serena [0:22:49]: First one supportive.

Serena [0:22:51]: And I think that was a great decision because, you know, we were able to show how we can help other teams.

Serena [0:22:58]: And I think that that naturally made them very receptive and excited about the program and just so I was I was just a bit surprised by just the positive feedback we got.

Serena [0:23:09]: Mh.

Serena [0:23:10]: But based on that kind of positioning.

Serena [0:23:12]: Yeah.

Serena [0:23:13]: So that was that was a very pleasant surprise.

Serena [0:23:15]: Yeah.

Erin [0:23:16]: Yeah.

Erin [0:23:16]: Nice.

Erin [0:23:17]: Nice.

Erin [0:23:17]: Jackie, any, surprises or or advice to others that are looking to build programs.

Jackie [0:23:24]: Yeah.

Jackie [0:23:24]: It's something that was both surprising, but I think is also per the formula of this program success is how collaborative this team is.

Jackie [0:23:31]: So even if Someone is sign to one of the programs or one of the studies.

Jackie [0:23:36]: There's still a lot of, you know, like, let's run the report by each other.

Jackie [0:23:40]: So I can come up with an initial, like, report right, and then Mary comes in and says, but what's a big picture story.

Jackie [0:23:46]: You know, let's rework this together.

Jackie [0:23:48]: So there's just a ton of collaboration to strengthen each step of the process for all of these programs.

Jackie [0:23:54]: So I I think that's definitely a formula for success and it...

Jackie [0:23:57]: It's not something you always experience.

Jackie [0:23:58]: Right?

Jackie [0:23:59]: Because some people think about it It's more efficient to kinda silo responsibilities, but I think we maintain a lot of quality and still that efficiency by being so collaborative.

Erin [0:24:09]: Yeah.

Erin [0:24:09]: Yeah.

Erin [0:24:09]: Awesome.

Erin [0:24:10]: And actually, if if you're able to share, I'm I'm curious about some of how you work together and manage these programs.

Erin [0:24:16]: So you all are on a team with lots of other researchers or certainly at Microsoft.

Erin [0:24:22]: But...

Erin [0:24:22]: Yeah.

Erin [0:24:23]: I how do you exist in the...

Serena [0:24:25]: We kind of have a hybrid approach.

Serena [0:24:27]: We...

Serena [0:24:27]: Where we have embedded researchers.

Erin [0:24:29]: Yep.

Serena [0:24:30]: In products.

Serena [0:24:30]: There's also...

Serena [0:24:31]: Teams that we work with that, have more of a centralized approach.

Serena [0:24:35]: And so...

Serena [0:24:36]: But in our case, mostly with the researchers we work with, They're pretty embedded in a product.

Serena [0:24:41]: Mh.

Serena [0:24:42]: And and so the Swift research team will work with that embedded researcher, to, you know, leverage the immense amount of knowledge they have in that domain, especially in a very technical domain.

Serena [0:24:53]: It's very important.

Erin [0:24:55]: Right.

Serena [0:24:55]: To build off their work to be, you know, good researchers in terms of, you know, not redoing work, not starting from scratch, you know, standing on the shoulder of those giants.

Erin [0:25:06]: Yeah.

Serena [0:25:06]: And and building off their work.

Serena [0:25:07]: So I think that's where the collaborative miss comes is is working so closely with those embedded researchers.

Erin [0:25:14]: And do each of these programs?

Erin [0:25:15]: Is there does someone own the program or how do you keep them running?

Erin [0:25:18]: Like, who's involved with with that structure?

Samira [0:25:21]: So I run the day to day of the program, and so I'm kind of, like, in charge of things happening and the time since, I have great support with Jackie.

Samira [0:25:32]: She's our full time researcher in Swift as well.

Samira [0:25:35]: And then we have another researcher who works on a specific part of the product, but through Swift.

Samira [0:25:41]: So we're running these three programs consecutively and in parallel.

Samira [0:25:47]: So it's just about kind of juggling who's got the time.

Samira [0:25:50]: Mh.

Samira [0:25:50]: Kind of all, you know, for example, Jack, he does a great job with terminology studies.

Samira [0:25:55]: So if there's something that comes along, I would always see if if Jackie is available to do that one.

Samira [0:26:00]: And then we have heuristic reviews, which is great for us to kind of...

Samira [0:26:04]: There there should be at least two researchers on one heuristic reviews.

Samira [0:26:08]: So we would...

Samira [0:26:09]: There would be one swift researcher and we try and get another embed...

Samira [0:26:13]: A researcher or maybe for that specific area to to bandwidth with us.

Samira [0:26:16]: So Mh.

Samira [0:26:17]: We kinda, like, you know, keep it going with all the collaborative.

Samira [0:26:20]: Like...

Samira [0:26:21]: And that's why the relationships come back into the picture as well because we're also leaning on other researchers in our team to keep the program of afloat going.

Erin [0:26:32]: Yeah.

Erin [0:26:32]: I imagine it would vary quite a bit depending on the structure of how research works in any organization, but that having someone to kind of own each program or to keep it running alongside stakeholders who are invested in its success would be important to to keep it running.

Serena [0:26:49]: From a manager point of view, what surprised me the most is the pride.

Serena [0:26:53]: It brought to my researchers that we're working the space.

Serena [0:26:57]: They had a program.

Serena [0:26:58]: They had something that belonged to them and that they could drive and really own, and I think that was an amazing thing to see and I was pleasantly surprised with that.

Serena [0:27:09]: For sure.

Erin [0:27:10]: Oh, Yeah.

Erin [0:27:10]: That's wonderful.

Erin [0:27:11]: It's great to hear.

Erin [0:27:12]: Sounds like lots of the...

Erin [0:27:15]: You talked the s is supportive, and it sounds like you all had confidence that this would be helpful, value add, but the level at which it has been helpful and supportive and the hundred percent satisfaction and, you know, talk people on the team feeling so great about it and all these wonderful things that happened that maybe weren't even part of the initial inspiration.

Erin [0:27:39]: Love to hear it.

Erin [0:27:40]: That's great.

Erin [0:27:41]: Alright.

Erin [0:27:42]: Rapid fire section.

Erin [0:27:43]: So what is and I'd love to hear from any and all of you for any of these who would like to share something.

Erin [0:27:48]: But the first one is your favorite, interview question as an I am conducting a research study and asking a participant a question.

Erin [0:27:55]: What are you...

Erin [0:27:56]: What's your guts see your your favorite questionnaire?

Samira [0:27:59]: I think my mine is definitely a tell me question, like, whatever they, say something.

Samira [0:28:03]: It's more about, like, tell me more about, you know, whatever they're doing?

Samira [0:28:06]: Mh.

Samira [0:28:07]: So just kinda, like, is open enough for them to go in whichever direction, and then just follow up on that bit.

Samira [0:28:13]: Whenever I'm just like, I don't know what's going on.

Samira [0:28:16]: Just ask them like, tell me, and then it just opens up a whole bunch of questions.

Erin [0:28:21]: Yep.

Erin [0:28:21]: What's that.

Erin [0:28:21]: And also...

Erin [0:28:22]: I'm...

Erin [0:28:23]: I think open ended right is a a go to or just add more so on, but I like to tell me emphasis you shared there of...

Erin [0:28:31]: I'm a person you're a person tell me I wanna hear.

Erin [0:28:33]: Right?

Erin [0:28:34]: It's kinda communicates I.

Erin [0:28:35]: I'm an open receptacle.

Samira [0:28:37]: Yeah.

Samira [0:28:37]: It it's also because of we're in such a technical space that sometimes I don't know when they're talking you about.

Samira [0:28:44]: Telling them I'm like...

Erin [0:28:48]: You to tell me more.

Samira [0:28:49]: Yeah.

Samira [0:28:49]: I know I don't know what you're saying right now.

Erin [0:28:54]: Right.

Erin [0:28:54]: Right.

Erin [0:28:54]: Yeah.

Erin [0:28:55]: On that.

Serena [0:28:56]: Yeah.

Serena [0:28:56]: I think for me, I like going down the route of, the last time you did an action, making them really ground it in the last time because you can see, know, from their kinda of more generic responses is how it changes how they have to go into a moment and reflect on it and and how much more richer that can be and and Mh.

Samira [0:29:15]: A

Serena [0:29:15]: lot more fun to kinda follow that path.

Serena [0:29:17]: So that's kinda of my go to, I think is Yeah.

Serena [0:29:19]: The last time you did something specific.

Erin [0:29:22]: Yeah The last time you're getting a recent and good memory and...

Erin [0:29:25]: Yeah.

Erin [0:29:25]: How's that.

Erin [0:29:26]: Yeah.

Erin [0:29:26]: Yep.

Jackie [0:29:27]: Oh, this is less, an interview question, but more one of those moments that I enjoy when they happen is when a participant asks you a question.

Jackie [0:29:34]: And you're...

Jackie [0:29:36]: You just have to say, you know, I'm.

Jackie [0:29:37]: I'm gonna frustrate you sometimes.

Jackie [0:29:38]: But, what if you ask yourself that question, you know, to

Erin [0:29:43]: turn it back.

Erin [0:29:44]: Feel.

Jackie [0:29:44]: I can't answer that for you.

Jackie [0:29:45]: Right?

Jackie [0:29:45]: Because we're looking your perspective.

Jackie [0:29:47]: I think that's funny when that happens.

Erin [0:29:50]: Yeah.

Erin [0:29:50]: Interesting.

Erin [0:29:51]: Okay.

Erin [0:29:53]: A few resources that you would recommend to folks, books, websites, people to follow anything along those lines.

Jackie [0:30:00]: Well, the mac's design library for, if that's okay.

Jackie [0:30:03]: Just because I totally love it.

Jackie [0:30:05]: It's one of the heuristic that we use for heuristic evaluations, and we also apply those heuristic if we're doing studies, and it's specifically about human Ai interact.

Jackie [0:30:15]: And principles.

Jackie [0:30:15]: Mh.

Jackie [0:30:16]: And just the website is really great to use in terms of you can, like, click on different, principal, different heuristic and seek examples.

Jackie [0:30:24]: So really great when you're doing a study with Ai features to think about is this following these best practice principles.

Erin [0:30:32]: Great.

Erin [0:30:32]: One.

Erin [0:30:32]: And we'll link all of this in the show notes for everybody, but it's...

Erin [0:30:35]: Yeah.

Erin [0:30:35]: Microsoft dot com slash hacks tool.

Erin [0:30:38]: I think we'll take you there, but we'll we'll link that in the show notes.

Serena [0:30:41]: So shout out to that team at Microsoft research to put in a lot of awesome work putting that together, way before kinda, Ai recently blew up.

Serena [0:30:50]: So very forward looking, I, concept and and we're playing we're getting dividends from it right now for sure.

Erin [0:30:58]: Yeah.

Erin [0:30:58]: Awesome.

Erin [0:30:58]: Yeah.

Erin [0:30:58]: I'd love to see when folks share their tools with with the community for the better vault.

Erin [0:31:04]: Creates to you.

Erin [0:31:05]: Alright.

Erin [0:31:07]: What else?

Serena [0:31:08]: I went...

Serena [0:31:08]: So this is a good plug.

Serena [0:31:10]: We have a book coming out hopefully, early next year, and there'll be a chapter, Jackie and Sphere and I wrote it on, doing research programs.

Serena [0:31:19]: So, basically, what we talked about here, but way more in depth a lot more detail.

Serena [0:31:23]: Mh.

Serena [0:31:23]: Would highly recommend when that book comes out.

Serena [0:31:25]: So it's on case studies, valued of case studies in H sci in the wild, would highly recommend that book.

Erin [0:31:33]: As Awesome.

Erin [0:31:33]: When does...

Erin [0:31:34]: When does that come out?

Erin [0:31:35]: Where can folks get it?

Serena [0:31:36]: Man.

Serena [0:31:36]: It is supposed to come out, probably, early twenty twenty six.

Serena [0:31:42]: It's through Spring.

Serena [0:31:45]: So it's with spring.

Erin [0:31:47]: Okay.

Erin [0:31:47]: So the book is gonna be published by Spring.

Erin [0:31:50]: It's called Ux research case book, valued case studies to influence product strategy, expected out in early twenty twenty six, and we'll toss the link in the show notes once it's available, but, definitely, everybody I'll look out for that one.

Erin [0:32:05]: And I'm sure there's lots of great stuff in there.

Samira [0:32:08]: And then I would plug for some other media articles that this dean written specifically Jackie and City.

Samira [0:32:15]: And we've listed those out in the doc we shared with you, Erin, but, I think Mh.

Samira [0:32:21]: Those are good resources in general.

Samira [0:32:23]: To kind of, like, even think about so, especially the one about prioritization.

Samira [0:32:29]: Mh.

Samira [0:32:30]: Will this stay knowing when to Nudge neil stage or let go.

Samira [0:32:33]: Like, that's a great one to know which ones would fit well in which program.

Erin [0:32:39]: Mh.

Samira [0:32:39]: Right?

Samira [0:32:39]: Like, what requests should go where and how to how to treat them.

Samira [0:32:42]: If a certain methods is gonna fit or not, or if how are you gonna align the kind of support that your team needs with the prioritization matrix.

Erin [0:32:52]: Yep.

Erin [0:32:52]: So I think that's

Samira [0:32:53]: a really good one.

Samira [0:32:54]: And then, Jackie, you can talk about the other one that you wrote as well.

Jackie [0:32:58]: Yes.

Jackie [0:32:58]: Ai is so hot right now.

Jackie [0:33:00]: Is my favorite say about it.

Jackie [0:33:02]: But, we did really interesting look at using Ai tools that exist for heuristic review seeing, you know, how much could they potentially do independently or with a human in the loop component.

Jackie [0:33:13]: Long story short, we still at this point recommend a human in the loop.

Jackie [0:33:18]: But looking at Ai tools, you know, as kind of like another colleague.

Jackie [0:33:23]: Right?

Jackie [0:33:23]: Is they can really help get you started rather than starting from a blank canvas, which can be the hardest.

Jackie [0:33:29]: Step.

Jackie [0:33:29]: Right?

Jackie [0:33:30]: Is there are tools that exist, including you can try any Ll, You could try C copilot.

Jackie [0:33:35]: You could try chat Gp and you can feed design images through it and feed heuristic through it and ask for evaluations and get a starting point there.

Jackie [0:33:44]: So I'm sure we're gonna see a lot more advancements and then figure out what our piece of it is to review the work of these Ll, and then, yeah, it moved even more quickly cover even more ground.

Erin [0:33:57]: Yeah.

Erin [0:33:57]: Also.

Erin [0:33:57]: Obviously could be could be a completely separate episode, but it feels like the common thread with Ai these days is that it it pretty much has a rule in everything, but how much you can let it go versus how much you need the human in the loop varies quite a bit.

Erin [0:34:15]: And obviously, as an evolving story as well.

Erin [0:34:19]: But I imagine Jackie, you know, with the heuristic, the more you know going into that, Ai support the better off you're gonna be.

Erin [0:34:26]: Right?

Erin [0:34:27]: Some basic heuristic standards.

Erin [0:34:30]: Right to begin with?

Jackie [0:34:32]: Exactly, totally.

Jackie [0:34:32]: And, you know, there's tons of great systems out there.

Jackie [0:34:35]: A lot of folks are familiar with, like, the Nielsen ten usability heuristic we use the Microsoft tenants and traps.

Jackie [0:34:42]: They really, like allow you two zero in on some best practices.

Jackie [0:34:45]: So definitely recommend those two as a resource.

Erin [0:34:49]: Wonderful.

Erin [0:34:49]: Again, we'll link everything in the show notes, and then Where can folks find you.

Erin [0:34:53]: Think Linkedin, I'll make it easy.

Erin [0:34:55]: It looks like Linkedin.

Erin [0:34:56]: They're.

Erin [0:34:57]: It looks like linkedin would be the best place and we'll link everybody, but Serena Hillman, Samira Jain and Jackie Ianni, so you can find everybody there.

Erin [0:35:07]: Thank you so much for joining.

Erin [0:35:09]: This is Been a lot of fun.

Erin [0:35:10]: I've learned a lot.

Erin [0:35:11]: I'm sure everyone listening has as well and congratulations on all of your success for these programs.

Samira [0:35:17]: Thank you for having us.

Serena [0:35:18]: Yeah.

Serena [0:35:18]: Thanks.

Erin [0:35:26]: Thanks for listening to Awkward Silences brought to you by User Interviews.

Erin [0:35:29]: Theme music by Fragile Gang.

Erin [0:35:31]: Hi.

Erin [0:35:42]: There Awkward Silences listener.

Erin [0:35:43]: Thanks for listening.

Erin [0:35:44]: If you like what you heard, we always appreciate a rating or review on your podcast app of choice.

Erin [0:35:51]: We'd also love to hear from you with feedback, guest topics or ideas so that we can improve your podcast listening experience.

Erin [0:35:57]: We're running a quick survey so you can share your thoughts on what you like about the show, which episodes you like best, Which subjects you'd like to hear more about, which stuff you're sick of and more just about you.

Erin [0:36:06]: The fans have kept us on the air for the past five years.

Erin [0:36:10]: We know surveys usually suck.

Erin [0:36:11]: See episode twenty one with Erica alcohol for more on that.

Erin [0:36:14]: But this one's quick and use fully promise.

Erin [0:36:17]: Thanks for helping us make this the best podcast it can be.

Erin [0:36:20]: You can find the survey link in the episode description of any episode or head on over to User Interviews dot com slash awkward survey.