Erin: [00:00:00]
Hello everybody and welcome back to Awkward Silences. Today we're here with Dave Chen, the Director of Consumer Insights at Flip. We're going to talk about market research and surveys and how those aspects of research can actually be really useful to insights and product teams and entire organizations and what that sort of looks like.
So, Dave, thanks so much for joining us. We've got JH here too.
JH: [00:01:04] Yeah, I'm excited. I feel like market research maybe has a little bit of a stigma on it in the user research world. So I'm I'm super curious to hear how how it can be part of the story and and help out.
Erin: [00:01:13] Fantastic. So, Dave, you lead an insights team that includes customer experience, AKA CX, a UX research, AKA UXR. We love our acronyms and market research. So just to kind of start off, how do you think about the role of each of those in the context of, you know, a larger insights team?
Dave: [00:01:34] Yeah, for sure. so for CX, the way that I think about it, CX breaks down to two parts, right? Like I think what people are most familiar with is the operational side of it. So this is where, like, you know, the troubleshooting, the responding to tickets the queue management. That's half the battle. And then the other half, well on the CX side is what we're calling internal education and product influence. And this in my mind is the true value of the CX.
Which is, how do we take the feedback that we get from the users through all kinds of feedback, channels, social, in-app feedback forms, app store reviews, and then aggregate that into themes and then surface that to product leaders in the organization. And so to me, that's an insights play. It's leveraging that data. So CX to me is an intake channel.
So leveraging that data Finding ways to analyze and synthesize the data and then drive actionable results by partnering with internal stakeholders . So that's how we think about CX.
And then, so on, on the research side, our two smaller teams as you said, Erin, so I have the UX research team.
The idea here is really about how do we, through conversations with users and it could be non-users in some cases, depending on the project, how do we understand them better at the human level? So the bottom line here Is to drive that empathy piece. It's really understanding the user or that user through conversation and not data, not numbers.
Where, and this is where it makes it UXR unique in a sense where you can invite stakeholders to these conversations, right? So they could experience and hear some of these pain points or opportunities firsthand.
And then lastly, the market research side. So market research. I do agree with your intro there, that it does carry a lot of connotation or stigma. And we can talk through that for sure.
The way that we think about market research, especially relative to UXR is it's about leveraging a higher sample size to build confidence through more structured data collection methods. And typically this means higher sample but also allows you the opportunity to dive into specific cohorts, through data cuts, analyses and whatnot.
So, so the way that I kind of, I talked to internally with my stakeholders around how you think about UXR versus market research is UXR is about making your heart smart. It's how do you drive empathy through conversation. And market research is about making your head smart. So it's, how do you drive confidence in in the answer or the objective that you're seeking through numbers and more so on the quantitative side of it.
Erin: [00:04:18] Yeah so if CX is sort of like you're saying it's intake, so like in our you know, UX research tools map, we put out, we kind of break research into passive or active, right. So like, is it coming in or am I launching a study? Right. And so that's passive. And then if we say like UXR and market research, are maybe more active. And so is it, are you saying like UXR is basically qualitative and market research is quantitative or is that too simplistic?
Dave: [00:04:47] I don't think it's overly simplistic. That's the way that we think about it at Flip and part of this is with the kind of like the people on my team where their core skill sets lie. So the UX researchers on my team. They are, like by training, a lot of them are like anthropologists or they've done on other things in their past life, you know, qualitative research agencies.
So they know how to execute a lot of these qualitative research methods really well. You know, via user interviews or shop-alongs or ethnographic types of work and they're skilled at interpreting qualitative information. This is where I think UXR is more of an art than science, if you will, because it's not something that, you know, you'd earn a certificate or like do a course.
And all of a sudden you become like a UXR a group. You have to do a lot of practice and there's a lot of feel to it. But I think in comparison market research can be trained. I mean, both, you wouldn't require a lot of training, but market research tend to be more rigid and structured. A lot of methods that falls under what you would consider a traditional marketing research study tend to be more rigid and structured.
And I would say that's where where some of the stigma comes from. Is it tends to be a little bit more rigid, a little bit more slow. It tends to follow a specific flow and is less flexible compared to, I guess, UXR research methodology or qualitative research methodology.
JH: [00:06:21] And so, because you have, so both teams sit under you. So when something that needs research comes up, is it coming to you for intake? And then you decide which team market research or UXR is best suited to take it. Or do those teams collaborate together and decide how we should go about learning about this problem area or this opportunity?
Like, what does that flow look like for you all?
Dave: [00:06:43] Yeah, so it's more the latter. I mean, the team has evolved quite a bit. When I first joined Flip, I was the very first researcher. Like I always joke. I say, like, I'm a patient zero. And I built a team. Oh, that may be a touchy metaphor to use. Now that I think about it.
JH: [00:07:00] You don't want to be patient zero anymore.
Dave: [00:07:02] But right now I'll have a fairly sizable team. And I have different leaders and managers that are dedicated to each functional vertical and work with their with the functional leaders. So for example, the UX researcher are mostly dedicated and embedded in the product design and product org.
So then when the product leaders have a question, they would go to the UX researchers more directly on those questions. And he would then be responsible to figure out the right methodology to use whether it's quantitative or qualitative, if it requires more of. You know, a market research teams help, this is where I think there's the benefit of having one insight that brings the team together to talk about high level insights, where research strategy on a regular basis.
Where we'd be able to have that collaboration across different members on the team to say, you know, we're going to be executing some one-on-one user interviews to get a sense of how users feel about this particular prototype, for example, but it would be cool to, execute a quantitative survey to understand, you know, at scale, how people You know, might feel about specific aspects of this prototype or even at a higher level, maybe more generative broad, overarching types of questions that can support kind of the story or the finding from the UXR side.
JH: [00:08:26] That makes sense. When I hear market research, I do just think surveys. Like, it just seems like I'm going to take a long maybe boring survey because some, you know, it landed in my inbox. I assume there's a lot more to it than that. And I think you touched on some of them, you mentioned alluded to some methodologies earlier, like how do you all think of that part? Like what when you're conducting the market research, how are you actually going about it?
Dave: [00:08:46] Yeah. So I would agree with you. I think when people hear market research, they think surveys and they think focus group. I think those are like the two most common methodologies that people tend to associate market research with. So I will say that surveys it's still today, the most cost efficient way to gather data, at least from a research perspective in terms of scale just because the delivery mechanism is easy to facilitate.
But to your point, it's not the only ones. One of the things that we do, at flip. In addition to what you would consider the, traditional standard market research, or talking to people in the market in the general population is we also manage an insights community.
So the insights community is only flip users. So we recruited them through multiple channels and we incentivize them to participate in the community on a regular basis. And then we would put out what we call engagement activities. Out to the community so that they can help answer questions that we have, whether it's, you know, product-related questions or could it be like a creative thing that we're thinking of.
And then within the community, we do surveys. Yes. For the most part. But we also have other engagement tactics, like forums, where we would put a topic out to the community and we would, we would work with the community members. To get them to engage in the form, ultimately addressing the question that the organization has.
So that to me is also market research. It may not be the traditional sense of the general population research, but it is about understanding the shopper or your customer at the end of the day,
JH: [00:10:29] And how big is that community? Like, I'd imagine again, for market research, I'm going to just go to all my preconceived notions. I assume it, I always think like volumes, you need a lot of people. So is that a very large community, or do you think about the context you do with them a little bit differently.
And when you're talking to just the general users in a market, that's where you really focus on volume and scale.
Dave: [00:10:50] yeah, there is a volume piece attached to it, for sure. John. So, our insights community is about 15,000 people and we divided roughly in half between the us and Canada. As we do operate in both markets and there are unique nuances in both markets that we gather insights from.
So we do, we do shoot for volume here. What's unique about the insights community is because these people are they volunteer to participate in this communities. So their engagement level on a community is higher than what you would typically see in like a general populations panel, or like if you were to work with a third party vendor sort of recruit for general population sample because they are intrinsically motivated to participate in these studies because they are, they feel passionate about the product.
They want to give feedback. So there's a volume piece, but we also leverage community for UXR recruitment too.
Because once you recruit these people into the community they already satisfy certain criteria. Right. Like we already know they're all Flip users. So for example, for one on one user interviews we don't necessarily have to go out to recruit for Flip users. Again, we can look for people inside the community because they are already Flip users and we know they can give us feedback on the product.
Erin: [00:12:07] Cool. So, we talked about some methods outside of focus groups and surveys the kind of stalwart methods folks associate with market research. So let's Let's further establish the stereotype. Let's talk about surveys and focus groups. So, you know, we see a lot of folks running focus groups at user interviews.
They're definitely still a thing. And in fact participants we see a lot of participants sign up coming, you know, say through search, looking to participate in focus groups. It's definitely a method people are aware of. But that, I think that there's probably a good amount of. You know, misconception about.
So, just curious, is that something you all are running a lot of focus groups or have you found, you know, kind of cases where they're very effective for your needs?
Dave: [00:12:53] Yeah, so I'll be transparent up front and say, we personally don't do any focus group or much focus group at all.
There are benefits to the focus group. So for example, I think it is a decently effective way to get, you know, anywhere between six to 12 people in a room and get them to react to an idea that you think aren't like, if we haven't an effective moderator, you can design an activity to engage the group to, you know, to provide opinions, the feedback on the stimuli or the prototype that you were thinking or the topic.
I do personally believe that the downside of the focus groups tend to outweigh the benefit and where I think like one-on-one user interviews, working with surveys, it might be a better form.
So for example, I think focus groups, you tend to get the louder voice in the room to dominate the conversation. And I've seen this a lot, like in my past experience and when I've worked in, you know, in the CPG world, CPG tend to do a lot of focus groups. So what happens is you get typically in the room, the one or two people that are more opinionated, it tend to speak up more.
And then over time, the quieter people tend to shy away of providing their opinions so that the voice in the room tend to skew. So that's one piece, I think the other piece is. Is you need very seasoned, experienced moderators to manage the room well. That's not an easy thing to do. And I haven't come across a ton of really good moderators can really manage the room as effective.
And if not done well you can lose control of the room and it put the kind of like the study or the project at risk because you can't manage the conversation. So those are, I think the downside potentially outweigh the benefits, which I think like one-on-one user interviews, for example, versus one-on-one conversations in general, tend to be a better a better route.
Erin: [00:14:46] Yeah. And then the other surveys you definitely, it sounds like are using surveys. Can you tell us a little bit about how surveys fit into your toolkit?
Dave: [00:14:55] Yeah. So surveys. It also has its cons for sure, but too, in terms of the benefits, in addition to it being more for more effective units in terms of achieving scale, it just it's fast. Right? If you can, if the turnaround is much quicker quantitative methods tend to be more labor intensive. So one piece, the other pieces you need experts.
People who have done this really well to manage what are the room where the ones that are very skilled at doing and having these conversations with the respondent. Surveys you can do a lot of this stuff offline and you can get alignment very quickly and a lot of eyes on the, on this kind of questions where the design of the survey with a small And then you can iterate on it.
If one survey doesn't work, it doesn't take a ton of effort to put out another survey to learn on the topic further. And then you also have a lot of flexibility on the back end after the fact to dive into different cohorts and data, to look at the specific segments within the study to better understand their behavior and deep dive.
So that's how we've been. We think about surveys. So we, the way that we kind of try to leverage all to different methods is triangulation is no one single method is going to give you the silver bullet solution. You have to be able to come at it all sides to kind of, leverage the different data points.
And this includes analytics and data science, by the way, although I don't manage the function, but we do very close to deal with the teams to be able to triangulate the data, to tell the full story.

JH: [00:17:19] I'm just curious since like you're overseeing the consumer insights group and you have these three areas. Just like, what is the most recent thing insights related at the CX UXR and market research teams have worked on, like, if you were going to like, just look back quickly, could you just give a snapshot of like what each group has been up to in recent examples?
Dave: [00:17:37] Yeah, so, so one example is we've been working on without getting into too specific detail, some innovations in a local market. We've identified an opportunity where we can. We can speed up the innovation in a specific local geo where we feel like there's enough traction, that we can test some further ideas in that local market.
So the way the workflow that we've done we first leverage the marketing research team to just put a feeler out around, you know, what are some of the bigger problem spaces that are out there from a shopping perspective. So think like the jobs to be done the types of work we've done some quantitative assessment in the market to say, you know, we've identified these opportunities.
These are white spaces. We feel these jobs are underserved and there's a potential opportunity for us to close the gap by offering solutions ABC. So then we take that insight and we went back to the product team, and product design team. We work on some prototypes. And then we engage the UX research team to say, can we just put these out in front of some users in that local geo to get a sense of, are we on the right track?
And this is like your typical Figma or Invision prototype, but a semi working in a sterile environment. But the point is just don't understand are we on the right track. So we've done a facilitated, you know, facilitated it, but also unfaciliatated user interviews or user tests just to get a sense of. What are some of the things that are working? What are some of the feedback we're hearing from the users based on these prototypes? How can we iterate on these to make them better? And then we take that feedback back. We will work with it, the product team, and then we shipped some features and we actually designed a different environment altogether for those shoppers in that local geo, put the stuff out in market to one.
Get AB test results. And this is where we partner closely with analytics and data science team to actually monitor product engagement, data retention, engagements, content, open rates. So those types of things to see are the features and the product resonating with our users. But also our CX team are able to closely monitor the feedback of the users in that local geo.
So then if something comes back through the CX channel or we have seen some of the feedback from the users, we would then be able to pass that feedback back to to the product team to say, you know, we tested this in research but maybe we missed the mark here. And this is how we close the loop.
If you will in that like end to end user experience or user insights gathering. So that we can take all that feedback back to the product team and we can iterate on it. That's like a, that's a very recent example.
JH: [00:20:35] and just make sure I'm understanding it. So is it pretty common that the CX UXR and market research teams are all kind of helping your understanding of one area simultaneously and they're all, and they're all just coming at it from different angles. And like you mentioned before triangulating on it, or are there cases where, you know, the CX team is interested is exploring this area and getting insights and they're kind of doing their own thing.
Are they always working in unison or sometimes do they go off and gather insights in different areas?
Dave: [00:21:01] There's definitely instances where they're working with it, but it's like to your point, it's not all the time. And I want to say like, not all initiatives and projects require all of the research effort, all of the CX effort all the time. Sometimes for example, like you hear one specific area on existing product feature throw a ton of CX feedback and that's enough. We don't need to do any more research on how people feel about it because we are getting tons and tons of feedback on this particular area. And we will have enough data and evidence to pass back to the product team to say, you know, we need to fix this.
Erin: [00:21:40] yeah, something we talk about a lot on the UXR side is kind of packaging your insights in a way that will be. You know, seen, understood, cared about right by your stakeholders. And I'm curious, you know, especially with such a large, potentially large volume of insight coming in across these three functions, have you thought about, you know, how to synthesize them across CX UXR and market research, and particularly on the market research side, I'm curious, you know, how do you package large volumes of kind of quantitative data in a way that I don't know if you have, you know, resistant stakeholders that are like, eh, market research. I don't know about this stuff, but have you been able to, you know, think about how do you present these insights and findings across the three functions in a way that is, you know, well understood and ultimately acted upon by your stakeholders.
Dave: [00:22:36] yeah, a hundred percent. Absolutely. It's one of the things that we hope we continue to work on even to this day. As we, you know, work with the stakeholders and adapt to you know, their comfort level their knowledge level when it comes to user data or customer data. I would say when I first joined the organization we, we tend to do more.
We used to do a lot of like the data dump. And I don't know how common this is across kind of like the, your other the other people in other research teams. I think there's tends to be a perception is more, is better. Or we want to give people all the data that they can sift through, but to your point, I think a lot of it is how do we figure out the best way to package it and tell the story in a way that it resonates.
So we've certainly scaled that significantly back now to we reserve a lot of that data for our own teams and for people who work like tend to want to get into the details before, you know, our product leaders, our exec team, especially We certainly put in a lot of effort trying to tell the story by triangulating the data from all sides.
So for example, if it's about a specific feature that we need to remember on shopping list, then we will call that specific data points. All shopping lists from the market research study, but then we will also try to accompany that data with your CX feedback as well as in UX research feedback, just to provide them with that verbatim.
And that user sentiment around how the feature can be further improved so that it's not just the quantitative data, because the quantitative data can only give you. The what, but it doesn't tell you as much the how and the why. And this is where like the CX feedback and the UX research, the quality of feedback can can provide a lot of the additional firepower here.
JH: [00:24:33] Do you find that when you're presenting stuff from the different methodologies and things you've sh and you're sharing, do stakeholders, are they able to kind of weave it all together or do some people like fixate on one type or the other? If somebody who's like, I just want to hear from users or show me the numbers.
I want to see what the trends are like. Do people actually take it all in? Or is it each stakeholder kind of like fixates on different results from different methodologies?
Dave: [00:24:55] There's definitely a level of that. I think there's a lot of like people, certain people will gravitate towards certain types of feedback. I think some people really love the quantitative verbatim feedback and they tend to take that away. And forget the, you know, 80% of people said this.
But the one thing that one person said really resonated. So we certainly tried to balance that and try to cater at the top side to to the stakeholder, but also keeping, helping them keep perspective that, you know, if it's an issue that impacts a large amount of users, That we have to we have to take a step back and not get fixated on like the one thing that one person said about this other thing about our product.
I think it's a lot of the ebbs and flows in the conversation is about stakeholder management is about understanding your stakeholder and build the report and finding ways to make sure that the story actually lands and resonates.
And I want to say like, We can't do this ourselves like researchers and research team or insights professional can only do so much. And this is where you have a lot of support within the organization to, you know, I don't know, through product design leaders who tend to be sometimes our closest allies or, you know, through analytics, but it's like some of the product leaders I've worked with probably dozen organizations that are very user centric and they are savvy in the data.
They understand. The difference between quantitative and qualitative data. And it helps a lot to have their support, to be able to you know, stand behind our insights and you know, help with the conversation and help manage the conversation in partnership with my team.
Erin: [00:26:33] Yeah. as you talk about this, I really, I feel like these cross-functional insights teams are really going to take over as how companies organize their insights. It just makes so much sense to, to start from the perspective of what's our question, you know, what's any person in the company's question and how are we going to answer it in, you know, the, how we're going to answer it part is going to have a lot of potential Methods, you know, that might be involved.
And so being able to bring that all together just makes so much sense. You know, you talked about, you know, market research being kind of having an edge over data science or analytics in that. It's literally designed to ask a specific question, which I think is really interesting, you know, it's one of the things that comes up with analytics where it's like, there's all this data everywhere. And maybe the answer to your question is somewhere, but you you're trying to sort of find it from data you're already collecting versus let's proactively go out and answer this question. And maybe market research is our best tool to do that.
And they think it's kind of cool to open up market research as a tool for people to answer questions.
Dave: [00:27:45] Yeah, absolutely. And that's how it works. We're certainly trying to steer the ship wherever we're trying to steer the ship is To educate our internal stakeholders less about on the solution itself is to push them on thinking about like, what is the question that you think you want to get answered?
What ultimately is the objective. And then it reading it's up to the insights leaders to researchers, and then same thing went and they'll partner with analytics and data science leaders to figure out, you know, this is the best method or methods that we can leverage to help address those questions. And then leave it up to us, you have to trust our expertise that we're gonna get you the answer versus, you know, the stakeholders dictating, you know, I want to do a survey to answer the question, ABC.
I think I agree with you Erin that I think that's is where the trend is. Certainly this is the effort that we're trying to, this is what we're trying to do within the Flip organization.