Erin: [00:00:00] Hello everybody. And welcome back to awkward silences. Today. We're here with Amanda Rosenburg. She is a staff UX researcher and team lead at Google. Specifically, she's working at a Google classroom, Google education products. And today we're going to talk about UX research as a tool for hyperscale in the age of COVID, which I think.
Erin: [00:00:50] When COVID sort of entered the public discussion in March, we maybe didn't think we'd still be very much in the midst of it in December, which is when we're recording. But here we are, a vaccine has just started going out a few days ago and we're excited to talk about how. the situation, the unfortunate situation of COVID has been actually an interesting learning and growth opportunity for Google and Amanda's experience with that.
So thank you so much for joining us. Amanda.
Amanda: [00:01:18] Thank you. I'm excited to be here.
Erin: [00:01:21] We've got JH here too.
JH: [00:01:23] Yeah, we're closing out 2020 with our most 2020 episode ever. So let's do it.
Erin: [00:01:28] Super 2020. So Amanda, tell us a little bit about your story of this last year and how your role in UX research has changed and the, the context of COVID and what that's meant for you.
Amanda: [00:01:43] Yeah. So, I mean, big changes on many fronts, one being we're working from home now. Right. And the other is the actual work we're doing. So. At around December last year. So a year ago we started to see the pandemic take foot in certain areas and Classroom was slowly starting to grow. But around February, early March, we just had a huge onboarding of people.
To give you an idea, we went from 30 million users consistently to hundreds of millions of users, almost overnight, where countries were shutting down and onboarding teachers and students onto the platform. So last year, we spent a lot of time in new markets with new users, with new behaviors, new needs.
During one of the most important times, I think in our history, when it comes to education and trying to figure out how we create. The best experience for our users during a time where things just aren't normal.
JH: [00:03:05] Just for my own education . Google classroom before COVID. Was that always for like fully remote kinds of situations or was it supplemental to in-person teaching or both and did that have to evolve as well? Or was it designed for this? Is it just the usage that scaled?
Amanda: [00:03:19] Yeah, that's a great question. Google classroom originally was created to be a tool to be used in the classroom. So a way to replace paper and pencil that allows for streamlined grading workflows for teachers and submission workflows for students, easy ways to collaborate. What was, what has really changed too, one aspect that we have seen is that, teachers were there, they were the facilitator with the students.
They could help students get in. They could help students figure out how to use certain things. And then when students moved to remote, that help was no longer there. So even the way students interacted with the platform has changed the way teachers use it. The fact that they're on it eight hours a day, and they're truly replicating their classroom experience through it.
And we've had to really think about how we embrace that and what changes we need to make and have made. To make the experience more suited for this current situation.
Erin: [00:04:23] Yeah. I imagine as this explosive growth started to happen, you're seeing growth. You said that, in new markets, new types of use cases, just a lot happening all at once. From a research perspective, how did you prioritize building out your research plans? Knowing that world might change again a lot in two weeks or, whatever timeframe you choose.
Amanda: [00:04:45] Yeah, that's a, that's a good question. The way we approached it is I have an international group of educators I have really easy access to. And those range from admins who are like decision-makers to teachers. And so when this first started. And we saw that schools weren't coming back in two weeks.
Like we thought that they would, I reached out to them to better understand, just in general what the blockers were. And from there, we did a lot of strategy and really thinking about what those blockers are to bring that into, to the research, to deep dive more into those issues.
JH: [00:05:30] Did this all happen pretty seamlessly or in retrospect, you're like, Ooh, we could have gone at this a different way, but it was such a crazy time that it was hard to know what to do.
Amanda: [00:05:40] I feel like I am extremely lucky because the team that I am on and Google in general is usually pretty good at this. My team. Really embraces research. We are a research driven team and my stakeholders. So my PM, my engineers, my designers, we all are in a decision making process together. So we come to the table as one and sit down and talk through issues and needs.
And. Really strategized together. So the research was a huge driver during this time. And it has always been a huge driver, but especially now when we were like, okay, this is a new place for us to be, what does this mean? And how do we incorporate it in? I think the big thing that changed on our side was the turnaround time.
So, research isn't something that can be done. You know, In 24, 48 hours, but when you are, experiencing such huge change and such big needs. And you're coming up with these priorities, you almost, need research within whatever timeframe it is possible. So I think the big thing for us was speeding up research, being able to get quality and unbiased recruits . getting that data from them and sometimes even scrapping the pretty reports and coming up with a quick email saying, okay, this is what we're seeing. This is the direction we need to go in a lot of iterative designs where we were making changes in the process of the research, just so we could like, just keep moving.
I think that was the biggest change that we saw in our team, but thankfully we all work really well together. And so I have to say going into this we were a pretty well-oiled machine and in the middle of it feeling really grateful of how well we work together.
Erin: [00:07:40] Yeah, I think there's a lot to be said for like team cohesion and communication above a set process. Everyone has a plan till they get punched in the face. Right. So yeah. Do something that people are starting to talk about a lot now, I think in particular that there's a vaccine in place and light at the end of the tunnel. What changes during this time are going to stick around, right?
Like what will we choose to do in person that we've learned how to do remote? What will we hurry to do in person once we can, again, and it sounds like a lot of what you did was force your research to move faster than you would have otherwise. I'm wondering are any of those tactics that you use to do those things that you'll keep around when you have the luxury of more time or were they methods of necessity that you'll be quick to get rid of when you have more time?
Amanda: [00:08:33] I think many of them were necessary. Some of them definitely keep around. I think the iterative process as we were going through the research was really nice instead of doing research with 12 to 15 participants in terms of usability going halfway through, sitting down and then figuring out if we needed a redirection and doing that.
And then finishing up the study instead of having like multiple phases, that's something that I think will definitely carry on. It was much more efficient. And it allowed for us to have collaboration every step of the way between us and design which we do, but it's an ongoing conversation, less of a check-in midpoint.
Another thing I think was really important is. For, The stakeholders to see that, yes, we are capable of operating at this velocity, but a conversation we're constantly having is like this velocity isn't maintainable for an extended period of time. So we need to be realistic about when we come back and what those expectations are in terms of slowing this down a bit.
But what we can do to have more of a regular cadence of data that we're putting out, like quicker turnaround. So we're still trying to figure that out.
JH: [00:09:54] it feels like you were hit with three really difficult things, right. Where you and your team are now interacting remotely. So that's the category. The product is being used differently because it's supposed to be like this in the classroom, and now it's fully remote. And then the scale of like usage has gone up a ton.
My first thought is, the use case of how the users interact with it, maybe as the hardest thing to react to. But for your team actually practically like those three, what has been like the hardest to juggle? Is it the scale? Is it the fact that you collaborate differently or is it that use case changing?
Amanda: [00:10:26] I think for us, it's the use case changing. Right? A lot of our work is really founded, it's founded in how our users are using the platform. And so having a change in the use case and how different markets are using it. Like we have new markets that came on this past year that look at education differently than markets we have notoriously worked with over the last five years. So for example, we have a feature within Classroom called rubrics, right? There are parts of the world that don't use rubrics when they grade. So taking things into account like that, really understanding education theory, more nuanced on the ground has been really important because really refining our product to meet needs has changed.
And getting access to those users are, in a different way too, because when you're in rural India, getting access to users during a pandemic is much more difficult than getting access to users in the United States. So we have had to really work to figure out how we do that, gain that access and understand those nuanced behaviors. So we can take every user into account when making these changes.
Erin: [00:11:44] Yeah, I wonder. Because you said after the first couple of weeks, you kind of realized this is not going to be a two week thing. But also we don't know how this will evolve or how long it will last or how those things will change. How do you prioritize? These new use cases in which ones we think are going to maybe make the product better for them, but they're not going to maybe stick around those users or are we going to make the product somehow worse for our kind of core user, if that exists and the process, how much do you bend?
The functionality of the product to meet the needs of some of these maybe seemingly fringe groups and use cases that you aren't sure if they're gonna, turn into long standing needs from a large number of customers.
Amanda: [00:12:30] Yeah. So for us, it's less about bending the product to a new set of users. And it's more about figuring out where that sweet spot is for everyone. Right? By adding certain things, we know that it could also improve a certain workflow for another type of user, if it's a subset of users and it's not actually breaking their user journey.
But making it maybe a little bit more difficult, but no one else is going to benefit from it. Is our value really there. So the way we've been looking at it is how we can make it. The changes that have the broadest impact and the deepest impact for the biggest set of users now.
JH: [00:13:14] And like what types of methodologies and approaches are you applying? Is this more like generative stuff to understand how people are using it, what they need, or is it more evaluative, usability or does it run the whole gamut?
Amanda: [00:13:25] Oh, it's the whole gamut. So it's a lot of generative stuff in our new markets. It's a lot of evaluative stuff in our current markets. Usability across the board, understanding how other tools come into play too, because Classroom's a hub. It's not necessarily the only thing that people are using, but they're bringing in other curriculum tools and, like writing tools and stuff like that.
We are looking at it from multiple angles. You never want, like in a situation like this, it's really important to understand all of the players. So understanding why admins are doing what they're doing, why teachers are doing, teaching a certain way or using certain tools, the impact on students, the impact on parents as well as.
Trying to see it through different lenses. So incorporating diary, studies, interviews, social analysis on media platforms, we get a lot of feedback on Twitter. And so taking the time to go through that and really understanding what those needs are we have in product surveys that we collect data through.
And so, yeah. Just taking all of the different avenues, we're getting research or getting data and trying to see it holistically. Right? What those different experiences are from the different players within the, within the whole system.

Erin: [00:15:35] And since you've been gathering so much data for a relatively long period of time now, nine months, or so have you seen how have things changed, right? Are people getting more used to using Google classroom now that maybe they were new to it in March and now it's not so new. Are you just constantly getting new people joining and so.
Yeah. And I'm curious to like you know, in your research, you're talking to so many different stakeholders, so many different users, you've got your teachers, your admins, your students. And I imagine like everyone's stressed out. I would. Right? So that's part of it too, in terms of how people are using my product under stress that maybe they didn't have as much before.
And how has that changed as you've continued
Amanda: [00:16:19] Yeah, I think that brings up a good point. Going back to the last point was actually taking into consideration the stress of the moment. We've had to be super careful in terms of what we ask our users for and being cognizant that teachers right now are just really stretched and getting data from them actually might take longer.
Right. We have a higher dropout rate. In our studies because of burnout. So being more flexible with that over recruiting and taking different methods to, to account for that. um, What I've seen, which is absolutely fascinating is teachers who were reluctant to adopt technology for education.
So those who we really never thought we could win in terms of bringing them on board and being a classroom user were almost forced while they were forced to go into tech, right. They had to adopt this technology to keep their classes running. And what we see is that they are. There's like this new wave of people opening up to the benefits of technology in the classroom.
So my future prediction is that as we move out of this, there's going to be more of an embrace of how tech is used in education, how we connect and communicate with each other and our flexibility for other people in their situations. So that's one major thing I see in terms of our users.
JH: [00:17:55] As you're moving faster, right? You've talked about this velocity that is not sustainable. It feels like it's also really important to nail the prioritization of, we have this new market coming on. We want to learn about that. We have this new feature rolling out because it's now remote instead of in-person all these different things, all these different methodologies are deploying.
Like how does your team balance, like what to do at any point in time and actually.
Amanda: [00:18:18] Yeah. So what we do is we go through a strategy process. So as this happened, we actually sat and we did a lot of hard conversations about what we need to prioritize and we went through. And based on the research, we do a stack rank of everything. That's come up, everything that we were focusing on and we do a re-prioritization and based on user need while also taking into account, like what the future should hold.
We pretty much have these really difficult conversations of what should be our number one priority. And that's for ENJ resourcing that's for, design as well as research. And then we go down the list and then what we do is we mark where our resourcing stops. So based on how many people we have on the team, this is what we can cover.
And we're really honest about that with the people above us. Right. And if there is a disagreement on what should be stack ranked, we go into negotiations about that, but everything is backed up with data that we have collected. And that is something that I, in past jobs during my career, isn't necessarily the case.
You're often having to argue why something's really important. It's almost. In a way it's funny to sit in a room and have everyone else talk about the research and you sit back and be like, yeah, they got it. So that's how we've done the prioritization. This is something we do on a regular cadence.
Actually we do it every quarter where we go through and we stack rank everything and figure out what we're going to be focusing on that quarter. The difference is that we had to choose that. A little bit mid-quarter and then we looked further down into the next few quarters, if this continues and we're not back, like, how do we think about education in a distance learning world.
JH: [00:20:18] And were you doing it like this before? Or is this like a new prioritization process?
Amanda: [00:20:21] This is something we've been doing. Since I actually joined the team, it's a way we handle all of our work before we enter into the quarter. So pre quarter, we do a stack rank and negotiation of that stack rank.
You were talking
Erin: [00:20:37] a little bit before about the variety of tools people are using to manage the education on the tech side. And as a parent, I'm certainly very familiar with that. And how many SaaS tools do I use at work everyday? Like a hundred? Like so, so many . It's chaos, the kids stuff, you know, cause it's all new and there's a new one all the time.
They're not all the best, Google Classroom excepted, of course, but you know, some of them are not the most user friendly. And so it's interesting cause you have all these consumers, parents, teachers, students, not like professional tech workers, so to speak building these tech stacks essentially for themselves, right.
On the fly and pretty rapidly I'm curious how you research those. You said you're researching, not just how people are using Google classroom, but really the rest of that stack of that workflow. And obviously you can't control all these other apps in the entire workflow, but what is some of the research you're doing?
And some of the changes you've been able to make based on any observations there.
Amanda: [00:21:42] Yeah, that's a really great question. A lot of what we do is really understanding the ecosystem, right? Because Classroom is a hub. And how teachers are providing assignments. If they're given a curriculum, where are they sending their students to? Is it in the hub? Are they leaving the hub? And how that workflow looks for on the teacher side and on the students side.
So if the student is saying, going to another website, what does that look like for them? We just don't stop in the classroom because that's not necessarily realistic. We need to know how they get back, how they actually submit the work from the other, the tool, how the teacher grades it, all of these other things that are really important.
And how like, parents manage that. How would your parents' magic manage all of the logins? How the safety concerns, all of that. So we're looking at things across the board. We're not like necessarily diving into other tools, but we're understanding how our tools touch those new tools.
JH: [00:22:44] How does everyone's evolving like tech and remote literacy affect all this stuff? Right? Like I think something we've seen from other user researchers is the efficacy or the quality of sessions remotely, maybe was a little different because participants felt a little less comfortable on a zoom call or whatever.
Right. And now everyone's doing it all the time and it's a little bit more normal. Is it like stuff you learned early in the pandemic is no longer relevant because people have a different baseline of how to use these tools and you have to like reassess or is that not part of it? Or how do you think about all that?
Amanda: [00:23:13] I mean, I feel like there was a point at the very beginning where there was certain data that was coming in that definitely doesn't really hit like it did then, as it does now. Right? Like it's old, but there's a part of that really lends to the story. We're still seeing new people come on.
Right. So understanding what those friction points are. Also like just the chaos aspect. That was another thing that we had to sift through that noise of chaos because there was, schools were closing without plans. Right. They hadn't onboarded teachers, students weren't ready, there weren't devices.
And so there was that noise in tool usage because people weren't able to log in, they didn't have their logins. Just in general. Or access, but it gave us a good understanding of what happens when students don't have easy access to their devices. What happens when they have trouble with their passwords, it might be on the extreme, but at a loss for us to really figure out how we think about these barriers going forward. And the support that we have to provide them. Not only do I have cross-functional stakeholders and enj and marketing or enj and design and PM, but I also have cross-functional stakeholders in marketing and support. So I've been working with them as well. How do we get the support out there in the languages that we need to be able to connect with our users to help them through these steps?
So that comes from the marketing side and we do hear that in our user research. We try to get our research open enough and more broad that it can really help multiple teams. So we stay connected and a really cohesive story and workflow
Erin: [00:25:08] Important aspects of doing faster research in this context of a rapidly growing and changing set of use cases as getting that research used, right. You're doing it to make an impact. And it sounds like you have a very sort of integrated team that is bought into research and has found ways to work well together.
But how have you changed any of your processes or adapted to this world to make sure that those insights are able to be used and that there's a feedback loop happening, right.
Amanda: [00:25:43] Yeah, that's a great question. I mean, there has been change on the research side that we had to make to stay afloat. I mean, I have multiple PMs on my team, so we were getting an abundance of research requests and the process we had before COVID was that our PMs would grab time on our calendars.
Ping us like, Oh, we need to look at this. And that cadence was not every day. I mean, it was like maybe once every few weeks and then COVID hit and it was like, we need to know so much more. We need to understand different people. Different uses like all of these things and. The influx of requests was so high that I had to take a step back and go, how do I refine this process?
Because we won't make it through this in this way. So. We started to stack rank research, which we hadn't done before, because we didn't need to, because there were a handful of studies that we were doing and consistent usability, but it wasn't like, This many. So we had one PM and myself, and we'd sit down and we'd stack rank research based on the urgency of it, the impact of it, the timeframe it would take, which wasn't something we had done before.
And that's something I think we'll be bringing forward because it allows for us to really think strategically about the questions we're answering. And then when it came to sharing our research. We have always sent our research out more broadly to the whole team with a summary.
But what we've been doing now is making sure that we're in all conversations and at the table, when we're talking about feature developments and changes that are being made. And that there's slides that are dedicated to why we're doing this. So our stakeholders understand the impact these changes have.
And I think that's really important, especially during this time, because there's so much to do that they need to understand the magnitude of why we're making even the slightest decisions.
Earlier
JH: [00:28:01] You mentioned a bit about, the teachers being so stretched and in a really tough spot. And accounting for that and your research and stuff. I'm curious, how are the students doing? Like, I guess like, you know, are the kids all right? Like the, are they adapting to this? Are they, does it seem like they're learning?
Are there signals that, they're, that this is going well and it's, it's working for them.
Amanda: [00:28:17] Yeah. So at the beginning, so the first three months I did a major social listening study where we analyzed open social media posts. So things that weren't behind sign-ins or anything like that, to really understand how these different user groups were doing and they have to say, kids are resilient.
They are really resilient, but this is a really difficult time. And one of the things that really came out of that is there is a rise in anxiety and depression and sadness. They want to be with friends from school, as great as it is that we have these platforms that are allowed to connect students with teachers. Education isn't just about learning, right? It's about social connection and kids were really starting to feel that isolation. So, thinking about how. Students can really connect with their peers and feel less isolated during this time. It became very, you know, was it a huge piece of the findings that we were getting .
Teachers were amazing. Teachers are amazing. Teachers are always amazing. But what we saw is this increase in flexibility and this drive from teachers to constantly check in on their students and their mental wellbeing, which I think, during something so scary as such a beautiful thing, it shows a piece of humanity of like how important we are to each other.
And so I do think the kids are going to be alright. I do think that, there is a lot that's going to happen post this. But children are resilient and thinking about how we can continue to help them communicate and connect with others is going to be extremely important. Even if w once they're back in the classroom.
JH: [00:30:14] Interesting other direction that I've seen as well. My wife is actually a mental health counselor and deals with some programs that are in schools and helping kids at the high school age. And there's been some kids on the other end of the spectrum that were really stressed and anxious in a school environment that have actually done really well in this.
And so hopefully there's a nice learning for everyone that, the one size fits all. Maybe there, we can be a little bit more adaptable to individual needs and stuff like that. Cause I think you've seen it in both directions and hopefully, maybe this has been eye opening in some or some regard there.
Amanda: [00:30:44] A hundred percent. I mean, we've heard from large district leaders saying that what's been. Really nice about the switches that it has given students who may have been bullied who were struggling with their identity, who were going through a transition. It has given them the space to be who they are without the negative influence that the negative pressure that they were getting from their peers.
So there is that opposite side in terms of it's a spectrum, right? And also, I mean, there's also a set of kids who really operate well at night. So the flexibility of being able to sleep in and then access your schoolwork a little bit later and allowing for autonomy, which children are always striving for has really lent itself well to this certain situation.
JH: [00:31:41] Some of my favorite stories from the pandemic, which I was in up in a great time, but especially early on when they're doing all zoom classes and stuff and you'd see the kids, they would get really creative and like game it, where they were putting up their face and like looking frozen intentionally. So they wouldn't have to pay attention.
And or doing other stuff. I always liked those kids. Those are my spirit animals. Just funny.
Amanda: [00:32:00] Children are very resourceful.
JH: [00:32:03] Yeah. Or there's one, I know there's some app that kids were being asked to use or whatever, and they all gave it such bad feedback on the app or whatever it was temporarily taken down and they didn't have to use it for awhile, which is like an awesome story. I don't know. I liked the kids messing with the system a little bit.
Erin: [00:32:16] Yeah, we gave Stella her own little, she's got the closet and the guest room is her. She calls it her office, which it is. But she's always in there like doing her yoga moves with her classroom and stuff. And I do think it's developed a lot of autonomy for her. And today's actually her birthday and talking about teaching.
Thank you talking about teachers being just emotionally awesome and available for their kids in this context. Um, Today it was a snow day, which is really fun. And the teacher's, “can I call you for 10 minutes on your birthday since we don't have school?” And I just thought that was so sweet. On a group on a Google meet.
JH: [00:32:50] Nice, but they preserved the snow day. They didn't make them do remote learning on a snow day.
Erin: [00:32:54] No, I was a little surprised no one had to go
JH: [00:32:56] I'm holding out for that. We need
Amanda: [00:32:58] Son had the same. We have a snow day and he's been remote. So he was super excited about
JH: [00:33:02] Thank God we need real snow days. That's important.
Erin: [00:33:04] Yeah. So that was fun, but yeah. How are you feeling about the future after this crazy year?
Amanda: [00:33:10] I mean I'm always excited about the future. I'm a very optimistic person. And I think that, we are a really, really interesting time and I think that. As we move into the future, we'll see huge changes in how we look at education and it's going to shift hopefully for the better, right. One of the greatest things to come out of this is the conversations we're having about equity in education, which, as someone whose background is an academics I think is so important that we really.
Have these open dialogues about what it means to make sure that our tools are accessible by all. And this has just put a spotlight on that importance. And I know a lot of people, multiple companies are reevaluating what that means. And I think that is a really good thing.
JH: [00:34:00] That's a great point. I mean, even just access to the internet, right? Like I know when this first started the local school system in Salem. It was actually able to spin up a thing where for families that didn't have internet access, they were able to provide a stipend to purchase it for them so that those kids could still attend and stuff like that's something you don't even really think about sometimes.
Amanda: [00:34:16] Yeah. I mean, there was a district in Texas, a low-income district in Texas that equipped multiple school buses with wifi access and they went into different neighborhoods and they would just set the bus there. So students had access to, to wifi that way. And so schools are also getting resourceful, but I also think like it has shown that our teachers are underpaid, that our schools need more resourcing, and we need to think about how we can get access for all which I don't think the conversation is going to end after all of this is said and done.
Erin: [00:34:50] Yep.