Level-up your UX Research with Diary Studies featuring Mike Green

AI-Generated Summary

In this episode, Mike Green discusses the use of diary studies in UX research, particularly in the context of public sector user research. Mike Green, a freelance user research lead based in the UK, shares his experiences using Google Forms, structuring incentives, and addressing challenges in recruitment for diary studies. The discussion emphasizes the longitudinal nature of diary studies, contrasting them with traditional usability tests, and highlights their effectiveness in capturing user feedback over a period of time. Mike also touches upon the importance of understanding user behavior, attitudinal responses, and the ethical considerations in dealing with vulnerable groups. The planning and design stage for diary studies is emphasized, stressing the need to align the research goals with the chosen methodology. Overall, the video aims to provide insights and tips for conducting impactful UX research using diary studies in the public sector.

Video

Audio

AI-Generated Transcript

Mike Green 0:00
It’s a bit jargony. But the best way of describing is kind of longitudinal research, ie, it’s not like usability tests where you get a user Well, in the old days into a lab, obviously, these days, it’s all remote. Where you’re testing, let’s say, a prototype for an hour in depth with an individual one on one or kind of two on one. For a sort of snapshot of their feedback, what you want to do is capture their feedback over a period of time. So it kind of goes back the first question you need to ask yourself is what do you want to learn? Because certainly not every research question is answered with a diary study, but any shape or form.

Tracy Laranjo 0:33
Experiment nation, it is Tracy, and I’m joined by a special guest, his name is Mike Green, and he runs his own podcast that will definitely appeal to the user researchers. And here, I have checked it out. I love it. So highly recommend, but I will let Mike take the floor and tell us a bit about himself and his podcasts. So Mike, welcome to the

Mike Green 0:54
show. Thanks, Tracy. lovely to be here. Good. Good to be chatting. So yeah. So I’m Mike. I’m a freelance user research lead, based in the UK. And I work mainly with public sector clients over the last few years since I’ve been a contractor. I’ve been dealing with lots of big government ministries in the UK. And I have a public sector client at the moment that I’m working with. And as you said, Yeah, I also have a podcast. So the podcast is understanding users. It’s a UX podcast aimed at UX designers, researchers, product managers, kind of digital folk in general, talking about digital transformation, and basically user centered design, how we go about building products in a user centered way. And the challenges we all face doing that. So it’s available on Google, Spotify, Amazon, Apple, usual places. So yeah, feel free to check it out. Yeah,

Tracy Laranjo 1:46
please do. And also the fact that you work public service, I’m sure there’s a lot of extra challenges that you deal with that are maybe not as easily spotted in the private sector. Can you speak a bit more to that? Yeah, that’s

Mike Green 2:01
a it’s a really good point. What’s interesting is actually, and I’ve said this kind of elsewhere that in terms of the British government early on, about, well, decade or so back, the British government identified problems with the way it procures it and delivers large scale it projects, not to say they’re all perfect now. But it set up this thing called the GDS, the Government Digital Service, which is it was designed to kind of embed digital delivery best practice across government, basically, to save taxpayer money to make processes more efficient. The upshot of that is that actually, over the last 10 years, lots and lots of really good digital delivery work has been done right across the British government. So things like, you know, renewing your passport, applying for tax credits, you know, applying for doctor’s appointments, all of that kind of stuff. People sort of say, Oh, the Gov. Gov uk is quite good, you know, surprising. And I’m like, well, having worked with many teams, kind of on those sort of pieces of work, that’s not an accident, it’s because lots of lots of people, digital experts, you know, from from a whole variety of disciplines who spend a lot of time and effort, ensuring that the services are built with users in mind. Because you know, the analogy I always use, unlike trainers, where you can, you know, you go onto Google, and you’ve got a dozen different companies that you can buy trainers from, if you’re trying to get a passport, or you’re trying to tax your car, there’s only one person you can do it, you know, and one place you can go and that’s the government. So it’s really important to in terms of accessibility, people with kind of sight issues, people with motor issues, that these digital services are built with the entire community in mind. So actually, yes, there are other challenges, to to your point around, you know, budgetary, you know, procurement issues kind of book doing everything in a in a taxpayer centric way.

Tracy Laranjo 3:38
But tape I’m sure. Exactly, exactly. But actually, I’ve

Mike Green 3:42
been I’ve stayed in public sector for quite a few years, because I, I feel like it you know, it’s a it’s good experience. And B, it’s actually in some ways, the better way better way of doing things. Not to say it’s perfect, but in many ways, it’s, it’s an exemplar of how it should be done. And actually, other countries have copied the British government in many ways in terms of doing it. Nice.

Tracy Laranjo 4:01
That was pretty illuminating. I look at my spouse’s experience working public sector for, like our provincial government in Canada, and I just I look at it and I’m like, I can’t I can’t deal with this. So that’s good that at least that’s a bit more streamlined in the UK. Now, just in preparation for this episode, you gave me a lot of really cool topics that we can discuss related to user research. I am a big user research fan myself. And you mentioned one particular method that I’ve never used myself personally. So I want to ask you a lot of questions about it. And that would be diary studies. I don’t I just don’t hear that term a lot. So can you describe to me and to our listeners who haven’t heard of a diary study what that actually is?

Mike Green 4:50
So yeah, absolutely. So I suppose it’s a bit jargony. But the best way of describing it is kind of longitudinal research, ie, it’s not like usability tests where you get a user willing The old days into a lab. Obviously, these days, it’s all remote. Where you’re testing, let’s say, a prototype for an hour in depth with an individual one on one or kind of two on one. For a sort of snapshot of their feedback, what you want to do is capture their feedback over a period of time. So it kind of goes back, the first question you need to ask yourself is, what do you want to learn? Because certainly not every research question is answered with a diary study for any shape or form. But where, you know, for example, behavior over a period of time, if you want to baseline behavior, if you want to test a new product or service and see if the behavior modifies, you know attitudinal responses. So if you want to see how, you know, citizens users are, how are they feeling at particular points in time. A diary study allows you to do that. And there’s different ways you can do that. So it’s normally run over, you know, one or two weeks, I think two weeks is kind of the sweet spot, the longer you run it, the less likely you are to have what the greater the level of attrition, probably the shorter you are, the kind of harder it is to get solid insights. But I mean, there’s one particular example I can talk about of a client I had a couple of years back, it was the middle of a pandemic, actually. And it was quite a high priority piece of work for the government and the department I worked with wanted some feedback on a particular service they wanted to launch. And I was concerned that we didn’t have enough clear user insight before, you know, the taxpayer studies. You know, what, on behalf of the taxpayer, lots of money was spent on this particular solution. So I suggested we run a diary study. So basically, we had a selection of families. And over a period of two weeks, we kind of gathered feedback from them pretty much every other day, to gather they’re trying out this this new solution to see what how they found the experience, basically. And the way we did that was, you know, we did a so for example, we did a setup call. So to kind of build the rapport with the family to introduce myself introduce what we’re up to upfront, I would do a kind of zoom call with each family and the word, let’s say, Can I think I can remember half a dozen families, I think, possibly eight family something like that, to explain what we’re doing and get the setup done. And then I would send them out to an SMS message. And we use the, you know, the government SMS facility, which was nice and anonymous, because it wasn’t coming from my individual phone, it was coming from, you know, the government phone number, to prod them and remind them. And in that reminder, there was a Google Forms link. And basically, all they had to do was basically click on the link, fill in the form. And the form was, you know, essentially, another kind of tip for Doris studies is keep the form as simple as possible. So that you’re not, you know, that again, the longer you make it, the more complicated they make it, the higher the likelihood of people just going, I can’t be bothered with this. Literally, I think we had three or four questions, bang, bang, bang, I don’t think there was even any other might have been one free text field, it was mainly kind of multiple choice or Likert scales. And then to do that every few days, over a couple of weeks, and then we had a final wrap up call at the end of the two weeks to go through with them kind of you know, thanks very much. And obviously, the beauty of doing it like that was we were gathering insight as we were going. So we could do the analysis based on the form submissions they were providing. So that by the time we did the final wrap up call, we had already kind of identified questions that we wanted to ask and areas we wanted to probe

Tracy Laranjo 8:13
for. That sounds kind of like a mix of a survey with not a user test, because I’m sure you’re not actually observing live behavior. But there’s definitely that level of participation that you wouldn’t see just from strictly a survey.

Mike Green 8:30
Yeah, exactly. Well, I mean, the reason they’re called diary studies is in the olden days, pre digital, they were literally diaries. And so this kind of ethnographic approach that has been around a long time, you know, digital can’t claim any credit for that. It’s just that, you know, rather than writing in a diary, sort of Dear Diary, entry, we now have the tools to be able to do something similar, but in the digital realm, and, you know, Google Forms is one tool. There’s endless the survey tools, there’s, there’s all sorts of capture tools. There’s, you know, Google Docs, as wikis, you could do, there’s SMS, but for our purposes, you know, and I think that’s another important point is the kind of ethical and accessibility issues need to be taken into account, because we were dealing with, let’s say, a vulnerable group, who, you know, most of them are suffering significant financial hardship. And they the, the tools have to be as simple and as low cost. And the process has to be as simple as possible. We also incentivize them, which is another kind of key point to mention, again, the chances are, you’re better off, you’re more likely to have a successful outcome. If you incentivize participants, and then it comes down to how do you incentivize them? Do you incentivize them just upfront? do you incentivize them at the end? Because that poses different issues, because if you centralize them upfront, then the chances are, they might drop out and disappear and you’ve lost money and you’ve got no insight. If you incentivize them at the end, there’s the chances are that they’re just going to give you what they think you they want you to hear, so that they get the money. So I think what we did from memory is we split it we did a payment upfront and a payment at the end just to sort of try and spread the spread the risk and still books still gather rich insights.

Tracy Laranjo 10:01
Do you remember which group was more likely to complete the study? As in? What the incentives? Were?

Mike Green 10:14
It’s a good question. No, I mean, I would always, like, you know, classic usability testing, where in the olden days, we used to bring people into a lab, and we’d pay them, we would always do that afterwards. So I think I would certainly err on the, you know, version of that. But at the same time, as I said, kind of the longer the diary study a week or two weeks is quite long time, you know, you’re asking for people to commit, I think the sort of the lip, the academic literature talks about kind of seven to 10 minutes a day maximum. If you’re asking people to do more than that, for the sake of a, I don’t know, let’s say 20 pound, 50 pound $50, Amazon voucher, they’re not gonna they’re not, you’re not gonna get that people are not gonna want to get that by doing hours and hours of input. So it’s got to be something as possible.

Rommil Santiago 10:57
This is Rommil. Santiago from experiment nation. Every week we share interviews with and conference sessions by our favorite conversion rate optimizers from around the world. So if you liked this video, smash that like button and consider subscribing, it helps us a bunch. Now back to the episode.

Tracy Laranjo 11:10
These are very good considerations that I’m sure go into your process of planning a diary study? What else would you consider as key considerations? In that planning and designing stage for the diary study? Like is, Do you often find you have to consider a diary study versus another research method? When would you choose a diary study over another method? And just yeah, what does that planning and design stage look like? For you? Yeah,

Mike Green 11:40
I mean, well, as always, with any kind of research, the planning stage is vital. And it always goes back to you know, the teams, I’ve worked with a bored of hearing me say it, but it’s kind of like, what do you want to learn? You always, always, always should start with that question. Because that will dictate the method you use, it will dictate the timescale, the budget, you know, the number of people involved, all of that kind of stuff. So, certainly for, you know, all sorts of outcomes or studies not applicable, but where it is. So as I was saying before, whether it’s kind of, you know, a longer term, you want to basically track behavior and or kind of feeling attitude over time. That’s really where a diary study comes into its own, rather than a bespoke kind of test of a new product. So I think you’d start with what do you want to learn. And then other tips, things like, you know, regular contact with participants is key. So that that’s one of the reasons I always do the kind of rapport building kind of intro at the beginning just to introduce myself. So even though we’re doing it remote, and because obviously, in the course of the diary study, they’re not seeing me or talking to me, it’s purely coming to them via SMS, or email, or whatever it might be filling in this form remotely. So I think it’s important that they know who they’re dealing with, what the purpose of the, you know, the session is. And, as always, you know, what’s, like the recent good research consent, informed consent, you know, what’s going to be done with the data? And who’s going to see it? Is it going to be attributable to them? You know, probably not, hopefully not, so that they can feel free to speak candidly and anonymously. And the other thing is, of course, accessibility issues, as I mentioned, so if you’re dealing with vulnerable users, you know, whether they be older, you know, with with physical and mental disabilities, all of that needs to be considered, because that should inform how you design your study. So important.

Tracy Laranjo 13:19
Yes, absolutely. So important. So let’s say you’ve decided you are ready to make a diary study? Where do you start? I know that there’s a form involved, and I’m sure there’s communication with your participants involved. But really, just where do you start?

Mike Green 13:37
Well, I mean, the step after deciding you want to do a diary study, I suppose really, is that is the recruitment piece. And that’s something that exercises user researchers and always has done, you know, how do you how do you then find the people and get them to agree to take part in this? So, you know, do you do you send out a an email to, you know, throw the net out wide and get people to kind of reply to you if they want to take part? I think that from memory, that’s what we did. And obviously, then you your chances of success, I’ll you know, you might get a five 10% response rate. So if you want half a dozen people, you’re probably going to need 50 to 100 emails to go out. If it’s a different type of service, you might have already identified particular, you know, if it’s, I don’t know, let’s say in an organization and you want to run a diary study with different branches of that organization or different offices in different countries, then that’s probably easier because you’ve already identified them, and then it becomes a lot easier up front. So it really depends on who uses already, where they are, how likely they are to engage all of that kind of stuff. Great.

Tracy Laranjo 14:41
So yeah, obviously recruitment is a big piece. Is there any way that you can make screening as easy as possible and like maximize that response rate, but also make sure that the quality of respondents is high?

Mike Green 14:57
Very good question. So my prayer Reference always, and this is a challenge I’m having with my current client is to use some kind of external recruiter. Because as you know, as a researcher within a digital team, there’s, there’s enough on our plates already with designing the research, running the research, analyzing the research, to also then have to go and find the people to do the research with, I would prefer to kind of outsource that, quite frankly, if I can. So what what I’ve done in the past is engage in not just for diary studies, but for kind of research in general as you engage. And there’s, there’s, you know, a number of, certainly in Britain, I’m sure there are all over the world, some really good organizations that are specialists in this, and you basically agree up front with them, you give them the criteria you want. And it’s their job to go and find you these people, and sometimes the most really obscure brief. So one particular example, I had a service I was working on a couple of years back was British citizens who had married Moroccan nationals in Morocco within the last six months. very specific, very specific. And even the agency was this recruitment agency was saying this is going to be a very hard ask, but they found they found the people and they were actually really, really good participants with some really valuable insights.

Tracy Laranjo 16:08
So I think that’s the thing. Also, I’m so sorry,

Mike Green 16:12
no, no, I was gonna say that’s one way of doing it. And that sort of takes the load off the researcher, but still, hopefully will guarantee you quality insight. Great.

Tracy Laranjo 16:20
Yeah, I noticed. I do user testing on the testing side. So I often read through a lot of screener questions. And sometimes I read the screener and I’m like, Whoa, you’re looking for someone who owns a Toyota speaks Korean, is married. And I’m like, This is so specific. How do you find these people? I often wonder if these very specific demographics even show up in the screeners as participants. So it’s kind of I see that on the other side, too.

Mike Green 16:58
Yeah, no, I agree. So so like, one example is the example I just gave, we use a an external agency. But obviously, there’s a cost involved with that. And it may be in lots of projects that the client doesn’t want to pay that cost, which is totally understandable. Because you’re having to pay them to find these people. You know, social media is another one. You know, I’ve seen it done on Facebook as well. You can use Facebook to find people and you can, using the kind of the Facebook targeting process, you can actually set up kind of targeted ads to the relevant demographics that you want. You could set up kind of surveys with screening questions built in kind of like the ones you were talking about where you, you know, you sort of have some kind of logic in Are you a Korean speaker? Yes or no? Do you have a Toyota yes or no, and you kind of filter people out that way. But obviously, the more barriers you put in front of the process, that the harder it will be to find the people. So it’s got to be as relatively straightforward as possible. Totally. Sorry. And the other thing to mention, while I’m thinking about it is particularly in the case of accessibility, so the the particular recruiter, I’m thinking about the one we used in London, had a specialist pool of, there was another piece of work we were doing for a different project where we wanted older users. And they had a specialist pool of older users, you know, they were mostly the ones we got in with between sort of 70 and 80. In the end, and they had sight issues, they had mobility issues, and it was really valuable to kind of be able to talk to them. And of course, finding them would have been even harder using some of that social media or something like that.

Tracy Laranjo 18:22
That’s so helpful to have that roster of participants already ready for you. Exactly. Yep. So screening is obviously or rather, screening and participant recruitment is obviously a challenge. Are there any other elements to diary studies that our listeners could maybe expect if they run their own diary study?

Mike Green 18:46
Um, I mean, I think we’ve touched on most of them, I would say, so set up is key. And that covers everything we’ve just talked about, I would say regular contact is key, over the cost of the diary study, I would also say being really frank with the participants upfront about the expectation, you need to say to them, we would like you to spend this much time a day or a week doing this and make sure they’re really clear what to expect to them upfront. Because if they start living Oh, and $50, Amazon voucher, great. And then you’re asking them to spend half an hour a day, right writing some kind of, you know, email, very quickly, they’re gonna go forget this. It’s not working.

Tracy Laranjo 19:22
Is the commitment worth the ends? Yes.

Mike Green 19:23
Correct. Exactly. And then just make it as easy as possible, you know, make it as easy for them to respond to you. And, you know, and I suppose also show gratitude, you know, the danger is sometimes this can become a scientific process where you’re just obsessed with your results. But these are human beings, you’re asking, you know, the maybe in a sense of them, maybe not, but still, they’re giving up their time to provide insight to you, which is valuable. Well, otherwise, why would you be gathering it? So it’s important to you know, I think show gratitude. I think

Tracy Laranjo 19:53
that’s a good that’s a really good tip. Now, let’s just assume you’ve gone through the process of creating in your diary study, all your participants have committed, you’re wrapping it up and you’re ready to digest everything you learned and put it in a format that’s going to make sense to the people who asked you to run this diary study. How do you go about analyzing the diary, study data and then extracting those insights? So it’s actually usable?

Mike Green 20:23
So that Tracy is the million dollar question that every researcher has at the end of around researchers, okay, you’ve got all these insights, you’ve got these, whether it be hours of interviews, whether it be kind of lots of survey responses, you know, how do you take all this data and turning into something meaningful? Well, the thing with a diary study is probably you’re going to end up with quant and qual data. So you’re going to have certainly in my case, we had quant data from, you know, attitudinal scales, we asked people to complete sort of, you know, hot numbers, if I can put it like that. But at the same time, we had qualitative data in the form of, you know, text responses that people are giving us. So we had free text boxes in, you know, albeit short, free text boxes in the form as well. So I think, well, first of all, is, that goes back to picking the tool, you want to pick a tool that allows you to get the information out as easy as possible. And of course, the beauty of Google Suite is that it’s pretty easy to export all that stuff, to Google Forms, drops it into a lovely spreadsheet that you can just export, which is what I did. And then these days, obviously, there’s there’s, you know, a huge plethora, and ever growing plethora of tools that allow you to do analysis. Back in the day, you know, my go to process was always Miro, and we’re all going to be mural or kind of in a similar tool. And then actually somewhat laborious ly kind of getting the, let’s say, getting the text quotes into post it notes, digital post it notes, and then kind of doing affinity mapping process, where you basically group, you get everything up on the board, virtually. I mean, if you can do it in person with real post it notes even better. But those days feel like they’re sort of gone. But and then kind of group in the post it notes thematically to say Well, look, these are the themes which are coming out because participant 1234 and five said this and participant six and seven said that. And that kind of handles the qual side, the quant side? Well, I mean it, how does one analyze quant generally you know you you can look at the you know the number of ways you can do it. Probably these days kind of AI will be able to help in the future with that kind of stuff. But again, I guess it depends on thinking about the design of the study upfront how, you know, what data do you want to gather, that will allow you to kind of make design decisions. That’s fundamentally what it comes down to. So if you’re less interested in the quant side, then probably kind of a more qualitative boasts approach in terms of you know, the forms they fill in, etc. is better? Mm hmm.

Tracy Laranjo 22:47
This is really good to like, I’m picturing this in my mind what that would look like as a mural board. And my big question and the question that I always come up with when I’m working with qualitative data, like a survey, for example, is how do you balance the knowledge that one person’s opinion is one person’s opinion, but at the same time, every participant counts? How do you know when something that a participant said is worth sharing as an insight? That

Mike Green 23:19
is a really, really good question. And it’s it’s come I’ve actually recently, again, with the with my current client, I mean, I think that’s why I’ve technique like affinity diagramming is so powerful, because, you know, we go I hate that old chestnut that’s always leveled at researchers around well, you know, so if you run around and research, let’s say, prototype tests, typically, what’s the Nielsen normal kind of metric, about half a dozen people will give you kind of the majority of the usability problems. And that’s become a sort of industry standard is also kind of manageable in terms of, let’s say, a two week sprint, to find interview, analyze six hours of user feedback. But at the same time, that’s not very many. And the argument can be leveled, particularly people who aren’t well versed in UX to say, well, that’s only six people or that, you know, how can you possibly know from talking to six people? And it’s a fair point. And I think with a diary study, therefore, probably you can go for a larger scale, you know, if you went for twice that number, it comes down to kind of that thematic analysis at the end. And if enough people are raising the same point again, and again, that’s where the light bulb should be coming on, you’re going, aha, this is this is a pain point that we need to focus on. I’m gonna always frame it in terms of pain points, and what are the pain points that are coming out? Because they’re the areas we need to kind of focus on? If if one person mentioned something and no one else does, then, you know, don’t ignore it, but it’s probably not as important as if everybody’s shouting, you know, raising the same pain point.

Tracy Laranjo 24:40
I think that’s really good advice just for finding the so what the responses and your research so I’m gonna definitely apply that to not just diary studies, but also to surveying user testing. So thank you for that. Is there anything else about diary studies that you feel our listeners need to No before maybe creating their first diary study.

Mike Green 25:06
I think now I think I think we’ve covered a lot of ground. I mean, I would say, I haven’t done a huge number, but the ones that I have done have been very instructive, destructive, if I can put it like that, I mean, I think probably one of the things says, It takes a lot of time, don’t be under any illusions that, to set this up, to run it, to analyze it to kind of to make the most of the opportunity that that particular methodology gives you is not something it’s not a quick and dirty solution. So you need to go into it with your eyes open. And again, go back to your chest, not what you want to learn. Because if if what you’re going to learn to read or study is not what you need or want to learn, then don’t do it. But if it is, then it’s a very powerful way of doing of learning that I think,

Tracy Laranjo 25:49
really good consideration. Thank you. Okay, so toughest question I’m gonna ask you today, what do you have going on, that you’d like our listeners to know about? And how should they find you?

Mike Green 26:02
Now, there’s a question. Well, I mean, I think I mentioned at the beginning, and you kindly mentioned it, as well, I think understanding users UX podcast that that’s what I spend, you know, when I’m not doing my day job, and when I’m not looking, looking after my small children with my wife, that’s what I devote kind of a lot of time to, it’s something I very much enjoy. I’ve got to meet some great people. We’ve had some really great guests on the show. We got some great guests coming up. Actually, we’ve got some kind of former senior product designers within Mehta talking about AI we’ve got you know, someone from NASA talking soon. Wow. So yeah, you know, I would, you know, encourage anyone to to, to have a look at that. And you can find me on LinkedIn as well and my website researchable.uk. And if they’ve got more questions or would like to reach out to me then feel very free. I’d love to love to engage. Love it.

Tracy Laranjo 26:51
Thank you so much, Mike. This is a pleasure talking to you about diary studies. And thanks again for joining the show.

Rommil Santiago 26:58
Thank you for having me. It’s been lovely. This is Rommil Santiago from experiment nation. Every week we share interviews with and conference sessions by our favorite conversion rate optimizers from around the world. So if you liked this video, smash that like button and consider subscribing it helps us a bunch

If you liked this post, sign up for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK