AI-Generated Summary

Dr. Elsa van der Berg, a B2B SaaS product management expert, shares some key advice: Don’t try to build for everyone. ICP discrepancies are a common problem. ICP descriptions need to be evidence-based. Embrace focused testing. Understand the difference between signal and noise: Not all feedback is actionable. Focus on pain points that users have actively tried to solve.

https://www.youtube.com/watch?v=FvBvNlMvKTg

AI-Generated Transcript

(00:00) there's a lack of clarity on the ICP there's a lack of clarity on what different personas or different icps might want what value means to them and that makes it very hard to guide people to set value if you don't know what it [Music] is welcome to experiment Nations podcast I'm your host Charlotte Bumford today we

(00:19) have a very special guest her name is Dr Elsa vanderberg who has worked on product management uh for B2B SAS startups and scale-ups so Dr Elsa would you be able to tell us more about you and your work experience sure um yeah um so I've been working in tech for about 15 years and I think um I think eight of those have

(00:47) been in product management um in various different roles um and I've been a solo preneur as they call it for the last two years which means that I've worked with a variety of of um B2B SAS startups and scale-ups in particular um many of them pre-product Market fit uh some of them thinking they have product Market fit but not quite

(01:09) having it um so kind of um in that space um and I focus very much on helping uh product teams become more outcom driven and evidence-backed so move from the classical feature Factory um to trying to achieve outcomes understanding which outcomes they should be driving and and figuring out good strategies uh to be

(01:29) doing that so that's been my focus for the last two years wow that's amazing so you're you have your own um business kind of like a consultancy type of thing or just me it's just me it's amazing yeah ex so it's a very tiny consulty so I wouldn't call it so I do some advising I do some coaching um my preferred way

(01:48) of working is as an interim um because I like really jumping in getting my hands dirty so having at least 20 hours a week with ay is is the best for me um even though I also do some advising on the site but of course then I can get a lot less um proactive right yes really good do the work yeah it's actually really good kind of like

(02:09) fits some of our um lifestyle nowadays the beauty of digital um work right so anyway I'm curious um because I want to know how like for us who are not in the product management area how does a day in your life look like well that depends a lot on the organization and on my role within that organization yeah um so I'm just going

(02:35) to think back about the last project I worked for which was actually very different from the project before um so depends right sometimes I'm brought into lead existing product teams so it's more like scale-ups they have already a pretty decent tech department a product Department that I'm there to create

(02:52) structure and and also maybe to up level that product team that is there um but in the last case I actually joined a very early stage startup where there was were zero product people and I was the first product person to kind of take over from a Founder um where of course then my work is way more individual

(03:10) contributor and and and very close to the leadership team trying to create leadership alignments um so that of course heavily impacts what my day looks like in any case there's always a lot of customer contact that sounds very blah and boring and like repeating what everyone says but I spent about well

(03:29) depending on the role right but in the last IC role I spent about 50 60% of my time talking to customers or audience members um talking to somebody exactly so somebody outside of the company so this is kind of like creating an MVP first and then um putting that out in the market and then eventually um finding out what the user

(03:52) behavior is before uplifting that product is that what you're trying to yeah so it depends like it it really depends right I hate this it depends on the maturity stage of a company but what I often see is I'm brought into a company they have a product out there I'm often brought in to optimize this existing product so they say hey we have

(04:13) a problem with our customer activation rates can you come in and Tackle our onboarding flow flow tackle our new user experience get them to Value more quickly and then it's going to be great and then I come in and I quickly realize that the problem isn't the onboarding journey the problem is that we don't

(04:30) really know what the value is yeah okay it's like another B2B s product it a c of B2B s products there's no clear competitive differentiation there's a lack of clarity on the ICP there's a lack of clarity on what different personas or different icps might want what value means to them and that makes

(04:49) it very hard to guide people to set value if you don't know what it is um so I'm often brought in with a pretty simple mandate and then it becomes quite complicated in it becomes this question of who are we serving and what is it that they want right like what strong recurring paino is still open for them that is currently unsolved

(05:11) um because we tend to stumble upon pain points that have already been solved oh that's true though yes yeah and that's I think a big part of good qualitative research is like digging a little bit deeper because when you do a lot of interviews or a lot of moderated user tests or any of the qualitative methods

(05:29) that I use people will give you a lot of pain points that's true but then if you Pro deeper right and ask the right questions you can figure out whether these are really strong pain points yes or just kind of nice have yeah and have they not already solved them yeah and and and also for me a good question is

(05:48) always like have you tried solving it before because often it's like nah and then it's kind of a hint that maybe they don't really care right there there might be exceptions to that so I think I think there's a lot of looking for that hook still that that angle still in the companies that I work with um which is a

(06:05) few steps before even delivering a good MVP right and is definitely many steps before running an AB test where you have very concrete idea it's going to be a or b right we're we're often not there yet in more mature companies we are there right or maybe that's like the fifth test that we layered upon other tests um

(06:23) but yeah there's there's no one siiz fits-all answer unfortunately true that's true it is kind of like difficult but if do you have like I know that there's no one siiz fits answer but do you have like a general type of process that you follow in terms of like when you go into a company what's the very

(06:42) first thing that you always do and then from there you go through the other Pro um processes or steps like how would that look like yeah so this is the beauty of being able to kind of have this same mandate over and over right do develop kind of a framework or process and the first thing I do always is I do

(07:02) one-on once with leaders of different departments to kind of see where we are at but oneon-one not in a group setting because I want to get right I want to see the differences and I always always find big discrepancies between important discrepancies between the leaders Department hi this is romal Santiago

(07:20) from experiment Nation if you'd like to connect with hundreds of experimenters from around the world consider joining our slack Channel you can find the link in the description now back to the episode um and one of the key things I look for is asking who is the ICP right who are we selling to who are we

(07:34) building for who are we messaging to yeah um which I think is fundamental to everything else right it's it's it's not um a marketing problem yeah it underpins everything we do right who's it for um so that's why step one I always find discrepancies there so I do a lot of work there in figuring out whether we

(07:56) can have more Focus there right and how we can best describe or Define this ICP and and where we need to create alignment and where it's maybe fine if we don't have alignment right maybe sales can do something slightly different from product but we need to make this explicit so that's like step one yeah um step two is understanding

(08:13) what is the core value to them right like and what is it what are we doing yeah do you think that most of the time most companies have not like especially startups right because again it's a startup do you find that most of their um icps are not defined and that's basically that may be one of the reasons

(08:33) why you're there to help them Define them or so they're often defined because a lot of people have read the books or looked in their LinkedIn feed and they've heard that you should Define an ICP right so yeah often they are defined okay the question is whether they are well defined right there's there's a

(08:49) variety of problems one is to have a very broad ICB to just say we're going to serve e-commerce shops sure which is super broad right it's that's not really an ICP that or that's that's that's not narrow at all and especially when you're just starting out when your startup or a scaleup I think you should be a lot more narrow than

(09:09) that because this is basically not having an ICP at all because it's a lot another one is of course having different departments working with different icbs and not knowing this of each other not being aware of this because it is sometimes fine right sometimes I have Founders or sales departments who are

(09:27) trying to sell more up market and product selling more for a medium tier or a building more for a medium tier and this can be okay if we have clear agreements about this but often they don't know from each other that they're serving completely different ICP problem um and then I see also ICP descriptions

(09:43) that are quite random where it was more like okay people told us we should describe our ICP yeah we'll just have a one- day brainstorming session we'll bring the leadership team together right and it's age yes and we wrote something down and it's narrow and it's cool and we wrote this down and then it went into our

(10:05) knowledge base and now we have an ICP right and that's also not really how it works right you want it to be evidence-based you want it to be iterative um it definitely has to be a living document right you can't just like park it somewhere and it should not be just a random fence just to be done like check off the

(10:22) homework um and I see a lot of ICP descriptions that Focus around certain attributes in B2B it's often about region company size Revenue right and I understand that these are nice attributes to work with because you can easily distill a a list right you can find a list of people who fit those attributes online and then sales can

(10:44) start harassing them for example um right it's term but usually it's not that those those are usually not the most important attributes to describe your specific ICP right maybe it's more about the specific pain point that they are looking for or it's about a specific Persona that's in there or it's a

(11:03) different way that the company is run or it's about the Tex stack that they use so there might be different attributes that are actually much more relevant for you to describe your ICP than company size so it feels a bit like this kind of hammer approach to defining an ICP which is not the best one so yeah problems

(11:20) that's true but having a narrow ICP means that also narrow also means a small data um so on on on um that perspective how do you deal with that small amount of data because again like we're about to delve into like experimentation and the product product management role that you have how do you usually um like how does it look like

(11:49) how does experimentation look like in your role and especially you are dealing with like you said a narrow ICP would that involve personalization I don't know what you mean by would that involve personalization what do you mean exactly by that so in terms of like uh the tests that you run would be specific for that

(12:13) um group only or what it would look like yes yes yes yes so so the whole reason why I care so much about narrowing down the ICP and I want the entire company align on that right I want the leadership team to to at least maybe not align but but agree with that this is what we're doing now right and and we will keep

(12:36) revisiting it but this is what we're testing now this is the one we're going for at least this quarter yeah Allin and because my Discovery and and and experimentation work will also focus on only that ICP yes I will not go outside of it I'm very bullish on that right and this is something where teams get lacks

(12:52) especially when you are relatively early stage and you struggle to find participants I see that teams are just happy that somebody signed up for the test right they're happy to get a participant and they are less strict on whether this participant fits the ICP um and that's to me very important I think

(13:11) one of the the biggest mistake that teams make because in the end you end up collecting a lot of noise you get feedback from people who do not fit your ICP who you're not building for um and you're LED astray right so that's something I I push quite a lot on and I think when again in early stage setting

(13:27) startup scaleup you are best off going quite niche in the beginning right I'm like one of million people who are saying this right I'm not going to stray too much on that but if you try to build for everyone it's going to be quite mediocre for everyone right you need to be quite specific on who you're building for so

(13:46) but then the discovery work also has to focus on these people and not anybody else right you have to be very strict with yourself there I agree I agree now um what are again like you've mentioned there are a lot of noises that would come along with some of the testings or you know the product management um

(14:06) process what are those common challenges that you face with during the testing or during the prototyping period or you know during user testing H and how would you usually um get past it yeah um so I think one of the most common challeng that we don't talk about enough is that we struggle to find enough participants

(14:31) oh yeah yes really like this is very very hard especially if you want to test with customers so people who are already using your product who fit your ICP right so you have two filters already they have to be customers there's a fit your IB and they have to say yes to being a participant so then you already

(14:49) have like like hardly anybody left and I've seen their statistics so I'm I'm a big fan or I I I I Rely a lot on Teresa torres's work um and I'm also part of her community where people talk to each other about their General statistics that they get when they're sending out an email blast to customers asking them

(15:08) to participate in interviews and how many people actually show up for an interview and it's a steep drop off right and this is quite depressing for teams because they're all excited to start with talking to customers within their ICP exactly and they can't get anyone on the phone so that's um challenge number one Challenge number

(15:25) two is then getting enough data where in in qualitative terms I think if I have conversations with 10 or 15 people I can already get quite a bit of signal out of that um in a in a good case right like that that's if I had a narrow ICP If I didn't if I went quite broad there will be no patterns because there will be

(15:45) like wide variety of people who have wide variety of problems so it's going be really hard to detect patterns and um the third big issue is um distinguishing signal from noise right so understanding yeah yeah what are they just kind of complaining about but they don't really care right what actually will move the

(16:04) needle um yeah so it's it's kind of those things but the beauty is that you can layer a variety of test methods upon each other right because it is super uncertain it is right 10 to 15 people is nowhere near statistically significant you have to kind of gauge whether they really mean it or not right there's like

(16:24) a lot of gut feeling involved in I agree test in Discovery and validation but that's okay for me because I layer it's not just one test right first I'll do interviews but then I will do um maybe a landing page test then I might do a moderated user test then I might do a prototype right I'll just for the same

(16:44) assumptions they will go through a variety of tests that will build my certainty would you would you use the same um people that you interviewed in um to go through the landing page tests um and other tests y yeah it depends on the case right but I I have done that in the past because sometimes for example I've also done moderated

(17:08) user tests with the first version of the product the old version of the product and then moderated user test with a new version of the product and then sometimes it's very interesting to do that with the same people who have seen both versions and can you can see how their response is different right how

(17:24) might they might respond way more fiery to the second thing and they will also tell you like w this is so much better than the first thing and that that gives me something of course y I was also about to ask that question that you've um touched on earlier how do you make sure that the findings from your user

(17:42) moderated test are actionable items like how would you define something that would move a needle that depends a lot on the target audience yeah um you have like an example or you know like yeah so one thing that came to mind just now like I I used to the the product I built before was a project management tool um which was sold to

(18:07) project managers and Founders at uh agencies for example and with them I really found that if they haven't looked for a solution right they're not going to be interested in buying it they have to have so so I would ask them okay um can you walk me through the last time you did X YZ and kind of PR for the pain

(18:28) points that they they experienced in that process right so I'm trying to outline the steps on their job map like what are they doing step by step and what are the pain points Associated to each step um and then for each of those I tried to figure out have you ever really tried solving that like have you

(18:45) looked for a solution and I found for this particular audience that was very important because they're they're the kind of people who if they didn't go out and look for a solution they're they're not going to care if you present them with a solution right but then I switched I worked for a different

(18:59) company that b built a developer tool um where this didn't work at all right because I kept asking of course like the same similar process and then asking have you looked for a solution and often they said I haven't oh but then they had switched to different tools that they read about on Hacker News that just

(19:18) looked shiny and cool right they just said a this looks shiny cool they started using it and they stuck with it forever which is a way of working which does not fit the old targeted audience at all but it is something that happened in this target audience so then I have to of course adapt how I interpret the

(19:35) things that they say based on behavior that I've seen for this particular target audience um so there's no one siiz fits all answer true it really depends on like how this specific target audience seems to tick and you can figure that out by talking to them and watching them that's really really interesting now um I think the other

(19:54) questions like you have touch on godfield cuz again like because it is a narrow um set of audience how do you um how do you usually remove the having that objective um Outlook in that specific Pro um product versus the subjective so like how does your empathy play a role in the things that you do um yeah so I'm

(20:21) going to be brutally honest it's pretty subjective I do right like there is no truth there is no spoon there is no real objectivity because even then when I talk to people I have to interpret In This Moment do I believe what they say right how serious do I think they are what is really important that that

(20:43) there's so many layers underneath right you try not to reach you try to take what people say and how they react at face value but there is a lot of interpretation of course there has to be a lot um and then also with qualitative tests right i' run I do maybe 30 interviews I do 10 usability tests I do

(21:00) 10 moderate us it's it's low numbers right so there's a lot of kind of making sense of that small amount of data so kind of putting a puzzle together and there is so much subjectivity in that so for me um this idea of saying evidence-based eradicates subjectivity or gut feeling or opinion nonsense and I've also worked

(21:26) with um so I I I'm I'm quite a fan of collecting evidence right but I love working with people who operate with a strong conviction so for example I I really like that like especially people who are very experienced right they're not throwing it out like no I just think it should be like this but they say I feel

(21:45) like this is wrong right so I can present them with all of this data and I say look this is the signal that I see and here how is it supported here's my opportunity solution tree look how heavy is this brand how many insights are here this seems to really be it there seems to be something here but then I'll work

(22:02) with somebody who says I don't think so you know like based on just my knowledge and my background right so I I'm convinced it's the other thing where we don't have that much evidence I I really feel like there's something there can we dig more there um and I quite enjoy working in this kind of um difficult envir right where it's not

(22:25) like because you can't blindly follow the evidence that would be too simple there is a lot of outof the-box thinking questioning critical assessment of what you find right so I quite enjoy having these conversations and then saying no we're we're going to keep exploring the thing where we have more insights or yes

(22:42) let's dig there even though we don't have many insights but maybe there is something there yeah um it's it's not an exact science is what I'm trying to say I agree with you like I think like you know the best analogy that I can think about is like when you're interviewing like crime and stuff like evidences

(22:57) based there will always be um a gut feel and there will always be while perspectives are different there will always be one thing that would be similar out of all the stories uh that would come out and that like it may be subjective but it would come out of your gut feeli as well that you are on the

(23:19) right path on finding out what the problem is um um it's not like a magic formula right like exactly it's not yeah yeah I like Rory southerland's thinking on this quite a lot from from ogl V this big marketing agency he wrote a book Alchemy and then the title is longer sorry with Alchemy and he talks a lot

(23:39) about this kind of creativity and and the fact that um running a business it's not like you follow a formula right you follow this process step by step by step and it splits out the answer because if that were the case everybody would be building the same products right exactly it's not how it works there is a lot of

(23:55) moonshots and and creativity and thinking out of the box and doing unexpected things exactly um question probably I think this is another good question how would you handle conflicting feedback or opinions from different users during a testing so it depends like of course assuming they're all from the same ICP

(24:17) right um this is not uncommon and I think um even within your ICP there's Al always segmenting to be done um so it's very interesting to understand understand is there like I always ask profiling questions to begin with to try to understand be able to segment them and to try to understand why is it different for this person than for this

(24:37) person it doesn't mean that it's not valid anymore it's just that there's maybe something else that they're doing or something in their background that makes something excruciatingly painful for customer a but not painful at all for user customer B right and I it's my job to try to understand what is the

(24:55) difference between them like why is that so that I can segment them even smaller and maybe I'll find within my ICP an even smaller like beach head underserved segments that is even more interesting okay awesome I think like we're almost out of time but I just want to know like for someone who is just

(25:16) starting in the field um who wants to go into product management or some something that you do what would you um what would be your advice on how to master the techniques that you have like what's the first place that they need to go who do they need to listen of course they have to listen to Dr Elsa

(25:38) vanderberg but aside from Dr Elsa vanderberg who else you know um so what would be your advice so I think for me I've learned the most on the job right and so I I like this advice of like don't look for a company but look for a kind of Mentor like somebody that you would like to work under right so it's

(25:58) it's it's not really about the company or the product or anything it's about the team that you work in right so I would um if LinkedIn is a very good source of information obviously and and you can see you can find people that you really respect right like I have a couple of people that I I found that I'm

(26:16) like okay these people are really really good like a Matt learner or Ed Biden or Ben Williams or like many others right there's like a list of people that I really respect because they put out really good content um I think if you follow these kind of people you can think okay if I could ever work with any

(26:32) of them right that would boost my career like crazy because for me I've worked with a variety of leaders I've worked with a variety of startups when you pick the right people that's where you learn the most right so I would um look for mentors or the the kind of people that you would like to work with um and and

(26:50) try to apply for jobs like that that's amazing so how would people get in touch with you they can always go to the experimentation website or your LinkedIn account LinkedIn I think is the best bet so feel free to find me on LinkedIn um maybe you can hyperlink it here my name is Elsa fber and you can find me and always

(27:10) reach out awesome I feel like there's a lot of things that we can talk about in your area but there you know we have like a certain amount of time but I think we should have like a session number two in the next few months to talk about more things and so I just want to thank you Dr Elsa for this interview and for your

(27:32) time really really really appreciate it um and we've learned a lot again like I feel like we're just scratching the surface with this interview and I really do think that there should be a second session so up to you once you're available by then um get in touch with us again and uh thank you so much again

(27:52) for your time I will do good thank you so much for having me I had a blessed thank you oh no worries thank you hi this is Romo Santiago from experiment Nation if you'd like to connect with hundreds of experimenters from around the world consider joining our slack Channel you can find the link in the

(28:05) description now back to the episode

If you liked this post, sign up for Experiment Nation's newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK