Nancy Kawatra on delivering value for the long-run
A Conversation with One & Body’s Nancy Kawatra about Experimentation
A lot of Experiments focus on driving revenue in the near term. However, overly focusing on revenue risks ignoring improving the customer experience. I recently spoke to One & Body’s Nancy Kawatra about running Experiments that focus on the customer experience, how companies are bringing Experimentation in-house, and the skills that Experimenters need in order to excel.
Hi Nancy! How are you?
Hi Rommil! Thanks for asking. I am doing pretty good and trying to cope up with the new normal and still enjoy the Canadian summers. How about you?
Could you share with our readers a bit about yourself and your career journey thus far?
Sure. I am an Optimization Specialist with over 3 years of experience in website optimization. My erstwhile role includes working as a Senior Optimization Consultant with VWO where I worked on improving user experience of client websites for better conversions.
Having worked with many companies, what are the most common misconceptions about Experimentation that clients have?
Having worked with companies from myriad industries such as SAAS, eCommerce, Blogs and Hospitality, a common misconception that I encountered is relying too much on macro conversions such as revenue for declaring a test as a winner & also, assuming that experimentation is time-bound activity. Yes, all the experiments are expected to deliver better revenue however experiments that are undertaken towards improving the user journey can improve all the other behavioural aspects with marginal difference in revenue figures. These types of experiments are ones which bear the fruits of improving the revenue in the long run. Also, it is very important for the clients to be aware of the fact that experimentation is an iterative process and we can never say that a website is fully optimized.
“[Experiments that are undertaken towards improving the user journey] are ones which bear the fruits of improving the revenue in the long run.”
I really like the way you phrased that. You’re so right, though. We often focus so much effort on generating revenue immediately, that we ignore the fact that a better customer experience has the potential to generate customer loyalty and in term much more revenue in the long run.
Changing gears. As you start working with clients?—?what are the questions you often ask them to decide where to start focusing?
Before jumping on a call with the client it’s very important to do the prep of learning about the business, by consuming information available online. This will help to form relevant questions to be asked that are important to form the foundation for all the experiments.
Recording a copy of the answers is also a good way to have a reference point in case of multiple clients.
The most important questions include having the client describe his business since it’s always important to hear directly from the horse’s mouth, including more open-ended questions which most of the time will stump you by always giving something indispensable to catch on, client expectations, their pain points, business operations and their past learnings with the experiments (if any…). While being on a kick-off call, it’s imperative to ask all the questions that might pop-up in your mind, no matter how vague they seem to be.
In your opinion, can you tell me a bit about the advantages and disadvantages of Bayesian vs Frequentist approaches?
Answer. I see a lot of A/B testing tools adopting the Bayesian methodology & eventually doing away with the Frequentist Model and they have their own ideologies around doing so. To be honest, a lot of times, this is not a hard question that would make or break the deal unless your client wants to have a hard look at the whitepapers. I have a one-shot answer to this question, that “What is the probability that A is better at converting visitors than B?” is far more conclusive than “If A will beat B?”. The frequentist method never tells the probability of the likelihood of A beating B and thus is somehow much more convoluted.
My opinion can be biased towards Bayesian since I have used tools with this approach and given the fact that we always intuitively agreed on the winners declared by the tool.
Some Experimenters feel that Frequestist is what people should use if they want to learn?—?what are your thoughts on this?
As I stated above, in the statistical world, we have proponents for both since it’s an independent decision to prefer one over the other and is almost similar to choosing your political party you want to vote for. Moreover, at the end of the day, they are all good measures to learn about the data results and make conclusions.
Bayesian uses prior knowledge i.e. the historical data in order to reach the statistically significant levels however contrary to this, Frequentist doesn’t rely on priors due to the risk associated with the previous experiments’ knowledge which has the tendency to skew the results. Thus it’s completely onto the decision-maker to responsibly choose the right tool which resonates with his/her affinity for either of the methods.
In your opinion, when should a company decide to take A/B testing in-house?
In house A/B testing is slowly gaining its way towards being recognized as an important internal function and its importance vis-à-vis conventional forms of marketing. If a company is just starting off, then in my opinion it’s best to outsource the task to gauge its impact and to gain a deeper understanding of the skills and tools required to accomplish testing before considering an in-house team. Adopting an experimentation culture is in itself a sign that a company takes its service quality seriously and wants the user to have a delightful, frictionless experience.
“Adopting an experimentation culture is in itself a sign that a company takes its service quality seriously and wants the user to have a delightful, frictionless experience.”
How do you demonstrate ROI for Experimentation?
It’s a no-brainer when it comes to eCommerce websites wherein most of the experiments, some metrics are sacrosanct which are the Ecommerce conversion rate, revenue, transactions and AOV. However, as stated earlier sometimes it takes a long time to bear the fruits of the changes and any improvement in the behavioural metrics is important to consider to pass the final verdict. Thus in my opinion, the reporting should not be restricted to only a few objectives but it’s important to dig deeper and understand the impact of the test on behavioural aspects as well.
In your opinion, what skills should Experimenters have to excel?
It’s imperative for the Experimenters to be able to think analytically to draw conclusions from the data derived from testing and also to be able to draw inferences from observations via qualitative tools such as session recordings, scroll maps, recordings, heatmaps etc. This will pave the way to deriving the hypothesis for the testing journey with the right prioritization of ideas.
Time for the Lightning round!
What would you do if you couldn’t work in Experimentation?
I would have been pursuing my Masters & beyond in Economics to become a Teacher.
What is your biggest pet peeve?
With the current pandemic sitch, the uncertainty that hangs around everything is my biggest peeve right now.
Bayesian or Frequentist?
Bayesian.
Describe Nancy in 5 words or less.
Ambitious, Hardworker, Adaptable, Introvert & Dog-Lover
Nancy, thank you for joining the conversation!