Designing an Experimentation strategy that supports product-development with Widerfunnel’s Zharina Pelea Experiment Nation

A conversation with Widerfunnel’s Zharina Pelea

I recently spoke to Zharina about how she crafts a customer-centric Experimentation strategy that helps drive product-development, how she gets buy-in for her strategies, and how she defines a good Experimentation strategy.

Rommil: Hi Zharina, thank you for taking the time away from your busy schedule to chat with me today. How are you?

Zharina: Hey Rommil! I’m doing well, thank you for asking and for reaching out. I hope all is well with you too.

Could you share with us what you do and a bit about yourself?

Happy to! I moved to Canada in October of last year from New Zealand. Two months later, I joined the Widerfunnel team as an Experimentation Strategist. Widerfunnel is a Vancouver- based company that provides full-service behavioural experimentation and program implementation to ensure that our enterprise clients are enabled to make evidence-based decisions for their business. Widerfunnel is part of the GO Group network, an international consultancy that offers experimentation services to enterprise multinational companies.

As an Experimentation Strategist, I manage a number of client accounts. On these accounts, I identify opportunities for revenue growth and develop strategies that improve on their online KPIs and overall business metrics that are validated through experimentation and actionable data.

Before moving to Canada, I was a Senior Conversion Consultant at Catchi Digital, an Auckland-based digital company specializing in Conversion Optimization and Data Analytics. Catchi is part of the wider Havas network, a leading global advertising and communications group based in France. During my time there, I led the Conversion Optimization Strategy of New Zealand and Australian based accounts and managed all aspects of the client relationship and CRO program life cycle.

Prior to my time in Catchi, I worked at FIRST Digital as an Inbound Marketing & Conversion Consultant. FIRST is a specialist digital agency that offers all facets of digital marketing from strategy, CRO, analytics, search, media, and email marketing. FIRST is a digital division of Beyond International, a renowned ASX-listed film and television production company based in Australia.

Prior to that, I worked in the Banking & Finance Industry for about 7 years in the Philippines. For the most part of my career in Finance, I was a Preferred Relationship Manager, managing the financial portfolios, providing banking solutions and wealth management services to high-tiered clients of the bank.

Wow. That’s quite the journey! Well, a happy belated welcome to Canada!

“Experimentation within user experience design is about ensuring that the value of the product is seamlessly integrated into the customer journey, across various customer touch points using a rigorous and cyclical scientific process…”

Can you share with us your thoughts about what role Experimentation should play in product development?

Wow! That’s a big one.

The Goals:

I think to be able to best answer that question is to first look at the purpose or goal of experimentation and product development.

The goal of product development is to transform an identified need of a specific target market to a viable solution or product using an agile process to allow for frictionless scalability.

Experimentation within user experience design is about ensuring that the value of this product is seamlessly integrated into the customer journey, across various customer touchpoints using a rigorous and cyclical scientific process that enables the business to constantly generate relevant insights that builds into the customer theory and overall improvement of the customer experience.

Looking at those goals, you can see that both product development and experimentation have two things in common?—?customer satisfaction and process. I think it’s common for product managers to find themselves facing a hundred really cool ideas or product concepts that need to be implemented straight away but that requires considerable investment in resources?—?time, effort and money. That’s most often, not realistic.

Product managers are then faced with finding answers to hard questions like: Which of those ideas do we need to prioritize based on our existing resources? Or, how much time do we dedicate on iterating on a product idea vs. moving on to the next? With every decision, there will definitely be opportunity costs and associated risks, aside from gains, of course. This is where experimentation comes in?—?to help answer these hard questions and mitigate any associated risks.

“Experimentation can help improve the ROI of product development efforts by reducing associated risks.”

The Risks:

There are many risks that are associated with new product development or a modification of an already existing product.

But let me focus on a few risks and where experimentation can help in reducing those associated risks.

The product doesn’t match customer needs and wants

This happens when the research and feature building stages of product development fails to really hone in on the customer problems and preferences that the product can solve and deliver. Or, it can also be the case that customer needs and wants have changed.

“Experimentation gives the product managers the confidence they need to pivot when things are not working or no longer relevant.”

Including experimentation as a way to mine for customer insights on an ongoing basis allows product managers to gain supplemental customer data points that evolve with the customer. Additionally, through testing, specifically through minimum viable testing (MVT), you can identify and eliminate features of the product which are not relevant to the customers and even modify your product based on an uncovered need.

Additionally, experimentation gives the product managers the confidence they need to pivot when things are not working or no longer relevant.

2. Product development comes with financial risk

It’s a possibility that the product is not able to generate enough customer appetite that would bring profit or that costs for bringing the product to market such as advertising won’t be covered by the product’s final price. Sometimes, this risk is increased when more and more resources are invested in the product build in the hopes of achieving the best outcome.

Experimentation helps reduce this risk through running ‘painted door’ tests. What’s a painted-door test? It’s used to gauge demand and interest from your target customer of a new product or feature. This type of test gives users a part of a feature in the user experience that hasn’t been fully built out to measure engagement and see if that’s something that your customers find valuable.

For example, you have an eCommerce business and you would like to know if adding product videos on your website’s product pages will increase engagement and conversion (purchase) rates.

One approach is to test adding a video thumbnail on a select group of product pages and making this appear as fully functional. When users click through the video thumbnail, they are shown a message that the feature is coming soon and thanking them for their interest. You can also choose to set up on-site surveys to supplement your research.

If there is considerable engagement with the video thumbnail and additional data from your qualitative research that shows customer demand for this feature, then you can make an informed decision to invest the resources for video production with reduced financial risk, compared to not running this test at all.

Furthermore, experimentation sets the measurement framework in place to evaluate the impact of the product being built quantitatively. By evaluating these ideas and making decisions of what to do next based on data, product managers can do away with any guesswork and prevent wasting resources

3. Risks associated with fearing failure

I think one of the main challenges that product managers (or most people, in general) face is the fear of failing. Depending on the product and expectations, some would tend to lean towards choosing solutions or approaches that can guarantee success. Having this kind of mindset can result in the product not meeting what customers really need and want.

Experimenting more doesn’t always result in guaranteed wins but getting a clearer understanding of the customer through early feedback and testing can be a good thing since product managers can identify what works and what doesn’t. Also, doing this early on in the initial stages of the product development is better than doing all testing after the final product is launched when a lot of resources have already been utilized.

Thus, Experimentation can help improve the ROI of product development efforts by reducing associated risks.

In your opinion, what makes a good CRO strategy?

A good CRO strategy looks at the big picture! The goal of CRO or experimentation is to improve the customer experience. Since we can’t ask every customer honestly how they feel, we use proxy metrics, such as purchases, to identify what is positive vs negative.

Additionally, it should not just be limited to isolated page tests but should consider testing user flows, identifying bottlenecks and plugging leaks. An often misconception about CRO or Experimentation is that people assume that there is just one big experience that will revolutionize the way your customers engage with and ultimately convert. Conversion rate optimization is meant to be a series of ever-evolving experiments?—?not one-off experiences?—?that continue to improve the customer experience over time and address customers’ constantly changing needs and preferences.

I also think sometimes businesses/brands get so caught up in achieving their own goals and key performance indicators (KPIs) that they forget about the goals, needs, challenges, and desires of the customer. CRO or Experimentation strategies need to not only be agile and data-backed but also customer-centric.

Additionally, there should be a rigorous and robust strategic and measurement framework. This establishes a consistent approach to experimentation across the entire organization, enabling more people to run more experiments and deliver value from the experiments.

“In product development, experimentation or CRO strategies need to, not only be agile and data-driven, but also customer-centric?—?solving for the main customer problem that the product was built for.”

How do you decide where to start working? Do you have a preferred order of things to work on?

It’s important for me to start with understanding why the client wants to start or scale experimentation as well as the ‘why’ of the business. This is normally answered through an in depth-conversation with the client about their business, goals and previous experience in experimentation or other methods. This initial conversation is not the first and final one, ongoing conversations are necessary to keep a pulse on any business or goal changes to ensure that the experimentation program is always aligned and always delivering value to the client.

Then, the conversion research comes in which involves a systematic search for data about the customer?—?jobs-to-be-done, preferences and online behaviour. This entails both quantitative and qualitative methodologies to create hypotheses that can be validated through testing.

How do you sell your strategy and ensure that teams follow through with it?

In presenting strategies to clients, I include an experimentation framework that ties in identified testing areas, levers and hypotheses to program KPIs and a clearly-defined and business-aligned overarching goal. By aligning the experimentation framework and ultimately, the experimentation program, to the overall business goals, you get the buy-in from the stakeholders because you speak their language! They know that you care about their goals and are there to work collaboratively with them to ensure not only the success of the experimentation program but their success as well.

To ensure continuous engagement with the stakeholders even after a new strategy has been presented, regular calls are scheduled to go through the experimentation pipeline and results analyses. These activities allow for constant collaboration, education and communication with the stakeholders.

Because CRO can be so unpredictable, what’s your approach to managing expectations?

I think, increasing trust and ensuring constant communication are the best ways to manage expectations.

From the beginning, it’s important to set mutually agreed-upon goals for the CRO or experimentation program that aligns with the client’s business goals. This ensures that everyone is on the same page and is working towards the same end result.

I also make sure to manage their expectations when experiments do not result in positive uplifts in the primary KPIs, for example, and communicate the value of the learnings and factor in the potential preventable losses from those experiments.

Depending on the experimentation maturity of the business, not finding a ‘winning experiment’ is seen as a failure. As long as your experiments can provide a deeper understanding of your target customer, every experiment should be seen as a success.

It’s important to keep up a culture of experimentation even when it doesn’t yield the expected outcome in the eyes of stakeholders.

Based on your experience, how can you tell if a company’s leadership embraces an Experimentation culture?

When they use experimentation for digital transformation and growth. This means that decisions are well supported by data and that the organization is willing and able to answer hard business questions and challenge ingrained assumptions. Experimentation is done across all business units and results and insights are shared across the whole organization.

Changing gears: What tests do you feel Experimenters need to stop running?

That would be tactical experiments. Each test should not exist in a vacuum but instead part of a strategic and continuously evolving way of testing.

Finally, it’s time for the Lightning round!

Bayesian or Frequentist?

Bayesian

What is your favourite test from recent history?

An experiment where the value propositions of the business were added and highlighted on the homepage of the site that led to a 50% increase in the CVR of the primary metric of subscriptions.

I love that one. Congrats!

If you couldn’t work in Experimentation – what would you do?

A neurosurgeon or work for the United Nations or a professional taste tester of Michelin restaurants.

Ha! I never doubted my career-choices until this moment.

Describe Zharina in 5 words or less.

Results-driven, Out-of-the-box thinker, Hardworking, Problem Solver & Kind.

I guess that’s less than 5 words, well played. 😉

Thank you, Zharina, for joining the conversation!

Thank you!

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.