Widerfunnel’s James Flory shares his perspective on how to succeed at Experimentation
A Conversion Conversation with Widerfunnel’s James Flory
As Experimentation matures and gains more traction as a discipline, more and more professionals ask, “How can we succeed at Experimentation?” Other than reading this blog, I (virtually) sat down with James from Widerfunnel to understand his career path and to get his take on what it takes for people and companies to excel in this exciting field.
Rommil: Hey James?—?Let’s start with the basics. Tell us a bit about yourself and what you do.
James: I’m a senior experimentation strategist at Widerfunnel, a leading conversion rate optimization and experimentation agency and consultancy. I’m responsible for managing, supplementing, or consulting on the experimentation programs of some of North America’s leading brands.
As a strategist, my key responsibility is identifying experimentation opportunities that will drive revenue and/or insights for clients. Analyzing data I’ve identified as an opportunity, I’m responsible for designing an experiment, getting it approved, supporting my team to design, develop, and launch it. I’m then responsible for monitoring, analyzing, interpreting, and reporting out on the results and impact of the experiment and, ideally, iterating on its success or failure.
Wow?—?that sounds like a cool gig. While I’m sure it’s challenging at times, it must be quite satisfying. For those new to the field, could you tell us a bit about how you got into experimentation?
I studied marketing and business at the British Columbia Institute of Technology. That, coupled with my experience as a self-taught freelance web developer uniquely positioned me for success when I applied at Widerfunnel. The technical know-how enabled me to understand the technologies, limitations and opportunities of experiment design, and catch on to UX and design principles very quickly. The background in marketing and business helped me understand my clients’ businesses and needs quickly and provide a depth of insight and outside perspective. Since I started in the industry it’s been immensely satisfying. From working with fantastic clients and partners across North America to meeting and collaborating with our GO Group Partners around the world.
What key skills should someone who’s interested in getting into experimentation have?
From my perspective I think it’s an important balance of technical, strategic/analytical, and creative skill sets.
On the technical front, being comfortable in the myriad of technologies you’re likely to encounter will be helpful. Analytics tools, testing tools, user research tools, CRM systems, GitHub, you name it, you may need to use it.
On the strategic and analytical side, making sense of data and being able to extract an insight or a pattern and turn it into an actionable finding is critical for creating experiment ideas and also understanding experiment results. Turning numbers into a story is hugely valuable to any organization.
On the creative side, it never hurts to be able to think outside the box. Great ideas are never driven by data alone. Data identifies the problem or opportunity, but there can be an infinite number of alternate solutions. That’s where creative thinking skills shine.
Here’s a tougher question?—?what does a company culture of experimentation mean to you?
I think a true culture of experimentation has to be grounded in humility, curiosity, a drive for growth, and a level of comfort with uncertainty. Experimentation can’t just be a means to an end, it needs to be seen as a cost of doing business and integrated in the foundations of the organization.
In a true culture of experimentation, people will not only be enabled, but encouraged to try new things and explore new opportunities. Importantly, they shouldn’t be punished if their explorations fail, but rewarded for their learnings. Everyone needs to be OK with being proven wrong, almost excited by it, and the organizational structure needs to be relatively flat with a low power distance to embrace this.
Why do you think some companies have a hard time embracing a culture of experimentation?
If an organization falls short in any of the areas I highlighted above they’re going to struggle. I think humility is a really difficult thing to come by in business, and that, coupled with commonly high power distance organizational structures incumbent in most large companies, is why experimentation programs are facing an uphill battle. HiPPOs don’t like to be proven wrong and as a result experimentation can be very politically frictional. It isn’t always the HiPPOs though, different teams are often in a power struggle for budgets, resources, authority, etc. that can all make experimentation difficult because most good experiments transcend one team’s domain.
A website is a perfect example of this. Copywriters, designers, developers, marketers, executives, and many other individuals or teams are all going to have had some input on a website. If you come along and propose an experiment that challenges someone’s contribution, their natural inclination is to push back. Politics make meaningful experimentation really hard.
“Experimentation can’t just be a means to an end, it needs to be seen as a cost of doing business and integrated in the foundations of the organization.”
If you could name 3 things that companies should do, not just to embrace experimentation, but to also see the rewards of it, what would they be?
- Understand and experiment on the entire customer journey, not just the page. The reason users may be struggling at a step or on a page may not be solely because of the page. Maybe an ad they encountered set the wrong expectation and you’re optimizing the wrong part of the journey. Don’t just analyze a single page, understand and analyze the journey and how it all fits together.
- Seek an outside perspective. I’d say one of the greatest values we bring to an organization is a fresh set of eyes with a ton of experience. All too often organizations and teams get in way too deep on their own products and services and develop tunnel vision that makes them blind to fairly obvious shortcomings of their experiences. This fresh perspective has provided a number of “Aha” moments to my clients over the years before we’ve even run an experiment.
- Don’t give up. Killing an experimentation program because it’s not producing the results you expected may be like throwing the baby out with the bathwater. You may be equating bad process and bad results when they may not be the same thing. A good process can result in a bad outcome and that’s OK. Make sure you’ve set your KPIs and metrics accordingly to make this distinction.
“If you come along and propose an experiment that challenges someone’s contribution, their natural inclination is to push back. Politics make meaningful experimentation really hard.”
If you could go back in time, what advice would you give your younger experimentation self?
Get into it earlier! Widerfunnel actually gave a guest lecture at my university while I was there but it didn’t click with me until a few years later when I joined the company how enjoyable the line of work was.
I’d also say pay attention to the behavioural sciences earlier. I think there is a tendency to equate experimentation to user experience and design very closely, so studying UX principles is likely an area where most people flock to to learn quickly (myself included). Later on I found that that’s really just a later stage component in the bigger picture of the customer journey. Instead the underlying motivations, behaviours, and biases exhibited by uses and customers can be far more fascinating and lucrative to design for than simply best practices. This finding is whats led Widerfunnel to take such a vested interest in our behavioural science team as well and really start to dig deeper into this well of commonly untapped insight.
Finally, Bayesian or Frequentist?
I don’t think I feel one is inherently better than the other. I think if you understand the method you’re using and what it requires of your test design you’re in good shape.
More controversially, I think the reliance on statistical significance in the world of business experimentation is actually hurting more than it’s helping these days, but that may be a bigger topic for another day.
Oh, that’s interesting! Maybe we need to host a panel one day and put that topic out there.