Kenya Davis on the challenges of running an Experimentation program at a large corporation
A Conversation Conversation with Lowe’s Kenya Davis
Leading an Experimentation program is never simple. You have to juggle a lot of stakeholders, stay on top of technology, and be the steward of the scientific process. Doing this for a multi-billion dollar international corporation is another level of hard. I recently spoke with Kenya from Lowe’s about how she got into the field, things to watch out for when dealing with execs, and the importance of a learning knowledgebase.
Rommil: Hi Kenya! How are you? Thanks for chatting with me today!
How about you tell us a bit about yourself and your role over at Lowe’s?
Kenya: Hi Rommil, It is a pleasure to chat with you on my experience at Lowe’s. I started out as a Resolutions Analyst, which was much like a liaison between what was then separated out as business and technology. My journey from there catapulted rather quickly as I moved into a newly created role managing content authoring for all Lowe’s digital platforms. At that time, I inherited 11 authors. We worked directly with marketing and merchandising, supporting all content swap outs. Naturally, leaders wanted to know more about the correlation between business efforts and revenue. I hadn’t thought much about A|B testing at that time until I noticed the many changes we were making that were not based on data. My manager at that time technically owned that space and graciously allowed me to explore what A|B testing would look like at Lowe’s. So now I’m here! We run the end-to-end process and govern the scientific login threaded through the final testing decisions.
“I hadn’t thought much about A|B testing at that time until I noticed the many changes we were making that were not based on data.”
So, I read that you have a Bachelor of Science in Physics. Can you share with us a little bit about your journey from the conservation of energy to the conversion of customers?
HAHA! That is a beautiful way to personify that transition. So I worked on a project at the University of Oklahoma back in 2013 that observed the presence of different gases in space objects. Specifically, Active Galactic Nuclei.
I sat in a room with 3 monitors and downloaded archival data from the Hubble Space Telescope, then ran it through a few python scripts that normalized the data, and configured it to predetermined spectral models. Often times, the data collection was so massive I found myself staring at screens for hours waiting for something more exciting to happen. Like NASA was going to burst through the doors and demand that I join their elite force of scientists including Michio Kaku , Bill Nye , the ghost of Nikola Tesla and the fictional character Samantha Carter (Stargate SG-1) on a mission to defeat aliens from a threatening galaxy that only I’ve identified in my research. Clearly, I was losing my mind and yearned for human interaction.
LOL!
So I pivoted my educational career to focus on the impact of technology in people’s everyday lives. I began working in a science museum (Discovery Place, Charlotte, NC) as an educational presenter. There was more opportunity to showcase my creative side with character reenactments like Berenib, an ancient queen of Egypt during the first appearances of Hieroglyphics and science shows blowing up hydrogen, setting ice on fire, and 3D printing dinosaur fossils. This wasn’t something that satisfies the full curiosity I had with technology but it opened the door to a rather large company called Maersk Line.
The oil and shipping company had just launched its first iteration of MyMaerskLine.com and hired a group of us to not only onboard their customers refusing to accept their first step to adding technology into the customer experience but also to identify areas of opportunity within that journey to add more iterations. This was the first time I started to see a real career in e-commerce.
That is quite the journey!
Changing gears. Every company is at a different comfort level in terms of Experimentation?—?what’s your approach to training stakeholders on the value of Experimentation?
It’s one thing to say you’ve A|B tested before, but it’s another to truly understand what it means to run a proper A|B test. I’ve found that the comfort levels of voicing the support of the idea are rather strong with many stakeholders. But now that we’ve crossed the line between back in the day remedial banner testing to strong feature canary tests, fallout redesign testing, and cross channel testing, the comfort levels have almost receded. As a way to counteract this insecurity, we work with our leaders to provide digital classes on scientific methods, KPI breakdowns, and analysis of how people’s social behaviours interrupt the conversion rate throughout the year.
“…we work with our leaders to provide digital classes on scientific methods, KPI breakdowns, and analysis of how people’s social behaviours interrupt the conversion rate throughout the year”
Of all the Experimenters I’ve interviewed very few, if any, venture offline. Can you share with us what kinds of Experiments you run offline? And how do you even run something like that?
Offline is one of the hardest areas to test. It is also uncharted territory for most retail companies that also have brick and mortar stores. We’ve only scratched the surface of this path by running basic buy online, pick up in-store promotional tests. Promotional tests are already hard to quantify as they are often threatening sample sizes and basic test run times. Honestly, the process is still undefined, but hypothetically, the factors that must be taken into consideration for offline ventures include, store conversion, foot traffic, online leads that connect to specific stores, and most of all inventory.
“Offline is one of the hardest areas to test.”
As I understand, you report to Execs on experimentation?—?what kinds of things are they usually interested in, and what kinds of things raise flags?
Execs are a different breed when it comes to testing. Their main objective, as it should be, are results that lead to increases in revenue. Whereas my team’s goals fluctuate with traffic flows between driving KPIs and collecting observational tests to increase our library of information. Red flags for both of us include more inconclusive tests than conclusive tests and the occasional slip of tests with technical setup issues. Keeping the number of occurrences for both of those reduces the trips to the principal’s office.
As you scale Experimentation at Lowe’s?—?what kinds of challenges do you face and how have you overcome some of them?
That’s funny because that’s a real issue we are facing today. We’ve mastered our process, but now the demands require that we reduce our lead time to execution. This demand attacks my team's process and the abilities of our developers/engineers to produce in a fraction of a normal 2-week sprint cycle. They face a need for speed that requires that we juggle client-side vs server-side testing more often. The ability to design a test that can be conclusive and immediately ramp-up to production at 100% with no heavy loaded releases is our number one priority. Additionally, keeping our technology stack for A|B testing relevant and scalable is much harder for a company as big as Lowe’s.
I like to ask this question for leaders in Experimentation, how do you ensure that learnings get shared with everyone and how do you combine these learnings with other data sources like Analytics Insights and User Research?
User research becomes partners with us and product to reduce unnecessary tests. We have a library of pre and post analyses that we refer to when qualifying a potential test. This library is open and searchable to all internal Lowe’s employees. It is encouraged that they also look at these learnings prior to submitting a request. Additionally, we use the old experiments to broaden our sensitivity of elemental metrics that tell other stories happening on the same page. The repository of information must always stay up to date to maintain its relevance for our stakeholders.
“We have a library of pre and post analyses that we refer to when qualifying a potential test. This library is open and searchable to all internal Lowe’s employees.”
Finally, it’s time for the Lightning Round!
Why does the world trend towards increased entropy?
Humans are becoming more complex in knowledge while fighting our basic nature to be destructive. The world is increasing in disorder, by design.
That’s deep. LOL
Do you think they’ll ever define a unifying theory?
No. To define a unifying theory, you would be voiding either macro or quantum level theories. Or hypothetically yes, but the grand unifying theory will exist on a plane of existence we cannot comprehend nor reach.
I understood half of that. LOL?—?I clearly was not meant for Physics.
Bayesian or Frequentist?
Oh boy…..Bayesian. I just can’t help it.
What is the oddest item that you know of that Lowe’s sells?
Pants…..very expensive pants…
Oh, I have to search Lowe’s now. LOL
With that, Kenya, thank you for joining our Conversation.