How Josh Sookman drives product development through Experimentation Experiment Nation

A Conversion Conversation with Ritual’s Josh Sookman

There is a lot of literature, courses, and fanfare given to Experimentation around Marketing. But some of the most impactful times to run Experiments is when developing products. The costs of building products can be mind-boggling. Learning early can save everyone a lot of headaches and cash. Despite this, very few PMs embrace Experimentation as part of their workflow. Today I chat with Josh, a PM from Ritual who embodies this approach. Read on to learn about his career, how he leverages Experimentation to build new features, and how he determines if a PM has Experimentation-chops.

Rommil: Hey Josh—it’s great to chat with you again. It’s been a while! Let’s pretend we don’t know each other and you can start by telling us a bit about yourself and what you do.

Josh: Sure thing, Rommil. Let’s start broad—I’m a born and raised Torontonian, coffee lover (thanks to Ritual), avid health and fitness enthusiast and biohacker, and a passionate entrepreneur—I really love building companies.

Since January 2015, I’ve been focused on building the Ritual business alongside the founding team—first as a consultant, then as a contractor, and then as our first Head of Expansion tasked with figuring out how to replicate our hyperlocal marketplace across different neighbourhoods in Toronto, followed by the US.

Wearing my expansion hat, one of the biggest challenges we had to overcome in our early days was increasing consumer demand. Much of my time and energy went into building, launching and managing 3 core channels of user acquisition including our core business development strategy (who we partner with to drive demand, how we structure those deals, honing the pitch), retail user acquisition strategy (a program to onboard our restaurants customers as our customers), and company meal plans (lets employers buy lunches and dinners for their staff).

Eventually, all this time spent on product-oriented work led me to take on the growth product role at Ritual, focused on consumer demand growth (user demand generation).

Over the last 2+ years, I’ve had the opportunity to work on many products, features, and experiments—most notably driving improvements to our referral funnels (k-factor), and new customer acquisition from our restaurant partners.

I’ve read somewhere you have a Masters in Biotechnology? That’s impressive! Can you share about your journey from biotech to social-ordering tech?

Don’t all mobile product folks have a background in biotech? Seems like an obvious jump to me (*laughing*), but glad to share my journey…

While completing my Master’s program, I was enamoured by venture capital after learning that most biotech companies effectively cannot launch without first burning tens to hundreds of millions of dollars on R&D and (even more so) fund the expensive clinical trials that are required before a drug is legally sold/dispensed to patients.

I had the opportunity to take internships. And I relentlessly pursued a VC analyst gig that wasn’t posted—but I knew that this team had taken on interns in the past and I was determined to ensure they would do it again. Luckily, I landed the gig at BDC Venture Capital in their life sciences group and soaked up all I could about how VCs operated and invested.

My biggest takeaways were that (1) I still had much to learn, (2) Toronto wasn’t a blossoming biotech community from an R&D, startups and investment perspective; and (3) I should utilize some of my other skills rather than my genetics/biotech background for my next career move.

Taking my own advice, I leveraged my other background in software development (self-taught, side hustle) to land a VC analyst role at RBC Venture Partners and the BlackBerry Partners Fund (now Relay Ventures), and supported the investment partners on executing 10 deals largely focused on the mobile and fintech sectors.

After evaluating 500+ business plans, hundreds of startup founder pitches, making investments, and following the growth (and failures) of those companies—my entrepreneurial itch grew strong. I left and founded a company called Guardly in the safety and security space, which helped people in emergencies summon assistance quickly.

My Guardly journey was tremendous in its experiences and learnings—having built a team, raised VC financing, created a technology that could pinpoint the floor/room people are located inside buildings from their smartphone (useful in an emergency), granted a patent, and won some world-class customers. Then it was time for my next step.

One of my investors at Guardly also happened to be a seed investor in Ritual, and ultimately connected me with Ray at Ritual—initially, I joined as a consultant to help with some financial modelling of the business. After I met the founding team, used the product, and better understood the market opportunity ahead (and boy, did I ever!)—I decided to make the jump!

Wow, that’s quite the career journey! So Josh, time to change gears. I’d love to hear your take on how Experimentation differs between product and marketing?

That’s a pretty broad question. Here’s my take…

Marketing experimentation typically means attracting new users to your product (eg. user acquisition), or bringing your customers back to use your product again from a medium outside of your product. It involves experiments within and between channels and understanding the costs of acquiring customers (CAC), long term retention, engagement, and revenue generated from those customers over time. You’ll find that some channels will be better than others and go on to produce great users, while others produce bad users who do not become a profitable investment for a business. Double down on the good channels. Kill (or completely reinvent) the bad channels. Find new channels. Repeat.

Product experimentation refers to additions (new features), deletions or updates to your core app experience which may include onboarding, first use, subsequent use, drip campaigns, push notifications, and more. Different products will measure different KPIs, but when making product experiments, you’ll look to assess these KPIs for all users across every marketing source acquisition channel using comparable experiment and control groups (I typically use 2 control groups for product experiments to also measure the variance in control samples). Similarly here, we will often pick KPIs to assess retention, engagement, and revenue generated and see whether our experiment group outperforms.

“Our product team at Ritual has been fortunate—especially during the early years—to run many, many experiments. The learnings have been incredibly valuable.”

You’re a pretty connected guy—it feels like you know everyone in Toronto! Thinking about all the PMs you know, do you feel many PMs embrace Experimentation?

Come on Rommil, I bet you just say that to everyone!

I certainly know some PMs who do, and it’s clear who they are as they’re able to articulate the direct improvements that they’ve been able to drive for their products—and variants that they’ve tested that did not yield any impact.

Unfortunately, some businesses don’t have as much of an experimental culture as others, and PMs within those orgs may not have as much opportunity to do so. Our product team at Ritual has been fortunate—especially during the early years—to run many, many experiments. The learnings have been incredibly valuable.

Have you ever used Experimentation to gauge market demand? How so?

Yes, on the marketing side. A more recent example involved running ads for a product in advance of it being built to inform feature development and act as a form of pre-sales. In this case, it was Facebook Lead Gen ads connected to a Zapier automation that sent out emails with Calendly links, which resulted in a hands-off approach to generating a calendar full of “customer development” discovery calls to better understand the needs of potential customers in the market segment.

“A more recent example involved running ads for a product in advance of it being built to inform feature development and act as a form of pre-sales.”

I love that kind of testing. Scrappy and cost-efficient. Josh, like many PMs that work for start-ups, you’re pushing the envelope into uncharted territory. How do you leverage Experimentation to help guide decisions?

When I write a product requirements document, I’ll define a primary measure of success KPI, and also a number of countermeasure KPIs to ensure that they are not negatively impacted.

Ah yes. Guardrail metrics.

So for example, if we are making a product change where we hope to improve referral (virality), and we do so successfully, but in the process, we find that users are not ordering as frequently, we may need to pause and dig in—to understand why that happened, assess whether there is a correlation here or whether this drop was due to an external variable or other product change/experiment that may have had an overlapping sample group—before scaling that experiment to 100% of users.

Experimentation tends to attract a lot of conflict and heated conversations. As you test new features—how do you manage stakeholder expectations and manage potential risks?

Yes, it certainly can—depending on the experiment and how it’s rolled out. No matter what you do, it’s hard to escape social media when someone posts about a product change or a product experience they had, and others notice that their version of the product looks/works differently.

Social media aside, there are definitely things you can do to prevent these cases from happening (a lot). More commonly, conflicts arise over experiments (product differences) when 2+ people are talking about or showing the app to friends or coworkers—so one easy way to prevent this from happening would be to release your experiment in a certain geographical market (if you know the locations of your users), or another similar variable.

In your opinion, what’s the best way to identify PMs who embrace Experimentation?

Speak to them. Ask them about experiments they’ve run. What they were testing. What they were hoping to learn. What metric they wanted to move. What metrics they were worried about moving in the opposite direction. And why they needed to run experiments at all. Then see how deep they go.

“Speak to other PMs who you know have been running experiments. Ask them about their best practices and what mistakes to avoid.”

That’s great advice. Do you have any tips for new PMs just starting to explore Experimentation?

Speak to other PMs who you know have been running experiments. Ask them about their best practices and what mistakes to avoid. Read; there is tons of information to consume online. Write your next PRD with an experimental design section and a measurement of success.

Oh, PRDs. That’s a topic for another time LOL! Sorry to interrupt, go on.

Be sure to include the size of your experiment and control groups and how much data you will require to ensure your experiment is sufficiently powered so that you can have confidence in the results after you’ve run your test. And have fun!

Perfect. Finally, it’s time for the Lightning round!

Are you a Ritual point hoarder or redeemer?

I may or may not have gone through a hoarding phase; but since I’ve become a proud redeemer.

Can you give me any of your Ritual points? Just kidding. Unless you can of course haha!

OK, but seriously—Frequentist or Bayesian?

Bayesian gut; frequentist mind. Is that cheating?

“When you can’t disprove an idea with data, logic or reason, it’s a good idea to see if you can validate it with data.”

Always covering your bases, as usual, Josh. Do you get ideas from data or do you have ideas that you validate with data?

Both. I’m a bit of a data nerd. So I’ll often write queries to inspect what data is being generated by users, review raw data, conduct a visual pattern analyses, ask myself questions about what real-world events might be occurring to drive patterns that I see in the underlying data.

In some cases, this leads to the formation of hypotheses that can be confirmed by surveying or interviewing those users who demonstrated those patterns. If those user interactions (which generated those patterns) are beneficial to the product or company, we can then form product hypotheses around changes we can make to get more users to take those actions.

Ideas come from many places—customers, leadership, customer support, and once and while I have one to throw into the pot—and many should be discarded (or disproved with data). When you can’t disprove an idea with data, logic or reason, it’s a good idea to see if you can validate it with data. Unbiased data stemming from a good powered experimental design.

(Oops, I forgot this was a lightning round…)

LOL—it’s all good. Josh, buddy, thank you.

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.