Lessons Learned: Building a Culture of Experimentation with Ben Wise

Spyros Tsoukalas: Hello, everyone. This is another episode of the Growth Mentor Podcast. And I’m excited to have Ben Wise with us today. Ben is the head of programmatic media at Google Canada. And he also does a few other amazing stuff. So Ben, welcome.

Ben Wise: Thank you, Spyros, I’m really excited to be here. I’ve been on the Growth Mentor platform for about six months, I had a lot of really great conversations and I love what you’re doing with the forum, so very happy to be a part of it.

Spyros Tsoukalas: So, today we’re gonna discuss Building a Culture of Experimentation. So, could you tell us something we don’t know about experimentation as a mindset?

Ben Wise: Sure thing, and I would say actually that experimentation and people listening, you can’t see me using air quotes on experimentation. I don’t actually think of that so much as a mindset more experimentation is really a tool. And like any tool, it can be extremely valuable when you start to put it to good use. But the flip side of that is, it can also be a giant waste of your time if you’re not doing it well. I think people like the sound of experimentation or creating a culture of experimentation, again, in air quotes. But if you take a step back and think about why you’re actually doing it, or at least what I think the reason is that you should be doing it is that you’re trying to learn something so that you can operate your business better, whether it’s better results, or more efficiency, or whatever the benefit is. So really, the mindset that I think people need to think about is a culture of learning. Experimentation is just a tool to do that. I know that sounds a little bit nitpicky, on wording and semantics. But I really think that if we’re talking about the mindset, people are approaching, I think the words that you use and the way that you think about it do really matter. So, I would tell anybody listening, to get rid of the idea of having a culture of experimentation and think about experimentation as your tool to implement a culture of learning.

Spyros Tsoukalas: So, what value would someone pursue to this way of thinking?

Ben Wise: For sure, so at a very high level, I think the idea of a culture of learning is all about constant improvement. That’s kind of the 30,000-foot view. When you break that down, it can be all sorts of different benefits. So a few examples, you could say, will help drive more efficiency. So you can learn ways to get more done or get the same things done more cheaply, you can be more effective, you can learn ways to do certain tactics more effectively, that are going to bring you better results, whether that’s more sales, or you close your sales at a higher rate. It can help you prioritize and learn where you should be focusing based on the things that you’ve learned. It helps you learn more about your users so that you can personalize to them differently, or kind of more or more in a more customized way. You can increase your customer buy-in and have to actually do less selling. I’ve got some examples of what that could look like. But really, the idea of learning means you should just constantly be improving. And that should be true across almost any part of your business.

Spyros Tsoukalas: So, what do you see going wrong in the startup or the tech space around this topic? Because obviously, people or companies don’t use it as you mentioned earlier. That’s my guess.

Ben Wise: So I have strong opinions on this one, I would say that what I see going wrong is people approaching experimentation for experimentation’s sake, they know they should be running an experiment. And so they constantly have one or multiple going, but they actually lack a plan for how they’re going to learn from it and the types of changes they’re going to make based on the learnings that they get. So I can give an example from one of the startups that I’m working with as an advisor.  They often go to a client and say, let’s test this product. And that’s kind of how they sell it. But they positioned it as an experiment. And the other side says, Okay, well, we’re going to test it, or we’re going to experiment with you for free. We’ll try it for a month and see how it goes. But it’s so unclear what the experiment is trying to learn that the follow-up, or the action that you want the client to take doesn’t actually happen. So instead of saying, let’s test this together, you need to kind of actually define what the success of that test looks like. And if you hit that success, what are you actually going to do differently if we reach certain metrics, if we reach certain adoption, we commit back to you that we will roll this out more broadly to more users that we will transition from an experiment to being a paid customer or whatever that may be, but at the end of the day, you’re trying to learn enough that the client wants to proceed with your product. If you just say let’s run an experiment, then you’re going to have no guard rounds, you’re going to have no target. And you’re going to say at the end of it, and we’ve seen this happen, the end of the experiment, and say, Well, that was really interesting. I really liked it. What about this? And they start to find different things and give you too much product feedback to say, Well, can we add this? Or can we add that and then try it again, and we add these other features, then we’ll try it one more time? And before you know it, you’ve spent six months and endless hours and cycles, experimenting with this client without actually selling them the product?

Spyros Tsoukalas: So given the mind the tool experimentations, a tool, my mindset, that mindset, like? Well, that’s my word, and that you introduced us earlier, and the mistakes that tech companies are doing, what should they be doing instead to change this? Because understanding the first thoughts that you mentioned, it doesn’t mean that they would necessarily go in the right direction. So how can they start going in that direction from the current processes they are having?

Ben Wise: Yeah, good question. And thank you for the semantics. I know I’m picky. But saying the culture of learning, I think, as I said earlier, that is important. So, when you think about the culture of learning you can go back to academics or scientists, and it should start with a hypothesis. So, this is something that I think makes a difference. And I want to test that out to figure out what we can learn and how we can actually implement it better. So a very, very simple example. And you know, my day job at Google works in advertising. So we could say, I have a hypothesis, that on car ads, the color of the car makes a difference in how it resonates with users, and some might be better than others. Now, we’ve started with a hypothesis, what we want to test, and we know what we want to learn. Next, we need to say we know what the potential outcomes could be. So, if we are going to test three or four different colors in the ad, we know that the options are some colors perform better than others. So, let’s just say blue cars, seem to resonate in the ads way better. If we find that statistically significant, then what action are we going to take based on that? Are we just gonna move on to the next test, you need to actually think about what are the actions based on those results. So, if a color performs much better than the others, we are going to roll that out across all of our advertising. If another potential outcome could be that the color doesn’t make a difference, and if you learn that the color doesn’t make a difference, you say, okay, great, that’s very valuable learning. We do not need to spend more time and effort testing that, let’s move on after that and say, Okay, maybe the language of the offer, and the ad are what make more of a difference. And we can test that next. And you can continually learn to refine that. So, then based on what those potential outcomes are, you can really pre-define what those next steps are. I think this is really what the key is because the idea of learning and experimentation as a tool, is that you actually are changing your behavior so that you’re becoming better at what you’re doing. And pre-planning based on outcome A, this is what we’re going to do, outcome B leads to the biggest steps, and outcome C we’re going to move on to another experiment to learn something else. But in each case, you should have a definition in advance of running your experiment of what steps you’re actually going to take. So again, if I go back for that, oh, sorry. Go ahead.

Spyros Tsoukalas: No, go ahead. I’m sorry.

Ben Wise: I was gonna say. So if I go back to that example, earlier, of the startup that I’m working with, that has kind of been pitching their product as an experiment or as a test, if you say if we reach a certain level of user activity, or you know, if we reach X number of signups or active users within the product, then what does it look like in terms of next steps, and if you can plan that in advance, you’ll get to that milestone, and your clients will say. Well, you know, we hit that marker that we’ve said, we’ve hit that milestone. So now we are going to convert to being a paying customer or now we are going to roll that out to more of our user base. And you don’t even have to sell it to them at that point, because they’ve already agreed to it. If you get them agreeing based on outcomes, or when we hit that in the experiment, when we hit that user base, or user activity, or whatever that metric is, you’ve already pre-sold your victory, which is wonderful. You don’t have to keep spending the time and effort and influence your client to sell to them. So if you have that agreement this is what success looks like in the experiment. And these are the actions we’ll take when we hit that success metric. Then you’re golden from there, I think.

Spyros Tsoukalas: Could you summarize the steps that you described as part of this process?

Ben Wise: Sure, thanks. So it was a starting piece was your hypothesis. This is what we want to learn. Second, based on that, what are the potential outcomes that could happen whether it’s you know, more clicks, more sales, whatever it may be? What and then based on those content outcomes, what are the steps and differences in behavior that you will take based on each of those individual outcomes? So I think it’s really those three key pieces that start with the hypothesis of what you want to learn, think about what the possible outcomes are. And then once you’ve learned from it, what are the steps that you’re going to take? What are the actions that you’re going to take or do differently?

Spyros Tsoukalas: This reminds me of a few different frameworks here and there, like the Lean Startup Framework in the space of startups or other frameworks in other spaces with different names. So, does this process have a name? Or do you follow these other frameworks when dealing with experimenting?

Ben Wise: As you said, it probably sounds a lot like many other frameworks that are out there, or tools or whatever they may be. I don’t have a formal name for this. I think in a lot of ways. It’s a lot of common sense. But you know, the cheesy expression, common sense is not common practice. And I see clients and I see startups, and I see it in the market all the time where people are just constantly experimenting, and testing because they can. And if they step back and think about the purpose of it, they’re trying to learn something, and they’re trying to improve their business and improve their behaviors. You know, those three steps of hypothesis potential outcomes and the next steps based on those are pretty straightforward. So it should hopefully be easy for people to implement that. And I’m sure there are frameworks out there that will give fancy names to each of those steps that are better than I make it sound. But it really is pretty, pretty simple, and pretty intuitive.

Spyros Tsoukalas: Pretty simple indeed, hard to accept it though sometimes.

Ben Wise: That is true.

Spyros Tsoukalas: So Ben, like, are there any favorite tools? This is the last question. So please give us the fancy tools that you use.

Ben Wise: Honestly, the fancy tool is usually for me a spreadsheet. For the most part, I think you probably have the tools that you need, you don’t need to go out and get many additional tools, in some cases, for specific use cases, you know, if you want to test different landing pages, you know, there are some great tools out there from Google or otherwise. But for the most part, if you know what those potential outcomes are, and you have a way to measure them, then that’s all that you need. Taking the step back and enforcing that you’re going to make those changes in behavior, takes discipline and takes forethought, but does not take any fancy toolkit to do it. So if you have the technology or the tools in place, just to track your business results. Because again, what you should be learning about is how to improve your business results. So if you can track those that for the most part is all that you’re going to need. Again, simple, intuitive, and not as common as we’d like it to be.

Spyros Tsoukalas: Spreadsheets, and pen and paper. I will add these are my favorite ones.

Ben Wise: I mean, I have awful handwriting. So I use Google Docs. But yeah, pen and paper, Google Docs, sticky notes whatever you want. Don’t over-engineer the process. Keep it keep it simple. Yeah.

Spyros Tsoukalas: Ben, thanks a lot for joining us today and sharing with us and the listeners, all these insights. It’s the first time that somebody addresses experimentation from this angle. So, I’m very happy. I learned a lot today. And I hope people will learn as much as I did.

Ben Wise: Thank you very much. It’s been great to be part of this. I love what you’re doing with GrowthMentor. And, as you said, a lot of startups and early tech companies are really focused on finding product market fit and that is all about learning. So just make sure you’re approaching it with a culture of learning and clear ideas and a clear hypothesis with the next steps, and you’re off to the races. So thank you.

Spyros Tsoukalas: Thanks a lot.