On company updates. Please be honest. They're for you and not for us.
And if you make them clearly crazy, like, you know, we're never, ever launching or launching in four million years.
We'll get the hint. So don't do that.
There have been a lot of questions about the graduation requirement, which was that you had to submit nine out of 10. Weekly updates it this course was originally a 10 week course and things changed around a little bit. It's actually more of a nine week course now. And so and it was a little bit difficult for some groups to get going because of our little snafu in the beginning of the course.
So so here's the deal. Eight out of nine updates is just fine, but we're also going to extend the class. Oh, actually, really two weeks beyond the last lecture so that there will be plenty of time to do 10 updates if you so choose and you choose to do nine out of ten, but eight out of nine will be the absolute requirement for consideration for the for the ten thousand grant.
So we'll be flexible, but making good updates, make them real updates.
Another reminder, those weekly updates are due 11:59 p.m. every Sunday Pacific Standard Time. There's been some confusion, confusion about that. We've written a number of times, but please recall when that is a. Questions again about anything about. About your moderators, about your groups, about your updates. Please send to start up school at Y Combinator dot com. And my very last note is. A reminder to launch if you haven't already. The only way to find out if you've made something that people really want is to have people really use it.
So get your products out there if you haven't.
Now I'm going to introduce our first speaker, who is my esteemed partner at Y Combinator. Gust of ostrum or who? In 2012 was part of the group that built the growth team at A, R, B and B. And if you know anything about Air B and B, you know, from two top 2012 for the next five years, they grew an enormous amount. This is something that costof knows better or at least as good as anyone in the entire world.
So I'm thrilled to have him talk today about growth. Thank you very much. Before I begin. Thank you so much to Stephen. Jeff Adora, Bill, for putting all this together so we can be here and do this today. I'm going to talk about growth. My background before joining y'see, last summer was being one of the first person on the growth team at age BMB and then growing that team from I think there were three people in the beginning.
This is how many people were in 2015 with anger. Two hundred and hundred and twenty people or something like that. Almost everything I'm talking talking about today here. I owe this group the sort of what I've learned, all this stuff. Some of the best ways to learn to work on user growth is actually work on a growth theme. And I was fortunate to do that for almost five years. BMB. So. Before I start. One of the questions that might be obvious, the answer might be obvious for some people, but not for everyone is sort of like, why should we do this?
Why is growth important? And I'm surprised how often I actually get this question, but it might be worth thinking about it for a second. So when you make a product. Some people think that if you just put that product into the market, people will come and people start using the product. In my experience, that's not really how the work world works. If you start a startup or a company that's aimed to be a startup, growth is important because that's actually what a startup is.
If you make something that has the potential to be really big, then making that really big is sort of what starts all about. There's actually a post that Polygram wrote that is called Startup Equals Growth, where he talks about specifically why growth is so important for startups and why it's not important. Why not? Every company is a startup. Why? Growth isn't important. frederich company before startups. It's it's really, really important. And I'll talk in the second sort of like what it means to be intentional by growth.
But but the first question you kind of have to answer yes to is sort of like I have a startup I want to grow. Now, how did it go go about doing that? So who's this talk for? In my experience, those kind of things you work on when you work on user growth started with consumer companies. Does those with the companies that start embracing a lot of the tactics and strategies and technologies that growth teams we're doing now? I would argue that almost every company that sells anything online or get users online.
Should be or could be using a lot of the skills and knowledge Damir talk about today. So you don't need to be just a consumer company. You don't need to be just a social network to apply these things. These are applicable to a wide range of companies. If you are a B2B company and are excellent at growth, you have a massive advantage to other companies in your space because they are probably not going to spend as much time during growth as you are.
Now, there is one type of companies who this is relevant for, but too early, and that actually might apply to most of you right now. And this is why this talk is dangerous, because working on growth the wrong time in your company's history can be different. Like really, really bad for you. It's sort of like when you are kind of off to the races a little bit too ahead of time. So the most important thing, which is was a lot of lectures here start school is about is how you make something people want and how you actually find product market fit.
If you apply a lot of the things I'm going to talk about here to a company that hasn't build something people want or haven't actually found product market fit, really bad things happen. You will very often then this is a very common graph grow like crazy in the beginning. But then when you realize that sort of like that fuel that you had isn't fueling anything, it just fueled kind of apparatus. You come straight down. So that's why it's dangerous.
And now I can go into more details about that in a second of applying some of these things before you actually have product market fit. So the first thing I'm talk about today is going to be around measurement around prac market fit because that's actually the most essential part for for continuing growth. Now, do growth themes have impact or our growth theme? Sort of like like how do you know the growth themes actually matter? How do you know that that these things that I'm talk about as well took my experimentation?
But before I get there, I won't tell you a story from Facebook. And this is not a story. I came up with myself. I found it. One of the talks from Facebook and I think it's so essential for describing the level of impact you can have if you are applying these kind of skill sets to a company. So this is the time on Facebook from Twitter, 2014/2015 is sort of their growth story. Some of this public some some of this wasn't the one thing.
Facebook kind of had an excellent data science team from the early days. They were really good at measuring sort of like forecasting how big Facebook was going to be. So back in 2006, 2007, when Facebook had just started the growth team, they had a bunch of data scientists doing isn't this sort of forecasting, how big will Facebook eventually be? And they looked at all the type of metrics that fake Facebook have. And and they came to a conclusion that Facebook will be roughly 400 million users by 2015.
That was their forecast by using all the data they had. Now, we know that wasn't really true. They actually start growing really fast in 2008, 2009.
And this actually as a result of something that the growth team came up with, anyone here had an idea of what would make Facebook grow so fast in 2000, 2008, roughly using 70 tonight opening to high school students.
Yes, they did. But that wasn't as big of a growth newsfeed with Susan five. So a recession, recession does not know 10 friends on that. That's good. Datchet is sort of like the growth story all across a mushing could've been that translation's is the answer. So this is surprising for many.
For many of you might maybe. But there's actually a core part of many growth teams is that you want to make your product available for as many people in the world. And Facebook star sort of hitting the limit on people who spoke English. And they assume that people just because Facebook was in English. And so they're the content where your local language and Facebook where in English it would continue to scale. That wasn't the case. So they implement its new platform that automatically translated Facebook into hundreds of languages and the growth through accelerated.
Now the same thing start to happen again. I think they call this lockdown boom. They were doing very important things in 2010, 2011, something very big was happening. And Facebook had to make some really large changes to some extent, also driven by sort of like using data out to figure out these things in India. What what, um, what happened in 2010 was a really big acceleration. Mobile. I heard mobile. So. So actually what's happening is that the data scientist Facebook, we're forecasting Facebook to be about 700 million users, something like that.
And because most of those people are using Facebook on computers, they hadn't intended this massive shift that was happening in 2010, just Levitte, which is people getting us smartphones and Facebook switching the entire team. Now even have large training classes of engineers, just our learning mobile. So so this was the the big change to happen then. And then actually one more thing happen, sort of like in twenty, thirteen, twenty, fourteen where they made a forecast, they were kind of kind of not hitting the ceiling of growth again.
But then something again happened. Any idea what happened.
So 2013, 2014, Instagram and WhatsApp could be good dancers. That's not what I'm looking for here. Messenger is a good answer, is not what I'm looking for. So Facebook actually running into the ceiling of the Internet. No people actually went line. So they started this and called in a dot dot org, which was intended to get more people online. And they went to carriers and they gave basic work with carriers to get people in line to get free Facebook.
And this actually were very important forces of the continuous growth of Facebook.
Now, what can we learn from this? Well, we can learn that the initial forecast of Facebook growth was about 400 million people. But Facebook today is at two point one 2.2 billion user platform. So it's a very large platform and the forecasts were wrong. Now, the forecast wasn't wrong. It just didn't intend taking account. Everything great that the company was going to do. So if there's anything we can learn from this is that if you're intentional by growth and you're really trying to sort of break through these forecasts in these ceilings, you can grow really, really fast.
So this applies to every company. And I show this graph to everyone that John, M.B. and me like Aaron, they join the growth team at M.B. and try to get them to think in the same way. So natural adoption, which is sort of like that initial forecast of how fast it is going without any of that work is always going to eventually slow down. But if you do the things that you do as the growth team, you can't sort of like continue to push push the growth of your company.
Now, I mention this before having a growth teams before if I'm pregnant. Market food isn't really useful. So let's talk about product market fit. What is it? So the important term, but it's not hard to hard to exactly define what it is. So the one way that I think about measuring primer, if it is, is these two things. First, you find the metric. You can do this for yourself. After this talk, find the metric that represents the value that your customers get from your product.
And then you measure the repeat usage of that metric. It sounds pretty simple and we'll see if it works. You can make this kind of table if you want. We have the company. Yeah, the metric that represents sort of the value used to get from your product. And then you have the ideal frequency. So let's take take aim in B.
So. You get value from A and B when you are booking and staying. So when you when you're actually traveling and B, that's when you find that M.B. is really about it valuable. It's an amazing macing appearance. Unfortunately, people don't do that more than annually. So if you want to figure out the park market fit of. You can always ask people after the first day, but you wouldn't really know until they book again. When I come back and book again.
So because the booking cycles for travel is so slow and it'll be a good metric here. Let's take Facebook. So Facebook, if you come back to Facebook being and voluntarily come back to Facebook as an active user, you're doing that because you find value. There's something that makes you come back to Facebook or Instagram or anything like that. And the question then is how often should I come back? Well, probably a daily or hourly or something like that.
They start this. They started looking monthly and then they went down to daily. I think that might be even more than that. Let's talk about gusto. What's the product value as a customer of gusto that you get out of using gusto? Well, when you run your payroll, which is sort of Augusta do for your employees. That's one of the one of the best ways to sort of measure the value to get out of gusto and how often you do that.
Well, you run paleo every other week or every month. I'm not gonna go into details of all of this, but for Lyft, it might be rides for check or it might be background checks for stripey, might be transactions. Now, for your company, you should figure out what is the one metric that I can measure easily everyday or every time it happens. And what's the ideal frequency through which that should happen? If you can answer those two questions, you can make a cohort analysis of your own company right now and you can start trying to figure out if you're parked mcgiffert.
So then you want to measure these things. So you take a graph on one axis, you have time and the other one you have the metric that we just decided. And from that, you can try to figure out if you have high point market fit. So this company is measuring this on a weekly basis. The ideal use case of this company is using this proc on a weekly basis. So every week here there's a dot, which is the percent of people they use this product on a week zero the first time they use it.
They used it again. So let's take the startup schooling's and sample. So I joined startup school and in week one, only 60 percent would come back to start school. And then in week four, only 30 percent will come back. Well, that would suggest this article wouldn't be a very good product, but that would be a good way to measure if starting school is actually a good product, people come back to all the content they were creating.
Let's take another product. This is a great product. This is kind of like a normal product. An oil pipe will look like because almost always you have little bit too many people in the beginning of the product. And then it will go down over time, but it will sort of flatten out where you keep measuring these these events of your metrics. This might sound technical, but it's actually just a representation of how many people each done is. How many people does using your product for that metric that you decided on that on that time window?
Most good products flatten out. Let's look at some examples here. So these are examples that based on payment retention. So this is a company that you have all heard of that retains 10 percent after one month and then 12 percent after twelve months. It's sort of like unusual. Who can that be? Stripe, as is your heart. This is Shopify. So shop if I have this product where a lot of people sign up. Not that many people continue in Montu.
But then they kind of stick on forever. So what do we know from from from this? Well, if if this curve would go down to zero or so to keep going down, then shop at five would be a bad product. It wouldn't be something that actually found pragma fed and should've been working on growth in this case. They have good, good product market fit and they should be working growth. They should do really all they can to contain work and growth.
They should do something about sort of like the initial onboarding here where they lose a lot of people in the beginning, but they should be working growth.
Here's another company. 50 percent retention at the one month and then 10 percent after 24 months. So a similar to Strive belike or similar to Shopify, but they continue to lose users all the way down to even 24 months. Maybe there's some flattening out at the very end here. But even twelve months after people start paying for this product, there's that every month fewer and fewer people pay for the product. Is this product McGiffert? Hard to say, but but is not an obvious case that they have found product market fit.
This is blue apron. Here's one that's pretty good. So 70 percent retention after 12 months and then 30 percent after seven years. This company definitely have product and product market fit. Any ideas? So which companies might be? You're all using it. Amazon. Amazon wasn't the right one. Apple no. This is Netflix. So 70 percent of people that start paying for Netflix pay for Netflix seven years down the line. That's definitely a company with private market fit to just spend every time everything they can working and growth.
So raise your hand if you're if you're measuring your attention right now. Not that many people. Well, there are other ways to figure out attention. If you're really small. You should go out and talk to users. You should ask them questions such as how would you feel if you can no longer use my product and should sort of like, say, as close as you can't be users. This is like measuring retention like this. It is very hard when you have 10 users, when you have a thousand.
It's easier. But when you have 10 users is very this is not the way to do it. You can actually go and just talk to any person. So there are other ways to measure it. But you sort of have to know that you have a product that's retaining. Otherwise you shouldn't be working on growth because you're going to end up burning cycle, something that doesn't matter. OK. So a lot of people might be wondering how does growth and marketing relate to each other, isn't it?
Is it the same thing? Or how should I think about it? Historically, the way you had a company 20 years ago is you had a product team that made the product and then you had the marketing team or product marketing team, the marketed the product. That's how things used to work. And a lot of the sort of like hierarchies and companies or sort of organizations are still based on this idea. They have a product team to separate from the marketing team.
And these are different teams and different skill sets. And an engineers work over in the product team and then the marketing people working with the marketing team. Actually, it's not how things work anymore.
So. The way you think about this is that I'm going to come to business in this and second is that there is three times three different types of people that can like organizations that can drive growth. There is what I call a pro-growth Grug growth engineering. This is different names. This is effectively product managers, engineers, data scientists and sometimes marketers that work on growth. But they work on growth using technology. So they're actually changing the product to drive more growth.
And much of this work here is about people that already arrived your product, but haven't really found the value of your product yet. And they're changing the product to make it grow faster. Conversion optimizations fall into this. This group, some of the growth challenge actually fall into this group. Now there's this big other group here, performance marketing, which is effectively Google and Facebook marketing, which is also super technical and super data driven. I would argue that these things are very, very, very similar.
So five or 10 years ago, you would go to a company, you would see these being very different groups, maybe different different floors in the office. Now, they, in my opinion, should sit together. So engineers should sit together with performance marketers and vice versa because they actually are doing very similar things. Now, there's this fifth button, fifth thing should be number three here, which is brand marketing. And brand marketing is sort of like the hardest to measure the the hardest to measure a type of marketing where you're not really having a direct response.
You're not having someone directly giving you feedback on how that marketing proceed if you can. It can't even measure that very easily. So this is not something the staff should be doing in the early days. My opinion staff should not be doing more brand marketing until way down the line when sort of like they hit the limits on these two things. So start ups should be doing these two things. And there are some qualifications, too, which you can do this.
But almost everyone can do this. If you're a startup, you have engineers, you can do product growth or growth, engineering or or or things like that. All right.
Let's talk about that first part, pro-growth. So your product is a funnel. What does that mean? Your product has many, many, many steps between the first user and the person, sort of like completing what they're trying to do on your product. Let's say I'm an e-commerce store and I'm trying to buy something. There are many, many steps in that funnel. And what the pro-growth team is working on is fall-off my station, a conversion without my station.
So if you look at what we do to be on the growth team, many of those growth teams work with conversion rate rate optimization teams. They're working in a specific part of the funnel. So the funnel could start with MCO. It can start with performance marketing. It could start with a referals of morality, but it would jump through a number of different steps, say sign up would be a common step or if you're an e-commerce site, payment conversion or or buying conversion.
All of these steps are things that as a conversion rate, optimizing part of the growth team should be working on. And these are some of the easiest things to get working on.
And I'm gonna be a little bit high level here because to go into depth about every single area here is gonna be. We to go on for hours. So some of the like, good I.D.s of areas just are working on for conversion realizations. One of them is translation. So if you have any national product that's not translated, you should be thinking about that. That merely drives up and drives more people to start using a product. The second thing is authentication.
So most of your products probably have some idea of user accounts so you can sign up to your product. And you can come back and log into your product. You'd be surprised how much how hard this is. And for me, I mean B I know we spent many years working on just authentication, signing up, logging in. It sounds so simple, but it turns out that's a very fragile moment of users sort of like flowing through your product that you can always continue to make optimizations.
If you go to A and B and you go to Pinterest today and assume that whatever is there in terms of authentication is the most opt optimized version, that these companies spend an enormous amount of time optimizing SINEM conversion. Another big area for commission itemization is onboarding. So when I come to your product, what's the first thing I experience and sort of like how do I what what can I do to bring me towards the value of the park as quickly as possible?
Those are things that you work on when you work on an onboarding. And then another big area here is purchase conversion. So like when I'm about to buy something, a product actually can take the final decision around what you do. There's so many things you can do here. All right. So then there is something called growth channels. So what is it, growth chance? Her challenge is sort of how people discover your product. Now, there is when you're a small company and I don't know how big you guys are, but let's say you have less than 50 users.
You shouldn't really be thinking Magro channels. Even have you less than five users, you probably shouldn't be thinking about this because it's too early. But the things that you do when you're small that don't scale, have the word don't scale in it for a reason, they don't actually scale. And there are very few companies who kind of growbag without growing on one of these scalable growth channels. They're not that many platforms and channels slash platform can be used in the same way.
They're not that many things. They're really, really big in the world on which you can build a large company. So let's talk about what those those channels are.
So the first thing here is basically you think through the behavior of of your product. And let's talk about the first one here, which is if the way you discover your product is a rare behavior where people use Google to find a solution, and this is actually how very a lot of products in the world are being discovered. If you have something through which to answer the question around what you do once a year, you go to Google. Probably if you're building a company trying to answer that question, you should be.
You should be on Google. So good example here. Might be buying a house. You aren't buying a house more than once or twice in your entire life. Which means when you go and buy a house, you're probably going to go to Google, which is not surprisingly, that's something like Zillow or Redfin or all of those different sites that allow you to buy homes online are entirely optimized for SEO and sometimes paid search. If you are using something every day, you're not going to go back to Google everyday.
You gonna go straight to that product. Can open the app on your phone. You can go to the straight to the Web site, whatever it might be. You're not going to go to Google anymore. Just kind of figure out which of the different things I'm going to use. Another goal is not the only searching, and there are other search engines are like that, you can optimize force while the Google Google is the one that still matters. Next behavior to people of my product already share the price.
Using word of mouth. So. If that's the case, then there are lots of things you can do around morality and referrals so you can grow your product by kind of accelerating that behavior from existing users. And you can either incentivize them with referrals we were you getting paid or you can do it for free and with with with using morality techniques. Thus, having more users on my product actually improve my experience. So what I mean by that? Well, if I am building the next link, then it makes sense.
The product is not as valuable when it's just all of us Symmington. But when it's a bunch of more people and companies on LinkedIn, it makes sense that the value increases. Well, in that sense, you should absolutely continue to do virality because everything on the user. Is there an opportunity to bring in more users? So here's another way to think about morality. And then this is that common question I ask companies in sort of the early days away, see, can you make a list?
Literally a spreadsheet of all the people in the world that would use my products. Let's say I sell to buyers who decides to. Doctors offices sell to doctors offices. Well, I can probably make a list of all the doctors offices in the United States or in California. It wouldn't be that hard. It's totally possible. So I would find a way to make a list and then would go on to sales. And this is surprisingly something often something that you should start with.
If you can make the list, if you know those people are, you should go on to sales. And the last one here is, does my users have high LTV? Does music does it does. Do I charge enough for my product for it to be valuable? Well, then I should definitely go and do paid acquisition. For example, Google and Facebook. I shouldn't do acquisitions or paid ads unless actually I'm trying for my product. All right.
I'm going to go through each one of these sort of like it's not going be possible to go into super deep detail for all of them. But I'm going to go into Libya deeper into some of them and not the others. All right. Let's talk about referrals. So I worked on a referral program at M.B. for a very long time.
The way to think about referrals is sort of that engineered word of mouth. So if if people are already talking about your product referrals is is a way through which you can engineer more people talking about your product. One way could be just making it easier. Another way could be by using financial incentives. In the M.B. referral program, we had a financial incentive where as a refer, I would make $10 for every users. I would sign up in travel credit.
And there's a free. I would I would get $40 as a new user of M.B. in travel credit. And we would we would kind of like start with that principle and then try to get as many people over, call through our friends, fans as possible. So you meant you notice I used the word funnel here again. So every product is a funnel and even there a full product, which is a sort of a product of his self. I mean, B had its own funnel.
So I'm not going to go through all of these details. But the way you think about something that's kind of engineering and product lead is that you break it down into into different steps and then you measure every single one of these steps and then you kind of measure the conversion rate to the very end. It's the first up here. Here is weekly active users that saw their full program. So how many users are there for a program? If I wasn't measuring this app, I wouldn't know how many people did that when we started measuring this step.
And I've seen this with many other companies when it who have a referral program. Is that a small percentage of your active users see their full program? Well, how could you be expected to use it if you can't see it? So we started mentioning this in the early days of A.P.B. It turns out that there's a lot of opportunity. I'm just telling people about you having your full program, but then there's all these opportunities to justify through that funnel.
So there is sort of people sending invites how many they sent to the conversion rate, to new users, to new guests, and finally to them booking their first timeit's. Here's another display. This line is going to be available online afterwards. You don't have to take notes or photos. We kind of separate the funnel into even more detail and we will continue to optimize this for years. Like this all matters a lot and we would continue to optimize for years.
I'm going to go through what one example of sort of like one of those conversion rate optimizations where did for their for program. So here's the referral invite email. So if if someone else if I invited someone, one of you guys, you'll get one of these e-mails. And the emails say Gustaf Ostomy Meitiv JVB Good sent you $40 on your first trip on air. BMB You can book rooms long. Just sign up at twenty fifth or May twenty eighteen.
And then there's a button to call, accept the invitation. And then there's a photo of me and my name. Looks pretty good. Now this email is a result of dozens of experiments. Nothing here is random. Everything here is for a reason. Let's talk about that. The first one is the subway line, the subway line has my name. If it's sent to any of my friends that make them more likely to open it. The sort of like headline other email have a clear value.
It's a very, very clear what this amounts about. You'd be surprised how many emails that don't have a clear headline and declare values are what the e-mail was about. These e-mails about that. I sent you $40 on your first trip on this Web site called SBB.
This, which is signed up by May 25th, 2018, is urgency. You should sign up by it by May 25th, 2018. That's the urgency that makes you actually go. That increases the chance of people to see this might actually go and do it. The name here, except imitation, it's the sense of exclusivity. It's not a doesn't say sign up for reason because sign up anyone can sign up by accepting an invitation sends an idea of X passivity.
And finally, it has my name where I live and how long I've been a user. May I be with my photo? That's really strong social proof that I endorse this website. I endorse this product. So the way you think about this is sort of hate to think about all the commercial randomisation. It's a set of optimizations. Pay growth. This is one year I'm not going to go too deep, too deep into the the options here. The things that matter a pay growth is that you shouldn't do it unless you have revenue.
Too many companies are buying ads when they don't have any revenue or know how they're gonna make any revenue. That's a big mistake. You shouldn't be doing that. If you have revenue, there's a couple of concepts that really matter. The first one, how much money am I paying for each new uses that I'm acquiring? It's called customer acquisition cost o'cIock. What's the lifetime value or payback time of the users that I've acquired through pay growth? So what that means is what's the longest that I can with some level of accuracy forecast how much these users will be worth.
If a user is paying $10 per month and then we can sort of have this cohre analysis that we had the beginning with the retention, we can figure out exactly how much our users gonna be worth. And if my customer acquisition costs is lower than my lifetime value, then that's good. That means you have a great payback time if you only pay $20 for a user. That means if you make $10 per month, you have probably around $20. Two months payback time.
That's more complicated in that list. So like its simplified version, these are the most important concepts to keep in mind for four pay growth attribution. So this matters if things get complicated. If using both Google and Facebook, if you're getting to some some level of scale each understand what it means to attribute a new paid user to a dollar that you spent. I'm not going to go too much into detail what this means, but but this is something that you should learn if you're thayne time on pay growth.
And finally, in my opinion, that only four channels this sort of matters that scale Facebook, Instagram, Google and YouTube, these are the very, very large pay growth channels through which many companies are actually built on these days. Sort of like one of the sort of unspoken truth, in my opinion, is that a lot of the free channels of growth is going down and the Internet and the pay channels are going up. Good example of that is that this hard to go on the Facebook newsfeed unless you paying Facebook money.
Same is true for Instagram. It's sort of changing always when there's a new platform. They often kind of promote free usage in beginning and then they start to my tistadt. And that's what was happening for both Facebook and Google and Instagram today. All right. Search engine optimization. So search engine optimization or. Some people might say, well, this is something of the past in my experience. It is not. It still matters because as long as we go back to Google to make our decisions about what we want to do in the future, this matters.
This is not what's gonna be delivered back to Google. Google is going to see a bunch of lines of text through which you a marked up in your code. And the Google have hopefully indexed if you've done the right thing. And if you haven't, then it's kind of like your fault. So there are a bunch of basics too that you can do with S. You just have to get right. The easier thing you can do is run your Web site through an SCA browser and try to figure out if you only saw this, would you understand what your practice about?
There are things that you can change on your Web site, and there are other people in the world linking to your Web sites.
This is the two main levers that you have in SVO. For the on page optimization. The right way to start is not to start with like a list. Make some changes, the right ways to start with the strategy. What am I trying to rank for? What keywords exist on Google today that I might want to be the number one result for now? To do that, you have to do what's called a keyword analysis or keyword research to try to figure out what are the things that people are searching for that might relate to my product.
And then hopefully they have some amount of large volume that they're searching for. And then I can try to say, well, these keywords to have sort of a medium to high volume and they're not that much competition around these keywords. Those are the ones I want to rank for. When you decide that that the other areas have on patients, messages now are easier because now, you know, we tried to to rank form at scale. But I've seen this as small scale to n.c.o.
at this point is about MCO experimentation, which means you can do the set of of of a of best practices in a. It might take you to be a decent sized company, but if you want to become a massive company, a really, really large company that's going through SVO, you have to use experimentation to make those decisions. The other thing, the matter here is off page optimization, who's linking to you? So there are lots of tools you can use to figure out who are all the Web sites online.
The links to you, what's their sort of domain authority? Are they actually authoritative on the Web to have something? There's a huge difference. If I will link to your Web site or New York Times thing to your Web site. It matters a lot in sort of how in Google's perception of you, of your of your site. So. One surprising way to M.B. is that we had a lot of press, a lot of people in the world for media would come and write about Obambi, and that was surprisingly important for our off page optimisations.
A few people writing my you either impress or in other areas. That's actually a great thing if you don't have anyone writing about you. Well, you're not going to have them any links because the web has changed. Aren't that many links anymore. It's not like everyone have a web site these days. They are linking to each other. So they said things have changed and had to be strategic about who are you getting links from? Easiest way you can do is whenever you get written out up by anyone.
Press or or whatever, just ask them to link to you. It matters to you. Last thing we're talking to here is on growth teams. So a growth team today is typically engineers, data scientists, designers, product managers and user researchers. These are not the all the people you might have in your company today, but these aren't company companies that you will have when you start making decisions around growth.
The way you organize the growth team is there's sort of two options. You either have the growth team as his own team and the wrestling practice is the product team. And that's actually sort of my the the challenge here is that when you when your small company and you're not kind of like a somewhat ignorant about growth, what you eat often say is like, I've hired a growth engineer, I hired a growth park manager. That person is going to be doing all the growth in the company.
That's sort of not really a recipe for success. So there's kind of a fine balance here between saying everyone is responsible for growth, which doesn't work and having a small team. They are responsible for all the growth. You kind of have to find a balance here and the right way to find that balance. It does set very clear goals and goals and very clear dividing lines in your product. So a good example here might be everyone that works on the core product, which is sort of the the value of your product.
Let's say I am gusto that the. Running payroll for employees is the core product. Now everyone trying to get to that experience. That's sort of like the growth area of gusto. So be one easy way to kind of make a distinction between growth and product.
So how do you decide what to work on? You make a simple calculation of what's the effort? What's the minimal bible. thing I want to try. Try here and sort of how big of an outcome can that be? So I always try to forecast out the outcomes before you actually start doing the work. Because to be forecasting something and a best case scenario, it's small. They shouldn't be doing it even though it's a low effort.
All right, two small short sections left of this talk. The first one is called Making Decisions. If you ask any proc manager, M.B., today, what is the most important tool or learning that you learned? Evan B you'll apply to your next thing they would say experiment or experiment framework or some way of making a/b tests inside the company. And I'm so happy that Soheil is talking later because I stole a quote from his investor deck that I found online.
This is the quote Most of the world will make decisions by either guessing or using the gut. They will either be lucky or wrong. If you keep making decisions without using data on experimentation, you will be lucky or wrong. And this is a huge problem. So. If you if you kind of get to a scale of R&B or something even smaller than that, then every decision you gonna make that you don't use a/b testing for, you won't actually know what the full true outcome of that decision might be.
So at every M.B., we use experimentation and a/b testing for every single major decision and then type and then type company. So this is how that tool worked. And B my talks are sort of like more about how they look like.
Let's say in your company you decide to ship a feature and this is how many kind of measurements I'm met that metric per day and the Wednesday here's my ship, then a feature. And then I look at it two weeks later and looks like that the metric of that feature went up. So it's a good thing, right? Well, it's not that easy because there's so many more different factors. Who knows what'll happen between here? If you're a soccer app and the World Cup, you started all that and you wouldn't know if the features have made a difference.
If this was just peak season of whatever you're doing, this is a school app, education app, and this is September and you wouldn't know either. So the only way you would know is what what's called a counterfactual. Basically, maybe test by running two different versions of the same feature, the same site, the same time, you would be able to know what the true difference between making decisions and not making it was.
This might sound a bit technical, but it's very, very important to internalize because you will get to a point where you were successful up until that point and you think that you're so good at making these decisions and then they get harder to make. And you have to have a framework to make these decisions. So will we. And then because this was so important that, B, we built something we call experiment, experiment, review, experiment review is one the whole growth team would meet in a room like this, would go through all the features that we had recently built.
And before we told you, which I wish of the different features in the experiment, the actual one, we would ask the audience. So I'm going do that with you guys here.
So here is a photo experiment of you. We would do this every two or three weeks. It's really fun. But it drove home this one thing, which is that making practitioners are really hard. All right, let's get started. So the goal of this product was to increase the number of shares from the Evan B mobile app, specifically the number of shares of listings and Bakhit at that time. We had two options. We had the native shochu that you'll know about.
And then we have this experiment, chargesheet, which takes up entire screen, has more colors. Same type of buttons, sort of. You can see the difference here. Now, before I show the answer, how many here think that the native share sheet led to more shares? Raise your hand. Half the room. How many here thought that the new sheriff sheet that one of our engineers built for more ships? Raise your hand. So most, many hands do not race.
I'm assuming you guys think there was no difference because this is so hard, you have to make a decision. You have to make a decision, you have to have an opinion if you don't have an opinion here. You're basically saying, I can't decide. So. This shit led to 40 percent more shares, so very important in this case that we'd use experimentation because we could just gone by a gut, we would've been wrong half of the room, more than half of the room would've been wrong.
Still, one more. So we sent out this email to existing users of Evan B to to sort of when I would make a booking at Bamby, we set up this email to the people that I listed as my co travelers. And the email will look in the control like this where it would have the itinerary of the address to that listing and the button on that email would say, join gustafer strip.
Now, we tried a new version where the e-mails but little bit different, less content. And then we have different Budden here called except Gustaf Strip imitation. How many here? The goal here was to get sign up. So people have to click on this button and then sign up. How many here think that John Gustav's trip and the control led to more sign ups?
Raise your hand. Some people, many here think that accept goosestep strip imitation is your hand. It's good. How many here think that there wasn't a difference? So this is a 14 percent increase of just change in the basically name of the button. Let's try one more for simplicity.
So in this case, another Shank's. I'm kind of giving you the very easy to understand ones.
So the control here was sort of like a bunch of sharing options on the M.B. listing. We had some Twitter icons and Facebook icons, demon icons. We had another version of that which icons were around. And then we have one that was called sort of square buttons and it was just displaying e-mail and Facebook. And then you can click on more. How many here thought if the goal was sharing that the control with the icons? 1. Raise your hand.
Some people may have thought that the round buttons were better raise your hand. Some people even thought the square buttons. Well, you guys are really good. How many thought there was no difference? Well, both of these were winners. This is by far the biggest winner. And these are the kind of things that we debated. And because we run experiments, you don't have to debate anymore. So practices are really hard at scale. You want to use experimentation, a/b testing to make these these decisions because otherwise it will be the loudest room, loudest voice in the room.
They'll make decisions that you don't want that. To summarize sort of my talk. If you're running a startup, you should be thinking about growth before you start working on growth tactics. You should be measuring your attention and knowing if people are using repeatedly using your product. You should pick a metric and then pick a goal for that metric and drive that metric toward that goal. Very simple. And then eventually. But not right away. You should start running a/b tests for the decisions that are hard to make.
In the early days, decisions are not that hard to make as they might be obvious. But the moment that not easy anymore, you should be. Doing this with experimentation. Thank you.
I think we have time for a couple of questions. Yes, experimentation. Or me or Theo. Do you have any framework for a way to think about trying to be honest beyond randomly?
So the question was around experimentation, Nasedo. How do you go about doing that? So what you're trying to test. We run it. And on page experiment, what are you trying to test? Is sort of the amount of organic traffic you get from Google and the sort of how your ranking shifts. So Google actually are changing their rankings all the time. So if you make a big change on your website, on, say, say half, let's say let's call you have for Amy's case listing pages.
Sorry. Search it searches all the pages. So Stockholm, London san, just go buy one group. And then you have New York, Paris and Barcelona. Another group, you would change the content on the first group pages, but not the others within. Not too long of a time you would see more or less traffic going to either of these groups from Google. Sometimes there's no difference. But if there is a difference, you'll see more traffic.
Give you an idea of every small change you can make. Let's change. Let's let's, for example, change the title tag from Evan B listings to the 20 top best listings in Barcelona. One of them is going to rank better because any more you're invited to click on. And then you run that experiment towards the search engine. And then the Trajtenberg will kind of tell you which one is better based on the of traffic and the search engine that you get.
Yes, they were totally blind. Yes.
The question was run a/b testing. How do I determine if it's right for me when I'm really early? So a/b testing is a function of change, so you can have a medium to small sized audience. If the change is large enough, it actually can be as significant like the numbers can be significant. Now what you can do is you go do can go to Google and typing a/b testing calculator and there's a kind of a form a pop up and you can type in sort of the the the metric that you have.
And then the change and you can see how big the change have to be for you. Just babble to see a difference. Now, if you're really, really small, you probably shouldn't be doing maybe testing at all. So. I would do a/b testing when you have large like enough traffic that you can see a medium to small sized change within, say, two or three weeks. So it's a small change would be a couple of percent. If you can see that about CalWORKs.
Kind of like that sort of the level you could be on. This probably sort of might. My general recommendation. I've certainly seen a lot of companies and their early embracing a/b testing relatively early and then can the guide sort of that their decision making. And I don't think that's a bad idea because I think that's much it's better to start a little bit too early, then you start way too late. If those are the only two options, of course you start the right time.
Yep. How do you apply growth to high barrier entry market like like health insurance? So I think you should separate sort of the acceleration of your growth in that market and the market itself. So if it's hard to reach people, that means you have to probably try to reach to reach a lot more people before you actually get sort of.
If if a high to me, what we mean by by sort of like a high barrier is regulation. Yes.
Yes. If you have a risk and regulation involved, I don't think that the principle of growth changes. But if you're purely growing through sales, for example, you can apply a lot of automation and technology to how to do sales. So if I'm a company and I'm selling to insurance buyers at startups, I might be starting by e-mailing 10 people. And in fact, I can email 100 people or a thousand people with the same type of level of.
The email itself is kind of a field esp as personal and asked direct to that to that receiver as if I send one email that is about myself. So that's certainly what kind of things you can apply to your growth that allows you to reach a lot more people. Now, the growth doesn't solve any of the risk challenges at all. It doesn't solve any kind of major market issues. And M.B. certainly had a lot of issues with wood legal like legal challenges in different markets and sort of growth.
Was this disconnect from that? That wasn't sort of ah ah, our goal. If you're a startup, you have those those problems. I don't think you should solve them. Solving them and solving growth are two different things and they're very separate from each other. And and growth is a way to accelerate you getting to more health insurance buyers. It doesn't actually say anything whether that's that's a good thing or a bad thing. Yeah.
Do I have any wisdom of using non-sustainable tactics in new market like Uber? So if you're doing things that don't scale, you sort of have two options. One is they don't scale, so you just stop doing them eventually or you build sort of a playbook where you take those skills and you kind of try them in a different city. Now, they might not eventually scale. So let's say I am out manually to recruit Uber. Drivers know that it might be an unsustainable growth tactic.
It might be working for the first 20 drivers, but eventually is not going to have high R.I. in comparison to, say, running ads on Facebook to recruit drivers. So. I think you kind of have to determine this manual thing that I'm doing in the very beginning, which is doing things that don't scale, which sort of recruiting every single user to become a user product eventually won't work anymore. And you kind of have to find another channel or you won't be.
Hi. Are y- in comparison to other channels and you'll most companies who go through this transition period where you go from, say, writing content manually to. Through engineering, changing a website so that we get more search traffic, for example.
And what about doing subsidies or incentives for rides? Well, I don't. I actually think the incentives are super scalable. Like everybody. We still have their full program. They're signing up hundreds of thousands of users every day, almost which and large portion of them are signing up through through referrals. So that is the example with subsidies are actually scalable now. Me handing you a coupon in your hand is not scalable, but doing that through an e-mail system or through WhatsApp or through Messenger or some other channel is actually scalable.
As long as you're not losing losing the money. So you have to have a very good hour calculation on the money that you're handing out and knowing that that's incremental money that if you didn't get get that money out, those users wouldn't actually start using your product. So as long as you have a good handle on that, that is totally fine. Yes, yes, yes, yes, yes, yes, yes, yes, yes.
So the question was around here. Two groups of users. So the users and the paying users and some of them are very active and some of them are not not as active, but at least no one to pay. How would you use growth there? So one of the things that we're trying to figure out is what is the retention rate of those paying users? Are they actually sticking around and using the product that they're paying for? And what's the retention rate for the fee users?
Do that change once they start paying? And then we'll look at the conversions are like how what percent of the fees is am I covering to paid? And there's many different ways to do that. There's one which is having the freemium concept. You can also have a free trial, which means you actually don't have freemium. You can just have a very limited set of free users and then they kind of have to go to paid. But ultimately, to the extent that you should be growing, all of this comes down to how much usage you have from these that the one you want the users that you value.
So if the paid users are the one you value the most. If if they stop using a proxy after three months, then you have a more fundamental problem, the conversion between these two groups. So I would still go back to retention and look at usage for those people and see are they actually sticking around and using the product? Last question from over here, yes, in the back. So the question is, and sometimes experimentation can be really hard to execute.
Is there an ideal sort of frequency or which which experience should I be actually be running for? Something like a marketplace? Experimentation is really hard. That is correct. So. In that case, it's easier, it's harder than just setting up a simple tool online or using mix panel to set up your simple a/b test. It's more difficult than that. If you are, it's kind of having a simple product funnel. It's generally not that hard to set up experimentation.
If you have an engineer and if you have decent traffic, it's not that hard to set it up. Now there's different kind of products that make experimentation harder. I think that if you're at this ad at the stage where you have a lot of traffic, you should in my opinion, you don't have really an option. You should have to invest in some level of infrastructure to use data to make decisions if you don't do that. The alternative cost is you end up making a lot of bad decision where they either long or they're lucky.
You're lucky because they're right or you're wrong. So you don't really have an option, but you run experiments. And it is true that there are some type of products through which expectation is easy. If you're running your mobile app, expectation is not that difficult. There are lots of tools you can use to run experimentations. If you're running out of ideas for experiments, well, you go back to user research and you should look at other products that are sort of like us, something like Pinterest or A.P.B.
Or Facebook. They're probably highly optimized. So a lot of the things in there funnels is there for very specific reasons because of a/b testing. So. I don't. I don't have any better answer than if you don't do it, you'll have a lot of other problems. And in terms of the frequency. There is a cost to setting up an experiment and that cost should be minimized as much as possible. But it's sort of like you should just try to get better at it and and pick the easiest, smallest thing that you can test and just test that if you have a big new feature, you can test, we'll just test the first part of that feature.
Don't test all of it and see how people react to that. The first part, the feature, I can talk for hours about this, but thank you very much.