CategoryEntrepreneurship

Good vs. Bad Retention — The User and Revenue Impact

I just published this piece on Medium, but am also cross posting to my blog. If you want to make sure you receive all of the content I put out, make sure to subscribe to my email list.


There are many things that set Facebook apart from your (or my) products. That said, there’s really one thing that it all boils down to: retention. Facebook has developed a product that people use indefinitely. The rest of us? We have a long way to go. What should you be doing to close the gap? Keep track of your retention numbers.

Most of the people I speak with have no idea how many people they expect to be using their product in a year, even though they are the ones ultimately responsible for the progress. If you do have some sort of goal, did you just pick a big hairy number? Did someone throw out a goal for you? If I could give you one piece of advice, it would be to build a simple model so you know what to expect.

After watching Phil Libin’s talk on retention and cohorts, I thought it would be interesting to model out what different types of retention look like for a SaaS product. What would it look like if you acquire the same number of users over time, but don’t hang onto them? What would it look like if you had really good retention? What are the tipping points for user growth? I built a couple of simple Excel models, and the graphs were quite shocking to me.

Let’s say you launch a new product, and as a good leader you track the people getting value from your product over time. Imagine it looks like this:

Congrats! You launched a new product to 1,000 users in January of 2016, and have grown it to over 8,000 monthly active users by the end of 2016. Your growth is slowing slightly, but you’re not too worried about it. Why should you be? You grew by 700% in 2016! That’s a cause for celebration.

Lets look at this graph in a slightly different way, by the cohorts of people who start using your product each month. In the example above, I assumed that 1,000 new people sign up for your service each month, and that some of them stop using it over time. Those people might find a different tool, unplug from the internet, or get a virus and blame your tool for the havoc it caused. Either way, of the 1,000 people who start each month, some of them quit using your product in the months after they sign up. This is what the active users chart looks like breaking down the cohorts over time:

In the chart above, the blue shape on the bottom represents the 1,000 people who signed up in January, and then how many of them are using it throughout the year. By December 2016, only 450 of them are still around. The cohorts “stack” on top of one another to produce your total active users in a given month.

If you develop a great product like Facebook or Uber, there’s some percentage of cohorts that use your product forever. They’re addicted to it. Even if they stop using it at some point, they come back. Facebook would have a hard time growing to 1.6 billion monthly active users if a lot of people used it once or twice and then never used it again.

Let’s see what happens to your growth if you weren’t like Facebook, and you didn’t hang onto your cohorts for a long time like Facebook. Let’s say you continue to have 1,000 people sign up every month, but over time those people end up quitting your product. This is what the chart looks like past 2016:

By the end of 2018, you’ll only have 13,000 users of your product. You had 11,700 people at the end of 2017! Even though you grew 700% in 2016, you only grew 11% in 2018. The rate at which you’re growing is slowing significantly, even though you continue to add 1,000 users a month. You can see this visually in the bottom right of the chart, where all of the cohorts seem to stack on top of one another, but don’t add up to anything. You can’t even tell the cohorts apart, they just look like a colorful set of stripes. By the end of 2018, the new people you’re adding every month are barely replacing the people who abandon your product from all of your previous cohorts.

What does it look like if you are able to build a product where 50% of your cohorts end up loving you product and sticking around for a long time. What would that look like? Let’s update our graphs:

Wow! Instead of 13,000 users, you will have 20,000 users by the end of 2018. You can see the big difference between the graphs. In the bottom right you have rectangles that build on top of one another. You overall growth rate is still decreasing (as a percentage of your install base), but your total number of active users continues to increase. In the previous example your growth had basically stalled, in this graph you are growing at a constant rate. The best products in the world retain a large percentage of cohorts over time, and the bars are a large percentage of the initial cohort size.

Up until this point, I’ve only been talking about retention of users. If you’re running a business you ultimately need to charge for your service (for example, a monthly subscription). Assume that a percentage of people will end up paying for your service, and that they slowly upgrade over time. If you can forecast how many people will be using your product, you should also be able to project how much money you’ll be making. Lets look at what your revenue looks like (again, broken down by cohort) when you have poor retention:

As your cohort sizes go to nothing, those people won’t keep paying for your product. This graph doesn’t look too bad, but what about if you look further into the future?

That doesn’t look good, you’re barely making any more money two years later. What about in the case where you have good retention? Assume that 10% of the long term users end up paying for your product, and they pay $50 / month. They don’t immediately upgrade — it happens slowly over time. What would that graph look like?

Holy crap! I like the slope of that line. In the bad retention example, you are making $15,000 / month in recurring revenue. What about in the good retention example? Over $80,000 / month.

Interested in playing with the different scenarios yourself? I uploaded my hypothetical data in an Excel file here, or in a Google Spreadsheet here. Google Spreadsheets is crappy for this kind of stuff, I’d recommend using Excel.

Facebook and Twitter Onboarding Emails November 2015

As part of my work on Sidekick and HubSpot’s sales platform, I focus a lot on the new user experience of our products. As Brian Balfour likes to say “user onboarding is the one element of your application that all users will use”. Can you think of better metrics to invest than getting your users activated and set up for success?

As part of thinking through what will help explain the value of our products to users, I like to evaluate what other successful companies are doing on a regular basis. Recently, I took a look at the emails they send to users as part of the signup process as they move them towards an activation event. I thought it was interesting to see how the emails for Facebook and Twitter stacked up against one another. They have a ton of signups and a lot of opportunity to tune these emails to get the best results. What are they doing that might be applicable for your personas / use case?

First Email:

  • Facebook subject: “Just one more step to get started on Facebook”
  • Twitter subject: “Confirm your account, FirstName LastName”
  • Facebook goes with an aggressive headline with “Action Required”. Grabs people’s attention!  Twitter uses your name in the subject, I’m surprised that Facebook doesn’t do the same.
  • Both Twitter and Facebook want you to “complete” your account in the paragraph above, but both say “confirm” in the CTA. The two sentences in the Facebook feel so robotic.
  • Facebook looks to reinforce its value by describing why it’s useful: “helps you communicate and stay in touch with all of your friends. Once you join Facebook, you’ll be able to share photos, plan events, and more”. Twitter doesn’t do anything like this. I guess it’s hard to describe what twitter is to everyone in one sentence.

Second Email:

  • Facebook subject: “Welcome to Facebook”
  • Twitter subject: “Follow Vogue Magazine, Jimmy Kimmel and Rihanna on Twitter!”
  • Twitter is focused on getting you to follow users, rather than build out your profile. I’d guess that twitter is less about making it so your friends can find you, and more about finding content you’re interested in.
  • Facebook is obsessed about getting you to enter your profile information. They try to hook you with content first, but I assume that profile information is the key to showing you friend suggestions and other information you might like.

Third Email:

  • Facebook subject: “You have more friends on Facebook than you think”
  • Twitter subject: “Eric Shawn tweeted: “Should we accept more #Syrianrefugees? A look at one man’s journey @Foxnews, @CWS_global, @John_Kass, Watch:
  • Twitter is all about information and news (granted, I picked some accounts to follow in their onboarding process), while Facebook is pushing you to connect your inbox so they can prompt you to add your friends. This is one long email with a lot of tweets embedded in it.
  • It’s weird that Facebook shows so many different email clients, when I signed up with a gmail.com test address. Feels like they haven’t optimized this email, but what do I know?

Fourth Email:

  • Facebook subject: “Robinson Cano and Tom Brady are Trending on Facebook”
  • No twitter email (other than more content to view / follow). Interesting that they don’t prompt you to connect your address book, everybody else does this.
  • I’m surprised that the content of the email isn’t more engaging. I’m surprised they aren’t using images more prominently as twitter is, or showing information as it would appear in your news feed.

Here’s the side by side comparison for Twitter and Facebook’s emails. Did I miss something? Are you impressed with their emails, or underwhelmed?

 

FBvsTwitter

 

 

The Hard Secret About Optimizing Week 1 Retention

Having data about how people are using features has revolutionized how I go about building and growing products. One of the key metrics that I’ve spent a lot of time optimizing is week 1 retention. It answers the question: of the people that start using your app, how many of them are still using it one week later?

Retention reports solve exactly this problem, but they can be confusing to interpret. MixPanel produces analytics software that produces retention reports, and they do a decent job of describing all of the information in their retention help article. While they are fairly intuitive, there is a subtlety that is hard to wrap your head around:

the beginning and ending of each bucket will be different for each customer in the cohort

That means that each person has their own “1 week” cohort, and that it takes 3 weeks for a single week to “mature” if you’re looking at week 1 retention. Here’s a good example:

retention cohort

In this example, we’re looking at users who activate the week of August 4th. That means that someone counts within this cohort when they sign up between 12:00 AM on Monday the 4th through 11:59pm on Sunday night the 10th. While people who sign up at different times both part of the 8/4 cohort, they each have different periods that represent week 0 and week 1. An example of two people in the 8/4 cohort:

  • Signup on 8/6 at 12pm:
    • Week 0: 8/6 12pm – 8/13 11:59am
    • Week 1: 8/13 12pm – 8/20 11:59am
  • Signup on 8/10 at 11:59pm:
    • Week 0: 8/10 11:59pm – 8/17 11:58pm
    • Week 1: 8/17 11:59pm – 8/24 11:58pm

This means that it takes 3 weeks for a cohort to fully mature so you know what the final week 1 retention percentage is. You won’t know the week 1 retention percentage for the 8/4 cohort until 8/25, when all of the users have been given a chance to be active in “their” week 1.

This makes analyzing and optimizing for week 1 retention very difficult, since it takes a long time to fully understand the implications of changes you’re testing. If at all possible, our team looks for early indicators of retention and leverage those as much as possible until we have our mature cohort percentages.

The ideal relationship between you and your boss

Working in the tech industry nowadays, it’s all about how to attract and retain the best teammates.  I think the most valuable piece to recruiting and retention is not perks – it is cultivating an environment where we can learn and grow more so than at any other company. How do you create an environment where employees become incredibly valuable, and would be bored anywhere else?

I love this excerpt from this piece in particular:

From the very beginning of my relationship with an employee- often during the interview process before they’ve even joined my team- I draw them a picture and make them a promise.

The promise sounds something like the following.

“Let me start with the bad news. The bad news is that you’re going to leave LinkedIn one day. I know you just got here and so it may be strange to think about leaving already, but I want to bring your attention to it so that you and I can partner together on making the most out of the time we spend together in this company. I don’t know if you’re going to spend 2 years here, or 5, or 10 or more, but I want to make sure that however much time you spend here with us on this journey, that when you look back on your whole career 20, 30, 40 years from now that you will look at the years you spent here as the most transformative years in your career. The years where you learned the most, grew the most quickly, were exposed to the most incredible people and the most innovative thinking. I want these years to be the years that literally change the trajectory of your career. I want you to enjoy more success in your life directly due to the experiences you had here at LinkedIn than you would have enjoyed had you never chosen to work here.

Have you ever done anything like this with an employee at the start of their tenure at your company? How can you do a better job of helping your employees accomplish their goals?

An Insider’s Look At HubSpot Sidekick’s Growth Approach

This post was originally published here on onstartups.com. It describes the process that I use in my day to day activities working on the sidekick team.

The Sidekick growth team is a small, data driven and aggressive group within HubSpot that works on new, emerging products with massive audiences and a freemium business model (similar to Dropbox and Evernote).  We are constantly pushing ourselves to learn new growth strategies, tactics, and techniques. I have personally become more data driven and model driven after joining the team, and wanted to walk through an example of one decision that became much easier with the use of our generic problem solving framework.

I am a big believer in the idea that complicated problems look simple when you are able to break them down.  Don’t take my word for it – this is what was attributed to Einstein:

If he had one hour to save the world he would spend fifty-five minutes defining the problem and only five minutes finding the solution

The Sidekick growth team follows a very straightforward process that strives to take complicated choices, and analyze them to produce areas of opportunity:

Step 1: Choose a goal

Step 2: Build a model

Step 3: Analyze the inputs

Step 4: Identify opportunities

These steps are generic enough that they can be applied to many kinds of problems.  Whether you’re on a sales, marketing, product, support, services, or any other type of team this framework is incredibly valuable.

Step 1: Choose a goal

Choosing the right metric / goal is very challenging and critical to our success.  If our team optimizes for the wrong metric, it doesn’t matter how well we execute because our efforts won’t translate into success.

Our identified goal:

We ultimately chose to define our goal as increasing the number of people active on a weekly basis.  Rather than pick any metric, a person has to take one of six key actions to demonstrate that they are getting value from our product.  Brian Balfour talks about the cycle of meaningless growth here, but the key takeaways for our product are that more people using Sidekick helps us to grow faster.

Some of the attributes we used when picking our goal:

  • It is a holistic representation of our product. We thought about all of the ways that someone uses Sidekick, and thought about the best way to represent them.
  • It’s authentic (hard to fake).  If you optimize for a hollow metric that is easy to attain, but doesn’t translate into success later on, you might fool yourself into thinking you’re making progress. If we were to pick signups, we might crush that goal and get a lot of users to sign up, but they might not stick around.
  • It represents real value.  If you solve for your own needs instead of the customer’s needs, you may be successful in achieving your goal but it won’t translate into true success down the road.  We tried to pick a goal that represented users getting value out of the product, which results in people upgrading to the paid version of Sidekick.

Step 2: Build a model

With the goal established, we then set out to build a model to understand what will have the biggest impact on weekly active users (WAUs).  If we simply tried to increase our top level goal on its own, we wouldn’t have an understanding of where to start.  In order to understand where to focus our efforts, the model breaks down the goal into manageable pieces.  The whole point of building the model is to understand what the inputs are, and what the biggest contributor to our goal is.

Our Excel model breaks down our goal (WAUs) into the individual components that drive it on a weekly basis.  It’s a simple equation:

WAU = (New people) + (People from previous weeks who continue to use Sidekick)

We broke down each of these two buckets into their individual components.

WAU =

  • New People:
    • Channels:
      • People we acquire through paid acquisition
      • People we acquire through content marketing
      • People we acquire through SEO
      • People we acquire virallly from existing Sidekick users (invites, etc)
      • Activation Rate
        • Not everyone who signs up ends up using the product.  Therefore, we measure the people who install our software and get it up and running correctly through an activation rate. Rather than look at the activation rate across all channels, it’s important to understand how each one is different, and if there are isolated pockets ripe for improvement. For example: users who read our content are more likely to set it up than someone who clicked through on an advertisement before signing up.
  • People from previous weeks who continue to use Sidekick
    • We look at the number of people who sign up each week, and then look to see how many of them are active each week since they signed up.
    • We look at retention, which is critical in freemium businesses.  In order to accomplish our goal of having millions of users, we have to retain the users we acquire.

This is a screenshot of our model:

excel growth model

The numbers in this screenshot have been changed so they are not reflective of our true numbers.

Step 3: Analyze the inputs

The model above is extremely valuable because it allows us to use our week-over-week growth to forecast the long term impact of any change. It’s incredibly hard to understand how multiple factors could interact over a long period of time. It might be possible for someone to reasonably predict the implications of any change, but without the model it is easy to be short sighted.

With our model built, it was easy for us to test the sensitivity of the inputs.  For example, if we were to increase the number of users we acquire from our paid acquisition budget, how would that impact our WAUs in a year? Instead, if we focused on retention and user acquisition rates stayed the same, would we have more users a year later?  What about if we improved the conversion rate for a different area of the funnel?

Rather than sporadically tackling new campaigns or projects, the goal is to understand what is the most impactful focus area for the business.

In looking at the Sidekick funnel, we found that two of our biggest drivers were retention and viral growth.  We modeled how changes to each of them would impact our goal, and decided to focus on retention first before looking at increasing the number of new people through viral channels.  At the end of the day, some of the factors that we always consider:

  • What is the current state of the metric?
  • How much do we think we can improve this metric?  What’s the ceiling on any improvement?
  • What are the resources required to have a meaningful impact?  How long would it take?

Factoring in answers to those questions, and including the estimates in our model, we decided to focus on improving our retention in Q4 2014.  There was a lot of analysis that went into picking retention; it was the result of repeating Steps 1 through 3 multiple times.  By going through the process of evaluating different levers in the model, it becomes much easier to weigh different options against one another and impartially judge alternatives.

For the Sidekick team, it wasn’t as simple as saying that we wanted to improve retention.  Just like WAU’s, retention in itself has many inputs that we had to evaluate.

  • Of the people that stop using Sidekick, we lose the majority of them in their first couple of weeks
  • In the hypothetical example below, we have sample numbers of how we retain users over time:
    • 45% of a cohort stops using Sidekick one week after signing up
    • 5% of a cohort stops using Sidekick two weeks after signing up
Cohort Size Signup Week Active 1 Week Later Active 2 Weeks later Active 3 Weeks Later
100 11/3 55 50 45
110 11/10 61 55 50
120 11/17 66 60 54

The numbers in this table have been changed from their real values for this post

  • Given the size of our user base, we determined that week 1 retention was our biggest issue and opportunity.  If our existing user base was larger, our long term retention might have been a more important issue.  The lesson is that your biggest areas of opportunity depend on your current context.

Once we isolated the fact that people stopped using it after their first week, we set out to understand why someone who installed Sidekick would stop using it after they signed up.

Step 4: Identify Opportunities

At this point of the process, we know what’s most important to our goal and the implications of an improvement.  The next step is to start identifying how we can make an improvement.   Depending on the lever, there’s a mix of elements that are helpful in breaking down the opportunity.  We used quantitative analysis to identify a problem segment, qualitative analysis to flush out its symptoms, and used our understanding of our product to come up with ideas to address the issue.

To identify a problem area, we did a quantitative analysis of the people that only used Sidekick for a single week.  We looked to segment these users to look for patterns, such as:

  • Where were these users coming from?  Was there an issue for a single channel of users?
  • What technology were these users using?  Was it an issue with Gmail, Microsoft Outlook, or Apple Mail?
  • What part of the application were they using the most?
  • How much do they use Sidekick?  How many days did they use it?  How much did they use it their first day?

In asking these questions, we found that Gmail users were more likely to stop using the product when compared with other email clients. This was a complete shock to us.  We had figured that Gmail would retain fairly well, and that an issue would be likely to exist in one of our other email clients.  We found that a large number of these people were only active the day that they signed up.  To understand their usage on their first day, we created a histogram that showed how many tracked emails this population of users sent their first day.

emails sent distribution of week 1 churn users

The numbers in this chart have been changed from their real values for this post

For the Sidekick team, it wasn’t surprising that people who only tracked a single email their first day didn’t come back.  The surprising element was that such a large % of these people were only tracking one email.  We wondered why someone would go through the Sidekick onboarding process only to never use it again.  Wouldn’t you at least test it out with a couple of friends or coworkers?

To understand why these people stopped using Sidekick, we sent out a simple email to a thousand users.  I emailed them individually by BCCing them from my HubSpot account, asking for feedback on a specific question designed to bring insight to the pattern we discovered.   We bucketed the replies to our email, and found that there were big opportunities to improve our week 1 retention.

reasons why people churned

The numbers in this chart have been modified from their real values for this post.

I was personally ecstatic when I saw these distributions.  It wasn’t that a competitor was better, or that there was a mismatch between the features people were looking for and what our product offered.  The issue was a psychological one:

We weren’t doing a very good job explaining what our product did, and how people could get value from using it.

Rather than having to build a lof of new features, we needed to experiment with explaining the value of the product.  It’s much easier to test out different ways of describing the product than addressing weird edge cases or building entirely new features.

With our quantitative analysis done and having received qualitative feedback from the segment of users we were most interested in, we spent time brainstorming ideas to address the opportunity.  We looked at how competitors accomplish the same task, how companies in other industries educate their new users, and researched why our most passionate users like Sidekick.  I’ve included a list of sample experiments we’ve tested:

  • Only show the Sidekick web application once we have value to demonstrate
  • Show a video of someone using Sidekick and how they get value out of it
  • Ask users whether they intended to use Sidekick for personal or business use cases, understand whether we should try to change their mind or give them examples that align with their mind set
  • Show a narrative of how someone uses Sidekick over a period of time
  • Incorporate our onboarding into the Gmail interface rather than in our web app

Conclusion

Looking at the opportunity we have focused on for Q4 2014, it seems kind of simple and obvious.  By setting an appropriate goal, understanding the inputs to that goal and finding the biggest contributor, it led us down a path to clearly define our next steps.  While finding a solution isn’t guaranteed, the team is confident that if successful it’ll have a big impact on our trajectory for 2015.

This framework isn’t perfect and isn’t for everyone, for instance, if you are creating a new product or process and have a small sample size.  However, for the Sidekick team, this process has been an enormous help in prioritizing where to focus energy and resources and get the team aligned behind a common goal.

This framework is incredibly valuable to the Sidekick team for multiple reasons:

  • It breaks down large, complicated problems into actionable and manageable tasks.
  • We have confidence that the opportunities we are working on will have a big impact.
  • We understand the relative importance of different initiatives and are able to make conscious decisions about areas to pursue and the resulting trade offs.  It’s also easier to decide what we shouldn’t be working on, even if it may feel important.
  • Our team can see the direct impact on individual metrics, and understand how any improvements translate into the success of our team.  Teams like being able to track their progress and see how their efforts translate into success.
  • It’s a repeatable and scalable process.
  • The insights aren’t isolated to technology solutions – they can be as simple as messaging and the steps instructions are displayed.

Great!  How do I apply this?

Your time is extremely valuable. Whenever you can, make data informed choices.  You don’t have to be a slave to your model, but you should be making conscious decisions about what you’re focusing on and why it’s most important. Hold your colleagues accountable – ask them why an initiative is important, and what the impact will be.

The mantra of our team is that we want to be the best at getting better.  If you’re interested in learning about growth and seeing your contributions all the way to a business’ bottom line, our team is hiring.  You can see a list of the open positions here.

Thanks to these wonderful readers for their feedback: Brian Balfour, Anum Hussain, Maggie Georgieva, Jeremy Crane, and Andy Cook.

© 2019 Dan Wolchonok

Theme by Anders NorénUp ↑