Having data about how people are using features has revolutionized how I go about building and growing products. One of the key metrics that I’ve spent a lot of time optimizing is week 1 retention. It answers the question: of the people that start using your app, how many of them are still using it one week later?
Retention reports solve exactly this problem, but they can be confusing to interpret. MixPanel produces analytics software that produces retention reports, and they do a decent job of describing all of the information in their retention help article. While they are fairly intuitive, there is a subtlety that is hard to wrap your head around:
the beginning and ending of each bucket will be different for each customer in the cohort
That means that each person has their own “1 week” cohort, and that it takes 3 weeks for a single week to “mature” if you’re looking at week 1 retention. Here’s a good example:
In this example, we’re looking at users who activate the week of August 4th. That means that someone counts within this cohort when they sign up between 12:00 AM on Monday the 4th through 11:59pm on Sunday night the 10th. While people who sign up at different times both part of the 8/4 cohort, they each have different periods that represent week 0 and week 1. An example of two people in the 8/4 cohort:
- Signup on 8/6 at 12pm:
- Week 0: 8/6 12pm – 8/13 11:59am
- Week 1: 8/13 12pm – 8/20 11:59am
- Signup on 8/10 at 11:59pm:
- Week 0: 8/10 11:59pm – 8/17 11:58pm
- Week 1: 8/17 11:59pm – 8/24 11:58pm
This means that it takes 3 weeks for a cohort to fully mature so you know what the final week 1 retention percentage is. You won’t know the week 1 retention percentage for the 8/4 cohort until 8/25, when all of the users have been given a chance to be active in “their” week 1.
This makes analyzing and optimizing for week 1 retention very difficult, since it takes a long time to fully understand the implications of changes you’re testing. If at all possible, our team looks for early indicators of retention and leverage those as much as possible until we have our mature cohort percentages.