What makes it worse is that nobody is giving others feedback to improve. I feel guilty I didn't provide better feedback to my direct reports when I was a manager, and I didn't provide enough feedback to my peers when I saw them kicking butt or when I saw how they could have done better.

It's uncomfortable to give feedback to others, that's the main reason that people aren't giving feedback. Companies have to mandate a once a year performance review to make sure that managers do a write up, have the conversation with their team member, and then have the person acknowledge it. If you talk with most people, they're not satisfied with the feedback they receive from their manager.

The thing that has changed recently is that AI is really good at giving feedback to help you improve. Mozilla shared how they found most bugs in April (due to AI) than they had in the entire previous year. If AI can find decades old bugs, I think it can tell you when you're undermining yourself or not speaking confidently.

I now have a windows app, a mac app, and a web app that all help you to analyze how you're showing up in your important meetings. The goal is to give you feedback into how you can do better in each meeting and spot patterns across them.

Why is this different from Granola, Wisper Flow, Otter, Fireflies, Fathom, TLDV, etc? They're all built for your company. They want to become the second brain for your organization. They want to sync to your CRM, they want to draft follow up emails, they want to draft the next steps in Jira, they want to kickoff workflows, etc. They're built for your company to automate your work, not help you improve how you communicate. Look at their pricing pages to identify who they're selling to. They have free plans to get you using it, and then their paid tiers are all about enterprise features. They want to sell SCIM, SSO, enterprise features, etc.

It's not about helping you do better work, it's about doing more work and selling to the head of your company's IT tools.

My goal is to build a tool that is focused solely on helping people communicate better. For all of the new fancy AI tools that we have, the most important part isn't your prompts, agent, model, harness, or anything else. It's still about making a convincing argument and telling it well to help tell your story, advocate for yourself and your ideas, and convince others to invest in you. Nobody is focusing on that.

That's why you should try out my new product, work coach. Download Work Coach for Mac or Windows here.

We use reverse ETL here at Reforge for a bunch of different use cases. They are:

• Sending data about our customers to our email automation system, Iterable
• Sending data that is aggregated in our data warehouse (snowflake) to a Google sheet
• Sending data to our behvaioral analytics system, Amplitude
• Sending data to our CRM (HubSpot) from our data warehouse
• Sending data to Slack from our data warehouse

Examples

We use Segment and Personas features to send data to our email system, Iterable.

There are cool features in Personas that allow you to use a GUI to define things like:

• Counters (example: how many page views that meet a certain criteria in the last X days)
• Aggregation (example: how much revenue have they generated in the past quarter)
• Most frequent (example: what's the most frequent blog post they've visited)
• First (example: what's the first marketing campaign that got a user to visit your site)
• Last (example: what's the last marketing campaign that got a user to visit your site)

They also have the ability to write arbitrary SQL to pull a list of people and any columns in the SQL can be traits associated with a user. So you can use GUI to define basic things or use complicated SQL to pull sophisticated stats.

An example of a user trait we have in Segment:

Important caveat: it's on our roadmap to pull all of this logic out of SQL and put it into a DBT model. The benefits there are better testing, understanding of data lineage in a DAG, and we have all of it under source control. In the future the SQL for this trait could be something like applications.user_application_values.

In our data dictionary we define these properties:

Then we can use the traits to send personalized campaigns in Iterable for many different scenarios:

• Application acceptance or denial
• Payment nudges to remind someone they've been accepted
• Stats about how many people from their company have participated

Here's an example of a campaign in Iterable:

You can see how we have a dynamic template in the email above that personalizes the email subject to have the company name they select as well as stats about the company with things like:

• How many people from their company have participated in Reforge in the past
• How many people from their company are current members
• How many people from their company have applied recently

I think it's pretty great that Segment has this functionality especially when you combine it with their other platform tools like Protocols. That said, I have some gripes about the tool as of November 2022:

• On our current tier the most frequent traits can run is hourly. Maybe I can pay for a higher frequency, but that drives me nuts and I'd have to negotiate with a sales rep for that.
• It's really hard to test a trait is working the way you want it, you can't trigger a sync. This drives me nuts because I always make a simple mistake and then either try to get it to trigger or have to wait an hour to see if an update fixed it.
• The destinations you can sync to are the typical ones for a CDP. I wish I could sync a result set with a spreadsheet, but that's not possible.
• I can't sync non-contact objects to our CRM. For instance, I can sync contacts and their properties to HubSpot but I can't sync subscriptions to a custom object or companies and their details to a company object in HubSpot.
• It feels like Segment has abandoned working on this feature. They're busy integrating with Twilio and building things like Twilio Engage rather than investing in making Personas great. I might have missed it, but it feels like they haven't made changes to the product in years.

How we use Hightouch

We use a second ETL provider at Reforge. It sucks that we have two, but my hope is that it's more of a transitionary period until we ultimately simplify with one. I started using Hightouch for free to sync data to a google sheet / to slack, and it became clear it was the vastly superior product to Segment Personas.

The world runs on spreadsheets, and often times it's really helpful to give teams a constantly updated resource that has data synced from your data warehouse. It's a great free path to get using their tools because once you have it connected, it's so easy to add additional sources.

Here's an example of how we send paid data into a spreadsheet for the marketing team (we pull paid numbers across all channels into our data warehouse and this helps them understand all of it and play with it in a spreadsheet environment rather than pulling it from a BI tool constantly):

The model can be custom SQL, a table in a DB, or a model in dbt or Looker. It makes it really easy.

We also use Hightouch to sync data to our CRM, HubSpot. It's just so easy.

We also use Hightouch to send customized messages to Slack. I've done this using Zapier in the past but it's nice to be able to have all of these syncs in one place. Here's an example of a daily ARR report that the team built:

Here's what it looks like in Slack:

I spoke with the co-founders of Hightouch when it was early days and that's when I found out they were part of the early team that built Personas at Segment. It feels like they were able to address a lot of their frustrations about how the product was built and make big improvements on it. Things that are just so much better than Segment:

• Their support is so much better than Segment. This should be every startup's advantage against the company acquired by a big behemoth, and they do a good job. I have been so disappointed with Segment's support over the years and Hightouch makes the overall experience so much better because you can get an answer to why something is broken quickly. With Segment support, I often times have to wait days which is so painful when you're working on a deeply technical integration like this.
• They have a "run now" button where you can run a sync anytime you want. This sounds so simple, but I'm sure it makes the technical implementation much harder. I can't tell you how many times I've messed up a piece of a sync in Segment personas and then I just have to wait until the sync runs again. It baffles me they haven't changed this.
• It feels like it's being worked on all the time. Lots of changes and improvements, new features, and integrations with the latest and greatest tools.
• The logging and insight into what happened is so much better with Hightouch. This is a real interaction I had with Segment's support team (no judgement on the support rep, this is a reflection of the state of their product). This type of response is infuriating when you are working to integrate two systems together and you don't know what's wrong or when it'll retry some logic.

Conclusion

I look forward to moving all of our ETL infrastructure over to Hightouch from Segment at annual contract renewal time. While there are benefits to having the reverse ETL logic and elements all in Segment (we pay for their protocols feature and there are benefits in blocking bad data / being alerted when your data doesn't adhere to the spec), I would rather use the better tool.

How do others do this type of work at their companies? I'd love to know if others solve their challenges similarly and if there are even better ways.

That led me to wonder if I could understand better what was driving our costs on snowflake and ensure that we didn't get an unexpected bill. Our storage costs are tiny (we're lucky at Reforge that we have enough data to make it useful, but not so much that the costs are prohibitive / make us move slowly).

I did two things as a result of this:

1. Visualized the costs over time.
2. We setup Metaplane so that it would monitor our daily usage and alert if it deviated from the historical norm. This was really easy.

Visualizing Costs over Time

I got some great feedback from the DBT community about the types of queries I should be running to run down costs. They asked good questions like:

• How are your warehouses configured?
• What warehouses are creating the most spend?
• Are the warehouses set to auto-suspend?
• What services are contributing to your bill?
• Is your spend mostly on storage or compute?
• What queries cost the most?

Ultimately, one of the things that came out of the analysis was that all of our bill was being used for compute. The sales team at snowflake said that most customers don't pay for "cloud services" and that we were an outlier. That led us to look into what was driving the overage on our cloud services. I visualized the cost over time with Deepnote for both compute and cloud services. You can see that "overage-cloud services" represent a big part of our bill, and we attributed most of that cost to the metabase queries.

We discovered the issue in April and took some action to try to reduce it. If you want to do the same analysis, I posted the notebook here. I'd love to build a flowchart of the types of things you should be looking at when analyzing your snowflake bill and adding them to the jupyter notebook.

Being Alerted to When Costs Go Haywire

Metaplane is a tool that looks at your data systems and lets you know when something is broken. It could be that tables have stopped being updated, it's not being updated at the same right, or someone is spamming events to your system. It has many different kinds of tests, but ultimately it helps to ensure that you know when things are going haywire so you can fix it before it causes problems or causes others to lose confidence in the reliability of your data and systems.

We configured Metaplane to run this query every day for both cloud services and compute spend, which are the vast majority of our spend. If it deviates from the historical norm it alerts the team so we can dig into what is increasing our usage and our bill. A big thanks to Ian Whitestone in the DBT community who helped with suggestions on this query as well as other ideas.

SELECT
  SUM(usage_in_currency) as total_cost
from SNOWFLAKE.ORGANIZATION_USAGE.USAGE_IN_CURRENCY_DAILY
where usage_type = 'overage-cloud services'
    and usage_date >= current_date - interval '1 day' 
    and usage_date < current_date
order by 1

Example Use

In the above image you can see how there was a big spike in usage that was much higher than the previous sixty days. Our team was notified, looked into the issue, and this was the result:

found two queries made in the Snowflake UI that had been running for 20 and 40 hours respectively.  I force canceled the queries and let the user know what happened.  We also decided to update the query timeout account wide from it’s default value of 48 hours down to 1 hour.

One of the data engineers on the team

This is a great example of how Metaplane ensures that we're on top of anything that is unexpected and can affect our team. It may be data quality issues, reporting issues, delays, logic issues, or something like costs. Examples like this confirm that data observability is a must-have that all companies will have in the next couple of years.

Next Steps

Ultimately, we didn't think it was worth our team's time to try to optimize our spend on Snowflake due to our metabase issues. I'd rather the team spend time on creating new value for our company and our customers rather than spending a lot of time to save $1,000 / month. There will come a point where it's worth it for someone on our team to look at this and a myriad of other areas for improvements, but until then we'll be monitoring our spend closely to ensure it doesn't change materially from where it is today. If anyone has any tips or insight into how to address the metabase problems, let me know!

One of the issues is that we ask people to be available for a week at a time to be the point person on responding to questions or issues that come up. We don't expect everyone to be the one to fix / respond to an issue, but we ask that they be the point person to help move it forward.

I built a really simple reminder workflow in Slack to remind people of what they should be doing when they are in this role because it's easy to forget. I thought I'd share how easy it was to setup and see if anyone had any other tools / processes they use.

We already had a google doc that listed the dates that each person was responsible for, and I built a quick mapping table that takes a person's name and puts in their Slack id. It's easy to get someone's slack id.

I then built a workflow in Zapier that:

• Runs every day at a standard time (12pm ET)
• It checks to make sure that it's a weekday (not a weekend)
• It looks up the row in the spreadsheet for the current day
• It only continues if there's a slack ID for that person that day (if someone doesn't want the reminders they can delete their slack id from the row)
• It sends them a direct message on slack from a reminder bot

This is the DM that people get in Slack:

How do others in the data space ensure that the team rotates through different responsibilities, get reminded of what they're supposed to do, and ensure they're aware of how to do it? I don't want to go full-blown pager service and this felt like the right level of reminder for our stage of company (the team has 5 analysts, a manager, a data engineer, and myself).

I built a weekly report that lays out how my company is using our BI tool, Metabase. It has the following reports:

• Dashboards created in the past 7 days
• Questions created in the past 7 days
• Individual Dashboard views by our exec team in the past 7 days
• Dashboard view counts by user in the past 7 days
• Activity by my boss (our CTO)
• Metabase MAUs
• Metabase WAUs

It helps me to understand at a high level what is happening inside the company from a data perspective:

• Who is building reports (questions / dashboards)
• What questions people are asking of our data
• What is most important to our exec team
• Who is looking at information on a regular basis
• What is top of mind for my boss
• How engaged our company is with our data & analytics

It was surprisingly easy to set this up. All we had to do was add the Metabase internal database as a source to Metabase (sounds confusing, but actually quite simple). It wouldn't shock me if this became a paid feature long-term for Metabase on their enterprise plan, but it took less than 30 minutes to figure out their data model and build some quick and dirty SQL questions that produced answers to our most important questions. Metabase is already storing all of the data we needed to visualize to get answers to our questions.

I set this up as a dashboard subscription so it is delivered to my email every week and I can quickly scan it to understand what's top of mind for the org.

Reforge is Data Driven

Since the beginning of the company we've looked at data to understand how well we're tracking against key goals for the company (revenue, applications, content completion, weekly activity, etc). These stats span across the entire company in terms of usefulness (support, sales, marketing, product, engineering, design, finance, etc) and we strive to make as much information accessible to everyone so that they can be data informed.

If you're considering joining a company - do you want to be at one that uses data to help inform decisions? You can see below that Reforge employees are using data on a regular basis and you can see how it scales as we've grown the team and hit key milestones. We've typically seen 75-100% of employees leveraging Metabase on a monthly basis.

The queries that we use to power the charts:

Dashboards created in the past week

select created_at, name, u.email, left(description, 40) as description, 'https://BI_HOSTNAME/dashboard/' || d.id as url
from report_dashboard d
    inner join core_user u 
        on d.creator_id = u.id
where created_at > current_date - interval '7 days'
order by 1 desc

Questions created in the past week

select created_at, name, u.email, 'https://BI_HOSTNAME/question/' || d.id as url
from report_card d
    inner join core_user u 
        on d.creator_id = u.id
where created_at > current_date - interval '7 days'
order by 1 desc

Exec team dashboard views in the past week

For this report, I wanted to know what the exec team was looking at in the past week. I want to know how much they're looking at the metrics and whether they're looking at the dashboards I think they should be looking at.

select u.email, d.name, count(v.id)
from view_log v
    inner join core_user u
        on v.user_id = u.id
    inner join report_dashboard d
        on v.model_id = d.id
where model = 'dashboard'
    and u.email in 
        (LIST_OF_EMAILS)
    and v.timestamp > (current_date - interval '7 days')
group by 1, 2
order by 3 desc
limit 200

Dashboard views leaderboard (views in the past week)

This shows the number of views by user in the past 7 days. It's always a good sign when your head of data isn't the most prolific viewer of dashboards.

select u.email, count(v.id)
from view_log v
    inner join core_user u
        on v.user_id = u.id
    inner join report_dashboard d
        on v.model_id = d.id
where model = 'dashboard'
    and v.timestamp > (current_date - interval '7 days')
group by 1
order by 2 desc
limit 200

Most popular dashboards by views

select d.name, count(v.id)
from view_log v
    inner join core_user u
        on v.user_id = u.id
    inner join report_dashboard d
        on v.model_id = d.id
where model = 'dashboard'
    and v.timestamp > (current_date - interval '7 days')
group by 1
order by 2 desc
limit 200

Questions created by a single person

select created_at, name, u.email, 'https://BI_URL/question/' || d.id as url
from report_card d
    inner join core_user u 
        on d.creator_id = u.id
where created_at > current_date - interval '28 days'
    and u.email = EMAIL_OF_YOUR_BOSS
order by 1 desc

Metabase MAUs

select date_trunc('month', timestamp), count(distinct user_id) 
from view_log 
group by 1
order by 1

Metabase WAUs

select date_trunc('week', timestamp), count(distinct user_id) 
from view_log 
group by 1
order by 1

We’ve been busy in this time. Some of the work that we’ve done:

• Migrated our data warehouse from postgres to snowflake
• Rebuilt our entire data warehouse schema organization
• Built countless models to model our programs
• Enabled the switch to a subscription model (reporting on subscriptions, change over time, renewals, etc)
• Worked with our R&D team to adopt and use a behavioral analytics system
• Defined our data dictionary
• Hired and onboarded 8 people
• Built data pipelines to enrich data in our system via Airflow
• Migrated our email automation system

Reforge as a company has also been doing very well during this time. Our business has grown tremendously, we raised our $60M series B round, and we have more than doubled the team in that time.

We had the good fortune of having raised our Series B earlier this spring before the recent economic turmoil and have the the money, time, and resources now to invest heavily in creating new experiences and products. It’s an exciting time to be investing and building when it feels like many companies are cutting back or deprioritizing investments.

Data engineering will be a critical component to our upcoming growth and I’m particularly excited about the projects we are undertaking:

• We will build data pipelines to enrich data about our prospects and customers. This will be the foundation for personalization and recommendations in the future. Having a reliable and performant system that can help us understand who someone is, where they work, and build classification models for their attributes will be critical for future product experiences as well as analyzing how our business is performing.
• We will build more sophisticated foundational reporting for our growing analyst team to work with. Our analysts work with stakeholders in every department to help find insight in data, build visualizations, and impact our strategy and future roadmap. You will have a ton of leverage in creating impact - the data sources you create will be used by every department.
• We will invest in additional data syncs between our systems to ensure that every customer interaction is excellent. Reforge is teaching how the best technology companies execute, and we aspire to be the best example of our own frameworks. Our customers are the leading practitioners in the tech industry and love to hold us accountable.
• Revamp the infrastructure to enable reporting for our marketing organization. When the data team was spun up we didn't have anyone in the marketing org, yet were growing 100% year over year due to word of mouth. The team has grown a lot since then, and we are becoming much more sophisticated in how we approach our marketing efforts. We need to empower our analysts and the marketing team with the data they need to understand how well we're attracting people to Reforge, where our best return on investments are, and what best explains the value of Reforge.

We use best of breed tools in our data stack. Here’s a video of our current data stack. Most of these tools have been implemented in the past year and we will continue to invest heavily as our business grows and we become even more sophisticated.

If you know of anyone who wants more responsibility and has a strong vision for how data engineering should work today, please send them this post or let me know if I should reach out to them. This is a critical role for us to fill before the end of the year.

Apply here for the role.

It's a very exciting time for Reforge and for the Data Team.

Helping People do the Best Work of Their Careers

Reforge is an amazing opportunity to work on a product that has a significant impact on people's lives. We are taking the untapped knowledge, frameworks, and practices from industry leaders and making it accessible to our customers. Our goal is to help our members do the best work of their career by unlocking insights and then helping them apply it back to their jobs right away.

We routinely here from our members that it was the best educational experience they've had (better than their MBA if they got one), and that it has helped them be more confident in their role and drive a big impact for their business.

$21 Million Series A Investment

We raised our Series A investment from A16Z in February. At the time, our CEO Brian Balfour wrote about:

• The history of Reforge
• Why there's a real need for our offering
• The solution we are building

I was lucky to be one of the earliest employees that took this from a nascent concept through today. It was an incredible time to be at the company, but today's inflection point is even more exciting. While we bootstrapped the company to eight figures of revenue, our recent fundraising round gives us the capital to invest even more aggressively.

We are the rare startup that has real revenue with generous margins and profits, yet also has raised venture capital and the ambition to continue growing 100% each year.

The Data Team at Reforge

The data team is just getting started here at Reforge. We're looking for people who want to be a part of a small team that is growing fast and are comfortable with ambiguity and fast paced change. It's a great opportunity for people who want to be a part of building a strong data practice at a company that sees data as invaluable in operations as well as strategic decisions.

The nature of our product is that it's cross functional. We create content, host events, and build relationships between people through marketing, community, operations, product, design, and engineering.

We have a central data team so that every part of the org has access to all of the data to deliver an exceptional experience. Every part of the experience should be tailored to your role, your company's business model, where you're located, how senior you are, what you've done in the platform, and what your goals are today. The data team is here to make sure teams have the information they need to deliver and iterate on an exceptional experience.

The data org owns the growth model for the business. It's our job to help bring together the metrics for the whole business to understand how we're trending towards our goals, what is performing well, where there is opportunity to improve, and what the greatest points of leverage are. This is especially important for a cross functional product like ours where opportunities require multiple groups within the company to collaborate.

What we're looking for

We're looking for someone who can take ambiguous questions, run independent analyses, and then clearly communicate their findings.

• Analysts will be required to field questions from many potential sources: the leadership team, PMs, Marketers, Engineering, Design, Operations, or support. Being able to listen to the questions of teams, clarify their goals, and then come up with the right way to analyze and summarize findings is critical.
• Often times the initial question asked is not the right one or needs to be clarified and adjusted. It is not an analysts job to mindlessly give teams the answers to their questions, but to push back when necessary and be a collaborator with their teams to gain insights that will push the product experience forward and improve the business.

An analyst should be deeply quantitative but naturally curious. This person should have opinions about software experiences and be passionate about finding ways to quantify value to end users, impact to the business, areas for improvement, and insights into behavior.

• Teams may not always know the right question to ask and the best analysts have their own opinions that they pursue.
• Analysts should understand how the product works, the user psychology of end users, and how the product is different from its alternatives. The quantitative metrics that are used to measure its performance are a direct result of finding ways to express a deep understanding of the product.
• The best analysts are looking around corners to help uncover pockets of strength, quantify areas that are under-performing, and are thinking about the best ways to influence people about what's important and should be focused on.

An analyst should be an excellent communicator, story teller, and consultant.

• We are not looking for analysts that are simply responsible for producing charts. They will need to be able to understand the motivation behind a request, suggest alternatives, and have the conviction to push back when they disagree to foster debate.
• We want analysts who are able to tell a story with the data, explain the technical elements of an analysis but communicate why it's important.
• We want analysts to be consultative with their peers - they need to understand what will be impactful and resonate with an audience and tailor the summary accordingly.

The Tools we Use

As much as possible, we use the best of breed technology stack available today. We have roots as a profitable, bootstrapped company so we haven't upgraded all of our tools yet but we are constantly evaluating each of the tools to ensure we're using the best tool for the job.

Our core sources of data:

• We have a read-replica of our production database so that we can query the latest and greatest production data in real time.
• We have a data warehouse that is populated hourly by our customer data platform, Segment.
• We are able to seamlessly run queries that mix and match data between the two systems so that we can use the latest production data with an analysis that uses raw event based data or models computed in the data warehouse.

Some of the tools we use:

• We use DBT for modeling and data transformation in our data warehouse
• We use github for our DBT models and all of our code
• We use metaplane to ensure that we are the first to know of any structural changes to our schemas as well as unexpected changes or trends within our databases.
• We use iterable as our email marketing tool
• We use Segment for our event pipeline, reverse ETL tool, and to populate our data warehouse with event level information from sources of data.
• We use Airflow for any jobs that connect to external services and to create more sophisticated models and jobs that can't be created in pure SQL.
• We use Amplitude as our behavioral analytics tool to get insights about how people are using our product, key conversion paths and funnels, and retention behavior.
• We use Metbase as our BI tool. I've written about examples of how we use it here and here.

How Teams are Structured

We are rapidly building our teams and how they are structured, but our strategy is a product led experience. That means that our product team is responsible for the core experience of our members. We are structuring teams in pods that own discrete elements of the experience. While every pod may vary, they will typically have:

• A product manager
• A product designer
• A Tech Lead
• Multiple software engineers
• A marketer
• A product analyst

It will be the product analysts responsibilities to:

• Be the expert on the team about data
• Empower the team to do their own reporting and answer the vast majority of their own questions
• Do deeper analyses than any other team member can do

If this is interesting to you, please apply here.

We use Segment as one of the key pieces of data our data infrastructure, and I recently turned on the Stripe integration. I was pleasantly surprised by how well it works. I authenticated into our Stripe account to configure it as a source within Segment. Segment then updates tables in our data warehouse with the latest and greatest data from Stripe. This is a screenshot of our database and the tables under the Stripe schema:

On one of our key dashboards showing our progress in generating revenue, I wanted a cumulative revenue chart. This is helpful to see how quickly we're generating revenue, what our total revenue in a period of time is, and how our revenue growth compares to previous periods.

This is a sample chart that's easy to setup in Metabase:

To generate it, this is the SQL I wrote to generate it:

The way it works:

• The first statement generates a table (that's the generate_series function) that has a start date and end date
• I join from that table with a left join to the stripe payments table. This ensures that if we don't have a day of revenue, the day still shows up in our table. This table gives me a total amount of revenue per day.
• In the final part of the statement, I use a window function to do a cumulative sum of all of the revenue per day, so that day 2 has all of day 1's revenue along with day 2's revenue added in.

I hope this helps you create similar charts of revenue per day or cumulative revenue reports. I've worked with teams where we've manually hit the Stripe API to pull all of this information, which is always a pain in the butt. The nice part of this solution is that Segment keeps the table up to date automatically, and then anyone can run these reports on the latest values in the data warehouse.

While we're still an early stage company, the type of integration is super powerful. There are a lot of interesting applications of being able to access failed charges, successful charges, refunds, and the various types of paying customers when joined to other data sources.

Metabase has been a huge help for me in democratizing access to our data. Metabase connects directly to any databases you want, and it allows anyone in the company (I have chosen not to set up advanced permissions yet) to manually inspect data, do advanced reporting, or view dashboards.

This is an example of what it looks like when someone looks at our program applications table (simple database table that keeps track of applications to our growth programs):

Benefits:

• Team members can look at the table as if it were a spreadsheet
• They can apply filters as if it was a spreadsheet
• They can visualize the results more easily than in a spreadsheet

I routinely build reports in metabase that filter to people with a certain condition and send it to teammates. It’s so easy to report on people who work at company X that are in participating in one of our programs. Much better to generate a simple report and then share it with a colleague knowing that it should always be up to date, even if our underlying data is updated.

You can also easily switch between a table view, and many other ways to visualize the data (table, line chart, area chart, bar chart, row chart, scatter chart, pie chart, and a map):

Once you’ve filtered your data set and chosen how to visualize it, it’s easy to then add it to a dashboard of other reports. It’s really nice that you can combine data from multiple databases into the same dashboard, and drag and drop the charts in any configuration you want. This is a dashboard that I setup to monitor the performance of applications submitted to our recent cohort of programs, as well as how people were paying for their spot. It has a segmentation of which programs they’re applying to, how much revenue we’re generated, how we’re comparing to previous periods, and where people are submitting applications from:

There are a ton of other features that I am a huge fan of. Some of them:

• Posting questions to slack at a regular interval:

Or via email:

It has been a huge help for me personally, and this doesn’t even cover all of the ways in which we use it. Best of all, it’s free and open source. We pay to host it ourselves via Amazon Elastic Beanstalk.

This kind of solution comes in incredibly handy in our overall data pipeline, especially when we can point it to a copy of our production database and our analytics data warehouse that is populated by Segment.

• Monitor my engagement metrics by email provider
• Remove non-engaging contacts from our email distribution list
• Monitor my long term retention of cohorts of contacts

These felt like classic behavioral analytic problems in the product space, but email focused. I assumed that somebody was enabling this kind of analysis for the email, space, right?

Nope. I worked at HubSpot for five years, and I have so much respect for that product team. They’re badass, plain and simple (crazy smart, humble, and get stuff done). They built some simple features to answer some of these questions, but don’t provide retention across all of your email campaigns. Does mailchimp offer anything like this? Nope. What about AutoPilot, the company we were using when I joined Reforge? Nope. I did a quick search and I didn’t find any company that provides this type of feature.

One of the core things we teach at Reforge is that retention is king - it makes or breaks your company (acquisition, monetization, payback period, competitive advantage). So I set out to measure it.

It was pretty simple, once I got the pieces working together:

• I turned on the AutoPilot source in Segment, and piped the data to our data warehouse. Luckily we’re not Amazon, so a simple postgres database easily housed the data for this new product.
• I turned on the Sendgrid source in Sendgrid, then spent weeks going back with Segment’s support department figuring out how to properly configure webhooks so email activity data flowed into our data warehouse.
• I wrote a Jupyter notebook that bucketed contacts into their weekly subscriber cohorts and then built retention heatmaps based on the email activity data from both our email marketing system and our transactional emails.
• I ran a script that queried DNS for a domain’s email provider so I could segment the retention curves by email provider (g-suite, microsoft, aol, yahoo, etc).

The outputs looked like this (non-segmented charts):

This helped us to answer key questions like:

• What percentage of our subscriber cohorts were active N weeks after they subscribed?
• Did we have a sticky email newsletter? Did people still around long term?
• Would we be able to sustainably grow our subscriber base over time, if we were able to keep acquisition constant / grow it over time?
• How did our retention curves look by email provider?
• Who were our most prolific consumers (forwarding emails to others, consuming regularly, etc)?
• Who should we be removing from our distribution lists (so that the email providers weren’t hurting our sender and reputation scores)?

It made me ask myself, why don’t email companies provide this kind of functionality? Some thoughts:

• Mailchimp, HubSpot, and companies like it are focused on all of the other aspects of email: helping people design emails, setup automation, and measure individual campaign performance. The bigger problem is not having enough contacts to email in the first place, not having a well designed email, or wanting to analyze a single campaign rather than look at the health of an entire contact database.
• Cohort analysis is not something many people find intuitive, and is a relatively advanced topic. There are still many product teams that don’t measure it, and I expect it’ll come to marketing tools eventually.
• This is a big company problem, and they'll end up writing custom software to solve it for themselves. For everyone else, this isn't a must have.

Is there some easier way to do this? Is there a company that enables this? Let me know, I’d love to use their less-buggy code. I am trying to clean up the code so it’s half respectable and will try to post when I can.

I work at Reforge, and we’re an education company. We offer programs for those located in SF, but we also have an equivalent online-only experience. For a bunch of our key initiatives this year, I felt that I didn’t fully understand what brought people back to our web app after some time away, and I wanted to dive deeper to improve it. Rather than do a one off survey, I setup a campaign that runs continuously to deliver this feedback on a daily basis.

Using Segment to create our list of “alumni” to survey

When I arrived at Reforge we were already using Segment, and we ended up buying their personas add-on. I’m a relatively happy customer (and happy they just raised $175 million), but used it because we were already paying for it. I'm a big fan of using the tools available to you.

Segment has a feature called Audiences that lets you create lists of people. Since most of our important data attributes and events were already flowing through Segment, it was very easy to define small segments or large swathes of our user base through a simple editor. While I live and breathe SQL, sometimes it’s really nice to build it out in a GUI. Here’s what I built:

The nice thing about this is that when someone enters this audience, it means that they’re an alum, they’re not in our most recent cohort, and they have viewed our online material. The cool thing is that you could specify anything about the users that you have available (role, country, seniority, depth of engagement, type of user, organization, etc).

A feature of Audiences is that you know when someone enters the audience. The way I’ve structured the audience, this will mean that they came back to our site and it has been more than 90 days since their last visit - otherwise they’d already been a member. So it’s a cool way to know when someone has come back.

Send the list of people to Zapier

Segment then allows you to send this information to any of their supported destinations. I sent this information to Zapier:

Segment tells Zapier that the user has come back to our site, and Zapier then writes it to a google spreadsheet. This is what our looks like:

What you can see here is that a user from HubSpot named Kieran has revisited how to build a qualitative growth model. I am also using a Segment API to pull in their first name and the title of the last page they visited, in case I want to include that in my outreach asking for feedback. Segment is continuously sending data about people coming back to Reforge, and each time it happens Zapier is writing it to a Google Spreadsheet for me.

Email the people revisiting the material

I then have another Zap that takes the rows from this spreadsheet and emails the person from my personal G-suite account.

You can see what it looks like if I look in my sent folder within gmail (and you can see that people have replied to the email from Gmail's threads):

Via this workflow, I’m automatically emailing people that are coming back to our site asking them what brought them back. I love getting a continual stream of this feedback. Because it’s in Gmail, I can go back and forth with them to clarify what they mean and to dive deeper to understand. If you have a huge user base, you can easily filter down the number of people you email with another step in Zapier (mod their user id by a number to make sure you don’t email too many people at once).

Collect all of the feedback in a Google Doc

At this point I’m automatically emailing everyone from the segment I care about, and I’m able to go back and forth to clarify any questions that I have or dig deeper. Rather than copy and paste their responses into a google doc to share snippets of feedback / soundbites, I hooked up another Zap to automatically pull in their feedback and put it in a google doc.

When the feedback emails come in, I setup a gmail filter to automatically apply a label to the email. I use Zapier to look for new emails under that label, and I exclude any emails from me (my replies to them). Zapier then puts the emails as a spreadsheet row in a google sheet:

Then I used a couple of simple formulas to combine all of the emails from a single user into a single row in another spreadsheet:

I separate the replies with a ---------, so the above row represents multiple emails back and forth with this person. You can see that my first question was about uses cases, and then the third one was about the ones that come up frequently.

Column A is set to be “=UNIQUE(Emails)”. That means that there will only be one row for each email address I have feedback from. The formula for column B is “=ARRAYFORMULA(TEXTJOIN(CHAR(10) & "--------" & CHAR(10), TRUE, IF(Emails=A2,Response,"")))”

Array formulas are really cool, I’ve only had the need to use them a couple of times but I am always so impressed with their functionality. Basically this formula tells Google Sheets to combine all of the emails from users (remember, each row is a single user) together with the “--------” separator.

This is pretty powerful. Now I have a spreadsheet that has the entire conversation with someone in a single spreadsheet row, and I can share that spreadsheet with my entire team. We can then add columns to categorize feedback into buckets for easy filtering / reading.

Why I love this approach:

• I get to read feedback from a critically important segment of users every day. I can define multiple segments to run simultaneously. The only limit is the limit on emails I can send from my Google account (2,000 messages), and the number of emails I have time to respond to.
• I get to follow up with them in my main email tool
• Their feedback then gets pulled into a spreadsheet automatically that I can share with my team members, categorized, and filtered.
• While this isn’t the easiest thing to put together, this is so much easier than it used to be: writing complex SQL by hand, setting up a cron job to run this, writing a custom Gmail script, and then store this information in some database / google sheet). This is so much easier. If you have read this far and are thinking about building this - let me know I’d love to try it out first.

Is there an easier way to accomplish this? Let me know, I'd love to switch to it.

As you monitor these key metrics, you’ll want to understand how and why it’s changing. I am a big fan of how Jonathan Hsu breaks it down in accounting for growth, breaking out the components that add or subtract from your top-level numbers. Understanding the dynamics that contribute to your growth (or lack thereof) are critical for members or your team to understand what drives success. Not just the PM that owns the feature, but many of the team members.

I’ve had conversations with PMs at many companies who may know their top-level numbers, but don’t have a good handle on why they’re going up or down. As Jonathan outlines in his post, two very different businesses can have the same MAU numbers, but one is much better than another.

There are a couple of scenarios I think are important to understand:

• If your retention rates are poor and new signups are helping you grow:
  • When will your growth rate flat line?
  • Could signups go down? What would happen if your growth flat lines or if your user base shrinks?
• If your active user numbers go up:
  • Is that because of a one-time increase in new signups? Do you expect that to continue?
  • Is it because of resurrecting users (making dormant users come back)? Is this a sign of a pattern you could invest in making stronger? Why do people stop using it in the first place?
• If your active user numbers go down:
  • What component decreased? Was it isolated to resurrected users / signups / retaining existing users? If it was isolated, was there a product issue that contributed to it?
• Are you soliciting feedback from each of these buckets of users? There is gold to be mined by segmenting and surveying the users with the most valuable feedback.

In addition to knowing your high-level active user numbers, you should know why you’re growing. It’s helpful to understand what the future looks like if things stay the same or worsen. Even if you know, does the rest of your team know? When everyone understands how your product and business will be successful, they’ll have the context to more effectively make the critical decisions in building tech products.

If you're looking to get started with this type of analysis, Jonathan Hsu's 8-ball analysis or the lifecycle feature in Amplitude is extremely helpful.

This isn’t helpful when you are a brand new startup without a product. At that stage, you should be sitting with your customers or talking to them all the time. You should be taking advantage of your ability to do things that don’t scale.

Once you have more users than you can speak with, behavioral analytics are crucial to being successful. I often have conversations with people that aren’t sure how to apply data enhanced methodologies when launching new features. They bring up a good point:

How are you supposed to leverage behavioral analytics when the feature doesn’t exist?

It’s a good question. At this stage of your lifecycle, your most important task is to speak with customers that may be a good fit for what you want to build. While you may not have people using this new feature yet, there are usually groups of people who could use it within your existing user base. You just need to find the right people to speak with.

These are the types of questions I ask:

• Which of your existing users are most likely to use the new feature?
• What characteristics do those users have?
• What actions are they taking?
• Do you know why they’re taking the actions they are and whether this new feature would be useful to them?

Take advantage of the existing users you have and their patterns of usage. If you’re building a new feature or iterating on an existing one, it’s helpful to understand the high-intensity users or the infrequent users. They are a goldmine of information and you’d be crazy if you didn’t ask them for feedback.

Talk to your existing high intensity users and your infrequent users, they are a goldmine of information. You’d be crazy to ignore their feedback when building something new.

I typically push PMs / researchers / marketers to spend 30 minutes looking at their analytics system to pick out a group of people they want to speak with. I think that anyone involved in building new products is already overloaded with too many tasks, and this can easily feel like unnecessary work or an endless process that will take too much time. I think the key is to timebox this type of analysis and have it point you in the right direction.

I agree with this Intercom post that more than half of a PM’s time should be spent understanding customers’ problems, doing research, and thinking about how design can be applied to those problems. In order to spend that time most effectively, I think behavioral analytics systems are crucial to making sure you maximize that time and understand what your existing users are doing.

It's consistent with anecdotal stories I've heard about Snap (they're serious about privacy and keeping their own information a secret), but I always try to take these stories with a grain of salt.

I immediately thought about how we try to do things differently at HubSpot. Two elements in the HubSpot culture code are using metrics and transparency in the organization. I thought "we're totally different than that here at HubSpot", and I bet many of you thought the same thing when reading the Snapchat article. While I think we strive to be different, we're far from perfect and are constantly trying to improve. Some of the questions I asked myself (and ways I want to hold myself and our teams accountable):

• Does everyone in the organization have access to data (behavioral analytics, data warehouse) that helps them make better decisions?
• For those that aren't technical, is it accessible with non-technical tools?
• Just because they have access to the data, do they leverage it in their ideas, analysis, and proposals?
• Do we have sufficient documentation about how to use the data that's available to all employees?
• Do we make an effort to train people on using the data so they are as self sufficient as possible?
• Do we create a culture of sharing and encouraging others to showcase their findings?
• Do we enable others to reproduce analysis that has been done in the past?

While I like to think we're better than the portrayal of Snapchat in the article, I'm not 100% satisfied with the answers to the questions above.

I’ve become increasingly obsessed with behavioral analytics over the years. Here’s a brief timeline of my experience with them:

• 2013:
  • Join HubSpot, start building a new product with Mixpanel
  • Blown away by the type of analysis it enables. Mind. Blown. It revolutionizes how I think about building products
• 2014:
  • Grow the new product to hundreds of thousands of users, start to get nervous about our Mixpanel bill (this was a huge mistake in hindsight)
• 2014 - 2015:
  • HubSpot decides to build its own internal behavioral analytics system
  • Rationale:
    • HubSpot is a public company at this stage, it’s a competitive advantage to have complete ownership and control over this system
    • If it costs as much as an engineer’s salary, why not pay someone to build a system customized for us?
    • We could solve our own problem, then turn the solution into a solution that could be sold to customers
• 2016:
  • Perform a vendor assessment of our internal tool vs. a vendor (for a variety of reasons, to be explained in a future post)
  • We choose to go with Amplitude as our new behavioral analytics system
• 2017:
  • Finish our migration to Amplitude, we currently have 250-300 HubSpotters using Amplitude on a monthly basis

Why did we pick Amplitude? Some key reasons:

• They allowed us to create charts that count by users or by other arbitrary identifiers. Since HubSpot is a B2B company, we want to track active companies, look at the conversion rates for key actions for all users in a company, and look at company retention. Amplitude had the best solution: it allowed us to change one option in an existing chart to toggle between users and organizations. Other companies could technically solve this, but I thought it was too cumbersome.
• They had an option to store our data in a SQL database (at the time Redshift, now it’s Snowflake). The important piece is that it allows our business intelligence team to ingest the data at a regular interval so it could be combined with other data sources. We use Looker internally, and we want to take behavioral data and combine it with financial data, CRM data, support data, and any other data loaded into our data warehouse.
• They were focused on product analytics. We felt that their roadmap aligned perfectly with our priorities and long-term goals.
• We had a team of 3 engineers and some of a PM’s time devoted to our internal tool. Amplitude has a much bigger engineering team and we didn’t think the customizations we would build were worth it. We felt the product team’s efforts were better spent generating value for the company, not in building a tool that was (at best and probably not the case) slightly better than Amplitude.
• Their dashboard and behavioral cohort features were just what we wanted
• It was fast. Our internal system had been plagued by slowness and outages (we had turnover on the team that built the internal tool and had then understaffed the team)

No solution is a panacea and I won’t say that Amplitude is perfect in every way, but I have been personally very happy with the decision we made. I’m pretty bullish on all of the companies in this space (I think they’re all powerful and worth the money), and unless there’s a fundamental shift in the technology required for these kinds of systems, I don’t want to be involved in building another one from scratch.

When I joined HubSpot I learned from many others about behavioral analytics. Sadly, I find myself constantly fighting responses when I speak with friends in the industry such as:

• “We forgot to add tracking”
• “We want to ship it and see how it does”
• “We don’t have any specific goals for this release other than to improve the design”
• “What should we measure?”
• “We can’t afford to use behavioral analytics, it’s too expensive”

This is how I want to react every time I hear one of those answers:

Just kidding. I am always asking questions to understand the rationale so I can try to help add perspective.

These are the tough questions I want to ask in response:

• What’s more expensive? A behavioral analytics system or shipping the wrong features / wasting the time of your product and engineering team?
• If you hear feedback from a couple of customers, is that representative of all users?
• How do you know that the users are actually doing what they say they’re doing?
• Do you think you’ll get a team’s best work if the only goal is to release their work?
• What do you think will garner more resources in the future? “We improved the experience, just look at it!” vs. “I increased conversion rates of signup to value by 10%, with an expected lift in revenue of Y”.

I don’t think you need to spend weeks off in a corner crunching numbers to come up with the answers to these questions. My suggestion is to spend 30 minutes thinking about a goal, why you’re working on something, and then a simple mechanism to measure success.

I push teams to answer these questions:

1. What represents success for this release/feature?
2. What is the current baseline?
3. What is the hypothetical ceiling of improvement?
4. Given the baseline and ceiling, how much do you think you can improve the metric?
5. What will be the mechanism to track success/failure?
6. When should you evaluate progress?

You don’t have to be super fancy and build Excel models, but at least spend 15-30 minutes thinking through the basics for a new feature. Regardless if you’re building something brand new or iterating on an old feature, I always think it’s worth considering the above questions.

As the saying goes, “if you can’t measure it, you can’t improve it”.

It appears it’s a big part of their recent announcement of how they are able to fund early-stage companies focused exclusively on their metrics.

One of the concepts that struck me was the depth of engagement. It shows you how engaged different portions of your user base are. You don’t need a ton of fancy data science techniques to get a glimpse into what your user base is doing. All you need a fairly straightforward SQL query to get you started.

It starts with a fairly simple concept: how many users are active for 1 day in the past month? How many are active for 2 days in the month? It’s really simple to generate a histogram (this is fake data) that looks like this:

In this fake example there are 100k monthly active users (MAUs) in this hypothetical product. I think this is very telling and interesting from a strategy and operational perspective, but there’s a different view that I now prefer. I prefer to look at this chart on a percentage basis (the % of MAUs), and look at it cumulatively. This is what it looks like:

How to read this chart: 33% of the MAUs are active for a single day of the month. It may be the first day of the month or the last, but the people that fall into this bucket were only active for a single day in the month. 53% of the MAUs were active for 2 or fewer days - you add up the 33k and the 20k from the histogram to get the 53%. In Jonathan’s example there’s a little bit of a spike of users that are active every day of the month - in a bunch of the examples I’ve seen in the B2B space there’s a nice healthy bump around 20 days, which makes sense when you consider that B2B apps are most likely used every business day, rather than every day.

This is a powerful way to slice up your install base very quickly. I push for taking the MAU install base and slicing it up into types of users. Here’s a hypothetical set of groupings:

• Low engaged users (66%): 3 days of activity or less
• Medium engaged users (14%): 4-10 days of activity
• Highly engaged users (10%): 11+ days of activity

There are a bunch of plays that I could see happening for each of these buckets:

• Sales: I could see sales following up with customers that fall into the highly engaged bucket. If they’re free, I could see them seeing value in paid tiers of your product. If they’re already paid customers, they are probably the most likely bucket to see value in additional paid options.
• Services: I could see customer success reaching out to the low engaged bucket to understand why they aren’t using the tool more frequently. In a B2B company where customer success is focused on retention, this is an area of high potential churn.
• Product: I could see the product team looking to build features that address the missing functionality users need to use it more. They could also work on retention hooks that pull users back into the product / get them to see more value in the tool.
• Marketing: I could see the marketing group targeting users based on the bucket they fall into and how they might see value from additional features.

If you're interested in doing this yourself, check out this Jupyter notebook for sample code.

Export the raw data from your analytics system

It should look like the below. The cohorts are in the first column, the size is next, and then the number from the original cohort that retained in each subsequent period.

Create another table that computes the percentage values of the original cohort size

Add a conditional formatting element to show the size of the cohorts over time (conditional formatting -> data bars)

Apply conditional formatting to the table that contains the percentages. It should now look like this:

Adding summary rows

Then I create two new rows at the bottom of the percentage table. The first is the average of each column, and the second is the percentage decrease for each additional week.

Wrap it up

Now you have a nicely formatted retention table that shows you:

1. The size in cohorts over time
2. At a glance how your cohorts retain over time
3. Where there are good pockets and bad pockets of retention
4. The average of cohort retention over time
5. Which weeks have the biggest drops in retention
6. Whether your retention levels off over the long term

Where there are many more graphs you could create off of this data set, I think this sets the right foundation for how to quickly look at the data and answer some important questions.
Are there other ways you visualize this information? Do you recommend doing this another way or formatting it differently? Let me know in the comments below.
Disclaimer: all of the data in these screenshots was made up for this example.

1. What do the cohorts measure?
2. What is the criteria for retention (the % of the cohort that returns)?
3. How can I segment the retention data?

There are many follow up questions you should be asking based on the answer.

Question 1: What do the cohorts measure?

• Is it any People? Companies? Revenue? People using it at a company? Is it a subset of any of those concepts?
• If it’s people — what kind of people? Is it people who did some action? Example: is it people who signed up for a product? Is it people who saw some value from your product? Bought a product? What is the trend of the cohorts over time? What could affect that? How do you expect the numbers to change over time?

Question 2: What is the criteria for retention?

• How is retention measured? If it’s people, what action do they need to take to be counted as having been retained? Why was that chosen? Why is it important?
• If it’s revenue, how is it measured?
• If it’s an attribute of a person or a company (maybe number of people active), how is that measured? Why was that chosen? How is it important to the business?
• Does it level off? If you have a budget to acquire people in the cohorts, how long must they stick around in order to sustain a profitable business?
• How does the data change over time? Are there patterns that you would expect to see, or cliffs where things drop off?
• Do you expect to see patterns horizontally (step function changes from one cohort to another), vertically (time barriers where cohorts worsen or improve), or diagonally (holidays, press spikes, re-engagement campaigns)?

Question 3: How can I segment the retention data?

• Are there pockets in the data that are better or worse than the overall average?
• Is there a feature that results in better retention?
• Is there a platform (ex: Android, iOS) that is better than another one?
• Is there a cohort acquisition source that performs better?
• Is there a geography that performs better in retention?
• Is there a cohort attribute that results in higher retention? Age / sex / profession of people? Size of company? Industry of company?

If you’re doing a retention exercise in an interview, you don’t even really need to look at the data, you could just ask questions the whole time.

As I worked on the HubSpot sales products and we grew it to hundreds of thousands of active users, I quickly realized I couldn’t speak to everyone. I needed to be strategic about who I reached out to, and the type of feedback I was looking for. It’s a dangerous path to go down, because you can end up wasting time by over-analyzing your data and getting into analysis paralysis. Even worse, in my opinion, is taking the existing feedback as representative of your user base, and only solving for the loudest segment of the customers or the customers that are most convenient to interact with.

I want to share the process that Brian Balfour showed me. It’s fairly easy (and inexpensive!) to identify who you should reach out to, and collect their feedback.

Step 1: Identify who you want to speak with

Are you working on activation? If so, you want to target people who sign up but don’t really get started or see value. If you’re working on retention, you need to dig into why someone would start to use it but end up quitting your product. Not sure which one you should be solving for? It depends.

Step 2: Find those people

Any good behavioral analytics system will allow you to look at the number of events that someone has done over time. Let’s say we want to find users who start to use the product, but then stop. Most analytics systems will tell you the raw number of times an event happened, or the unique users that did it. Here’s a sample from the Amplitude demo for when someone plays a song:

The graph above is great for evaluating trends, but the goal is to speak with individual users about their experience. You need to go a level deeper and visualize this data on an individual user level. You should be able to get it by the user’s email. Once you have that, export it to Excel so you can play with it more closely.

For the next images, imagine I exported data that has the number of events someone performed on a daily basis for my app’s most important events. I then would create a pivot table:

This pivot table has a list of email addresses in the first column, alongside the number of times an event happened per day. You could do the same on an hourly / daily / weekly / monthly basis, depending on what makes sense for your app.

Next, I typically do some more filtering to evaluate who I could speak with. I set up this filter to find the people that weren’t active on 6/18, but were active on 6/16 or 6/17. I expect people to use my app every day, and I am shocked when they don’t. I’m curious to know why they didn’t use it.

You could spend a lot of time doing this type of analysis. I’d try to limit yourself to less than thirty minutes of playing with a spreadsheet like the above. I’d bucket your search into a couple of categories:

Super active users:

• They use the crap out of your product, consistently. I’d be curious to speak with them to understand what they like about your product and why it’s so valuable to them.

Drive by users:

• People who check out your application quickly and then leave, never to return. What was their impression? Why didn’t they stick around? If you’re using a product like Fullstory, are you able to see what they did in your product?

Engaged people who quit

• I’d look for people who used it for a minimum of some period (a month? Multiple weeks?) and then stopped using it. These people presumably understand at least a portion of your product, but then took action to stop using it.
• As you can see in the screenshot, I added a column where I computed the number of days someone had performed an action. If done over weeks / months, this is helpful to quickly find the users who were long term users.

Step 3: Reach out to users

This should produce some users that you want to speak with. Now comes the fun part! I took the list of emails, and I send them an email soliciting feedback. This is where I hear a ton of complaints from product managers like “I don’t have enough people to email” or “people never click through to the survey.”

I typically get 10–20% response rates on the emails that I send out. My secret is that I send out the emails from my Gmail account. I BCC lists of users and when I contacted them so I have a record and don’t email people multiple times. This works for multiple reasons:

1. It ends up in their inbox, not the promo inbox
2. It feels more personal, there’s no professional email template
3. One final key: they only have to respond to give you feedback. Nobody wants to click on a link to fill out a survey.

Here’s the email that I send out:

Nobody wants to click on a link to fill out a survey, even if it’s a one question survey. I often get back soliloquies from users with incredibly valuable feedback. I’m grateful, and I also reply back to the emails multiple times doing a typical five whys analysis. This is another reason why it’s superior to embedding a google form into an email.

Step 4: Collect / Analyze the feedback

If you choose to send out this survey to a large group of people, it’ll quickly become difficult to report off the trends and high level information. This is where I use Zapier.

I connect the Gmail and Google Docs zaps like this:

I filter the responses to the survey into a custom label so it doesn’t overload my inbox. I use Zapier to pull the responses into a Google Spreadsheet, so I can easily read them and bucket the responses:

You can see that I have manually gone through and added a feedback bucket for each of the responses. I try to bucket reasons in a handful of categories, and look for common language patterns for how people describe problems. That allows me to create reports for development teams that look like the following:

The value isn’t in doing this with free or low friction tools — the value is in the insight you get into your users and what they love or hated about your product. I’m sure that there are fancy tools that help with this type of analysis, but the bottom line is that you don’t have an excuse for not doing this type of process. It works incredibly fast, produces results, and is free (other than your existing analytics system). I like this because it gets you really close to your users and their behavior, and allows you to quickly get your hands dirty and get some actionable feedback.

How could this process be better? How do you find and the solicit feedback from the users that will have the biggest impact on your company? Let me know through a comment, or drop me an email (it’s on my personal blog).

Thanks to Magdalena Georgieva, Lars Osterberg, and Brian Balfour for reading drafts of this post.