Authordanwolch

Being the closest to the customer

If you read about the strategy of successful tech companies today, it’s all about having “obsessive customer focus” (Jeff Bezos’ 2016 Amazon annual shareholder letter). You’ll hear that “whoever gets closest to the customer wins” (Drift), and that companies want to “solve for the customer” (HubSpot culture code). Ultimately, the question isn’t about whether you’re focused on your customers, but how you go about evaluating who they are, what they’re trying to accomplish, and how they’re interacting with your product. I recently started using a workflow that gives me a continual stream of feedback, allows me to go back and forth to dig deeper and clarify important questions, and then also easily share the results with my team. It also required no effort from our engineering team to setup and didn’t require additional budget.

I work at Reforge, and we’re an education company. We offer programs for those located in SF, but we also have an equivalent online-only experience. For a bunch of our key initiatives this year, I felt that I didn’t fully understand what brought people back to our web app after some time away, and I wanted to dive deeper to improve it. Rather than do a one off survey, I setup a campaign that runs continuously to deliver this feedback on a daily basis.

Using Segment to create our list of “alumni” to survey

When I arrived at Reforge we were already using Segment, and we ended up buying their personas add-on. I’m a relatively happy customer (and happy they just raised $175 million), but used it because we were already paying for it. I’m a big fan of using the tools available to you.

Segment has a feature called Audiences that lets you create lists of people. Since most of our important data attributes and events were already flowing through Segment, it was very easy to define small segments or large swathes of our user base through a simple editor. While I live and breathe SQL, sometimes it’s really nice to build it out in a GUI. Here’s what I built:

The nice thing about this is that when someone enters this audience, it means that they’re an alum, they’re not in our most recent cohort, and they have viewed our online material. The cool thing is that you could specify anything about the users that you have available (role, country, seniority, depth of engagement, type of user, organization, etc).

A feature of Audiences is that you know when someone enters the audience. The way I’ve structured the audience, this will mean that they came back to our site and it has been more than 90 days since their last visit – otherwise they’d already been a member. So it’s a cool way to know when someone has come back.

Send the list of people to Zapier

Segment then allows you to send this information to any of their supported destinations. I sent this information to Zapier:

Segment tells Zapier that the user has come back to our site, and Zapier then writes it to a google spreadsheet. This is what our looks like:

What you can see here is that a user from HubSpot named Kieran has revisited how to build a qualitative growth model. I am also using a Segment API to pull in their first name and the title of the last page they visited, in case I want to include that in my outreach asking for feedback. Segment is continuously sending data about people coming back to Reforge, and each time it happens Zapier is writing it to a Google Spreadsheet for me.

Email the people revisiting the material

I then have another Zap that takes the rows from this spreadsheet and emails the person from my personal G-suite account.

You can see what it looks like if I look in my sent folder within gmail (and you can see that people have replied to the email from Gmail’s threads):

Via this workflow, I’m automatically emailing people that are coming back to our site asking them what brought them back. I love getting a continual stream of this feedback. Because it’s in Gmail, I can go back and forth with them to clarify what they mean and to dive deeper to understand. If you have a huge user base, you can easily filter down the number of people you email with another step in Zapier (mod their user id by a number to make sure you don’t email too many people at once).

Collect all of the feedback in a Google Doc

At this point I’m automatically emailing everyone from the segment I care about, and I’m able to go back and forth to clarify any questions that I have or dig deeper. Rather than copy and paste their responses into a google doc to share snippets of feedback / soundbites, I hooked up another Zap to automatically pull in their feedback and put it in a google doc.

When the feedback emails come in, I setup a gmail filter to automatically apply a label to the email. I use Zapier to look for new emails under that label, and I exclude any emails from me (my replies to them). Zapier then puts the emails as a spreadsheet row in a google sheet:

Then I used a couple of simple formulas to combine all of the emails from a single user into a single row in another spreadsheet:

I seperate the replies with a ———, so the above row represents multiple emails back and forth with this person. You can see that my first question was about uses cases, and then the third one was about the ones that come up frequently.

Column A is set to be “=UNIQUE(Emails)”. That means that there will only be one row for each email address I have feedback from. The formula for column B is “=ARRAYFORMULA(TEXTJOIN(CHAR(10) & “——–” & CHAR(10), TRUE, IF(Emails=A2,Response,””)))”

Array formulas are really cool, I’ve only had the need to use them a couple of times but I am always so impressed with their functionality. Basically this formula tells Google Sheets to combine all of the emails from users (remember, each row is a single user) together with the “——–” separator.

This is pretty powerful. Now I have a spreadsheet that has the entire conversation with someone in a single spreadsheet row, and I can share that spreadsheet with my entire team. We can then add columns to categorize feedback into buckets for easy filtering / reading.

Why I love this approach:

  • I get to read feedback from a critically important segment of users every day. I can define multiple segments to run simultaneously. The only limit is the limit on emails I can send from my Google account (2,000 messages), and the number of emails I have time to respond to.
  • I get to follow up with them in my main email tool
  • Their feedback then gets pulled into a spreadsheet automatically that I can share with my team members, categorized, and filtered.
  • While this isn’t the easiest thing to put together, this is so much easier than it used to be: writing complex SQL by hand, setting up a cron job to run this, writing a custom Gmail script, and then store this information in some database / google sheet). This is so much easier. If you have read this far and are thinking about building this – let me know I’d love to try it out first.

Is there an easier way to accomplish this? Let me know, I’d love to switch to it.

Accounting for Growth

You’d be crazy not to measure your active users, at the appropriate frequency for your app. That may be daily, weekly, monthly, or yearly. (DAU / WAU / MAU).

As you monitor these key metrics, you’ll want to understand how and why it’s changing. I am a big fan of how Jonathan Hsu breaks it down in accounting for growth, breaking out the components that add or subtract from your top-level numbers. Understanding the dynamics that contribute to your growth (or lack thereof) are critical for members or your team to understand what drives success. Not just the PM that owns the feature, but many of the team members.

I’ve had conversations with PMs at many companies who may know their top-level numbers, but don’t have a good handle on why they’re going up or down. As Jonathan outlines in his post, two very different businesses can have the same MAU numbers, but one is much better than another.

There are a couple of scenarios I think are important to understand:

  • If your retention rates are poor and new signups are helping you grow:
    • When will your growth rate flat line?
    • Could signups go down? What would happen if your growth flat lines or if your user base shrinks?
  • If your active user numbers go up:
    • Is that because of a one-time increase in new signups? Do you expect that to continue?
    • Is it because of resurrecting users (making dormant users come back)? Is this a sign of a pattern you could invest in making stronger? Why do people stop using it in the first place?
  • If your active user numbers go down:
    • What component decreased? Was it isolated to resurrected users / signups / retaining existing users? If it was isolated, was there a product issue that contributed to it?
  • Are you soliciting feedback from each of these buckets of users? There is gold to be mined by segmenting and surveying the users with the most valuable feedback.

In addition to knowing your high-level active user numbers, you should know why you’re growing. It’s helpful to understand what the future looks like if things stay the same or worsen. Even if you know, does the rest of your team know? When everyone understands how your product and business will be successful, they’ll have the context to more effectively make the critical decisions in building tech products.

If you’re looking to get started with this type of analysis, Jonathan Hsu’s 8-ball analysis or the lifecycle feature in Amplitude is extremely helpful.

Using analytics before any code is written

I was very lucky to have worked at HubSpot during a pivotal transition in its evolution. I managed a team of Product Analysts and data scientists, and we were charged with improving our product and the overall customer experience. We did this by analyzing what our customers did in our product.

This isn’t helpful when you are a brand new startup without a product. At that stage, you should be sitting with your customers or talking to them all the time. You should be taking advantage of your ability to do things that don’t scale.

Once you have more users than you can speak with, behavioral analytics are crucial to being successful. I often have conversations with people that aren’t sure how to apply data enhanced methodologies when launching new features. They bring up a good point:

How are you supposed to leverage behavioral analytics when the feature doesn’t exist?

It’s a good question. At this stage of your lifecycle, your most important task is to speak with customers that may be a good fit for what you want to build. While you may not have people using this new feature yet, there are usually groups of people who could use it within your existing user base. You just need to find the right people to speak with.

These are the types of questions I ask:

  • Which of your existing users are most likely to use the new feature?
  • What characteristics do those users have?
  • What actions are they taking?
  • Do you know why they’re taking the actions they are and whether this new feature would be useful to them?

Take advantage of the existing users you have and their patterns of usage. If you’re building a new feature or iterating on an existing one, it’s helpful to understand the high-intensity users or the infrequent users. They are a goldmine of information and you’d be crazy if you didn’t ask them for feedback.

Talk to your existing high intensity users and your infrequent users, they are a goldmine of information. You’d be crazy to ignore their feedback when building something new.

I typically push PMs / researchers / marketers to spend 30 minutes looking at their analytics system to pick out a group of people they want to speak with. I think that anyone involved in building new products is already overloaded with too many tasks, and this can easily feel like unnecessary work or an endless process that will take too much time. I think the key is to timebox this type of analysis and have it point you in the right direction.

I agree with this Intercom post that more than half of a PM’s time should be spent understanding customers’ problems, doing research, and thinking about how design can be applied to those problems. In order to spend that time most effectively, I think behavioral analytics systems are crucial to making sure you maximize that time and understand what your existing users are doing.

Snapchat’s Secrecy and DAU Metrics

I was pretty interested to read about Snapchat’s DAU numbers and their culture of secrecy. At first, I was pretty shocked to read about the stories employees told about the lengths the company goes to in order to keep its information private.

It’s consistent with anecdotal stories I’ve heard about Snap (they’re serious about privacy and keeping their own information a secret), but I always try to take these stories with a grain of salt.

I immediately thought about how we try to do things differently at HubSpot. Two elements in the HubSpot culture code are using metrics and transparency in the organization. I thought “we’re totally different than that here at HubSpot”, and I bet many of you thought the same thing when reading the Snapchat article. While I think we strive to be different, we’re far from perfect and are constantly trying to improve. Some of the questions I asked myself (and ways I want to hold myself and our teams accountable):

  • Does everyone in the organization have access to data (behavioral analytics, data warehouse) that helps them make better decisions?
  • For those that aren’t technical, is it accessible with non-technical tools?
  • Just because they have access to the data, do they leverage it in their ideas, analysis, and proposals?
  • Do we have sufficient documentation about how to use the data that’s available to all employees?
  • Do we make an effort to train people on using the data so they are as self sufficient as possible?
  • Do we create a culture of sharing and encouraging others to showcase their findings?
  • Do we enable others to reproduce analysis that has been done in the past?

 

While I like to think we’re better than the portrayal of Snapchat in the article, I’m not 100% satisfied with the answers to the questions above.

Choosing a behavioral analytics system: our journey to Amplitude

As part of my role at HubSpot, I run a team of analysts and data scientists that leverage quantitative analysis to inform our product development and improve the customer experience. It’s our goal to make teams self-sufficient in answering questions like: “How many people are using this feature?” or “What percentage of signups do X?” or “How sticky is this feature?”. In addition, we perform analysis and build models to identify and act upon areas of opportunity. One of the main tools we use on a daily basis is our behavioral analytics system, which helps us understand what our customers are doing inside our product.

I’ve become increasingly obsessed with behavioral analytics over the years. Here’s a brief timeline of my experience with them:

  • 2013:
    • Join HubSpot, start building a new product with Mixpanel
    • Blown away by the type of analysis it enables. Mind. Blown. It revolutionizes how I think about building products
  • 2014:
    • Grow the new product to hundreds of thousands of users, start to get nervous about our Mixpanel bill (this was a huge mistake in hindsight)
  • 2014 – 2015:
    • HubSpot decides to build its own internal behavioral analytics system
    • Rationale:
      • HubSpot is a public company at this stage, it’s a competitive advantage to have complete ownership and control over this system
      • If it costs as much as an engineer’s salary, why not pay someone to build a system customized for us?
      • We could solve our own problem, then turn the solution into a solution that could be sold to customers
  • 2016:
    • Perform a vendor assessment of our internal tool vs. a vendor (for a variety of reasons, to be explained in a future post)
    • We choose to go with Amplitude as our new behavioral analytics system
  • 2017:
    • Finish our migration to Amplitude, we currently have 250-300 HubSpotters using Amplitude on a monthly basis

Why did we pick Amplitude? Some key reasons:

  • They allowed us to create charts that count by users or by other arbitrary identifiers. Since HubSpot is a B2B company, we want to track active companies, look at the conversion rates for key actions for all users in a company, and look at company retention. Amplitude had the best solution: it allowed us to change one option in an existing chart to toggle between users and organizations. Other companies could technically solve this, but I thought it was too cumbersome.
  • They had an option to store our data in a SQL database (at the time Redshift, now it’s Snowflake). The important piece is that it allows our business intelligence team to ingest the data at a regular interval so it could be combined with other data sources. We use Looker internally, and we want to take behavioral data and combine it with financial data, CRM data, support data, and any other data loaded into our data warehouse.
  • They were focused on product analytics. We felt that their roadmap aligned perfectly with our priorities and long-term goals.
  • We had a team of 3 engineers and some of a PM’s time devoted to our internal tool. Amplitude has a much bigger engineering team and we didn’t think the customizations we would build were worth it. We felt the product team’s efforts were better spent generating value for the company, not in building a tool that was (at best and probably not the case) slightly better than Amplitude.
  • Their dashboard and behavioral cohort features were just what we wanted
  • It was fast. Our internal system had been plagued by slowness and outages (we had turnover on the team that built the internal tool and had then understaffed the team)

No solution is a panacea and I won’t say that Amplitude is perfect in every way, but I have been personally very happy with the decision we made. I’m pretty bullish on all of the companies in this space (I think they’re all powerful and worth the money), and unless there’s a fundamental shift in the technology required for these kinds of systems, I don’t want to be involved in building another one from scratch.

© 2019 Dan Wolchonok

Theme by Anders NorénUp ↑