Dan Wolchonok

Product Guy at Reforge

Show menu Hide menu

CategoryProduct Design

Flying blind: not setting or measuring product metric goals

January 4, 2018 / / 0 Comments

I love building new products. Ever since I was building junky web apps as a geeky high schooler, I always get excited the first time something actually works. It has always felt like magic. Now that I’m older, I increasingly feel the pressure of showing my impact. After the initial euphoria passes, I now immediately measure the metrics that represent success. Something that has been bothering me lately is that regardless of your methodology (waterfall, agile, scrum, burndown, trello anarchy, etc), I never hear others talk enough about product success metrics.

When I joined HubSpot I learned from many others about behavioral analytics. Sadly, I find myself constantly fighting responses when I speak with friends in the industry such as:

  • “We forgot to add tracking”
  • “We want to ship it and see how it does”
  • “We don’t have any specific goals for this release other than to improve the design”
  • “What should we measure?”
  • “We can’t afford to use behavioral analytics, it’s too expensive”

This is how I want to react every time I hear one of those answers:

A guy in a panda suit breaks a computer on someone's desk

Just kidding. I am always asking questions to understand the rationale so I can try to help add perspective.

These are the tough questions I want to ask in response:

  • What’s more expensive? A behavioral analytics system or shipping the wrong features / wasting the time of your product and engineering team?
  • If you hear feedback from a couple of customers, is that representative of all users?
  • How do you know that the users are actually doing what they say they’re doing?
  • Do you think you’ll get a team’s best work if the only goal is to release their work?
  • What do you think will garner more resources in the future? “We improved the experience, just look at it!” vs. “I increased conversion rates of signup to value by 10%, with an expected lift in revenue of Y”.

I don’t think you need to spend weeks off in a corner crunching numbers to come up with the answers to these questions. My suggestion is to spend 30 minutes thinking about a goal, why you’re working on something, and then a simple mechanism to measure success.

I push teams to answer these questions:

  1. What represents success for this release/feature?
  2. What is the current baseline?
  3. What is the hypothetical ceiling of improvement?
  4. Given the baseline and ceiling, how much do you think you can improve the metric?
  5. What will be the mechanism to track success/failure?
  6. When should you evaluate progress?

You don’t have to be super fancy and build Excel models, but at least spend 15-30 minutes thinking through the basics for a new feature. Regardless if you’re building something brand new or iterating on an old feature, I always think it’s worth considering the above questions.

As the saying goes, “if you can’t measure it, you can’t improve it”.

Segment Your User Base: Depth of Engagement

December 30, 2017 / / 7 Comments

If you haven’t read Jonathan Hsu’s 8 part Medium series on Social Capital’s diligence process, add it to your reading list. I didn’t immediately grok all of the concepts in the post, but it has had an incredible impact on how I look at product metrics.

It appears it’s a big part of their recent announcement of how they are able to fund early-stage companies focused exclusively on their metrics.

One of the concepts that struck me was the depth of engagement. It shows you how engaged different portions of your user base are. You don’t need a ton of fancy data science techniques to get a glimpse into what your user base is doing. All you need a fairly straightforward SQL query to get you started.

It starts with a fairly simple concept: how many users are active for 1 day in the past month? How many are active for 2 days in the month? It’s really simple to generate a histogram (this is fake data) that looks like this:

Count of Users by Days Active

In this fake example there are 100k monthly active users (MAUs) in this hypothetical product. I think this is very telling and interesting from a strategy and operational perspective, but there’s a different view that I now prefer. I prefer to look at this chart on a percentage basis (the % of MAUs), and look at it cumulatively. This is what it looks like:

CDF of Monthly Active Users by Days Used

How to read this chart: 33% of the MAUs are active for a single day of the month. It may be the first day of the month or the last, but the people that fall into this bucket were only active for a single day in the month. 53% of the MAUs were active for 2 or fewer days – you add up the 33k and the 20k from the histogram to get the 53%. In Jonathan’s example there’s a little bit of a spike of users that are active every day of the month – in a bunch of the examples I’ve seen in the B2B space there’s a nice healthy bump around 20 days, which makes sense when you consider that B2B apps are most likely used every business day, rather than every day.

This is a powerful way to slice up your install base very quickly. I push for taking the MAU install base and slicing it up into types of users. Here’s a hypothetical set of groupings:

  • Low engaged users (66%): 3 days of activity or less
  • Medium engaged users (14%): 4-10 days of activity
  • Highly engaged users (10%): 11+ days of activity

There are a bunch of plays that I could see happening for each of these buckets:

  • Sales: I could see sales following up with customers that fall into the highly engaged bucket. If they’re free, I could see them seeing value in paid tiers of your product. If they’re already paid customers, they are probably the most likely bucket to see value in additional paid options.
  • Services: I could see customer success reaching out to the low engaged bucket to understand why they aren’t using the tool more frequently. In a B2B company where customer success is focused on retention, this is an area of high potential churn.
  • Product: I could see the product team looking to build features that address the missing functionality users need to use it more. They could also work on retention hooks that pull users back into the product / get them to see more value in the tool.
  • Marketing: I could see the marketing group targeting users based on the bucket they fall into and how they might see value from additional features.

If you’re interested in doing this yourself, check out this Jupyter notebook for sample code.

Facebook and Twitter Onboarding Emails November 2015

December 2, 2015 / / 1 Comment

As part of my work on Sidekick and HubSpot’s sales platform, I focus a lot on the new user experience of our products. As Brian Balfour likes to say “user onboarding is the one element of your application that all users will use”. Can you think of better metrics to invest than getting your users activated and set up for success?

As part of thinking through what will help explain the value of our products to users, I like to evaluate what other successful companies are doing on a regular basis. Recently, I took a look at the emails they send to users as part of the signup process as they move them towards an activation event. I thought it was interesting to see how the emails for Facebook and Twitter stacked up against one another. They have a ton of signups and a lot of opportunity to tune these emails to get the best results. What are they doing that might be applicable for your personas / use case?

First Email:

  • Facebook subject: “Just one more step to get started on Facebook”
  • Twitter subject: “Confirm your account, FirstName LastName”
  • Facebook goes with an aggressive headline with “Action Required”. Grabs people’s attention!  Twitter uses your name in the subject, I’m surprised that Facebook doesn’t do the same.
  • Both Twitter and Facebook want you to “complete” your account in the paragraph above, but both say “confirm” in the CTA. The two sentences in the Facebook feel so robotic.
  • Facebook looks to reinforce its value by describing why it’s useful: “helps you communicate and stay in touch with all of your friends. Once you join Facebook, you’ll be able to share photos, plan events, and more”. Twitter doesn’t do anything like this. I guess it’s hard to describe what twitter is to everyone in one sentence.

Second Email:

  • Facebook subject: “Welcome to Facebook”
  • Twitter subject: “Follow Vogue Magazine, Jimmy Kimmel and Rihanna on Twitter!”
  • Twitter is focused on getting you to follow users, rather than build out your profile. I’d guess that twitter is less about making it so your friends can find you, and more about finding content you’re interested in.
  • Facebook is obsessed about getting you to enter your profile information. They try to hook you with content first, but I assume that profile information is the key to showing you friend suggestions and other information you might like.

Third Email:

  • Facebook subject: “You have more friends on Facebook than you think”
  • Twitter subject: “Eric Shawn tweeted: “Should we accept more #Syrianrefugees? A look at one man’s journey @Foxnews, @CWS_global, @John_Kass, Watch:“
  • Twitter is all about information and news (granted, I picked some accounts to follow in their onboarding process), while Facebook is pushing you to connect your inbox so they can prompt you to add your friends. This is one long email with a lot of tweets embedded in it.
  • It’s weird that Facebook shows so many different email clients, when I signed up with a gmail.com test address. Feels like they haven’t optimized this email, but what do I know?

Fourth Email:

  • Facebook subject: “Robinson Cano and Tom Brady are Trending on Facebook”
  • No twitter email (other than more content to view / follow). Interesting that they don’t prompt you to connect your address book, everybody else does this.
  • I’m surprised that the content of the email isn’t more engaging. I’m surprised they aren’t using images more prominently as twitter is, or showing information as it would appear in your news feed.

Here’s the side by side comparison for Twitter and Facebook’s emails. Did I miss something? Are you impressed with their emails, or underwhelmed?

 

FBvsTwitter

 

 

The Search Button is Dead

March 20, 2011 / / 0 Comments

The search button is dead.  You shouldn’t even have one.

You need a search box – but you don’t need a button for someone to click on.  This might seem like a inconsequential UI change, but it’s an important mindset to integrate into your site design.  It’s called query completion – giving users an indication of what results they’ll see, or steer them to use query suggestions before they hit return.  By thinking of a search box as a central piece of navigation within your site, it helps users find what they’re looking for more quickly.  Rather than having a hierarchical menu of options, let users tell you what they’re looking for and put the appropriate results into categories.  I’ve seen this a lot recently – and I am starting to expect it from every site I visit.  I’ve highlighted some of the sites that I think are doing a great job:

Greplin

I don’t need to review greplin, as I think it’s a great idea.  Here’s a solid write-up of greplin if you haven’t heard of it already.  As you can see in the screenshot, they don’t even have a search button.

Plancast

Another site doing a great job with this paradigm is plancast.  While greplin is a search engine and it makes sense that search is very important in their interface, this next example shows how search can be central to any site.  Plancast is a site that allows you to subscribe to the calendar of someone else.  For example, you could follow a famous designer and go to the same conferences they attend in the hopes of meeting them and speaking with them.  Here’s a journalist’s take on plancast, and I’ve taken a screenshot of a query for brad feld.  In the screenshot, it lists the plans for Brad Feld, as well as techstars information and a meeting he is attending.  An example of excellent execution:

 

The final example is from LinkedIn.  I typed in bro, and they gave me suggested items for a lot of different categories.  They clearly get it as well.  I did a search for bro, and it showed me Peter Bromka, one of my close friends and LinkedIn connections.

Even after you do a search, when you click/unclick certain check boxes you don’t have to hit a ‘submit’ button to make the search effective.  They automatically do the search for you again (with visual feedback the query is being executed).

People think that search is just showing a bunch of links to other pages on their site, while these three examples show how it’s more than that.  How you seen examples of other sites using search effectively?  I’d like to keep a list of sites that get it (other than, you know, the big elephant in the room).

© 2024 Dan Wolchonok

Theme by Anders Norén — Up ↑