← Back
LinkedIn
Resume

Doubling new user activation with Heap's first LLM offering, interactive sign up, improved onboarding

🚧 Warning: This case study is a work in progress 🚧

Design is iterative, right? Well, this is one of the very first iterations of turning ~3 years of work into a compelling story. There is still a lot of context missing, but you can get the jist of what I (and my teams) accomplished by jumping to Goals and results

Before we jump into it, people had some really nice things to say about our partnership during our time at Heap...

"Thanks for being such a thoughtful and empathetic manager, as well as an insanely good designer-I've learned so much from working with you."

Mikel McCavana
Senior Product Designer @ Heap

"A great design leader and a lovely human being."

Sarrah Figueroa
Senior Staff Designer @ Heap

Overview

Company

Heap Analytics is the leader in self-serve product analytics. Most product analytics tools require manually tracking every click, tap, and pageview in code. With Heap, product managers and analysts can manage their own dataset without needing to pull additional resources from engineering.

When I joined, a completely self-serve setup process was possible, but it was technical and challenging. A self-serve purchase process was not possible at all.

My role

During my time at Heap, I was responsible for building our Growth product area from the ground up. I (alongside one of our product directors) defined the strategy that grew Heap's self-serve revenue from $0 to >$1m ARR and nearly doubled our new user activation rate.

I designed the overall vision of what our onboarding and new user experience could look like, and partnered with our product and engineering teams to implement features and systems that moved us toward that vision.

As I (and the team) grew, I moved from my role as a Senior Product Designer into a Product Design Manager role overseeing multiple designers in both our Growth and Data Ecosystem product areas.

Goals and results

Here are just a couple things we accomplished:

  • Nearly doubled our new user activation rate and met our yearly activation goal in 2 quarters
  • Implemented self-serve purchase and upgrade flows for multiple products and plans, and going from $0 to >$1m self-serve ARR
  • Led the creation of A/B testing, feature gating, and rollout processes that were adopted by the entire company
  • Validated our first LLM offering by developing meaningful training data and testing the outputs directly with users

Leading analytics into the future with natural language analysis

LLMs are definitely having a moment right now, and Heap is no exception. However, we didn't want to spend a lot of time and energy unless it was something that could provide real value to our customers. To do that, this project started as a small working group focused on testing both the technology itself as well as the outcomes that could be generated with our users' datasets. The design portion was led by one of the ICs on my team, but we worked closely together at all levels of fidelity.

Discovering where there is opportunity

Even moreso than a typical design project, developing functionality with a new technology means starting from the very beginning. The team was already actively using machine learning for a number of features, and it seemed like LLMs could benefit Heap in some of the same ways.

We identified three potential areas of opportunity:

  1. Data summaries: Starting with the raw data we were already capturing, the LLM could complete tasks such as summarizing the data, starting to interpret it, and presenting relevant segments
  2. Natural language analysis: Rather than requiring users to navigate a sometimes complex set of steps to generate a chart or graph, they could type the question they have, and the LLM would generate the chart
  3. Insight generation: Lots of customers already had 1000s of charts though. Instead of starting from scratch, the LLM could take a chart as an input and turn that into insights about key takeaways, inflection points, and other anomalies in the data

Although initially leaning toward option 1 (summarizing), we discovered that option 2 (natural language analysis) provided the most value for Heap and its customers by getting clearer on our target audience. Non-technical users joining existing accounts are often faced with an overwhelming amount of analysis tools and existing data. Allowing them to get started with a much simplier, chat-based interface could greatly decrease these barriers to success.

Training the model

With our audience in mind, we started to reach out to them to understand the exact questions they were likely to ask. This was possible through a combination of surveying and looking at existing charts and graphs.

Core experience

Although we explored experiences that would sit alongside or being integrated with our manual query creation process, it quickly became clear this would create an even more overwhelming experience than the one we were already trying to solve.

This led to a chat experience that was simplified and independent from the typical chart generation process. The initial empty states, especially, were able to be reduced to only a text input and a few other elements.

Relying on progressive disclosure as they interacted with the tool, we could slowly build out the chart they were looking for as well as introduce them to existing charts.

An imperfect science

With a limited training set, our LLM was bound to make mistakes. This definitely happened, but we were also pleasantly surprised at how often is was right. User research showed that users often put extreme trust in it though, even when the data was unreliable or innaccurate.

This led to us to increase the likelihood of it returning 'unable to generate chart' messages to avoid false positives. To get around that though, we plan to add followup questions that allow the user more direct interaction with their existing data set. This combines the best of both worlds. Easy-to-use, natural language chart generation with powerful data-driven fallbacks.

Entrypoints & education

The independent experience provided more advantages than drawbacks, but it did mean we had to dive deeper into how users would discover the functionality. This work is still ongoing, but we have a few very promising directions.

By including LLM processing with our existing search, it uses a surface that is already regularly use and introduces new functionality without any new interactions from users. Similarly, a simple callout on the typical chart generation page allows an easy shortcut to the natural language flow.

Making SaaS signup fun and educational!

Prefer watching rather than reading or want to try it out yourself? Here's a quick Loom about this project or jump straight to the final experience

Since Heap is such a complex product, we wanted to highlight it's value and competitive advantage before someone even signed up. If we could increase signup conversion by doing that, even better!

Building the experimentation process

A/B testing is the perfect venue for something like this, but we didn't really have the tools or process in place. Partnering with product and engineering, I developed a framework for how we can run and evaluate experiments. Then, we launched this experience as part of a multivariate test. Typically, we'd use a traditional A/B test, but this experience being at the very top of the funnel allowed us to test 4 different approaches.

Over time, the experimentation process we developed as a part of this project was adopted by the whole company, and the tool we used to run it (LaunchDarkly) is now used for feature gating and rollouts.

Showing our competitive advantage instead of telling

With the process in place, it was time to design the actual experience. Rather than try to explain why Heap is great, I wanted to show it. This led to the additon of interactive content that updated as someone filled out the form. The user profile and interactions it displayed mirror the same content someone sees once they're using the product.

This was also an opportunity to demonstrate the personality of Heap. Including celebratory moments when someone finally hit 'Sign up', and acknowledging critical security moments (like adding a password) helped people feel more confident about signing up.

How it worked out

So what was the outcome? While none of the approaches led to a statistically significant conversion rate improvement, we did get multiple pieces of unprompted feedback from prospects about how much they enjoyed the experience and how it helped them understand the product

Now that you have some of the why and how of this signup page, try it out yourself.

Try it out yourself

Doubling activation rate with a holistic approach to onboarding

Research, roadmaps, and rough drafts

In an effort to move from an onboarding and sales process that was often driven by sales and sales enablement, we needed to rebuild our onboarding flow from the ground up. As a data-heavy product it was extremely important we provided guidance every step along the way to ensure users got started successfully.

This was the first step of building out our Growth product area and a detailed testing and experimentation process, but it all began with foundational research and planning conducted by me and one of our product directors.

Initial research with new customers helped us discover why they signed up for Heap, how they intended to use it, and what their initial experience was. From this feedback, we were able to put together what we believed would be an 'ideal' onboarding flow for new accounts. Because Heap often requires multiple users within a single account to be success, we made sure to map out how each of those users overlapped.

Roadmaps meet reality

Even the most well research plan changes once users starting interacting with it. This project was no different. Some pieces of it performed even better than expected, while some we had to scrap completely.

Once someone adds the code snippet to their website or app, there was often a delay before Heap had data coming in. We explored adding a fun, low stakes quiz about how people use data just to ease the wait. However, research showed that users believed this was customizing their in-app experience before they even had an idea how or if they wanted it customized.

On the other hand, experiments that made it easier to involve teammates proved extremely successful! By making it simple to contact the right person, we increased installation rates in spite of adding extra steps to the process. Before adding this feature, non-technical users were faced with multiple technical questions.

After, it was a simple choice between 'I can do this myself' or 'Someone else can do this'.

The final outcome

When planning for our yearly goal, getting to a 35% activation rate seemed ambitious even for an entire year. Our team hit that goal in two quarters! This not only made it significantly easier for customers to get value out of Heap, it also set the stage for our self-serve payments and continued product-led growth.

That's all for now! Jump back to the homepage, check out another project, or send me a message at bryan@zavzen.com

Have a great day! 👋