Leading analytics into the future with natural language analysis
LLMs are definitely having a moment right now, and Heap is no exception. However, we didn't want to spend a lot of time and energy unless it was something that could provide real value to our customers. To do that, this project started as a small working group focused on testing both the technology itself as well as the outcomes that could be generated with our users' datasets. The design portion was led by one of the ICs on my team, but we worked closely together at all levels of fidelity.
Discovering where there is opportunity
Even moreso than a typical design project, developing functionality with a new technology means starting from the very beginning. The team was already actively using machine learning for a number of features, and it seemed like LLMs could benefit Heap in some of the same ways.
We identified three potential areas of opportunity:
- Data summaries: Starting with the raw data we were already capturing, the LLM could complete tasks such as summarizing the data, starting to interpret it, and presenting relevant segments
- Natural language analysis: Rather than requiring users to navigate a sometimes complex set of steps to generate a chart or graph, they could type the question they have, and the LLM would generate the chart
- Insight generation: Lots of customers already had 1000s of charts though. Instead of starting from scratch, the LLM could take a chart as an input and turn that into insights about key takeaways, inflection points, and other anomalies in the data
Although initially leaning toward option 1 (summarizing), we discovered that option 2 (natural language analysis) provided the most value for Heap and its customers by getting clearer on our target audience. Non-technical users joining existing accounts are often faced with an overwhelming amount of analysis tools and existing data. Allowing them to get started with a much simplier, chat-based interface could greatly decrease these barriers to success.
Training the model
With our audience in mind, we started to reach out to them to understand the exact questions they were likely to ask. This was possible through a combination of surveying and looking at existing charts and graphs.
Although we explored experiences that would sit alongside or being integrated with our manual query creation process, it quickly became clear this would create an even more overwhelming experience than the one we were already trying to solve.
This led to a chat experience that was simplified and independent from the typical chart generation process. The initial empty states, especially, were able to be reduced to only a text input and a few other elements.
Relying on progressive disclosure as they interacted with the tool, we could slowly build out the chart they were looking for as well as introduce them to existing charts.
An imperfect science
With a limited training set, our LLM was bound to make mistakes. This definitely happened, but we were also pleasantly surprised at how often is was right. User research showed that users often put extreme trust in it though, even when the data was unreliable or innaccurate.
This led to us to increase the likelihood of it returning 'unable to generate chart' messages to avoid false positives. To get around that though, we plan to add followup questions that allow the user more direct interaction with their existing data set. This combines the best of both worlds. Easy-to-use, natural language chart generation with powerful data-driven fallbacks.
Entrypoints & education
The independent experience provided more advantages than drawbacks, but it did mean we had to dive deeper into how users would discover the functionality. This work is still ongoing, but we have a few very promising directions.
By including LLM processing with our existing search, it uses a surface that is already regularly use and introduces new functionality without any new interactions from users. Similarly, a simple callout on the typical chart generation page allows an easy shortcut to the natural language flow.