Aaru

How Aaru uses Exa to evaluate simulations of human behavior

7x
Lower query latency compared to a leading competitor
5x
Reduction in screening cost for verifying data is contamination-free
81%
Preference rate when benchmarked against other providers
Employees
<100
Industry
Software, Startup
Product
Search, Snapshot
“The general pattern with good developer tooling is that it enables engineers to focus on what they do best. We’re glad that Exa has allowed us to concentrate on advancing human simulations rather than search indexes.”
John Kessler
Co-founder & CTO, Aaru
John Kessler

About Aaru

Aaru builds simulations of human behavior that help companies make better decisions. Aaru’s simulations are able to model opinions, sentiment, and purchasing behavior for nearly any population. Aaru works with financial services clients such as EY and RBC as well as consumer insights clients such as Comcast, SharkNinja, and Bayer. Customers use Aaru to make better decisions on products, messaging, and strategy. For simulations to be accurate, the underlying models require immense quantities of data. Aaru uses large-scale proprietary datasets across many markets – foot traffic, transactions, social media, point-of-sale, among others – alongside using web search data. Aaru uses Exa to provide high-quality, accurate web search for its simulations.

Good, fast, or cheap? Pick three

Previously, Aaru used a variety of sources and providers when trying to find a long-term solution for high-quality web search. Each provider came with trade-offs, forcing Aaru to pick two of three: speed, quality, or cost. Furthermore, Aaru spends a large amount of time backtesting simulations to verify accuracy across many environments and subject matters. This lends itself to needing support for “cutting off” data past a certain date to ensure that new information doesn’t leak into a backtested simulation.

Finding solutions for these cutoffs was onerous for Aaru. For instance, Common Crawl is a large-scale web archive that adds ~344 TiB of uncompressed content per month. However, their API is volunteer-led and frequently abused, leading to rate limits that aren’t suitable for scale. Similarly, the Internet Archive wasn’t suitable for high-volume use despite being a fantastic service. In January of 2026, Aaru spent approximately 300 engineer hours building a solution that supported these cutoffs robustly. While a success internally, Aaru also incurred a maintenance burden. Aaru eventually moved over to Exa after learning more about their Snapshot feature.

Finding relevant evidence without looking ahead

Building out accurate simulations is a highly complex research problem. Simulations must be accurate across hundreds of traits and questions, capturing (1) univariate distributions (2) multivariate frequency distributions (3) coherent individual profiles. Because customers make decisions on these simulations, high-quality evaluation infrastructure is paramount. If a simulation were to consume data that was after the release date of a study Aaru wanted to replicate, it would immediately contaminate the rollout, making the output useless for accuracy testing.

Exa’s Snapshot helps solve this problem. Snapshot is designed to support querying a cached index of the web with specific historical captures, matching Aaru’s needs. Aaru gained access as the first user of Snapshot and has since provided feedback to shape the product into what it is today. Snapshot empowers Aaru’s engineers to focus more on their mission – making the most accurate simulation system possible – without worrying about search indexes. Furthermore, Snapshot drove a 5x decrease in LLM spend within the filtering component of the pipeline by removing the need to verify all inputs as contamination-free.

Faster retrieval and reduced screening

Aaru ran a head-to-head evaluation of Exa’s Search API against a leading competitor. Aaru’s criteria for quality was fairly standard – could the API return relevant content quickly while having low-to-no false positives (e.g. irrelevant information)?

“For the queries we care about, Exa was #1 on every axis of our evaluation.”
Shashank Vemuri
Research Engineer
Shashank Vemuri

Across 10,000 tested queries, Exa’s results were preferred over the competitor’s 81% of the time. Since choosing Exa after this initial evaluation, Aaru routinely evaluates other search providers to ensure simulations are using the highest quality search tool on the market. In a recent evaluation, Aaru tested Exa against another alternative web search provider, evaluating on five dimensions: relevance, coverage, content quality, signal density, and recency. In this evaluation, Exa won on all five criteria while returning results with 7x lower latency.

Snapshots in practice

Population Simulation at the Replication Floor

Aaru recently released Population Simulation at the Replication Floor, which they believe is one of the largest open evaluations of a behavioral simulation system to date. Across 2,993 questions, Aaru evaluated how closely AI simulations could replicate the results of real-world studies, the proof customers need before acting on a simulation. Aaru came away with three key results.

First, predicting what people say. Aaru’s errors against survey data are comparable in magnitude to published survey-to-survey disagreement rates, with a sample-adjusted mean absolute error of 2.78 percentage points.

Second, predicting what people do. Aaru compared simulations with observed purchasing behavior in a separate evaluation. Across a set of brand-period observations backed by credit card data, Aaru’s predictions ranked brands closer to the real results than published purchase-consideration scores did, with approximately 66% fewer brand-ranking errors than surveyed purchase consideration.

Last, predicting the accuracy of a simulation without the results. Aaru observed an area under the receiver operating characteristic curve (AUROC) of 0.845 for differentiating TVD levels below 16%, compared with 0.736 for the strongest method reported by a separate evaluation.

Aaru used Exa’s Snapshots product as an input into this study. Because these were recreating studies that had already been published, Aaru needed to be selective around the timeframe. Aaru chose studies that were past the knowledge cutoff of any enterprise-ready language model available during the evaluation to ensure that potential memorization (e.g. from a data analysis agent) didn’t contaminate the results. Aaru used Exa’s Snapshots to provide a time-indexed sample of the web as a data source, facilitating a contamination-free evaluation.

What’s next

Exa is excited to be a partner to Aaru as they continue to scale behavioral simulation capabilities. Aaru is looking forward to working with Exa on larger-and-larger evaluation projects and production deployments.