A Perfect Search Engine
Before exploring other worlds, we should fully understand our own
Before exploring other worlds, we should fully understand our own
WRITTEN BY
PUBLISHED ON
Jan 6, 2025
Isn't it strange that LLMs can solve graduate-level math questions, but today's search engines can't understand a 3 word phrase? For example, if you search "shirts without stripes" on google images, you get exclusively shirts that **do** have stripes.
I started Exa to build a better search engine than Google, one that can actually understand you like an LLM does.
As our company has grown and LLMs have advanced, so have our ambitions. We no longer just want search that's better than Google. We want search that's perfect.
A perfect search engine would help the world more than people realize. And as far as I'm aware, Exa is the only organization on the planet building it.
Let me tell you what we see.
Despite all the hype about AI search in recent years, search engines themselves haven't actually changed.
Google search and Bing search are still very similar to what they were a decade ago. When you type a query like "shirts without stripes", they match the keywords in your query to all the documents on the web. The search fails here because "without stripes" requires understanding that goes beyond keyword matching.
While search algorithms haven't changed, all the AI search hype has come from the introduction of AI summaries. Google AI overviews, SearchGPT, Perplexity -- each of these uses an old-school search engine like Google or Bing under the hood, and then uses an LLM to generate a summary.
LLM summaries are great and save us time. But unfortunately the LLM is still limited by the quality of the underlying search engine. If Bing can't find something, then SearchGPT (which relies on Bing) can't either. And so today's "AI search" is mostly a time saver, it's not actually a better search engine that finds us things we couldn't before.
I remember when I told my dad I was building a new search engine in 2021, he replied, "Isn't Google already good enough?".
It's not good enough. Google's old school search algorithm works great for simple queries -- "Taylor Swift boyfriend" or "Walmart homepage". But it fails miserably the moment a query gets more complex -- "phds in the Bay area who've written about flying cars".
If you ever go to Linkedin to find people, X to find interesting articles, or your friends to find good startups, then you are well aware that Google isn't good enough because you didn't choose Google for that information. But that's ridiculous because these are literally information searches over the web, and search engines in 2025 should be able to handle them correctly.
It's not that Google hasn't *indexed* all this information -- Google has in fact indexed nearly all pages on the web (roughly a trillion pages). It's just that Google's algorithm was fundamentally not designed to handle complex searches over a trillion pages.
But now there exist AI systems that can handle complex requests nearly perfectly. We should expect the same from our search engines. The world deserves a perfect search engine.
A perfect search engine is one that can find you any information you want no matter how complex your request.
It's a database of humanity's collective knowledge, fully organized however you want in real time.
That's hard to imagine because it doesn't yet exist, so here are some cool examples of what perfect search would be like:
Say you have an idea and you want to find similar ideas across the web. It's impossible to do that right now with traditional search engines. For example, I have this idea about flying cars that use roofs to recharge, and I want to find the hundreds of essays, tweets, youtube videos, and experts that discuss this idea and not some tangential idea. Google completely fails here, because this search requires real understanding. If I had a perfect search engine that could match content based on precise meaning, I'd instantly find everything I need.
Perfect search by idea would have huge impact across our lives. Here are some downstream effects:
We humans are constantly seeking out other people for friendship, collaboration, and community. But the best people search our advanced society has built is Linkedin -- ie basic keyword filters over linkedin profiles. The internet is so much richer than that. With perfect search, you should be able to find whoever you want. If I'm an undergrad who's studying AI alignment, I should be able to easily find "undergrads who took time off to study AI alignment and who have a blog". Try searching that on Google. Perfect web search would convert the messy web into a social network more powerful than Meta or X.
Some major downstream effects:
The web contains billions of videos, images, and songs, but because we can't search through them well, most of that value goes unseen. For example, I really like video clips where people make some sort of self-sacrifice for the world. Youtube search is horrendous at these searches. Perfect search would find you any media you want no matter how complex your search.
Some major downstream effects:
Google provides little control over search results. You type in a few keywords and hope it understands you. You never think to type a long sentence explaining exactly what you want because you know it won't work. Perfect search lets you add whatever filters you want, and it always works. If you're looking for people to hire, you'd be able to add modifiers like "who've worked at startups", complex conjunctions like "who know both rust and C++", and negatives like "who didn't study at these schools". The web would feel like a database you can filter arbitrarily however your mind wishes.
Some major downstream effects:
We each walk around with an incomplete understanding of nearly everything. Whether you're looking for companies, blog posts, people, products, papers, etc, Google will give you a small sample of results, but you'll miss everything else that's out there. Neverending FOMO. Perfect search fixes this -- if there are 387 entities in the world who match your criteria, you should be able to find all 387. Not 10, not 389, but 387. Exactly what you asked for. Perfect search would give us all a complete understanding of our world.
Some major downstream effects:
Perfect search is most powerful when paired not with a human, but with an agentic LLM (which is coming in 2025). The human chats with the LLM and the LLM agentically searches in the background, potentially hundreds of times.
As a final example, imagine I have an idea for how flying cars can leverage city infrastructure. I ask my LLM to generate a report on everything I can do to make this work.
The LLM first searches for all similar ideas across the web, in posts, tweets, research papers, news articles, videos, etc. For each idea, the LLM finds the best counterarguments on the web if they exist. The LLM then creates a fully informed step by step plan based on all those ideas. Then for each step of the plan, the LLM finds people in my current city who can be helpful to implement the step, from suppliers to independent researchers. The LLM gathers the contact info of the ones with publicly available info and creates a personalized message for each.
With an LLM + perfect search, it takes a few minutes to gather a comprehensive action plan. With an LLM + traditional search, it would never fully work -- I'd forever only know a small percentage of all the relevant ideas and people out there, and my flying car idea would never take off.
I've only listed a small subset of all the downstream effects of perfect search. Similar to LLMs, it's difficult to predict all the ways perfect search would be used and integrated into our tools.
I'd go as far as to say that the future of humanity depends on getting this right. Because the information we consume has immense power over us. It determines what we know, which determines how we think, which determines how we act.
Right now the world's information is a complete mess. There's a firehose of content pouring onto the world each day, without any sort of organization. Is it any wonder that people's perspectives are all over the place? It's hard to know what is true, what is radical, what is reasonable, who thinks what, why they think it, where did those ideas originate, etc.
Despite all the advances in AI, our society’s main information tools are still Google, social media, and the mainstream media. None of these are capable of nor devoted to providing a comprehensive, unbiased, high quality understanding of the world.
This is a dangerous situation for a nuclear armed society rapidly approaching AGI. We need to fix our information ecosystem, so that we can navigate the next decade as a well-informed species.
Perfect search is by definition a solution to our information ecosystem because it gives everyone full control over the world's information -- our information -- so that it's digestible, legible, useful, actionable.
Perfect search is civilizational infrastructure that needs to exist and no one else is building it. That's why Exa is building it.
So why has no one built a perfect search engine yet? Three reasons -- money, technology, and insanity.
To build perfect search, you need an organization with the right financial incentives. Google makes 200 billion dollars a year from search ads. Perfect search wouldn’t make Google more ad revenue – it could even make them less. Exa makes money from API usage and subscriptions. We are strongly incentivized to improve search until it’s perfect.
To build perfect search, you need to redesign the search algorithm using novel neural methods, not keyword methods. Traditional search engines were designed two decades ago when computers could not think. Neural search engines are more chaotic and unpredictable, but in time they will win over traditional ones. The big players won’t build neural search engine because their infrastructure and products are built around an old paradigm. And for new players it’s quite hard to develop the novel ML architectures and web-scale infrastructure that’s required. At Exa, we have the freedom, experience, and now resources to build search algorithms from scratch the right way.
To build perfect search, you need to be a bit insane. Since we started the company in 2021, pretty much everyone thought what we were doing was insane. “What’s wrong with search?” “Isn’t Google good enough?” You need a group of people that refuse to accept the way things are and work for years until an abstract vision is made real, even if no one else gets it. That’s what OpenAI did with intelligence, and it’s what we intend to do with knowledge.
SEE MORE
Before exploring other worlds, we should fully understand our own
Will Bryk
January 7, 2025
It uses clustering, matryoshka embeddings, binary quantization, and SIMD operations. Written in rust of course 🦀
The Exa Team
December 17, 2024
Exa is pitching a new spin on generative search. It uses the tech behind large language models to return lists of results that it claims are more on point than those from its rivals, including Google and OpenAI.
The Exa Team
December 3, 2024