Build LLM pre-training corpora from a 100B-document index
The index behind Exa, delivered as a clean, comprehensive corpus you can train on. Draw from over 2T tracked URLs and the billions of topics about the world, including news, case law, financial markets, research papers, points of interest, code documentation, and more.
We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.