Building a News Summarizer

In this example, we will build an LLM-based news summarizer with the Exa API to keep us up-to-date with the latest news on a given topic. We’ll do this in three steps:

Generate search queries for Exa using an LLM
Retrieve relevant URLs and their contents using Exa
Summarize webpage contents using GPT-3.5 Turbo

This is a form of Retrieval Augmented Generation (RAG), combining Exa’s search capabilities with GPT’s summarization abilities. The Jupyter notebook for this tutorial is available on Colab for easy experimentation. You can also check it out on Github, including a plain Python version if you want to skip to the complete product.

Get Started

Pre-requisites and installation

Install the required packages:

pip install exa_py openai

You’ll need both an Exa API key and an OpenAI API key to run this example. You can get your OpenAI API key here.

Get your Exa API key

Set up your API keys:

from google.colab import userdata # comment this out if you're not using Colab

EXA_API_KEY = userdata.get('EXA_API_KEY') # replace with your Exa API key
OPENAI_API_KEY = userdata.get('OPENAI_API_KEY') # replace with your OpenAI API key

Initialize the clients

Import and set up both the OpenAI and Exa clients:

import openai
from exa_py import Exa

openai.api_key = OPENAI_API_KEY
exa = Exa(api_key=EXA_API_KEY)

Generate a search query

First, we’ll use GPT to generate an optimized search query based on the user’s question:

SYSTEM_MESSAGE = "You are a helpful assistant that generates search queries based on user questions. Only generate one search query."
USER_QUESTION = "What's the recent news in physics this week?"

completion = openai.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": SYSTEM_MESSAGE},
        {"role": "user", "content": USER_QUESTION},
    ],
)

search_query = completion.choices[0].message.content

print("Search query:")
print(search_query)

Search for recent articles

Now we’ll use Exa to search for recent articles, filtering by publication date:

from datetime import datetime, timedelta

one_week_ago = (datetime.now() - timedelta(days=7))
date_cutoff = one_week_ago.strftime("%Y-%m-%d")

search_response = exa.search(
    search_query,
    start_published_date=date_cutoff,
    contents={"text": True}
)

urls = [result.url for result in search_response.results]
print("URLs:")
for url in urls:
    print(url)

We use start_published_date to filter for recent content.

Get article contents

Because we requested contents={"text": True}, the article contents are available directly:

results = search_response.results
result_item = results[0]
print(f"{len(results)} items total, printing the first one:")
print(result_item.text)

Unlike traditional search engines that only return URLs, Exa gives us direct access to the webpage contents, eliminating the need for web scraping.

Generate a summary

Finally, we’ll use GPT to create a concise summary of the article:

import textwrap

SYSTEM_MESSAGE = "You are a helpful assistant that briefly summarizes the content of a webpage. Summarize the users input."

completion = openai.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": SYSTEM_MESSAGE},
        {"role": "user", "content": result_item.text},
    ],
)

summary = completion.choices[0].message.content

print(f"Summary for {urls[0]}:")
print(result_item.title)
print(textwrap.fill(summary, 80))

And we’re done! We’ve built an app that translates a question into a search query, uses Exa to search for useful links and their contents, and summarizes the content to effortlessly answer questions about the latest news.Through Exa, we have given our LLM access to the entire Internet. The possibilities are endless.

​Get Started

Get your Exa API key

Get Started