# Monitoring OpenAI news in Python, where entity filters fail

We wanted to know when OpenAI ships a model, loses an executive, or gets sued, without refreshing anyone's timeline. The script that does this is about eighty lines. Getting to those eighty lines took a week of measurement, and most of what we measured contradicted the advice we started with.

**A news watcher is a scheduled query against a news index plus the filtering you write yourself: deduplication, theme routing, and a noise threshold.** The query is the easy part. Every vendor blog stops there. This one doesn't, because the query returned 820 articles in a week and we only wanted about fifteen a day.

The watcher runs four steps in this order, and every one of them turned out to matter:

1.  **Query** one day of coverage from a news index, paging until the index says there's no next page.
    
2.  **Deduplicate** by title token overlap, because the same filing reaches you from a dozen newsrooms.
    
3.  **Route** each article into a theme — launch, people, legal, outage, money — with regex over title and description.
    
4.  **Threshold** on source authority, then deliver what's left.
    

All numbers below come from one collection run against the [APITube](https://apitube.io) news API: English-language articles with "OpenAI" in the title, 30 August to 6 September 2026, 820 articles across 372 sources. Disclosure: we work on APITube. Where it fell over, we say so — most of this article is about it falling over.

## The obvious approach, and why it failed for OpenAI

Every guide to company monitoring says the same thing: don't search for a keyword, filter by the company entity. Keyword search catches "Apple" the fruit; entity resolution catches Apple Inc. It's good advice. It also assumes the entity exists in the vendor's graph.

APITube has an `organization.name` filter. Here is what a week of coverage looks like through it:

| Filter value | Articles, 30 Aug – 6 Sep | What came back |
| --- | --- | --- |
| `organization.name=Microsoft` | 4,123 | Microsoft coverage |
| `organization.name=Anthropic` | 2,080 | Anthropic coverage |
| `organization.name=Nvidia` | 1,770 | Nvidia coverage |
| `organization.name=Apple` | 4,123 | includes *Gooey Apple Cinnamon Rolls* |
| `organization.name=Tesla` | 11 | Tesla coverage, almost none of it |
| `organization.name=Meta` | 0 | nothing |
| `organization.name=OpenAI` | — | `entity organization name 'OpenAI' not found` |
| `organization.name=Alphabet` | — | `entity organization name 'Alphabet' not found` |
| `title=OpenAI` | 828 | OpenAI coverage |

OpenAI isn't in the entity graph at all. Neither is Alphabet. Meta resolves to something with zero articles, Tesla to something with eleven, and Apple returns the same count as Microsoft while serving us a cinnamon roll recipe. We also pulled every organization entity out of 195 OpenAI articles: Hugging Face, SES S.A., Microsoft, Anthropic, `Gruppo API`, and a person called Joseph Altman. Not one mention of OpenAI.

So the rule isn't "use entity filters" or "use keywords". It's **check whether your entity exists before you design around it**, which is one request:

```bash
curl -s "https://api.apitube.io/v1/companies?name=OpenAI" \
  -H "X-API-Key: $APITUBE_API_KEY"
# {"status":"ok","limit":100,"page":1,"has_next_pages":false,"results":[]}
```

Empty array. The autocomplete endpoint agrees — `GET /v1/suggest/entities?prefix=OpenAI` returns *Openair Frauenfeld*, a Swiss music festival. Unlike a keyword search, an entity filter fails silently when the entity is missing: you get zero rows and no error telling you the concept doesn't exist, which means a watcher built on it stays quiet forever and looks like it's working.

For OpenAI, the watcher runs on `title=OpenAI` and does its own filtering downstream.

## What the watcher queries

```bash
curl -s -G "https://api.apitube.io/v1/news/everything" \
  -H "X-API-Key: $APITUBE_API_KEY" \
  -d title=OpenAI \
  -d language.code=en \
  -d published_at.start=2026-09-05 \
  -d published_at.end=2026-09-06 \
  -d sort.by=published_at -d sort.order=desc \
  -d per_page=100
```

One result, trimmed to the fields the watcher touches:

```json
{
  "id": 3079204730,
  "title": "Seattle Times, Newsday Take OpenAI And Microsoft To Court Over Copyright Claims",
  "href": "https://www.timesnownews.com/technology-science/seattle-times-newsday-take-openai-and-microsoft-to-court-over-copyright-claims-article-156045463",
  "published_at": "2026-09-05T04:23:43.000Z",
  "source": { "domain": "timesnownews.com", "rankings": { "opr": 6 } },
  "sentiment": { "overall": { "polarity": "neutral" } },
  "story": { "id": 3079204730 },
  "is_duplicate": false
}
```

Two things bit us here, both worth knowing before you schedule anything.

`published_at.end` **is exclusive.** Asking for `start=2026-09-06&end=2026-09-06` returns zero rows; `end=2026-09-07` returns the day. A daily watcher written the obvious way skips every day it runs and never errors.

**Unknown parameters are sometimes ignored in silence.** `entity.name`, `body` and `keyword` at least come back flagged in `meta.warnings`, so log that array. `search` doesn't: `search=zzqqxwv` returned three cheerfully unrelated articles. Before trusting any filter, send it a nonsense value and check you get nothing back.

## Deduplication is the part nobody writes about

820 articles from 372 sources over seven days, and the same story arrives four or five times. We tried four ways of collapsing them on the same dataset:

| Method | Duplicates caught | Share of 820 |
| --- | --- | --- |
| The API's `is_duplicate` flag | 0 | 0.0% |
| Grouping by `story.id` | 8 | 1.0% |
| Exact match on normalised titles | 106 | 12.9% |
| Token-set overlap ≥ 0.6 | 179 | 21.8% |

![Of 820 articles, the API's own is_duplicate flag caught 0 duplicates and story.id clustering caught 8, while exact title matching caught 106 and token overlap caught 179](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/k03ulbtlwauldegzt2ts.png align="center")

The vendor's own duplicate flag was `false` on all 820 rows. Story clustering caught eight. Whatever your provider promises about deduplication, measure it on your own query before you rely on it — ours cost us one line of Python to check and would have cost us a fifth of our alert volume to skip.

Token overlap is what the watcher uses. Strip punctuation and stopwords, compare titles as sets, drop anything that overlaps an already-kept title by 60% or more:

```python
STOP = set("a an the of in on at to for and or with by from is are was were as "
           "its it this that new says said after over into be will has have "
           "how why what when who".split())

def tokens(title):
    return {w for w in re.sub(r"[^a-z0-9 ]", " ", title.lower()).split()
            if w not in STOP and len(w) > 2}

def dedup(articles):
    kept, seen = [], []
    for a in articles:
        t = tokens(a["title"])
        if any(len(t & s) / len(t | s) >= 0.6 for s in seen if t and s):
            continue
        seen.append(t)
        kept.append(a)
    return kept
```

It catches syndication, not rewriting. On 5 September the copyright suit reached us as four separate alerts — *"Seattle Times, Newsday sue OpenAI, Microsoft"*, *"Seattle Times, Newsday Take OpenAI And Microsoft To Court"*, *"US newspapers sue OpenAI, Microsoft"*, and *"Newsday sues OpenAI and Microsoft"* — because four newsrooms wrote four genuinely different sentences. Dropping the threshold to catch those starts merging unrelated stories. We kept the duplicates and moved on; embeddings would fix it, and would also turn eighty lines into a service.

## Routing into launches, departures and lawsuits

Five regexes over title plus the first 400 characters of the description:

```python
THEMES = {
    "launch": r"(launch\w*|releas\w*|unveil\w*|introduc\w*|roll\w* out|debut\w*|announc\w*|general availability|preview)",
    "people": r"(hir\w+|join\w+|depart\w+|resign\w*|step\w* down|quit\w*|fired|ousted|appoint\w*|new (ceo|cto|cfo|president))",
    "legal":  r"(lawsuit\w*|sue[sd]?\b|suing|court|judge|settl\w+|copyright|antitrust|subpoena|alleg\w+|litigation|injunction|regulat\w+|probe|investigat\w+)",
    "outage": r"(outage\w*|downtime|degrad\w+|incident\w*|status page|disruption\w*|restored|went down)",
    "money":  r"(funding|fundrais\w*|valuation|\bround\b|invest\w+|acqui\w+|\bipo\b|billion (deal|agreement)|stake)",
}
THEMES = {k: re.compile(v, re.I) for k, v in THEMES.items()}

def route(a):
    text = a["title"] + " " + (a.get("description") or "")[:400]
    return [name for name, rx in THEMES.items() if rx.search(text)]
```

Across the week: launch 284 articles (34.6%), legal 191 (23.3%), money 68 (8.3%), outage 62 (7.6%), people 53 (6.5%). 259 articles (31.6%) matched nothing and 90 (11.0%) matched more than one theme.

Reading the description roughly doubles recall and imports someone else's summary into your routing. Of 62 outage matches, 38 matched only in the description — including *"OpenAI Says It Wants to Create a Standard for Revealing AI Alignment Meltdowns"*, which is a policy story that says "incidents" in its second sentence. Match `people` and `legal` across title and description, because a departure is often buried in paragraph three. Match `outage` on the title alone.

That 31.6% residual is not a bug to tune away. It's GPT-6 benchmark scores, congressional candidates expensing subscriptions, and agents editing a German wiki — real coverage that no keyword set anticipated. Send it as a low-priority bucket rather than dropping it.

## How loud should the watcher be

Every article carries `source.rankings.opr`, an authority score from 0 to 8. Pick your threshold by how many alerts a day you'll actually read:

| Threshold | Articles kept | Share | Per day |
| --- | --- | --- | --- |
| `opr >= 3` | 783 | 95.5% | 112 |
| `opr >= 4` | 696 | 84.9% | 99 |
| `opr >= 5` | 531 | 64.8% | 76 |
| `opr >= 6` | 246 | 30.0% | 35 |
| `opr >= 7` | 78 | 9.5% | 11 |

Below 5 you're subscribing to the whole internet — the top ten domains are only 17.2% of volume, so the long tail is where the count lives. At 7 you get eleven a day and you're reading TechCrunch, which you already do. We run at 6.

![820 raw articles fall to 641 after dedup, 177 after the opr filter and 118 after theme routing — from 117 a day down to 17](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/rz4g6w8g4a3zimxqmrla.png align="center")

The table above applies the threshold to raw articles. In the watcher it runs after dedup, so the numbers differ: **820 raw → 641 after dedup → 177 above** `opr >= 6` **→ 118 with a theme.** Seventeen alerts a day, of which the ones we'd have wanted to know about within the hour — the GPT-6 Astra launch, four newsrooms filing suit, the wiki incident — all survived.

Cost, since nobody mentions it: a day of OpenAI coverage is one to two requests at `per_page=100`, so a daily digest is about 500 requests a year and a fifteen-minute loop is roughly 35,000. Both fit inside a free tier on most providers. The expensive version is the one that polls every source separately.

## The whole script

```python
import os, re, sqlite3, datetime as dt
import requests

API = "https://api.apitube.io/v1/news/everything"
KEY = os.environ["APITUBE_API_KEY"]
MIN_OPR = 6

def fetch(day, query="OpenAI"):
    """One day of coverage. published_at.end is exclusive — pass the NEXT day."""
    out, page = [], 1
    while True:
        r = requests.get(API, headers={"X-API-Key": KEY}, timeout=30, params={
            "title": query,
            "language.code": "en",
            "published_at.start": day.isoformat(),
            "published_at.end": (day + dt.timedelta(days=1)).isoformat(),
            "sort.by": "published_at", "sort.order": "desc",
            "per_page": 100, "page": page,
        })
        r.raise_for_status()
        body = r.json()
        for w in (body.get("meta") or {}).get("warnings", []):
            print("API warning:", w["message"])
        out += body.get("results", [])
        if not body.get("has_next_pages"):
            return out
        page += 1

def loud_enough(a):
    return (a["source"].get("rankings", {}).get("opr") or 0) >= MIN_OPR

def main():
    db = sqlite3.connect("seen.db")
    db.execute("CREATE TABLE IF NOT EXISTS seen (id INTEGER PRIMARY KEY)")
    yesterday = dt.date.today() - dt.timedelta(days=1)

    raw = fetch(yesterday)
    fresh = [a for a in raw
             if not db.execute("SELECT 1 FROM seen WHERE id=?", (a["id"],)).fetchone()]
    digest = [a for a in dedup(fresh) if loud_enough(a) and route(a)]

    for a in digest:
        print(f"[{'/'.join(route(a))}] {a['title']}  "
              f"({a['source']['domain']}, opr {a['source']['rankings']['opr']})")

    db.executemany("INSERT OR IGNORE INTO seen VALUES (?)", [(a["id"],) for a in raw])
    db.commit()

if __name__ == "__main__":
    main()
```

The SQLite table exists because reprints trickle in for days and the dedup pass only sees one run at a time. Point `print` at a Telegram `sendMessage` call or an SMTP digest and you're done. Run it on cron at 08:00 for a morning digest, or every fifteen minutes if you want `legal` and `outage` fast — the freshest article in our collection was 53 minutes old when we pulled it, so a quarter-hour loop is not obviously wasteful.

Real output, 5 September:

```plaintext
2026-09-05: 87 articles -> 78 after dedup -> 15 alerts
[outage] OpenAI confirms 'wiki incident,' says it's 'working on a framework'  (techcrunch.com, opr 8)
[launch] OpenAI Launches GPT-6 Astra With Computer Use Tools  (dev.to, opr 7)
[legal] Newsday sues OpenAI and Microsoft over copyright infringement  (newsday.com, opr 6)
[launch/money] Nvidia inks $13 billion deal to buy the AI startup hacked by OpenAI  (egyptindependent.com, opr 6)
```

## Swap the query, keep the pipeline

Change `title=OpenAI` to `organization.name=Anthropic` and everything downstream still works — dedup, routing and thresholds don't care where the rows came from. That's the point of doing the filtering yourself. Just re-run the entity check first, and re-measure the duplicate rate: 21.8% is what OpenAI's coverage looked like, not a constant.

## Questions we had while building this

### How do I monitor OpenAI news automatically?

Monitoring OpenAI news automatically means scheduling a daily query against a news index for articles with OpenAI in the title, then deduplicating, routing into themes, and thresholding on source authority before anything is delivered. The query takes ten lines; the filtering is where the work is. On our dataset that pipeline reduced 820 articles a week to 17 alerts a day.

### Does OpenAI have an RSS feed?

OpenAI publishes an RSS feed at `https://openai.com/news/rss.xml`, free and complete for anything the company says about itself. It will never carry a lawsuit filed against OpenAI, an executive leaving, or an outage the company hasn't acknowledged yet. Subscribe to it *and* run a third-party watcher; they cover different things.

### How do I track a company's news with Python?

Tracking a company's news in Python starts with checking that the entity exists (`/v1/companies?name=...`), falling back to title search if it doesn't, paging through with `per_page=100`, remembering what you've already sent in SQLite, and deduplicating by token overlap. That's roughly eighty lines with `requests` and the standard library.

### How do you filter duplicate news articles?

Duplicate news articles are best filtered by comparing normalised titles as token sets and dropping anything that overlaps an already-kept title by 60% or more. On our dataset that caught 179 of 820 articles, against 106 for exact title matching and 0 for the API's own duplicate flag. Rewritten headlines still slip through; embeddings are the next step.

### What's the best API for AI news?

The best API for AI news is whichever one resolves the entities you actually care about, and that is a single request to find out — don't take a vendor's entity coverage on faith, including ours. We used APITube because it exposes source authority scores and a `count` endpoint, which is what made the thresholds in this article measurable. APITube is one of the APIs we mentioned — there's a free tier at [apitube.io](https://apitube.io).

## Resources

*   [APITube news API docs](https://docs.apitube.io) — endpoints and filters used here
    
*   [OpenAI news RSS feed](https://openai.com/news/rss.xml) — first-party announcements
    
*   [perigon.io on monitoring OpenAI](https://www.perigon.io/blog/monitor-openai-news) — the entity-first playbook this article argues with