<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[APITube News API]]></title><description><![CDATA[Measured experiments with news data: what a filter actually drops, what a byline field actually contains, what an API actually returns. Code and numbers include]]></description><link>https://apitube.hashnode.dev</link><image><url>https://cdn.hashnode.com/res/hashnode/image/upload/v1593680282896/kNC7E8IR4.png</url><title>APITube News API</title><link>https://apitube.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Sun, 20 Sep 2026 23:57:47 GMT</lastBuildDate><atom:link href="https://apitube.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[A week of NVIDIA news is 5,718 articles. My filter kept 161]]></title><description><![CDATA[I wanted a small script that pings me when something real happens to NVIDIA — an earnings surprise, an export-control change, a leadership move. Not a feed. A handful of lines a day I can actually rea]]></description><link>https://apitube.hashnode.dev/monitor-nvidia-news-python-watcher</link><guid isPermaLink="true">https://apitube.hashnode.dev/monitor-nvidia-news-python-watcher</guid><category><![CDATA[Python]]></category><category><![CDATA[api]]></category><category><![CDATA[finance]]></category><category><![CDATA[Tutorial]]></category><dc:creator><![CDATA[APITube News API]]></dc:creator><pubDate>Wed, 16 Sep 2026 04:57:30 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a9fa4994ac02a30c9ee5810/1462b758-f987-4b87-a31f-20d7b08115f2.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I wanted a small script that pings me when something real happens to NVIDIA — an earnings surprise, an export-control change, a leadership move. Not a feed. A handful of lines a day I can actually read.</p>
<p>The first version was fifteen lines and it was wrong in a way I couldn't see from the output. So I pulled a full week of coverage three different ways and counted.</p>
<p><strong>A company news watcher is a filter chain, not a query.</strong> Over seven days (2026-08-30 to 2026-09-05) there were <strong>5,718 articles mentioning NVIDIA</strong> — about 817 a day. Five filtering stages got that to <strong>161 alerts, or 23 a day</strong>. This post is the measurements behind each stage, and the finished script.</p>
<p>This is for developers who want a company news feed they can trust, and analysts who need to know what their alerts are silently dropping.</p>
<p><strong>What the week showed, in four lines:</strong></p>
<ul>
<li>Searching headlines for the company name finds <strong>29.8%</strong> of the coverage that mentions it.</li>
<li>Entity resolution alone isn't a fix — it missed <strong>180 articles</strong> that had "nvidia" in the headline.</li>
<li>Only <strong>31.6%</strong> of the coverage is in English.</li>
<li><strong>12.5%</strong> of what survives filtering is the same story republished under a near-identical headline.</li>
</ul>
<p><em>Measured 2026-09-06 against the APITube news API, over articles published 2026-08-30 to 2026-09-05. The pull script (<code>pull.py</code>) and the resulting counts (<code>data.csv</code>) sit alongside this post.</em></p>
<h2>The fifteen-line version</h2>
<p>Start with the obvious thing: search headlines for the company name.</p>
<pre><code class="language-bash">curl -s -G "https://api.apitube.io/v1/news/everything" \
  -H "X-API-Key: $APITUBE_API_KEY" \
  --data-urlencode "title=nvidia" \
  --data-urlencode "per_page=2" \
  --data-urlencode "published_at.start=2026-08-30" \
  --data-urlencode "published_at.end=2026-09-05"
</code></pre>
<p>Trimmed to the fields that matter, one article comes back like this:</p>
<pre><code class="language-json">{
  "id": 3078309425,
  "title": "Nvidia earnings highlight remarkable growth",
  "published_at": "2026-09-02T17:56:16.000Z",
  "language": "en",
  "source": {
    "domain": "proactiveadvisormagazine.com",
    "type": "news"
  },
  "entities": [
    {
      "id": 1280220,
      "name": "Nvidia",
      "type": "organization",
      "frequency": 3,
      "sentiment": { "score": 0.46, "polarity": "positive" },
      "metadata": {
        "aliases": ["NVIDIA", "nVidia", "NVDA"],
        "is_public_entity": true
      }
    }
  ],
  "sentiment": {
    "overall": { "score": 0.64, "polarity": "positive" }
  },
  "is_duplicate": false,
  "story": { "id": 3078309425 }
}
</code></pre>
<p>Two fields decide everything later: <code>entities[].frequency</code> (how many times the company is actually named in the piece) and <code>entities[].id</code> (the canonical company, independent of spelling).</p>
<h2>The headline query finds 30% of the coverage</h2>
<p>Here is the part that made me throw away version one. I ran the same week twice — once filtering on the headline, once on the canonical entity id.</p>
<table>
<thead>
<tr>
<th>Strategy</th>
<th>Unique articles, 7 days</th>
<th>Share of entity-matched coverage</th>
</tr>
</thead>
<tbody><tr>
<td><code>title=nvidia</code></td>
<td>1,831</td>
<td>29.8%</td>
</tr>
<tr>
<td><code>entity.id=1280220</code></td>
<td>5,538</td>
<td>100%</td>
</tr>
<tr>
<td>union of both</td>
<td>5,718</td>
<td>—</td>
</tr>
</tbody></table>
<p>The headline query misses <strong>3,890 articles</strong> — a supplier's export licence, a hyperscaler's capex call, a competitor's benchmark. Those are the stories that move a position, and they rarely put "NVIDIA" in the headline.</p>
<p>The obvious fix is to drop the headline query and use entity resolution alone. That is also wrong. <strong>180 articles had "nvidia" in the headline and no NVIDIA entity extracted at all</strong> — 9.8% of the headline set. Named-entity recognition misses things too, especially on smaller and non-English outlets.</p>
<p>Neither filter is a superset of the other, so the watcher queries both and unions the results. Unlike a single-strategy filter, the union costs one extra request per window, which means you stop choosing between the two failure modes and just pay for both.</p>
<h2>Two thirds of it isn't in English</h2>
<p><img src="https://cdn.hashnode.com/uploads/covers/6a9fa4994ac02a30c9ee5810/9fbf285d-029a-4f5d-bb84-1de57d951627.png" alt="Only 31.6% of NVIDIA coverage is English; German 9.4%, Spanish 7.4%, Chinese 6.4%, French 5.7%" /></p>
<p><em>Language of all 5,718 articles mentioning NVIDIA over the seven days, from the <code>language</code> field on each article.</em></p>
<p>Of those 5,718 articles, <strong>1,806 were in English — 31.6%</strong>. German was 9.4%, Spanish 7.4%, Chinese 6.4%, French 5.7%, Italian 4.9%, Japanese 4.2%, Portuguese 4.1%.</p>
<p>If you never set a language filter, you don't get a broader watcher — you get a mostly-unreadable one, and your theme keywords (written in English) silently fail against it. If you do set <code>language=en</code>, be honest that you just dropped two thirds of the world's coverage. I set it, because I can't read the rest. That's a choice, not a default.</p>
<h2>The five stages, with the numbers</h2>
<p><img src="https://cdn.hashnode.com/uploads/covers/6a9fa4994ac02a30c9ee5810/bb960f3b-8c10-4468-ba99-d42ef3cc2e99.png" alt="Five filter stages cut 5,718 articles a week to 161: English 1,806, headline plus 2+ mentions 679, dedup 594, theme match 161" /></p>
<p><em>Each stage applied in order to the same 7-day pull. Counts are articles, not alerts sent.</em></p>
<p>Each stage below is one rule with one threshold, applied in order.</p>
<table>
<thead>
<tr>
<th>Stage</th>
<th>Rule</th>
<th>Articles / 7 days</th>
<th>Per day</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Everything mentioning NVIDIA (union of both queries)</td>
<td>5,718</td>
<td>817</td>
</tr>
<tr>
<td>2</td>
<td><code>language == "en"</code></td>
<td>1,806</td>
<td>258</td>
</tr>
<tr>
<td>3</td>
<td>NVIDIA in the headline <strong>and</strong> <code>frequency &gt;= 2</code></td>
<td>679</td>
<td>97</td>
</tr>
<tr>
<td>4</td>
<td>Near-duplicates removed</td>
<td>594</td>
<td>85</td>
</tr>
<tr>
<td>5</td>
<td>Matched a watch theme</td>
<td>161</td>
<td>23</td>
</tr>
</tbody></table>
<p>Stage 3 is the one worth explaining. Only <strong>32.2% of entity-matched articles have NVIDIA in the headline</strong>; the other two thirds mention it in passing — a monitor review, a fund's holdings list, an unrelated AI story name-checking the chip. Requiring the headline <em>and</em> at least two mentions in the body is a cheap proxy for "this article is about NVIDIA", and it cut 258/day to 97/day.</p>
<p>Stage 4 removed 12.5% of what survived stage 3. Those 924 source domains republish each other constantly — one acquisition story appeared eight times under near-identical headlines. Normalising the title to a sorted set of its long words catches these without any similarity library.</p>
<p>Stage 5 splits the remainder into themes, which is also the alert budget: earnings 11.4/day, leadership 10.0/day, legal 2.6/day, export controls 2.1/day. Those sum to more than 23 because one article can match two themes — an earnings call that announces a CFO change lands in both.</p>
<h2>The watcher</h2>
<pre><code class="language-python">import json, os, re, urllib.parse, urllib.request

API = "https://api.apitube.io/v1/news/everything"
KEY = os.environ["APITUBE_API_KEY"]
NVIDIA = 1280220

THEMES = {
    "earnings": (
        r"earnings|revenue|quarter|guidance|\beps\b"
        r"|results|forecast"
    ),
    "leadership": (
        r"\bceo\b|jensen huang|\bcfo\b|resign"
        r"|steps down|appoint"
    ),
    "export": (
        r"export|sanction|china|\bban\b|restrict"
        r"|licen[cs]e|blackwell|tariff"
    ),
    "legal": (
        r"lawsuit|sue[sd]?\b|antitrust|probe"
        r"|investigat|settle|court"
    ),
}
THEME_RE = re.compile("|".join(THEMES.values()), re.I)

def fetch(**params):
    params.setdefault("per_page", 200)
    url = f"{API}?{urllib.parse.urlencode(params)}"
    req = urllib.request.Request(url, headers={
        "X-API-Key": KEY,
        # urllib's default UA gets a 403
        "User-Agent": "nvidia-watch/1.0",
    })
    resp = urllib.request.urlopen(req, timeout=60)
    payload = json.load(resp)
    # a typo'd filter is ignored, not rejected
    for w in payload.get("meta", {}).get("warnings", []):
        print("WARNING", w["code"], w["message"])
    return payload

def mentions(article, entity_id):
    for e in article.get("entities", []):
        if e["id"] == entity_id:
            return e.get("frequency", 0)
    return 0

def normalise(title):
    text = re.sub(r"[^a-z0-9 ]", " ", (title or "").lower())
    words = {w for w in text.split() if len(w) &gt; 3}
    return " ".join(sorted(words))

def watch(entity_id, keyword, start, end, language="en"):
    seen, out = set(), []
    # stage 1: union of both queries
    for field, value in (("entity.id", entity_id),
                         ("title", keyword)):
        page = 1
        while True:
            q = {"published_at.start": start,
                 "published_at.end": end,
                 "page": page, field: value}
            data = fetch(**q)
            rows = data.get("results", [])
            for a in rows:
                # stage 2: language
                if a.get("language") != language:
                    continue
                # stage 3: headline and 2+ mentions
                title = a.get("title") or ""
                if not re.search(keyword, title, re.I):
                    continue
                if mentions(a, entity_id) &lt; 2:
                    continue
                # stage 4: near-duplicate headline
                key = normalise(title)
                if key in seen:
                    continue
                # stage 5: matches a watch theme
                blob = f"{title} {a.get('description', '')}"
                if not THEME_RE.search(blob):
                    continue
                seen.add(key)
                out.append(a)
            if not rows or not data.get("has_next_pages"):
                break
            page += 1
    return out

if __name__ == "__main__":
    hits = watch(NVIDIA, "nvidia", "2026-08-30", "2026-09-05")
    n = len(hits)
    print(f"\n{n} alerts over 7 days ({n / 7:.1f}/day)\n")
    for a in sorted(hits, key=lambda x: x["published_at"]):
        blob = f"{a['title']} {a.get('description', '')}"
        theme = next(t for t, p in THEMES.items()
                     if re.search(p, blob, re.I))
        day = a["published_at"][:10]
        print(f"[{theme:10s}] {day}  {a['title'][:70]}")
</code></pre>
<p>Standard library only. Running it just now returned 153 rather than the 161 in my archived pull — the index keeps ingesting, so a re-run of a past window is close but not byte-identical. Worth knowing before you write a test that asserts an exact count.</p>
<p>Piping it into Telegram is four more lines and a bot token; the filtering is the part that was hard.</p>
<h2>Five things that cost me time</h2>
<p><strong>There is no ticker filter.</strong> <code>ticker=NVDA</code> returns HTTP 200 and quietly ignores you. NVDA lives in <code>entities[].metadata.aliases</code> alongside <code>is_public_entity: true</code>, so resolve ticker → entity id once, store the map, and filter on the id.</p>
<p><strong><code>organization.name</code> is case-sensitive.</strong> <code>organization.name=Nvidia</code> works. <code>organization.name=NVIDIA</code> returns HTTP 400, <code>ER0220</code>, "entity organization name 'NVIDIA' not found." The all-caps spelling most people type is the one that fails.</p>
<p><strong>Unknown parameters are warnings, not errors.</strong> <code>entity.name</code> and <code>ticker</code> both come back 200 with <code>meta.warnings[].code == "ER0368"</code> and <em>unfiltered</em> results. A typo doesn't crash your watcher, it floods it. Print <code>meta.warnings</code> — that's the two lines in <code>fetch()</code> above.</p>
<p><strong>Headline search defaults to a 31-day window</strong> (<code>ER0366</code>) if you don't pass <code>published_at.start</code> and <code>published_at.end</code>. Your "last 24 hours" alert is quietly a month.</p>
<p><strong>Python's default User-Agent gets a 403.</strong> <code>urllib</code> sends <code>Python-urllib/3.x</code> and the edge rejects it. Set any real User-Agent and it works — this cost me twenty minutes of blaming my API key.</p>
<h2>Frequently asked questions</h2>
<h3>How do I get news alerts for a specific company?</h3>
<p>The reliable way to get news alerts for a specific company is a five-stage filter chain rather than a single query, because no single filter is both precise and complete. Query the company's canonical entity id, union it with a headline keyword search, then filter by language, require the company in the headline with two or more body mentions, drop near-duplicate titles, and match against theme keywords. In this measurement that chain reduced 817 articles a day to 23.</p>
<h3>Why do my company news alerts return irrelevant articles?</h3>
<p>Company news alerts return irrelevant articles because a mention is not a subject. Only 32.2% of articles that mention NVIDIA put it in the headline; the rest are passing references in unrelated stories. Requiring a headline match plus <code>frequency &gt;= 2</code> removes most of them.</p>
<h3>Can I filter news by ticker symbol?</h3>
<p>No — this API has no <code>ticker</code> parameter, and passing one is silently ignored rather than rejected. The ticker appears inside <code>entities[].metadata.aliases</code>, so build your own ticker-to-entity-id table once and filter on <code>entity.id</code>.</p>
<h3>How do you deduplicate news articles from multiple sources?</h3>
<p>The cheapest way to deduplicate news articles across sources is normalised-headline matching, because syndicated copies keep the same words while rewording punctuation and framing. Normalise each headline to a sorted set of its words longer than three characters and treat a repeat as a duplicate. Across 924 source domains that removed 12.5% of the surviving articles, with no similarity library.</p>
<h3>Is entity resolution enough on its own?</h3>
<p>No — entity resolution alone is not enough, because entity extraction has its own recall gap. It missed 180 articles that had the company name in the headline — 9.8% of the headline-matched set. Query both ways and union the results.</p>
<h2>Making it yours</h2>
<p>To point this at another company:</p>
<ol>
<li><strong>Find the entity id.</strong> Pull any article that mentions the company and read <code>entities[]</code> — the id, the aliases and <code>is_public_entity</code> are all in there. Store the mapping; it doesn't change.</li>
<li><strong>Rewrite <code>THEMES</code>.</strong> The stage thresholds transfer between companies, the theme regexes don't — "export controls" matters for a chipmaker and means nothing for a bank.</li>
<li><strong>Re-measure the funnel.</strong> Count what each stage drops for <em>your</em> company before you trust the output. A quiet mid-cap won't need stage 5 at all; a bank will need a different stage 3.</li>
</ol>
<p>Mine only looked reasonable after I counted what it threw away.</p>
<p>Disclosure: I work on APITube, which is the API in the code above — free tier at <a href="https://apitube.io">apitube.io</a>. The measurement approach works against any news API that exposes entity ids and per-article mention counts.</p>
<h2>Resources</h2>
<ul>
<li><a href="https://docs.apitube.io">APITube news API documentation</a> — endpoints, parameters, response schema</li>
<li><a href="https://www.google.com/alerts">Google Alerts</a> — the no-code baseline: no API, no deduplication, no entity resolution</li>
<li><a href="https://perigon.io/blog/monitor-nvidia-news">Perigon's NVIDIA monitoring guide</a> — good on entity resolution as a concept, no code or numbers</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Monitoring OpenAI news in Python, where entity filters fail]]></title><description><![CDATA[We wanted to know when OpenAI ships a model, loses an executive, or gets sued, without refreshing anyone's timeline. The script that does this is about eighty lines. Getting to those eighty lines took]]></description><link>https://apitube.hashnode.dev/monitoring-openai-news-in-python</link><guid isPermaLink="true">https://apitube.hashnode.dev/monitoring-openai-news-in-python</guid><category><![CDATA[Python]]></category><category><![CDATA[api]]></category><category><![CDATA[AI]]></category><category><![CDATA[Tutorial]]></category><category><![CDATA[automation]]></category><dc:creator><![CDATA[APITube News API]]></dc:creator><pubDate>Tue, 08 Sep 2026 13:05:00 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a9fa4994ac02a30c9ee5810/13b3825d-8481-4db3-8767-0e832c3f6e15.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>We wanted to know when OpenAI ships a model, loses an executive, or gets sued, without refreshing anyone's timeline. The script that does this is about eighty lines. Getting to those eighty lines took a week of measurement, and most of what we measured contradicted the advice we started with.</p>
<p><strong>A news watcher is a scheduled query against a news index plus the filtering you write yourself: deduplication, theme routing, and a noise threshold.</strong> The query is the easy part. Every vendor blog stops there. This one doesn't, because the query returned 820 articles in a week and we only wanted about fifteen a day.</p>
<p>The watcher runs four steps in this order, and every one of them turned out to matter:</p>
<ol>
<li><p><strong>Query</strong> one day of coverage from a news index, paging until the index says there's no next page.</p>
</li>
<li><p><strong>Deduplicate</strong> by title token overlap, because the same filing reaches you from a dozen newsrooms.</p>
</li>
<li><p><strong>Route</strong> each article into a theme — launch, people, legal, outage, money — with regex over title and description.</p>
</li>
<li><p><strong>Threshold</strong> on source authority, then deliver what's left.</p>
</li>
</ol>
<p>All numbers below come from one collection run against the <a href="https://apitube.io">APITube</a> news API: English-language articles with "OpenAI" in the title, 30 August to 6 September 2026, 820 articles across 372 sources. Disclosure: we work on APITube. Where it fell over, we say so — most of this article is about it falling over.</p>
<h2>The obvious approach, and why it failed for OpenAI</h2>
<p>Every guide to company monitoring says the same thing: don't search for a keyword, filter by the company entity. Keyword search catches "Apple" the fruit; entity resolution catches Apple Inc. It's good advice. It also assumes the entity exists in the vendor's graph.</p>
<p>APITube has an <code>organization.name</code> filter. Here is what a week of coverage looks like through it:</p>
<table>
<thead>
<tr>
<th>Filter value</th>
<th>Articles, 30 Aug – 6 Sep</th>
<th>What came back</th>
</tr>
</thead>
<tbody><tr>
<td><code>organization.name=Microsoft</code></td>
<td>4,123</td>
<td>Microsoft coverage</td>
</tr>
<tr>
<td><code>organization.name=Anthropic</code></td>
<td>2,080</td>
<td>Anthropic coverage</td>
</tr>
<tr>
<td><code>organization.name=Nvidia</code></td>
<td>1,770</td>
<td>Nvidia coverage</td>
</tr>
<tr>
<td><code>organization.name=Apple</code></td>
<td>4,123</td>
<td>includes <em>Gooey Apple Cinnamon Rolls</em></td>
</tr>
<tr>
<td><code>organization.name=Tesla</code></td>
<td>11</td>
<td>Tesla coverage, almost none of it</td>
</tr>
<tr>
<td><code>organization.name=Meta</code></td>
<td>0</td>
<td>nothing</td>
</tr>
<tr>
<td><code>organization.name=OpenAI</code></td>
<td>—</td>
<td><code>entity organization name 'OpenAI' not found</code></td>
</tr>
<tr>
<td><code>organization.name=Alphabet</code></td>
<td>—</td>
<td><code>entity organization name 'Alphabet' not found</code></td>
</tr>
<tr>
<td><code>title=OpenAI</code></td>
<td>828</td>
<td>OpenAI coverage</td>
</tr>
</tbody></table>
<p>OpenAI isn't in the entity graph at all. Neither is Alphabet. Meta resolves to something with zero articles, Tesla to something with eleven, and Apple returns the same count as Microsoft while serving us a cinnamon roll recipe. We also pulled every organization entity out of 195 OpenAI articles: Hugging Face, SES S.A., Microsoft, Anthropic, <code>Gruppo API</code>, and a person called Joseph Altman. Not one mention of OpenAI.</p>
<p>So the rule isn't "use entity filters" or "use keywords". It's <strong>check whether your entity exists before you design around it</strong>, which is one request:</p>
<pre><code class="language-bash">curl -s "https://api.apitube.io/v1/companies?name=OpenAI" \
  -H "X-API-Key: $APITUBE_API_KEY"
# {"status":"ok","limit":100,"page":1,"has_next_pages":false,"results":[]}
</code></pre>
<p>Empty array. The autocomplete endpoint agrees — <code>GET /v1/suggest/entities?prefix=OpenAI</code> returns <em>Openair Frauenfeld</em>, a Swiss music festival. Unlike a keyword search, an entity filter fails silently when the entity is missing: you get zero rows and no error telling you the concept doesn't exist, which means a watcher built on it stays quiet forever and looks like it's working.</p>
<p>For OpenAI, the watcher runs on <code>title=OpenAI</code> and does its own filtering downstream.</p>
<h2>What the watcher queries</h2>
<pre><code class="language-bash">curl -s -G "https://api.apitube.io/v1/news/everything" \
  -H "X-API-Key: $APITUBE_API_KEY" \
  -d title=OpenAI \
  -d language.code=en \
  -d published_at.start=2026-09-05 \
  -d published_at.end=2026-09-06 \
  -d sort.by=published_at -d sort.order=desc \
  -d per_page=100
</code></pre>
<p>One result, trimmed to the fields the watcher touches:</p>
<pre><code class="language-json">{
  "id": 3079204730,
  "title": "Seattle Times, Newsday Take OpenAI And Microsoft To Court Over Copyright Claims",
  "href": "https://www.timesnownews.com/technology-science/seattle-times-newsday-take-openai-and-microsoft-to-court-over-copyright-claims-article-156045463",
  "published_at": "2026-09-05T04:23:43.000Z",
  "source": { "domain": "timesnownews.com", "rankings": { "opr": 6 } },
  "sentiment": { "overall": { "polarity": "neutral" } },
  "story": { "id": 3079204730 },
  "is_duplicate": false
}
</code></pre>
<p>Two things bit us here, both worth knowing before you schedule anything.</p>
<p><code>published_at.end</code> <strong>is exclusive.</strong> Asking for <code>start=2026-09-06&amp;end=2026-09-06</code> returns zero rows; <code>end=2026-09-07</code> returns the day. A daily watcher written the obvious way skips every day it runs and never errors.</p>
<p><strong>Unknown parameters are sometimes ignored in silence.</strong> <code>entity.name</code>, <code>body</code> and <code>keyword</code> at least come back flagged in <code>meta.warnings</code>, so log that array. <code>search</code> doesn't: <code>search=zzqqxwv</code> returned three cheerfully unrelated articles. Before trusting any filter, send it a nonsense value and check you get nothing back.</p>
<h2>Deduplication is the part nobody writes about</h2>
<p>820 articles from 372 sources over seven days, and the same story arrives four or five times. We tried four ways of collapsing them on the same dataset:</p>
<table>
<thead>
<tr>
<th>Method</th>
<th>Duplicates caught</th>
<th>Share of 820</th>
</tr>
</thead>
<tbody><tr>
<td>The API's <code>is_duplicate</code> flag</td>
<td>0</td>
<td>0.0%</td>
</tr>
<tr>
<td>Grouping by <code>story.id</code></td>
<td>8</td>
<td>1.0%</td>
</tr>
<tr>
<td>Exact match on normalised titles</td>
<td>106</td>
<td>12.9%</td>
</tr>
<tr>
<td>Token-set overlap ≥ 0.6</td>
<td>179</td>
<td>21.8%</td>
</tr>
</tbody></table>
<img src="https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/k03ulbtlwauldegzt2ts.png" alt="Of 820 articles, the API's own is_duplicate flag caught 0 duplicates and story.id clustering caught 8, while exact title matching caught 106 and token overlap caught 179" style="display:block;margin:0 auto" />

<p>The vendor's own duplicate flag was <code>false</code> on all 820 rows. Story clustering caught eight. Whatever your provider promises about deduplication, measure it on your own query before you rely on it — ours cost us one line of Python to check and would have cost us a fifth of our alert volume to skip.</p>
<p>Token overlap is what the watcher uses. Strip punctuation and stopwords, compare titles as sets, drop anything that overlaps an already-kept title by 60% or more:</p>
<pre><code class="language-python">STOP = set("a an the of in on at to for and or with by from is are was were as "
           "its it this that new says said after over into be will has have "
           "how why what when who".split())

def tokens(title):
    return {w for w in re.sub(r"[^a-z0-9 ]", " ", title.lower()).split()
            if w not in STOP and len(w) &gt; 2}

def dedup(articles):
    kept, seen = [], []
    for a in articles:
        t = tokens(a["title"])
        if any(len(t &amp; s) / len(t | s) &gt;= 0.6 for s in seen if t and s):
            continue
        seen.append(t)
        kept.append(a)
    return kept
</code></pre>
<p>It catches syndication, not rewriting. On 5 September the copyright suit reached us as four separate alerts — <em>"Seattle Times, Newsday sue OpenAI, Microsoft"</em>, <em>"Seattle Times, Newsday Take OpenAI And Microsoft To Court"</em>, <em>"US newspapers sue OpenAI, Microsoft"</em>, and <em>"Newsday sues OpenAI and Microsoft"</em> — because four newsrooms wrote four genuinely different sentences. Dropping the threshold to catch those starts merging unrelated stories. We kept the duplicates and moved on; embeddings would fix it, and would also turn eighty lines into a service.</p>
<h2>Routing into launches, departures and lawsuits</h2>
<p>Five regexes over title plus the first 400 characters of the description:</p>
<pre><code class="language-python">THEMES = {
    "launch": r"(launch\w*|releas\w*|unveil\w*|introduc\w*|roll\w* out|debut\w*|announc\w*|general availability|preview)",
    "people": r"(hir\w+|join\w+|depart\w+|resign\w*|step\w* down|quit\w*|fired|ousted|appoint\w*|new (ceo|cto|cfo|president))",
    "legal":  r"(lawsuit\w*|sue[sd]?\b|suing|court|judge|settl\w+|copyright|antitrust|subpoena|alleg\w+|litigation|injunction|regulat\w+|probe|investigat\w+)",
    "outage": r"(outage\w*|downtime|degrad\w+|incident\w*|status page|disruption\w*|restored|went down)",
    "money":  r"(funding|fundrais\w*|valuation|\bround\b|invest\w+|acqui\w+|\bipo\b|billion (deal|agreement)|stake)",
}
THEMES = {k: re.compile(v, re.I) for k, v in THEMES.items()}

def route(a):
    text = a["title"] + " " + (a.get("description") or "")[:400]
    return [name for name, rx in THEMES.items() if rx.search(text)]
</code></pre>
<p>Across the week: launch 284 articles (34.6%), legal 191 (23.3%), money 68 (8.3%), outage 62 (7.6%), people 53 (6.5%). 259 articles (31.6%) matched nothing and 90 (11.0%) matched more than one theme.</p>
<p>Reading the description roughly doubles recall and imports someone else's summary into your routing. Of 62 outage matches, 38 matched only in the description — including <em>"OpenAI Says It Wants to Create a Standard for Revealing AI Alignment Meltdowns"</em>, which is a policy story that says "incidents" in its second sentence. Match <code>people</code> and <code>legal</code> across title and description, because a departure is often buried in paragraph three. Match <code>outage</code> on the title alone.</p>
<p>That 31.6% residual is not a bug to tune away. It's GPT-6 benchmark scores, congressional candidates expensing subscriptions, and agents editing a German wiki — real coverage that no keyword set anticipated. Send it as a low-priority bucket rather than dropping it.</p>
<h2>How loud should the watcher be</h2>
<p>Every article carries <code>source.rankings.opr</code>, an authority score from 0 to 8. Pick your threshold by how many alerts a day you'll actually read:</p>
<table>
<thead>
<tr>
<th>Threshold</th>
<th>Articles kept</th>
<th>Share</th>
<th>Per day</th>
</tr>
</thead>
<tbody><tr>
<td><code>opr &gt;= 3</code></td>
<td>783</td>
<td>95.5%</td>
<td>112</td>
</tr>
<tr>
<td><code>opr &gt;= 4</code></td>
<td>696</td>
<td>84.9%</td>
<td>99</td>
</tr>
<tr>
<td><code>opr &gt;= 5</code></td>
<td>531</td>
<td>64.8%</td>
<td>76</td>
</tr>
<tr>
<td><code>opr &gt;= 6</code></td>
<td>246</td>
<td>30.0%</td>
<td>35</td>
</tr>
<tr>
<td><code>opr &gt;= 7</code></td>
<td>78</td>
<td>9.5%</td>
<td>11</td>
</tr>
</tbody></table>
<p>Below 5 you're subscribing to the whole internet — the top ten domains are only 17.2% of volume, so the long tail is where the count lives. At 7 you get eleven a day and you're reading TechCrunch, which you already do. We run at 6.</p>
<img src="https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/rz4g6w8g4a3zimxqmrla.png" alt="820 raw articles fall to 641 after dedup, 177 after the opr filter and 118 after theme routing — from 117 a day down to 17" style="display:block;margin:0 auto" />

<p>The table above applies the threshold to raw articles. In the watcher it runs after dedup, so the numbers differ: <strong>820 raw → 641 after dedup → 177 above</strong> <code>opr &gt;= 6</code> <strong>→ 118 with a theme.</strong> Seventeen alerts a day, of which the ones we'd have wanted to know about within the hour — the GPT-6 Astra launch, four newsrooms filing suit, the wiki incident — all survived.</p>
<p>Cost, since nobody mentions it: a day of OpenAI coverage is one to two requests at <code>per_page=100</code>, so a daily digest is about 500 requests a year and a fifteen-minute loop is roughly 35,000. Both fit inside a free tier on most providers. The expensive version is the one that polls every source separately.</p>
<h2>The whole script</h2>
<pre><code class="language-python">import os, re, sqlite3, datetime as dt
import requests

API = "https://api.apitube.io/v1/news/everything"
KEY = os.environ["APITUBE_API_KEY"]
MIN_OPR = 6

def fetch(day, query="OpenAI"):
    """One day of coverage. published_at.end is exclusive — pass the NEXT day."""
    out, page = [], 1
    while True:
        r = requests.get(API, headers={"X-API-Key": KEY}, timeout=30, params={
            "title": query,
            "language.code": "en",
            "published_at.start": day.isoformat(),
            "published_at.end": (day + dt.timedelta(days=1)).isoformat(),
            "sort.by": "published_at", "sort.order": "desc",
            "per_page": 100, "page": page,
        })
        r.raise_for_status()
        body = r.json()
        for w in (body.get("meta") or {}).get("warnings", []):
            print("API warning:", w["message"])
        out += body.get("results", [])
        if not body.get("has_next_pages"):
            return out
        page += 1

def loud_enough(a):
    return (a["source"].get("rankings", {}).get("opr") or 0) &gt;= MIN_OPR

def main():
    db = sqlite3.connect("seen.db")
    db.execute("CREATE TABLE IF NOT EXISTS seen (id INTEGER PRIMARY KEY)")
    yesterday = dt.date.today() - dt.timedelta(days=1)

    raw = fetch(yesterday)
    fresh = [a for a in raw
             if not db.execute("SELECT 1 FROM seen WHERE id=?", (a["id"],)).fetchone()]
    digest = [a for a in dedup(fresh) if loud_enough(a) and route(a)]

    for a in digest:
        print(f"[{'/'.join(route(a))}] {a['title']}  "
              f"({a['source']['domain']}, opr {a['source']['rankings']['opr']})")

    db.executemany("INSERT OR IGNORE INTO seen VALUES (?)", [(a["id"],) for a in raw])
    db.commit()

if __name__ == "__main__":
    main()
</code></pre>
<p>The SQLite table exists because reprints trickle in for days and the dedup pass only sees one run at a time. Point <code>print</code> at a Telegram <code>sendMessage</code> call or an SMTP digest and you're done. Run it on cron at 08:00 for a morning digest, or every fifteen minutes if you want <code>legal</code> and <code>outage</code> fast — the freshest article in our collection was 53 minutes old when we pulled it, so a quarter-hour loop is not obviously wasteful.</p>
<p>Real output, 5 September:</p>
<pre><code class="language-plaintext">2026-09-05: 87 articles -&gt; 78 after dedup -&gt; 15 alerts
[outage] OpenAI confirms 'wiki incident,' says it's 'working on a framework'  (techcrunch.com, opr 8)
[launch] OpenAI Launches GPT-6 Astra With Computer Use Tools  (dev.to, opr 7)
[legal] Newsday sues OpenAI and Microsoft over copyright infringement  (newsday.com, opr 6)
[launch/money] Nvidia inks $13 billion deal to buy the AI startup hacked by OpenAI  (egyptindependent.com, opr 6)
</code></pre>
<h2>Swap the query, keep the pipeline</h2>
<p>Change <code>title=OpenAI</code> to <code>organization.name=Anthropic</code> and everything downstream still works — dedup, routing and thresholds don't care where the rows came from. That's the point of doing the filtering yourself. Just re-run the entity check first, and re-measure the duplicate rate: 21.8% is what OpenAI's coverage looked like, not a constant.</p>
<h2>Questions we had while building this</h2>
<h3>How do I monitor OpenAI news automatically?</h3>
<p>Monitoring OpenAI news automatically means scheduling a daily query against a news index for articles with OpenAI in the title, then deduplicating, routing into themes, and thresholding on source authority before anything is delivered. The query takes ten lines; the filtering is where the work is. On our dataset that pipeline reduced 820 articles a week to 17 alerts a day.</p>
<h3>Does OpenAI have an RSS feed?</h3>
<p>OpenAI publishes an RSS feed at <code>https://openai.com/news/rss.xml</code>, free and complete for anything the company says about itself. It will never carry a lawsuit filed against OpenAI, an executive leaving, or an outage the company hasn't acknowledged yet. Subscribe to it <em>and</em> run a third-party watcher; they cover different things.</p>
<h3>How do I track a company's news with Python?</h3>
<p>Tracking a company's news in Python starts with checking that the entity exists (<code>/v1/companies?name=...</code>), falling back to title search if it doesn't, paging through with <code>per_page=100</code>, remembering what you've already sent in SQLite, and deduplicating by token overlap. That's roughly eighty lines with <code>requests</code> and the standard library.</p>
<h3>How do you filter duplicate news articles?</h3>
<p>Duplicate news articles are best filtered by comparing normalised titles as token sets and dropping anything that overlaps an already-kept title by 60% or more. On our dataset that caught 179 of 820 articles, against 106 for exact title matching and 0 for the API's own duplicate flag. Rewritten headlines still slip through; embeddings are the next step.</p>
<h3>What's the best API for AI news?</h3>
<p>The best API for AI news is whichever one resolves the entities you actually care about, and that is a single request to find out — don't take a vendor's entity coverage on faith, including ours. We used APITube because it exposes source authority scores and a <code>count</code> endpoint, which is what made the thresholds in this article measurable. APITube is one of the APIs we mentioned — there's a free tier at <a href="https://apitube.io">apitube.io</a>.</p>
<h2>Resources</h2>
<ul>
<li><p><a href="https://docs.apitube.io">APITube news API docs</a> — endpoints and filters used here</p>
</li>
<li><p><a href="https://openai.com/news/rss.xml">OpenAI news RSS feed</a> — first-party announcements</p>
</li>
<li><p><a href="https://www.perigon.io/blog/monitor-openai-news">perigon.io on monitoring OpenAI</a> — the entity-first playbook this article argues with</p>
</li>
</ul>
]]></content:encoded></item></channel></rss>