About the WorldFeedMonitor crawler

What it is

WorldFeedMonitor is a news aggregator. It reads the RSS and Atom feeds that publishers make public, and serves them to our customers as a structured API.

If you are reading this, you probably found the address in a User-Agent header in your logs. This page exists so you can see exactly what that request was and decide what you want to do about it.

How to identify it

Every request we make sends this User-Agent:

Mozilla/5.0 (compatible; WorldFeedMonitor/1.0; +https://worldfeedmonitor.com/crawler)

We do not disguise our requests as a browser, and we do not rotate identities. A request that does not carry that string is not from us.

What it fetches

Only feed URLs — the RSS or Atom document itself. We do not crawl your site, follow links out of the feed, fetch article pages, or request images and assets. One feed URL is one request.

We store what the feed publishes: headline, summary, link, publication date, categories and, where the feed supplies one, the image URL. Our customers are always given the link back to your page.

How often

Each feed is read at most once every fifteen minutes. If a feed has not changed since the last read, we notice and do nothing further with it.

We do not fetch in parallel bursts against a single host, and we do not retry a request that has been answered — including one answered with a refusal.

If you refuse it

A 403 or 429 is treated as your decision, not as an error to retry around. We slow down on it: after a few refusals we drop to hourly, then to every six hours, then to once a day. We do not change our User-Agent, our address, or anything else to get a different answer.

We keep the feed on that slow schedule rather than dropping it, because blocks are sometimes temporary or accidental. If yours is deliberate and permanent, tell us and we will remove the feed entirely.

How to block it

Any of these works, and none of them needs our cooperation:

• Disallow the feed path for WorldFeedMonitor in your robots.txt.
• Return 403 to the User-Agent above. We will back off automatically.
• Email us and we will stop, whatever your server says.

Allowlisting and contact

If you would rather we kept reading your feed and something in front of it is refusing us — a WAF rule or an address-reputation filter, usually — allowlisting the User-Agent above is enough.

Either way, write to legal@worldfeedmonitor.com. A request to stop is honoured without argument, and we do not ask for a reason.