echo-robots-guard: honour robots.txt before an import download

A WordPress plugin that reads the publisher's robots.txt under RFC 9309 before Echo RSS Post Importer downloads an article page, and skips the item when the rules disallow it for your product token.

echo-robots-guard is a WordPress plugin that reads a publisher’s robots.txt before the Echo RSS Post Importer downloads an article page. It skips the item when robots.txt disallows it for your product token. It is for site operators who import news from feeds and want a /bot page that tells the truth. The code is at https://github.com/IndianOpinion/echo-robots-guard, under the MIT licence, and v0.1.0 was released on 12 November 2026.

It is a companion to that importer, through its public echo_filter_dup_check filter. We wrote it from scratch. It is independent of and not affiliated with the importer it integrates with.

The problem

A site that imports news with the importer reads a feed, then downloads each linked article page. Nothing in that path reads the publisher’s robots.txt. The operator cannot honestly say “we honour your robots.txt” on a /bot page. A publisher who writes User-agent: YourBot and Disallow: / is ignored.

The importer exposes one public point before the download: the filter echo_filter_dup_check, applied to each item URL, where a true return skips the item. The plugin uses it to ask whether robots.txt lets this product token fetch this URL. That filter is the only vendor fact the project relies on.

What it does

  • Checks each item URL against its host’s robots.txt under RFC 9309, and skips the item when the rules disallow it for your product token.
  • Has three modes: enforce (the default), shadow (log what would be skipped, skip nothing) and off.
  • Caches parsed rules per origin, with a configurable lifetime of 24 hours by default.
  • Lets you choose a failure policy for an unreachable robots.txt: closed (the RFC rule, and the default) or open.
  • Generates the text of your /bot page from live settings with a shortcode, so the page says what the plugin actually does.
  • Offers WP-CLI commands to check URLs, list and clear the cache, and read counters and the blocked log. check exits with status 1 when any URL is disallowed.
  • Exposes may_fetch() and a filter, so another importer can ask the same question.

How it works

The plugin hooks echo_filter_dup_check at priority 20, after duplicate checks at priority 10. For each URL it skips non-http values and hosts on a skip list. It then finds the robots.txt rules for the origin through a per-request memo, the object cache, the plugin’s own table, and finally a fetch. It matches them against your token and returns true to skip, or the original value to allow. It never returns false, so another callback’s true is never overwritten.

The parser and matcher are plain PHP classes under src/Robots/ with no WordPress dependency. They implement the RFC’s rules:

  • Groups are matched by exact product token, never a prefix. Groups naming the token are merged. The * group is a fallback only when none names the token.
  • The longest matching pattern wins, and Allow wins a tie. * and $ work, percent-encoding is normalised on both sides, and /robots.txt itself is always allowed.
  • Parsing stops at 500 KiB.

For HTTP status, 2xx is parsed and 4xx means no file, so allow all. 5xx, timeouts, DNS and TLS errors mean unreachable. A 429 is also read as unreachable. That departs from the RFC, which groups it with 4xx, and it is deliberate: 429 says “come back later”, not “there is no file”.

The cache is one row per origin in a network-wide table, with the object cache in front. It stores parsed groups, never the raw body and never the group chosen for a token, so changing the token takes effect at once. Errors are cached for an hour and successes for a day. A stale copy fetched within the last 30 days decides when the origin is unreachable.

The checks from the README:

wp echo-robots-guard check <url>... [--token=<token>] [--fresh]
wp echo-robots-guard check - < feed-urls.txt

$decision = may_fetch('https://news.example/story');
$decision->allowed;   // true, false, or null when not checked
$decision->reason;    // rule, no_rule, unreachable, stale_copy, fetch_cap, ...

Guard rails

  • It never stops an import. Any error inside the plugin is caught at the hook, counted, logged once, and the item passes.
  • robots.txt is fetched with wp_safe_remote_get(), so a feed item cannot make the site probe private, loopback or link-local addresses. No cookies, referrer or credentials are sent.
  • Matching is linear, so a hostile pattern such as /*a*a*a*a*b cannot stall an import.
  • At most 20 robots.txt fetches run per request by default. Items past the cap use a stale copy if one is held, else the failure policy, and are checked properly on a later run.
  • A cache lifetime above 24 hours is accepted with a warning, because the RFC says a cached copy should not be used for longer.

What review changed

The reviewer pushed five commits before the merge:

  • The parser could silently drop rules. A single line padded with spaces (5,000 were enough) hit the PCRE backtrack limit, the matching call returned early, and every rule after that line was lost, so the origin failed open on rules. Lines are now split with plain string functions. A differential run over 4,000 random inputs gives identical groups to the old parser.
  • A strict timing test failed. Parsing 500 KiB took about 56 ms against a 50 ms budget, because strspn() with a long mask is a nested loop in PHP. A fixed character class replaced it, with byte-identical output.
  • The address check had gaps. IPv4-mapped and NAT64 forms of loopback addresses passed. The check now inspects the embedded IPv4 address, and userinfo in a redirect target is stripped before the next hop.
  • wp echo-robots-guard check was capped at 20 fetches, so an audit of a long feed list reported fetch_cap for the rest. The command now lifts the cap for its own run.

The review also accepted one departure from the PRD: fresh installs default to enforce, so the public statement is true from the first import. Shadow mode is documented for a trial.

Limits and what is next

  • The feed URL itself is never checked. The importer applies no filter before it fetches the feed, so the plugin cannot see that request. The README gives an operator policy: check a feed with WP-CLI before adding it, review existing feeds on a schedule, and publish a contact address on the /bot page.
  • Only the article download is guarded, and only for this importer. A callback at a higher priority than 20 can still change the result.
  • The importer’s own User-Agent is not changed. The plugin names itself only on its robots.txt fetch, so decisions are keyed on the product token.
  • Redirect links from aggregators are checked against the aggregator’s robots.txt, which does not speak for the publisher. Put such hosts on the skip list.

Test status: this is v0.1.0. At review, 198 tests passed with PHPStan level 8 and phpcs clean. They ran against stubs, mocks and fakes, not a live importer. The CI workflow has not run on GitHub yet, because the token lacked the scope to push it. The plugin has not run on a live site. There are no open issues.

Install

composer require indianopinion/echo-robots-guard

The package is not on Packagist yet, so add a Composer vcs repository entry for https://github.com/IndianOpinion/echo-robots-guard first.

It needs WordPress 6.4 or later and PHP 8.1 or later, with no Composer at runtime. Without Composer, put the folder in wp-content/plugins/echo-robots-guard/ and activate it, or copy it to mu-plugins with the one-line loader.

All 47 utilities · echo-robots-guard on GitHub

Search Trending
Loading

Signing-in 3 seconds...

Signing-up 3 seconds...

All fields are required.