echo-robots-guard is a WordPress plugin that reads a publisher’s robots.txt before the Echo RSS Post Importer downloads an article page. It skips the item when robots.txt disallows it for your product token. It is for site operators who import news from feeds and want a /bot page that tells the truth. The code is at https://github.com/IndianOpinion/echo-robots-guard, under the MIT licence, and v0.1.0 was released on 12 November 2026.
It is a companion to that importer, through its public echo_filter_dup_check filter. We wrote it from scratch. It is independent of and not affiliated with the importer it integrates with.
A site that imports news with the importer reads a feed, then downloads each linked article page. Nothing in that path reads the publisher’s robots.txt. The operator cannot honestly say “we honour your robots.txt” on a /bot page. A publisher who writes User-agent: YourBot and Disallow: / is ignored.
The importer exposes one public point before the download: the filter echo_filter_dup_check, applied to each item URL, where a true return skips the item. The plugin uses it to ask whether robots.txt lets this product token fetch this URL. That filter is the only vendor fact the project relies on.
enforce (the default), shadow (log what would be skipped, skip nothing) and off.check exits with status 1 when any URL is disallowed.may_fetch() and a filter, so another importer can ask the same question.The plugin hooks echo_filter_dup_check at priority 20, after duplicate checks at priority 10. For each URL it skips non-http values and hosts on a skip list. It then finds the robots.txt rules for the origin through a per-request memo, the object cache, the plugin’s own table, and finally a fetch. It matches them against your token and returns true to skip, or the original value to allow. It never returns false, so another callback’s true is never overwritten.
The parser and matcher are plain PHP classes under src/Robots/ with no WordPress dependency. They implement the RFC’s rules:
* group is a fallback only when none names the token.* and $ work, percent-encoding is normalised on both sides, and /robots.txt itself is always allowed.For HTTP status, 2xx is parsed and 4xx means no file, so allow all. 5xx, timeouts, DNS and TLS errors mean unreachable. A 429 is also read as unreachable. That departs from the RFC, which groups it with 4xx, and it is deliberate: 429 says “come back later”, not “there is no file”.
The cache is one row per origin in a network-wide table, with the object cache in front. It stores parsed groups, never the raw body and never the group chosen for a token, so changing the token takes effect at once. Errors are cached for an hour and successes for a day. A stale copy fetched within the last 30 days decides when the origin is unreachable.
The checks from the README:
wp echo-robots-guard check <url>... [--token=<token>] [--fresh]
wp echo-robots-guard check - < feed-urls.txt
$decision = may_fetch('https://news.example/story');
$decision->allowed; // true, false, or null when not checked
$decision->reason; // rule, no_rule, unreachable, stale_copy, fetch_cap, ...
wp_safe_remote_get(), so a feed item cannot make the site probe private, loopback or link-local addresses. No cookies, referrer or credentials are sent./*a*a*a*a*b cannot stall an import.The reviewer pushed five commits before the merge:
strspn() with a long mask is a nested loop in PHP. A fixed character class replaced it, with byte-identical output.wp echo-robots-guard check was capped at 20 fetches, so an audit of a long feed list reported fetch_cap for the rest. The command now lifts the cap for its own run.The review also accepted one departure from the PRD: fresh installs default to enforce, so the public statement is true from the first import. Shadow mode is documented for a trial.
Test status: this is v0.1.0. At review, 198 tests passed with PHPStan level 8 and phpcs clean. They ran against stubs, mocks and fakes, not a live importer. The CI workflow has not run on GitHub yet, because the token lacked the scope to push it. The plugin has not run on a live site. There are no open issues.
composer require indianopinion/echo-robots-guard
The package is not on Packagist yet, so add a Composer vcs repository entry for https://github.com/IndianOpinion/echo-robots-guard first.
It needs WordPress 6.4 or later and PHP 8.1 or later, with no Composer at runtime. Without Composer, put the folder in wp-content/plugins/echo-robots-guard/ and activate it, or copy it to mu-plugins with the one-line loader.