# echo-skip-seen: stop importers re-downloading seen articles

echo-skip-seen is a WordPress plugin for sites that import articles from feeds. It answers the importer's duplicate check from a small table of URLs the site has already imported, so a seen article is never downloaded again. The code is at [https://github.com/IndianOpinion/echo-skip-seen](https://github.com/IndianOpinion/echo-skip-seen), under the MIT licence, and v0.1.0 was released on 12 November 2026.

It is a companion to Echo RSS Post Importer, through that plugin's public echo_filter_dup_check filter. Any other importer can use it through a function, a filter or an action. We wrote it from scratch. It is independent of and not affiliated with the importer it integrates with.

## The problem

Feed importers fetch the article page for every feed item, and only then decide whether the post already exists. A feed lists one article for hours or days, and a site that follows many feeds sees it through several of them. Each run downloads pages it then throws away. That costs bandwidth, PHP time and importer run time. It also costs goodwill with publishers whose servers see the same fetch again and again, and some of them rate-limit or block the fetching host.

Echo RSS Post Importer applies echo_filter_dup_check to each item URL before it downloads the page, and skips the item if a callback returns true. Nothing in WordPress answers that filter by default, so the hook goes unused. This plugin is the callback that was missing.

## What it does

- Returns true from the importer's duplicate filter for a URL whose article was already imported, before any download.

- Treats common URL variants as one article: scheme, mobile and AMP hosts, AMP paths, tracking parameters, fragments, trailing slashes, default ports, index.html and percent-encoding case.

- Records a URL as seen when a post is created with a source URL in its meta, with no change to the importer.

- Seeds the table from existing posts, from an admin button or with wp skip-seen backfill.

- Expires old rows after a retention period, 90 days by default.

- Shows counters on a Tools page and through WP-CLI.

- Runs as a normal plugin or a must-use plugin, on single sites and multisite.

## How it works

The plugin keeps one table per site, {prefix}echo_skip_seen. A row holds a 16-byte MD5 hash as the primary key, a key kind, a key version, the post id and two UTC times. No URL is stored, so the table is about 60 bytes a row and holds nothing beyond "some post came from some URL".

Each URL gets two keys. The canonical key collapses the variants listed above. On a slug path only a short list of id parameters survives. On a script endpoint such as /?p=1 the query is kept, because there it names the page. The plain key is a frozen, conservative exact-match key. It keeps matching if the canonical rules change in a later version.

A lookup computes both hashes and runs one indexed query, cached per request. A match counts a skip and refreshes last_seen at most once an hour per row. A miss returns the value the callback received. It does not return false, because that could overwrite another callback's true.

Rows are written only when a post exists: on wp_after_insert_post, or when the configured source meta key is added or updated, since importers often write it after the insert. A lookup never records.

The quick start, from the README:

wp skip-seen backfill --meta-key=<your_key> --dry-run
wp skip-seen backfill --meta-key=<your_key>
wp skip-seen check 'https://example.org/news/story-1?utm_source=rss'

You first find the meta key your importer stores the original URL under, and set it on the Tools page. The key list ships empty on purpose. A detector would have to guess which URL-valued field is the source, and a wrong guess would poison the table quietly.

The public interface is three functions (echo_skip_seen_is_seen(), echo_skip_seen_record(), echo_skip_seen_keys()), a set of actions and filters for tuning and observing, and the WP-CLI commands backfill, check, record, stats, prune and verify-schema.

## Guard rails

- Fail open. A missing table, a database error, a broken filter callback or a URL that cannot be keyed all mean "not seen", and the importer downloads as usual. After one database error in a request, the plugin stops querying for the rest of it.

- A kill switch: the ECHO_SKIP_SEEN_DISABLED constant wins over the setting and passes every value through.

- Non-URLs, non-http values, the site's own host and configured redirect-wrapper hosts are left alone.

- Every statement with a value goes through $wpdb->prepare(). Hashes cross the connection as hex through UNHEX().

- After creating or upgrading the table, the plugin reads the real columns and indexes back, repairs missing ones with ALTER TABLE, and stores the schema version only then. We added this because dbDelta can skip column adds without saying so.

- Uninstall drops only the plugin's table, options, transient and scheduled event.

## What review changed

The independent reviewer's comment on the build pull request lists five fixes made before the merge:

- The last line of fail-open was missing. Hook wrappers caught errors while running services but not while building them, so a failure in construction, such as no usable database object, threw into the importer and into wp_insert_post. Every wrapper now catches Throwable around both.

- Counters were wrong on multisite. Pending increments were flushed into whichever site was current, so an importer that switched blogs wrote one site's counts to another. They are now kept and written per site.

- The backfill ran out of memory on large sites, because it never released post and meta caches between batches. The CLI now clears the object cache after each batch.

- A test fixture used PHP 8.2 syntax (a standalone false return type), which would have failed the PHP 8.1 jobs. It was fixed and the analyser was pinned to the 8.1 target.

The review also reversed a build decision on path case. The builder had made paths case-sensitive. The reviewer restored the PRD default of lower-casing, since the plain key already lower-cases the whole URL, and the reversal only lost variants that differed in case and something else.

## Limits and what is next

- It only knows URLs. A syndicated copy, or an article with a new slug, counts as a different article unless an article id rule or the host filter joins them.

- Redirect wrappers are left alone. Without an unwrap tool on the echo_skip_seen_url filter, those items download as before.

- It knows only what was imported after install or backfill. Run the backfill once.

- It needs the source URL meta key. With none set, only direct calls record.

- Paths compare without case by default. Two articles whose URLs differ only in path case are treated as one, unless you opt that host out.

Test status: this is v0.1.0. At review, PHPUnit ran 176 tests with PHPStan level 8 and phpcs clean, against a fake database object, stubs and mocks. The CI workflow has not run on GitHub yet, because the token lacked the scope to push it. The plugin has not run on a live site, and there are no open issues.

## Install

composer require indianopinion/echo-skip-seen

The package is not on Packagist yet, so add a Composer vcs repository entry for https://github.com/IndianOpinion/echo-skip-seen first.

It needs WordPress 6.4 or later, PHP 8.1 or later, and MySQL 5.7 or MariaDB 10.4. The intl extension is optional. Without Composer, put the folder in wp-content/plugins/echo-skip-seen and activate it, or copy it to mu-plugins with the one-line loader from the examples.

[All 47 utilities](https://indianopinion.org/blog/open-source/) · [echo-skip-seen on GitHub](https://github.com/IndianOpinion/echo-skip-seen)

Canonical: https://indianopinion.org/blog/open-source/echo-skip-seen/

AI disclosure: Every summary and comment here is written by AI and published automatically, without a person reviewing each article. We do no original reporting. How this works: https://indianopinion.org/ai-use-policy/
