import-scheduler: one cron lane and one lock per import rule

A WordPress plugin that runs each import rule on its own interval with its own database lock and history, so one slow or failing source cannot starve the others.

import-scheduler is a WordPress plugin for sites that import from many sources. It runs each import rule as its own unit of work on a dedicated cron lane, with per-rule last run, next run and duration on one screen and in WP-CLI. It is MIT licensed, and v0.1.0 was released on 15 November 2026.

The problem

The usual set-up runs every import rule inside one scheduled hook, with one lock so two passes never overlap. That fails quietly as the rule count grows.

  • One pass takes longer than the cron cadence. Every tick that fires during a pass finds the lock held and does nothing, so the intervals on the settings screen are nominal and the real cadence is the length of the whole pass.
  • One slow or hung source holds the lock for everyone.
  • When the import shares the general WordPress cron run, it also delays scheduled posts and other plugins’ jobs.
  • One fatal error in one rule skips every rule after it until the next pass.
  • Nobody can see any of this. The operator only notices that imports feel slow.

The plugin makes each rule its own unit of work with its own interval, lock and history, and a tick that stops choosing new work when its time budget is spent.

What it does

  • Registers rules in code with an interval from 60 seconds to 30 days and a lane name. Any callable can be a runner.
  • Runs lanes from a system cron line, one tick per minute by default, instead of WP-Cron.
  • Records each run with a status of success, failure, overrun, abandoned or lock_lost, a message and a monotonic duration.
  • Backs off failing rules exponentially and can pause a rule after a set number of failures in a row.
  • Ships a reference adapter, import_scheduler_register_feed(), that reads one feed through core’s fetch_feed() and emits items on an action. Turning items into posts is the consumer’s job.
  • Gives a screen under Tools and WP-CLI commands including tick, list, status, run, pause, pause-lane, history, unlock and cron-line.

How it works

Rules are registered unconditionally on the import_scheduler_register action, so the WP-CLI process that runs the lane sees them too:

add_action('import_scheduler_register', function () {
    import_scheduler_register('partner-api', function (\IndianOpinion\ImportScheduler\RunContext $context) {
        $response = wp_safe_remote_get('https://example.com/api/items', ['timeout' => 15]);
        if (is_wp_error($response)) {
            return $response;              // failure, with its message
        }
        // ... import, checking $context->shouldStop() between items ...
        return 12;                         // success, 12 items
    }, ['interval' => 600, 'lane' => 'default']);
});

Then one system cron line per lane, printed by wp import-scheduler cron-line --lane=default --user=www-data --path=/var/www/site. Each tick reads the database clock once, runs pre-flight guards, takes a lane slot, finds due rules with one indexed query capped at 200 rows, oldest due first, and runs them one by one until its budget is spent. Rules whose turn did not come stay due and lead the next tick.

A lock is a row with a name, owner token, acquired time, expiry and heartbeat. Taking one is an insert. If the name is taken, a single conditional update takes it over only when it has expired. All lock times come from UNIX_TIMESTAMP() in the statement, so web nodes with skewed clocks agree. There are no lock files, no flock and no MySQL named locks, and a crashed process never wedges a lane for longer than the expiry.

After a success the next run is the scheduled time plus the interval, or now plus the interval if that is past, so there is never a burst of catch-up runs. After an overrun it is the finish plus a full interval. After a failure it is the finish plus the interval times 2 to the power of failures minus 1, capped. Storage is four tables: rules, runs, locks and ticks.

Guard rails

  • Kill switch: IMPORT_SCHEDULER_DISABLE in wp-config.php wins over the setting. Running rules see shouldStop() turn true. A single lane can be stopped with pause-lane.
  • A tick has a time budget (default 55 seconds), a memory ceiling (default 80 per cent of memory_limit), and optional disk and available-memory floors, below which it stands down.
  • An exception or fatal error in one rule is recorded against that rule only. A shutdown handler closes the run, and the next tick marks a run that died without it as abandoned once its lock expires.
  • A long run is never interrupted, never stacked and never blocks other lanes. Out-of-range intervals, lane names and rule ids are rejected, not clamped.
  • Lock and state reads always go to the database, never the object cache.
  • The README says never to run the cron line as root, to send its output to a log file and not to /dev/null, and to give every HTTP call in a runner a timeout.

What review changed

  • The tick read its due list once, so a rule that another slot or a manual run had finished, or that an operator had paused or moved, could run again. The runner now re-reads the state row after taking the rule lock and skips the rule if it is no longer due.
  • Running run <id> on a rule that was not yet due pushed its schedule out by a full interval. The scheduled time is now the smaller of the stored next run and the run’s start.
  • Lane slot expiry used registered lanes and intervals only, so a slow rule moved into a lane by an operator could outlive the slot and let a second tick in. The tick now stretches the slot when overrides need it. A failed slot renew, previously ignored, now stops the tick choosing rules.
  • Every WP-CLI command other than tick now reports missing tables or a database error as an error with exit code 1. Each fix has a regression test that fails on the previous code.

Limits and what is next

  • It cannot kill a running callable. Overruns are handled by recording, cooperative stop signals and lock expiry.
  • It does not create posts or de-duplicate items.
  • It never installs or removes a crontab line. You remove the line yourself on uninstall.
  • One slot per tick. Multi-slot claims are on the roadmap.
  • Single site per tick. On multisite each site runs its own lanes with --url.
  • A slow rule on a one-slot lane makes the other rules on that lane wait, so give slow rules a lane of their own.
  • There are no open issues at the time of writing.

Test status: this is v0.1.0. The review reports 74 PHPUnit tests on PHP 8.3, PHPStan level 8 and phpcs clean. One test, the MySQL lock race test, is gated on a database connection and was not run locally, so it is left to a CI job. The other tests use an in-memory double and WordPress stubs. CI has not yet run on GitHub, and the plugin has not run against a live site. It was specified, built and reviewed in separate roles by language-model coding agents under the editor’s direction.

Install

It needs PHP 8.1 or later, WordPress 6.4 or later, WP-CLI 2.8 or later, MySQL 5.7 or later or MariaDB 10.4 or later, and a host where you can add a system cron line.

The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/import-scheduler.

composer require indianopinion/import-scheduler

Without Composer, unzip a release into wp-content/plugins/import-scheduler/ and activate it.

All 47 utilities · import-scheduler on GitHub

Search Trending
Loading

Signing-in 3 seconds...

Signing-up 3 seconds...

All fields are required.