# finetune-export: export posts as JSONL without leaking drafts

[finetune-export](https://github.com/IndianOpinion/finetune-export) is a WordPress plugin for site owners who want their own writing as a fine-tuning set. It exports posts as JSONL, redacts personal data on a best-effort basis, and makes no model calls and no uploads. It is MIT licensed, and v0.1.0 was released on 19 November 2026.

It was written from scratch from an idea, as part of a set of utilities built between 11 and 19 November 2026 by language-model coding agents under the editor's direction. It has not run on a live site.

## The problem

Getting a site's writing out of WordPress in a usable shape is harder than it looks, and the easy routes are unsafe.

- A database dump or the core XML export carries drafts, private and password-protected posts, revisions, author email addresses and comment data.

- Post bodies hold block markup, shortcodes, scripts and embeds. A model trained on that learns markup, not prose.

- Posts name people and quote addresses, phone numbers and email addresses. A model can memorise and repeat them, and fine-tuning providers keep uploaded files for a time.

- Nobody knows the size of the set until it is uploaded and billed. Records over a context limit fail late or are cut silently.

- The usual place for a plugin to write a file is the uploads directory. That is public by design, and on many sites it is mirrored to storage or a CDN that serves files without asking the web server. A private export written there is public to anyone with the URL.

## What it does

- Exports published posts in three shapes: chat, prompt_completion and text, as one JSON object per line.

- Lets you choose post types, date range, categories, tags, authors, length limits and fields, and apply a validation split.

- Cleans block comments, shortcodes, scripts, iframes and forms, and outputs text, Markdown or a small HTML subset.

- Redacts URLs, emails, IPs, phones, addresses and names, with optional consistent pseudonyms per record.

- Shows a preview of the first records after cleaning and redaction, and a count with skip reasons, before anything is written.

- Estimates tokens locally and applies a per-record limit by skipping or truncating at a paragraph break.

- Offers a per-post and a per-category switch to exclude content from training exports.

## How it works

One Pipeline turns a post into a record or a skip reason, and preview, count and export all share it. Posts are read through get_posts in ascending id order, in batches. Revisions, attachments, trash and autosaves are never selected. Each dropped post counts a reason such as too_short, opted_out or duplicate.

Redaction runs in a fixed order: URLs, emails, IPs, phones, addresses and postcodes, then names. Placeholders are protected markers, so a later rule cannot match an earlier placeholder. Name sources are site users, a list you supply, honorifics and a filter.

The token estimate counts ASCII letter and digit runs as the length divided by four, rounded up, and adds a fixed overhead per chat message. It can be off by 25 percent or more, mostly for non-Latin scripts. An offline script, tools/calibrate.php, compares it with a real tokeniser, and an estimator can be plugged in.

Jobs run in batches through WP-Cron, write to a .part file with mode 0600, and rename it on completion. A dead batch is cut back to the last good offset and resumed. A download link is an 83-character token with an HMAC over job, user, expiry and a nonce, keyed from the site salts and a per-site secret.

wp finetune-export check-directory
wp finetune-export preview --limit=3
wp finetune-export count
wp finetune-export run --foreground
wp finetune-export link 12 --user=1 --ttl=30

## Guard rails

- Published posts only and redaction on, by default. Drafts, password-protected posts, turning off a redaction type or a name source, and acting on other users' jobs need the finetune_export_sensitive capability.

- The plugin refuses to run if no safe directory exists. It never writes into uploads, even with an acknowledgement.

- A directory inside the web root needs an acknowledgement and a self-check. The self-check writes a random canary file and requests it over HTTP. If it is served, jobs and downloads stop. Only a 4xx answer counts as safe.

- Download links work only for the signed-in user they were issued to, for 60 minutes by default. Anonymous requests get a 401. Every other refusal, including an expired link, returns the same 404 body.

- Files are removed only by the retention setting or an explicit operator delete, and each removal is logged. Uninstall never touches export files.

- The plugin's only outbound request is the self-check to the site's own URL.

- The README says plainly that redaction is best-effort, and that the export is the operator's responsibility under the data-protection law that applies to them.

## What review changed

- The batch lock was not a mutex. add_option sends INSERT ... ON DUPLICATE KEY UPDATE, so two requests could both get a truthy result, shown against MySQL 8.0. The lock is now a plain INSERT, with takeover by compare-and-swap, read from the database and not the object cache.

- The protected directory could fail open. With WordPress in a subdirectory, the default location sat inside the served home directory but counted as outside the web root. The home directory and document root now count as web root.

- The self-check reported a redirect as safe. Now only a 4xx answer is safe.

- Switching every name source off turned name redaction off without the sensitive capability. Any name source off now counts as a sensitive choice.

- Deleting the files of a stalled job let a resume reopen the removed file at its old offset. Delete now needs a final state.

## Limits and what is next

- Redaction is lists and patterns. It misses names not on a list and not after an honorific, unusual address and phone formats, and personal data in images. It also replaces some text that is not personal.

- The token estimate is advisory.

- No multi-turn chat records in 0.1.0.

- Downloads have no byte ranges or resume.

- The user list for redaction is capped at 50,000 names. The 200,000 post figure in the README is a design target, not a measurement.

There are no open issues. This is v0.1.0. The suite has 191 tests and 1101 assertions, with 12 regression tests for the review fixes, mostly on stubs and mocks. The database layer was run once against a throwaway MySQL 8.0, and the lock was tested with eight racing processes over 30 rounds. Static analysis and code style checks pass. No real export has been uploaded to a provider, and the CI workflow has not run on GitHub yet.

## Install

Requires WordPress 6.4 or later and PHP 8.1 or later with dom, json and mbstring. There are no runtime Composer dependencies.

composer require indianopinion/finetune-export

The package is not on Packagist yet, so add a Composer vcs repository entry for https://github.com/IndianOpinion/finetune-export first.

Or unzip the release into wp-content/plugins/finetune-export/ and activate it. Set FINETUNE_EXPORT_DIR to a directory outside the web root first.

[All 47 utilities](https://indianopinion.org/blog/open-source/) · [finetune-export on GitHub](https://github.com/IndianOpinion/finetune-export)

Canonical: https://indianopinion.org/blog/open-source/finetune-export/

AI disclosure: Every summary and comment here is written by AI and published automatically, without a person reviewing each article. We do no original reporting. How this works: https://indianopinion.org/ai-use-policy/
