readable-extract is a PHP library that extracts the article body, title, authors, dates and lead image from a news page. It reports where each field came from and how sure it is. It treats Indic scripts as first-class. It is for developers who handle news links: importers, newsroom tools, aggregators and archivists. The code is at https://github.com/IndianOpinion/readable-extract, licensed MIT. Version v0.1.0 was released on 15 November 2026.
We wrote it from scratch from an idea. The scoring heuristic is our own and is not a port of an existing library. It has not run on a live site. The 47 utilities in this series were specified, built, reviewed and released between 11 and 19 November 2026 by language-model coding agents under the editor’s direction, in five roles: a spec writer, a spec reviewer, a builder that saw only the spec, an independent code reviewer and a releaser. The editor made the rulings on every open question.
Importers usually take whatever text a feed or a generic scraper hands them. On news sites that text is often wrong in ways that are hard to see:
src and the photo in data-src or srcset.The PRD’s conclusion is that a wrong body is worse than an empty one. An empty body is noticed and retried. A wrong body is trusted and repeated.
Article with a page type (article, liveblog, video, gallery, index, paywalled, script_rendered, wrapper, empty or unknown), body as allowlisted HTML and plain text, authors, dates in UTC, lead image, canonical URL, language and detected script.json_ld, microdata, meta, visible or none), a per-field confidence and an overall confidence from 0 to 1.biography_only, not_an_article and visible_body_short.extract, score, baseline and fixture commands.Extractor runs the steps in order. It decodes the character set (byte-order mark, declared charset, meta tags, then UTF-8), parses with libxml’s HTML parser, reads JSON-LD, microdata and meta tags before anything is removed, classifies the page, finds the body, cleans it and then fills each field from a fixed source order. No field is found by running a regular expression over the whole page.
Body scoring walks the DOM. A paragraph of at least 25 characters scores one point, plus one per comma, half a point per sentence mark (the danda counts) and up to three for length. The parent gets the full score and the grandparent half. Class and id hints add or subtract 25, and the result is multiplied by one minus the link density. The visible body wins when it meets the prose minimum of 400 characters. Otherwise a structured body wins. Two long bodies that disagree are never merged.
The confidence is a base from the body (1.0 when a structured body confirms the visible one, 0.9 for a visible body that meets the minimum, 0.8 for a structured fallback) multiplied down by the warnings. A biography or a non-article multiplies it by zero. Every number sits in ScoringWeights::DEFAULTS and can be overridden.
$extractor = new Extractor((new ExtractorOptions())->withOwnHosts(['www.my-site.example']));
$article = $extractor->extract($html, 'https://www.daily.example/city/story-1001');
if (!$article->isReadable()) {
// Not an article, too short, or not confident enough: hold it for a human.
}
robots.txt with no off switch, plus robots meta tags and X-Robots-Tag, and takes a per-host rate-limit slot.429 or 503 with Retry-After blocks the host. Nothing is retried on its own. No cookies are stored or sent.script_rendered.javascript: URLs. The README still says to run your own sanitiser before rendering.The independent code review found and fixed these before the merge:
Authorization header went to every redirect hop, so a redirect to another host or an https to http downgrade leaked the credential. It now goes only to the requested origin.robots.txt wildcard rule hit the regex backtrack limit, which dropped the rule and failed open. It is replaced with a linear glob match.base64, uuencode, html-entities and utf-7 were passed to mbstring, which could wipe the page or decode escaped script tags into markup. They are now ignored with a warning.Test status: this is v0.1.0. 200 PHPUnit tests run with no network, on synthetic fixtures and not on live pages. Static analysis and code style checks passed. CI has not yet run on GitHub: the commit adding the workflow stayed local because the token lacked the workflow scope. In the repo’s own timing on a quiet two-core machine, a synthetic 300 KB page extracted in about 33 ms and the 1.15 MB fixture in about 145 ms.
It needs PHP 8.1, 8.2 or 8.3 with ext-dom, ext-libxml and ext-mbstring. ext-curl is needed only for the default fetch transport, and ext-intl is optional. There are no runtime Composer dependencies.
The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/readable-extract.
composer require indianopinion/readable-extract
Without Composer, require 'path/to/readable-extract/bootstrap.php';.