readable-extract: article body extraction with confidence

A plain PHP library that extracts the body, title, authors, dates and lead image from a news page, says where each field came from, and refuses to pass off a biography or a section front as an article.

readable-extract is a PHP library that extracts the article body, title, authors, dates and lead image from a news page. It reports where each field came from and how sure it is. It treats Indic scripts as first-class. It is for developers who handle news links: importers, newsroom tools, aggregators and archivists. The code is at https://github.com/IndianOpinion/readable-extract, licensed MIT. Version v0.1.0 was released on 15 November 2026.

We wrote it from scratch from an idea. The scoring heuristic is our own and is not a port of an existing library. It has not run on a live site. The 47 utilities in this series were specified, built, reviewed and released between 11 and 19 November 2026 by language-model coding agents under the editor’s direction, in five roles: a spec writer, a spec reviewer, a builder that saw only the spec, an independent code reviewer and a releaser. The editor made the rulings on every open question.

The problem

Importers usually take whatever text a feed or a generic scraper hands them. On news sites that text is often wrong in ways that are hard to see:

  • The wrong block wins. A short author biography sits before the article in the markup and a naive extractor returns it as the body.
  • The lead image is a spacer. Lazy-loading pages put a one-pixel placeholder in src and the photo in data-src or srcset.
  • Video pages, playlists, galleries and section fronts carry a headline and some text, and look like articles.
  • Dates and bylines drift between visible text and structured data, and bylines mix people, desks and wire agencies.
  • Nobody notices regressions when a site changes its template.

The PRD’s conclusion is that a wrong body is worse than an empty one. An empty body is noticed and retried. A wrong body is trusted and repeated.

What it does

  • Extracts offline from HTML you already hold. Extraction never touches the network.
  • Returns an immutable Article with a page type (article, liveblog, video, gallery, index, paywalled, script_rendered, wrapper, empty or unknown), body as allowlisted HTML and plain text, authors, dates in UTC, lead image, canonical URL, language and detected script.
  • Records the source of each field (json_ld, microdata, meta, visible or none), a per-field confidence and an overall confidence from 0 to 1.
  • Reports warnings such as biography_only, not_an_article and visible_body_short.
  • Offers an optional fetcher and a command line with extract, score, baseline and fixture commands.

How it works

Extractor runs the steps in order. It decodes the character set (byte-order mark, declared charset, meta tags, then UTF-8), parses with libxml’s HTML parser, reads JSON-LD, microdata and meta tags before anything is removed, classifies the page, finds the body, cleans it and then fills each field from a fixed source order. No field is found by running a regular expression over the whole page.

Body scoring walks the DOM. A paragraph of at least 25 characters scores one point, plus one per comma, half a point per sentence mark (the danda counts) and up to three for length. The parent gets the full score and the grandparent half. Class and id hints add or subtract 25, and the result is multiplied by one minus the link density. The visible body wins when it meets the prose minimum of 400 characters. Otherwise a structured body wins. Two long bodies that disagree are never merged.

The confidence is a base from the body (1.0 when a structured body confirms the visible one, 0.9 for a visible body that meets the minimum, 0.8 for a structured fallback) multiplied down by the warnings. A biography or a non-article multiplies it by zero. Every number sits in ScoringWeights::DEFAULTS and can be overridden.

$extractor = new Extractor((new ExtractorOptions())->withOwnHosts(['www.my-site.example']));
$article = $extractor->extract($html, 'https://www.daily.example/city/story-1001');

if (!$article->isReadable()) {
    // Not an article, too short, or not confident enough: hold it for a human.
}

Guard rails

  • Fetching happens only when you call the fetcher. It checks scheme and port, resolves the host and refuses private, loopback, link-local, multicast, reserved and cloud metadata addresses, on every redirect hop. It honours robots.txt with no off switch, plus robots meta tags and X-Robots-Tag, and takes a per-host rate-limit slot.
  • A 429 or 503 with Retry-After blocks the host. Nothing is retried on its own. No cookies are stored or sent.
  • The user agent must name your product and a contact. A browser-style agent string is refused.
  • It does not bypass paywalls, logins, consent walls or bot challenges, and it does not run JavaScript. Pages that render only in script are reported as script_rendered.
  • Body HTML is allowlisted, with no scripts, event handlers, iframes or javascript: URLs. The README still says to run your own sanitiser before rendering.
  • Extracted text belongs to its publisher. The library is not a licence to republish.
  • Aggregator wrapper links are recognised and refused, not resolved. That job belongs to gnews-unwrap.

What review changed

The independent code review found and fixed these before the merge:

  • A comment thread outscored the article. A paragraph inside a container with more negative than positive hints still scored, so 25 short comments could beat a four-paragraph article. Such paragraphs no longer score, and the walk reruns without the rule when nothing is left, so a biography-only page is still found and refused.
  • The Authorization header went to every redirect hop, so a redirect to another host or an https to http downgrade leaked the credential. It now goes only to the requested origin.
  • A hostile robots.txt wildcard rule hit the regex backtrack limit, which dropped the rule and failed open. It is replaced with a linear glob match.
  • Charset labels such as base64, uuencode, html-entities and utf-7 were passed to mbstring, which could wipe the page or decode escaped script tags into markup. They are now ignored with a warning.

Limits and what is next

  • libxml’s parser predates HTML5. Its tree for some malformed markup differs from a browser’s.
  • DNS is checked before each request, but curl resolves again when it connects, so a host that changes its answer in between is not caught. A custom transport or an egress proxy can close the gap.
  • Month names are read in English, Hindi and Marathi. Visible dates in other scripts are read only in ISO or numeric form.
  • The 56 fixtures are synthetic pages modelled on common layouts, with reserved example hosts and original placeholder text, in eleven languages. They were written alongside the extractor, so a perfect score is a regression guard, not an accuracy claim about real pages.
  • One open issue: add fixtures captured under permissive terms, with outlet names replaced, to measure accuracy on real pages.

Test status: this is v0.1.0. 200 PHPUnit tests run with no network, on synthetic fixtures and not on live pages. Static analysis and code style checks passed. CI has not yet run on GitHub: the commit adding the workflow stayed local because the token lacked the workflow scope. In the repo’s own timing on a quiet two-core machine, a synthetic 300 KB page extracted in about 33 ms and the 1.15 MB fixture in about 145 ms.

Install

It needs PHP 8.1, 8.2 or 8.3 with ext-dom, ext-libxml and ext-mbstring. ext-curl is needed only for the default fetch transport, and ext-intl is optional. There are no runtime Composer dependencies.

The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/readable-extract.

composer require indianopinion/readable-extract

Without Composer, require 'path/to/readable-extract/bootstrap.php';.

All 47 utilities · readable-extract on GitHub

Search Trending
Loading

Signing-in 3 seconds...

Signing-up 3 seconds...

All fields are required.