# gnews-unwrap: turn Google News links into publisher URLs

gnews-unwrap is a PHP library that turns Google News wrapper links back into the publisher's own article URLs. It also gives you an outlet key and an article key, so attribution and duplicate checks work on the real source. It is for anyone who ingests Google News feeds. The code is at [https://github.com/IndianOpinion/gnews-unwrap](https://github.com/IndianOpinion/gnews-unwrap), under the MIT licence, and v0.1.0 was released on 12 November 2026.

It has no runtime dependencies and needs PHP 8.1 or later. We wrote it from scratch. It is independent of and not affiliated with any third-party plugin it integrates with.

## The problem

Google News feeds do not link to articles. Each item links to a Google wrapper page. A system that stores that link as the item's source gets three things wrong at once:

- Attribution. Every item is credited to Google News, so counts of distinct outlets per story collapse onto one fake outlet.

- De-duplication. The same article arriving from a direct publisher feed and from a Google News feed has two different URLs, so URL-based dedup never matches them.

- Reader links. A source link shown to readers lands on an interstitial that needs JavaScript, consent or both.

A smaller fault sits beside these. Outlets serve one article from several hosts, such as www., amp. and m.. Counted as raw host strings, one outlet counts twice. Rows already stored with wrapper links also need repairing, which the library supports.

## What it does

- Detects eight known wrapper shapes (S1 to S8) from the URL alone, with no network.

- Decodes legacy article ids offline.

- Resolves current, opaque ids over HTTP, within a time budget, with a circuit breaker and a cache.

- Returns a Result that says what was tried, what it cost and, when it fails, a reason code.

- Builds an outlet key from a host (www, amp, m and mobile labels stripped, never past the registrable domain) and an article key from a URL (tracking parameters, fragments and trailing slashes dropped).

- Leaves any URL that is not a wrapper unchanged, with the reason not_google_news, and no request. It is safe to run every stored URL through it.

## How it works

For each link the Unwrapper runs a chain of methods. The names are cache, query_param, offline_decode, batch_execute, http_redirect and html_parse, and each can be switched off.

- Detect the shape. Pure, no network.

- Look in the cache, keyed on the normalised wrapper.

- For the shapes that carry the target in a url or q parameter, read it. Nested redirectors unwrap up to three levels.

- For legacy ids, decode the base64 protobuf message whose field 4 is the publisher URL. The parser is bounded on every length. Ids issued since July 2024 hold an opaque token and go to the network.

- Fetch the article page, read its signature and timestamp, and post them to an undocumented internal endpoint. This is isolated in one class, BatchExecute. It reports format_changed when the answer changes shape.

- Follow Location headers between Google hosts, and stop at the first hop that leaves Google. The publisher is never requested.

- Read an interstitial page already fetched.

Each link's method chain is a PHP generator that yields a request when it needs the network. resolve() drives one generator. resolveMany() can drive several through curl_multi. Both paths therefore share one piece of resolution logic.

The usage from the README:

$unwrapper = new Unwrapper();
$result = $unwrapper->resolve($link);

if ($result->resolved) {
$url = $result->output; // publisher URL
$outlet = $result->outletKey;
$key = $unwrapper->articleKey($url);
} else {
// $result->output is the wrapper, $result->reason says why.
}

Unwrap at ingest, before any hash, key, outlet count or dedup check. Store the resolved URL, and do not re-resolve on read.

## Guard rails

- Never throws for bad input. Only programming errors throw: a bad setting, or a transport that returns the wrong type.

- Fail open on the item, closed on the target. Every candidate must be an absolute http or https URL, not a Google host, not one of your own hosts, not an IP literal or private address, at most 2048 characters, with no control characters, and not the wrapper itself. In doubt, the wrapper comes back.

- Defaults: 3 second connect timeout, 8 second request timeout, 15 second budget per link, 5 redirects at most, 512 KB read per response.

- A 429 or a redirect to a rate-limit page stops network methods for that link. Three of them in 60 seconds open a circuit breaker for 5 minutes. Offline methods keep running while it is open.

- Resolved results are cached for 30 days and unresolved results for one hour. Transient reasons such as rate_limited, timeout and circuit_open are never cached.

- Both transports verify TLS with no switch to turn it off, keep no cookies, never follow redirects themselves and cut the body at the byte cap.

- The consent form is never posted and consent cookies are never forged.

- An optional generic mode follows redirects for any URL, with checks on every hop that refuse private, loopback and link-local addresses.

## What review changed

The reviewer called the build solid and fixed these directly before the merge:

- Two defaults had drifted from the PRD. Concurrency now defaults to 1, and AMP-marker stripping in article keys to off. Both remain opt-in.

- The address guard for generic mode used filter_var flags, whose coverage differs by PHP version. It now uses explicit IPv4 and IPv6 range tables, checks embedded IPv4 addresses in mapped, NAT64 and 6to4 forms, and looks up AAAA records as well as A records. Before, a host with only a loopback AAAA record passed. Hops with userinfo or a backslash are now refused, because parsers disagree on which host they reach.

- A concurrent transport whose sendMany() threw escaped resolveMany(). It now fails that round's requests with network_error.

- A cached positive result is now checked again against today's acceptance rules. If an own host was added since, or the shared store was tampered with, the entry is deleted and treated as a miss.

## Limits and what is next

- Current ids need the network. No offline decode for them is known, checked on 11 November 2026. Resolving one costs about a second and two requests.

- The batch endpoint is undocumented and may break. When it does you see format_changed or signature_missing, and the link comes back unresolved, not wrong. Complying with Google's terms is your responsibility, and you can switch the method off.

- Consent walls. Where Google sends requests to a consent page, network methods return consent_wall. A live check on 11 November 2026 hit this. Run network resolution from a host outside those regions.

- Bulk resolution draws rate limits. Use pauseMs and keep concurrency low.

- Generic mode cannot fully stop DNS rebinding. Do not enable it on untrusted input without an egress firewall.

Test status: this is v0.1.0. At review, 196 tests ran on PHP 8.3 against recorded fixtures and a fake transport (no live site), with PHPStan level 8 and phpcs clean. A live smoke test is opt-in. The README carries a CI badge, but the workflow has not run on GitHub yet, because the token lacked the scope to push it, so PHP 8.1 and 8.2 are untested. There are no open issues.

## Install

The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/gnews-unwrap.

composer require indianopinion/gnews-unwrap

It needs PHP 8.1 or later. ext-curl is used when present, with a fallback to PHP streams, and ext-intl is used for IDN hosts when present. Without Composer, require bootstrap.php from a release download.

[All 47 utilities](https://indianopinion.org/blog/open-source/) · [gnews-unwrap on GitHub](https://github.com/IndianOpinion/gnews-unwrap)

Canonical: https://indianopinion.org/blog/open-source/gnews-unwrap/

AI disclosure: Every summary and comment here is written by AI and published automatically, without a person reviewing each article. We do no original reporting. How this works: https://indianopinion.org/ai-use-policy/
