simhash-unicode is a PHP library that turns a short text, such as a headline, into a 64-bit simhash fingerprint. It measures the Hamming distance between fingerprints and finds near-duplicates in a list. It is for anyone who needs to spot the same story arriving from several sources, in any script. The code is at https://github.com/IndianOpinion/simhash-unicode, under the MIT licence, and v0.1.0 was released on 12 November 2026.
It has no runtime dependencies and needs PHP 8.1 or later. We wrote it from scratch. It is independent of and not affiliated with any third-party plugin it integrates with.
Most PHP simhash code splits text on anything that is not a-z or 0-9. That works for English and fails silently for everything else. An ASCII-only tokeniser gives every Devanagari, Telugu and Tamil title zero tokens. All of them hash to 0000000000000000 and sit at distance 0 from each other, so unrelated stories fall into one group.
Nothing raised an error. The hash looked valid, and the distance looked like a perfect match. The bug shows up only as wrong grouping downstream. The library fixes this at the root. Tokens are letters, digits and combining marks in any script. An empty token set is reported as empty, not as a real fingerprint. A near-duplicate check refuses to match on too little signal.
isNearDuplicate() with a reason: near, far, empty, config_mismatch or too_short.Batch).A hash goes through these steps:
InvalidInputException and names the byte offset. A regex over invalid UTF-8 fails, and naive code reads that as “no tokens”, which is exactly the zero-hash bug.Every setting that changes the bits feeds a configuration id, a short string such as shu1-57555a8696b2, stamped on each fingerprint. Comparing fingerprints made under different ids returns config_mismatch instead of a quiet wrong answer.
From the README:
$simhash = new Simhash();
$a = $simhash->hash('भारत में मानसून समय से पहले पहुँचा, किसान बुवाई की तैयारी में');
$b = $simhash->hash('भारत में मानसून समय से पहले पहुंचा। किसान बुवाई की तैयारी में');
$match = $simhash->isNearDuplicate($a, $b);
var_dump($match->isNear); // true
echo $match->reason; // near
echo $match->distance; // 5
Store the fingerprint with its configuration id and token count (toArray() and Fingerprint::fromArray()), so a later comparison can apply the same guards. The default threshold is 6, with minTokens at 3.
minTokens word tokens (default 3) gives too_short, whatever the distance.ext-intl, normalisation is skipped, a flag is set and -nonorm is added to the configuration id. Those fingerprints never match ones from a host with intl.weightsId, because a closure cannot be identified by value.Batch in auto mode uses it only where it is exact.setlocale().The reviewer pushed five commits before the merge. The main changes:
scrub replace valid text. scrub and the e-mail pattern were quadratic as well. All are now linear or bounded, with near-1 MB hostile tests.weightsId is now rejected. Weights in the id no longer depend on the precision ini settings.minTokens stops false matches but does not create recall.digits: mask makes them identical, and if numbers matter you compare them separately.Test status: this is v0.1.0, tested on stubs and a self-written fixture corpus, not on a live site. At review, 193 tests passed with PHPStan level 8 and phpcs clean. Without intl, 184 passed and 9 were skipped. CI has not yet run on GitHub, because the token lacked the scope to push it. There are no open issues.
The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/simhash-unicode.
composer require indianopinion/simhash-unicode
It needs PHP 8.1 or later and ext-mbstring. ext-intl is strongly recommended, and a 64-bit build is recommended. Without Composer, require bootstrap.php from a release download.