simhash-unicode: 64-bit simhash for short text in any script

A PHP library for 64-bit simhash fingerprints of short text, with Hamming distance and near-duplicate search. It tokenises every script and refuses to call two texts near when there is too little signal.

simhash-unicode is a PHP library that turns a short text, such as a headline, into a 64-bit simhash fingerprint. It measures the Hamming distance between fingerprints and finds near-duplicates in a list. It is for anyone who needs to spot the same story arriving from several sources, in any script. The code is at https://github.com/IndianOpinion/simhash-unicode, under the MIT licence, and v0.1.0 was released on 12 November 2026.

It has no runtime dependencies and needs PHP 8.1 or later. We wrote it from scratch. It is independent of and not affiliated with any third-party plugin it integrates with.

The problem

Most PHP simhash code splits text on anything that is not a-z or 0-9. That works for English and fails silently for everything else. An ASCII-only tokeniser gives every Devanagari, Telugu and Tamil title zero tokens. All of them hash to 0000000000000000 and sit at distance 0 from each other, so unrelated stories fall into one group.

Nothing raised an error. The hash looked valid, and the distance looked like a perfect match. The bug shows up only as wrong grouping downstream. The library fixes this at the root. Tokens are letters, digits and combining marks in any script. An empty token set is reported as empty, not as a real fingerprint. A near-duplicate check refuses to match on too little signal.

What it does

  • Hashes a short text in any script Unicode encodes, to hex, a signed integer or an unsigned decimal string.
  • Keeps combining marks inside their syllable, so an Indic syllable is never split.
  • Reports empty input as an empty fingerprint that never matches anything.
  • Answers isNearDuplicate() with a reason: near, far, empty, config_mismatch or too_short.
  • Finds near pairs and groups in a list, with a band index for speed (Batch).
  • Offers word shingles, character n-grams, stop words and token weights, all caller-supplied. The package ships no stop-word lists.
  • Prints a full explanation of one hash: tokens, features, weights, the 64 sums, flags and the library versions seen.

How it works

A hash goes through these steps:

  1. Validate UTF-8. Invalid input raises InvalidInputException and names the byte offset. A regex over invalid UTF-8 fails, and naive code reads that as “no tokens”, which is exactly the zero-hash bug.
  2. Clean: control characters become separators, and invisible joiners and marks are removed without splitting the word.
  3. Normalise: Unicode form (NFKC by default), full case folding, then the form again.
  4. Tokenise: a token is a run of letters, digits and combining marks. A token also ends where one letter script meets another. Scripts written without spaces, such as Han, Thai and Myanmar, become grapheme bigrams.
  5. Build features: each token, plus optional shingles and character n-grams cut on graphemes, so a conjunct is never split.
  6. Hash each feature with XXH64 and, for each of the 64 bits, add the weight where the hash has a 1 and subtract it where it has a 0. A bit is 1 when its sum is above zero.

Every setting that changes the bits feeds a configuration id, a short string such as shu1-57555a8696b2, stamped on each fingerprint. Comparing fingerprints made under different ids returns config_mismatch instead of a quiet wrong answer.

From the README:

$simhash = new Simhash();
$a = $simhash->hash('भारत में मानसून समय से पहले पहुँचा, किसान बुवाई की तैयारी में');
$b = $simhash->hash('भारत में मानसून समय से पहले पहुंचा। किसान बुवाई की तैयारी में');

$match = $simhash->isNearDuplicate($a, $b);
var_dump($match->isNear);   // true
echo $match->reason;        // near
echo $match->distance;      // 5

Store the fingerprint with its configuration id and token count (toArray() and Fingerprint::fromArray()), so a later comparison can apply the same guards. The default threshold is 6, with minTokens at 3.

Guard rails

  • Fewer than minTokens word tokens (default 3) gives too_short, whatever the distance.
  • Input is capped at 4096 bytes and 2048 features by default. Over-long input is cut at a grapheme boundary and flagged, or it throws.
  • Without ext-intl, normalisation is skipped, a flag is set and -nonorm is added to the configuration id. Those fingerprints never match ones from a host with intl.
  • A weight callback requires a weightsId, because a closure cannot be identified by value.
  • The band index is exact up to threshold 3 and a candidate filter above it. It refuses thresholds over 16. Batch in auto mode uses it only where it is exact.
  • Case folding does not depend on setlocale().

What review changed

The reviewer pushed five commits before the merge. The main changes:

  • Hostile input was not linear. Truncation was quadratic over a run of combining marks, and at the largest input cap it ran for more than ten minutes. The UTF-8 offset scan used one unbounded regex that, without PCRE’s JIT, hit the backtrack limit, named the wrong byte and made scrub replace valid text. scrub and the e-mail pattern were quadratic as well. All are now linear or bounded, with near-1 MB hostile tests.
  • The built-in grapheme rule disagreed with ICU for Thai and Lao sara am, halfwidth katakana sound marks and Hangul jamo. Four Thai fixtures hashed differently on hosts with and without intl under equal configuration ids. The rule was fixed, and a test now hashes the whole corpus through both paths. One gap is documented.
  • Configuration ids did not fail closed. Two different weight callbacks could share one id and compare as equal. A callback without a weightsId is now rejected. Weights in the id no longer depend on the precision ini settings.
  • Fourteen fixture titles read like real headlines, with real teams, brands or organisations. They were replaced with invented ones. The README performance section dropped an unmeasured claim and now states the measured numbers with their variance.

Limits and what is next

  • Short text is weak. Under about five tokens, one changed word moves the fingerprint a long way. minTokens stops false matches but does not create recall.
  • Paraphrase and translation are not detected. Use embeddings for that.
  • Numbers: two titles that differ only in a figure are near by design. digits: mask makes them identical, and if numbers matter you compare them separately.
  • Boilerplate such as “Live updates:” pulls unrelated titles together. Remove it with stop words or weights.
  • A fingerprint is not anonymised. A short text can be guessed by brute force from it.
  • Brute-force search is quadratic. The benchmark in the README comes from one loaded two-vCPU VM and varies about two times, so measure on your own hardware.

Test status: this is v0.1.0, tested on stubs and a self-written fixture corpus, not on a live site. At review, 193 tests passed with PHPStan level 8 and phpcs clean. Without intl, 184 passed and 9 were skipped. CI has not yet run on GitHub, because the token lacked the scope to push it. There are no open issues.

Install

The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/simhash-unicode.

composer require indianopinion/simhash-unicode

It needs PHP 8.1 or later and ext-mbstring. ext-intl is strongly recommended, and a 64-bit build is recommended. Without Composer, require bootstrap.php from a release download.

All 47 utilities · simhash-unicode on GitHub

Search Trending
Loading

Signing-in 3 seconds...

Signing-up 3 seconds...

All fields are required.