# script-share: letter share by writing system, no model

script-share is a PHP library for anyone who needs to check which writing system a piece of text is in: Devanagari, Bengali, Tamil, Telugu, Latin and 19 more. It uses no model and makes no network calls. The code is at [https://github.com/IndianOpinion/script-share](https://github.com/IndianOpinion/script-share), under the MIT licence. Version v0.1.0 was released on 14 November 2026.

## The problem

A pipeline asks a language model to write in English, Hindi or Tamil. Sometimes the model answers in another language, usually because the prompt says "write in the language of the source" and the source was Telugu or Kannada. The reply is fluent, well formed and wrong, and nothing downstream notices. The same failure appears as a translated headline over an untranslated body, and as one Indian script bleeding into text meant for another.

The cheap test is the writing system, since each script has its own Unicode script property. The naive version is easy to get wrong. The naive version fails in three ways:

- Dividing by every character, so spaces, digits and punctuation dilute the share. On an example Hindi news sentence the Devanagari share over all characters was 0.70, which forced an absurdly low threshold.

- Counting only letters. Indic vowel signs and the virama are combining marks, not letters. In the same example sentence 37 percent of the Devanagari code points were marks, so ignoring them makes Indic text look about a third less Indic.

- Treating a regex failure as zero. PHP's PCRE functions return false on invalid UTF-8, which cast to a number is zero letters in every script. A broken reply was reported as English and passed.

## What it does

- Counts letters per script and reports shares, with letters() + ignored() === codePoints() always true.

- Reports the dominant script, or null on a tie, on no letters or under a threshold.

- Checks text against an expected script or language tag with caller-supplied thresholds, returning a verdict of pass, fail or too short, and a stable one-line reason that never contains the input text.

- Analyses text in chunks, with the same result as analysing the whole string.

- Optionally skips links, emails, mentions and hashtags, and stops after a letter limit.

## How it works

A code point counts when it is a letter, or in the default mode a combining mark, and its Unicode Script property is a real script. Digits, punctuation (including the danda), symbols, emoji and whitespace never count. Inherited marks count for the script of the code point before them.

One compiled PCRE pattern tokenises the text, with one alternative per script, so each code point is read once and nothing is rescanned. Text is processed in 16 KiB windows cut after whitespace, which keeps memory flat. The main files are Accumulator.php (buffering, windows, UTF-8 validation), Internal/Scanner.php (counting), Internal/Pattern.php (the pattern), Result.php, Check.php and LanguageMap.php (language tag to expected scripts).

Script data comes from the PCRE library PHP was built with. On PCRE2 10.40 and later the pattern asks for the Script property explicitly with \p{sc:Devanagari}, because from 10.40 a bare script name means Script_Extensions. A golden corpus of 308 lines checks that results match across builds.

use IndianOpinion\ScriptShare\Script;
use IndianOpinion\ScriptShare\ScriptShare;
use IndianOpinion\ScriptShare\Verdict;

$check = ScriptShare::checkLanguage($reply, 'hi', 0.80);
if ($check->verdict() !== Verdict::Pass) {
error_log($check->reason());
}

ScriptShare::isMostly($body, [Script::Deva, Script::Latn], 0.95);

In the repo's own benchmark on synthetic text (PHP 8.3, PCRE2 10.42), 1 MiB of mixed Hindi and English takes about 70 ms with PCRE JIT on and under 0.4 MiB of extra memory. With JIT off it takes about 200 ms.

## Guard rails

- Invalid UTF-8 throws InvalidUtf8Exception with the byte offset, or in substitute mode counts each invalid subpart as one ignored code point.

- A PCRE failure throws RegexException. It is never reported as zero letters.

- Thresholds outside 0 to 1, NaN and unknown languages throw InvalidArgumentException. They are never clamped.

- check() has no default minShare, on purpose: every caller's tolerance for quoted English differs.

- An empty string has no dominant script, not Latin, so broken replies cannot pass an English check.

- No exception message or log line contains the input text.

## What review changed

- The build had turned skipLinks on by default. The PRD fixes it as off, and on by default also dropped Hindi hashtags (one test string gave 5 Devanagari letters instead of 9). The default is back to off.

- When a run of over 16 KiB had no whitespace, a link straddling the forced cut leaked up to 18 letters as Latin. Forced cuts now move back before any ASCII link prefix, and a new test covers every offset near the cut for five link kinds.

- The PCRE failure test passed or failed depending on test order, because an already-compiled JIT pattern ignored the changed setting. It now runs in a separate process.

- The README listed the wrong Unicode version for PCRE2 10.45 onward, and its speed claim ignored the JIT-off case. Both were corrected.

## Limits and what is next

- It reports scripts, not languages. Hindi, Marathi, Nepali and Sanskrit all look like Devanagari, and romanised Hindi is 100 percent Latin.

- Input is plain text, with no normalisation. Counting is by code point, not grapheme cluster.

- Scripts outside the 23 named ones are grouped as Zzzz (Armenian, Georgian and so on).

- A link that starts inside a run of more than 256 ASCII link characters in a stretch of over 16 KiB without whitespace can still be split and counted as letters.

- There are no open issues on the repo.

Test status: this is v0.1.0. 198 PHPUnit tests (3,454 assertions) pass, with PHPStan level 8 and phpcs clean, on fixed corpora and stand-in inputs. It has not run on a live site. Only PHP 8.3 was run. PHP 8.1 and 8.2 are covered by a CI workflow that is parked on a separate branch, so CI has not yet run on GitHub.

## Install

Requires PHP 8.1 or newer with ext-mbstring and PCRE with Unicode support. No ext-intl, no runtime dependencies.

The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/script-share.

composer require indianopinion/script-share

Without Composer, copy the repository (or just src/) and require src/bootstrap.php.

[All 47 utilities](https://indianopinion.org/blog/open-source/) · [script-share on GitHub](https://github.com/IndianOpinion/script-share)

Canonical: https://indianopinion.org/blog/open-source/script-share/

AI disclosure: Every summary and comment here is written by AI and published automatically, without a person reviewing each article. We do no original reporting. How this works: https://indianopinion.org/ai-use-policy/
