Text is not ASCII: what Indian scripts taught our PHP

Seven assumptions about text that break on Devanagari, Tamil and Bengali, with the code points that break them, and how our open-source PHP utilities handle each one, with and without ext-intl.

Indian Opinion has published in English only since September 2026. The sources we read are not all in English, and neither are the readers who comment. All of it passes through PHP whose defaults assume ASCII.

Several of our open-source utilities exist because of that gap. This post collects their shared lessons, one wrong assumption at a time, with the code points that break each one.

The utilities, part of the 47 we published, were specified, built, reviewed and released between 11 and 19 November 2026 by language-model coding agents working under the editor’s direction, in five separate roles: a spec writer, a spec reviewer, a builder that saw only the spec, an independent code reviewer, and a releaser. The editor made the rulings on every open question. Every repo named here is at v0.1.0 and tested on stubs, mocks and a throwaway MySQL. None has run its CI on GitHub yet, and some have not run on a live site.

The samples below ran under php -r on PHP 8.3 with PCRE2 10.42 and ICU 74.2.

A character is a byte

UTF-8 spends one byte on an ASCII letter and three on every Devanagari, Bengali or Tamil code point. Anything that counts with strlen or cuts with substr is wrong by a factor of three.

Take the word हिन्दी. It is six code points: U+0939 HA, U+093F VOWEL SIGN I, U+0928 NA, U+094D VIRAMA, U+0926 DA, U+0940 VOWEL SIGN II.

$s = "हिन्दी"; // U+0939 U+093F U+0928 U+094D U+0926 U+0940
echo strlen($s), ' ', mb_strlen($s, 'UTF-8'), ' ', grapheme_strlen($s), "\n";
// 18 6 2

Eighteen bytes, six code points, two things a reader sees. substr($s, 0, 4) leaves invalid UTF-8.

Invalid UTF-8 then breaks the next layer quietly. preg_match_all with the u flag returns false on it, and false / $n is 0 in PHP. In script-share that turned broken input into a pass, so the library now throws InvalidUtf8Exception with the byte offset. simhash-unicode does the same through TextNormalizer::assertUtf8().

Bytes also go wrong before the text reaches PHP. A page that declares windows-1252 but sends UTF-8 turns न (bytes E0 A4 A8) into न. In readable-extract, Charset::toUtf8() reads such a page as UTF-8 when its bytes are valid UTF-8, and records a charset_mismatch warning. Bytes that are still invalid become U+FFFD and are counted.

Without ext-intl: nothing here needs it. Encoding checks and conversion use mbstring.

A character is a code point

Indic scripts write most vowels as combining marks. In हिन्दी, U+093F and U+0940 are vowel signs and U+094D is a virama. Unicode puts them in General Category M, not L, and no reader counts them as characters.

That matters twice. First, a count of \p{L} drops them. In the sentence “RBI ने बुधवार को रेपो दर में कोई बदलाव नहीं किया।”, script-share counts 35 Devanagari letters with marks and 21 without. Counting letters alone makes a Hindi sentence look 87.5 per cent Devanagari instead of 92.1. ScriptShare::analyse() counts a mark with its script by default (MarkMode::Count), and divides by letters only, so spaces, digits and the danda (U+0964) do not dilute the share.

Second, length limits. A headline limit of 70 characters means 70 things the reader sees. In headline-options, the Length class counts grapheme clusters with grapheme_strlen() when ext-intl is loaded. A candidate over the limit is dropped, never cut.

Grapheme counts are not stable across ICU versions either. ICU 74 (Unicode 15.1, rule GB9c) keeps न्द together, so हिन्दी is two clusters. Older ICU gives three. simhash-unicode adds one rule on top, Graphemes::mergeConjuncts(): a cluster ending in a joining virama merges with the next letter. The Tamil pulli (U+0BCD) is left out on purpose, because Tamil writes the dead consonant as its own visible letter. தமிழ் (U+0BA4 TA, U+0BAE MA, U+0BBF VOWEL SIGN I, U+0BB4 LLLA, U+0BCD PULLI) stays three clusters: த, மி, ழ்.

Without ext-intl: headline-options counts code points minus marks and format characters (\p{M} and \p{Cf}), which gives 3 for हिन्दी and 3 for தமிழ். simhash-unicode’s Graphemes::fallback() attaches marks and joiners to the character before them, then applies the same conjunct rule, so both paths give the same tokens. script-share never needs ext-intl: the Script property comes from PCRE.

Words are separated by spaces, and letters are a to z

Indic scripts do put spaces between words. The bug is the character class written for “a word”. Splitting on [^a-z0-9] leaves nothing at all, and splitting on \p{L} cuts words apart at their vowel signs.

$t = "भारत में मानसून";
echo count(preg_split('/[^a-z0-9]+/i', $t, -1, PREG_SPLIT_NO_EMPTY)), "\n";
echo implode(' | ', preg_split('/[^\p{L}\p{N}]+/u', 'मानसून', -1, PREG_SPLIT_NO_EMPTY)), "\n";
echo implode(' | ', preg_split('/[^\p{L}\p{M}\p{N}]+/u', $t, -1, PREG_SPLIT_NO_EMPTY)), "\n";
// 0
// म | नस | न
// भारत | में | मानसून

In मानसून, U+093E (VOWEL SIGN AA) is a spacing mark (Mc) and U+0942 (VOWEL SIGN UU) is a non-spacing mark (Mn), so a \p{L} split breaks the word at both.

The first line is the bug that started simhash-unicode. In our own use, an ASCII-only tokeniser gave every Devanagari, Telugu and Tamil title zero tokens. They all hashed to 0000000000000000 and matched each other at distance 0. The library’s Tokenizer splits on letters, digits and marks by script, and a text with no tokens gets an empty fingerprint that isNearDuplicate() never matches (reason empty).

For word counts, readable-extract’s WordCounter splits on anything that is not a letter, mark, digit, ZWNJ or ZWJ. Scripts written without spaces (Han, Kana, Thai, Lao, Khmer, Myanmar) go to IntlBreakIterator. The same “not a letter in any script” edge defines a whole word in comment-draft-replies: Text::matchesBlocklist() uses (?<![\p{L}\p{N}\p{M}]) rather than \b. On PCRE2 10.42 a vowel sign is not a word character, so /\bमानस\b/u matches inside मानसून.

Encoding matters too. readable-extract reads Hindi and Marathi month names in visible dates. फ़रवरी (February) can arrive with a precomposed U+095E or as U+092B PHA plus U+093C NUKTA, and NFC turns the first into the second. The month table in Dates lists both, so a match does not depend on whether normalisation ran.

Without ext-intl: readable-extract counts every letter of a no-space script as one word and reports the method as characters; simhash-unicode uses grapheme bigrams for those scripts on either path. Without NFC, simhash-unicode adds -nonorm to its configuration id, so fingerprints from hosts that normalise differently are never compared.

Uppercase marks a name

A common check on a generated headline is “every name in it appears in the article”, and names are found by capital letters. Devanagari, Bengali, Tamil and Arabic have no case. “दिल्ली में बारिश” has no \p{Lu} code point at all, and mb_strtoupper() returns it unchanged. A capital-letter rule finds no names and passes.

headline-options treats that as its own outcome. Headline::isMostlyCaseless() is true when more than half the letters are \p{Lo} (letters with no case). EntitySupportRule then uses only Latin-script words in the headline, the post’s tags and categories, and a list of known entities. If none of them finds a name, the rule returns na and leaves the total alone. Numbers::digitTokens() maps native digits such as ४ (U+096A) to ASCII and drops grouping, so 1,00,000 and 100000 compare equal.

Without ext-intl: no change. The General Category comes from PCRE.

Four characters make a token

“About four characters per token” is a fair rule for English. Non-Latin scripts use far more tokens per character, so a budget built on length over four undercounts them, which is the dangerous direction for a spending cap.

In post-embeddings, TokenEstimator::estimate() counts characters per script group and divides each by its own ratio: 4.0 for Latin, 2.5 for Cyrillic and Greek, 2.0 for Arabic and Hebrew, 1.5 for Indic and Southeast Asian scripts, 1.0 for CJK. For “भारत में मानसून समय से पहले पहुँचा” (34 code points), a flat divide-by-four gives 9; the estimator gives 21. The ratios pass through a filter, and the estimate is used only when the provider reports no usage.

audio-to-post estimates before any transcription is paid for, from minutes of audio, so it works per word. TokenRatios defaults to 1.3 tokens per word for Latin, 2.5 for Indic and Arabic scripts, and 1.0 per character for Chinese, Japanese and Korean. The operator tunes them against actual usage. Its quote checks keep \p{M} when they strip punctuation, so vowel signs survive.

Without ext-intl: the estimates need nothing from it. audio-to-post skips NFC and compares text as given.

Strip the invisible characters

Zero-width characters can hide text from a reviewer and break the markers that fence reader input inside a prompt. The tempting fix is to strip the whole U+200B to U+200F block. That block includes U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER, which Indic scripts use to choose a visible form.

क्‍ष written as U+0915 KA, U+094D VIRAMA, U+200D ZWJ, U+0937 SSA shows a half KA beside SSA. Without the ZWJ the same letters render as the conjunct क्ष. In Bengali, র‍্যাব (U+09B0 RA, U+200D, U+09CD VIRAMA, U+09AF YA, U+09BE, U+09AC) uses a ZWJ so the RA keeps its full form instead of becoming a reph. Strip the joiner and you change the word on screen.

$word = "\u{0915}\u{094D}\u{200D}\u{0937}"; // ka, virama, ZWJ, ssa
$naive   = preg_replace('/[\x{200B}-\x{200F}\x{FEFF}]/u', '', $word);
$careful = preg_replace('/[\x{200B}\x{200E}\x{200F}\x{FEFF}]/u', '', $word);
var_dump($naive === $word, $careful === $word);
// bool(false)
// bool(true)

comment-draft-replies puts reader comments into a prompt, so it must clean them. Generation\Prompt::clean() drops U+200B, the direction marks and controls, the BOM and most C0 controls, but keeps ZWNJ and ZWJ. It also breaks double brackets with joiners hidden between them, so a reader cannot fake a data block marker. A unit test checks that "क्\u{200D}ष" comes back unchanged.

The output side does not yet match. Text::hasInvisible(), which Generation\Validator runs on a model’s reply, still uses the range U+200B to U+200F. A correct reply that needs a joiner is rejected as invisible and retried, and the publish step warns about the same text. We found this while writing this post; it is a gap in v0.1.0, not a design choice.

Other tools make the opposite choice for good reasons. simhash-unicode strips joiners inside words by default (joiners: strip), because two spellings of one word should hash alike. It first maps the older Malayalam chillu encoding (consonant, virama, ZWJ) to the atomic chillu letter. ascii-slugs drops joiners too, because its output is ASCII.

The reply is in the language you asked for

A model asked for Hindi can answer in English, in Marathi, or half in each, and nothing raises an error.

ScriptShare::checkLanguage($reply, 'hi', 0.80) returns a Check holding a Verdict and a reason such as “fail: Deva 0.00 < 0.80 (dominant Latn, 43 letters)”. It reports scripts, not languages: Devanagari serves Hindi, Marathi and Nepali. “No letters” is its own answer, never Latin by default.

translate-sidecar is for sites that publish translations. Our own site stopped its machine-translated editions in September 2026. Its Guards class fails the whole job, storing nothing, when any call is truncated, drops a segment or breaks a placeholder. checkUnchanged() catches the partial translation that returns some paragraphs still in the source language. checkScript() needs 70 per cent of a title’s letters and 60 per cent of a body’s in the target script, at most 5 per cent in a third script. On the page, Head::hreflangLinks() prints the same cluster on every member, including the page itself and x-default, and a translation’s canonical points at itself.

comment-draft-replies is softer on purpose. A Hindi comment with English words is normal, so Text::scriptMismatch() only flags a reply whose dominant script holds less than 20 per cent of the comment’s letters, and texts under five letters never flag. The flag is script_mismatch for the editor to see; it never blocks.

Without ext-intl: script-share needs none. translate-sidecar applies NFC only when Normalizer exists and otherwise counts the text as given.

A slug can hold anything

WordPress percent-encodes a slug made from a non-Latin title, and the post_name column holds 200 bytes. Every Devanagari code point costs nine bytes once encoded. “चंद्रयान मिशन की सफलता पर देश में जश्न” is 38 code points and 286 encoded bytes, so core stores a slug cut at 200 bytes, about 22 code points and usually mid-word. Nothing errors.

ascii-slugs hooks sanitize_title at priority 9, in the save context only, and only when core would percent-encode something. Slugger::slug() splits the title into runs of one script and transliterates each with a built-in table that handles the inherent vowel, matras, virama conjuncts, anusvara and nukta letters. The title above becomes chandrayan-mishan-ki-saphalata-par-desh-men-jashn. தமிழ் becomes tamizh, and বাংলা সংবাদ becomes bangla-sangbad. A result with fewer than three letters or digits, or with under half the title’s letters transliterated, falls back to a dated hash such as post-20261112-459eae08. Uniqueness always finishes with core’s own function.

Existing URLs do not move. A stored encoded slug is pure ASCII (%e0%a4...), so it takes the fast path untouched, and the plugin never acts in the query context, so old encoded URLs keep resolving. Repair is a separate WP-CLI command that dry-runs by default; with --yes it saves through wp_update_post(), so core redirects the old slug with a 301.

Without ext-intl: this was the original failure. When transliteration relied on intl alone, every title on a host without it went to the hash. Built-in tables now cover the main Indic scripts, Urdu, Cyrillic and Greek. In the default auto mode, ICU is used only for scripts with no table, so a slug comes out the same on every host.

Search Trending
Loading

Signing-in 3 seconds...

Signing-up 3 seconds...

All fields are required.