packed-vectors: exact vector search in a flat file

A PHP library that stores embeddings in a packed float32 file, rebuilds it atomically, and scans it by exact cosine similarity in bounded memory, for corpora too small to justify a vector database.

packed-vectors is a PHP library for applications that keep embeddings in database rows and need to find the most similar ones. It stores them in a packed float32 file with a small header, rebuilds that file atomically, and scans it by exact cosine similarity in bounded memory. The code is at https://github.com/IndianOpinion/packed-vectors, under the MIT licence. Version v0.1.0 was released on 14 November 2026.

The problem

When embeddings live in database rows as JSON text, similarity search hits two walls as the corpus grows.

  • Memory. Fetching every vector in one result set holds each row at least twice, once in the driver buffer and once in the returned array. With enough rows of 512 dimensions, one result set can exhaust a typical memory_limit.
  • Time. Decoding a JSON vector for every row on every request is slow. Reading the same vectors as packed float32 with unpack() is far cheaper.

A vector database is often not available on shared or small hosts, and for tens of thousands of rows it is not needed. Teams that write the packed file by hand then meet the same traps: unpack() arrays that start at index 1, machine-order floats, unchecked writes that rename a truncated file into place, a trailing partial record silently dropped, and code that quietly truncates vectors of the wrong length.

What it does

  • Defines a versioned binary format: a 40-byte header, then fixed-size records of two unsigned 32-bit integers (id and ref) and dim little-endian float32 values.
  • Rebuilds the file from any iterable, streaming, in bounded memory. The result either replaces the file completely or leaves the old one untouched.
  • Scans by exact cosine similarity with top-k, an optional minScore, and an optional best-hit-per-ref mode.
  • Detects corruption through magic, version, header checksum, exact file size and a payload CRC-32 computed during the scan.
  • Provides ChunkedScanner, a fallback with identical ranking over pages your own code reads from your store.
  • Has a CLI with info and verify.

How it works

The build step checks the target directory, takes a non-blocking flock, and creates a temporary file in the same directory with exclusive create. It writes a placeholder header, streams records through a 1 MiB buffer while updating the CRC, writes the real header, calls fsync, checks the size, and calls rename(). It then syncs the directory. Vectors are normalised at build time by default, so a scan is one dot product per record.

The scan opens the file once and takes its size from that handle with fstat. It checks the header and the exact size before reading a record, then reads whole-record chunks and decodes floats with one unpack() per block, using explicit 1-based offsets. The top-k is a bounded min-heap with a deterministic tie-break on score, then id, then ref. Memory stays flat whatever the file size.

A scan that opened the old file keeps reading the old inode, and a later scan sees the whole new file. Scanners never take the lock. The main classes are Builder, Scanner, ChunkedScanner, the immutable BuildOptions and ScanOptions, and Internal\Header, Internal\Vectors and Internal\TopK.

$report = (new Builder($path, $dim))->build($records);

$result = (new Scanner($path, new ScanOptions(timeBudgetSeconds: 0.5)))->top($queryVector, 10);
foreach ($result->hits() as $hit) {
    echo $hit->id, ' ', $hit->ref, ' ', $hit->score, "\n";
}

For serving, catch UnavailableException, queue one rebuild, and fall back to the chunked scanner for that request. The repo’s own benchmark script, on synthetic vectors, a loaded two-core box, PHP 8.3 and no JIT, reports 0.84 s for 10,000 vectors of 512 dimensions, with about 8 MiB of added memory.

Guard rails

  • Every failure is a typed exception under PackedVectorsError. The “fall back and rebuild” family (missing, unreadable, corrupt, wrong version, wrong dimension) also implements UnavailableException. There is no @ suppression anywhere.
  • The scanner never truncates or pads a vector. A query of the wrong length throws DimensionMismatchException.
  • Bad build records are skipped and counted by reason (length, non_finite, zero_norm and others), or abort the build with onBadRecord: 'throw'. The build report is meant to be logged.
  • A time budget aborts a long scan with BudgetExceededException.
  • The temporary file is created with exclusive mode and the file mode is set before data is written (default 0640). A pre-existing name is never followed or removed.
  • A second builder returns at once with a “lock busy” report, or waits if lockWaitSeconds is set.
  • The file should live outside the web root, since embeddings can leak information about the text they came from.
  • It refuses to load on a 32-bit PHP build.

What review changed

  • The time budget was checked every 256 records, so a file with fewer than 256 records was never checked. A reproduction with 200 records of 65,535 dimensions and a one-microsecond budget ran for 5.4 seconds and returned hits. The check interval now depends on the dimension.
  • An exception thrown by the records factory passed to rebuildIfStale() escaped unwrapped. It now raises BuildFailedException with the original as the previous exception.
  • An IteratorAggregate whose getIterator() threw escaped unwrapped in the same way. The fix is the same, and a test checks the old file is unchanged and no temporary file remains.

The review also noted two behaviours it left alone: a payload bit flip in a float exponent can report non_finite before the checksum, and a flipped version bit reports an unsupported version. Both are still corrupt and still UnavailableException.

Limits and what is next

  • Exact linear scan only. It suits hundreds to about 100,000 vectors. Beyond that, or if you need filters inside the search, use a real vector index.
  • No metadata filtering in the scan, and no appends, updates or deletes. The file is always rebuilt whole.
  • Ids and refs are unsigned 32-bit integers in format version 1.
  • 64-bit PHP and local POSIX filesystems only for concurrent build and scan.
  • The library produces no embeddings and writes no logs of its own.
  • There are no open issues on the repo.

Test status: this is v0.1.0. 101 PHPUnit tests (6,620 assertions) pass on PHP 8.3, including the memory-budget group. Two were skipped when run as root. PHPStan level 8 and phpcs are clean. PHP 8.1 and 8.2 are verified only by a CI workflow that has not yet run on GitHub. Timings are printed by the benchmark and not asserted. It has not been run on a live site.

Install

Requires PHP 8.1 or later, 64-bit, with no runtime dependencies.

The package is not on Packagist yet, so first add a Composer vcs repository entry for https://github.com/IndianOpinion/packed-vectors.

composer require indianopinion/packed-vectors

Without Composer, download a release and require bootstrap.php once.

All 47 utilities · packed-vectors on GitHub

Search Trending
Loading

Signing-in 3 seconds...

Signing-up 3 seconds...

All fields are required.