Build the build-time English text ingestion pipeline #108

Closed
opened 2026-07-01 00:54:15 +00:00 by manlycucumber · 1 comment
manlycucumber commented 2026-07-01 00:54:15 +00:00 (Migrated from github.com)

Problem

English translations must be ingested from public-domain sources at build time (Node only, no Python) into the corpus with provenance. A reproducible pipeline is needed.

Acceptance criteria

  • Pipeline ingests English source text into corpus strings
  • Each string is tagged with source and licence
  • Running twice yields identical output
  • No Python is used anywhere

Notes

Runtime remains PHP+MySQL; ingestion is build-time.

## Problem English translations must be ingested from public-domain sources at build time (Node only, no Python) into the corpus with provenance. A reproducible pipeline is needed. ## Acceptance criteria - [ ] Pipeline ingests English source text into corpus strings - [ ] Each string is tagged with source and licence - [ ] Running twice yields identical output - [ ] No Python is used anywhere ## Notes Runtime remains PHP+MySQL; ingestion is build-time.

Done in 04f37a2 (ingestion core), bebb2a4 (psalters) and 51a683a (canticles).

ACs: pipeline ingests English source text into corpus strings ✓ — tools/generator/src/ingest.mjs + parse-douay.mjs; each string tagged with source and licence ✓ — every row carries source + rights, enforced by the born-cited gate; running twice yields identical output ✓ — streams are NFC-normalized, sorted by (psalm, verse) and dedup'd fail-closed, and the committed MANIFEST sha256s are re-verified by verify.mjs; no Python is used anywhere ✓ — verified, the repo contains no .py file and the toolchain is Node-only.

Done in `04f37a2` (ingestion core), `bebb2a4` (psalters) and `51a683a` (canticles). ACs: pipeline ingests English source text into corpus strings ✓ — `tools/generator/src/ingest.mjs` + `parse-douay.mjs`; each string tagged with source and licence ✓ — every row carries `source` + `rights`, enforced by the born-cited gate; running twice yields identical output ✓ — streams are NFC-normalized, sorted by `(psalm, verse)` and dedup'd fail-closed, and the committed MANIFEST sha256s are re-verified by `verify.mjs`; no Python is used anywhere ✓ — verified, the repo contains no `.py` file and the toolchain is Node-only.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Directorium/Core#108
No description provided.