feat: citable & reproducible dataset (#237 / #238, #239, #241) #473

Merged

manlycucumber merged 1 commit from feature/237-citable-dataset into develop

2026-07-10 22:03:42 +00:00

manlycucumber commented

2026-07-10 22:02:10 +00:00

(Migrated from github.com)

What & why

Epic #237 — citable & reproducible dataset: make the CC0 corpus something scholars and other engines can cite and verify. The corpus was already deterministic (verify.mjs gates determinism + freshness); this hardens that into an enforced integrity check, a versioned checksummed release artifact, and DOI archival prep.

#238 — manifest integrity (the check a consumer runs)

New tools/generator/src/manifest.mjs — verifyManifest() recomputes every file's SHA-256 and asserts it matches MANIFEST.json (+ no missing files).
verify.mjs now enforces three guarantees in CI (corpus-verify runs npm run verify): determinism · freshness · integrity.

#239 — versioned CC0 release artifact

New tools/generator/src/package.mjs — prepareRelease() verifies the corpus against its manifest, then emits SHA256SUMS (covering every file in the archive, including the CC0 schema files the manifest doesn't itself list) + release-metadata.json.
release-please.yml — on a cut release, assembles a deterministic corpus tarball (--sort=name, fixed mtime/owner, gzip -n) and attaches it with its .sha256, SHA256SUMS, and metadata. The archive is named with the corpusVersion (data-version, #54).

#241 — citable / DOI

CITATION.cff — fixed the stale repo URL (Directorium/Core → DirectoriumApp/Core) and added a version release-please keeps current.
.zenodo.json — metadata for per-release Zenodo DOI archival (AGPL engine / CC0 dataset split noted).
docs/citable-dataset.md — how to cite, how to verify a download (sha256sum -c), the reproducibility guarantees, and the one-time maintainer step to enable the GitHub↔Zenodo integration.

Verification

node --test ✅ 40 tests (+8: manifest good/wrong-hash/missing, packaging incl. schema coverage + manifest-mismatch refusal).
npm run verify ✅ (determinism + freshness + integrity, 32 files) · npm run package dry-run ✅ (SHA256SUMS covers all 44 files incl. the 12 schemas). All JSON/YAML validated. No corpus/data bytes changed.
Adversarial review: GO, 0 pipeline-breaking defects. The one confirmed correctness defect — SHA256SUMS not covering the tarball's schema files (documented as "every file") — is fixed in this PR by walking the whole tree; release-metadata.json is now uploaded (was dead); a test locks the schema coverage.

Closes #238, closes #239, closes #241.

## What & why Epic **#237 — citable & reproducible dataset**: make the CC0 corpus something scholars and other engines can **cite and verify**. The corpus was already deterministic (`verify.mjs` gates determinism + freshness); this hardens that into an enforced integrity check, a versioned checksummed release artifact, and DOI archival prep. ## #238 — manifest integrity (the check a consumer runs) - New `tools/generator/src/manifest.mjs` — `verifyManifest()` recomputes every file's SHA-256 and asserts it matches `MANIFEST.json` (+ no missing files). - `verify.mjs` now enforces **three** guarantees in CI (`corpus-verify` runs `npm run verify`): determinism · freshness · **integrity**. ## #239 — versioned CC0 release artifact - New `tools/generator/src/package.mjs` — `prepareRelease()` verifies the corpus against its manifest, then emits `SHA256SUMS` (covering **every file in the archive**, including the CC0 schema files the manifest doesn't itself list) + `release-metadata.json`. - `release-please.yml` — on a cut release, assembles a **deterministic** corpus tarball (`--sort=name`, fixed mtime/owner, `gzip -n`) and attaches it with its `.sha256`, `SHA256SUMS`, and metadata. The archive is named with the `corpusVersion` (data-version, #54). ## #241 — citable / DOI - `CITATION.cff` — fixed the stale repo URL (`Directorium/Core` → `DirectoriumApp/Core`) and added a `version` release-please keeps current. - `.zenodo.json` — metadata for per-release Zenodo DOI archival (AGPL engine / CC0 dataset split noted). - `docs/citable-dataset.md` — how to cite, how to verify a download (`sha256sum -c`), the reproducibility guarantees, and the one-time **maintainer** step to enable the GitHub↔Zenodo integration. ## Verification - `node --test` ✅ **40 tests** (+8: manifest good/wrong-hash/missing, packaging incl. schema coverage + manifest-mismatch refusal). - `npm run verify` ✅ (determinism + freshness + integrity, 32 files) · `npm run package` dry-run ✅ (SHA256SUMS covers all **44** files incl. the 12 schemas). All JSON/YAML validated. No corpus/data bytes changed. - Adversarial review: **GO, 0 pipeline-breaking defects.** The one confirmed correctness defect — `SHA256SUMS` not covering the tarball's schema files (documented as "every file") — is fixed in this PR by walking the whole tree; `release-metadata.json` is now uploaded (was dead); a test locks the schema coverage. Closes #238, closes #239, closes #241.

manlycucumber referenced this pull request from a commit

2026-08-05 19:02:29 +00:00

feat: citable & reproducible dataset — integrity gate + release artifact + DOI prep (#238, #239, #241) (#473)

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Directorium/Core!473

No description provided.

Rows
Columns