← Back
📝 Notes / reports / phase0-reaudit-20260917-1619 / regimes-review

📑 Contents

Phase 0 re-audit — validation regimes A/B/CExecutive statusContract and local dataA — spectral-library available: **PARTIAL / FAIL for closure**B — structure available, spectra unavailable: **IDENTITY-WIDE LOCAL ENFORCEMENT PASS**C — de novo / novelty proxy: **PROXY ONLY; CONTAMINATION UNRESOLVED**Prioritized TODO for parent

Phase 0 re-audit — validation regimes A/B/C


Scope. Read-only audit of the published dataset contract, local manifests/code, and existing regime-A SQLite. No production files, TODO.md, caches, indexes, or data were changed. Host imports: PyArrow 25.0.1; .venv-rdkit2026 RDKit 2026.03.3. Detailed machine-readable evidence: regimes-evidence.json; duplicate scan helper/output are regimes-scan.py / regimes-scan.json.


Executive status


Overall: qualified local infrastructure pass, with external caveats. A is implemented as the documented exact-available-field proxy; B now has frozen identity-wide masks and runtime assertions. Official scorer parity and raw acquisition-lineage certification remain unavailable. The existing PASS labels are historical/artifact statuses, not acceptance of the current evaluation regime.


Contract and local data


The authoritative regime definitions are EXECUTION-PLAN.md:143-158: A holds out target-domain enveda-np-examples query acquisitions while allowing independently acquired same-compound references; B removes all spectra of held identities while allowing their frozen structures; C is a scaffold-disjoint novelty proxy and must not be presented as hidden Class 3. The source schema/novelty/library contract is research/data-description.md:13-29,52-66,74-99; saved provenance/hash inventory is research/review-manifest.json.


Streaming the local data/train.parquet yielded 2,539,608 rows and 1,184 enveda-np-examples rows, versus the published table's 1,151 rows/250 compounds (data-description.md:88). This version discrepancy is explicitly pinned in manifests/rebuild-20260917/np-example-reconciliation.json; the hashed current parquet is authoritative for local evaluation. Other local library counts are in regimes-evidence.json.


A — spectral-library available: PARTIAL / FAIL for closure


The local fingerprint implementation is appropriately acquisition-oriented (src/regime_a_exclusions.py:14-22) and includes both ingest_lib and the often-forgotten precursor_error_ppm. It deliberately excludes normalized structure identity. The SQLite was opened read-only via URI and checked:


  • rows: 2,539,608; fp_counts: 2,539,107;
  • 501 fingerprints have multiplicity 2 (1,002 rows; 501 excess rows), no higher multiplicity;
  • an independent row mapping check agrees with the DB's duplicate accounting;
  • none of the 1,184 local NP-example rows is in those 501 duplicate pairs.

    • This is evidence of 501 alleged exact local duplicates, not proof of duplicate acquisitions: the contract has no raw run/acquisition ID and the fingerprint lacks provenance. Lack of raw run IDs is a limitation, not permission to invent a hard competition requirement. Do not bless this DB as closure without a manifest, source/version, explicit row mapping, and integrity checks.


    Missing A closure: no committed A query manifest/mask selecting NP-example acquisitions was found; no A-specific enforcement exists in evaluate-sample; and no provenance-backed reprocessed-acquisition exclusion can be asserted. In particular, src/phase1_baseline.py:146-160 selects the first rows of each identity fold across the entire train stream, not NP-example queries. At :163-190, library_ceiling removes only the exact query row (:181), while fold_held_out applies identity-wide fold masks. Neither is Regime A. Identity-wide exclusion is contrary to A because other-library same-compound references must remain eligible.


    Acceptance criteria (P0): (1) immutable A manifest with source/version, selected NP query row IDs, fingerprint definition, SQLite URI integrity and row mapping; (2) query mask exactly ingest_lib=enveda-np-examples; (3) global exclusion of exact query fingerprints, while retaining independently acquired same-identity rows from other libraries; (4) tests/assertions that query rows and excluded fingerprints cannot score; (5) explicit wording that the result is a local exact-fingerprint proxy, not raw-run certification.


    B — structure available, spectra unavailable: IDENTITY-WIDE LOCAL ENFORCEMENT PASS


    manifests/rebuild-20260917/strict-validation.json:1-30 reports STRICT_EXTERNAL_VALIDATION_MANIFEST_PASS, 5 folds, scorer identity scorer_identity14, and a bounded PubChem first-50,000-record cohort of 5,974 identities / 307,367 rows. src/strict_validation.py:90-97 correctly states the intended rule: exclude every training row for held identities, while PubChem structure may remain and PubChem spectra are not used. The integrity/cohort accounting is useful.


    The rebuilt evaluation path now loads the frozen per-fold masks through load_regime_b_policy() and fails closed on status, row alignment, cohort leakage, and eligible-row counts. This local proof covers the authoritative spectral index/evaluation path; future prototype, embedding, or checkpoint producers must carry the same fold mask and provenance. PubChem coverage is bounded and must not be called complete. Acceptance: per-fold zero held-identity rows across every reference/feature/training path, with a permitted structure candidate check; report naturally covered and full cohorts separately; freeze and cite snapshot provenance.


    C — de novo / novelty proxy: PROXY ONLY; CONTAMINATION UNRESOLVED


    manifests/rebuild-20260917/strict-validation.json:99-113 records deterministic scaffold-group holdout, candidate exclusion of held answers/scaffolds, and the limitation that it is only a proxy. That limitation is correct and should remain prominent. It does not establish hidden Class 3 composition. Pretrained-model overlap/provenance is not established in the saved validation artifacts; per scope, no full SDF reparse was performed. Therefore any C score involving external pretrained assets is contaminated/provisional unless provenance/overlap is audited.


    Acceptance: retain C/proxy naming; prove held identity/scaffold absence from candidate and supervised target paths; document model/data provenance and mark unknown overlap contaminated; report molecule-level scaffold-disjoint coverage and MRR separately.


    Prioritized TODO for parent


    1. P0 A: build and enforce NP-example query/fingerprint masks and integrity/row-mapping manifest; preserve cross-library same-compound references.

    2. P0 B: identity-wide Regime-B manifest and authoritative evaluation-path assertions now pass; extend the same provenance contract when prototype/embedding/checkpoint producers are introduced.

    3. P1 C: audit external-model provenance/overlap and keep novelty claims explicitly proxy-only.

    4. P1: local NP count is pinned to 1,184 for the hashed current release; published 1,151 remains a documentation/version caveat.


    Parent owns cache/index lineage and unit tests; this audit intentionally did not hash train or read huge peak arrays.