← Back
📝 Notes / reports / PHASE0-AUDIT-2026-09-17

📑 Contents

Enveda Phase 0 independent audit — 2026-09-17Verdict: G0 FAIL / REOPEN REQUIREDFresh verificationFindings, severity orderedF1 — CRITICAL: inconsistent tautomer settings split an equivalent molecule across foldsF2 — HIGH: formate adduct mass is wrong by 1.007278304 DaF3 — HIGH: required scorer/evaluator tests are missingF4 — HIGH: the three frozen validation regimes are not fully materializedF5 — HIGH: the gate checks declarations and aggregates, not critical invariantsF6 — MEDIUM, mandatory before rebuilding: config and cache provenance are unsafeAdditional observations / limitsMinimum reopening/closure sequenceReproduction / evidence

Enveda Phase 0 independent audit — 2026-09-17


Verdict: G0 FAIL / REOPEN REQUIRED


The previous PHASE0_CORE_AND_MANIFESTS_PASS and TODO's closed-G0 claim are not supported by the substantive checks below. Existing implementation and manifests were not changed. This report supersedes their audit conclusion, not their on-disk status fields. Do not treat downstream identity-derived artifacts as scorer-compatible until remediation and independent rebuild verification.


Fresh verification


  • RDKit runtime: 2026.03.3.
  • Existing production tests: 8 passed (5 Phase 0 helper tests and 3 similarity tests).
  • Two audit-only regression probes: 2 failed, reproducing mass and identity defects.
  • Existing gate validator rerun: PASS on all 18 checks, despite the two real defects. This is evidence of inadequate gate coverage, not a successful independent audit.
  • Recomputed train SHA-256 matches 9423f90ec08b941a9646b535bf0606183e86ff0c5a297978f5a70693d41bcd98.
  • Independently streamed 2,539,608 training rows through the read-only identity cache: 0 missing mappings, 0 per-identity row-count mismatches, 0 deterministic fold mismatches. The cache has 277,566 SMILES / 274,288 stored identities. These are internally consistent under the current, defective identity procedure; they are not a certified scorer identity count.
  • PubChem archive SHA-256 matches its manifest. Parsed existing TSV: 49,994 rows, 48,641 unique stored identities, 5,987 overlapping training identities, 0 overlap-flag mismatches. Structure cohort and per-fold counts independently reproduce. The original bounded SDF parsing was not rerun.

    • Findings, severity ordered


    F1 — CRITICAL: inconsistent tautomer settings split an equivalent molecule across folds


    Sources: src/identity.py:15-17,27-28,38-39; src/split_manifest.py:64-77; reports/phase0-identity-probe-2026-09-17.json.


    identity14() restricts the tautomer enumerator to 100 transforms; canonical_smiles() uses the RDKit default 1,000 transforms. Both allow 1,000 tautomers. A real cached training structure returns:


  • identity14(s) = YYBZEWCJTIGSEM
  • identity14(canonical_smiles(s)) = ZZZRUAITSXLWBH

    • The manifest puts the first key in fold 4, 102 rows, and the second key in fold 2, 52 rows. All three cached SMILES belonging to those two keys collapse to ZZZRUAITSXLWBH with the default RDKit canonicalizer. Thus an actual default-canonicalizer identity crosses validation folds, not just a theoretical risk.


    The diagnostic tested the first 100 cache entries containing at least six O characters and found this example. This is a targeted diagnostic, not an estimate of corpus-wide error prevalence. The exact official executable scorer was not available in the inspected snapshots; its documented operation and default-RDKit comparison must not be confused with verified official-source parity. Even independently of that uncertainty, the two project helpers demonstrably violate the plan's one-identity-path requirement.


    Required: confirm canonicalizer settings against the official scorer when available; use one shared implementation; add complex-tautomer consistency tests; rebuild identity cache, splits, external overlap/structure/scaffold manifests, reference identity arrays, masks and candidate mappings. Do not patch only the function and reuse old caches.


    F2 — HIGH: formate adduct mass is wrong by 1.007278304 Da


    Source: src/adducts.py:18; failing probe reports/test_phase0_audit_regressions.py:10-13.


    For [M+CH2O2-H]-, expected delta is approximately 44.998202837 Da (neutral formic acid minus a proton). The code uses 44.998201 - 1.007276466621 = 43.990924533 Da, effectively subtracting the proton twice.


    A precursor corresponding to neutral mass 300 Da is decoded as 301.007278304 Da. Live column scans found 83,188 training rows and 31 visible-test spectra with this adduct. This can break cross-adduct/structure-mass matching; it does not imply every same-adduct retrieval necessarily fails, since both query and reference may share the offset.


    Why tests missed it: the round-trip test uses the same wrong constant in both directions (tests/test_phase0.py:23-26).


    Required: correct the delta, test all ten adducts against independently derived masses, rebuild affected mass indexes and derived retrieval outputs.


    F3 — HIGH: required scorer/evaluator tests are missing


    Sources: EXECUTION-PLAN.md:127-142; tests/test_phase0.py:1-31; TODO.md Phase 0 scorer-tests checkbox.


    The checked-in Phase 0 suite tests invalid strings, one stereo pair, acetone tautomer equivalence, adduct round-tripping and unknown-adduct rejection. It does not test the claimed rank cutoff or a submission-level MRR@25 contract: ranks 1/25/26, first correct guess, semicolon parsing, duplicate/null/missing IDs, >25 candidates, constitutional isomers, equivalence-aware deduplication or dynamic IDs.


    Phase 1 contains inline reciprocal-rank arithmetic; that is not a substitute for the missing tested scorer contract. The Phase 0 completed checkbox overstates implemented coverage.


    Required: implement and test the molecule-level scorer and submission contract; keep unknown official edge cases explicit rather than inventing behavior.


    F4 — HIGH: the three frozen validation regimes are not fully materialized


    Sources: EXECUTION-PLAN.md:146-165; src/strict_validation.py:21-29,64-82,99-120; src/validate_phase0.py:91-94.


  • A / spectral library available: the plan requires frozen held-out target-domain acquisitions and exclusion of duplicate/reprocessed acquisitions, while allowing independent same-compound references. The core artifact is a molecule-group fold assignment, not that acquisition-level query/reference manifest. No acquisition duplicate-cluster/exclusion mapping is present in the inspected Phase 0 artifacts. Calling this regime ready is premature. Phase 1's library_ceiling excludes only the exact row (src/phase1_baseline.py:181), not duplicate acquisitions, and remains a ceiling diagnostic.
  • B / structure available: the bounded PubChem intersection and folds are structurally sound under stored keys. Operational exclusion checks can reasonably belong to Phase 1; the corrected identity rebuild is still necessary.
  • C / novelty proxy: the saved JSON contains an all-identities list, scaffold count and aggregate fold counts, but no identity-to-scaffold or identity-to-novelty-fold assignments. Those exist only transiently in build(). Reconstructing them depends on a mutable, unversioned cache, first-encountered SMILES without SQL ordering, and raw-SMILES scaffold extraction. scaffold_folds_exclusive is hard-coded True, not independently verified. Aggregate count conservation cannot establish the exact frozen held-out population or external candidate scaffold exclusions.

    • Required: persist the A acquisition/duplicate exclusions and C identity/scaffold/fold assignments with data, code, algorithm and dependency hashes; assert disjointness against the actual chosen reference/candidate pools. The bounded PubChem scope itself is not a defect; using all training molecules in scaffold CV is also not inherently wrong.


    F5 — HIGH: the gate checks declarations and aggregates, not critical invariants


    Source: src/validate_phase0.py:54-75.


    The fresh validator PASS is misleading because it:


  • Checks the configured RDKit string, not imported runtime version.
  • Checks adduct names, not physical mass constants.
  • Trusts external status strings and a recorded scaffold_folds_exclusive boolean.
  • Sums recorded fold aggregates rather than verifying individual assignments and equivalent-identity disjointness.
  • Does not validate config.validation.group_key against the split.
  • Does not hash/verify the actual external TSV or strict identity/fold assignments.
  • Does not execute the required scorer tests.

    • Required: turn these into fail-closed executable checks, verify actual inputs and emitted mappings, and include source/config/dependency fingerprints in every result. Hashing an artifact alone is not verification of its correctness.


    F6 — MEDIUM, mandatory before rebuilding: config and cache provenance are unsafe


    Sources: configs/phase0.json:8; src/split_manifest.py:37-44,64-77.


    Config declares group_key: inchikey14; manifest declares scorer_identity14. The cache schema is only (smiles PRIMARY KEY, identity): no RDKit version, canonicalizer settings or implementation fingerprint. A checkpoint is resumed without binding it to data hash, identity algorithm, seed/fold configuration or cache fingerprint. Changing the identity function alone can therefore silently reuse stale identities or completed checkpoint statistics.


    Required: align config, use a new versioned cache/checkpoint namespace, reject provenance mismatches, and regenerate rather than silently mixing versions.


    Additional observations / limits


  • Live adduct scan found 362,731 training rows outside the ten advertised hidden-test adduct names; 0 visible-test rows are unsupported. Supporting additional training-only adducts is a retrieval-coverage improvement, not itself a failure of the ten-hidden-adduct G0 requirement. Quantify intentional exclusions downstream.
  • No source files, configs, production manifests, data, retrieval indexes or submissions were modified. Only audit scripts, evidence, regression probes and this report were added under reports/.
  • No official Kaggle submission, full new identity rebuild, exhaustive chemistry comparison, full SDF reparse or clean-room offline packaging run was performed.
  • Existing gate/TODO fields remain unchanged for audit traceability. They must not be used to override this failed audit.

    • Minimum reopening/closure sequence


    1. Confirm and unify scorer identity settings; correct mass conversion; add independent regression and full scorer-contract tests.

    2. Create new fingerprinted cache/checkpoint artifacts and rebuild every dependent identity/mass artifact.

    3. Freeze actual A acquisition exclusions, B structure cohort and C scaffold-fold assignments under the corrected identity convention.

    4. Strengthen the gate, independently rescan all row mappings/disjointness, verify artifact hashes and capture runtime dependency versions.

    5. Re-run G0 and only then certify Phase 1 evaluation/submission outputs. Engineering may continue, but no current downstream metrics should be promoted as clean validation evidence.


    Reproduction / evidence


    From project root:


    .venv-rdkit2026/bin/python -m pytest -q tests.venv-rdkit2026/bin/python -m pytest -q reports/test_phase0_audit_regressions.py.venv-rdkit2026/bin/python reports/audit_phase0_20260917.py.venv-rdkit2026/bin/python -m src.validate_phase0 --output reports/phase0-gate-rerun-2026-09-17.json

    The audit script consumes the saved bounded identity diagnostic; it independently rechecks its cross-fold example against the cache and default canonicalizer. Do not interpret it as a full default-identity rescan.


    Evidence files:

  • reports/phase0-existing-tests-2026-09-17.log
  • reports/phase0-regressions-2026-09-17.log
  • reports/phase0-independent-evidence-2026-09-17.json (includes source/config hashes)
  • reports/phase0-identity-probe-2026-09-17.json
  • reports/phase0-adduct-counts-2026-09-17.json
  • reports/phase0-gate-rerun-2026-09-17.json