Enveda CASMI progress audit — 2026-09-18 evening

Verdict

Audit completed; G1 remains OPEN. Substantial local fixes are real, but the active evaluation's eventual exit/report will not by itself satisfy the gate. The highest-priority newly reproduced defects are in the downstream wrapper/validation path. No evaluator, production source, input artifact, running shell, or scheduler was changed by this audit.

This audit updated TODO.md and created only isolated audit evidence. It did not submit to Kaggle, acquire a GPU, train a model, rerun full-data inference, or rescan/re-hash the training file.

Scope and evidence

Verified progress

1. Training release comparison exists and matches. The saved Kaggle-CLI redownload comparison has identical SHA-256, 3,033,286,496 bytes, 2,539,608 rows, 21 row groups and schema. Authoritative file remains present; staged/trash duplicates are absent after the user's permanent deletion request. This establishes equality to the September 18 download, not a fresh recheck against future Kaggle changes.

2. Rebuilt namespace is the operational target. Root retrieval-index/cache symlinks resolve to rebuild-20260917. Saved qualified reports contain 274,195 identities, 2,160,035 valid reference spectra, A's 1,184 query rows, and identity-wide B exclusions. Historical 274,288/275,810 counts belong to other namespaces.

3. A query selection fix is effective. Independent fixture: one allowed query and one eligible same-identity reference produce exactly one query/reference comparison, without promoting the reference into another query or self-match.

4. All-spectrum metric fixes are effective. Fixtures exercise ranks 1/25/26/100/1,000/1,001/absent. Pool coverage is 6/7 while Recall@1,000 is 5/7. Conditional MRR uses all six pool-covered targets; global MRR equals coverage times conditional MRR. Explicit tied-score and empty-pool fixtures also pass.

5. Basic strict-output fixes are effective. Current validator rejects invalid SMILES, equivalent-identity duplicate guesses and >25 nonempty guesses.

6. Basic formate alias support exists. Both conversion functions use the canonical alias table; the independent mass check and current alias/rejection tests pass. Broader numeric/polarity validation is not complete.

7. Progress reporting exists. Tiny fixture confirms a readable atomically replaced JSON progress record with no leftover .tmp. The active library-ceiling scan reached 621 batches and 113,376,230 comparisons; B has begun. Counters are not saved predictions or restart state.

8. Planning artifacts exist, not implementations. Optimization design and Phase 3A structure-index draft are present. PubChem format/count/hash audit is saved. No completed structure index, COCONUT acquisition, learned ranker, analogue system or GPU evaluator is evidenced by these documents.

Confirmed blockers / remaining limitations

A1 — Wrapper masks failed stages (P0)

scripts/run-g1-20260918.sh:2,8-12 enables pipefail but does not check the two preceding sequential stage exits. Exact-copy tests with a fake evaluator exiting 41, or fake submission exiting 42, still produced [G1_DONE], [SUBMISSION_DONE], [VALIDATION_DONE] and exit 0 when the fake validator succeeded. Only validator failures propagate.

Action: checked stage exit codes, run-scoped logs/output paths and failure-injection tests. Preserve the active evaluation; do not edit a live shell script in place or trust DONE markers alone.

A2 — Broken direct entry point and metadata overwrite (P0)

The helper begins with from src.scorer_contract import validate_submission, but its direct-script invocation supplies no project import path. A clean fixture reproduces ModuleNotFoundError: No module named 'src'; the live launch's PYTHONPATH is unset. When supplied the explicit project path, the helper successfully validates runtime IDs, but overwrites phase1-retrieval-rebuilt-20260918.json, the same file written by make_submission() for generation metadata. The synthetic test proves loss of spectra/coverage/status fields.

Action: tested module/package entry point; separate generation and .validation.json outputs. The long evaluator need not be restarted to repair a separately prepared postprocessing path.

A3 — Raw CSV and runtime-ID validation remains incomplete (P0)

The validator accepts missing/empty SMILES fields, an interior empty candidate token, 26 raw slots when 25 are empty, and an extra CSV field. The runtime helper accepts null Parquet IDs after converting them to the literal string None.

Action: validate raw fields/slots before sanitizing, reject malformed records and null/blank runtime IDs, and document the intended zero-candidate policy. These are local pipeline guarantees, not invented official-scoring rules.

A4 — Run reproducibility is not fully bound (P0)

Source paths under this project are untracked in the parent repository. The all-spectrum report embeds literal train/index hashes rather than calculating/binding a complete run manifest. Existing namespace guards compare counts/membership and selected metadata; they do not certify all input arrays, row mappings, splits, masks and SMILES mappings by content hash.

src/adducts.py changed at 17:47 after this process started at 12:39. Python's already-imported evaluator does not automatically acquire that change; later subprocesses may import newer on-disk code. This is a provenance gap, not proof that canonical-adduct numerical results changed.

Action: immutable source/config/environment/input manifests for future runs; explicitly qualify this run. Keep chunked hashing—an earlier inline mask audit used train.read_bytes(), which is not a bounded streaming hash.

A5 — Gate evidence still incomplete (P0)

The evaluator stores aggregate cohort summaries only. Per-molecule predictions/ranks/full-pool membership, fold/stratum reports, complete invalid/zero-valid query accounting and C-proxy evaluation are not produced. g1_molecule_eval.py and the sample evaluator retain the older conditional-MRR convention. Submission generation silently drops candidates with missing SMILES mappings and collects all test spectra in memory.

Action: persisted auditable predictions, explicit C=NOT_EVALUATED until evaluated, legacy-path deprecation/fixes, identity-map fail-closed checks and offline/resource validation. Do not declare G1 complete on the aggregate report alone.

A6/A7 — Test and checkpoint scope (P1)

Maintained test_g1_contract.py has three tests; it does not cover beyond-1,000 targets, all regime candidate-loop branches, real multi-query max aggregation or checkpoint interruption/resume. Its rank-26 fixture does exercise a lexicographic tie, but the separately named empty/tie test only checks an empty pool. This audit adds stronger independent evidence without pretending it has already been integrated into the maintained suite.

A null precursor currently raises before the invalid-spectrum handler. Checkpoints serialize counters, not scores/query state, and there is no resume loader. REGIME_COMPLETE is written before ranks/metrics are calculated; final JSON is not written atomically, and per-regime results are not independently saved. A crash late in B can lose the completed library metrics held only in RAM.

Action: promote audit probes, test invalid inputs, add bounded/time-based progress plus a distinct, tested resume/results format.

A8 — Phase 3 draft and optimization/GPU claims (P1, eligibility fix P0)

The draft's default normalization.training_overlap exclusion conflicts with the intended B task. Eligible held-out structures must remain searchable; exclude their spectra/prototypes/derived features, not their candidate identities. C's structure/scaffold exclusion is separate.

The PubChem .tsv.gz is still plain text at its original path. The word QUARANTINED in the draft is a status label, not physical quarantine. Stored identity/overlap/novelty counts are not re-normalized chemical validation. A legacy 50,000-record audit does not certify the different 240,536-row extract.

The optimization draft attributes 69,211,929 comparisons to batch 400; the recorded conversation checkpoint associated that figure with batch 500 (batch 400 was 59,662,024). Its 90-minute automatic-stop rule is not authorization to override the user's instruction to let G1 continue. No measured GPU speedup or full optimized implementation exists. A GPU kernel must preserve the current tolerance-based greedy peak matching, not substitute unrelated dense/binwise cosine.

Status/ETA corrections

Progress percentages are scan counts, not completed-work percentages. Record the checkpoint's own timestamp and age; do not present its old elapsed_seconds as current regime runtime. B and library must be compared at the same checkpoint, not partial B versus completed library. At batch 300, B had 5.54M comparisons / 1,324s versus library 8.73M / 1,488s—this does not support earlier claims of a dramatically shorter whole pass. CPU activity alone does not prove advancement or diagnose a hang. Prior ETA/GPU speed figures were unbenchmarked guesses and must not drive gate decisions.

Next safe order

1. Prepare a fail-closed, import-safe, metadata-preserving postprocessing path and raw-output tests in isolation.

2. Preserve ongoing run results/logs without claiming full G1 closure; add immutable provenance and per-molecule evidence for subsequent accepted runs.

3. Promote independent regression probes; fix the Phase 3 B-eligibility draft and PubChem source contract.

4. Benchmark CPU optimizations/GPU feasibility against exact oracle fixtures before costly reruns or training.

External qualifications remain: official CASMI scorer parity unavailable; A is an exact-available-field acquisition proxy, not lineage certification; C is a bounded novelty proxy, not hidden-test evidence.

Post-audit remediation (2026-09-18 19:xx MYT)

The following audit findings were fixed in the working tree and verified without restarting PID 198260:

Verification: 28 tests passed, shell syntax passed, adduct edge checks passed, and the isolated wrapper/runtime probes were rerun. The active process was not retroactively changed; G1 remains open because provenance, per-molecule/stratified evidence, resumable state, and C evidence are still outstanding.