Enveda CASMI 2026 Todo

Legend: [ ] pending · [-] in progress · [x] complete · [!] blocked

The priority order is deliberate: establish trustworthy evaluation and a strong retrieval floor before investing in larger models.

Phase 0 — Ground truth and infrastructure

[x] P0 Download the current train.parquet release from Kaggle.

[x] P0 Record file size, SHA-256, Kaggle metadata and download timestamp in manifests/.

[x] P0 Inspect the actual training schema, row groups, library counts and identity counts.

[x] P0 Add a low-RAM data access path using PyArrow row-group streaming and selected columns.

[x] P0 Create a pinned RDKit 2026.03.3 environment for scorer-compatible identity matching.

[x] P0 Implement tautomer-canonicalized InChIKey14 identity logic in src/identity.py.

[x] P0 Add scorer unit tests for tautomer equivalence, stereochemistry, invalid SMILES and rank cutoff.

[x] P0 Implement all ten advertised adduct mass conversions and round-trip tests.

[-] P0 Freeze molecule-grouped validation manifests for library, database-only and novelty-proxy regimes. The rebuilt scorer-compatible five-fold and strict external manifests are internally consistent; qualified Regime A query/reference masks are now operational and runtime-validated, while Regime B identity-wide exclusion manifests and runtime checks now pass locally, and Regime C remains proxy-only.

[x] P1 Add a reproducible experiment configuration format under configs/.

Resource constraint: the local development host has approximately 11 GiB RAM. Full-data operations must stream or use bounded artifacts; do not load the entire training table into pandas.

Gate G0: [-] Qualified local infrastructure only. Rebuilt cache/split/index/masks pass local checks and the root retrieval artifact resolves to the rebuilt 274,195-identity namespace; official scorer parity is unavailable and Regime A is only a qualified exact-available-field proxy. Regime B identity-wide local enforcement is now frozen and tested; do not report unconditional G0 closure.

Phase 1 — Spectral retrieval baseline

[x] P0 Build a streaming Parquet reader and compact peak-array representation.

[x] P0 Implement precursor/adduct to neutral-mass filtering.

[x] P0 Implement peak cleaning variants: intensity floor, precursor exclusion and top-N/windowed peaks.

[x] P0 Implement fragment cosine similarity.

[x] P0 Implement spectral entropy similarity.

[x] P0 Implement neutral-loss similarity.

[-] P0 Build an indexed reference search over training spectra. Root artifacts/retrieval-index was repaired on 2026-09-17 to resolve to rebuild-20260917/retrieval-index; the legacy index was preserved under artifacts/legacy-20260917/retrieval-index-root-legacy-20260917-1717. Rebuilt index verification: 2,539,608 rows, 2,160,035 valid spectra, 274,195 identities, all five fold masks. Persistent provenance assertions and qualified Regime-A runtime checks now pass; fresh G1 metrics remain open.

[ ] P0 Aggregate evidence across all spectra for each molecule_id. Source code contains molecule-level aggregation, but the saved main submission/evaluation artifacts are stale or not sufficient as fresh G1 evidence. Regenerate from the authoritative rebuilt index and verify hashes.

[-] P0 Deduplicate candidates by scorer-compatible identity. The implementation is present and locally tested; end-to-end rebuilt-index output and official scorer parity remain unverified.

[-] P0 Generate and validate a first local submission file. The molecule-level fixture exists, but the main submissions/phase1-retrieval.csv is historically spectrum-level/stale and current output provenance must be regenerated and validated against the rebuilt namespace.

[ ] P1 Add source, adduct, polarity and collision-energy compatibility features.

[-] P1 Add timing, RAM and candidate-coverage measurements. Historical index timing is 149.85 s for the legacy index and 157.30 s for rebuilt; prior submission/RSS numbers are not fresh rebuilt-index G1 evidence. Re-measure together with frozen-fold metrics.

Gate G1: [!] OPEN. Regime-B smoke enforcement is complete, but still missing fresh rebuilt-namespace molecule-level MRR@25, Top-1, Top-5, Recall@25, candidate recall, pool coverage, conditional MRR, runtime, peak memory, and stratified Regime A/B/C results.

Phase 2 — Learned ranking and domain adaptation

[ ] P0 Generate out-of-fold candidate features without molecule leakage.

[ ] P0 Train a first molecule-grouped LightGBM/CatBoost ranker.

[ ] P0 Compare learned ranking against hand-tuned retrieval aggregation.

[ ] P1 Balance or reweight training sources to reduce synthetic-library dominance.

[ ] P1 Add reference-count bias correction.

[ ] P1 Add confidence margins and uncertainty diagnostics.

[ ] P1 Run ablations for fragment, entropy, neutral-loss and multi-spectrum evidence.

Gate G2: learned ranking improves grouped validation without severe regression in any intended novelty regime.

Phase 3 — Known structures without reference spectra

[ ] P0 Acquire and document an eligible COCONUT snapshot.

[ ] P1 Build a mass/formula index for competition structures and COCONUT.

[ ] P1 Add a broader PubChem-derived structure candidate source if storage and licensing permit.

[ ] P1 Implement spectrum-to-fingerprint neighbour transfer.

[ ] P1 Test pretrained spectrum embeddings such as DreaMS/MIST after license and offline checks.

[ ] P1 Train a spectrum-to-fingerprint retrieval model.

[ ] P1 Evaluate formula inference and candidate-pool recall.

[ ] P2 Add a forward-spectrum reranker for shortlisted structures.

Gate G3: measurable improvement in the structure-known/no-spectrum validation regime, reported separately as pool coverage and conditional MRR.

Phase 4 — Novel structures and de novo generation

[ ] P1 Establish a clean novelty-proxy split with structure and spectral exclusions.

[ ] P1 Build a formula-, adduct- and collision-energy-conditioned spectrum encoder.

[ ] P1 Implement a bounded SMILES decoder/generator.

[ ] P1 Add beam search and controlled sampling with valid-SMILES filtering.

[ ] P1 Add analogue editing from close spectral neighbours.

[ ] P1 Deduplicate generated outputs by scorer-compatible identity.

[ ] P2 Add forward-spectrum and formula consistency reranking.

[ ] P2 Measure unique candidate recall and actual MRR contribution, not validity alone.

Gate G4: de novo candidates add measurable clean-validation MRR or candidate coverage within the offline runtime budget.

Phase 5 — Unified ensemble

[ ] P0 Union retrieval, database and de novo candidate pools.

[ ] P0 Train a final OOF molecule-level ensemble ranker.

[ ] P0 Produce exactly one row per runtime molecule with at most 25 unique candidates.

[ ] P1 Calibrate route confidence and preserve exploration for uncertain molecules.

[ ] P1 Stress-test rare adducts, high masses, one-spectrum molecules and noisy spectra.

[ ] P1 Compare library-heavy, balanced and novelty-heavy validation mixtures.

[ ] P1 Perform paired-bootstrap comparisons and record promotion decisions.

Gate G5: robust integrated model beats the retrieval baseline across the selected validation scenarios.

Phase 6 — Kaggle packaging and final submission

[ ] P0 Package model weights, indexes, tokenizers and dependencies for offline notebook execution.

[ ] P0 Run with internet disabled.

[ ] P0 Test against dynamically loaded test IDs; do not rely on placeholder IDs.

[ ] P0 Verify submission.csv columns, nulls, duplicate IDs and maximum candidate count.

[ ] P0 Complete two clean-room notebook reruns within the 9-hour limit.

[ ] P1 Prepare two complementary final candidates only if validation supports both.

[ ] P1 Record final code, configuration, hashes, licenses and reproduction instructions.

[ ] P0 Select final submissions explicitly before the deadline.

Gate G6: two successful offline reruns, reproducible output and final compliance audit.

Active experiment log

Date	Experiment	Hypothesis	Result	Decision
2026-09-16	Competition review and workspace setup	Establish a complete execution baseline	Plan and project scaffold created; no model run	Start Phase 0
2026-09-16	Phase 0 data acquisition	Verify current Kaggle release and characterize it under low RAM	3,033,286,496-byte file; SHA-256 recorded; 2,539,608 rows, 21 row groups, 275,810 unique InChIKey14 identities; audit complete	Proceed to grouped split manifests
2026-09-16	Molecule-grouped split	Prevent spectral/structure leakage in local validation	Deterministic 5-fold manifest validated: 275,810 groups and 2,539,608 rows; fold assignment is group-exclusive	Use fold 0 as initial validation; keep novelty labels separate
2026-09-16	G0 data/infrastructure gate	Confirm reproducible Phase 0 inputs and assumptions	All eight gate checks passed; config and split hashes recorded in `manifests/phase0-gate.json`	Close data/infrastructure work; proceed to retrieval baseline
2026-09-16	Phase 0 audit	Verify that validation grouping uses scorer identity	Failed: 12/100 sampled normalized SMILES differed between stored and scorer-derived InChIKey14; full probe hit pathological tautomer enumeration	Reopen G0; rebuild identity-safe split before retrieval evaluation
2026-09-16	Phase 0 corrective audit	Verify corrected scorer-identity split and reproducibility gate	Passed: 8 tests; compilation passed; 2,539,608 rows and 274,288 scorer groups validated; cache/checkpoint agree; full G0 gate passed; 100-row identity probe had 0 invalid structures	Proceed to retrieval baseline; retain external novelty regimes as pending
2026-09-16	Phase 1 retrieval baseline	Build a low-memory, identity-safe spectral retrieval floor	Index built over all 2,539,608 rows; 2,160,035 valid spectra; fold-0 mask excludes all 503,842 held-out rows; molecule-level fixture submission has 400/400 covered molecules and passes schema/count checks	Keep baseline; run grouped MRR/Top-1/Recall evaluation before closing G1

Blockers and decisions

Current blockers: (1) Regime A is operational only as the documented exact-available-field local proxy; it is not acquisition-lineage certification; (2) official scorer executable/reference vectors are unavailable; (3) local NP-example count is now pinned to 1,184 for the hashed current release, versus stale published documentation of 1,151; (4) full G1 evidence remains open.

Do not start: large model training before the qualified G0 issues above are resolved and G1 is measured on the rebuilt namespace.

Do not submit: until a regenerated molecule-level output passes local schema checks, official/parity caveats are documented, and offline packaging uses only rebuilt artifacts.

📑 Contents