← Back
📝 Notes / TODO-BASELINE-TOP5

📑 Contents

Enveda CASMI26 — Baseline-to-Top-5 TODO0. Baseline decision and scope1. Review result: what we actually haveVerified on 26 SeptemberActual architecture, not the notebook marketing labels2. P0 — Establish a trustworthy baseline measurement3. Primary paper references and adaptation boundariesP1 — First ranking experiments (one controlled change at a time)P2 — Conditional additions selected by measured failure causes4. Evaluation, promotion and leaderboard gatesExperiment record required for every challenger5. Immediate execution queue

Enveda CASMI26 — Baseline-to-Top-5 TODO


Created: 26 September 2026 (MYT)

Owner: CC

Goal: improve the selected Enveda CASMI26 Baseline using arXiv-inspired techniques, targeting a top-five official leaderboard finish. Track public rank during development; final private rank is the outcome. Neither a paper result nor a fixed score guarantees that rank.


0. Baseline decision and scope


  • Our baseline B0 is the bank-INCLUDED replica: [kaggle-kernel/haideptry-v32-replica/](kaggle-kernel/haideptry-v32-replica/).
  • Kaggle: [jamesl8/enveda-casmi26-baseline](https://www.kaggle.com/code/jamesl8/enveda-casmi26-baseline), Enveda CASMI26 Baseline, private; initial push was version 1.
  • Executed source: haideptry-v32-replica.ipynb. It writes casmi_engine.py, pv.py, pv_fp.py, and regio_generator.py into the Kaggle working directory. Editing a local .py alone does not update the notebook submitted to Kaggle.
  • Keep the existing isomer-bank input and behavior in B0. Do not recreate the deleted no-bank variant. Improvements are incremental challengers to this baseline, not an excuse to replace the user's baseline decision.
  • Preserve B0's source, inputs and outputs as the rollback/reference snapshot. Make later approved changes separately/version them; do not rewrite historical results.
  • This is the new execution roadmap for this baseline. The older [TODO.md](TODO.md) and [EXECUTION-PLAN.md](EXECUTION-PLAN.md) remain historical/reference material; their competition-only candidate pool, absent-FP-model status, expired quota warning and old launch order are not the starting point here.
  • Existing notebook models/data are retained because CC explicitly requested this replica. For new paper-inspired additions, preserve the prior reference-only policy: independently implement ideas; do not introduce additional external paper code, pretrained weights, tokenizers or training archives without an explicit change of scope and rights review. A published algorithm is not permission for every associated asset.
  • Model training/heavy inference belongs on authorized Kaggle resources, not this ARM host. No new paid compute. Recheck available hardware/quota before each launch. This review creates a plan, not authorization for automatic training, pushes or competition submissions.

    • Status: [x] verified at the stated scope · [-] partial · [ ] pending. Tasks below are future work unless checked.


    1. Review result: what we actually have


    Verified on 26 September


  • [x] B0-SOURCE: the four local modules equal their notebook %%writefile bodies, and all executable notebook cells match the preserved upstream source. The upstream URL says V32 while its original header says V33; refer to the pinned snapshot, not an assumed historical version.
  • [x] B0-RUN: selected Kaggle kernel is COMPLETE, checked at approximately 10:11 MYT. Downloaded execution modules match our local modules. The log records 41.1 minutes for the visible run, 2,539,608 reference spectra, 712,199 candidate records, one single-spectrum and one merged-spectrum FPNet on CUDA, and bank entries used for 354 visible molecules. Candidate-record counts are not exhaustive scorer-unique identity counts.
  • [x] B0-OUTPUT: downloaded submission.csv, submission_pv.csv, and submission_ours.csv each contain 400 unique molecule IDs matching the retained visible test, 2–25 parseable SMILES per row, and zero CCO guesses. Each CSV is byte-identical to the corresponding preserved upstream visible-run CSV. This establishes visible-output reproduction, not hidden accuracy.
  • [x] B0-WIDTH: 17 rows per CSV contain fewer than 25 guesses. The project production validator rejects those under its local exact-25 policy; a separate read-only audit passes the faithful replica's 1–25 policy, which the stored official scorer permits. Do not pad or change B0 to make an unrelated policy check green. Repetitions would add no ranking credit.
  • [-] B0-SCORE: submission 56566539 completed with 0.337 public score at 02:06:26 UTC (10:06 MYT). Its timestamp is consistent with the selected B0 kernel run (01:24:16 UTC), but Kaggle does not expose the source-kernel/version binding or uploaded-file hash. Record this as a real account result with probable but unproved B0 linkage, not a version-bound baseline metric; private score is not exposed. The earlier 0.175 belongs to another BASE-02/V9 submission.
  • [x] LB-SNAPSHOT: the 26 September ~11:37 MYT CLI snapshot shows top-five public scores 0.425, 0.412, 0.401, 0.400, 0.400. Treat 0.400 as a moving, rounded cutoff—not guaranteed fifth place. B0's account score is 0.337; the snapshot does not establish our exact rank.

    • Evidence: [inventory/hashes](reports/baseline-top5-review-20260926/baseline-inventory.json), [source parity](reports/baseline-top5-review-20260926/remote-parity.json), [run log](reports/baseline-top5-review-20260926/kaggle-v1-output/remote-log-final.txt), [replica-policy validation](reports/baseline-top5-review-20260926/scorer-input-validation.json), [project exact-25 result](reports/baseline-top5-review-20260926/output-validation.json), [leaderboard snapshot](reports/baseline-top5-review-20260926/leaderboard-final.csv), [scored submission](reports/baseline-top5-review-20260926/own-submissions-scored-20260926.json), [provenance freeze](reports/baseline-top5-review-20260926/baseline-provenance-freeze-20260926.json), [runtime-ID writer tests](experiments/top5-01-runtime-contract/test_runtime_output.py), [visible CSV round-trip](experiments/top5-01-runtime-contract/results/visible-v1-roundtrip/audit.json), [visible scorer-identity audit](experiments/top5-02-fallback-identity/results/visible-output-scorer-identity-audit.json), [fallback helper tests](experiments/top5-02-fallback-identity/test_candidate_fallback.py), [COCONUT identity audit script](experiments/top5-02-fallback-identity/audit_coconut_identity.py).


    The source run did not emit hashes of its runtime test/weights/feature archives. Matching visible IDs and output bytes does not fill that provenance gap or reveal hidden-run routes.


    Actual architecture, not the notebook marketing labels


    References below are to the unchanged casmi_engine.py unless otherwise stated.


    1. Candidate generation: COCONUT + training structures; median inferred neutral mass across a molecule's spectra; 10-ppm window, 30 ppm only when empty (:120–157,307–340).

    2. Spectral channels: direct entropy and precursor-shifted matching; top 80 analogue structures inside ±200 Da; overall and polarity-specific representative libraries (:165–245; pv.py:10–24). COCONUT contributes structures/fingerprints, not new reference spectra.

    3. Neural channel: single-spectrum + merged-input FPNet, fingerprint-logit scores (pv_fp.py). The supplied checkpoint pair already exists; do not restart the old FP-01 from-scratch project as a prerequisite to using B0.

    4. Fusion: eight 51-feature classifiers plus eight 31-feature PV classifiers; 88% PV / 12% other within-molecule rank blend. Direct-library similarity ≥0.85 adds +10 before ranking (:368–410,567–574). This is not calibrated probability averaging.

    5. Postprocessing: retain the first two base predictions, use bank rows with fitness >0.05, supplement with generated isomers, preferentially insert variants at ranks 4, 6, 8, 10 (:447–545). Protection of ranks 1–2 is unconditional, despite “high-confidence” wording.

    6. All three output variants share the bank/isomer postprocessor. submission_ours.csv is not a fully FP-free ablation, because generated-isomer ranking can still use FPNet logits.


    Review judgment: we already have a working multi-channel retrieval/ranking system. The highest-priority research is better ordering of mass-compatible candidates and more reliable evidence fusion, not indiscriminate database expansion or an immediate large de-novo model.


    2. P0 — Establish a trustworthy baseline measurement


    Do these before interpreting a new score. Keep changes bounded; do not repeat unrelated historical audits.


  • [-] B0-01 — Bind the official baseline result. Submission 56566539 is complete at 0.337 public; its timestamp follows the B0 kernel run and is consistent with a B0 submission. Kaggle CLI does not expose the uploaded-file hash or source-kernel/version binding. Retain it as an account score with probable but unproved B0 linkage; resolve only if binding evidence appears. Do not resubmit merely to confirm the old result.
  • [-] B0-02 — Freeze inputs and reproducible packaging. [Provenance snapshot](reports/baseline-top5-review-20260926/baseline-provenance-freeze-20260926.json) records hashes for the local notebook, train/visible-test/sample files and RDKit wheel; current inventories and declared licenses for all eight attachment slugs; plus kernel/run/submission timestamps. Recursive basename lookup chooses the shortest path (casmi_engine.py:26–30); current inventories show unique required filenames, but do not establish which historical dataset versions/content hashes run v1 mounted. Kaggle also omits remote checkpoint/ranker/bank hashes. The visible log confirms the precomputed pool was absent and COCONUT+training structures were rebuilt. Still unresolved: historical asset versions, runtime NumPy/sklearn/Torch/CUDA/seed/hardware, checkpoint/ranker hashes, and actual fingerprint-bit order versus the hardcoded 6,930-bit unpacker. The optional precomputed-pool loader catches broad exceptions and silently rebuilds (:120–137).
  • [-] PKG-01 — Fix output handling in an isolated engineering challenger. experiments/top5-01-runtime-contract/ now has an isolated writer deriving unique nonblank IDs from runtime spectra; sample order is accepted only for an exact ID-set match; scorer width defaults to 1–25 with exact width opt-in; it validates rows/SMILES, serializes, reparses, then atomically replaces. 26 tests pass: repeated spectra, reordering, changed/disjoint/malformed/empty IDs, short valid output, blank/invalid guesses, width/schema, replacement failure and preserving/cleaning a prior file. The helper round-tripped all three archived B0 visible CSVs against runtime test IDs byte-for-byte. It is not integrated in the next notebook; B0 still joins to sample IDs and writes before the notebook-level check. The detached baseline_output.py remains unimported. Next: integrate this helper only in a separate authorized challenger and run visible end-to-end parity.
  • [-] ID-01 — Unify chemical identity without silently changing B0. Static review found base-shortlist candidates are keyed via tautomer-canonical InChIKey14 but parsing/canonicalization failure falls back to stored pool keys (casmi_engine.py:452–487); bank/generated entries are mostly filtered by raw SMILES with no common final scorer-identity dedup (:464–538; generator uses ordinary InChIKey14). An audit of the three archived visible CSVs finds 0 invalid/unidentified guesses, but repeated scorer identities occur in 11/400, 12/400, and 10/400 rows (13, 14, and 12 repeated slots respectively); every repeated slot is a different SMILES spelling of an identity already present earlier in the row (0 literal text repeats). See [identity audit](experiments/top5-02-fallback-identity/results/visible-output-scorer-identity-audit.json) and [duplicate-type crosswalk](experiments/top5-02-fallback-identity/results/visible-output-duplicate-types.json). This establishes identity-level redundancy, not correctness or score impact. The [challenger helper](experiments/top5-02-fallback-identity/candidate_fallback.py) has focused cross-source tautomer-dedup tests, but is not integrated. A read-only full 436,389-row local COCONUT stored-key vs pinned-scorer audit remains active in tmux session casmi-top5-coconut-id; as of 12:25 MYT it had not produced a report (log still empty), so no mismatch rate is verified. Its dated log is reports/baseline-top5-review-20260926/coconut-identity-audit.log; do not start a duplicate run. This local bundle is not proven identical to the historical Kaggle mount. Next: collect the existing run if it finishes, then bank/generated crosswalk where evidence is available; quantify changes only in matched, answer-key-authorized offline evaluation.
  • [x] ID-01 visible-output subcheck — all 39 repeated slots across the three output variants were crosswalked to their literal SMILES strings: 0 literal duplicates, 39 alternative SMILES spellings collapsing to an identity already ranked earlier. This narrows the duplicate type but does not establish whether canonicalizing all tautomers/representations at output time changes ranking behavior or accuracy. Evidence: [duplicate-type crosswalk](experiments/top5-02-fallback-identity/results/visible-output-duplicate-types.json).
  • [-] BANK-01 — Retain and qualify the selected bank. Preserve the included bank and B0 identity; the visible run used entries for 354 IDs. The consumer selects by molecule_id, applies physical_fitness > 0.05, and does not verify query-content provenance or recompute fitness (casmi_engine.py:464–470,499–505). Current Kaggle metadata declares CC0-1.0, but bank source ancestry, producer/spectrum/fitness provenance and historical attached version/hash remain unaudited. ID-specificity alone proves neither leakage nor a rules issue. Preserve it; never recreate the deleted no-bank variant. Important bug: empty base candidates cause 25 CCO outputs and continue before a loaded bank can be used (:490–492). The isolated [fallback helper](experiments/top5-02-fallback-identity/candidate_fallback.py) now has 10 passing fixture tests for bank-first/nearest-mass fallback, scorer-identity dedup, bounds, and fail-closed malformed inputs; it is not integrated or evaluated. Test and fix only in a bounded challenger.
  • [-] CHEM-01 — Qualify mass inference and retained-bank fallback. B0 estimates one median neutral mass, which can hide acquisition/adduct disagreement. Its no-base-candidate branch bypasses bank entries and emits CCO placeholders; add a tested bank-first, then nearest-mass eligible fallback only in a challenger. The isolated helper now rejects malformed/non-finite/negative nearest-mass errors rather than treating unparseable errors as unbounded candidates, and rejects a non-finite fitness threshold; the helper suite passes. This is not production integration or performance validation. Still audit unknown/adduct-conflict cases and finite/sorted/aligned candidate arrays. The formate mapping in this code is correct; do not repeat the unrelated old fix. The 10/30-ppm windows are model heuristics, not scorer rules.
  • [ ] EVAL-01 — Freeze the paired evaluation contract. Use all scoreable spectra per molecule, canonical scorer identities, identical candidate pools/eligibility and denominators for compared rankers. Preserve source/instrument/adduct/mass/spectrum-count strata. A: exclude the query acquisition/duplicates while permitting independent same-identity references. B: retain the structure but remove every held-out identity's spectra from references, training pairs, prototypes and spectral-derived features. C proxy: additionally embargo target structures/scaffolds from training targets, stored candidate databases and precomputed bank/generation assets. Legitimate new generation from allowed runtime inputs remains eligible; never use evaluation truth to delete newly generated answers. Keep failures in the denominator. Done when: masks/joins and scorer agreement pass fixtures and each comparison is explicitly scoped.
  • [ ] EVAL-02 — Audit inherited checkpoint and feature exposure. Shipped FPNet and ranker archives may already have seen our training-derived evaluation identities; masking retrieval alone does not undo this. Establish their split provenance. If full exclusion cannot be proved, label end-to-end results exposed diagnostics and use a genuinely independent evaluation set or retrained controlled research branch before making clean-generalization claims. All new learned features/calibrators need query/identity-grouped OOF or nested development splits. Do not score/tune the frozen reports/hidden-sim-diagnostic-panel-20260926/ panel, whose manifest forbids it. Its 15/50/35 quotas are not hidden class estimates. Reuse appropriate existing contracts, not incompatible legacy metrics such as 0.959 vs 0.227.
  • [-] OBS-01 — Save candidate-level evidence and cache expensive stages. Static review confirms recs contains useful features/routes but notebook cell 11 discards it after generating submissions. Challenger should persist candidate scorer ID, source, eligibility, direct/shifted/analogue/FP/fragment scores, bank/generator provenance, pre/post ranks, final ranks, runtime/memory and fallbacks. Key caches by source + query content + preprocessing + weights + bit schema, never ID alone. Configure BLAS/Numba/Torch threads before imports—the notebook imports NumPy before casmi_engine.py sets OpenBLAS—and reuse variants across output branches. Evidence persistence/parity is not implemented.

    • First gate: B0 source and visible outputs remain preserved. The account has a completed 0.337 public score with probable, unproved B0 linkage. PKG-01 is isolated, tested, and byte-round-tripped. Visible-output identity audit found 39 repeated scorer-identity slots, all different SMILES spellings and no invalid guesses. The isolated fallback helper passes 10 tests; the full local COCONUT crosswalk remains active but has not produced a report. Challenger integration, historical consumed-asset binding, exposure checks, and paired accuracy evaluation remain open. No improvement claim or authorization to alter B0 follows from these diagnostics.


    3. Primary paper references and adaptation boundaries


    Primary abstract/method snapshots were retrieved on 26 September and retained under [reports/baseline-top5-review-20260926/literature/](reports/baseline-top5-review-20260926/literature/). Paper percentages are not expected CASMI gains. The proposed implementations below are adaptations, not reproduced paper results.


    RefVerified paper/versionTechnique to borrowImportant limit
    P1[Small molecule retrieval from tandem mass spectrometry: what are we optimizing for?](https://arxiv.org/abs/2602.16507v1), 18 Feb 2026Candidate-aware/listwise contrastive objectives; study mass/formula hard negatives instead of optimizing fingerprint similarity aloneIts retrieval/hit-rate analysis is not direct optimization or proof of CASMI MRR@25; candidate composition matters
    P2[JESTR: Joint Embedding Space Technique for Ranking Candidate Molecules for the Annotation of Untargeted Metabolomics Data](https://arxiv.org/abs/2411.14464v3), v1 18 Nov 2024; v3 7 Jun 2025Joint spectrum/molecule embeddings; cosine retrieval; late regularization with structurally similar same-formula candidatesOur full high-resolution/multi-spectrum input and candidate pool differ from its binned-spectrum pipeline; do not blindly copy preprocessing or oracle formula access
    P3[GLACIER: Rethinking Mass Spectrum Prediction as an Object Detection Problem](https://arxiv.org/abs/2606.29161v1), 28 Jun 2026Molecular-graph fragment detection and intensity prediction, followed by spectrum-comparison/contrastive retrievalHigher-cost forward-model research; no imported paper checkpoint/code; its training needs fragmentation supervision/representation work
    P4[MIST-CF: Chemical formula inference from tandem mass spectra](https://arxiv.org/abs/2307.08240v1), 17 Jul 2023Rank formula/adduct hypotheses using fragment-subformula evidence rather than mass error alonePaper assumes positive singly charged ions; negative/multicharge adaptation is separate. Formula does not distinguish same-formula isomers
    P5[MSAlign: Aligning Molecule and Mass Spectra Foundation Models for Metabolite Identification](https://arxiv.org/abs/2605.19752v1), 19 May 2026Cheap cached representations, lightweight projections, candidate-based InfoNCE, explicit leakage/domain-shift checksPaper uses frozen DreaMS/ChemBERTa. They are not authorized new assets here; a head on existing baseline features is only inspired by its training/caching idea, not MSAlign reproduction

    P1 — First ranking experiments (one controlled change at a time)


  • [ ] RANK-01 — Retrieval-aligned head / objective pilot (P1; P5-inspired caching). On cached, exposure-qualified baseline spectrum representations and candidate fingerprints, independently train a small candidate scorer. Compare plain candidate scoring, BCE where applicable, and candidate-based contrastive softmax; use the same queries/pool/budget. Sample negatives from the actual mass-eligible candidate distribution, add same-formula hard negatives as a separate ablation, exclude scorer-equivalent positives, and weight molecules rather than letting large candidate lists dominate. Start frozen-backbone; do not change B0 checkpoints. Insert: new score alongside blk_base / BLOCKS, then controlled fusion—not a replacement pipeline. Deliver: per-molecule ranks, errors and paired MRR@25 versus B0 and mass-only on the same pool. Reject: better fingerprint metrics/training loss without incremental end-to-end MRR, leakage, or prohibitive cost.
  • [ ] FUSE-01 — Refit fusion from honest OOF features. Retain B0's rank blend, +10 library boost, protected first two ranks and insertion order as the control. Test calibrated confidence/margins and cross-channel agreement one at a time using query-grouped training. Recompute all set-relative features when pool/channel definitions change; version both feature schema and fitted ranker. Do not feed truth availability, label-derived regime or sampler-specific source flags. Deliver: displacement/win/loss counts per molecule and regime, including formerly-correct high ranks. Reject: in-sample AUC gains, a bundled hyperparameter sweep, or unexplained A/B-domain damage. Do not assume more/deeper trees must help.
  • [ ] VIEW-01 — Metadata-preserving multi-spectrum evidence (engineering hypothesis, not a claimed paper result). Keep the existing two model views. Compare per-adduct/polarity/charge-compatible aggregation or a small metadata-aware set head with the current merged-peak/median-precursor/first-adduct behavior (pv_fp.py:132–210). Preserve each surviving spectrum's own metadata and missingness. Avoid naively feeding a pretrained model inputs outside its trained distribution. Deliver: same-data single/merged/compatible-group comparison and spectrum-order/empty-view fixtures. Reject: a gain restricted to contaminated or hand-selected queries; unsupported adducts must retain a declared fallback.
  • [ ] JESTR-01 — Add a complementary learned graph-retrieval score (P2). Independently implement a compact spectrum/structure dual encoder with contrastive training and a separately ablated late hard-candidate regularizer. Use only permitted training pairs, audit equivalent/held-out negative exclusions, and cache molecular embeddings. Keep the full baseline mass-eligible candidate pool for the rank-only experiment. Insert: a new versioned BLOCKS score into the baseline fusion. Deliver: cosine-rank component results and B0-plus-component MRR@25 on matched queries, including a predefined same-formula stratum. Reject: no complementarity after fusion, loss of recall, or insufficient memory/runtime headroom. Do not port the old competition-only JESTR contract without rebinding it to B0's pool.

    • Suggested first research run: RANK-01 on cached features, then FUSE-01. VIEW-01 is a separate low-cost hypothesis; JESTR-01 is the next representation investment if simple ranking changes leave useful headroom. Do not launch all at once.


    P2 — Conditional additions selected by measured failure causes


  • [ ] FRAG-01 — Candidate-to-spectrum reranking (P3). Proceed if the true structure is usually in the eligible shortlist but loses among isomers. Begin with a bounded independently trained graph-to-fragment/intensity predictor; design the required fragment targets on eligible training data rather than pretend a full GLACIER implementation is a small wrapper. Score predicted vs observed peaks across actual adduct/energy conditions. Choose shortlist size by measured truth retention and runtime, not a fixed small mass-sorted cap. Controls: B0, current MetFrag-lite and the best new ranker on the same shortlist. Deliver: shortlist recall, full-cohort/conditional MRR, per-molecule changes and worst-case cost. Reject: improved spectrum similarity without retrieval gain. On model/coverage failure keep the upstream ranking rather than dropping all predictions.
  • [ ] FORM-01 — Soft formula/adduct evidence (P4). Proceed when error decomposition shows neutral-mass/adduct or formula confusion rather than purely same-formula isomers. Rank multiple formula/adduct hypotheses from observed peaks and precursor; use them as soft features first. Never read query truth formula/scaffold during inference. Treat positive-mode proof separately from negative/multicharge extensions. Deliver: correct-formula retention, rare-adduct failures and full-cohort MRR after the feature is fused. Reject: a hard filter that deletes more useful candidates than it improves, or using formula accuracy as a substitute for molecule ranking.
  • [ ] ISO-01 — Better ranking of bank and generated isomers (P1/P3-informed adaptation). Keep the bank enabled. Score bank-derived and on-the-fly variants with query-dependent evidence, deduplicate under the scorer and compare against B0's fixed insertion/protection policy. Preserve distinct constitutional isomers and verify formula/valence behavior of generated candidates; do not treat a raw MMFF94 energy threshold as calibrated identity confidence. Deliver: number of newly correct structures, rank promotions, displaced correct candidates and total MRR. Reject: gains measured only on the visible-ID bank or on cases selected because B0 failed. Bound expansion count and wall time during enumeration: the current generator applies max_variants only after it has enumerated variants (regio_generator.py:26–191).
  • [ ] LIB-01 — Optional reference-library expansion, not the main paper track. Only revisit MassBank harmonized or other data if reference-missing error strata justify it. Audit record-level licenses, overlap with current training spectra/structures, acquisition duplicates and historical CASMI subsets; distinguish structure additions from spectral additions. Apply the same evaluation exclusions to external spectra. Deliver: incremental eligible references and a paired result, not raw downloaded counts. No broad PubChem/COCONUT rebuild solely to enlarge a number.
  • [ ] GEN-01 — Defer general de-novo generation until pool absence is demonstrated. First exhaust bounded parent/isomer improvements already supported by B0. A later paper-inspired generator requires a separately justified design, permissible training assets, deployable conditioning, useful exact-identity novelty and measured marginal MRR after shared reranking. Valid SMILES or a lower generation loss alone are insufficient. Do not revive an old Phase-4 run simply because it exists.

    • 4. Evaluation, promotion and leaderboard gates


  • [ ] GATE-01 — Cheap controls and denominators. Evaluate every ranker against B0 and mass-error ordering on identical eligible candidates. Use random/tie-order and spectrum-independent controls to detect shortcuts. Define hard-isomer strata by chemistry before predictions—not by cases a component already failed. Score full cohort first; publish covered-only metrics separately.
  • [ ] GATE-02 — Incremental value and reproducibility. Save per-molecule first-correct ranks and paired delta-MRR; estimate uncertainty by resampling whole molecules, not spectra/candidate rows. Repeat finalists with recorded seeds; investigate any unexplained prediction differences. No universal ±0.006 noise rule from another notebook. A small uncertain gain stays provisional; require no unexplained severe regression in an intended regime and actual offline cost headroom.
  • [ ] GATE-03 — Development submission decision. Bind a single hypothesis, immutable code/assets and validated output to each authorized submission. Check current daily limits and outstanding evaluations; change one experimental lever at a time. Record official result without claiming which hidden route caused it unless evidence exists. Never probe hidden labels or tune directly to hidden answers. Public LB is a noisy selection signal, not a replacement for validation.
  • [ ] GATE-04 — Finalist acceptance. Reproduce each finalist twice offline on Kaggle; measure actual hardware/memory/wall time and boundary-query behavior. Reconfirm current competition rules, runtime ceiling and final-selection deadline before final submission. Existing project notes say ≤9 hours and recommend ≤6 hours for headroom; these are to be reverified, not assumptions that every new model fits. Select only eligible, reproducible best/complementary entries allowed by the current rules. Record final private MRR and official rank; mark the top-five objective complete only on that evidence.

    • Experiment record required for every challenger


    experiment_id, parent baseline/champion, paper + version + exact borrowed mechanism, change hypothesis, code/notebook hashes, consumed asset versions/hashes/licenses, fold/exposure masks, candidate/bit-schema hashes, query-content IDs, seeds, training/validation/report partitions, per-molecule component/final ranks, bank-hit/fallback provenance, MRR@25, Top-1/5, Recall@25 and shortlist/pool recall, same-formula/domain strata, candidate displacement, uncertainty, wall time, peak RAM/VRAM, official submission ID/result if authorized, and keep / reject / inconclusive decision.


    MRR@25 is the mean reciprocal rank of the first matching scorer identity; absent answers and rank >25 contribute zero. AUC, hit rate, fingerprint similarity, candidate coverage and valid CSVs answer different questions.


    5. Immediate execution queue


    1. B0-01: score lookup is complete: 0.337 public, likely but not conclusively version-bound to B0. Do not resubmit just to establish provenance.

    2. B0-02 + PKG-01 + OBS-01: provenance snapshot and isolated output helper are captured; bind historical assets/checkpoint/bit schema where possible, finish integrating PKG-01 only in an authorized challenger, and build candidate-level evidence/caches. Keep B0 immutable.

    3. ID-01 + BANK-01 + CHEM-01: visible B0 identity audit is complete; its archived-output report found repeated identities but no invalid guesses. The isolated bank-first/nearest-mass fallback helper has 10 passing tests, including malformed mass-error and threshold handling. Collect the already-running COCONUT crosswalk (do not launch a duplicate); then qualify bank provenance and quantify fallback/dedup effects only in a separate challenger. Preserve the selected bank; do not change B0.

    4. RANK-01 → FUSE-01: small candidate-aware ranking experiment on cached evidence, with a documented decision before any larger GPU run.

    5. VIEW-01 / JESTR-01, then FRAG-01 / FORM-01 / ISO-01 only as error evidence warrants. Carry successful changes forward against the latest measured champion while retaining original B0 comparisons.


    Work performed for this file: read-only code/attachment and paper review; retrieval of completed B0 run; official score/leaderboard snapshots; local hash/provenance audit; isolated runtime-output helper (26 tests + byte-identical round-trip of three outputs); separate bank-first/identity fallback helper (10 passing tests, now fail-closed on malformed mass errors and non-finite threshold); visible-output scorer-identity audit (0 invalids, 39 repeated alternative-SMILES identity slots) and duplicate-type crosswalk. The full public-derived COCONUT identity audit remains active in the existing tmux session with no report yet; this run is not evidence about the historical Kaggle dataset mount. No B0 source edit, no no-bank recreation, no training, kernel push, or new competition submission.