Analog-propagation review — decisions and plan revision
Date: 22 September 2026 (MYT).
Request: review [prvsiyan's CASMI 2026 notebook](https://www.kaggle.com/code/prvsiyan/analog-propagation-casmi-2026-baseline), identify useful improvements, and update EXECUTION-PLAN.md and TODO.md.
Outcome: revised research priorities and falsifiable acceptance tasks. No new model training, heavy inference, notebook execution/push, dataset expansion, production promotion or automation changes.
1. Evidence inspected
research/pages.json, especially the competition-specific winner-license and external-data/tool clauses; this was not a fresh legal approval or a new rules scrape.The notebook was retrieved at 19:04:12–14 MYT and matched the previously retained 15:15 source exactly. SHA-256: 3ce550ab18441573ee519a4e1df88ae03090e00e424c16671e02bc6b76214a9e. The pull did not establish a version number. It contains 37 cells, zero saved outputs, and null execution counts. All performance tables are author-reported; none were reproduced here. Six attached dataset names do not prove asset licensing, availability, training provenance or fold isolation.
2. Revised next-work order
1. Repair the comparison contract: scorer-identity grouping, exposure ledger, baseline effective configuration, ablation semantics and independent audit checks. Preserve useful historical evidence without overclaiming gate closure.
2. Freeze a bounded eligible candidate pool and non-neural controls: use a documented natural-product-focused source plus permitted training structures; independently measure membership coverage. Do not resume unbounded PubChem slice acquisition.
3. Prioritize spectrum-to-fingerprint learning: binary fingerprint BCE, candidate score f · z, then same-mass and same-formula hard-negative ranking loss. Compare compatible multi-spectrum and domain-balanced variants.
4. Add mass-shifted analogue evidence in parallel: transfer structural evidence from eligible chemical neighbours onto candidate structures. This is not exact-library lookup and not copying a neighbour's SMILES.
5. Calibrate complementary channels: OOF ranker against single-channel/static controls; same-formula diagnostics; bounded fragment explanation and candidate–peak cross-encoding only if justified by errors and latency.
6. Qualify, then consider extending the generator: validate the current objective and output contract first. Predicted-fingerprint/formula/valence conditioning is a possible follow-on, not an automatic long training run or a fixed five-slot fallback.
The user has approved a documentation review, not execution of every new experiment. Previously authorized Phase 4 work is not stopped, relaunched or promoted by this revision. Model training remains off the local ARM host, including smoke training.
3. Concrete notebook lessons mapped to TODOs
| Lesson / code evidence | Adopted work | Acceptance / caution |
|---|---|---|
| Four complementary channels; wider mass-shifted analogue search, C006/C007/C012 | FP-01/02/03 and AN-01/02 | Same folds/pool; identity-wide reference exclusions; retain per-channel evidence. Current narrow direct-cosine transfer is not already this route. |
| Multi-family binary fingerprints and BCE + 63 same-mass decoys, C009/C010/C001:TRAIN | FP-01/02 | Compare Morgan control and documented bit union. f · z is independent-Bernoulli log-likelihood up to a query-only constant, not calibrated exact correctness. Audit false-negative/duplicate sampling. |
| Reference formula-derived mass option, C007/C020 | CHEM-02 | Compare eligible reference metadata to measured precursor mass; keep unavailable query truth formulas out of inference. |
| Multi-spectrum benefits but unsafe mixed-adduct metadata, C018 | FP-03 | Preserve each surviving spectrum's metadata; compatible ion groups or explicit set models, matched training/inference distributions. |
| Provenance-feature leakage and attached ranker features, C022/C026 | RIGHTS-01, FUSE-01 | No in-sample upstream features or dataset-origin shortcuts; retain licensing/OOF provenance and group-aware calibration. |
| Unused knobs and cap changes affecting all channels, C002/C012/C026/C027 | BASE-02, FUSE-02 | Bind effective parameters to execution, record answer retention and regenerate/refit set-relative features after pool/pruning changes. |
| Failure-conditioned isomer benchmark is biased, C033 | ISO-01, RERANK-01 | Define full/cohort/isomer panels independently of ranker failures; report shortlist retention before reranker quality. |
| Unsuccessful logits–fingerprint interaction heads, C032 | RERANK-01 | A candidate–peak cross-encoder is a distinct hypothesis; use OOF trunk features and compare against simple scoring under latency constraints. |
| DreaMS not an automatic analogue upgrade, C031 | EMB-01 | Optional challenger only; the reported negative test changed analogue similarity, not fingerprint-head fine-tuning. |
| Rerun score differences and unknown test mixture, C025/C035 | REPRO-01, VAL-02 | Estimate our own numerical/seed uncertainty; no universal leaderboard noise threshold or inferred exact class labels. |
4. Corrections to earlier conversational conclusions
0.151 / 0.93 ≈ 16% public Class-1 share rests on transfer assumptions. The remainder can include both Class 2 and Class 3; it is not a known 84% Class-2 fraction.5. Local issues that must not be hidden by the new roadmap
The local audit found substantive contract issues; it did not alter source or repair them in this review:
1. Disputed baseline lock: configured 20 ppm versus 30 ppm evidence; 200 declared peaks versus 64 in the index; raw split bound versus scorer split consumed; existing entry-point defaults still at 10 ppm. All ten listed hashes match, demonstrating that hash consistency alone is insufficient.
2. Ablation defects reproduced on synthetic records: widest-window identity aggregation can delete valid narrow-window evidence; a binding mass cap leaves mass order as final rank; target rank 1001 is incorrectly counted as not in the pool. Candidate counts also mean unique identities rather than spectral comparisons. Existing mass/cap/compute conclusions need corrected replay, not a new label.
3. Exposure/namespace: saved G1 records contain 1,163 scorer-fold-0 identities in each library/B cohort and 47 in A; Phase 4 repeatedly validates raw-key fold 0. These facts contradict a blanket untouched-fold claim, without proving every fold-0 identity leaked into training.
4. Generator source versus acceptance: shifted targets and unconditional causal masking are present locally. Raw-identity grouping, greedy top-1 metrics, optional-RDKit fallback and missing immutable run binding remain acceptance issues. The earlier claimed RDKit auto-install is not in either current local source copy.
5. Audit scope: stronger exact cohort, mask and row-identity joins and fail-on-drift behavior are needed for the independent G1 acceptance audit. That deficiency alone is not evidence the historical run leaked or drifted.
6. Do not redo fixed work: all-spectrum terminal-checkpoint ordering is now corrected in source. Genuine resume state remains unfinished; the old before-metrics ordering bug is not still an implementation TODO.
These are tracked as VAL-01/02, BASE-01/02, AUDIT-01, GEN-01/02 and PKG-01. Existing artifacts remain intact and unpromoted. This task has not changed the remote kernel, its current status or its monitoring schedule.
6. Files changed and verification boundary
Updated:
EXECUTION-PLAN.md](../../EXECUTION-PLAN.md): current priorities, class/coverage caveats, fingerprint/analogue objectives, chemistry/metadata handling, feature and validation contracts, generator role, rights review and revised execution ladder. Original calendar and early data inspection are explicitly historical.TODO.md](../../TODO.md): concrete task IDs, dependencies, P0 acceptance repairs, source-present versus validated status, bounded-pool preference, and corrected stale blocker/progress language. New experiments remain unchecked.Supporting evidence consists of this report, the two reviews, the read-only notebook snapshot/metadata and original-document backups. The [independent final QA](final-qa.md) requested two scope/staleness corrections; both were applied: G1-05 now claims only the exclusion check its auditor actually performed, and G1-08/A6/precursor/packaging descriptions distinguish historical blockers from later completed fixes. Its two minor wording corrections were also applied.
Final [verification.json](verification.json) records document/report hashes, link/task checks and QA-fix assertions; [changes.diff](changes.diff) preserves the final document diff. All 26 new experiment/acceptance tasks remain pending. Verification here is documentation/static evidence verification, not a full model test suite or G1/G4 promotion.