Local evidence audit for the 22 September analogue-plan review
Scope: read-only inspection of current local source, small saved reports/manifests, NPY headers and retained Phase 4 output text. No training, model/checkpoint inference, training-data scan, Kaggle/network job, pipeline/config edit or gate promotion. The only new file is this report. Notebook-source review belongs to the other reviewer.
Source anchors below refer to the files inspected on 2026-09-22. Parent is editing the plans concurrently; original-document anchors therefore use reports/plan-review-20260922-analog/before/. Completed results remain historical diagnostics unless their stronger acceptance contract is separately established.
Highest-priority corrections and explicit TODOs
P0-1 — Qualify, reconcile and revalidate the purported locked retrieval baseline
Verified conflicts:
manifests/retrieval-baseline-locked-20260921.json:5-16 configures 20 ppm / uncapped / top_n=200, but :42-48 copies 30 ppm selected-setting metrics. The saved 300-identity report gives 20 ppm MRR 0.870833 at reports/g1-ablation-mass-window-cap-20260921-scale300/results.json:138-149, versus 30 ppm 0.879722 at :194-205. Neither is a newly validated optimum.artifacts/rebuild-20260917/retrieval-index/metadata.json:1-5. Both NPY headers were read independently: (2539608, 64). The ablation uses top_n=mz.shape[1], scripts/run_g1_ablation.py:69-73.manifests/molecule-grouped-5fold.json, at :112-115. That file declares group_key=inchikey14, 275,810 groups at :2-7. The ablation actually reads the rebuilt scorer split and strict cohort at scripts/run_g1_ablation.py:183-190; the authoritative split declares 274,195 scorer identities at manifests/rebuild-20260917/molecule-grouped-5fold-scorer.json:2-9.src/g1_molecule_eval_all.py:156,189-196 uses 10 ppm; src/phase1_baseline.py:374,458-485 likewise defaults to 10 ppm and its CLI submission call supplies no override. A manifest is not proof deployed behavior changed.TODO: preserve the existing lock as disputed historical configuration. Establish a single effective-config contract covering query/reference peak width, ppm, absolute floor if used, cap unit/order, similarity, aggregation, actual scorer split/strict masks, index/cache and report hashes. Distinguish a consciously chosen cost/quality tradeoff from the highest saved MRR. Require resolved-config output and a fail-closed match at execution entry points before a new lock. Do not silently change 20 to 30 or promote either setting from the present ablation.
P0-2 — Repair filter/cap/aggregation order before interpreting the mass-window sweep
collect_and_score keeps only the highest-scoring reference/query pair per candidate identity at the widest window, saving that pair's mass distance (scripts/run_g1_ablation.py:81-103). Narrow windows subsequently filter those identity winners (:130-137). It also uses the first valid query's mass as the tolerance scale for all winner pairs (:81-82,125-131).
This loses lower-scoring, narrower-window-eligible evidence; filter and max-by-identity do not commute. The report's claim of equivalence at scripts/run_g1_ablation.py:294-295 and saved report :258 is false despite cosine itself being window-independent.
Fresh synthetic reproduction, using AST-extracted current functions and fake in-memory spectra only: same identity has a 5 ppm reference scoring 0.6 and a 25 ppm reference scoring 0.9. Widest-window aggregation retains only the latter; the 10 ppm setting incorrectly reports target coverage 0 instead of 1.
TODO: specify per-query eligibility → mass filter → explicitly defined row/identity cap → spectral scoring/eligible aggregation → scorer-identity ranking. Retain sufficient per-reference/per-query records (or a proven equivalent sufficient statistic) to replay each setting. Persist source rows, individual query masses, masks, cohort IDs and all retained/dropped target evidence. Prove replay against direct per-setting retrieval on synthetic/adversarial fixtures before any bounded rerun; rederive bootstrap comparisons only from corrected, paired molecule records.
P0-3 — Restore cosine order after candidate-cap selection
evaluate_setting starts from cosine-sorted records, but when the cap binds it mass-sorts and truncates, then uses that mass order as final rank, scripts/run_g1_ablation.py:133-144. The bootstrap helper repeats this at :233-240.
Fresh synthetic reproduction: cap=2 retains target score 0.99 and decoy score 0.5, but the nearer-mass decoy is ranked first; actual MRR is 0.5 instead of cosine-ranking MRR 1.0. This changes the ranking algorithm as well as membership.
TODO: select cap membership with an explicit deterministic tie rule, then restore the documented final score ordering. Include cap=0, nonbinding cap, binding cap with target retained, target dropped, equal-distance and equal-score cases. Current claims that cap=100 harms retrieval cannot separate real retention loss from this bug. Moreover, the recorded loss is not approximately 0.09 at every window: saved 300-cohort differences are about 0.0381 / 0.0462 / 0.0902 / 0.1127 for 8.5/10/20/30 ppm, before semantic correction.
P0-4 — Separate rows, identities, comparisons, pool coverage and ranked recall
candidate_rows_scored increments by the length of already-deduplicated identity records, not reference rows or actual cosine comparisons (scripts/run_g1_ablation.py:52,95-103,136,154). Caps likewise apply to identities after aggregation, not spectral rows. This cannot be used to prove a 50% inference-work increase, or that a row cap >=1,000 never binds.pool_coverage is calculated from the target's presence after top-1000 truncation, and conditional MRR uses the same truncated denominator (:137-158).pool_coverage=0; actual pre-top-K pool coverage is 1. The coverage×conditional-MRR identity alone would not catch this shared-denominator mistake.runtime_seconds is only filtering/metric time (:119,162), not retrieval/scoring; the shared scoring pass happens earlier.TODO: separately emit reference rows before/after filters/caps, unique identities before/after aggregation/truncation, actual similarity comparisons, cap-bind rate, correct-candidate retention at each stage, full eligible pool coverage, Recall@25/100/1000, and MRR@25 conditional on full coverage. Persist query counts/invalid/zero-valid outcomes. Report one-off scoring cost versus replay cost honestly. Add rank 25/26/1000/1001/absent and empty-cohort fixtures.
P0-5 — Enforce one scorer identity across Phase 4 splits, evaluation and generated outputs
Current Phase 4 computes folds from raw df.inchikey14, kaggle-kernel/phase4-denovo/main.py:671-688, while src/split_manifest.py:79-89,110-126 groups src/identity.py:15-39 tautomer-canonicalized identities. Matching the blake2b hash function (main.py:82-91) does not make different identity namespaces equivalent. Validation also compares raw InChIKey14, main.py:613-619; it does not invoke the scorer canonicalizer. phase4_train.py is byte-identical to main.py, so these findings apply to both.
TODO: attach/version the authoritative scorer-group assignments or identically derive and validate them under pinned RDKit; assert no scorer identity crosses train/dev/report or regime exclusions across all libraries, tokenizer/structure targets, spectral prototypes and derived features. Use the same identity in all candidate dedup/evaluation/submission paths. Keep B structure eligibility separate from excluded spectra; C also excludes structure/target exposure. This audit verifies a split-contract mismatch, not a dataset-wide quantified leakage count; do not claim clean novelty or unseen-identity performance from this split.
P0-6 — Withdraw the absolute “fold 0 untouched” claim and create an exposure ledger
The lock and ablation report call fold 0 untouched (manifests/retrieval-baseline-locked-20260921.json:41,132,138; scripts/run_g1_ablation.py:281,293). Existing completed G1 output already contains those identities.
Independently joined saved per-molecule results to the authoritative split, without scanning training data:
reports/g1-acceptance-run-20260920-rerun/g1-results.json:168-171 records a rank-1 result for BGBDMEPBLWJCHI; rebuilt split :1141668-1141671 assigns it fold 0. Another is SRYYOKKLTBRLHT, report :132-135, split :1083033-1083036.main.py:552-556,682-703,738-745,782-792; retained /tmp/kaggle-phase4-logs/phase4-metrics.json:2-38). This is not an untouched report workflow.TODO: distinguish excluded-from-training, inspected-for-diagnostics, tuned-on and never-inspected. Record identity/cohort/version/decision exposure across old retrieval, Phase 3A and generator work. The present 300-identity sweep excludes fold 0, but cannot retroactively make the whole fold untouched. Reserve a genuinely unexposed group/scaffold-aware report cohort or use nested/grouped validation; changing seeds alone does not erase exposure. Do not claim every fold-0 identity is contaminated either—the saved evidence proves specific prior exposure and defeats the blanket claim.
P0-7 — Verify teacher-forcing/generation equivalence; do not mistake low loss for identification
Current main.py:298-325 does contain the unconditional causal mask and shifted next-token CE. These two source fixes are already present, not new pending implementation work. Historical pulled source /tmp/kaggle-pulled/enveda-casmi26-phase-4-de-novo-train.py:298-319 instead has a training-only mask and unshifted targets. The diff contains precisely these corrections; no RDKit auto-install was found in either current local copy.
Retained metrics show final loss 0.0003058483, validity 0, exact match 0 and rdkit_available=false (/tmp/kaggle-phase4-logs/phase4-metrics.json:29-38). They establish completed diagnostic output, not a successful generator. No checkpoint/model inference was performed in this audit; source-to-run/checkpoint immutability remains unproven.
TODO: before scaling or accepting a new run, add deterministic prefix-logit parity between teacher-forced next-token scoring and incremental generation, tests of shifted BOS/EOS/PAD targets, future-token perturbation invariance, causal-mask device/dtype, and cumulative beam-score recomputation (including EOS stopping). Treat source inspection/compilation as insufficient evidence these contracts run correctly. Reject copy-bug-era checkpoints as task-quality evidence; retain them as diagnostic artifacts. Separate fixed-source existence from verified corrected-run results.
P0-8 — Make chemistry/scorer dependencies fail closed and package them offline
main.py:28-33 silently makes RDKit optional. :561-637 falls back to a permissive syntax checker and string equality, returning the same metric keys; with RDKit present it still uses raw identity. kernel-metadata.json:4-13 selects main.py with internet disabled, but declares no offline wheel/model dataset attachment. Source has no pinned RDKit assertion or offline wheel installation. A recorded rdkit_available flag does not make syntax validity chemistry validity.
TODO: freeze RDKit 2026.03.3 plus compatible dependencies and wheels; verify offline import/version and scorer fixtures before scientific validation/submission. Fail closed or explicitly mark diagnostic-only when unavailable—never report fallback exact match as the official metric. Test invalid valence, invalid/empty molecules, tautomer/stereo equivalence, duplicate identities and dependency failure. Do not plan online pip install rdkit as an offline solution, or assert the memory note's install fix is present.
P0-9 — Bind every run to immutable source, resolved config, tokenizer, data and checkpoint
Phase 4 epoch checkpoints contain only model weights/epoch (main.py:545-550); the final checkpoint contains architecture and fold seed/number (:796-807), while tokenizer and metrics are separate (:810-825). There is no source/data/split/tokenizer/checkpoint-hash binding, complete optimizer/scaler/RNG resume state, or explicit Torch/NumPy training seed. Source hash alone cannot identify a historical checkpoint's training semantics.
The existing G1 prelaunch manifest exists and its evaluator hash matches current source (reports/g1-acceptance-prelaunch-20260920/launch-manifest.json:34). However its command is a template and lacks the evidence-dir flag (:82); post-run binding explicitly admits no immutable launch proof (manifests/g1-06-provenance-snapshot-20260921.json:175-180). Do not falsely say there was no prelaunch file, or upgrade it to proof of the exact executed launch.
TODO: immutable run/version directory containing exact entry source, dependency/accelerator metadata, CLI/resolved configuration, kernel revision, dataset/checkpoint/tokenizer/split/mask hashes, all seeds and molecule-level outputs; checkpoint embeds the run-manifest identity and loaders reject mismatches. Save reproducible resume state separately from final inference assets. Record prelaunch verification and post-run output hashes. Treat /tmp artifacts as discoverable evidence, not durable archival provenance.
P0-10 — Keep G1 diagnostic; strengthen the independent audit before any fresh acceptance decision
The latest saved report itself says official_g1_closed=false (reports/g1-acceptance-run-20260920-rerun/g1-results.json:2-6). Current evaluator computes pre-top-K coverage correctly (src/g1_molecule_eval_all.py:103-125), unlike the new ablation. B=0 remains expected for a library matcher with target spectra excluded, not evidence about a fingerprint/database ranker.
The post-run auditor checks row bounds, not row→identity/query/mask membership (scripts/audit_g1_post_run.py:88-93). Its B guard only prohibits candidate==current target (:147-155), not every held-out identity in that fold. It records metric drift but does not incorporate it into PASS (:134-145,201-217). Its “target sets identical” advertised check actually compares denominator counts (:205-215,237). These are audit-scope deficiencies, not proof the saved run actually leaked or has numerical drift.
TODO: independently validate every evidence source/query row against authoritative identities, fold/query/reference masks and expected cohort membership; check exact target sets and zero-candidate/zero-query identities, not equal counts. Fail acceptance on metric drift and malformed/missing evidence; test adversarial drift/set-permutation/mask violations. Preserve A's acquisition-lineage proxy caveat and C=not evaluated. A post-run arithmetic PASS, 400 placeholder rows or source fixes do not automatically promote G1.
Bounded modeling TODOs, not execution authorization
P1-1 — Evaluate unique structures, not a nominal beam width
main.py:63 says width 25 is “exactly the 25 guesses”; :329-399 actually returns raw token beams with cumulative, unnormalized sequence log probabilities and an 80-token cap. Validation hardcodes beam_width=1, :589-596; N_BEAM_VAL=10 is unused. The final metrics are loss, validity, raw exact top-1 and Tanimoto (:632-637), not MRR@25/unique recall. There is no final molecule-level prediction/submission path in this training script despite its header's dedup claim.
TODO: measure decoded/valid/EOS-complete/mass-and-formula-compatible/unique scorer identities per molecule after pooling permitted spectra; then exact Recall@K and MRR@25. Dedup before final truncation, calibrate length/route/spectrum-count scores, and measure marginal unique-correct recall and final MRR per runtime increment. Compare beam and controlled sampling as hypotheses, not a guaranteed MRR improvement. Do not pad repeated guesses or equate width 25 with 25 usable structures.
P1-2 — Add actual metadata conditioning and calibrated chemistry constraints
Current input consists only of precursor m/z as an artificial-intensity peak plus MS2 peaks (main.py:438-460,671-672,731-733; encoder :417-428). No adduct, polarity, CE, predicted formula or fingerprint conditioning is provided. Morgan fingerprints appear only in the Tanimoto diagnostic (:570-572,621-623), not as training supervision/ranking constraints. RDKit parsing is post-generation validation, not a formula/valence-constrained decoder.
TODO: implement and ablate adduct/polarity/CE/missingness-aware inputs and molecule-level multi-spectrum aggregation. Add predicted-formula/neutral-mass posterior checks, valence validity and fingerprint consistency with recall-preserving soft gates when uncertain. Distinguish beam-time grammar constraints from post-filtering; measure target retention after each filter. Never gate with oracle validation formulas. A bounded spectrum→fingerprint ranker on the same eligible pool is a defensible primary learned baseline; generator complement must show incremental held-out contribution under calibrated shared ranking.
P1-3 — Keep catalogue scope bounded and asset permissions separate
Original plan before/EXECUTION-PLAN.md:237 and TODO before/TODO.md:148 still prescribe broader PubChem expansion; supersede that standing direction with the user's no-unbounded-expansion preference. Retain a bounded, independently evaluated eligible pool; no new source ingestion is authorized by this review.
A more recent bounded artifact manifest exists at artifacts/phase3a/pubchem-structure-index-rebuilt-20260921.json:3-25: 444,792 source rows / 424,291 unique identities, explicitly post-hoc metadata and uncertified C labels. Its progress JSON still uses the old total 240,536, producing fraction 1.849 and negative ETA (.progress.json:3-8); terminal status is not a source/license certificate. This review did not rescan that database or rederive these counts.
manifests/phase3a-pubchem-provenance-recovered-20260919.json:2,10-17,24-27 records a recovered mutable CURRENT-Full URL and a license string but still says not certified. TODO: keep separate paper, repository code, checkpoint, training-data, database/source and derived-asset permission/provenance records (version, authoritative license evidence, access/offline location, attribution, competition/winner-use compatibility, overlap limits). An MIT code repository does not license all associated weights or datasets. Do not turn recovered local prose into blanket eligibility.
Documentation reconciliation that can be made now
before/EXECUTION-PLAN.md:6,449) with dated history and current unqualified experimental artifact status; retained Phase 4 logs/metrics prove a training run occurred. Do not assert a current remote run state from these local files.before/TODO.md:62,103: current src/g1_molecule_eval_all.py:136-152 writes terminal checkpoint after metrics/evidence, and saved checkpoints now have metric fields (reports/phase1-g1-checkpoints-20260918/library_ceiling.json:13-16). True resumable score/model state remains absent. Do not leave the already-fixed ordering defect as pending implementation work.before/TODO.md:58-62,101-105,255), retaining the difference between historical evidence packaging, verified current fixes and full gate acceptance.before/TODO.md:187-198) with “experimental implementation/run evidence exists; scorer/leakage/offline/scoring-equivalence/unique-recall acceptance open; no further execution authorized here.” Existing precursor conditioning is real; adduct/CE/formula conditioning is not.Verification record
evaluate_setting/collect_and_score with in-memory fake spectra and fake cosine values. Reproduced narrow-window target loss, cap-induced rank change and rank-1001 pool-coverage error. No real spectrum/model was evaluated and no source file was changed.fb89d8dd90d28938efe07acd1468a4935bbb6f2e8c61736ae7e917e89b932515.c2c3124ba39ba6af7683ad75378f26bdd161efc524c9e3b5ab511829ffff7d11; saved metrics: 55ce11853dd8720819e0362d2e683f28dcdb6fa7965057c5f1bbd995b3680d0c; saved log: ddb0d64455448b6ab34605029d255cb52a04a1a6f34015866a7a1b914b4943f5. These are audit-time hashes, not an asserted immutable source/checkpoint/run binding.Recommended decision: update plans with the above falsifiable acceptance work and bounded learned-ranking/generator-complement direction; retain current results as diagnostics; no automatic gate promotion, new experiment execution, submission or catalogue expansion.