G1 evaluation validity audit (2026-09-19)

Scope: read-only audit of current source and saved phase1-g1-all-20260918.json; no production artifacts changed. Independent probe output is probes.json in this directory.

Gate verdict

Do not gate this as a CASMI26 G1 score or a leakage-free prospective benchmark. The saved result is usable as qualified local diagnostic evidence only. The current all-spectrum scorer fixes the previously observed coverage/MRR denominator issue and row/query grouping issue, but its headline library ceiling is intrinsically an upper-bound/self-library experiment, and Regime A is a local exact-fingerprint proxy that deliberately retains 56,089 other-library spectra sharing a validation identity. There is no acquisition/run provenance or lineage guarantee. Regime B's zero coverage is an exclusion-control result, not model quality.

Findings (prioritized)

F1 — High: library-ceiling score is not prospective leakage-free performance

src/g1_molecule_eval_all.py:48-64 scores every valid training row as a query and searches the same global training index; for library_ceiling, it excludes only the exact query row (ref == row_offset, line 59), not the query identity's other spectra. It then takes the maximum score per target identity (lines 61-64). This is correctly labelled library_ceiling in the saved report, but it means the 83.53% Top-1 / 86.24% MRR@25 result is a same-library ceiling, not hidden-test performance. The saved result explicitly has 294,944 queries for 5,974 identities and 97 identities with zero valid queries (reports/phase1-g1-all-20260918.json:12-27; independent recomputation in probes.json).

The all-spectrum max aggregation also lets a molecule's best of hundreds of query spectra determine its rank (maximum query count 1,422). This is a valid diagnostic definition only, not a single unknown-spectrum estimate. Per-molecule query counts and ranks are retained in the report, which is a positive provenance feature.

F2 — High: Regime A target identity leakage is explicitly retained

src/regime_a_exclusions.py:25-31,55-64,70-73 excludes exact available-field fingerprints, but intentionally retains non-exact rows from other libraries with the same scorer identity. The frozen policy reports 56,089 such rows (manifests/rebuild-20260917/regime-a-policy.json, reference.retained_other_library_same_identity_rows). Independent row-level checks reproduced: 1,184 query rows, 2,538,424 eligible reference rows, zero mask overlap, and 56,089 retained same-identity rows (probes.json).

Consequently Regime A's 88.0% Top-1 and 92.97% MRR@25 (phase1-g1-all...json: regime_a) measure retrieval where the exact answer identity remains in the reference library. They are not novelty or acquisition-independent generalization. The code and policy do disclose this limitation (regime_a_exclusions.py:73), so this is a claim/gate limitation rather than an unnoticed implementation bug.

F3 — High (latent/stale path): bounded evaluator reports wrong conditional-MRR denominator

src/g1_molecule_eval.py:48-51 defines conditional_mrr_at_25 using len(r25), i.e. only targets rankable within 25, rather than all pool-covered targets. This is inconsistent with the corrected all-spectrum evaluator (src/g1_molecule_eval_all.py:70-76, which uses covered). It can inflate conditional MRR whenever a target is pool-covered but rank >25. The saved all-spectrum report does include pool ranks >25: 13 library-ceiling identities are pool-covered beyond 25 (probes.json), so the stale evaluator is not semantically interchangeable with the reported all-spectrum result. The maintained all-spectrum artifact appears internally consistent; this finding blocks relying on the older evaluator for acceptance, not the saved corrected metric.

F4 — Medium: strict novelty proxy is tautologically the full training identity set

src/strict_validation.py:71-74 defines novelty_ids as every identity whose scaffold fold is in range(folds). Since scaffold_fold is assigned to every scaffold in that range, novelty_ids == identities by construction. The later manifest text calls this a scaffold-group holdout (lines 99-113), but the target set is not a held-out subset; only the fold-specific stats partition it. This does not alter the current 5,974 structure-only G1 cohort or the saved A/B metrics, but it invalidates any future claim that strict-validation.json's novelty_proxy itself is a novelty evaluation.

F5 — Medium: no lineage/fingerprint guarantee beyond the local available fields

src/regime_a_exclusions.py:15-23 fingerprints selected metadata, precursor and peak arrays but excludes normalized structure identity by design. The module documents that no raw acquisition/run IDs exist and that exact matching is only a proxy (lines 1-7,71-73). Thus it cannot detect same acquisition lineage when metadata/serialization differs, nor establish organizer-level reprocessing separation. Regime B does provide identity-wide spectral exclusion (src/regime_b_exclusions.py:1-5,54-64 and src/phase1_baseline.py:147-156), but B intentionally makes every target absent from the local spectral pool, yielding zero target coverage. No current artifact establishes true CASMI hidden-test leakage status.

Checks that passed / discrepancies not found

Train/index row identity alignment independently checked over all 2,539,608 rows: 0 mismatches; index has 2,160,035 valid rows (probes.json).

Regime-A query/reference masks are disjoint (0 overlap), and the recomputed retained same-identity count matches policy (56,089).

Strict cohort IDs are all present in the frozen split (0 missing); Regime-B fold masks and the all evaluator's fold selection are identity-wide and fail-closed through load_regime_b_policy (src/phase1_baseline.py:128-157).

Current all evaluator's global MRR denominator is all molecules (g1_molecule_eval_all.py:70-76), and pool_coverage includes targets found at ranks beyond 25. Saved library values show 13 pool-covered ranks >25 and 575 pool misses; they are not silently treated as rank-25 successes.

* Tie ordering is deterministic by (-score, identity) (g1_molecule_eval_all.py:65-68); empty pools produce None and zero contribution rather than a fabricated rank.

Recommended disposition

Qualified local PASS only, with explicit non-submission/non-prospective caveat; CASMI G1 gate: FAIL/NOT ESTABLISHED. Preserve the current report as a diagnostic ceiling and exclusion-control artifact. Do not report 83.53% or 88% as official/hidden-test predictive accuracy. A valid prospective gate requires organizer scorer parity, true test/query provenance, and acquisition/lineage-safe references (or a clearly accepted identity-wide holdout); the present local artifacts do not provide those.