G1 independent result audit — 2026-09-19

Decision

Saved-result arithmetic and current local submission: PASS. Full G1 acceptance: OPEN.

The 2026-09-19 acceptance report's PASS WITH CAVEATS is defensible only for a qualified local diagnostic baseline. It does not establish complete reproducibility, robust offline competition packaging, unseen-compound performance, an official score, or a Top-5 trajectory. This audit found concrete remaining local issues as well as external qualifications. None of the reproduced issues demonstrates that the current saved all-spectrum numbers are arithmetically wrong.

G1 is an internal project milestone, not an organizer-issued certification. There is no official G1 score in these artifacts.

Scope and preservation

Audited current source, rebuilt index/cache/masks, saved per-molecule ranks, generation/validation metadata, and the local submission.

Recomputed reported metrics independently from saved ranks; checked current hashes and every train/index row identity against the authoritative identity cache.

Independently recomputed the documented local RDKit identity for 9,106 distinct submitted SMILES.

Ran maintained tests, two separate regression tests, current wrapper failure injection, and new synthetic runtime/provenance probes.

Did not rerun the 9-hour full spectral-ranking job, independently recompute all 274,195 cached chemical identities, submit to Kaggle, train a model, alter production source, or restart PubChem.

Source/script/test hashes and the saved report/submission were unchanged during this audit. The PubChem process was still running at the final inspection.

1. Verified numerical results

All reported metrics agree with independent aggregation of saved per-molecule ranks (floating-point residual below 1e-12). Cohort membership and rank truncation checks have zero discrepancies; no duplicate evaluation identities were found.

Regime	Molecules	Valid query spectra	MRR@25	Top-1	Top-5	Recall@25	Full target-pool coverage
Library ceiling	5,974	294,944	0.862357	83.5286%	89.5715%	90.1573%	90.3750%
A: local acquisition-exclusion proxy	250	1,184	0.929650	88.0000%	98.8000%	100.0000%	100.0000%
B: held-out identity spectra excluded	5,974	294,944	0.000000	0.0000%	0.0000%	0.0000%	0.0000%

For the library ceiling, conditional MRR@25 is 0.954199 and MRR@25 = coverage × conditional MRR@25 holds. Target-rank counts: 4,990 at rank 1; 361 at ranks 2–5; 35 at 6–25; 12 at 26–100; one at 101–1,000; 575 absent. Thus the 13 covered-but-outside-top-25 targets contribute zero to MRR@25; they are not dropped from its conditional denominator.

What the numbers mean

Library ceiling is an optimistic self-library diagnostic. It excludes the exact query row, not all other spectra of the same identity (src/g1_molecule_eval_all.py:75-89). Molecules have a median of 17 and a maximum of 1,422 valid query spectra, combined by maximum evidence. This differs substantially from an unknown runtime molecule with a few spectra.

The library cohort includes 97 zero-valid-query identities in the denominator. Another 445 identities have exactly one valid spectrum and zero target coverage after their own row is excluded. Among identities with more than 100 valid query spectra, Top-1 is 98.44%. These are descriptive strata, not a causal improvement estimate or matched ablation.

A intentionally permits eligible same-identity reference spectra. Its query/reference row masks are disjoint, with 1,184 query rows and 2,538,424 reference-eligible rows. The reference side retains 56,089 other-library same-identity rows, as the frozen policy requires. This is appropriate for known-compound retrieval, not itself a B-style identity-leakage bug. Exact acquisition lineage remains uncertified.

B=0 is expected for direct-library retrieval when answer spectra are removed. It verifies a limitation/exclusion control, not successful structure-only identification. A separate permitted structure candidate route is necessary to make B a useful identification benchmark.

C is NOT_EVALUATED, not a zero score. The union of scaffold-CV validation identities covering the whole training set is not intrinsically a defect; fold-specific exclusion/evaluation is what matters. No executed C result exists here.

None of these values can be directly compared with Kaggle public/private leaderboard MRR to infer a competition rank.

2. Fresh checks that passed

Tests and output contract

.venv-rdkit2026/bin/python -m pytest -q tests: 28 passed in 27.76s, including the real-index A/B mask tests.

.venv-rdkit2026/bin/python -m pytest -q reports/test_phase0_audit_regressions.py: 2 passed in 1.95s.

Total explicit selection: 30 passing tests. Archived copies under old reports were not included in discovery.

Submission: 400/400 expected runtime molecule IDs, correct first-seen order, no duplicate/missing/extra IDs, maximum 25 raw candidates, no blank rows, no invalid SMILES or duplicate scorer identities. 9,106 unique submitted SMILES were independently canonicalized under the local documented identity convention.

Current copied wrapper exits 41 / 42 / 43 for injected evaluator/submission/validator failure, without false downstream completion markers.

Clean direct validator invocation works without PYTHONPATH; generation metadata survives validation; a null runtime molecule ID is rejected.

Index/data checks

2,539,608 index rows, 2,160,035 valid reference spectra, 274,195 scorer identities.

All 2,539,608 train/index row labels match the authoritative cache; zero cache misses. This is a full label-alignment check, not a new full chemical re-normalization.

Current train/index identity hashes agree with the values reported by the all-spectrum evaluator.

Key current hashes match the prior acceptance report:

- Train: 9423f90ec08b941a9646b535bf0606183e86ff0c5a297978f5a70693d41bcd98

- Test: 2fa29c4aa06e93b48e0cd6619fd2de3f4e956f0019d242141678b7287f2b43e0

- Evaluation JSON: 34dd4a40db9178a3900a5e319c1d08f83da01bf9ff1a9a6157fe5d62bfc221d4

- Submission CSV: 83e3209f445f76e91ba59117fb3b6dd656699a1f1d45a2370d04432bcd1ae6d4

Pinned environment inspected: Python 3.10.12, RDKit 2026.03.3, NumPy 2.2.6, PyArrow 25.0.1.

3. Reproduced local issues and acceptance gaps

P1 — Provenance guard can accept row-misaligned artifacts

src/phase1_baseline.py:34-51 checks row counts and the set of identities against the cache. It does not bind ordered identities to peaks/masses, all array contents, runtime versions, or source data by hashes.

Fresh reproduction: swapping the two identity labels of an otherwise valid synthetic index leaves its identity set and counts unchanged; assert_index_provenance accepts it. Evidence: adversarial-probes.json → identity_permutation_provenance_probe.tamper_rejected = false.

Impact: a mixed/reordered artifact can pass the guard. No such corruption was found in the current full row-label audit. The evaluator's reported train/index hashes are literal source strings (g1_molecule_eval_all.py:161-162), not hashes computed at launch. Current post-hoc hash matches are useful but cannot replace an immutable launch manifest.

Before a strong reproducibility claim: bind source/config/runtime, training data, ordered identity labels, peak/mass/order/valid arrays, masks, SMILES mapping, and outputs in a content-addressed run manifest; reject deliberate permutations/tampering.

P1 — One null/nonnumeric precursor can abort submission generation

src/phase1_baseline.py:396-398 calls float(precursor_mz) while collecting queries, before the later protected per-spectrum handler (415-423).

Fresh reproduction on a real synthetic two-isomer index:

Valid precursor: one complete, valid dynamically named output row.

Null precursor: TypeError at line 397, aborting inference.

Nonnumeric precursor: ValueError at line 397, aborting inference.

NaN precursor: safely produces an explicit blank no-candidate row.

This corrects the preliminary runtime review's broader claim that all invalid precursor inputs were already fail-safe. The inspected current train and test files contain zero null precursor values, so this does not explain or invalidate their saved results. Define a per-spectrum malformed-input policy, move coercion into the guarded path, and add regression tests before runtime hardening is marked complete.

P1 — Legacy evaluation paths still have incompatible semantics

src/g1_molecule_eval.py:53-55 divides conditional MRR by only top-25 hits and computes pool coverage after truncation; query selection at 28-34 is identity-based even for A, without enforcing the A row query mask.

src/phase1_baseline.py:359 also uses top-25 hits as the conditional-MRR denominator.

The current all-spectrum report uses the corrected code (g1_molecule_eval_all.py:93-124) and passed independent arithmetic checks.

Do not compare legacy and all-spectrum reports as if they share definitions. Repair or explicitly deprecate the old entry points before using them for ablations or gate evidence.

P1 — Reproducible offline package and complete evaluation evidence remain unproven

No top-level dependency lock/environment manifest was found. The wrapper assumes .venv-rdkit2026 already exists; it does not enforce/record the required RDKit/NumPy runtime. The pinned local environment works, which is not a clean-machine packaging test.

scripts/validate-runtime-submission.py:13-17 hard-codes one dated submission path and data/test.parquet; a renamed output can accidentally be validated separately from the intended inference artifact. Add explicit input/output arguments and hash binding.

Saved target ranks/query counts now exist: do not repeat the stale claim that all per-molecule evidence is missing. However, complete ranked candidate IDs/scores/source rows, full quality/adduct/mass/source strata, independent ranking reruns and resumable score state are absent.

This audit supplies post-hoc frozen-fold and query-count breakdowns. They do not establish an untouched reporting partition or substitute for a predeclared tuning policy.

C-proxy execution, official scorer/reference-vector parity, and two clean offline notebook runs are not established. Exact acquisition lineage is an external qualification, not something to fabricate from the available fields.

4. Resource evidence

The actual saved completion log starts 2026-09-18 20:02:14 MYT, not the earlier 12:39 run described by stale TODO snapshots. File modification times show current evaluator/wrapper edits before this later launch, but this is not immutable launch-time source proof.

From logs/g1-completion-20260918.log:

Full local evaluation: 9h 04m 20s, peak RSS 5,137,132 KiB (~4.90 GiB), exit 0.

Placeholder inference: 1m 49.03s, peak RSS 1,506,848 KiB (~1.44 GiB), exit 0; validation completed afterward.

The evaluation exceeding nine hours is not automatically a Kaggle inference failure: the offline submission notebook need not rerun the local validation benchmark. Conversely, 1,213 placeholder spectra are not evidence of full hidden-test runtime headroom. Package inference separately and measure the actual allowed workload.

5. Disposition and next acceptance steps

1. Preserve the current baseline/report. The saved metrics and submission are useful, consistent local evidence; do not discard/rebuild them merely because external qualifications remain.

2. Keep full G1 acceptance OPEN. Narrow the earlier blanket reproducibility wording to qualified local diagnostics; do not present it as submission-ready or Top-5 evidence.

3. Before training/promotion: close the provenance and runtime-input gaps, retire inconsistent evaluators, freeze the model-selection/reporting policy, and test the offline package against dynamic inputs.

4. Continue the already authorized structure-only candidate-index work without altering the live PubChem job. Validate its completed artifact independently; do not use an interrupted/in-progress index. Candidate coverage comes before larger learned models.

5. Report C explicitly as not evaluated until its own exclusion and evaluation evidence exists. Obtain official score evidence only through an authorized compliant submission; none was attempted here.

Evidence/reproduction

All new persistent audit work is under reports/g1-audit-20260919-1005/:

artifact-evidence.json: independently aggregated metrics, rank histograms, fold/query-count strata, identity alignment, submitted-chemistry validation, runtime versions, current hashes.

recompute_evidence.py / recompute.log / recompute.exit: read-only artifact audit, successful exit 0.

adversarial_probes.py / adversarial-probes.json: current inference/provenance reproductions on synthetic data.

maintained-tests.txt, additional-regressions.txt: explicit fresh test selections.

probe-output-wrapper.py / submission-wrapper-probes.json: fresh wrapper/validator failure-injection evidence; historical probes were copied, not overwritten.

evaluation/ and runtime/: preliminary independent review notes. The present consolidated report supersedes their overbroad A-leakage/C-union terminology and the runtime review's invalid-precursor claim, as explained above.

Reproduce without launching the full evaluator or changing production outputs:

.venv-rdkit2026/bin/python -m pytest -q tests.venv-rdkit2026/bin/python -m pytest -q reports/test_phase0_audit_regressions.py.venv-rdkit2026/bin/python reports/g1-audit-20260919-1005/recompute_evidence.py.venv-rdkit2026/bin/python reports/g1-audit-20260919-1005/adversarial_probes.pyPYTHONPATH="$PWD" .venv-rdkit2026/bin/python reports/g1-audit-20260919-1005/probe-output-wrapper.py

📑 Contents