← Back
📝 Notes / reports / g1-audit-20260919-1005 / runtime / AUDIT

📑 Contents

G1 runtime/submission/scorer contract audit — 2026-09-19 10:05 MYTVerdictEvidence and probesFindings (prioritized)R1 — Runtime dependency contract is not repository-enforced (P1, reproducibility/readiness)R2 — Validator is fixed but operationally hard-coded (P1, portability)R3 — Contract API permits custom limits while reporting fixed `mrr_at_25` (P2, API clarity)Positive current-source checksRemaining scope caveats (not newly reproduced)

G1 runtime/submission/scorer contract audit — 2026-09-19 10:05 MYT


Verdict


PASS WITH CAVEATS at the requested local contract scope; no new P0 runtime failure reproduced in the current maintained source. Current source contains the previously reported fixes for wrapper stage propagation, direct validator importability, metadata separation, raw-field validation, and null/blank runtime IDs. The maintained acceptance evidence is consistent with these fixes (400/400 rows, <=25 raw candidates, zero duplicate scorer identities, offline reordered-ID smoke pass).


This audit was read-only against production/data/artifacts. Only this report and probes.json were written. No full evaluation, index rebuild, wrapper launch, external network, submission, or process intervention was performed.


Evidence and probes


  • Current source reviewed: src/scorer_contract.py, src/identity.py, src/adducts.py, src/phase1_baseline.py, src/g1_molecule_eval_all.py, scripts/validate-runtime-submission.py, scripts/run-g1-20260918.sh, tests/test_g1_contract.py, tests/test_scorer_contract.py.
  • Acceptance evidence: reports/g1-acceptance-20260919/acceptance-evidence.json, G1-ACCEPTANCE-REPORT.md.
  • Prior findings cross-checked against current source: reports/audit-20260918-1923/AUDIT.md and probe JSONs. The old report's A1–A3 defects are not present in current files.
  • Current generated submission metadata remains separate: submissions/phase1-retrieval-rebuilt-20260918.json and .validation.json are both present; validation evidence reports 400 expected/400 actual and no errors.
  • probes.json records current Python/system-RDKit and adduct failure probes. The system interpreter has RDKit 2022.09.5 built against NumPy 1.x while installed NumPy is 2.2.6 (_ARRAY_API warnings/errors on import). This does not invalidate the pinned .venv-rdkit2026 acceptance run, but demonstrates that the repository does not itself pin/enforce the runtime environment: no requirements/lock/environment manifest was found at repository top level (only research/code-requirements.md, which is competition prose).

    • Findings (prioritized)


    R1 — Runtime dependency contract is not repository-enforced (P1, reproducibility/readiness)


    Evidence: src/identity.py:11-12 imports RDKit at module import; no top-level requirements.txt, pyproject.toml, lockfile, or environment manifest was found. Running the ordinary system python probe produced repeated NumPy ABI diagnostics and RDKit _ARRAY_API not found errors before the adduct/scorer probe completed. Acceptance uses .venv-rdkit2026/bin/python, but the wrapper (scripts/run-g1-20260918.sh:8-12) assumes that directory exists and does not verify interpreter/RDKit/NumPy versions or record them in its output metadata.


    Impact: a clean machine or differently provisioned offline notebook can fail before submission validation, and identity values can vary with RDKit version. This is especially material because IDENTITY_ALGORITHM names transform limits but not RDKit release/InChI implementation.


    Repro:

    cd /home/ubuntu/.openclaw/workspace/enveda-casmi26python - <<'PY'import rdkit, numpyprint(rdkit.__version__, numpy.__version__)from src.identity import identity14print(identity14('CCO'))PY

    Observed system environment: Python 3.10.12, RDKit 2022.09.5, NumPy 2.2.6, _ARRAY_API ABI errors. Do not treat this as a failure of .venv-rdkit2026; run maintained tests in that pinned environment as parent is doing.


    Action: add a committed offline dependency/environment manifest and a preflight that checks executable, RDKit, NumPy, and InChI availability; include versions/hashes in submission metadata. Do not silently fall back to another identity implementation.


    R2 — Validator is fixed but operationally hard-coded (P1, portability)


    Evidence: scripts/validate-runtime-submission.py:13-14,16 hard-codes submissions/phase1-retrieval-rebuilt-20260918.csv, its .validation.json, and data/test.parquet. The inference CLI accepts --root/--output, but the validator has no corresponding arguments and cannot validate an arbitrary output produced by a renamed clean-environment run.


    Impact: the acceptance smoke with renamed/reversed IDs validates the inference path only; a genuinely dynamic offline package or future run name requires editing/copying the script, weakening reproducibility and creating risk of validating a stale fixed artifact.


    Repro (static, no production execution):

    nl -ba scripts/validate-runtime-submission.py | sed -n '1,35p'

    Action: add explicit --root, --submission, --test, and --validation-output arguments with current paths as defaults; reject output metadata if it does not identify the same source/test artifact.


    R3 — Contract API permits custom limits while reporting fixed mrr_at_25 (P2, API clarity)


    Evidence: src/scorer_contract.py:44-64: score_rows(..., limit=...) passes the caller's limit into reciprocal_rank, but always emits keys mrr_at_25, recall_at_25, and status. In probes.json, a valid target at rank 2 with limit=1 correctly returns no hit, but the metric label still says “at_25”. The production call uses the default 25, so this is not a current submission defect.


    Action: either remove public custom-limit semantics or emit mrr_at_{limit}/recall_at_{limit} (and preserve aliases only when limit == 25). Add a test to prevent misleading non-default reports.


    Positive current-source checks


  • validate_submission now checks exact columns, row.get(None) extra fields, missing IDs/SMILES, empty interior tokens, raw token count >25, invalid SMILES, duplicate scorer identities, missing and unexpected IDs (scorer_contract.py:76-130). Empty entire smiles is explicitly accepted as no-candidate.
  • make_submission groups all test spectra by runtime molecule_id, rejects null/blank IDs before string conversion (phase1_baseline.py:386-403), preserves test/sample order when sets match (404-410), uses deterministic sorted(..., key=lambda ident: (-score, ident)) (435), and emits at most top_k (435-445).
  • Invalid/nonfinite precursor/adduct input is fail-safe in inference: neutral_mass raises for null, NaN, and unsupported adduct; make_submission catches TypeError, ValueError and yields a blank candidate row rather than crashing (phase1_baseline.py:413-423). Direct adduct probes recorded in probes.json confirmed ValueError for null adduct, null/NaN m/z, and unknown adduct.
  • Identity deduplication and tie order are scorer-identity based and lexicographically deterministic. Current acceptance evidence independently reports no duplicate identity rows and max raw candidates 25.
  • scripts/run-g1-20260918.sh currently uses set -Eeuo pipefail plus ERR trap, and stages are sequential with separate completion markers; this addresses the prior false-success wrapper behavior.

    • Remaining scope caveats (not newly reproduced)


    Official CASMI scorer parity remains unavailable; acceptance is local contract evidence. No official hidden-test generalization, acquisition-lineage certification, Regime-C result, full content-hash run manifest, or resumable per-molecule evidence is claimed. These are correctly listed as unresolved by the current acceptance report and remain reasons not to call this an official G1 pass.