Enveda evidence and TODO review — 2026-09-19

Decision

Useful local diagnostics exist; full G1 and Phase 3A promotion remain OPEN.

This review reconciles the roadmap, current source, prior G1 audit, Phase 3A reports,

shadow database and maintained tests. It does not re-execute the nine-hour spectral

benchmark or every historical experiment. No production database, model or inference

source was modified; no external submission or training was performed.

Fresh verification

integrity ok; identity sets match. Full row comparison includes source name,

source record, mass, formula, scaffold, overlap, novelty and source digest.

Production has seven unparseable representatives; shadow has zero. Each repaired

representative freshly reproduces its stored scorer identity and exact mass.

checks are in [evidence.json](evidence.json); reproduction: pinned Python +

[verify.py](verify.py). These are audit-time hashes, not launch-time provenance.

reports/test_phase0_audit_regressions.py: 23 passed in 51.77s.

tests/test_scorer_contract.py tests/test_similarity.py: 13 passed in 0.51s.

This covers every current maintained test file plus the two report regressions:

36 passed, not just the previously reported 23-test subset. Archived report

copies were excluded. Passing tests do not establish missing acceptance coverage.

P0: scaffold classification was wrong

scripts/classify_phase3a_pubchem_scaffolds.py computes same_overlap from

ident in train_ids versus the stored identity training-overlap flag, then labels

its category as scaffold overlap. The saved per-row scaffold membership fields

show the actual result:

The net change is -849, matching 163,344 → 162,495 in the earlier label audit.

The seven unparseable production structures were excluded from this classification.

Zero identity-overlap mismatches does not establish zero scaffold-overlap changes.

Nor is any changed string automatically a chemically equivalent serialization.

Keep novelty/scaffold filtering quarantined; fix the classifier and independently

recompute from source/representatives before closing normalization.

Follow-up completed: the classifier was corrected to use the stored and

recomputed scaffold membership booleans. The rerun reproduced 7,853 string-only

differences and 1,741 membership changes, with zero identity-overlap changes;

tests/test_phase3a_regressions.py now asserts these distinctions (7 passed).

P0: analogue metrics and validation claims need narrowing

Source inspection of scripts/diagnose_phase3a_analogue_transfer.py establishes:

1. mrr and fallback_mrr append 1/rank at all ranks. Thus the saved combined

0.469755 is untruncated MRR, not MRR@25. Exact corrected MRR@25 cannot be

recovered from aggregate summaries; preserve per-query ranks on a corrected run.

2. The registered set contains 739 identities, but the runner selects the first

100 rows/fold, yielding 462 valid spectra, not 462 distinct molecules. It uses

raw Parquet inchikey14 against scorer-derived split IDs without authoritative

cache reconciliation. Audit identity alignment and unique-molecule coverage.

3. The manifest excludes only the prior modulo-5 cohort. 164 selected identities

are in fold 4, already inspected in phase3a-fold4-policy-readout.json (1,221

identities). “Registered before this run” is supportable; “genuinely untouched”

is not established. Earlier broad diagnostics add further exposure to audit.

4. target_present_no_eligible_reference increments even when target is absent.

The 92 P0 cases are no-positive-reference routes, only 71 with target recovered

by fallback; they cannot all be called target-present cases. target_rankable

means route availability, not guaranteed target rankability.

5. target_outside_1000 increments when target is absent from a score map, not

when its rank exceeds 1,000. eligible_reference_no_positive_candidate_score

tests an empty dict, not all-zero scores. Adduct/mass-bin pool counts omit

fallback target hits because of an early continue.

6. P2's reference_fallback field is declared but execution uses fallback and

checks an empty raw mass window, not absence of eligible/positive neighbours.

7. Current regression tests mostly inspect manifest arithmetic, text/AST names,

a standalone sort and saved repair booleans. They do not execute the actual

fallback/category/repair behavior end-to-end.

P0's saved Recall@25 88.10% is retained as a bounded query-level diagnostic, not

an independent corrected ranking result or competition score. No new ranker or

policy promotion follows from these aggregates.

Corrected run completed: scripts/diagnose_phase3a_analogue_transfer_corrected.py

produced a separate report with 500 unique sampled identities / 480 valid queries,

an enforced top-1000 cap and true MRR@25. P0 combined MRR@25 over all queries is

0.2883 and Recall@25 is 0.8188; P2 is 0.2903 / 0.8229. These are not promotion

metrics: the partition has prior fold exposure, and raw-to-scorer identity and

per-query provenance audits remain open.

Identity/exposure follow-up completed: the cache-backed audit confirms all

739 registered identities are present and 100/100 direct identity14 samples

match the cache. However, the full training file contains 1,559 rows across

44 stored→derived mappings where Parquet inchikey14 differs from the cache

identity for a registered-derived identity; 33,283 rows agree and 1,559 are

derived-only under the registered set. The registered set also intersects the

prior fold-4 diagnostic in 164 identities, while its modulo-5 exclusion holds.

Therefore the corrected analogue metrics cannot be treated as authoritative or

untouched until query extraction is switched to the reconciled scorer namespace

and prior exposure is handled explicitly. Evidence: reports/phase3a-partition-identity-exposure-audit-20260919.json and scripts/quantify_phase3a_identity_mismatches.py.

Namespace correction completed: the corrected runner now derives query IDs

from identity-cache.sqlite keyed by normalized_smiles; raw Parquet

inchikey14 is not used for selection. The cache-aligned report is

reports/phase3a-analogue-transfer-preregistered-corrected-cache-aligned-20260919.json.

P0/P2 combined MRR@25 are 0.3003/0.3024 and Recall@25 are 0.8167/0.8208.

These remain diagnostic because 164 identities overlap prior fold 4 and

per-query provenance is not yet persisted.

Per-query provenance completed: the cache-aligned report now persists 1,440

policy/query records for 500 sampled identities and 480 valid queries. Each

record includes source row, cache-derived identity, adduct, precursor m/z,

candidate identity list, ranked identity list, reference identity list, route,

fallback state, target-in-pool state, and target rank. The maintained Phase 3A

regression suite passes 10 tests. This closes the evidence-persistence portion

of S3/S4, but not the prior-exposure or production/shadow parity gates.

Shadow repair: real but not fully accepted

SHADOW_REPAIR_PATCH_COMPLETE (copy plus seven repairs). The adjacent progress

file still says RUNNING, 5,000 rows, PID 327512: stale evidence from the abandoned

build, not current activity. No matching build/evaluation process was found.

does not contain; its default progress path refers to production. The earlier

/dev/null JSON failure does not imply a correctly invoked verifier would accept

the shadow. Define a distinct, fail-closed repair acceptance contract.

index. That supports pool equivalence for identical query/tie/cap policies.

It does not prove fingerprint/analogue ranking parity: seven formerly unavailable

fingerprints now become available. Analogue runner also hardcodes production.

no completed shadow recall or analogue comparison was found. Do not claim those

runs succeeded. Production recall covers 5,820/5,974 identities; 154 omitted

targets require separate accounting, and formula coverage uses oracle metadata.

representative. Builder silently unlinks an existing output and lacks stored

representative round-trip validation. A repeatable guarded patch/build path,

provenance hashes, and acceptance tests remain needed.

Identity mismatch against source-declared labels must not automatically reject

a valid recomputed scorer identity; namespace reconciliation is intentional.

local source hash is not a replacement for those fields or catalogue completeness.

G1 and whole-roadmap reconciliation

The prior independent [G1 audit](../g1-audit-20260919-1005/AUDIT.md) already verified

saved target-rank arithmetic, current index/cache alignment and the local 400-row

submission (9,106 unique submitted SMILES). These artifacts are not missing. It

records evaluation 9h04m20s / ~4.90 GiB peak RSS and inference 1m49.03s / ~1.44 GiB.

This review did not rerun that expensive result audit; its evidence remains qualified.

Still open: row-identity permutation accepted by the provenance guard, null/string

precursor coercion outside protected handling, inconsistent legacy metrics, immutable

launch inputs, full candidate-level records/strata, dynamic offline packaging and C.

Current source still shows the precursor/legacy-metric paths. Official scorer parity

and acquisition-lineage certification remain unavailable.

Phase 0/1 local infrastructure and diagnostics are substantial but not full gate

closure. Phase 3A/3B is partly built/tested, not a licensed/promoted universe. COCONUT,

formula inference, complete non-neural comparisons and true-novelty evaluation remain

open. Phases 2/4/5/6 training/generation/ensemble/final packaging remain deferred or

pending; no Top-5 or official-score claim is supported. Project is still untracked in

the parent repository; current filesystem hashes cannot prove historical execution.

Next execution order

1. Fix classifier semantics and actual executable regressions; retain raw evidence.

2. Correct analogue identity, sampling, MRR@25, counters and per-query provenance;

audit previously exposed identities before defining a clean evaluation policy.

3. Create reproducible guarded shadow repair/verification with truthful manifests;

run fingerprint/analogue comparison on explicit old/new paths; complete provenance.

4. Close reproduced G1 runtime/provenance gaps and packaging checks.

5. Only then reconsider baseline promotion and a bounded learned ranker.

TODO.md now distinguishes completed evidence tasks, in-progress composite work

and still-blocked promotion. This review updates the backlog, not production code.

Guarded repair path completed: manifests/phase3a-shadow-repair-20260919.json

binds the exact source hash, base artifact, seven expected old rows and seven

replacement SMILES. scripts/build_phase3a_shadow_repair_guarded.py refuses a

source-hash mismatch, an existing output, an unexpected base row count, or any

old-row mismatch, and removes a partial output on failure. A temporary guarded

build produced 231,749 rows and matched the existing repaired shadow at the

full semantic row-field level; raw SQLite file hashes are not used as equality

evidence because SQLite page layout is not a semantic hash.

Fingerprint availability parity completed: on the exact 1,440-record

cache-aligned evidence set, production and shadow both provide fingerprints for

11,106/11,106 candidate identities. For references, both provide 420/2,095;

the remaining 1,675 reference identities are unavailable in both artifacts.

There are zero candidate- or reference-availability changes. This proves

artifact parity for the evaluated evidence, not complete fingerprint coverage,

source licensing, or a clean untouched evaluation partition.

Provenance audit completed: reports/phase3a-pubchem-provenance-audit-20260919.json

confirms the local source is reproducible (plain TSV, 240,536 rows, 231,755

declared identities, SHA-256 6006275e…91152) but fails the provenance gate.

The manifest lacks a source URL, upstream version, download date, license, and

derivation commit. Therefore the bounded source/index remains diagnostic and

must not be promoted as complete PubChem coverage or used for novelty claims.

Production/shadow same-query parity completed: the exact cache-aligned

evidence set was rerun against pubchem-structure-index-repaired.sqlite.

All 1,440 policy/query records matched production for candidate IDs, ranked

IDs, reference IDs, route, rank, candidate/reference counts, and aggregate

P0/P2 metrics. The artifact parity audit is also PASS: 231,749 rows, identical

identity/mass/label fields, seven repaired representative SMILES, and zero

invalid shadow representatives. This closes the parity comparison portion of

S3; guarded rebuild/provenance/licensing and prior-exposure limitations remain.

Final checklist verification: 107 bullet tasks = 32 complete / 17 in progress /

58 pending (29.9% complete by count). Four repair tasks were added. Final SHA-256

rechecks of production and shadow match the hashes in evidence.json; neither

database changed during this review.