V6 hidden-data CSV failure review
Verdict
The strongest reproduced defect in the preserved v6 generation logic is an empty candidate list becoming a blank smiles field. Padding a short nonempty list does not repair an empty list. This can happen with a supported adduct; an unknown adduct is not required.
This review establishes a failure mechanism, not the exact trigger inside Kaggle's rejected hidden rerun. Its generated CSV and execution log remain unavailable. Kaggle's total_bytes=0 field is rejection metadata, not proof that the notebook created a zero-byte CSV.
Evidence and version limits
d077484b727d1db2582e1ec2c2ec43b061df894744a17ff4ab3cb7edb33f9ad8, matching its runtime report. That report records 400 molecules, all 400 routed through spectral retrieval, zero mass-only routes, and 53 padded rows.main.py is v7, not an independently authenticated historical v6 source snapshot. Its source SHA-256 is 0f4d79adbf29b7bd54b1ccbf9a14619ceb766daff5e62ed647b98a49f8ecb6e0./tmp/base02-kernel-v4/ is labelled as such, not relabelled v2. The earlier /2 output retrieval returned a later artifact and is excluded as v2 proof.See [provenance.json](provenance.json) and [inputs/](inputs/).
1. Reproduced blank-cell path
Source anchors below refer to the frozen v7 snapshot with shared v6 candidate logic, not historical v6 line numbers: [inputs/base02-v7-current.py](inputs/base02-v7-current.py).
1. mass_only_ranked_ids, lines 50–74, searches only ±10 ppm. If no spectrum produces a hit, it returns [] (lines 60–67). The manifest says the 274,195-structure pool is training-derived; it is not an exhaustive database of hidden molecules.
2. The gate at lines 243–252 uses only mass candidates when no library candidate reaches cosine 0.15. Thus weak available spectral candidates can be discarded even when the mass window is empty.
3. Chemical parsing/canonicalization can remove every remaining candidate, leaving hybrid_smiles=[] (lines 255–264).
4. Padding only happens for a nonempty list:
if hybrid_smiles: hybrid_smiles.extend([hybrid_smiles[0]] (25 - len(hybrid_smiles)))rows.append({'molecule_id': mid, 'smiles': ';'.join(hybrid_smiles[:25])})For [], the serialized value is '': a row such as synthetic_query,. A normal pandas read treats that field as NaN, and the stored official scorer raises ParticipantVisibleError: NaN in required columns.
The recorded v6 sequence writes the file before its final assertions. The local reproduction leaves the blank CSV on disk and then fails the post-write assertion. How Kaggle maps that notebook exception to its generic error remains unverified. V7's additional pre-write checks abort earlier; they do not supply a missing candidate or solve the coverage problem.
Executed local fixtures
Six tiny source-extracted candidate fixtures, three ID fixtures and four scorer-contract fixtures completed once, using RDKit 2026.03.3. The test worker timed out without artifacts; the parent completed these remaining local fixtures directly. The earlier successful notebook review was preserved, not rerun.
| Synthetic condition | Candidate result | Re-read CSV/scorer result |
|---|---|---|
Supported [M+H]+, no spectral or mass-window hits | Empty | Blank field; scorer rejects NaN |
| Supported adduct, mass hit whose SMILES is invalid | Empty after filtering | Blank field; scorer rejects NaN |
| Weak spectral hit, cosine approximately 0.1, empty mass window | Available spectral candidate discarded | Blank field; scorer rejects NaN |
| One valid candidate | Padded to 25 | Accepted by stored scorer |
| Two tautomer-equivalent mass candidates | Deduplicated, then padded | Accepted by stored scorer |
| Synthetic unsupported adduct | Empty | Blank field; not evidence hidden data uses this label |
Toy index/peak data demonstrate program behavior only, not hidden-data prevalence or prediction accuracy. Full reproducible artifacts: [candidate-probes.py](candidate-probes.py), [candidate-probes.json](candidate-probes.json), [candidate-probes.md](candidate-probes.md).
The preserved earlier v4-path source has unconditional hybrid_smiles.extend(['CCO'] max(0, 25 - len(hybrid_smiles))). Executing that exact padding statement on an empty list produces 25 valid placeholder entries. It closes this specific empty-list gap, though it does not improve identification accuracy.
2. Shared notebooks: the important difference is fallback, not CSV punctuation
Static cached-source inspection only; all four public snapshots have null execution counts and no saved outputs. These are implementation comparisons, not verified leaderboard or accepted-submission claims. Citations use zero-based cells and one-based source lines. Detailed hashes, excerpts and limitations are in [public-notebooks.md](public-notebooks.md).
| Public notebook | Handling an empty/weak candidate pool | Submission writer |
|---|---|---|
| [prvsiyan — analog propagation](https://www.kaggle.com/code/prvsiyan/analog-propagation-casmi-2026-baseline) | Retries 10→30 ppm; if still empty, smis=['CCO'] | C027:L034–037,071–084: semicolon join, maximum 25, fills missing values and writes index=False. Fallback is one guess, not 25. |
| [haideptry — fast spectral cosine](https://www.kaggle.com/code/haideptry/enveda-casmi-2026-fast-spectral-cosine-baseline) | Strong hits → COCONUT → weak hits → global library fallback | C016:L021–043; C020:L003–012: validates before to_csv(index=False). Its exact-25 and sample-order checks are stronger local policies. |
| [octaviograu — evidence ranking](https://www.kaggle.com/code/octaviograu/enveda-molecule-evidence-ranking) | Nearest-mass fallback if local window is empty | C010:L069–075; C012:L069–085,102–103: validates runtime test ID set, 1–25 guesses and chemical validity before writing. No-repeat rule is stricter than official scorer. |
| [dmitriigluzdov — EDA/search](https://www.kaggle.com/code/dmitriigluzdov/casmi-26-eda-search-model) | Nearest-mass training-structure search beyond initial ppm window; filter/dedup candidates | C082:L016–054: builds IDs from current test, rejects blank lists before to_csv(index=False), permits up to 25. |
These examples also have weaknesses: prvsiyan's final left join to sample IDs can drop runtime-only IDs, fillna does not repair existing empty strings, haideptry can fail if its fallback has too few structures, and nearest-mass methods do not guarantee a result if their global candidate table is empty. Do not copy any notebook wholesale as a format guarantee.
The separately inspected private Bob comparator is not another public notebook. Its static embedded path can reach np.stack([]) before final CCO padding; that padding is not an end-to-end no-hit guarantee.
3. Runtime IDs versus sample IDs
V6's preserved ID selector falls back to runtime query IDs when the sample and test sets differ, then its post-write assertions still require equality with the sample.
The local ID fixtures showed:
The mismatch is an inconsistent validation assumption, not proof runtime IDs are wrong. Whether Kaggle leaves its sample unchanged in this competition is unverified. Sample-ID left joins in some public examples could silently mask the same mismatch rather than solve it. V7's new unconditional sample/test set guard similarly aborts on this scenario; it should not be described as a proven hidden-runtime fix.
4. Corrections to earlier diagnoses
CCO for an empty pool. A sample containing 25 placeholders alone does not establish mandatory list width; local exact-25 checks are policy.src/submission_validation.py accepted a blank field and rejected repeated valid guesses. The newer sample-matching validator rejected one valid guess solely for width. Neither is a faithful end-to-end hidden-runtime acceptance oracle.Recommended repair order — not applied in this review
1. Use the actual runtime test's unique, nonblank molecule IDs as the coverage contract; use sample order only when sets match. Do not hardcode 400 or silently drop runtime IDs in a sample merge.
2. Keep the best available valid candidates when confidence is low; do not replace a usable weak spectral list with an empty mass list.
3. Implement an explicit no-candidate policy: validated nearest/wider-pool candidates where appropriate, and a clearly logged valid last-resort placeholder if needed. A placeholder prevents a format failure, not a low score; report fallback rows separately from genuine retrieval coverage.
4. Validate schema, all runtime IDs, nonempty 1–25 lists, and token validity before publishing the final CSV. Use a temporary file, re-read/validate it, then atomic rename. Unify validator behavior with the pinned official scorer rather than sample placeholder counts.
5. Add these no-hit, weak-route, invalid-candidate and replacement-ID fixtures as regressions. Preserve immutable source hashes and launch-time metadata for the next version.
Bottom line: the empty-list route is now reproduced with real source blocks and a real invalid CSV. It is a stronger actionable finding than guessing about delimiters, duplicate padding, or new adducts. The exact Kaggle hidden rejection remains unconfirmed; no push or resubmission was made.
5. Follow-up implementation status (2026-09-25)
The recommendations above were not applied during the original read-only review. They have since been implemented in the local, unpushed workspace source kaggle-kernel/base02-cpu/main.py (revision base02-v8-zero-candidate-runtime-id-atomic-20260925), derived from the retrieved V7 source. The immutable V7 snapshot analyzed above remains at inputs/base02-v7-current.py, SHA-256 0f4d79adbf29b7bd54b1ccbf9a14619ceb766daff5e62ed647b98a49f8ecb6e0; the current modified workspace source is a later file and is not that snapshot.
| Original review item | Current local status | Evidence |
|---|---|---|
| Use runtime IDs as coverage contract; sample order only when ID sets match | Implemented. Reject invalid/duplicate runtime IDs; use sample order only on an exact ID-set match, otherwise use sorted runtime IDs. Validate exact runtime ID coverage. | main.py: select_runtime_ids, lines 107–117; runtime extraction and selection lines 353–374; writer validation lines 188–216. Regression tests cover mismatched IDs/counts and invalid IDs. |
| Preserve weak valid spectral candidates and make no-hit behavior explicit | Implemented. Weak spectral guesses backfill mass candidates; if primary candidates are empty, try bounded nearest-mass candidates, then a valid CCO last resort explicitly logged and reported as format-only, excluded from genuine coverage. | main.py: prepare_predictions, lines 120–185; fallback diagnostics/logging lines 382–437. Tests cover weak spectral, invalid/no-hit, nearest-mass and placeholder routes. |
| Validate schema, IDs and chemistry before publishing; use temporary file and atomic replacement | Implemented. Writer validates exactly 25 nonempty parseable guesses per runtime ID, writes/fsyncs a same-directory temporary file, rereads/revalidates it, and atomically replaces the final CSV only after success. | main.py: validate_output_rows and atomic_write_submission, lines 188–243. Tests cover invalid input preservation, reread failure cleanup, and roundtrip. |
| Require exactly 25 guesses, per CC's instruction | Implemented as local policy. Candidate lists are padded to 25; repetitions remain allowed. The archived scorer evidence still permits 1–25, so this is an output requirement, not a claim about scorer necessity. | main.py: prepare_predictions lines 181–185 and validation lines 203–214; shared validators in src/submission_validation.py and src/submission_format.py. Tests reject non-25 counts. |
| Add no-hit, weak-route, invalid-candidate and replacement-ID regressions | Implemented and passing. | Full repository tests: 134 passed; Python compilation passed on the kernel and validator modules on 2026-09-25. |
Not yet done / remaining limitation: no source has been pushed to Kaggle, so published V7 itself is unchanged. No Kaggle rerun or hidden-data execution has verified the new code. The empty-list format failure path is addressed in local source, but fallback placeholders protect formatting rather than identification quality; hidden-run cause and score impact remain unverified. See also ../v6-hidden-output-fix-20260925/RESULTS.md for artifact hashes and verification details.