CalCOFI.io CalCOFI.io Storage

ducklake/releases/v2026.10.21 / RELEASE_NOTES.md · raw markdown ↗

CalCOFI integrated database release v2026.10.21

RELEASE_NOTES.md on gs://calcofi-db/ducklake/releases/v2026.10.21

Release date: 2026-10-21 · promoted (latest.txt)

The October 2022 eDNA survey moves to its corrected v1.3, and every CTD source archive is kept

This release fixes v2026.10.20's two metadata defects, takes the provider's corrected eDNA archive and closes the gap that let calcofi.org's 1995-98 CTD files disappear with no copy kept. The sections below give each change; the bottle, ichthyoplankton, zooplankton and other datasets carry the same rows as v2026.10.20. Cite this release, not v2026.10.20, for the measurement categories and faces.

eDNA: archive v1.3, the ASVs assigned to taxa, and the provider's answers

calcofi_2022-edna now reads dwca-calcofi_october2022_edna-v1.3.zip (IPT, published 2026-10-19), in which Garret O'Donnell corrected the "Total number of OTUs or ASVs assigned to taxa": through v1.2 it was a summed read count, so it was held back (question Q04). It is published as two new types, edna_asvs_dloop and edna_asvs_12s: the distinct sequences over a filter's libraries of each assay, not the sum of the per-library values, because PCR replicates share sequences. A filter with one library of an assay carries exactly the provider's number, which the ingest asserts. The occurrence and DNA tables are unchanged from v1.2. The citation names Version 1.3; the dataset links the bioinformatics scripts (github.com/CalCOFI/eDNA_patch) beside the paper. Questions Q04, Q06, Q07, Q08 and Q10 are answered: the archive carries detections only, because edna2obis drops blanks and filters with no detection, and it stays that way, as the provider publishes it.

Every CTD archive the ingest reads is kept on GCS, and a failed download stops the render

calcofi.org listed FinalQC zips for nine cruises that returned 404, and the CTD ingest only warned; v2026.10.20 recovered eight of them from our archived FinalDB copies. Two gaps let that happen. On the Mac mini, where releases are cut, the downloads folder is not synced back to GCS, so a zip fetched from calcofi.org existed on one machine. And a failed download was a warning(), which a rendered notebook hides. Now calcofi4db::archive_sources() (4.29.0) keeps each archive the ingest reads, once by content, at gs://calcofi-files-public/archive/sources/calcofi/ctd-cast/{md5}/{file}; a corrected file re-published under the same name lands beside the old one. metadata.json sources[] names each archive's md5, calcofi.org url and archive copy, so the sidecars record which bytes a release read. Every failed download is listed in the render, and one that leaves its cruise with no final or preliminary archive stops it unless a provider question names it.

The rebuild was diffed against v2026.10.20 before release (diff_stage_vs_release()): the headline obs rows are identical, and obs_ctd_full differs by five values, the station-corrected nitrate and four oxygen estimates of a single upcast scan (cast 0604_014u, cruise 2006-04-32NM, 403 m), which the rebuild no longer emits; no value or quality flag changed anywhere else. Its obs_id sequence shifts by those five rows, which is why 22 cruise partitions have new signatures with the same content. Why that one scan dropped is not yet known. The ingest recorded 295 source archives; two of them, the raw-cast zips of 2208BH and 1203NH, are 196-byte error pages saved in October 2025 in place of the archive (the ingest does not read raw-cast zips, so no data are affected).

farallon_bird-mammal was recorded under custom terms, pointing at the Farallon Institute's Data Sharing Agreement (23 July 2025); the agreement releases the data under CC BY 4.0, so the licence is now CC-BY-4.0, and the dataset carries the agreement's requested attribution statement as its acknowledgement.

sio_pic-zooplankton names the Pelagic Invertebrate Collection's page as its data source, as promised with the front-door review. Its question Q07 (preservative and net side) is answered by Linsey Sala's Parent Net Event table and the net crosswalk (workflows#198). New questions: the MOCNESS mouth area for gear M2 (Ed Weber, swfsc_ichthyo), cruise 1501NH's dual processing and reused cast numbers (CTD team, calcofi_ctd-cast, workflows#134), and bottle cast 24415 of cruise 1986-03-32NM, whose position falls on land near Paso Robles (calcofi_bottle, a likely 1° longitude typo; the position is published as given). Four more came out of measuring the bottle provider's zero rule and the TEOS-10 sigma-theta: est_nitrate_cruise_corr is 0 on every row of cruises 2301, 2304, 2307 and 2411, which looks like a placeholder and gives anomalies of -4 to -12 µmol/L (calcofi_ctd-cast Q48); the July baseline of est_chlorophyll_a_cruise_corr is dominated by values above 20 µg/L from 2000-10 and 2001-07 (Q49); bottle nitrate's code 4 on 277 zeros of 2015-01 (calcofi_bottle Q16); and two surface salinities on 2020-07-33P4 (Q17). All four series are published as given until the provider answers.

The sixteen METS keys get their category back

v2026.10.20 shipped 16 calcofi_mets keys with a NULL category (its known issue). The ingest rebuilds the measurement_type rows it owns and carried only the bounds and derivation across the rebuild; it now carries every registry column, so a blank is filled from the registry and a value the notebook sets still wins, and it stops if an owned row comes out with a blank where it went in with a value. metadata/measurement_type.csv was never committed blank, so the next re-ingest writes it unchanged. sst_c, sss_psu, tsg1_temp_c, tsg1_salinity_psu, oxygen, uws_flow and bottom_depth_m return to Physical Oceanography, par_surf to Productivity & Pigments and sw_ph to Carbonate System.

TEOS-10 sigma-theta has a face of its own, and the face registries are checked at release

sigma_theta_teos10 was registered as a stand-in for sigma_theta's face while carrying its own NERC P01 (SIGTPR03). A stand-in borrows the lender's chain, so calcofi.io's face check refused it at the v2026.10.20 deploy, and the key has no face on calcofi.io until the next release. It now takes scale, like sigma_theta. release_database.qmd runs validate_measurement_faces() with the measurement type registry before it builds measurements.json and stops on any finding, including the new standsin_own_p01 rule (calcofi4db 4.28.1).

Contents (generated)

table rows
climatology 1,878,006 partitioned
cruise 869
dataset 20
dataset_taxon 1,940
fish 10,796
grid 225
grid_crosswalk 673
ichthyo_net 77,888
lookup 26
measurement_type 234
obs 35,358,143 deprecated → obs_bio, obs_env (objects removed in next)
obs_attribute 455,300
obs_bio 1,586,795
obs_env 33,771,348 partitioned
region 4
sample 1,467,375
sample_measurement 659,307
sample_spatial 1,996,331
ship 49
spatial 3,390
spatial_attribute 37,943
taxon 2,606
taxon_group 438
obs_ctd_full 286,148,056 supplemental
obs_mets_full 19,926,523 supplemental
sample_root 419,418 supplemental

26 tables, 383,803,703 rows, 2.85 GB.

Datasets (20): calcofi_2022-edna, calcofi_bottle, calcofi_bottle-derived, calcofi_ctd-cast, calcofi_ctd-derived, calcofi_dic, calcofi_mets, calcofi_phyllosoma, calcofi_phytoplankton, cce-lter_euphausiids, cce-lter_iron, cce-lter_picoplankton-bacteria, cce-lter_zoodb, cce-lter_zooscan, cdfw_dungeness-crab, farallon_bird-mammal, sio_mesopelagic-fish, sio_pic-zooplankton, swfsc_cufes, swfsc_ichthyo

Validation: 97 pass / 0 fail / 4 skip (consumer-contract suite, 2026-10-21T10:39:31Z).

Software: calcofi4db 4.29.0, calcofi4r 1.28.1.

How to cite

CalCOFI (2026). CalCOFI Integrated Database, release v2026.10.21 [Data set]. Scripps Institution of Oceanography, NOAA Fisheries, and California Department of Fish and Wildlife. https://doi.org/10.5281/zenodo.23566406

Cite the source datasets you use alongside the release:

Access

con <- calcofi4r::cc_get_db(version = "v2026.10.21")
con = calcofi4py.cc_get_db("v2026.10.21")

Parquet: https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.10.21/parquet/{table}.parquet; full history: RELEASES.md.