CalCOFI.io CalCOFI.io Storage

ducklake/releases/v2026.11.12 / RELEASE_NOTES.md · raw markdown ↗

CalCOFI integrated database release v2026.11.12

RELEASE_NOTES.md on gs://calcofi-db/ducklake/releases/v2026.11.12

Release date: 2026-11-12 · promoted (latest.txt)

The 1 m CTD table is whole again: 22 cruises had been published with most of their casts missing

obs_ctd_full, the supplemental table of 1 m CTD bins, was published short in v2026.11.05 and v2026.11.09: for 22 cruises between 2015-04 and 2026-04 it carried 2 to 68 casts where there are 92 to 232 (2015-11-32OC: 2 of 140), so 47,189,291 of its 287,281,956 rows were missing. The release had built the table whole; the loss happened on the way to the bucket. DuckDB 2.0.0 starts another file for a partition once more than 100 are open, so 24 of the table's 143 cruises were written as two or three files. Each was given the cruise's one path, the uploads overwrote each other and the last, smallest file stayed. The catalog listed each of those 24 paths once per file (169 objects for 143 cruises), so the two whole cruises among them, 1994-03-31JD and 2005-11-32NM, were read twice. v2026.11.04 has the same repeated listings, but its objects are whole: every row is there, and those 24 cruises' rows are read two or three times. v2026.10.30 and earlier are right. No other table was affected (obs_env has 96 partitions, below the limit), and nothing upstream was lost: the staged ingest output matches the release export row for row.

This release publishes one object per cruise again (calcofi4db 4.38.0: the export keeps every partition open for the write and stops unless each holds exactly one file; release_objects() and freeze_plan() stop on a second file or a repeated path). It replaces the 22 short objects in the shared store, where their content, and so their path, is unchanged: the upload step now re-uploads an object the plan would reuse when its size in the bucket differs from what the previous catalog recorded. Because v2026.11.05 and v2026.11.09 point at the same objects, they now read whole cruises too, though their catalogs still list those 24 paths two or three times.

test_release.qmd now checks every table in the catalog, supplemental ones included, before promotion: the paths are unique, a partitioned table has one object per partition value, and the rows in the published objects' parquet footers equal the catalog's rows (check_release_objects()). Against v2026.11.09 as published it fails those three checks for obs_ctd_full and passes the other 75. The CTD derived-products ingest applies the same check before it falls back to the published table.

Consumers: read obs_ctd_full from this release, or pass refresh = TRUE to calcofi4r::cc_get_db() if you opened v2026.11.05 or v2026.11.09 with supplemental = TRUE: the cached connection holds the repeated file list. calcofi4r, calcofi4py and db-query now drop a repeated path when they read a catalog.

sample_root says which stage its data are at, so an app can label a preliminary cruise

sample_root gains data_stage as its last column (calcofi4db 4.37.0, build_sample_root()): the root sample's own sample.data_stage, NULL where the dataset records none. Until now the stage was only on sample, which the Explorer does not load, so it drew the preliminary fall cruise with no label while the CTD transects app marked it. The release notebook's column-order check accepts hex7 followed by data_stage.

Consumers: additive. A reader that selects sample_root columns by name is unaffected; one that relies on hex7 being the table's last column should select by name instead.

Contents (generated)

table rows
climatology 1,921,596 partitioned
cruise 869
dataset 20
dataset_taxon 1,940
fish 10,796
grid 225
grid_crosswalk 673
ichthyo_net 77,888
lookup 26
measurement_type 298
obs 36,096,173 deprecated → obs_bio, obs_env (objects removed in next)
obs_attribute 455,300
obs_bio 1,586,795
obs_env 34,509,378 partitioned
region 4
sample 1,467,430
sample_measurement 715,139
sample_spatial 2,296,225
ship 49
spatial 3,395
spatial_attribute 37,968
taxon 2,606
taxon_group 438
obs_ctd_full 287,281,956 supplemental
obs_mets_full 19,926,523 supplemental
sample_root 419,489 supplemental

26 tables, 386,813,199 rows, 2.89 GB.

Datasets (20): calcofi_2022-edna, calcofi_bottle, calcofi_bottle-derived, calcofi_ctd-cast, calcofi_ctd-derived, calcofi_dic, calcofi_mets, calcofi_phyllosoma, calcofi_phytoplankton, cce-lter_euphausiids, cce-lter_iron, cce-lter_picoplankton-bacteria, cce-lter_zoodb, cce-lter_zooscan, cdfw_dungeness-crab, farallon_bird-mammal, sio_mesopelagic-fish, sio_pic-zooplankton, swfsc_cufes, swfsc_ichthyo

Validation: 184 pass / 0 fail / 4 skip (consumer-contract suite, 2026-11-13T06:01:18Z).

Software: calcofi4db 4.38.0, calcofi4r 1.31.1.

How to cite

CalCOFI (2026). CalCOFI Integrated Database, release v2026.11.12 [Data set]. Scripps Institution of Oceanography, NOAA Fisheries, and California Department of Fish and Wildlife. https://doi.org/10.5281/zenodo.24187468

Cite the source datasets you use alongside the release:

Access

con <- calcofi4r::cc_get_db(version = "v2026.11.12")
con = calcofi4py.cc_get_db("v2026.11.12")

Parquet: https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.11.12/parquet/{table}.parquet; full history: RELEASES.md.