CalCOFI.io CalCOFI.io Storage

ducklake/releases/v2026.10.01 / RELEASE_NOTES.md · raw markdown ↗

CalCOFI integrated database release v2026.10.01

RELEASE_NOTES.md on gs://calcofi-db/ducklake/releases/v2026.10.01

Release date: 2026-10-01 · promoted (latest.txt)

CTD: the corrected 2607 file, provider flags on every series, cruise-corrected oxygen

Five recent cruises regain their offshore stations. The first 20-2607SH_CTDPrelim.zip wrote station numbers of 100 and above with three digits, so 100/110/120 read as 000/010/020. The distance filter then dropped every such cast as a position error: in v2026.09.11, 2026-07-3322 has 122 casts instead of 144, and lines 80, 83.3, 86.7, 90 and 93.3 stop at station 90. The provider fixed the file on 2026-09-14 (Kelsey Vogel). Only Sta and Sta_ID differ between the two files, not a single measured value, so this release adds those 22 casts back: 11,372 rows per series, and no value changes on any other cruise. The same truncation is in the still-current preliminary files for 2507SR, 2511SR, 2601RL and 2604SH, which the provider has not re-issued. The ingest repairs a station to station + 100 only where the cast's own GPS position confirms it: within 10 km of station + 100 and farther than that from the station as written. Applied to the old 2607 file, that rule reproduces the provider's correction on all 56,993 rows. Every repaired cast is listed in the ingest's report, and questions.csv Q39 asks for re-issued files. The ingest also fails the render if any dropped cast still carries the signature, so a truncation cannot drop stations quietly again. A zip re-published under the same name also invalidates the ingest's fingerprint, its checkpoint and its extraction now; before this, the corrected file would have read as "inputs unchanged" (CalCOFI/workflows#104).

Every CTD series carries the provider's own flag. The CTD team's 2026 processing software writes a quality code for each corrected, averaged and derived series (SaltAve_CorrQ, OxAve_StaCorrQ, EstChl_StaCorrQ, EstNO3_CruiseCorrQ, BATQ, PoT1Q, DynHtQ, … 27 columns, first in the corrected 2607 file). Each series' measurement_qual now comes from that column, falling back to the sensor flag it inherited before for a file in the legacy layout, so no flag on a legacy cruise moves. The two-sensor averages are still rebuilt by Rasmus Swalethorp's flag rule (combine_sensor_pair_sql()); the provider's flag on its own average rides on ours, and a cross-check table compares the two averages per file layout for the CTD team (#105).

2607 casts 1–6: the faulty secondary temperature is flagged. The provider's cruise notes say the secondary temperature sensor was faulty on casts 1–6. It reads a median 11 °C from the primary, up to 42 °C, and nothing flagged it, so 1,775 of those casts' temperature_ave values averaged it in (up to 11.3 °C off). A new committed registry, metadata/calcofi/ctd-cast/flag_overrides.csv (one row per cruise, cast range, direction and series, each with a reason, a source and a question), flags that sensor 9 on those casts, together with every series the provider derives from it (secondary salinity, sigma-theta, potential temperature and oxygen). It is applied after the provider's flags and before any average, so temperature_ave there is the primary sensor alone; the ingest asserts it. This is provisional until the CTD team flags at source (questions.csv Q37).

oxygen_ml_l_ave_cruise_corr, the cruise-corrected DO average, is new. It is built from Ox1_CruiseCorr / Ox2_CruiseCorr by the same flag rule; the source ships no average of that pair. It is canonical (in obs via ctd_thin) and bounded 0–15 ml/L like its station-corrected sibling (#106).

A new dataset: calcofi_ctd-derived, hydrographic products computed from the CTD casts

Rasmus Swalethorp asked for values derived from the CTD profiles beside the measured series, to show how much upwelling, productivity and California Undercurrent there is on each cruise (CalCOFI/workflows#98). ingest_calcofi_ctd-derived.qmd computes them from the full-resolution 1 m bins of calcofi_ctd-cast after the provider's 8/9 flags are dropped, with one tested calcofi4db (≥ 4.16.0) function per rule. It publishes:

  • obs (so obs_env and the climatology): spiciness0 (TEOS-10 spice at 0 dbar) and sigma_theta_ave (the sensor pair combined by its flags), at the depths ctd-cast publishes;
  • sample_measurement, on the ctd-cast cast:
    • mixed-layer depth by three criteria (mld_sigma_theta_003, mld_sigma_theta_0125, mld_temperature_02);
    • chl_max_depth / chl_max;
    • chl_integrated (0–200 m) / chl_integrated_depth;
  • ctd_geostrophic, a new table: relative geostrophic velocity between adjacent stations of each line, referenced to 500 dbar, in 10 dbar bins (no anomaly: relative flow).

preliminary_without_bottle casts get only mld_temperature_02, since nothing salinity-based applies. The defaults (10 m / Δσθ 0.03, 5 m median, 0–200 m, p_ref 500, 10 km minimum spacing, down cast) are provisional until Rasmus answers the dataset's questions Q01–Q05. The dataset emits no sample: every row keys to the ctd-cast cast, declared in relationships_cross.csv.

Dataset metadata: the 16 CalOOS-sheet proposals reviewed into the record (#79, #96)

Each dataset_meta.yml proposal imported from the CalOOS sheet on 2026-09-05 was reviewed field by field against its dataset (Betty Huang). What was accurate was merged, what wasn't was rejected with a reason, and what was uncertain became a proposed question for the provider. The new fields are abstracts, creators, associated parties, contacts, keywords, QC statements and maintenance, which flow into every dataset page, citation and EML document.

  • DIC: creators are now the seven authors of its NCEI citation, in order. The two PIs stay in pi_names.
  • Dungeness crab: its data source is CDFW's CNRA Open Data page (Christy Juhasz, 2026-09-14).
  • sio/pic-zooplankton: the proposal is rejected; it described the pending biovolume table (Q01).
  • The proposal files are removed now that they are reviewed.

Why each measurement matters: one cited pick, the alternatives beside it

metadata/measurement_why.csv now carries, for every measurement key with a face, one authored rank-1 sentence about the health of the ocean or the California Current, each tied to a citation whose DOI resolves, plus the alternatives a reader can open instead: a second authored line, the GOOS Essential Ocean Variable's scientific question quoted verbatim from its specification sheet, the Wikipedia article the page leads from, and calcofi.org's own words where a methods page states why. The why covers the concept, not the key, so the fifteen oxygen keys share one pick and a stand-in inherits the pick of the face it borrows, with its own reason ranked beneath it. The BibTeX entries live in ../docs/refs/refs.bib.

measurements.json is schema 1.1: an anomaly per depth band, the face registries, n_flagged

The measurements catalog record gains three things, all additive — a 1.0 reader sees the record it always saw. anomaly: for every key the release's climatology covers, the yearly departure from that baseline in each depth band it is measured in (obs_env joined to climatology on dataset_key, measurement_type, site_key, calendar month and 10 m bin), with a 1984–2021 least-squares trend per decade, the band's extremes, one symmetric ymax shared by the key's bands and the bands that hold values but reach no baseline listed under deeper[] rather than dropped. The mean is taken per cruise and then over cruises, and a year of fewer than two cruises stays in the series but never steers a trend, an extreme or the scale. A unified key merges its series before averaging, each series against the normal of its own measurement_type. The five face registries metadata/measurement_{face,chem,method,scale,why}.csv ride on the key when they carry a row for it; a kind = computed scale mark is recomputed at build from its own how (TEOS-10's gsw) and never read as a typed number, and one this package cannot evaluate says so (computed_at_build: false). n_flagged (n_values − qual_ok_n) is now on every series and every key, and observed{} — the range a page quotes — is computed inside the declared bounds and inside qual_ok: a provider's quality flag outranks a physical bound, so a flagged extreme that happens to be inside the bounds is no longer advertised as the observed maximum. Measured on v2026.09.11: 89 keys, 71 with an anomaly over 362 bands with a normal and 164 deeper bands named, 48,427 values that a provider flag keeps out of observed{}; the record grows 189 KB → 551 KB.

The climatology goes to the bottom, not to 500 m

build_climatology()'s depth_max_m defaulted to 500 m — the depth the Explorer's Sections lens draws to — and the cap travelled well beyond that lens: on v2026.09.10, 117,302 temperature, 105,689 salinity, 81,423 oxygen and 26,076 nitrate values below 500 m had no normal to depart from, so no product could show a deep anomaly at all. The default is now no depth ceiling: the ≥ 5-cruise rule the release already passes is the only rule that decides whether a cell is a baseline, and it stops the table where the data thins. Measured on v2026.09.10 at min_cruises = 5: 714,882 → 734,410 rows (+19,528, +2.7 %), deepest bin 500 → 720 m, 8.98 → 9.27 MB of parquet. The change is additive: no cell at or above 500 m is added, lost, or changed (the Explorer's own section_clim.sql returns the identical 21,306 cells above 500 m and 533 new ones at 510–720 m), so every existing anomaly reads exactly as before.

Picoplankton counts move to Biology, with a taxon on every row

The CCE-LTER flow-cytometry counts are observations of organisms, so they are now obs_bio rather than obs_env (measurement-faces plan D9; Ben, 2026-09-12). All 60,802 rows carry one measurement_type, picoplankton_abundance (cells per ml), and the population counted moves into obs.taxon_key: worms:160572 Synechococcus (16,002 rows, 2004–2023) and worms:345515 Prochlorococcus (12,789), which therefore get species pages at /species/worms-160572/ and /species/worms-345515/. The flow cytometer's other two populations are operational, not taxonomic — a size fraction and a trophic fraction — so they keep their identity in a declared dataset-local key (cce-lter_picoplankton-bacteria:picoeukaryotes, 16,009 rows; …:het_bacteria, 16,002), the same mechanism ZooScan's eggs / multiples / nauplii / others use, and both are declared in check_taxon_ids()'s allowlist.

Consequence for consumers: the four measurement keys synechococcus, prochlorococcus, picoeukaryotes and het_bacteria leave the measurements catalog — 89 keys / 94 series on v2026.09.11 become 85 / 90 — and obs_env loses those 60,802 rows (29,838,093 → 29,777,291). Their metadata/measurement_type.csv rows are not deleted: a registry row is statused, never removed, and they remain the vocabulary of the dataset's own picoplankton_bacteria_measurement table, which is still served as a compat view carrying the source's four column names.

Contents (generated)

table rows
climatology 821,001 partitioned
cruise 842
dataset 17
dataset_taxon 1,921
grid 218
lookup 26
measurement_type 211
obs 33,139,449 deprecated → obs_bio, obs_env (objects removed in next)
obs_attribute 458,184
obs_bio 1,319,467
obs_env 31,819,982 partitioned
region 4
sample 1,469,239
sample_measurement 652,879
sample_spatial 929,664
ship 49
spatial 13,206
spatial_attribute 148,461
taxon 2,623
taxon_group 441
obs_ctd_full 284,379,888 supplemental
obs_mets_full 19,926,523 supplemental
sample_root 421,538 supplemental

23 tables, 375,505,833 rows, 2.73 GB.

Datasets (17): calcofi_bottle, calcofi_ctd-cast, calcofi_ctd-derived, calcofi_dic, calcofi_mets, calcofi_phyllosoma, calcofi_phytoplankton, cce-lter_euphausiids, cce-lter_picoplankton-bacteria, cce-lter_zoodb, cce-lter_zooscan, cdfw_dungeness-crab, farallon_bird-mammal, sio_mesopelagic-fish, sio_pic-zooplankton, swfsc_cufes, swfsc_ichthyo

Validation: 87 pass / 0 fail / 4 skip (consumer-contract suite, 2026-10-01T14:22:24Z).

Software: calcofi4db 4.17.2, calcofi4r 1.24.2.

How to cite

CalCOFI (2026). CalCOFI Integrated Database, release v2026.10.01 [Data set]. Scripps Institution of Oceanography, NOAA Fisheries, and California Department of Fish and Wildlife. https://doi.org/10.5281/zenodo.23085119

Cite the source datasets you use alongside the release:

Access

con <- calcofi4r::cc_get_db(version = "v2026.10.01")

con = calcofi4py.cc_get_db("v2026.10.01")

Parquet: https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.10.01/parquet/{table}.parquet; full history: RELEASES.md.