{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "source": [ "\n", " \"Open\n", "" ], "metadata": { "id": "mzo92lGD-0MZ" } }, { "cell_type": "markdown", "source": [ "# Exploring Global Multi-Spectral Satellite Mosaics with Zarr on Google Cloud Storage and SatWeave\n", "\n", "**Welcome!** This section demonstrates how to load, slice, and visualize the **SatWeave** dataset available in Zarr format on Google Cloud Storage, developed by Google DeepMind and Google Research.\n", "\n", "**Dataset Overview**\n", "\n", "SatWeave is a global, analysis-ready dataset of multi-spectral geostationary satellite observations that harmonizes imagery from **five operational geostationary satellite positions** (**GOES-East**, **GOES-West**, **Meteosat Prime 0°**, **Meteosat Indian Ocean (IODC)**, and **Himawari**) into a single global mosaic across **17 spectral bands**:\n", "* **7 Visible, Near-IR & SWIR Reflectance Bands:** `0 to 100 %`\n", "* **10 Shortwave, Water Vapor & Thermal IR Brightness Temperature Bands:** `K`\n", "\n", "SatWeave provides:\n", "* **Near-Real-Time Stream & Archive (`2026 to present`):** Updated **every hour** at **0.1° (~10 km)** resolution with **~1 hour latency** (24 timesteps/day).\n", "* **Historical Archive (`2016–2025`):** Continuous multi-year record at **30-minute intervals** (`2016-01-01T00:00:00` to `2025-12-31T23:30:00` UTC, 48 timesteps/day).\n", "\n", "SatWeave provides the live satellite observations used to initialize WeatherNext 3 every hour and to generate near-real-time precipitation estimates in PARDIG-NRT.\n", "\n", "**Notebook Purpose**\n", "\n", "This section aims to:\n", "\n", "* Familiarize users with the structure, 17 spectral channels, and coordinate conventions of SatWeave in Zarr on Google Cloud Storage.\n", "* Demonstrate how to query and slice multi-spectral satellite channels using Xarray and obstore.\n", "* Showcase practical meteorological applications, including tropical cyclone cloud-top brightness temperature analysis, multi-spectral visible vs. water vapor vs. IR comparisons, and global interactive Folium mapping.\n", "\n", "**Copyright & Attribution Information**\n", "\n", "Copyright 2026 Google LLC. All materials and data are licensed under the [Creative Commons Attribution 4.0 International License (CC-BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Publicly accessible with any Google Cloud account (no access request form required).\n", "\n", "> **Attribution Notice:**\n", "> © 2026 Google LLC's machine learning models and data processing pipelines used to create the experimental data made available under CC BY 4.0 license terms. This data is intended for experimental modeling only and is not intended, validated, or approved for real-world operational warning use.\n", "\n", "**Additional Documentation**\n", "\n", "* [SatWeave Developer Guide](https://developers.google.com/weathernext/guides/satweave)\n", "* [Xarray Library Documentation](https://docs.xarray.dev)" ], "metadata": { "id": "kdwgcj8Y13zl" } }, { "cell_type": "markdown", "metadata": { "id": "cell_1793968725108051899_19" }, "source": [ "# SatWeave Dataset Schema & Cloud Storage Structure\n", "\n", "SatWeave Zarr datasets are published in Google Cloud Storage under:\n", "`gs://satweave_1/`\n", "\n", "### Directory Structure & Storage Paths\n", "\n", "```none\n", "gs://satweave_1/\n", "└── zarr/\n", " ├── 2016_to_2025/\n", " │ └── mosaic.zarr # Historical archive (2016-01-01 to 2025-12-31 UTC · 30-min cadence)\n", " └── 2026_to_present/ # Live operational mosaics (~1 hour latency)\n", " ├── /\n", " │ └── mosaic.zarr # Single hourly mosaic · 1h cadence · (1, 1801, 3600)\n", " └── ...\n", "```\n", "\n", "### Dimensions & Coordinates\n", "\n", "
\n", "\n", "| Coordinate / Dimension | Shape & Type | Description |\n", "| --- | --- | --- |\n", "| `time` | `datetime64[ns]` | Observation valid timestamp (UTC, 30-min intervals in `2016_to_2025/`; 1-hour intervals in `2026_to_present/`). |\n", "| `latitude` | `1801` (`float64`) | Global 0.1° grid ordered north-to-south from `90.0°` to `-90.0°` (`degrees_north`). |\n", "| `longitude` | `3600` (`float64`) | Global 0.1° grid in **0° to 360° convention** (`0.0°` to `359.9°`). Convert standard `-180°..180°` using `lon % 360`. |\n", "\n", "
\n", "\n", "### Supported 17 Spectral Bands\n", "\n", "
\n", "\n", "| Variable Name | Central Wavelength | Channel Type | Units | Physical Application |\n", "| --- | --- | --- | --- | --- |\n", "| `mosaic_470_nm` | 0.47 µm (470 nm) | Blue Visible | % | Aerosols, dust, smoke, daytime clouds |\n", "| `mosaic_509_nm` | 0.51 µm (509 nm) | Green Visible | % | True-color composites, vegetation, aerosols |\n", "| `mosaic_640_nm` | 0.64 µm (640 nm) | Red Visible | % | Daytime clouds, fog, snow & ice discrimination |\n", "| `mosaic_863_nm` | 0.86 µm (863 nm) | Near-IR | % | Vegetation index, land/water contrast |\n", "| `mosaic_1370_nm` | 1.37 µm (1370 nm) | Near-IR | % | Thin high-altitude cirrus cloud detection |\n", "| `mosaic_1610_nm` | 1.61 µm (1610 nm) | SWIR | % | Water vs. ice cloud phase, snow vs. cloud |\n", "| `mosaic_2250_nm` | 2.25 µm (2250 nm) | SWIR | % | Cloud droplet effective radius, hot spots |\n", "| `mosaic_3830_nm` | 3.83 µm (3830 nm) | Shortwave IR | K | Nighttime fog/low clouds, fire & hotspot detection |\n", "| `mosaic_6200_nm` | 6.20 µm (6200 nm) | Upper Water Vapor | K | Upper-tropospheric moisture, jet streams |\n", "| `mosaic_6900_nm` | 6.90 µm (6900 nm) | Mid Water Vapor | K | Mid-tropospheric moisture, atmospheric wind flow |\n", "| `mosaic_7330_nm` | 7.33 µm (7330 nm) | Lower Water Vapor | K | Lower/mid-tropospheric moisture, convection initiation |\n", "| `mosaic_8590_nm` | 8.59 µm (8590 nm) | Cloud-Top Phase IR | K | Ice vs. water cloud phase, volcanic SO₂ |\n", "| `mosaic_9620_nm` | 9.62 µm (9620 nm) | Ozone Absorption IR| K | Total column ozone, stratospheric dynamics |\n", "| `mosaic_10400_nm` | 10.40 µm (10400 nm) | Clean Longwave IR | K | Cloud-top temperature, surface temperature |\n", "| `mosaic_11200_nm` | 11.20 µm (11200 nm) | Standard Longwave IR| K | Deep convection, cloud-top & sea surface temperature |\n", "| `mosaic_12400_nm` | 12.40 µm (12400 nm) | Split-Window IR | K | Low-level moisture, volcanic ash, SST |\n", "| `mosaic_13300_nm` | 13.30 µm (13300 nm) | CO₂ Absorption IR | K | Cloud-top height assignment, air temperature |\n", "\n", "
\n", "\n", "> **Note on Sensor Coverage & Missing Values (`NaN`):**\n", "> * **Geostationary Viewing Limits:** Geostationary coverage extends to approximately `±60°–65°` latitude; polar grid cells beyond the viewing limb are `NaN`.\n", "> * **Nighttime Reflectance:** Solar reflectance channels (`0–100 %`) are `NaN` during local nighttime.\n", "> * **Constellation Band Availability:** 11 bands are carried on all 5 satellites (~85–87% global coverage). Specific channels carried on a subset of sensors (e.g. `mosaic_509_nm` on Himawari only; `mosaic_1370_nm` on GOES only; `mosaic_470_nm`, `mosaic_2250_nm`, `mosaic_6900_nm`, `mosaic_10400_nm` on GOES and Himawari) contain `NaN` outside their respective satellite coverage sectors." ] }, { "cell_type": "markdown", "metadata": { "id": "vEwggdFJIgj_" }, "source": [ "## Import Python Libraries & Setup\n" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "zYFw6t_gIgj_", "outputId": "038776f5-a461-47ef-d2a3-2281254b9258" }, "source": [ "!pip install zarr obstore\n" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting zarr\n", " Downloading zarr-3.4.2-py3-none-any.whl.metadata (5.8 kB)\n", "Collecting obstore\n", " Downloading obstore-0.12.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB)\n", "Collecting donfig>=0.8 (from zarr)\n", " Downloading donfig-0.8.1.post1-py3-none-any.whl.metadata (5.0 kB)\n", "Requirement already satisfied: google-crc32c>=1.5 in /usr/local/lib/python3.13/dist-packages (from zarr) (1.9.0)\n", "Collecting msgspec>=0.19 (from zarr)\n", " Downloading msgspec-0.22.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (6.0 kB)\n", "Collecting numcodecs!=0.17.0,>=0.16.6 (from zarr)\n", " Downloading numcodecs-0.17.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (3.2 kB)\n", "Requirement already satisfied: numpy>=2 in /usr/local/lib/python3.13/dist-packages (from zarr) (2.4.6)\n", "Requirement already satisfied: packaging>=22.0 in /usr/local/lib/python3.13/dist-packages (from zarr) (26.3)\n", "Requirement already satisfied: typing-extensions>=4.14 in /usr/local/lib/python3.13/dist-packages (from zarr) (4.16.0)\n", "Requirement already satisfied: pyyaml in /usr/local/lib/python3.13/dist-packages (from donfig>=0.8->zarr) (6.0.3)\n", "Downloading zarr-3.4.2-py3-none-any.whl (410 kB)\n", "Downloading obstore-0.12.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m \u001b[33m0:00:00\u001b[0m\n", "\u001b[?25hDownloading donfig-0.8.1.post1-py3-none-any.whl (21 kB)\n", "Downloading msgspec-0.22.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (230 kB)\n", "Downloading numcodecs-0.17.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.4 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m16.6 MB/s\u001b[0m \u001b[33m0:00:00\u001b[0m\n", "\u001b[?25hInstalling collected packages: obstore, numcodecs, msgspec, donfig, zarr\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5/5\u001b[0m [zarr]\n", "\u001b[1A\u001b[2KSuccessfully installed donfig-0.8.1.post1 msgspec-0.22.0 numcodecs-0.17.1 obstore-0.12.1 zarr-3.4.2\n" ] } ] }, { "cell_type": "code", "metadata": { "id": "HhROsdRWIgj_" }, "source": [ "from google.colab import auth\n", "import folium\n", "from ipyleaflet import basemaps, basemap_to_tiles\n", "import pandas as pd\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "import matplotlib.dates as mdates\n", "import seaborn as sns\n", "import numpy as np\n", "import obstore\n", "from scipy import interpolate as scipy_interpolate\n", "import xarray as xr\n", "import zarr\n", "\n", "import warnings\n", "# Suppress FutureWarning\n", "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "cell_1793968737311255793_20" }, "source": [ "# SatWeave Example Queries\n", "\n", "First, open the multi-spectral SatWeave Zarr dataset from Google Cloud Storage using `obstore` and `xarray`." ] }, { "cell_type": "code", "metadata": { "colab": { "height": 409, "base_uri": "https://localhost:8080/" }, "id": "cell_1793968741473200722_21", "outputId": "7a02e3c5-b1b2-495c-c13f-14c2e02c55f3" }, "source": [ "# Open an hourly SatWeave Zarr store on Google Cloud Storage\n", "\n", "auth.authenticate_user()\n", "\n", "gcs_satweave = obstore.store.GCSStore(\n", " bucket=\"satweave_1\",\n", " prefix=\"zarr/2026_to_present/20261001T120000/mosaic.zarr\",\n", ")\n", "\n", "satweave_store = zarr.storage.ObjectStore(gcs_satweave)\n", "satweave_ds = xr.open_zarr(satweave_store, chunks={})\n", "\n", "satweave_ds" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " Size: 441MB\n", "Dimensions: (time: 1, latitude: 1801, longitude: 3600)\n", "Coordinates:\n", " * time (time) datetime64[ns] 8B 2026-10-01T12:00:00\n", " * latitude (latitude) float64 14kB 90.0 89.9 89.8 ... -89.9 -90.0\n", " * longitude (longitude) float64 29kB 0.0 0.1 0.2 ... 359.7 359.8 359.9\n", "Data variables: (12/17)\n", " mosaic_10400_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_11200_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_12400_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_13300_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_1370_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_1610_nm (time, latitude, longitude) float32 26MB dask.array\n", " ... ...\n", " mosaic_640_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_6900_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_7330_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_8590_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_863_nm (time, latitude, longitude) float32 26MB dask.array\n", " mosaic_9620_nm (time, latitude, longitude) float32 26MB dask.array\n", "Attributes:\n", " Conventions: CF-1.8\n", " title: Calibrated Geostationary Satellite Mosaic (Earl...\n", " institution: Google Research / Google DeepMind\n", " source: Calibrated geostationary satellite constellatio...\n", " grid_resolution_degrees: 0.1\n", " crs: EPSG:4326" ], "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 441MB\n",
              "Dimensions:          (time: 1, latitude: 1801, longitude: 3600)\n",
              "Coordinates:\n",
              "  * time             (time) datetime64[ns] 8B 2026-10-01T12:00:00\n",
              "  * latitude         (latitude) float64 14kB 90.0 89.9 89.8 ... -89.9 -90.0\n",
              "  * longitude        (longitude) float64 29kB 0.0 0.1 0.2 ... 359.7 359.8 359.9\n",
              "Data variables: (12/17)\n",
              "    mosaic_10400_nm  (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_11200_nm  (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_12400_nm  (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_13300_nm  (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_1370_nm   (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_1610_nm   (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    ...               ...\n",
              "    mosaic_640_nm    (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_6900_nm   (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_7330_nm   (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_8590_nm   (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_863_nm    (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "    mosaic_9620_nm   (time, latitude, longitude) float32 26MB dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>\n",
              "Attributes:\n",
              "    Conventions:              CF-1.8\n",
              "    title:                    Calibrated Geostationary Satellite Mosaic (Earl...\n",
              "    institution:              Google Research / Google DeepMind\n",
              "    source:                   Calibrated geostationary satellite constellatio...\n",
              "    grid_resolution_degrees:  0.1\n",
              "    crs:                      EPSG:4326
" ] }, "metadata": {}, "execution_count": 4 } ] }, { "cell_type": "markdown", "metadata": { "id": "cell_1793969239676568924_33" }, "source": [ "## Interactive Global Satellite Mosaic Map (`mosaic_3830_nm`)\n" ] }, { "cell_type": "code", "metadata": { "colab": { "height": 880, "base_uri": "https://localhost:8080/" }, "id": "cell_1793969246421667909_34", "outputId": "9ce3c1a8-9230-474a-bcef-1ea44d865a2a" }, "source": [ "import folium\n", "import matplotlib\n", "import numpy as np\n", "from scipy import interpolate as scipy_interpolate\n", "\n", "global_ir = satweave_ds[\"mosaic_3830_nm\"].isel(time=0).load()\n", "\n", "y_dim, x_dim = \"latitude\", \"longitude\"\n", "\n", "# Flatten coordinates and values\n", "var_flat = global_ir.stack(lat_lon=(y_dim, x_dim))\n", "x = var_flat[x_dim].values\n", "y = var_flat[y_dim].values\n", "z = var_flat.values\n", "\n", "# Convert longitudes from [0, 360) to [-180, 180] for Web Mercator mapping\n", "x = np.where(x > 180, x - 360, x)\n", "\n", "nx = global_ir.sizes[x_dim]\n", "ny = global_ir.sizes[y_dim]\n", "\n", "dx = 360.0 / nx\n", "dy = 180.0 / ny\n", "lon_bins = np.linspace(-180.0 - dx / 2, 180.0 - dx / 2, nx + 1)\n", "lat_bins = np.linspace(-90.0 - dy / 2, 90.0 + dy / 2, ny + 1)\n", "\n", "# Bin data onto regular grid\n", "weighted_sum, _, _ = np.histogram2d(x, y, bins=(lon_bins, lat_bins), weights=z)\n", "counts, _, _ = np.histogram2d(x, y, bins=(lon_bins, lat_bins))\n", "\n", "with np.errstate(divide=\"ignore\", invalid=\"ignore\"):\n", " grid = np.where(counts > 0, weighted_sum / counts, np.nan)\n", "\n", "# Transpose and flip so row 0 is North (+90) and col 0 is West (-180)\n", "grid_equirectangular = np.flipud(grid.T)\n", "\n", "# Reproject equirectangular latitudes to Web Mercator latitudes (EPSG:3857)\n", "max_merc_lat = 85.0511287798\n", "y_merc = np.linspace(np.pi, -np.pi, ny)\n", "lat_target = np.degrees(2 * np.arctan(np.exp(y_merc)) - np.pi / 2)\n", "lat_source = np.linspace(90.0, -90.0, ny)\n", "\n", "f_interp = scipy_interpolate.interp1d(\n", " lat_source,\n", " grid_equirectangular,\n", " axis=0,\n", " kind=\"linear\",\n", " bounds_error=False,\n", " fill_value=np.nan,\n", ")\n", "grid_mercator = f_interp(lat_target)\n", "\n", "# Normalize brightness temperatures (190 K to 310 K) and apply colormap\n", "norm = matplotlib.colors.Normalize(vmin=190.0, vmax=310.0, clip=True)\n", "cm = matplotlib.colormaps[\"magma\"]\n", "rgba_img = cm(norm(grid_mercator))\n", "\n", "# Set alpha to 0 for missing values (NaN beyond limb or polar limbs)\n", "rgba_img[np.isnan(grid_mercator), 3] = 0.0\n", "basemap = basemaps.OpenStreetMap\n", "tile_layer = basemap_to_tiles(basemap.DE)\n", "\n", "tile_url = tile_layer.url\n", "attribution = tile_layer.attribution\n", "# Create Folium base map\n", "m_sat = folium.Map(\n", " location=[15, 0],\n", " zoom_start=2,\n", " tiles=tile_url,\n", " attr=attribution\n", ")\n", "\n", "# Overlay raster satellite field\n", "folium.raster_layers.ImageOverlay(\n", " image=rgba_img,\n", " bounds=[[-max_merc_lat, -180.0], [max_merc_lat, 180.0]],\n", " mercator_project=False,\n", " opacity=0.8,\n", ").add_to(m_sat)\n", "\n", "# Display interactive map\n", "m_sat" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ] }, "metadata": {}, "execution_count": 5 } ] }, { "cell_type": "code", "source": [], "metadata": { "id": "_z1TDKe__8mg" }, "execution_count": null, "outputs": [] } ] }