Data Organization: S3 Buckets & Available Dates

Last updated: 08 JUL 2026

This document describes where the LLC4320 data used by this codebase lives, what data sits under each prefix, and which dates are available. Two storage locations are involved:

s3://dbof (NRP Nautilus, endpoint https://s3-west.nrp-nautilus.io) — our internal bucket. Requires credentials; this section is for internal users only. External users of this code will not be able to read these prefixes.
OSN (endpoint https://mghp.osn.xsede.org) — a public, anonymous bucket hosting kerchunk references to LLC4320 surface data, readable by anyone. Used by the SURF and OSN pipelines.

Store readiness on s3://dbof (grid.zarr present, timestamps readable by the global pipeline) is verified by tests/test_raw_store_readiness.py:

pytest --run-integration -m s3_dbof tests/test_raw_store_readiness.py

Date stamps in store names use YYYYMMDDTHH (e.g. 20121109T12.zarr = 2012-11-09 12:00 UTC).

Internal bucket layout (`s3://dbof`, credentialed)

s3://dbof/
├── LLC4320_RAW/                      # raw data transferred from MIT
│   ├── SURFACE/                      # 2D surface fields + grid
│   │   ├── grid.zarr
│   │   └── {YYYYMMDDTHH}.zarr
│   ├── DEPTH/                        # full-depth 3D fields + grid
│   │   ├── grid.zarr
│   │   └── {YYYYMMDDTHH}.zarr
│   └── CHUNKS/                       # single 720x720 native chunks
│       └── {chunk_name}/
│           ├── grid.zarr
│           └── {YYYYMMDDTHH}.zarr
├── surface_fields/                   # global pipeline output (SURF/OSN)
│   └── {run_id}/{date}/{subset}.zarr
├── depth_fields/                     # global pipeline output (DEPTH)
│   └── {run_id}/{date}/{subset}.zarr
├── LLC4320_GRID_2D/                  # derived 2D lat/lon grid
│   └── llc4320_grid.zarr
├── native_grid_dbof_training_data/   # legacy generate-global output + old grid copy
│   └── llc4320_grid.zarr
├── LLC4320/                          # LEGACY — original first S3 copy
└── LLC4320_v1/                       # LEGACY — superseded by LLC4320_RAW/DEPTH

Data transferred from MIT

Transferred with transfer-timestep (src/dbof/cli/transfer_llc4320.py); configs in configs/transfer/. Each prefix holds one grid.zarr (static grid variables, written once) plus one store per date containing the time-varying fields and a time variable.

LLC_SURFACE

Surface-only native fields and grid.

Location: s3://dbof/LLC4320_RAW/SURFACE/{date}.zarr/{field}
Config: configs/transfer/run_surface.yaml
Fields: Eta, SIarea, oceTAUX, oceTAUY, oceQnet, time
Dates (23 stores):
- 20111209T12
- 2012MM01T12 and 2012MM09T12 for MM = 01–11 (1st and 9th of each month, Jan–Nov 2012, 12:00 UTC)

Known-corrupt date: 20120801T12 is corrupt in the MIT source (tracked as KNOWN_CORRUPT_DATES in tests/test_transfer_integrity.py; xfail until the source is repaired).

LLC_DEPTH

Full-depth (51-level) 3D native fields and grid. Chunk layout (k=51, face=1, j=720, i=720) — one S3 GET retrieves the full water column for one face tile.

Location: s3://dbof/LLC4320_RAW/DEPTH/{date}.zarr/{field}
Config: configs/transfer/run_depth.yaml
Fields: Theta, Salt, U, V, W, Eta, SIarea, oceTAUX, oceTAUY, oceQnet, time
Dates (1 store):
- 20121109T12

LLC_CHUNKS

Single native 720×720 spatial chunk (all 51 depth levels) around a chosen lat/lon, transferred across many timestamps.

Location: s3://dbof/LLC4320_RAW/CHUNKS/{chunk_name}/{date}.zarr/{field}
Config: configs/transfer/run_chunks_monterey_bay.yaml
Chunk names: monterey_bay (36.8°N, −121.9°E)
Fields: Theta, Salt, U, V, W, Eta, oceTAUX, oceTAUY, SIarea, oceQnet, time
Dates (17 stores):
- Daily at 12:00: 20120629T12 – 20120707T12 (9 stores)
- 3-hourly around Jul 3: 20120703T00, T03, T06, T09, T15, T18, T21, and 20120704T00 (12:00 covered by the daily set)

Data generated by the `global` pipeline

Produced by generate-global (src/dbof/cli/generate_global.py). See Global_Maps.md for the fields contained in each subset.

depth_fields

Global fields calculated at various depths (DEPTH pipeline).

Location: s3://dbof/depth_fields/{run_id}/{date}/{subset}.zarr
Dates: 20121109T12
Current version: V4 [16 JUN 2026] — OUT OF DATE, needs regeneration to match the current structure [08 JUL 2026]

surface_fields

2D global fields calculated at the surface (SURF or OSN pipeline).

Location: s3://dbof/surface_fields/{run_id}/{date}/{subset}.zarr
Dates: none yet
Current version: N/A [08 JUL 2026]

LLC4320_GRID_2D

Derived 2D lat/lon grid written by generate_grid_global.py (generate-global-grid-zarr): the native 13-face grid converted to a single rectangular (j, i) = (12960, 17280) image. Contains 12 surface-relevant variables (XC, YC, rA, Depth, SN, CS, dxC, dyG, dyC, dxG, rAz, hFacC at k=0); no vertical grid information. Companion to the surface_fields/depth_fields outputs — read via GridAccessConfig / grid_access configs, not by the raw-data pipelines.

Location: s3://dbof/LLC4320_GRID_2D/llc4320_grid.zarr
History: copied on 08 JUL 2026 from s3://dbof/native_grid_dbof_training_data/llc4320_grid.zarr (the original copy remains there; versioned legacy configs, e.g. access_depth_V4.yaml, still reference it). When regenerating, pass --folder LLC4320_GRID_2D to generate-global-grid-zarr.

OSN public bucket (anonymous access)

The SURF and OSN pipelines read LLC4320 surface data from a public bucket on the Open Storage Network — no credentials needed, so this is the path available to external users of the code.

Endpoint: https://mghp.osn.xsede.org
Location: s3://cnh-bucket-1/{llc_surf,llc_wind}/kerchunk_files/
Format: kerchunk JSON references, one per face and iteration (see dbof.llc4320_ingestion.get_raw_data); grid references at llc_surf/kerchunk_files/llc4320_grid_f{face}.json
Fields:
- llc_surf: Eta, U, V, W, Theta, Salt
- llc_wind: KPPhbl, PhiBot, oceTAUX, oceTAUY, SIarea
Dates: hourly LLC4320 snapshots addressed by iteration number (dbof.llc4320_ingestion.date_iterations converts dates → iterations; note the wind store’s FIRST_WIND_RECORD_OFFSET)

The SURF pipeline combines OSN (core ocean variables) with s3://dbof/LLC4320_RAW/SURFACE (forcing variables + timestamp cross-check), so it still requires dbof credentials; the pure-OSN pipeline does not.

Legacy prefixes

s3://dbof/LLC4320/ — original first S3 copy; superseded by the 2026 migration to LLC4320_RAW/SURFACE No longer referenced by the pipelines (the legacy grid_folder override was removed from LLC_DEPTH_SOURCE on 08 JUL 2026); still used as the comparison baseline (OLD_TMPL) in tests/test_transfer_integrity.py. Retire that comparison before deleting this prefix.
s3://dbof/LLC4320_v1/ — superseded by the 2026 migration to LLC4320_RAW/DEPTH (verified with dbof/transfer/verify_copy.py). Candidate for deletion after the readiness tests pass against LLC4320_RAW.

Keeping this document current

tests/test_raw_store_readiness.py hardcodes the expected date lists above. If a transfer adds or removes stores, update both this document and the expected_stamps lists in that test — test_bucket_matches_expected_inventory warns about undocumented stores it finds in the bucket.