# Data Organization: S3 Buckets & Available Dates

*Last updated: 08 JUL 2026*

This document describes where the LLC4320 data used by this codebase lives,
what data sits under each prefix, and which dates are available. Two storage
locations are involved:

* **`s3://dbof`** (NRP Nautilus, endpoint `https://s3-west.nrp-nautilus.io`)
  — our internal bucket. **Requires credentials**; this section is for
  internal users only. External users of this code will not be able to read
  these prefixes.
* **OSN** (endpoint `https://mghp.osn.xsede.org`) — a **public, anonymous**
  bucket hosting kerchunk references to LLC4320 surface data, readable by
  anyone. Used by the SURF and OSN pipelines.

Store readiness on `s3://dbof` (grid.zarr present, timestamps readable by
the global pipeline) is verified by `tests/test_raw_store_readiness.py`:

```bash
pytest --run-integration -m s3_dbof tests/test_raw_store_readiness.py
```

Date stamps in store names use `YYYYMMDDTHH` (e.g. `20121109T12.zarr` =
2012-11-09 12:00 UTC).

## Internal bucket layout (`s3://dbof`, credentialed)

```
s3://dbof/
├── LLC4320_RAW/                      # raw data transferred from MIT
│   ├── SURFACE/                      # 2D surface fields + grid
│   │   ├── grid.zarr
│   │   └── {YYYYMMDDTHH}.zarr
│   ├── DEPTH/                        # full-depth 3D fields + grid
│   │   ├── grid.zarr
│   │   └── {YYYYMMDDTHH}.zarr
│   └── CHUNKS/                       # single 720x720 native chunks
│       └── {chunk_name}/
│           ├── grid.zarr
│           └── {YYYYMMDDTHH}.zarr
├── surface_fields/                   # global pipeline output (SURF/OSN)
│   └── {run_id}/{date}/{subset}.zarr
├── depth_fields/                     # global pipeline output (DEPTH)
│   └── {run_id}/{date}/{subset}.zarr
├── LLC4320_GRID_2D/                  # derived 2D lat/lon grid
│   └── llc4320_grid.zarr
├── native_grid_dbof_training_data/   # legacy generate-global output + old grid copy
│   └── llc4320_grid.zarr
├── LLC4320/                          # LEGACY — original first S3 copy
└── LLC4320_v1/                       # LEGACY — superseded by LLC4320_RAW/DEPTH
```

## Data transferred from MIT

Transferred with `transfer-timestep` (`src/dbof/cli/transfer_llc4320.py`);
configs in `configs/transfer/`. Each prefix holds one `grid.zarr` (static
grid variables, written once) plus one store per date containing the
time-varying fields and a `time` variable.

### LLC_SURFACE

Surface-only native fields and grid.

* **Location:** `s3://dbof/LLC4320_RAW/SURFACE/{date}.zarr/{field}`
* **Config:** `configs/transfer/run_surface.yaml`
* **Fields:** `Eta`, `SIarea`, `oceTAUX`, `oceTAUY`, `oceQnet`, `time`
* **Dates (23 stores):**
  * `20111209T12`
  * `2012MM01T12` and `2012MM09T12` for `MM` = 01–11
    (1st and 9th of each month, Jan–Nov 2012, 12:00 UTC)

> **Known-corrupt date:** `20120801T12` is corrupt in the MIT
> source (tracked as `KNOWN_CORRUPT_DATES` in
> `tests/test_transfer_integrity.py`; xfail until the source is repaired).

### LLC_DEPTH

Full-depth (51-level) 3D native fields and grid. Chunk layout
`(k=51, face=1, j=720, i=720)` — one S3 GET retrieves the full water column
for one face tile.

* **Location:** `s3://dbof/LLC4320_RAW/DEPTH/{date}.zarr/{field}`
* **Config:** `configs/transfer/run_depth.yaml`
* **Fields:** `Theta`, `Salt`, `U`, `V`, `W`, `Eta`, `SIarea`, `oceTAUX`,
  `oceTAUY`, `oceQnet`, `time`
* **Dates (1 store):**
  * `20121109T12`

### LLC_CHUNKS

Single native 720×720 spatial chunk (all 51 depth levels) around a chosen
lat/lon, transferred across many timestamps.

* **Location:** `s3://dbof/LLC4320_RAW/CHUNKS/{chunk_name}/{date}.zarr/{field}`
* **Config:** `configs/transfer/run_chunks_monterey_bay.yaml`
* **Chunk names:** `monterey_bay` (36.8°N, −121.9°E)
* **Fields:** `Theta`, `Salt`, `U`, `V`, `W`, `Eta`, `oceTAUX`, `oceTAUY`,
  `SIarea`, `oceQnet`, `time`
* **Dates (17 stores):**
  * Daily at 12:00: `20120629T12` – `20120707T12` (9 stores)
  * 3-hourly around Jul 3: `20120703T00`, `T03`, `T06`, `T09`, `T15`,
    `T18`, `T21`, and `20120704T00` (12:00 covered by the daily set)

## Data generated by the `global` pipeline

Produced by `generate-global` (`src/dbof/cli/generate_global.py`). See
[Global_Maps.md](Global_Maps.md) for the fields contained in each subset.

### depth_fields

Global fields calculated at various depths (DEPTH pipeline).

* **Location:** `s3://dbof/depth_fields/{run_id}/{date}/{subset}.zarr`
* **Dates:** `20121109T12`
* **Current version:** V4 [16 JUN 2026] —
  **OUT OF DATE, needs regeneration to match the current structure [08 JUL 2026]**

### surface_fields

2D global fields calculated at the surface (SURF or OSN pipeline).

* **Location:** `s3://dbof/surface_fields/{run_id}/{date}/{subset}.zarr`
* **Dates:** none yet
* **Current version:** N/A [08 JUL 2026]

### LLC4320_GRID_2D

Derived 2D lat/lon grid written by `generate_grid_global.py`
(`generate-global-grid-zarr`): the native 13-face grid converted to a single
rectangular `(j, i)` = (12960, 17280) image. Contains 12 surface-relevant
variables (`XC`, `YC`, `rA`, `Depth`, `SN`, `CS`, `dxC`, `dyG`, `dyC`,
`dxG`, `rAz`, `hFacC` at k=0); no vertical grid information. Companion to
the `surface_fields`/`depth_fields` outputs — read via `GridAccessConfig` /
`grid_access` configs, not by the raw-data pipelines.

* **Location:** `s3://dbof/LLC4320_GRID_2D/llc4320_grid.zarr`
* **History:** copied on 08 JUL 2026 from
  `s3://dbof/native_grid_dbof_training_data/llc4320_grid.zarr` (the original
  copy remains there; versioned legacy configs, e.g. `access_depth_V4.yaml`,
  still reference it). When regenerating, pass
  `--folder LLC4320_GRID_2D` to `generate-global-grid-zarr`.

## OSN public bucket (anonymous access)

The SURF and OSN pipelines read LLC4320 surface data from a public bucket on
the Open Storage Network — no credentials needed, so this is the path
available to external users of the code.

* **Endpoint:** `https://mghp.osn.xsede.org`
* **Location:** `s3://cnh-bucket-1/{llc_surf,llc_wind}/kerchunk_files/`
* **Format:** kerchunk JSON references, one per face and iteration
  (see `dbof.llc4320_ingestion.get_raw_data`); grid references at
  `llc_surf/kerchunk_files/llc4320_grid_f{face}.json`
* **Fields:**
  * `llc_surf`: `Eta`, `U`, `V`, `W`, `Theta`, `Salt`
  * `llc_wind`: `KPPhbl`, `PhiBot`, `oceTAUX`, `oceTAUY`, `SIarea`
* **Dates:** hourly LLC4320 snapshots addressed by iteration number
  (`dbof.llc4320_ingestion.date_iterations` converts dates → iterations;
  note the wind store's `FIRST_WIND_RECORD_OFFSET`)

The SURF pipeline combines OSN (core ocean variables) with
`s3://dbof/LLC4320_RAW/SURFACE` (forcing variables + timestamp cross-check),
so it still requires dbof credentials; the pure-OSN pipeline does not.

## Legacy prefixes

* `s3://dbof/LLC4320/` — original first S3 copy; superseded by the 2026 migration to
  `LLC4320_RAW/SURFACE` No longer referenced by the
  pipelines (the legacy `grid_folder` override was removed from
  `LLC_DEPTH_SOURCE` on 08 JUL 2026); still used as the comparison baseline
  (`OLD_TMPL`) in `tests/test_transfer_integrity.py`. Retire that comparison
  before deleting this prefix.
* `s3://dbof/LLC4320_v1/` — superseded by the 2026 migration to
  `LLC4320_RAW/DEPTH` (verified with `dbof/transfer/verify_copy.py`).
  Candidate for deletion after the readiness tests pass against
  `LLC4320_RAW`.

## Keeping this document current

`tests/test_raw_store_readiness.py` hardcodes the expected date lists above.
If a transfer adds or removes stores, update **both** this document and the
`expected_stamps` lists in that test —
`test_bucket_matches_expected_inventory` warns about undocumented stores it
finds in the bucket.