# Nua Sapphire on WGO-Bench: public results

This folder holds every per-clip output, score and judge verdict behind the WGO-Bench numbers we publish for Nua Sapphire and NVIDIA Flora, so anyone can check them.

| Model | Step timing F1 (IoU ≥ 0.75) | Steps + captions F1 |
|---|---|---|
| **Nua Sapphire** (pre-release, 2026-11-16) | **55.9** | **34.9** |
| NVIDIA Flora | 50.6 | 31.2 |

Both scores are the average over WGO's three subsets, the way NVIDIA's Flora paper reports WGO. The per-subset and pooled numbers are in `scores.json`.

## Benchmark

- **WGO-Bench** by Macrodata: https://huggingface.co/datasets/macrodata/WGO-Bench, revision `1c86a62e53029ff85cea3a238bc58af3a40055b6`, file `data/annotations.parquet` (SHA-256 `24c6c23447753385b9cea0d17f17d7a8aa74131d800d9bc9ab74d7669d7404ab`).
- The benchmark has 100 episodes and 743 reference steps: 25 Galaxea (robot), 25 HomER (human, first-person) and 50 DROID-derived (robot).
- We do not redistribute the videos or the reference labels. Get them from the link above (licence CC BY-NC-SA 4.0).

## Metrics

We follow the benchmark authors' grading definition (https://macrodata.co/blog/annotating-robot-video-subtasks#grading) and their reference implementation (https://github.com/macrodata-labs/refiner, commit `5f7698228074f0c1163cdb830185c20dfecb3f41`).

- **Step timing F1:** a predicted step matches a reference step when their time overlap (IoU) is at least 0.75. Labels are ignored. Before scoring, the first predicted start and last predicted end are snapped to the reference's outer boundaries, as the protocol specifies. Unsnapped numbers are also given in `scores.json`.
- **Steps + captions F1:** a match must also pass a semantic check. `gemini-3.5-flash`, using the official WGO rubric, judges whether the predicted and reference captions describe the same event, object and end state. Different wording is allowed.
- Both scores are computed per subset and then averaged across the three subsets.

## How the models were run

- **Nua Sapphire:** the pre-release model dated above, one request per video. Clips are not edited or trimmed.
- **NVIDIA Flora:** the publicly released model, run by Nua on the same 100 videos. Flora's input limits required re-encoding videos to 20 fps. We got no usable output from Flora for one clip (`homer_7`). It is included with no steps and counts as a miss, as it would for any model.
- Neither model saw the reference steps.

## Files

| File | Contents |
|---|---|
| `outputs/sapphire.jsonl`, `outputs/flora.jsonl` | One line per clip: `{"id", "segments": [{"start_sec", "end_sec", "subtask"}]}`, exactly as each model produced it. |
| `judgments/*.json` | The judge's verdict for every pair that passed the timing match, keyed `"predicted_index:reference_index"` per clip. |
| `scores.json` | Subset-average, pooled and per-subset scores, with matched, predicted and reference counts, precision and recall. |
| `manifest.json` | Versions, sources and file checksums. |
| `SHA256SUMS` | Checksums for every file in this folder. |

## Verify it yourself

1. Check the files: `gsutil -m cp -r gs://nua-sapphire-benchmarks/wgo/v2026-11-16 . && cd v2026-11-16 && shasum -a 256 -c SHA256SUMS`
2. Download WGO-Bench at the revision above and confirm the parquet checksum.
3. **Step timing:** score `outputs/*.jsonl` against the references with Macrodata's reference implementation. This part has no judge, so it should reproduce our timing scores.
4. **Steps + captions:** re-run the semantic judge on the matched pairs, or audit our verdicts in `judgments/` line by line. An LLM judge varies slightly between runs, typically by under one point.

Questions, or a different result? Contact us through platform.usenua.com.
