# MINERVA Checkpoints — MQuAKE-ST Multi-Answer

These files contain the pretrained **MINERVA** checkpoints used for the corresponding experiments in [*Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation*](https://arxiv.org/abs/2609.14528).

Three independently trained runs are released, using seeds **0**, **42**, and **100**. Each seed is distributed as a self-contained ZIP archive containing the trained TensorFlow checkpoint together with the configuration and evaluation artifacts retained from that run.

- **THESEUS:** https://github.com/HalcyonSolutions/THESEUS
- **MINERVA implementation:** https://github.com/HalcyonSolutions/MINERVA


## Release summary

| Field | Value |
| --- | --- |
| Model | MINERVA |
| Dataset | MQuAKE-ST |
| QA setting | Multi-answer |
| Question hops | 1–4 |
| Reasoning horizon | 4 hops |
| Seeds | 0, 42, 100 |
| Dataset page | https://storage.googleapis.com/halcyon_data/multihop_ds/datasets/MQuAKE_ST/index.html |

## Downloads

| Seed | Archive |
| ---: | --- |
| 0 | [seed0.zip](https://storage.googleapis.com/halcyon_data/multihop_ds/conferences/all/minerva/mquake_st/multi_answers/seed0.zip) |
| 42 | [seed42.zip](https://storage.googleapis.com/halcyon_data/multihop_ds/conferences/all/minerva/mquake_st/multi_answers/seed42.zip) |
| 100 | [seed100.zip](https://storage.googleapis.com/halcyon_data/multihop_ds/conferences/all/minerva/mquake_st/multi_answers/seed100.zip) |

## Archive layout

```text
ma_qa_nhop_reason_4hop_seed<seed>/
├── model/
│   ├── checkpoint
│   ├── model.ckpt.data-00000-of-00001
│   ├── model.ckpt.index
│   └── model.ckpt.meta
├── config.json
├── scores.txt
├── scores_updated.txt
├── scores_rephrasing.txt
├── test_beam/
│   └── test_paths.txt
└── LICENSE
```

## Archive contents

Each `seed<seed>.zip` contains one experiment directory. The `model/` directory is the loadable TensorFlow checkpoint; the remaining files document the original run and its evaluations.

| Artifact | Description |
| --- | --- |
| `model/checkpoint` | TensorFlow checkpoint-state file. |
| `model/model.ckpt.data-00000-of-00001` | TensorFlow checkpoint parameter data. |
| `model/model.ckpt.index` | TensorFlow checkpoint index. |
| `model/model.ckpt.meta` | TensorFlow graph metadata. |
| `config.json` | Effective configuration saved during the original run for reproducibility. It records the resolved run settings and original paths, but is **not** intended to be used directly as an evaluation configuration. |
| `scores.txt` | Score history/evaluation output retained from the original run. |
| `test_beam/test_paths.txt` | Human-readable per-question evaluation output for the top-ranked beam-search trajectory. |

The four files under `model/` belong to the same TensorFlow checkpoint and should be kept together.


### Additional evaluation artifacts

| Artifact | Description |
| --- | --- |
| `scores_updated.txt` | Re-evaluation using the current THESEUS metric definitions. Use this file for the corrected final metrics, including the current Multi-Answer path-fidelity evaluation. |
| `scores_rephrasing.txt` | Evaluation using the released MQuAKE-ST question rephrasings. |

### Updated metrics

For MQuAKE-ST, `scores_updated.txt` should be treated as the **authoritative evaluation output under the current THESEUS metric definitions**.

The updated evaluation reflects the metric corrections made after the original run outputs were produced:

- **PED** and **RED** are reported as unnormalized edit distances.
- In the **Multi-Answer** setting, the current evaluation additionally supports **PED** and **F1_SG**.

`scores.txt` is retained unchanged as an original-run artifact for provenance.


## Loading a checkpoint

Extract one seed archive, then point a compatible MINERVA evaluation configuration to the checkpoint prefix:

```yaml
load_model: true
model_load_dir: /path/to/extracted/experiment/model/model.ckpt
```

Use the matching [MINERVA implementation](https://github.com/HalcyonSolutions/MINERVA) and the corresponding THESEUS-formatted dataset. `config.json` is provided as a record of the original run; paths stored in it may refer to the training environment in which the checkpoint was produced.

## Integrity

SHA-256 hashes for the three seed archives are provided in `SHA256SUMS.txt`. After downloading the files, they can be checked with:

```bash
sha256sum -c SHA256SUMS.txt
```


## Dataset attribution

These checkpoints use the **Multi-Answer** MQuAKE-ST setting, in which a question may have multiple semantically valid answer entities. MQuAKE-ST is a static navigation-ready reconstruction built from MQuAKE, MQuAKE-Remastered, and Wikidata-derived graph data. See the [MQuAKE-ST dataset page](https://storage.googleapis.com/halcyon_data/multihop_ds/datasets/MQuAKE_ST/index.html) for full construction, licensing, answer semantics, and attribution details.

## Citation

If you use these checkpoints, please cite MINERVA and THESEUS. When using MQuAKE-ST, also cite the upstream MQuAKE and MQuAKE-Remastered works listed on the dataset page.

### MINERVA

```bibtex
@inproceedings{minerva,
  title     = {Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning},
  author    = {Das, Rajarshi and Dhuliawala, Shehzaad and Zaheer, Manzil and Vilnis, Luke and Durugkar, Ishan and Krishnamurthy, Akshay and Smola, Alex and McCallum, Andrew},
  booktitle = {International Conference on Learning Representations},
  year      = {2018}
}
```

### THESEUS

```bibtex
@article{hernandez2026theseus,
  author  = {Hernandez, Eduin E. and Garcia, Luis F. and Askar, Nurassyl and Diaz, Sergio A. and Rini, Stefano},
  title   = {Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation},
  journal = {arXiv preprint arXiv:2609.14528},
  year    = {2026}
}
```


### MQuAKE

```bibtex
@inproceedings{zhong2023mquake,
  title     = {{MQuAKE}: Assessing Knowledge Editing in Language Models via Multi-Hop Questions},
  author    = {Zhong, Zexuan and Wu, Zhengxuan and Manning, Christopher D. and Potts, Christopher and Chen, Danqi},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  pages     = {15686--15702},
  year      = {2023},
  doi       = {10.18653/v1/2023.emnlp-main.971}
}
```

### MQuAKE-Remastered

```bibtex
@inproceedings{zhong2025mquake,
  title     = {MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations},
  author    = {Shaochen Zhong and Yifan Lu and Lize Shao and Bhargav Bhushanam and Xiaocong Du and Louis Feng and Yixin Wan and Yucheng Shi and Daochen Zha and Yiwei Wang and Ninghao Liu and Kaixiong Zhou and Shuai Xu and Vipin Chaudhary and Xia Hu},
  booktitle = {The Thirteenth International Conference on Learning Representations},
  year      = {2025},
  url       = {https://openreview.net/forum?id=m9wG6ai2Xk}
}
```

## License

The pretrained MINERVA checkpoints and associated experiment artifacts in this release are distributed under the **Apache License 2.0**, consistent with the MINERVA implementation used to produce them.

A copy of the Apache 2.0 license is provided alongside the downloads and is also included inside each seed archive. See [`LICENSE`](./LICENSE) for the complete terms.

The datasets used to train and evaluate these checkpoints are distributed separately and remain subject to their respective licenses and attribution requirements.