# MQuAKE-ST KG–QA

A large **knowledge-graph + question answering (KGQA)** resource adapted from **MQuAKE-CF** and **MQuAKE-Remastered** and reconstructed over a fixed Wikidata-derived knowledge graph for multi-hop graph navigation.

**MQuAKE-ST** (where **ST** denotes *Static*) provides a shared, materialized KG together with natural-language multi-hop questions, explicit topic entities, answer entities, and path-level supervision for **MultiHop KGQA**.

## Key Features

- Large **general-knowledge static knowledge graph**
- **38,516 entities**, **665 relations**, and **724,141 triples**
- Wikidata **QID/PID identifiers** with human-readable metadata
- Templated **1–4 hop QA pairs**
- **Explicit start/topic entity** for navigation-based models
- **Gold/reference reasoning paths** for single-answer questions
- **Single-answer and multi-answer** evaluation settings
- **Paraphrased question variants** for robustness evaluation
- Shared KG snapshot designed for reproducible graph-navigation experiments

---

## Contents

### 1) Knowledge Graph (KG) Triples

KG triples are stored under the `kg/` directory.

```text
kg/
├─ train.txt
├─ dev.txt
├─ test.txt
└─ triplets.txt
```

All triple files follow the format:

```text
head relation tail
```

- Files are **tab-separated**.
- Entities are represented by **Wikidata QIDs**.
- Relations are represented by **Wikidata PIDs**.

**File descriptions**

| File | Description |
|---|---|
| `train.txt` | Training triples |
| `dev.txt` | Validation triples |
| `test.txt` | Test triples |
| `triplets.txt` | Full static graph used for navigation |

---

### 2) Metadata

Metadata is stored in the `metadata/` directory and maps machine-readable Wikidata identifiers to human-readable entity and relation information.

The release contains two metadata files:

- `node_data.csv`
- `relation_data.csv`

#### Node Data CSV schema

| Column | Description |
|---|---|
| `QID` | Wikidata entity ID |
| `Title` | Preferred entity label |
| `Description` | Wikidata description |
| `Alias` | `|`-separated aliases |
| `MID` | Freebase-compatible ID when available |
| `URL` | Wikipedia URL |
| `Forwarding` | Redirect / forwarding target |

#### Relation Data CSV schema

| Column | Description |
|---|---|
| `Property` | Wikidata relation ID |
| `Title` | Preferred relation label |
| `Description` | Wikidata description |
| `Alias` | `|`-separated aliases |

---

### 3) Question Answering (QA) Files

QA datasets are stored under the `qa/` directory.

```text
qa/
├─ single_answers/
│  ├─ qa_2hop.csv
│  ├─ qa_3hop.csv
│  ├─ qa_4hop.csv
│  └─ qa_nhop.csv
└─ multi_answers/
   ├─ qa_2hop.csv
   ├─ qa_3hop.csv
   ├─ qa_4hop.csv
   └─ qa_nhop.csv
```

| File | Description |
|---|---|
| `qa_2hop.csv` | Contains only 2-hop questions |
| `qa_3hop.csv` | Contains only 3-hop questions |
| `qa_4hop.csv` | Contains only 4-hop questions |
| `qa_nhop.csv` | Mixed dataset containing 1–4 hop questions |


### 4) Question Templates

The verified question templates used to construct MQuAKE-ST are released under the `templates/` directory.

```text
templates/
├─ single_hop_question_templates.json
└─ multi_hop_question_templates.json
```

The template files preserve the mapping between the **Wikidata relation semantics** and the natural-language questions used during QA construction. The placeholder `[X]` is replaced by the human-readable label of the question's start/topic entity.

#### `single_hop_question_templates.json`

This file maps an individual Wikidata property ID (**PID**) to a list of manually verified one-hop question templates.

- **44 relation types**
- **3 templates per relation**
- **132 one-hop templates in total**

Example:

```json
{
  "P50": [
    "Who is the author of [X]?",
    "By whom is [X] written?",
    "Who is the writer of [X]?"
  ]
}
```

Here, `P50` denotes the Wikidata relation **author**.

#### `multi_hop_question_templates.json`

This file maps a **relation-chain key** to the natural-language templates that express that ordered multi-hop composition. The key uses the same `PID->PID->...` representation stored in the QA files as `Path-Key`.

The released file contains **574 relation chains**:

| Hop length | Relation chains |
|---|---:|
| **2-hop** | 221 |
| **3-hop** | 186 |
| **4-hop** | 167 |
| **Total** | 574 |

Across these chains, the file contains **2163 multi-hop templates**. Each relation chain has between **3 and 7** manually verified templates, with at least three variants available for every released chain.

For example:

```json
{
  "P112->P69": [
    "Which educational institution educated the founder of [X]?",
    "What educational institution did the founder of [X] attend?",
    "At which educational institution did the founder of [X] receive their education?"
  ]
}
```

The key `P112->P69` denotes the ordered relation chain **founded by → educated at**. A three-hop example is `P112->P69->P159`, corresponding to **founded by → educated at → headquarters location**.

During QA construction, the relation sequence of a valid path is used to select the corresponding template bank. Question variants are then sampled from that bank, with one selected as the canonical question. The released template files therefore make the mapping from **graph relation chains to natural-language questions** explicit and allow additional QA instances to be generated over the same static KG while preserving the intended relation semantics.


---

### Important Note on Single-Answer and Multi-Answer Splits

The **Single-Answer** and **Multi-Answer** variants are alternative QA settings built over the **same static KG** and should **not be assumed to form mutually disjoint train/dev/test benchmarks**.

Questions or semantically equivalent queries may occur in both variants, and their split assignments are not guaranteed to remain aligned across the two releases. In particular, the same or equivalent `(Source-Entity, Path-Key)` query may appear in the training partition of one variant and in the development or test partition of the other.

Therefore, **do not train on one answer setting and evaluate on the other using the provided splits as-is** if the goal is a leakage-free evaluation. For example, training on `single_answers/` and testing on `multi_answers/` may expose the model during training to a query that is used for evaluation in the other setting.

Using the **same underlying KG** across the two variants is intentional and is **not itself QA leakage**. The concern is overlap among the **QA instances and their train/dev/test assignments**, not access to the shared graph.

If Single-Answer and Multi-Answer data are combined for training or cross-setting evaluation, construct a **unified de-duplicated split** first. At minimum, all QA instances sharing the same source entity and relation-chain semantics (for example, the same `Source-Entity` and `Path-Key`) should be assigned to the same partition.

---

### Important Note on 1-Hop Questions

1-hop questions are used **only for training** in the mixed-hop dataset.

This design choice is motivated by two factors:

- KG embedding models are trained directly on graph triples, so evaluating 1-hop QA can conflate question answering with prior exposure to the corresponding KG edge.
- Multi-hop reasoning (2–4 hops) requires learning the underlying **1-hop relations**.

We therefore reserve 1-hop questions for training and evaluate reasoning performance on 2–4-hop questions. Using 1-hop questions during training helps models **learn the base relational structure** needed for multi-hop reasoning.

---

### Triple example

```text
Q130283	P50	Q692
```

Human-readable form:

```text
Macbeth author William Shakespeare
```

### CSV schema

| Column | Description |
|---|---|
| `Question-Number` | Integer question ID |
| `Question` | Natural-language question |
| `Question-Paraphrased` | List of paraphrased questions |
| `Source` | Start/topic entity (human-readable) |
| `Source-Entity` | Encoded start/topic entity (QID) |
| `Answer` | Gold answer, or answer list in the multi-answer setting |
| `Answer-Entity` | Encoded answer QID, or QID list in the multi-answer setting |
| `Hops` | Reasoning hop count (1 / 2 / 3 / 4) |
| `Paths` | Gold reasoning path as an ordered list of triples `(h, r, t)` from source to answer. Only available in Single-Answer setting. |
| `Paths-Label` | Human-readable version of `Paths`. Only available in Single-Answer setting. |
| `Path-Key` | Relation-sequence key such as `P50->P27` |
| `SplitLabel` | Dataset split (`train` / `dev` / `test`) |

**Multi-Answer Note**

Because the Multi-Answer setting may contain multiple branching entity-level paths for the same source entity and relation chain, `Paths` and `Paths-Label` are not stored as a single gold path. Instead, `Source-Entity` together with `Path-Key` defines the intended relation-chain semantics and can be used to construct the semantically valid entity-level paths lazily over the released KG.

### Path encoding example

```text
[['Macbeth', 'author', 'William Shakespeare'],
 ['William Shakespeare', 'country of citizenship', 'Kingdom of England']]
```

---

## Usage Notes

- **Directionality.** Triples are directed. The THESEUS/MQuAKE-ST experiments use the KG as a **directed graph**. Mirroring edges changes the navigation protocol and should only be done intentionally for a different experimental setting.
- **Shared graph.** Single-Answer and Multi-Answer use the same static KG; only the QA answer semantics differ.
- **Reference paths.** Gold paths should be interpreted as reference evidence paths. Other valid routes may exist in the KG, particularly in the Multi-Answer setting.

---

## Quick Example

**Question:** “What is the *nationality* of the *author* of Macbeth?”  
**Question-Paraphrased:** “What is the *country of citizenship* of the *author* of Macbeth?”  
**Start:** `Macbeth`  
**Gold path:** `(Macbeth, author, William Shakespeare) → (William Shakespeare, country of citizenship, Kingdom of England)`  
**Encoded path:** `(Q130283, P50, Q692), (Q692, P27, Q179876)`  
**Path-Key:** `P50->P27`  
**Answer:** `Kingdom of England` (2-hop)

---

## Dataset Construction and Upstream Sources

MQuAKE-ST is built from resources released with **MQuAKE** and **MQuAKE-Remastered**, together with a static Wikidata reconstruction.

#### Upstream resources

- **MQuAKE repository:** https://github.com/princeton-nlp/MQuAKE/
- **MQuAKE-Remastered repository:** https://github.com/henryzhongsc/MQuAKE-Remastered
- **MQuAKE-Remastered dataset:** https://huggingface.co/datasets/henryzhongsc/MQuAKE-Remastered

MQuAKE-CF provides the original multi-hop examples, annotated reasoning paths, and Wikidata-grounded entity/relation identifiers used as seeds. MQuAKE-Remastered provides improved question resources that address several quality issues in the original benchmark, including under-specified questions and duplicate cases.

#### Why a static reconstruction is needed

MQuAKE-CF was designed for **knowledge editing**, not graph navigation. Using only the triples appearing in its annotated paths produces a highly fragmented graph, while the original question texts can under-specify the intended relation chain. The source data also contains duplicate paths associated with different question texts and facts that may differ across Wikidata snapshots.

MQuAKE-ST therefore reconstructs both the graph and QA layers for a fixed graph-navigation setting.

#### Static KG reconstruction

The entities and relations appearing in the annotated MQuAKE-CF paths are used as **Wikidata seeds**. For each seed entity, Wikidata is queried for triples in which the entity appears as either the head or tail.

The construction pipeline expands the initial seed collection from roughly **9K triples** to more than **5M candidate triples**. The expanded graph is then filtered to obtain a manageable navigation space. The resulting released graph forms a single weakly connected component:

- Redirected QIDs are canonicalized to a single identifier, preferring the original MQuAKE-CF QID when available.
- A frequency threshold of **40** is applied to control graph size and remove very infrequent entities and relations.
- Original seed entities and relations are exempt from this frequency pruning so that the semantic vocabulary required by the source paths is retained.
- Original seed **triples are not forcibly reinserted** when they are absent from the reconstructed snapshot, because MQuAKE-CF and MQuAKE-ST rely on different Wikidata snapshots. This avoids introducing temporal inconsistencies for facts such as office holders, spouses, team memberships, or organizational affiliations.

#### QA and template reconstruction

The QA layer is rebuilt around **verified relation-chain templates** rather than directly reusing every original MQuAKE-CF question.

Improved question templates from **MQuAKE-Remastered** are aligned with the relation sequence of each reasoning path. The templates are further polished using relation metadata, with GPT-assisted editing followed by manual verification, so that each question preserves the intended relation semantics. Each released one-hop relation or multi-hop relation chain has at least **three manually verified question templates**.

For original question/path pairs that are not usable in the reconstructed static graph, replacement paths are sampled from the reconstructed KG using the same relation-chain semantics. Retained and newly sampled paths must be:

- contiguous;
- present in the released KG; and
- unique at the Wikidata-identifier level.

For each path, three template variants are sampled and one is selected as the canonical question. When a higher-hop relation chain contains a verified shorter prefix, the corresponding lower-hop question may also be instantiated.

#### Answer settings

MQuAKE-ST provides two QA settings over the same static KG:

- **Single-Answer:** each topic entity and relation chain is constrained to yield a single semantically valid answer.
- **Multi-Answer:** all semantically valid answer entities reachable from the topic entity under the same relation-chain semantics are retained. Reaching any valid answer is considered correct.

For QA splitting, the datasets use an approximately **80/10/10 train/dev/test split**, while **1-hop questions are reserved for training**.

Please cite **MQuAKE**, **MQuAKE-Remastered**, and the MQuAKE-ST / THESEUS work when using this resource; see the [Citation](#citation) section below.

---

## Dataset Statistics

### Knowledge Graph

- Entities: **38,516**
- Relations: **665**
- Triples: **724,141**

### Single-Answer QA

- Total questions: **29,081**

| Hop | Total | Train | Dev | Test |
|---|---:|---:|---:|---:|
| **1-hop** | 13,733 | 13,733 | 0 | 0 |
| **2-hop** | 9,432 | 7,586 | 938 | 908 |
| **3-hop** | 4,215 | 3,356 | 439 | 420 |
| **4-hop** | 1,701 | 1,346 | 179 | 176 |

### Multi-Answer QA

- Total questions: **21,979**

| Hop | Total | Train | Dev | Test |
|---|---:|---:|---:|---:|
| **1-hop** | 12,959 | 12,959 | 0 | 0 |
| **2-hop** | 4,723 | 3,780 | 488 | 455 |
| **3-hop** | 2,418 | 1,953 | 231 | 234 |
| **4-hop** | 1,879 | 1,513 | 185 | 181 |

---

## Loading Tip

Several QA columns are serialized list literals in CSV form. In Python, parse them with `ast.literal_eval`.

```python
import ast
import pandas as pd

single_df = pd.read_csv("qa/single_answers/qa_nhop.csv")
single_df["Question-Paraphrased"] = single_df["Question-Paraphrased"].map(ast.literal_eval)
single_df["Paths"] = single_df["Paths"].map(ast.literal_eval)
single_df["Paths-Label"] = single_df["Paths-Label"].map(ast.literal_eval)

multi_df = pd.read_csv("qa/multi_answers/qa_nhop.csv")
multi_df["Question-Paraphrased"] = multi_df["Question-Paraphrased"].map(ast.literal_eval)
multi_df["Answer"] = multi_df["Answer"].map(ast.literal_eval)
multi_df["Answer-Entity"] = multi_df["Answer-Entity"].map(ast.literal_eval)
```

---

## Citation

If you use this resource, please cite the original **MQuAKE** and **MQuAKE-Remastered** works together with the paper introducing **MQuAKE-ST**.

### MQuAKE

```bibtex
@inproceedings{zhong2023mquake,
  title={{MQuAKE}: Assessing Knowledge Editing in Language Models via Multi-Hop Questions},
  author={Zhong, Zexuan and Wu, Zhengxuan and Manning, Christopher D. and Potts, Christopher and Chen, Danqi},
  booktitle={Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  pages={15686--15702},
  year={2023},
  doi={10.18653/v1/2023.emnlp-main.971}
}
```

### MQuAKE-Remastered

```bibtex
@inproceedings{zhong2025mquake,
  title={MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable Evaluations},
  author={Shaochen Zhong and Yifan Lu and Lize Shao and Bhargav Bhushanam and Xiaocong Du and Louis Feng and Yixin Wan and Yucheng Shi and Daochen Zha and Yiwei Wang and Ninghao Liu and Kaixiong Zhou and Shuai Xu and Vipin Chaudhary and Xia Hu},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=m9wG6ai2Xk}
}
```

### MQuAKE-ST / THESEUS

```bibtex
@article{hernandez2026theseus,
  author = {Hernandez, Eduin E. and Garcia, Luis F. and Askar, Nurassyl and Diaz, Sergio A. and Rini, Stefano},
  title = {Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation},
  journal = {arXiv preprint arXiv:2609.14528},
  year = {2026}
}
```

---

## License

The **MQuAKE-ST-specific contributions** to this release, including the static KG organization and preprocessing, reconstructed QA pairs, verified relation-chain templates, reasoning-path supervision, dataset splits, answer-setting definitions, and paraphrased question variants, are licensed under the [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) license, except where upstream materials retain their original licenses or terms.

The original **MQuAKE** resources are distributed through the [Princeton NLP MQuAKE repository](https://github.com/princeton-nlp/MQuAKE/), which is released under the **MIT License**.

The **MQuAKE-Remastered** dataset is distributed through [Hugging Face](https://huggingface.co/datasets/henryzhongsc/MQuAKE-Remastered) under the **Creative Commons Attribution 4.0 International (CC BY 4.0)** license.

The static MQuAKE-ST knowledge graph additionally contains structured data obtained from **Wikidata**. Wikidata structured data is made available under the [Creative Commons CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/) public-domain dedication.

Please cite **MQuAKE**, **MQuAKE-Remastered**, and **Theseus in the Graph / MQuAKE-ST** when using this resource. See the [Citation](#citation) section above for the corresponding references.