One mask, one footprint
A matched building.
A projected map polygon is a guess about where a building is. An independent detector gives a second opinion. Only buildings where the two agree become ground truth, and they anchor a graph of buildings, roads and vehicles that every question is generated from.
Registration places each UAV image on the map to within a few metres, which is not enough to tell neighbouring buildings apart. We therefore pair what the map says with what the image shows, and keep only the buildings both agree on.

OpenStreetMap footprints of the buildings inside the registered UAV footprint, with their names, types and addresses.

Projected through the registration, the footprints land a few metres off. The homography assumes a flat scene seen straight down, so slightly oblique viewing angles, roof–footprint parallax of tall buildings and residual registration error shift them, often onto the building next door.

SAM3 segments buildings from the text prompt "building", without seeing the map. Its masks show where each building really is in the image.

The offset shared by the whole image is estimated from the two sources and removed, and masks are matched one-to-one to footprints. A pair survives only if it is unambiguous and consistent; only these pairs anchor VQA questions.
The shared shift is estimated by cross-correlating the union of projected footprints with the union of SAM3 masks and refined with a RANSAC similarity fit; corrected footprints and masks are matched one-to-one by Hungarian assignment on IoU. A pair is accepted only with IoU ≥ 0.5, a SAM3 score ≥ 0.5, a margin ≥ 0.2 over the runner-up, agreement (IoU ≥ 0.75) with a second SAM3 mask prompted by the corrected OSM box, and no contact with the image border. Images used for building questions need at least four accepted pairs, which annotators then verified.
Pick a frame, toggle the layers, and ask a question by clicking objects in the image or on the map. Level 0 retrieves map knowledge from visual evidence (the name, type or street of the building you point at); levels I to IV follow the benchmark. Auto VQA generates a few questions for each UAV-MapVQA task, T1 to T8, from the same graph.
30 frames per source, 120 in total, are included in this static copy. The demo runs entirely in your browser; running new frames or asking a model needs the research server. Best on a large screen.
SAM3 may draw one mask over a row of attached houses that OSM maps one by one, or several masks over one large mapped building. The scene graph matches groups instead of forcing one-to-one pairs.
A matched building.
Joined into one building.
Split with SAM3 box prompts at each footprint. If the split fails, the group stays one block and is never used as an answer.
Change is decided from coverage, never from a failed match: a building counts as not on the map, or missing from the image, only when less than 20% of it is covered by the other source. Anything in between is left out.
All geometry lives in a local east/north frame in metres, so relations are metric and north-referenced whatever the UAV heading. The registration residual, reg_error_m, sets the confidence of distance and direction answers.
| Node | What it carries |
|---|---|
| building | Source: matched (tier high, low, split or parts), visual only, or map only, with change unmapped, missing or ambiguous. OSM tags, outlines in the image and on the map, nearest road and address road. |
| block | A merged mask SAM3 could not split, with its member OSM buildings. |
| car | A SAM3 car that passes the metric size and shape filter, with its ground position, size, long-axis bearing and road status: on, junction, parking, off road or no road. |
| road | Drivable OSM ways grouped by name, with class, reference, width and length in view. |
| Edge | Between |
|---|---|
| on | car and the road whose carriageway it occupies |
| at_junction_of | car and each road meeting at its junction |
| in | car and a parking area |
| near | car and a road or building within 30 m, with compass direction |
| adjacent_to | building and its address street, or its nearest road |
| intersects | road and road |
Examples from all four sources. Matching colour in the map and the UAV image marks the same building. Unmapped buildings, buildings cut by the border and look-alike neighbours are left out.















