UAV-MapVQA

UAV × map scene graph

A projected map polygon is a guess about where a building is. An independent detector gives a second opinion. Only buildings where the two agree become ground truth, and they anchor a graph of buildings, roads and vehicles that every question is generated from.

Reliable correspondences from an imperfect registration

Registration places each UAV image on the map to within a few metres, which is not enough to tell neighbouring buildings apart. We therefore pair what the map says with what the image shows, and keep only the buildings both agree on.

  1. OpenStreetMap buildings around the footprint

    Map knowledge

    OpenStreetMap footprints of the buildings inside the registered UAV footprint, with their names, types and addresses.

  2. OSM footprints projected into the UAV image, visibly offset

    A plain overlay is not enough

    Projected through the registration, the footprints land a few metres off. The homography assumes a flat scene seen straight down, so slightly oblique viewing angles, roof–footprint parallax of tall buildings and residual registration error shift them, often onto the building next door.

  3. SAM3 building masks detected in the UAV image

    Visual evidence

    SAM3 segments buildings from the text prompt "building", without seeing the map. Its masks show where each building really is in the image.

  4. Aligned and verified building pairs, numbered

    Combine, then keep only confident matches

    The offset shared by the whole image is estimated from the two sources and removed, and masks are matched one-to-one to footprints. A pair survives only if it is unambiguous and consistent; only these pairs anchor VQA questions.

The matching rules

The shared shift is estimated by cross-correlating the union of projected footprints with the union of SAM3 masks and refined with a RANSAC similarity fit; corrected footprints and masks are matched one-to-one by Hungarian assignment on IoU. A pair is accepted only with IoU ≥ 0.5, a SAM3 score ≥ 0.5, a margin ≥ 0.2 over the runner-up, agreement (IoU ≥ 0.75) with a second SAM3 mask prompted by the corrected OSM box, and no contact with the image border. Images used for building questions need at least four accepted pairs, which annotators then verified.

Explore a scene graph

Pick a frame, toggle the layers, and ask a question by clicking objects in the image or on the map. Level 0 retrieves map knowledge from visual evidence (the name, type or street of the building you point at); levels I to IV follow the benchmark. Auto VQA generates a few questions for each UAV-MapVQA task, T1 to T8, from the same graph.

Open full screen

30 frames per source, 120 in total, are included in this static copy. The demo runs entirely in your browser; running new frames or asking a model needs the research server. Best on a large screen.

When masks and footprints do not split buildings the same way

SAM3 may draw one mask over a row of attached houses that OSM maps one by one, or several masks over one large mapped building. The scene graph matches groups instead of forcing one-to-one pairs.

One mask, one footprint

A matched building.

Several masks, one footprint

Joined into one building.

One mask, several footprints

Split with SAM3 box prompts at each footprint. If the split fails, the group stays one block and is never used as an answer.

Change is decided from coverage, never from a failed match: a building counts as not on the map, or missing from the image, only when less than 20% of it is covered by the other source. Anything in between is left out.

What the graph contains

All geometry lives in a local east/north frame in metres, so relations are metric and north-referenced whatever the UAV heading. The registration residual, reg_error_m, sets the confidence of distance and direction answers.

NodeWhat it carries
buildingSource: matched (tier high, low, split or parts), visual only, or map only, with change unmapped, missing or ambiguous. OSM tags, outlines in the image and on the map, nearest road and address road.
blockA merged mask SAM3 could not split, with its member OSM buildings.
carA SAM3 car that passes the metric size and shape filter, with its ground position, size, long-axis bearing and road status: on, junction, parking, off road or no road.
roadDrivable OSM ways grouped by name, with class, reference, width and length in view.
EdgeBetween
oncar and the road whose carriageway it occupies
at_junction_ofcar and each road meeting at its junction
incar and a parking area
nearcar and a road or building within 30 m, with compass direction
adjacent_tobuilding and its address street, or its nearest road
intersectsroad and road

The verified correspondence pool

Examples from all four sources. Matching colour in the map and the UAV image marks the same building. Unmapped buildings, buildings cut by the border and look-alike neighbours are left out.

Map, MagdeburgUAV image, Magdeburg
VETRA, Magdeburg, Germany
Map, KemptenUAV image, Kempten
VETRA, Kempten, Germany
Map, downtown AustinUAV image, downtown Austin
AirLock, downtown Austin
Map, residential AustinUAV image, residential Austin
AirLock, residential Austin
Map, Florida after Hurricane IanUAV image, Florida after Hurricane Ian
USGS, Florida after Hurricane Ian
Map, residential FloridaUAV image, residential Florida
USGS, residential Florida
Map, industrial area in ChinaUAV image, industrial area in China
UAV-VisLoc, industrial area, China
Map, city block in ChinaUAV image, city block in China
UAV-VisLoc, city block, China