UAV-MapVQA

Leaderboard

Exact-match accuracy per task with zero-shot chain-of-thought prompting. Proprietary models are scored on Test-0.8K (100 questions per task), the others on Test-3.2K (400 per task). Visual and Tabular share question IDs, so the comparison is paired.

Select a column header to sort within each group. Outlined cells are the best in their column. Chance is 25% per single-answer task and 7.1% on T3, which has 14 possible answer sets; overall is the mean over the eight tasks. 95% confidence half-widths are at most ±4.9 points per task and ±1.7 overall on Test-3.2K, and ±9.8 and ±3.5 on Test-0.8K.

Overall accuracy

GeoChat, a remote-sensing model, scores below chance on Visual and cannot take Tabular inputs at all: its 4K context is smaller than the serialized maps, which take 8.5K to 71K tokens.

    Is it the orientation?

    Rotating the map to the UAV heading removes the view-orientation gap. It helps strong models by 4 to 16 points per level, so orientation errors propagate into reasoning, but even the aligned GPT-5.5 stays far below people.

    ModelVisualTabular
    North-upRandomAlignedΔ Level IΔ Level IIΔ Level IVNorth-upRandomAligned
    GPT-5.546.345.655.0+10.0+10.0+7.849.648.452.6
    Claude Opus 4.741.939.149.9+11.5+16.0+4.343.745.047.9
    Qwen2.5-VL-32B23.322.924.1+2.4+0.5+0.233.031.935.2
    Qwen2.5-VL-7B24.424.226.6+2.8+6.3+0.924.525.126.0

    Accuracy (%) over all tasks except T4: north-up map, randomly rotated map (control) and heading-aligned map. Δ: aligned minus north-up per level, Visual. GPT-5.5 and Claude Opus 4.7 on 700 questions, Qwen on 2,800.

    Dense scenes favour the raster

    Scenes are split into sparse and dense by the number of OSM features in the map window, at a single dataset-wide threshold chosen with Otsu's method. The serialized map grows with every entity; the raster does not. Fine-tuned Tabular models lose 11.8 (7B) and 13.7 (32B) points in dense scenes, while the raster loses at most 2.0.

    Fine-tunedVisual sparseVisual denseTabular sparseTabular dense
    Qwen2.5-VL-7B48.246.651.239.4
    Qwen2.5-VL-32B58.156.160.546.8
    Median map tokens5295296.2K16.6K

    How models are evaluated

    All models share one prompt template: "Think step by step, then end with Answer: <letter>.", with comma-separated letters for the multi-answer T3. The prediction is parsed from the last Answer: line; unparseable answers and over-length prompts count as wrong, so no question is dropped.

    Models without built-in reasoning decode greedily. GPT-5, GPT-5.5 and the Claude models reason at medium effort within 16K tokens. In Visual, the UAV image is sent at 640 px with the map.

    Fine-tuning and people

    Qwen2.5-VL-7B and 32B are fine-tuned with 4-bit QLoRA on a single 40 GB A100, on a 12.8K-question training split that shares no scene with the test set (1,600 per task, one adapter per representation).

    As a human reference, six college students who took no part in building the dataset each answered 240 Visual questions, 30 per task, with the same inputs and one minute per question. There is no Tabular human reference: entity tables are not a practical interface for people.