Spatial Retrieval Augmented Autonomous Driving

🗺️ Task: End-to-End Planning

📖 Introduction

This repository contains the official implementation of the End-to-End Planning task from our paper: "Spatial Retrieval Augmented Autonomous Driving".

We introduce a novel Spatial Retrieval Paradigm that retrieves offline geographic images (Satellite/Streetview) based on GPS coordinates to enhance autonomous driving tasks. For End-to-End Planning, we design a plug-and-play Spatial Retrieval Adapter and a Reliability Estimation Gate to robustly fuse this external knowledge into BEV representations.

This main branch provides the implementation based on VAD.

🚀 News

• [2025-12-09] Code and checkpoint for End-to-End Planning are released!

📊 Model Zoo & Main Results

End-to-end Planning Results

All methods are reproduced. Introducing geographic information could improve the performance at long horizons. Further, on the night subset (10% of val), VAD+Geo shows substantial collision reduction at long horizons.

Method	Split	L2 1s	L2 2s	L2 3s	L2 Avg	Col 1s (%)	Col 2s (%)	Col 3s (%)	Col Avg (%)	Config	Download
VAD	Full	0.34	0.61	0.98	0.64	0.16	0.29	0.66	0.37	-	-
VAD + Geo	Full	0.35	0.62	0.96	0.64	0.14	0.29	0.64	0.36	config	model
VAD	Night	0.44	0.82	1.35	0.87	0.10	0.24	1.30	0.55	-	-
VAD + Geo	Night	0.46	0.83	1.33	0.87	0.0	0.15	1.30	0.48	config	model

📦 Installation

Prerequisites

• Python 3.8
• CUDA 11.3
• PyTorch 1.9.1

Step 1: Create and Activate Environment

conda create -n newvad python=3.8 -y
conda activate newvad

Step 2: Install CUDA Toolkit

conda install nvidia/label/cuda-11.3.0::cuda

Step 3: Install PyTorch

pip install https://mirror.sjtu.edu.cn/pytorch-wheels/cu111/torch-1.9.1+cu111-cp38-cp38-linux_x86_64.whl
pip install torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

Step 4: Install MMDetection Ecosystem

pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.html
pip install mmdet==2.14.0
pip install mmsegmentation==0.14.1
pip install timm

Step 5: Install MMDetection3D

git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout -f v0.17.1
pip install -v -e .
cd ..

Step 6: Install Other Dependencies

pip install nuscenes-devkit==1.1.9
pip install yapf==0.40.1
pip install setuptools==59.5.0

📂 Data Preparation

Step 1: Prepare Base Dataset (Following VAD)

NuScenes Dataset

Download nuScenes V1.0 full dataset and CAN bus expansion data from HERE.

Download CAN bus expansion:

# Download 'can_bus.zip'
unzip can_bus.zip 
# Move can_bus to data directory

Prepare nuScenes data:

python tools/data_converter/vad_nuscenes_converter.py nuscenes --root-path ./data/nuscenes --out-dir ./data/infos --extra-tag vad_nuscenes --version v1.0 --canbus ./data

This will generate vad_nuscenes_infos_temporal_{train,val}.pkl.

Step 2: Generate Geographic Data (nuScenes-Geography)

Following the instructions in the nuScenes-Geography project, prepare both the nuScenes-Geography dataset and its devkit.

After preparation, place the dataset directory nuScenes-Geography under data/

Step 3: Add Retrieved Geographic Data

After preparing the base dataset, merge geographic information:

python tools/merge_data.py

Step 4: Generate nuScenes-night Subset for Evaluation (Optional)

python tools/extract_night_subset.py \
    --input_pkl data/infos/vad_nuscenes_infos_temporal_val.pkl \
    --output_pkl data/infos/val_night_subset.pkl \
    --nusc_root data/nuscenes \
    --version v1.0-trainval \
    --verbose

Final Folder Structure

End2End-Planning/
├── mmdetection3d/
├── projects/
├── tools/
├── ckpts/
├── data/
│   ├── infos/
│   │   ├── vad_nuscenes_infos_temporal_train.pkl
│   │   ├── vad_nuscenes_infos_temporal_val.pkl
│   │   ├── val_night_subset.pkl
│   ├── can_bus/
│   ├── nuscenes/
│   │   ├── maps/
│   │   ├── samples/
│   │   ├── sweeps/
│   │   ├── v1.0-test/
│   │   ├── v1.0-trainval/
│   ├── nuScenes-Geography-Data/

🚄 Training & Evaluation

Train VAD_Geo with 8 GPUs

./scripts/train.sh

Eval VAD_Geo with 1 GPU

./scripts/test.sh

🖊️ Citation

@misc{spad,
      title={Spatial Retrieval Augmented Autonomous Driving}, 
      author={Xiaosong Jia and Chenhe Zhang and Yule Jiang and Songbur Wong and Zhiyuan Zhang and Chen Chen and Shaofeng Zhang and Xuanhe Zhou and Xue Yang and Junchi Yan and Yu-Gang Jiang},
      year={2025},
      eprint={2512.06865},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.06865}, 
}

🙏 Acknowledgements

This work is based on VAD. It is also greatly inspired by the following outstanding contributions to the open-source community: BEVFormer, PETR.