Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild

Conference on Robot Learning (CoRL) 2025


Derek M. S. Tan1,4, Shailesh3, Boyang Liu1, Alok Raj3, Qi Xuan Ang4, Weiheng Dai1,
Tanishq Duhan1, Jimmy Chiun1, Yuhong Cao1, Florian Shkurti2, Guillaume Sartoretti1
1National University of Singapore, 2University of Toronto,
3IIT-Dhanbad, 4Singapore Technologies Engineering
Paper Code Data ICRA-W Demo (TTA) Demo (LISA)

Visual search for bears in simulated Yosemite Valley (colored path = simulation path).

TL;DR

Search-TTA is a multimodal test-time adaptation framework that significantly corrects poor VLM predictions due to domain mismatch or the lack of training data, given various input modalities (e.g. image, text, sound) and planning methods (e.g. RL) to achieve efficient visual navigation and search in-the-wild.

AVS-Bench is a visual search dataset based on internet-scale ecological data that contains up to 380k diverse satellite images, each with taxonomies and their corresponding locations, ground image, taxonomic label, and sound data.

Abstract

To perform outdoor visual navigation and search, a robot may leverage satellite imagery to generate visual priors. This can help inform high-level search strategies, even when such images lack sufficient resolution for target recognition. However, many existing informative path planning or search-based approaches either assume no prior information, or use priors without accounting for their origins. Recent work instead utilizes large Vision Language Models (VLMs) for generalizable priors, but their outputs can be inaccurate due to hallucination, leading to inefficient search. To address these challenges, we introduce Search-TTA, a multimodal test-time adaptation framework with a flexible plug-and-play interface compatible with various input modalities (e.g., image, text, sound) and planning methods (e.g., RL-based). First, we pretrain a satellite image encoder to align with CLIP’s visual encoder to output probability distributions of target presence used for visual search. Second, our TTA framework dynamically refines CLIP’s predictions using uncertainty-weighted gradient updates online during search. To train and evaluate Search-TTA, we curate AVS-Bench, a visual search dataset based on internet-scale ecological data containing 380k images and taxonomy data. We find that Search-TTA improves planner performance by up to 30.0%, particularly in cases with poor initial CLIP predictions due to domain mismatch and limited training data. It also performs comparably with significantly larger VLMs, and achieves zero-shot generalization via emergent alignment to unseen modalities.

Test-Time Adaptation Feedback

In this work, we use CLIP as our lightweight VLM, and first align a satellite image encoder to the same representation space as a vision encoder through patch-level contrastive learning. This enables the satellite image encoder to generate a score map by taking the cosine similarity between its per-patch embeddings and the embeddings of other modalities (e.g., ground image, text, sound). We then introduce a novel test-time adaptation feedback mechanism to refine CLIP’s predictions. To achieve this, we take inspiration from Spatial Poisson Point Processes to perform gradient updates to the satellite image encoder based on past measurements. We also enhance the loss function with an uncertainty-driven weighting scheme that acts as a regularizer to ensure stable gradient updates.

Search-TTA Framework

TTA Example

Search-TTA is agnostic to different robots (e.g. UAV / UGV / AUV) and planners (e.g. RL, Info Surfing).

AVS-Bench

To validate Search-TTA, we curate AVS-Bench, a visual search dataset based on internet-scale ecological data. It comprises Sentinel-2 level 2A satellite images with unseen taxonomic targets from the iNat-2021 dataset (released for the FGVC8 workshop at CVPR'21), each tagged with ground-level image and taxonomic label (some with sound data). One advantage of using ecological data is the hierarchical structure of taxonomic labels (seven distinct tiers), which facilitates baseline evaluation across various levels of specificity. AVS-Bench is diverse in geography and taxonomies to reflect in-the-wild scenarios. Our dataset offers 380k training images and 8k validation images (in- and out-domain). Aside from Search-TTA, we use AVS-Bench to finetune LISA that output score maps and text explanations (demo link at the top).

AVS-Bench covers a diverse set of taxonomies across the world.

Results

Performance Analysis

TTA Performance Gain with Pretraining Data

VLM Inference Time Analysis



Multimodality

Emergent Alignment to Unseen Modalities




Combating Hallucination

TTA Performance Gain with Extent of Hallucination

TTA Comparisons

We validate Search-TTA on a wide variety of taxonomies. Feel free to play these videos to see the effectiveness of TTA.

With Real UAV Dynamics

Comparison without TTA

Comparison with LISA

🤗 HuggingFace Demo


Spotlight Presentation

BibTeX

    @inproceedings{tan2025searchtta,
      title        = {Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild},
      author       = {Derek Ming Siang Tan, Shailesh, Boyang Liu, Alok Raj, Qi Xuan Ang, Weiheng Dai, Tanishq Duhan, Jimmy Chiun, Yuhong Cao, Florian Shkurti, Guillaume Sartoretti},
      booktitle    = {Conference on Robot Learning},
      year         = {2025},
      organization = {PMLR}
    }

This work is completed during Derek's visiting researcher stint at the University of Toronto. Shailesh and Alok contributed to this work as interns at the National University of Singapore.