Robust Promptable Video Object Segmentation

The performance of promptable video object segmentation (PVOS) models substantially degrades under input corruptions, which prevents PVOS deployment in safety-critical domains. This paper offers the first comprehensive study on robust PVOS (RobustPVOS). We first construct a new, comprehensive benchmark with two real-world evaluation datasets of 351 video clips and more than 2,500 object masks under real-world adverse conditions. At the same time, we generate synthetic training data by applying diverse and temporally varying corruptions to existing VOS datasets. Moreover, we present a new RobustPVOS method, dubbed Memory-object-conditioned Gated-rank Adaptation (MoGA). The key to successfully performing RobustPVOS is two-fold: effectively handling object-specific degradation and ensuring temporal consistency in predictions. MoGA leverages object-specific representations maintained in memory across frames to condition the robustification process, which allows the model to handle each tracked object differently in a temporally consistent way. Extensive experiments on our benchmark validate MoGA's efficacy, showing consistent and significant improvements across diverse corruption types on both synthetic and real-world datasets, establishing a strong baseline for future RobustPVOS research. Our benchmark will be made publicly available.

We propose Memory-object-conditioned Gated-rank Adaptation (MoGA) for RobustPVOS. MoGA decomposes the weight matrix of a low-rank adapter into rank-1 components and selectively activates them using object-specific representations from the memory bank. This design enables the adapter to handle distinct objects differently while ensuring temporal consistency.

We present the first RobustPVOS benchmark suite that includes manually annotated real-world evaluation datasets under adverse conditions and a synthetic corruption dataset with temporally varying degradation patterns.

We release the two real-world evaluation test sets of our RobustPVOS benchmark. Each dataset contains video clips captured under natural adverse conditions (fog, rain, snow, nighttime, low light, motion blur, etc.) with dense, per-object, pixel-level annotations.