Abstract
Robust semantic segmentation under adverse conditions is of great importance in real-world applications. To address this challenging task in practical scenarios where labeled normal condition images are not accessible in training, we propose FREST, a novel feature restora- tion framework for source-free domain adaptation (SFDA) of semantic segmentation to adverse conditions. FREST alternates two steps: (1) learning the condition embedding space that only separates the condi- tion information from the features and (2) restoring features of adverse condition images on the learned condition embedding space. By alternat- ing these two steps, FREST gradually restores features where the effect of adverse conditions is reduced. FREST achieved a state of the art on two public benchmarks (i.e., ACDC and RobotCar) for SFDA to adverse conditions. Moreover, it shows superior generalization ability on unseen datasets.
Keywords: semantic segmentation, feature restoration, robustness, source-free domain adaptation
Overview of FREST
The overall architecture and training strategy. The segmentation network is pre-trained using a labeled source dataset. For each iteration, the condition strainer and segmentation network are trained alternatingly. The frozen modules are shown in gray, the trainable modules are highlighted in red, and “sg” denotes the stop gradient. (Step 1) The condition strainer and projection head are trained to learn the condition embedding space. (Step 2) The segmentation network is trained to restore features from adverse to normal conditions on the condition embedding space. For evaluation, only the encoder φenc and decoder φdec of the segmentation network are utilized.
Detail of proposed methods
(a) Detail of positive embedding sampling strategy in the condition-specific learning. (b) Detail of the condition discriminator. (c) Detail of our network computing condition-infused feature. The condition strainers are connected to the original feed-forward layer (FFN) and multi-head selfattention layer (MHSA) through the residual connections.
Experimental Results
Comparison with previous methods
Comparison with existing methods on Cityscapes → ACDC. The results are reported in mIoU (%) on the ACDC test set.
(a) Comparison with previous methods on City → RobotCar. (b) Comparison with UDA methods on City → ACDC. (SF: source-free method). (c) Generalization performance of models adapted from Cityscapes to ACDC on ACG and Cityscapes-lindau40.
Ablation Study
Loss analysis on (a) Step 1 and (b) Step 2 on ACDC val set. (c) Analysis of the structure and training strategy in FREST on ACDC val set.
In-depth Analysis
(a) Performance according to positive embedding selection strategies and loss functions. Cls. and Contra. denote classification and contrastive loss. (b) The number of parameters for additional modules. (c) Impact of restored features.
Empirical Justification
Image reconstruction results from segmentation features. (a) Input target image. Image reconstructed by the (b) Baseline and (c) FREST. (d) Reference image. (b) The number of parameters for additional modules. (c) Impact of restored features.
Empirical analysis on the impact of feature restoration during training FREST. (a) Inter-domain shift between adverse and normal conditions. (b) Intra-domain shift within each condition. (c) Convergence in total losses for both Step 1 and Step 2.
Qualitative Results
Qualitative results of FREST (Ours), its baseline (SegFormer), and CMA on ACDC and RobotCar.
BibTeX
@inproceedings{lee2024frest,
title={FREST: Feature RESToration for Semantic Segmentation under Multiple Adverse Conditions},
author={Lee, Sohyun and Kim, Namyup and Kim, Sungyeon and Kwak, Suha},
journal={European Conference on Computer Vision},
year={2024},
}