Publications

2024

1. Paper for Continual Test-time Adaptation

   Sohyun Lee, Juwon Kang, Nayeong Kim, Seongjoon Oh, and Suha Kwak

   Under Review 2024

2. 

   FREST: Feature RESToration for Semantic Segmentation under Multiple Adverse Conditions

   Sohyun Lee, Namyup Kim, Sungyeon Kim, and Suha Kwak

   European Conference on Computer Vision (ECCV), 2024

   Abs arXiv Website

   Robust semantic segmentation under adverse conditions is of great importance in real-world applications. To address this challenging task in practical scenarios where labeled normal condition images are not accessible in training, we propose FREST, a novel feature restoration framework for source-free domain adaptation (SFDA) of semantic segmentation to adverse conditions. FREST alternates two steps: (1) learning the condition embedding space that only separates the condition information from the features and (2) restoring features of adverse condition images on the learned condition embedding space. By alternating these two steps, FREST gradually restores features where the effect of adverse conditions is reduced. FREST achieved a state of the art on two public benchmarks (\ie, ACDC and RobotCar) for SFDA to adverse conditions. Moreover, it shows superior generalization ability on unseen datasets.

2023

1. 

   Active Learning for Semantic Segmentation with Multi-class Label Query

   Sehyun Hwang,  Sohyun Lee, Hoyoung Kim, Minhyeon Oh, Jungseul Ok, and Suha Kwak

   Conference on Neural Information Processing Systems (NeurIPS), 2023

   Abs arXiv

   This paper proposes a new active learning method for semantic segmentation. The core of our method lies in a new annotation query design. It samples informative local image regions (, superpixels), and for each of such regions, asks an oracle for a multi-hot vector indicating all classes existing in the region. This multi-class labeling strategy is substantially more efficient than existing ones like segmentation, polygon, and even dominant class labeling in terms of annotation time per click. However, it introduces the class ambiguity issue in training since it assigns partial labels (, a set of candidate classes) to individual pixels. We thus propose a new algorithm for learning semantic segmentation while disambiguating the partial labels in two stages. In the first stage, it trains a segmentation model directly with the partial labels through two new loss functions motivated by partial label learning and multiple instance learning. In the second stage, it disambiguates the partial labels by generating pixel-wise pseudo labels, which are used for supervised learning of the model. Equipped with a new acquisition function dedicated to the multi-class labeling, our method outperformed previous work on Cityscapes and PASCAL VOC 2012 while spending less annotation cost.

2. 

   Human Pose Estimation in Extremely Low-Light Conditions

   Sohyun Lee*, Jaesung Rim*, Boseung Jeong, Geonu Kim, ByungJu Woo, Haechan Lee, Sunghyun Cho, and Suha Kwak (*equal contribution)

   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

   Abs arXiv Code Website

   We study human pose estimation in extremely low-light images. This task is challenging due to the difficulty of collecting real low-light images with accurate labels, and severely corrupted inputs that degrade prediction quality significantly. To address the first issue, we develop a dedicated camera system and build a new dataset of real low-light images with accurate pose labels. Thanks to our camera system, each low-light image in our dataset is coupled with an aligned well-lit image, which enables accurate pose labeling and is used as privileged information during training. We also propose a new model and a new training strategy that fully exploit the privileged information to learn representation insensitive to lighting conditions. Our method demonstrates outstanding performance on real extremely low-light images, and extensive analyses validate that both of our model and dataset contribute to the success.

2022

1. 

   Combating Label Distribution Shift for Active Domain Adaptation

   Sehyun Hwang,  Sohyun Lee, Sungyeon Kim, Jungseul Ok, and Suha Kwak

   European Conference on Computer Vision (ECCV), 2022

   Qualcomm Innovation Fellowship Winner

   Abs arXiv Website

   We consider the problem of active domain adaptation (ADA) to unlabeled target data, of which subset is actively selected and labeled given a budget constraint. Inspired by recent analysis on a critical issue from label distribution mismatch between source and target in domain adaptation, we devise a method that addresses the issue for the first time in ADA. At its heart lies a novel sampling strategy, which seeks target data that best approximate the entire target distribution as well as being representative, diverse, and uncertain. The sampled target data are then used not only for supervised learning but also for matching label distributions of source and target domains, leading to remarkable performance improvement. On four public benchmarks, our method substantially outperforms existing methods in every adaptation scenario.

2. 

   FIFO: Learning Fog-invariant Features for Foggy Scene Segmentation

   Sohyun Lee, Taeyoung Son, and Suha Kwak

   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

   (Best Paper Finalist, Oral Presentation)
   Invited paper talk at V4AS Workshop @ CVPR 2022
   Qualcomm Innovation Fellowship Winner

   Abs arXiv Code Website

   Robust visual recognition under adverse weather conditions is of great importance in real-world applications. In this context, we propose a new method for learning semantic segmentation models robust against fog. Its key idea is to consider the fog condition of an image as its style and close the gap between images with different fog conditions in neural style spaces of a segmentation model. In particular, since the neural style of an image is in general affected by other factors as well as fog, we introduce a fog-pass filter module that learns to extract a fog-relevant factor from the style. Optimizing the fog-pass filter and the segmentation model alternately gradually closes the style gap between different fog conditions and allows to learn fog-invariant features in consequence. Our method substantially outperforms previous work on three real foggy image datasets. Moreover, it improves performance on both foggy and clear weather images, while existing methods often degrade performance on clear scenes.

3. 

   Style Neophile: Constantly Seeking Novel Styles for Domain Generalization

   Juwon Kang,  Sohyun Lee, Namyup Kim, and Suha Kwak

   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

   Qualcomm Innovation Fellowship Winner

   Abs Website

   This paper studies domain generalization via domain-invariant representation learning. Existing methods in this direction suppose that a domain can be characterized by styles of its images, and train a network using style-augmented data so that the network is not biased to particular style distributions. However, these methods are restricted to a finite set of styles since they obtain styles for augmentation from a fixed set of external images or by interpolating those of training data. To address this limitation and maximize the benefit of style augmentation, we propose a new method that synthesizes novel styles constantly during training. Our method manages multiple queues to store styles that have been observed so far, and synthesizes novel styles whose distribution is distinct from the distribution of styles in the queues. The style synthesis process is formulated as a monotone submodular optimization, thus can be conducted efficiently by a greedy algorithm. Extensive experiments on four public benchmarks demonstrate that the proposed method is capable of achieving state-of-the-art domain generalization performance.