Sohyun Lee | Publications

2026

Robustness to Robot Hardware Imperfections: A Benchmark and a Self-Compensating VLA

Sohyun Lee*, Yoonjae Baek*, Jaesang Won, Jinnyeong Kim, Seung-Hwan Baek, Ivan Laptev, and Suha Kwak (*equal contribution)

Under Review, 2026

Abs

In this paper, we investigate the robustness of vision-language-action (VLA) models against hardware imperfections. To circumvent the high costs of physical testing, we first introduce a new simulated benchmark, termed Simulated Hardware IMperfections (SHIM), for rigorous evaluation of VLA robustness under precisely controlled hardware imperfections. SHIM models physical hardware realities by incorporating state-dependent, non-Markovian, and joint-level noises, and offers deployment scenarios that combine these noises to approximate real-world composite imperfections. We further present Self-Compensating VLA, a deployment-time adaptation framework that enables a VLA policy to adapt to hardware imperfections on the fly. Our framework adapts online from the residual between its commanded and executed motion, compensating for them without labels. Experiments on SHIM show that our framework broadly improves upon the base policy across diverse settings and successfully adapts to two physical robots with distinct imperfections.
TestDG: Test-time Domain Generalization for Continual Test-time Adaptation

Sohyun Lee, Nayeong Kim, Juwon Kang, Seong Joon Oh, and Suha Kwak

Transactions on Machine Learning Research (TMLR), 2026

Abs arXiv Code

This paper studies continual test-time adaptation (CTTA), the task of adapting a model to constantly changing unseen domains in testing while preserving previously learned knowledge. Existing CTTA methods mostly focus on adaptation to the current test domain only, overlooking generalization to arbitrary test domains a model may face in the future. To tackle this limitation, we present a novel online test-time domain generalization framework for CTTA, dubbed TestDG. TestDG aims to learn features invariant to both current and previous test domains on the fly during testing, improving the potential for effective generalization to future domains. To this end, we propose a new model architecture and a test-time adaptation strategy dedicated to learning domain-invariant features, along with a new data structure and optimization algorithm for effectively managing information from previous test domains. TestDG achieved state of the art on four public CTTA benchmarks. Moreover, it showed superior generalization to unseen test domains.
De-occluding Broadband Metalens

Seungwoo Yoon*, Dohyun Kang*, Eunsue Choi, Sohyun Lee, Seoyeon Kim, Minho Choi, Hyeonsu Heo, Dong-Ha Shin, Suha Kwak, Arka Majumdar, Junsuk Rho+, and Seung-Hwan Baek+ (*equal contribution)

Nature Communications, 2026

Abs HTML Code

Obstructions such as raindrops, fences, or dust degrade images, especially when cleaning is infeasible. Conventional approaches use bulky compound-lens arrays or computational inpainting, compromising compactness or fidelity. Metalenses promise compact imaging, but achieving both broadband and obstruction-free imaging remains challenging, since a metalens cannot simultaneously focus distant scenes and defocus nearby occlusions across a broadband spectrum. We introduce a de-occluding broadband metalens that suppresses obstructions while enabling broadband imaging with a learned phase profile. Our approach divides the spectrum of each color channel into pass and stop bands with multi-band spectral filtering; the metalens focuses light from far objects through pass bands, then light from nearby obstructions falls in the stop bands and is rejected. A neural network further enhances imaging quality. Compared with a conventional hyperbolic metalens under obstructions, our system improves PSNR by 32.29% and raises performance on object detection and two semantic segmentation benchmarks by 13.54%, 48.45% and 20.35%, respectively. This promises robust obstruction-free sensing and vision for compact systems, including mobile robots, drones, and endoscopes.
Robust Promptable Video Object Segmentation

Sohyun Lee, Yeho Gwon, Lukas Hoyer, Konrad Schindler, Christos Sakaridis, and Suha Kwak

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

Abs arXiv Website

The performance of promptable video object segmentation (PVOS) models substantially degrades under input corruptions, which prevents PVOS deployment in safety-critical domains. This paper offers the first comprehensive study on robust PVOS (RobustPVOS). We first construct a new, comprehensive benchmark with two real-world evaluation datasets of 351 video clips and more than 2,500 object masks under real-world adverse conditions. At the same time, we generate synthetic training data by applying diverse and temporally varying corruptions to existing VOS datasets. Moreover, we present a new RobustPVOS method, dubbed Memory-object-conditioned Gated-rank Adaptation (MoGA). The key to successfully performing RobustPVOS is two-fold: effectively handling object-specific degradation and ensuring temporal consistency in predictions. MoGA leverages object-specific representations maintained in memory across frames to condition the robustification process, which allows the model to handle each tracked object differently in a temporally consistent way. Extensive experiments on our benchmark validate MoGA’s efficacy, showing consistent and significant improvements across diverse corruption types on both synthetic and real-world datasets, establishing a strong baseline for future RobustPVOS research. Our benchmark will be made publicly available.

2025

GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation

Sohyun Lee, Yeho Gwon, Lukas Hoyer, and Suha Kwak

Conference on Neural Information Processing Systems (NeurIPS), 2025
ICCV 2025 Workshop on Building Foundation Models You Can Trust
Abs arXiv Website

Improving robustness of the Segment Anything Model (SAM) to input degradations is critical for its deployment in high-stakes applications such as autonomous driving and robotics. Our approach to this challenge prioritizes three key aspects: first, parameter efficiency to maintain the inherent generalization capability of SAM; second, fine-grained and input-aware robustification to precisely address the input corruption; and third, adherence to standard training protocols for ease of training. To this end, we propose gated-rank adaptation (GaRA). GaRA introduces lightweight adapters into intermediate layers of the frozen SAM, where each adapter dynamically adjusts the effective rank of its weight matrix based on the input by selectively activating (rank-1) components of the matrix using a learned gating module. This adjustment enables fine-grained and input-aware robustification without compromising the generalization capability of SAM. Our model, GaRA-SAM, significantly outperforms prior work on all robust segmentation benchmarks. In particular, it surpasses the previous best IoU score by up to 21.3%p on ACDC, a challenging real corrupted image dataset.

2024

FREST: Feature RESToration for Semantic Segmentation under Multiple Adverse Conditions

Sohyun Lee, Namyup Kim, Sungyeon Kim, and Suha Kwak

European Conference on Computer Vision (ECCV), 2024

Abs arXiv Code Website

Robust semantic segmentation under adverse conditions is of great importance in real-world applications. To address this challenging task in practical scenarios where labeled normal condition images are not accessible in training, we propose FREST, a novel feature restoration framework for source-free domain adaptation (SFDA) of semantic segmentation to adverse conditions. FREST alternates two steps: (1) learning the condition embedding space that only separates the condition information from the features and (2) restoring features of adverse condition images on the learned condition embedding space. By alternating these two steps, FREST gradually restores features where the effect of adverse conditions is reduced. FREST achieved a state of the art on two public benchmarks (\ie, ACDC and RobotCar) for SFDA to adverse conditions. Moreover, it shows superior generalization ability on unseen datasets.

2023

Active Learning for Semantic Segmentation with Multi-class Label Query

Sehyun Hwang, Sohyun Lee, Hoyoung Kim, Minhyeon Oh, Jungseul Ok, and Suha Kwak

Conference on Neural Information Processing Systems (NeurIPS), 2023

Abs arXiv Code

This paper proposes a new active learning method for semantic segmentation. The core of our method lies in a new annotation query design. It samples informative local image regions (, superpixels), and for each of such regions, asks an oracle for a multi-hot vector indicating all classes existing in the region. This multi-class labeling strategy is substantially more efficient than existing ones like segmentation, polygon, and even dominant class labeling in terms of annotation time per click. However, it introduces the class ambiguity issue in training since it assigns partial labels (, a set of candidate classes) to individual pixels. We thus propose a new algorithm for learning semantic segmentation while disambiguating the partial labels in two stages. In the first stage, it trains a segmentation model directly with the partial labels through two new loss functions motivated by partial label learning and multiple instance learning. In the second stage, it disambiguates the partial labels by generating pixel-wise pseudo labels, which are used for supervised learning of the model. Equipped with a new acquisition function dedicated to the multi-class labeling, our method outperformed previous work on Cityscapes and PASCAL VOC 2012 while spending less annotation cost.
Human Pose Estimation in Extremely Low-Light Conditions

Sohyun Lee*, Jaesung Rim*, Boseung Jeong, Geonu Kim, ByungJu Woo, Haechan Lee, Sunghyun Cho, and Suha Kwak (*equal contribution)

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Abs arXiv Code Website

We study human pose estimation in extremely low-light images. This task is challenging due to the difficulty of collecting real low-light images with accurate labels, and severely corrupted inputs that degrade prediction quality significantly. To address the first issue, we develop a dedicated camera system and build a new dataset of real low-light images with accurate pose labels. Thanks to our camera system, each low-light image in our dataset is coupled with an aligned well-lit image, which enables accurate pose labeling and is used as privileged information during training. We also propose a new model and a new training strategy that fully exploit the privileged information to learn representation insensitive to lighting conditions. Our method demonstrates outstanding performance on real extremely low-light images, and extensive analyses validate that both of our model and dataset contribute to the success.

2022

Combating Label Distribution Shift for Active Domain Adaptation

Sehyun Hwang, Sohyun Lee, Sungyeon Kim, Jungseul Ok, and Suha Kwak

European Conference on Computer Vision (ECCV), 2022
Qualcomm Innovation Fellowship Winner
Abs arXiv Code Website

We consider the problem of active domain adaptation (ADA) to unlabeled target data, of which subset is actively selected and labeled given a budget constraint. Inspired by recent analysis on a critical issue from label distribution mismatch between source and target in domain adaptation, we devise a method that addresses the issue for the first time in ADA. At its heart lies a novel sampling strategy, which seeks target data that best approximate the entire target distribution as well as being representative, diverse, and uncertain. The sampled target data are then used not only for supervised learning but also for matching label distributions of source and target domains, leading to remarkable performance improvement. On four public benchmarks, our method substantially outperforms existing methods in every adaptation scenario.
FIFO: Learning Fog-invariant Features for Foggy Scene Segmentation

Sohyun Lee, Taeyoung Son, and Suha Kwak

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
(Best Paper Finalist, Oral Presentation)
Invited paper talk at V4AS Workshop @ CVPR 2022
Qualcomm Innovation Fellowship Winner
Abs arXiv Code Website

Robust visual recognition under adverse weather conditions is of great importance in real-world applications. In this context, we propose a new method for learning semantic segmentation models robust against fog. Its key idea is to consider the fog condition of an image as its style and close the gap between images with different fog conditions in neural style spaces of a segmentation model. In particular, since the neural style of an image is in general affected by other factors as well as fog, we introduce a fog-pass filter module that learns to extract a fog-relevant factor from the style. Optimizing the fog-pass filter and the segmentation model alternately gradually closes the style gap between different fog conditions and allows to learn fog-invariant features in consequence. Our method substantially outperforms previous work on three real foggy image datasets. Moreover, it improves performance on both foggy and clear weather images, while existing methods often degrade performance on clear scenes.
Style Neophile: Constantly Seeking Novel Styles for Domain Generalization

Juwon Kang, Sohyun Lee, Namyup Kim, and Suha Kwak

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
Qualcomm Innovation Fellowship Winner
Abs Website

This paper studies domain generalization via domain-invariant representation learning. Existing methods in this direction suppose that a domain can be characterized by styles of its images, and train a network using style-augmented data so that the network is not biased to particular style distributions. However, these methods are restricted to a finite set of styles since they obtain styles for augmentation from a fixed set of external images or by interpolating those of training data. To address this limitation and maximize the benefit of style augmentation, we propose a new method that synthesizes novel styles constantly during training. Our method manages multiple queues to store styles that have been observed so far, and synthesizes novel styles whose distribution is distinct from the distribution of styles in the queues. The style synthesis process is formulated as a monotone submodular optimization, thus can be conducted efficiently by a greedy algorithm. Extensive experiments on four public benchmarks demonstrate that the proposed method is capable of achieving state-of-the-art domain generalization performance.