publications

2024

1. ARXIV

   First Place Solution to the ECCV 2024 BRAVO Challenge: Evaluating Robustness of Vision Foundation Models for Semantic Segmentation

   Tommie Kerssies, Daan Geus, and Gijs Dubbelman

   arXiv preprint arXiv:2409.17208, Sep 2024

   Abs arXiv Bib PDF Code

   In this report, we present the first place solution to the ECCV 2024 BRAVO Challenge, where a model is trained on Cityscapes and its robustness is evaluated on several out-of-distribution datasets. Our solution leverages the powerful representations learned by vision foundation models, by attaching a simple segmentation decoder to DINOv2 and fine-tuning the entire model. This approach outperforms more complex existing approaches, and achieves first place in the challenge

   @article{kerssies2022evaluating,
     title = {First Place Solution to the ECCV 2024 BRAVO Challenge: Evaluating Robustness of Vision Foundation Models for Semantic Segmentation},
     author = {Kerssies, Tommie and de Geus, Daan and Dubbelman, Gijs},
     journal = {arXiv preprint arXiv:2409.17208},
     year = {2024},
     month = sep,
   }

2. ECCV

   The BRAVO Semantic Segmentation Challenge Results in UNCV2024

   Tuan-Hung Vu, Eduardo Valle, Andrei Bursuc, Tommie Kerssies, Daan Geus, Gijs Dubbelman, Long Qian, Bingke Zhu, Yingying Chen, Ming Tang, Jinqiao Wang, Tomáš Vojíř, Jan Šochman, Jiří Matas, Michael Smith, Frank Ferrie, Shamik Basu, Christos Sakaridis, and Luc Van Gool

   In Proceedings of the European Conference on Computer Vision (ECCV) Workshops , Sep 2024

   Abs arXiv Bib Code

   We propose the unified BRAVO challenge to benchmark the reliability of semantic segmentation models under realistic perturbations and unknown out-of-distribution (OOD) scenarios. We define two categories of reliability: (1) semantic reliability, which reflects the model’s accuracy and calibration when exposed to various perturbations; and (2) OOD reliability, which measures the model’s ability to detect object classes that are unknown during training. The challenge attracted nearly 100 submissions from international teams representing notable research institutions. The results reveal interesting insights into the importance of large-scale pre-training and minimal architectural design in developing robust and reliable semantic segmentation models.

   @inproceedings{vu2024bravo,
     title = {The BRAVO Semantic Segmentation Challenge Results in UNCV2024},
     author = {Vu, Tuan-Hung and Valle, Eduardo and Bursuc, Andrei and Kerssies, Tommie and de Geus, Daan and Dubbelman, Gijs and Qian, Long and Zhu, Bingke and Chen, Yingying and Tang, Ming and Wang, Jinqiao and Vojíř, Tomáš and Šochman, Jan and Matas, Jiří and Smith, Michael and Ferrie, Frank and Basu, Shamik and Sakaridis, Christos and Van Gool, Luc},
     booktitle = {Proceedings of the European Conference on Computer Vision (ECCV) Workshops},
     year = {2024},
     month = sep,
   }

3. CVPR

   How to Benchmark Vision Foundation Models for Semantic Segmentation?

   Tommie Kerssies, Daan De Geus, and Gijs Dubbelman

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , Jun 2024

   Abs arXiv Bib PDF Code Project page

   Recent vision foundation models (VFMs) have demonstrated proficiency in various tasks but require supervised fine-tuning to perform the task of semantic segmentation effectively. Benchmarking their performance is essential for selecting current models and guiding future model developments for this task. The lack of a standardized benchmark complicates comparisons. Therefore, the primary objective of this paper is to study how VFMs should be benchmarked for semantic segmentation. To do so, various VFMs are fine-tuned under various settings, and the impact of individual settings on the performance ranking and training time is assessed. Based on the results, the recommendation is to fine-tune the ViT-B variants of VFMs with a 16x16 patch size and a linear decoder, as these settings are representative of using a larger model, more advanced decoder and smaller patch size, while reducing training time by more than 13 times. Using multiple datasets for training and evaluation is also recommended, as the performance ranking across datasets and domain shifts varies. Linear probing, a common practice for some VFMs, is not recommended, as it is not representative of end-to-end fine-tuning. The benchmarking setup recommended in this paper enables a performance analysis of VFMs for semantic segmentation. The findings of such an analysis reveal that pretraining with promptable segmentation is not beneficial, whereas masked image modeling (MIM) with abstract representations is crucial, even more important than the type of supervision used. The code for efficiently fine-tuning VFMs for semantic segmentation can be accessed through the project page.

   @inproceedings{Kerssies_2024_CVPR,
     author = {Kerssies, Tommie and De Geus, Daan and Dubbelman, Gijs},
     title = {How to Benchmark Vision Foundation Models for Semantic Segmentation?},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
     month = jun,
     year = {2024},
     pages = {1162-1171},
     url = {https://openaccess.thecvf.com/content/CVPR2024W/2WFM/html/Kerssies_How_to_Benchmark_Vision_Foundation_Models_for_Semantic_Segmentation_CVPRW_2024_paper.html},
   }

4. CVPR

   Exploring the Benefits of Vision Foundation Models for Unsupervised Domain Adaptation

   Brunó B. Englert, Fabrizio J. Piva, Tommie Kerssies, Daan De Geus, and Gijs Dubbelman

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , Jun 2024

   Abs arXiv Bib PDF Code

   Achieving robust generalization across diverse data domains remains a significant challenge in computer vision. This challenge is important in safety-critical applications, where deep-neural-network-based systems must perform reliably under various environmental conditions not seen during training. Our study investigates whether the generalization capabilities of Vision Foundation Models (VFMs) and Unsupervised Domain Adaptation (UDA) methods for the semantic segmentation task are complementary. Results show that combining VFMs with UDA has two main benefits: (a) it allows for better UDA performance while maintaining the out-of-distribution performance of VFMs, and (b) it makes certain time-consuming UDA components redundant, thus enabling significant inference speedups. Specifically, with equivalent model sizes, the resulting VFM-UDA method achieves an 8.4x speed increase over the prior non-VFM state of the art, while also improving performance by +1.2 mIoU in the UDA setting and by +6.1 mIoU in terms of out-of-distribution generalization. Moreover, when we use a VFM with 3.6x more parameters, the VFM-UDA approach maintains a 3.3x speed up, while improving the UDA performance by +3.1 mIoU and the out-of-distribution performance by +10.3 mIoU. These results underscore the significant benefits of combining VFMs with UDA, setting new standards and baselines for Unsupervised Domain Adaptation in semantic segmentation.

   @inproceedings{Englert_2024_CVPR,
     author = {Englert, Brun\'o B. and Piva, Fabrizio J. and Kerssies, Tommie and De Geus, Daan and Dubbelman, Gijs},
     title = {Exploring the Benefits of Vision Foundation Models for Unsupervised Domain Adaptation},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
     month = jun,
     year = {2024},
     pages = {1172-1180},
     url = {https://openaccess.thecvf.com/content/CVPR2024W/2WFM/html/Englert_Exploring_the_Benefits_of_Vision_Foundation_Models_for_Unsupervised_Domain_CVPRW_2024_paper.html},
   }

2023

1. PMLR

   Neural Architecture Search for Visual Anomaly Segmentation

   Tommie Kerssies, and Joaquin Vanschoren

   In Proceedings of the Second International Conference on Automated Machine Learning , Nov 2023

   Abs arXiv Bib PDF Code

   This paper presents the first application of neural architecture search to the complex task of segmenting visual anomalies. Measurement of anomaly segmentation performance is challenging due to imbalanced anomaly pixels, varying region areas, and various types of anomalies. First, the region-weighted Average Precision (rwAP) metric is proposed as an alternative to existing metrics, which does not need to be limited to a specific maximum false positive rate. Second, the AutoPatch neural architecture search method is proposed, which enables efficient segmentation of visual anomalies without any training. By leveraging a pre-trained supernet, a black-box optimization algorithm can directly minimize computational complexity and maximize performance on a small validation set of anomalous examples. Finally, compelling results are presented on the widely studied MVTec dataset, demonstrating that AutoPatch outperforms the current state-of-the-art with lower computational complexity, using only one example per type of anomaly. The results highlight the potential of automated machine learning to optimize throughput in industrial quality control.

   @inproceedings{pmlr-v224-kerssies23a,
     title = {Neural Architecture Search for Visual Anomaly Segmentation},
     author = {Kerssies, Tommie and Vanschoren, Joaquin},
     booktitle = {Proceedings of the Second International Conference on Automated Machine Learning},
     pages = {20/1--14},
     year = {2023},
     editor = {Faust, Aleksandra and Garnett, Roman and White, Colin and Hutter, Frank and Gardner, Jacob R.},
     volume = {224},
     series = {Proceedings of Machine Learning Research},
     month = nov,
     publisher = {PMLR},
     url = {https://proceedings.mlr.press/v224/kerssies23a.html},
   }

2022

1. ARXIV

   Evaluating Continual Test-Time Adaptation for Contextual and Semantic Domain Shifts

   Tommie Kerssies, Mert Kılıçkaya, and Joaquin Vanschoren

   arXiv preprint arXiv:2208.08767, Aug 2022

   Abs arXiv Bib PDF Code

   In this paper, our goal is to adapt a pre-trained convolutional neural network to domain shifts at test time. We do so continually with the incoming stream of test batches, without labels. The existing literature mostly operates on artificial shifts obtained via adversarial perturbations of a test image. Motivated by this, we evaluate the state of the art on two realistic and challenging sources of domain shifts, namely contextual and semantic shifts. Contextual shifts correspond to the environment types, for example, a model pre-trained on indoor context has to adapt to the outdoor context on CORe-50. Semantic shifts correspond to the capture types, for example a model pre-trained on natural images has to adapt to cliparts, sketches, and paintings on DomainNet. We include in our analysis recent techniques such as Prediction-Time Batch Normalization (BN), Test Entropy Minimization (TENT) and Continual Test-Time Adaptation (CoTTA). Our findings are three-fold: i) Test-time adaptation methods perform better and forget less on contextual shifts compared to semantic shifts, ii) TENT outperforms other methods on short-term adaptation, whereas CoTTA outpeforms other methods on long-term adaptation, iii) BN is most reliable and robust.

   @article{kerssies2022evaluatinh,
     title = {Evaluating Continual Test-Time Adaptation for Contextual and Semantic Domain Shifts},
     author = {Kerssies, Tommie and K{\i}l{\i}{\c{c}}kaya, Mert and Vanschoren, Joaquin},
     journal = {arXiv preprint arXiv:2208.08767},
     year = {2022},
     month = aug,
   }