https://doi.org/10.26089/NumMet.v27r444

Assessing the resilience of guided segmentation models under adversarially perturbed point cues

Authors

  • Andrey V. Moskalenko
  • Vladislav I. Shakhuro
  • Dmitriy S. Vatolin

Keywords:

computer vision
interactive segmentation
promptable segmentation
user clicks
adversarial attack
resilience assessment

Abstract

Guided segmentation models are widely used to extract objects and accelerate data annotation through user-provided prompts. The simplest, most intuitive, and most common type of such prompts in most annotation systems is clicks on objects of interest. However, most existing approaches to training and evaluating these models rely on unrealistic synthetically generated click prompts, while the impact of real user inputs remains insufficiently studied. Based on a crowdsourcing study involving 1800 users, we show that standard click generation strategies do not always reflect real user behavior, and that strong performance on conventional benchmarks does not guarantee reliable operation in practical settings. An exhaustive search over all possible user input locations could address the problem of assessing model resilience, but it would require enormous computational resources and is infeasible within a reasonable time. We therefore reformulate the problem as an efficient white-box adversarial attack on an interactive segmentation model. We propose resilience evaluation metrics and use them to experimentally compare models on popular datasets.


Published

2026-11-17

Issue

Section

Methods and algorithms of computational mathematics and their applications

Authors

Andrey V. Moskalenko

Vladislav I. Shakhuro

Dmitriy S. Vatolin


References

  1. C. Rother, V. Kolmogorov, and A. Blake, “GrabCut – Interactive Foreground Extraction Using Iterated Graph Cuts,” ACM Transactions on Graphics (TOG) 23 (3), 309–314 (2004).
    doi 10.1145/1186562.1015720
  2. D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics,” in Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vancouver, BC, Canada, July 07–14, 2001(IEEE Press, New York, 2001), pp. 416–423.
    doi 10.1109/ICCV.2001.937655
  3. K. McGuinness and N. E. O’Connor, “A Comparative Evaluation of Interactive Segmentation Algorithms,” Pattern Recognition 43 (2), 434–444 (2010).
    doi 10.1016/j.patcog.2009.03.008
  4. N. Xu, B. Price, S. Cohen, et al., “Deep Interactive Object Selection,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, Nevada, USA, June 26–July 1, 2016(IEEE Press, New York, 2016), pp. 373–381.
    doi 10.1109/CVPR.2016.47
  5. M. Everingham, L. Van Gool, C. K. I. Williams, et al., The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results(2012).
  6. T.-Y. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” in Proceedings 13th European Conference. Computer Vision – ECCV 2014, Zurich, Switzerland, September 6–12, 2014.ECCV 2014. Lecture Notes in Computer Science, Vol. 8693. (Springer, Cham, 2014), pp. 740–755.
    doi 10.1007/978-3-319-10602-1_48
  7. Z. Li, Q. Chen, and V. Koltun, “Interactive Image Segmentation with Latent Diversity,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 18–23, 2018(IEEE Press, 2018), pp. 577–585.
    doi 10.1109/CVPR.2018.00067
  8. F. Perazzi, J. Pont-Tuset, B. McWilliams, et al., “A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 27–30, 2016(IEEE Press, 2016), pp. 724–732.
    doi 10.1109/CVPR.2016.85
  9. B. Hariharan, P. Arbelaez, L. Bourdev, et al., “Semantic Contours from Inverse Detectors,” in 2011 International Conference on Computer Vision, Barcelona, Spain, November 06–13, 2011(IEEE Press, 2011), pp. 991–998.
    doi 10.1109/ICCV.2011.6126343
  10. W.-D. Jang and C.-S. Kim, “Interactive Image Segmentation via Backpropagating Refinement Scheme,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, June 15–20, 2019(IEEE Press, 2019), pp. 5292-5301.
    doi 10.1109/CVPR.2019.00544
  11. K. Sofiiuk, I. Petrov, O. Barinova, and A. Konushin, “f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 13–19, 2020(IEEE Press, 2020), pp. 8620–8629.
    doi 10.1109/CVPR42600.2020.00865
  12. K. Sofiiuk, I. A. Petrov, and A. Konushin, “Reviving Iterative Training with Mask Guidance for Interactive Segmentation,” in 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, October 16–19, 2022(IEEE Press, 2022), pp. 3141–3145.
    doi 10.1109/ICIP46576.2022.9897365
  13. B. Cheng, R. Girshick, P. Dollár, et al., “Boundary IoU: Improving Object-Centric Image Segmentation Evaluation,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, June 20–25, 2021(IEEE Press, 2021), pp. 15329–15337.
    doi 10.1109/CVPR46437.2021.01508
  14. H.-E. Gueziri, M. McGuffin, and C. Laporte, “Latency Management in Scribble-Based Interactive Segmentation of Medical Images,” IEEE Transactions on Biomedical Engineering 65 (5), 1140–1150 (2018).
    doi 10.1109/TBME.2017.2777742
  15. E. Agustsson, J. R. R. Uijlings, and V. Ferrari, “Interactive Full Image Segmentation by Considering All Regions Jointly,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, June 15–20, 2019(IEEE Press, 2019), pp. 11614–11623.
    doi 10.1109/CVPR.2019.01189
  16. D. P. Papadopoulos, J. R. R. Uijlings, F. Keller, and V. Ferrari, “Extreme Clicking for Efficient Object Annotation,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, October 22–29, 2017(IEEE Press, 2017), pp. 4940–4949.
    doi 10.1109/ICCV.2017.528
  17. H. Cheng, S. Xu, X. Jiang, and R. Wang, “Deep Image Matting with Flexible Guidance Input,” in Paper presented at 32nd British Machine Vision Conference, BMVC 2021, Virtual, Online
    doi 10.5244/C.35.198
  18. A. Kirillov, E. Mintun, N. Ravi, et al., “Segment Anything,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, October 01–06, 2023(IEEE Press, 2023), pp. 3992–4003.
    doi 10.1109/ICCV51070.2023.00371
  19. C. Kamann and C. Rother, “Benchmarking the Robustness of Semantic Segmentation Models,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 13–19, 2020(IEEE Press, 2020), pp. 8825–8835.
    doi 10.1109/CVPR42600.2020.00885
  20. E. A. Vorob’ev and D. E. Namiot, “Adversarial Testing of Image Segmentation Models,” Programmnaya Inzheneriya 16 (4), 190–198 (2025). [in Russian].
    doi 10.17587/prin.16.190-198
  21. N. D. Lokshin, A. V. Khvostikov, and A. S. Krylov, “Augmenting the Training Set of Histological Images with Adversarial Examples,” Programming and Computer Software 49 (3), 187–191 (2023).
    doi 10.1134/S0361768823030027
  22. Z. Guan, M. Hu, Z. Zhou, et al., “BadSAM: Exploring Security Vulnerabilities of SAM via Backdoor Attacks (Student Abstract),” in Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI-24), Vancouver, Canada, February 20–27, 2024Vol. 38, Iss. 21. (AAAI Press, Washington, DC, USA, 2024), pp. 23506–23507.
    doi 10.1609/aaai.v38i21.30448
  23. C. Zhang, C. Zhang, T. Kang, et al., “Attack-SAM: Towards Attacking Segment Anything Model with Adversarial Examples,” arXiv preprint arXiv:2305.00866 (2023).
    doi 10.48550/arXiv.2305.00866
  24. Z. Zhou, Y. Song, M. Li, et al., “DarkSAM: Fooling Segment Anything Model to Segment Nothing,” Advances in Neural Information Processing Systems 37, 49859–49880 (2024).
    doi 10.52202/079017-1579
  25. Y. Qiao, C. S. Hong, T. Kang, et al., “Robustness of SAM: Segment Anything under Corruptions and Beyond,” IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (10), 12082–12099 (2026).
    doi 10.1109/TPAMI.2026.3698837
  26. Y. Wang, Y. Zhao, and L. Petzold, “An Empirical Study on the Robustness of the Segment Anything Model (SAM),” Pattern Recognition 155 (12), Article Number 110685 (2024).
    doi 10.1016/j.patcog.2024.110685
  27. W. Brendel, J. Rauber, and M. Bethge, “Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models,” in 6th International Conference on Learning Representations, Vancouver, Canada, April 30–May 3, 2018
    https://openreview.net/group?id=ICLR.cc/2018/Conference.
  28. A. N. Bhagoji, W. He, B. Li, and D. Song, “Practical Black-Box Attacks on Deep Neural Networks Using Efficient Query Mechanisms,” in Proceedings of the European Conference on Computer Vision (ECCV 2018)Lecture Notes in Computer Science, Vol. 11216. (Springer, Cham, 2018), pp. 158–174.
    doi 10.1007/978-3-030-01258-8_10
  29. J. Su, D. V. Vargas, and K. Sakurai, “One Pixel Attack for Fooling Deep Neural Networks,” IEEE Transactions on Evolutionary Computation 23 (5), 828–841 (2019).
    doi 10.1109/TEVC.2019.2890858
  30. X. Chen, Z. Zhao, F. Yu, et al., “Conditional Diffusion for Interactive Segmentation,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, October 10–17, 2021(IEEE Press, 2021), pp. 7325–7334.
    doi 10.1109/ICCV48922.2021.00725
  31. Q. Liu, Z. Xu, G. Bertasius, and M. Niethammer, “SimpleClick: Interactive Image Segmentation with Simple Vision Transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, October 01–06, 2023(IEEE Press, 2023), pp. 22233–22243.
    doi 10.1109/ICCV51070.2023.02037
  32. S. Sun, M. Xian, F. Xu, et al., “CFR-ICL: Cascade-Forward Refinement with Iterative Click Loss for Interactive Image Segmentation,” in Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI-24), Vancouver, Canada, February 20–27, 2024Vol. 38, Iss. 5. (AAAI Press, Washington, DC, USA, 2024), pp. 5017–5024.
    doi 10.1609/aaai.v38i5.28306
  33. M. Zhou, H. Wang, Q. Zhao, et al., “Interactive Segmentation as Gaussian Process Classification,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, June 17–24, 2023(IEEE Press, 2023), pp. 19488–19497.
    doi 10.1109/CVPR52729.2023.01867
  34. L. Ke, M. Ye, M. Danelljan, et al., “Segment Anything in High Quality,” Advances in Neural Information Processing Systems 36. 29914–29934 (2023).
    doi 10.52202/075280-1303
  35. C. Zhang, D. Han, Y. Qiao, et al., “Faster Segment Anything: Towards Lightweight SAM for Mobile Applications,” arXiv preprint arXiv:2306.14289 (2023).
    doi 10.48550/arXiv.2306.14289
  36. X. Ma, Y. Zhou, X. Xu, et al., “Towards Layer-Wise Image Vectorization,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, June 18–24, 2022(IEEE Press, 2022), pp. 16293–16302.
    doi 10.1109/CVPR52688.2022.01583
  37. D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980 (2014).
    doi 10.48550/arXiv.1412.6980
  38. L. R. Dice, “Measures of the Amount of Ecologic Association between Species,” Ecology 26 (3), 297–302 (1945).
    doi 10.2307/1932409