Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Models

Adrien Le Coz (1, 2) , Houssem Ouertatani (1, 3) , Stéphane Herbin (2) , Faouzi Adjed (1)

1 IRT SystemX
2 DTIS, ONERA, Université Paris Saclay [Palaiseau]
3 Inria Lille - Nord Europe

Image classifiers should be used with caution in the real world. Performance evaluated on a validation set may not reflect performance in the real world. In particular, classifiers may perform well for conditions that are frequently encountered during training, but poorly for other infrequent conditions. In this study, we hypothesize that recent advances in text-to-image generative models make them valuable for benchmarking computer vision models such as image classifiers: they can generate images conditioned by textual prompts that cause classifier failures, allowing failure conditions to be described with textual attributes. However, their generation cost becomes an issue when a large number of synthetic images need to be generated, which is the case when many different attribute combinations need to be tested. We propose an image classifier benchmarking method as an iterative process that alternates image generation, classifier evaluation, and attribute selection. This method efficiently explores the attributes that ultimately lead to poor behavior detection.

Keywords: Generative AI; Bayesian optimization; Computer vision

Domains: Computer Vision and Pattern Recognition [cs.CV]; Artificial Intelligence [cs.AI]; Machine Learning [cs.LG]

Adrien Le Coz, Houssem Ouertatani, Stéphane Herbin, Faouzi Adjed. Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Models. Generative Models for Computer Vision - CVPR 2024 Workshop, Jun 2024, Seattle, United States. ⟨hal-04549384v2⟩