ImmUQBench: A Benchmark on Uncertainty Quantification of Protein Immunogenicity Prediction

Abstract Objectives Discovering antigen proteins, capable of eliciting desired immune responses, is of paramount importance in developing immunogenic therapeutics for combating various diseases, particularly autoimmune disorders, infectious diseases, as well as cancers. Despite recent advances in artificial intelligence (AI) and machine learning (ML), accurate and generalizable immunogenicity prediction remains challenging due to limited labeled data and model over-simplifications. Uncertainty Quantification (UQ) approaches are commonly used to address the aforementioned challenges when applying AI/ML methods with limited training data, aiming to reduce the risk of catastrophic errors. This study aims to systematically evaluate the performance of UQ methods for antigen immunogenicity prediction and to establish a benchmark for assessing model reliability in data-scarce setting. Methods We here present ImmUQBench, a comprehensive benchmark that compares several well-known UQ methods for antigen immunogenicity prediction tasks. The benchmark assesses models in terms of predictive accuracy, calibration, and robustness under both in and out of distribution settings, providing standardized evaluation framework. Results Our evaluation reveals that different UQ strategies exhibit varying capabilities in capturing predictive uncertainty and maintaining robustness. This work yields critical insights into the performance and reliability of various UQ methods when applied to immunogenicity data, helping to identify which methods offer the most trustworthy predictions. Conclusion ImmUQBench provides a unified platform for assessing UQ approaches in immunogenicity prediction, facilitating the development of more trustworthy AI/ML models for therapeutic antigen design. By offering insights into the strengths and limitations of existing UQ methods, our work facilitates more effective and reliable immunogenic therapeutic discovery.