Uncertainty modeling in multimodal speech analysis across the psychosis spectrum

Speech provides a rich behavioral signal of psychosis, yet its diagnostic use remains limited because speech patterns vary widely across individuals and contexts. We model this variability as uncertainty, capturing how consistently speech features indicate symptom expression. We introduce a multimodal model that integrates acoustic and linguistic information to predict symptom severity and psychosis-related traits across the spectrum, from high schizotypy to clinical psychosis. By estimating uncertainty for each modality, the model learns when to rely on specific signals, adapting to speech quality and task context to improve accuracy and interpretability. Using speech from 114 participants-32 with early psychosis and 82 with low or high schizotypy-recorded in German across structured and narrative tasks, the model achieved an F1-score of 83% (ECE = 0.045), demonstrating robust and well-calibrated performance. Uncertainty estimation further revealed which speech markers most reliably indicated symptoms, including pitch variability, fluency disruptions, and spectral instability.