Objectives/Goals: To validate 20 culturally tailored clinical vignettes for a translational study assessing ChatGPT’s diagnostic support in adolescent and young adult mental health. The validation ensures methodological, ethical, and cultural rigor to examine ChatGPT’s potential in collaborative clinical decision-making. Methods/Study Population: Twelve licensed clinical psychologists evaluated 50 culturally adapted vignettes using a nine-item instrument assessing diagnostic agreement (Fleiss κ), content validity (I-CVI, S-CVI/Ave), comorbidity sensitivity, difficulty, ethical sensitivity, and perceived risk. Qualitative feedback refined clarity and realism. Vignettes meeting κ ≥ 0.60, S-CVI ≥ 0.80, and ethics ≥ 3 were retained, ensuring one easy, two moderate, and one difficult case per condition. The resulting 20 validated cases will serve as standardized stimuli to assess ChatGPT’s diagnostic accuracy and its ability to collaborate effectively with clinical psychologists. Results/Anticipated Results: High inter-rater reliability (κ ≥ 0.60) and strong content validity (S-CVI ≥ 0.85) are expected. Balanced difficulty and accurate risk detection will confirm calibration. Qualitative feedback will enhance realism and cultural appropriateness. Validated vignettes will enable rigorous assessment of ChatGPT’s diagnostic accuracy and clinical decision-support potential within a translational research framework. Vignette cases will be presented to three groups: 1) ChatGPT app, 2) Clinical mental health care providers, and 3) Clinical mental health care providers collaborating with ChatGPT. Each group will provide a diagnosis and confidence level to compare accuracy, sensitivity, and collaborative value across human-AI and human-only conditions. Discussion/Significance of Impact: This validation provides a rigorous foundation for evaluating ChatGPT’s translational role in mental health diagnostics. Psychometrically and ethically sound vignettes strengthen evidence-based integration of AI tools into precision mental health care and promote responsible human-AI collaboration.
Rosa et al. (Wed,) studied this question.