MILU: a consensus ensemble benchmark for multimodal medical imaging lecture understanding

Key Points

• This research aims to establish a benchmark for assessing the understanding of scientific lecture slides across multiple formats.
• Developed a comprehensive benchmark for evaluating structured understanding
• Assessed formatted reliability and semantic consistency of various models
• Identified gaps in current modeling approaches for lecture interpretation
• Current models perform well in formatting but poorly in semantic consistency
• Benchmark lays groundwork for future expert-annotated evaluations
• Highlights need for improved techniques in scientific lecture interpretation