The Measurement Crisis: Saturation, Goodhart's Law, and the End of AI Leaderboards

Key Points

• The aim is to explore the impact of Goodhart's Law and saturation on AI benchmark systems.
• Literature review of existing AI performance metrics
• Analysis of the impact of saturation on measurement validity
• Discussion of implications of Goodhart's Law in AI systems.
• Identified that performance metrics lose their validity once they become targets
• Demonstrated that saturation leads to diminishing returns in AI benchmarking
• Highlighted the need for new measurement approaches to advance AI research.