Evaluating automatic label noise detection in 3D segmentation with realistic label noise

Automatic label correction methods have been developed to address issues that come with a growing volume of annotated data in Machine Learning pipelines. The majority of these methods have been primarily evaluated on artificial noise in Euclidean data. Yet, particularly the subject of Geometric Deep Learning could benefit from well-performing label correction algorithms since non-Euclidean data, such as curved surfaces in 3D, is especially cumbersome and error prone to annotate. To assess the current applicability of existing label correction methods to 3D data, we evaluate automatic noise detection methods using realistic and artificial label noise in the 3D domain. We not only evaluate their effectiveness in 3D, but also examine general correction behavior regarding properties like carefulness and aggressiveness and compare those between both noise types. Our investigation shows that finding realistic noise in a 3D segmentation problem is of different nature compared to finding artificial noise. • We evaluate automatic label detection methods with realistic label noise in 3D. • Automatic label correction methods make different suggestions than human experts. • Automatic label correction methods that carefully select a smaller subset of corrections that would be suggested by a human too perform better than over-correcting, aggressive methods, that propose a lot of corrections a human would not suggest. • While well selected corrections positively affect the generalization performance in a downstream task, the opposite is not necessarily true: A good downstream task performance does not imply a good selection of corrections. • Automatic label detection methods are mostly evaluated with artificial label noise. • Finding artificial noise is an easier task compared to finding real label noise in a 3D-setting.