HeuCI Usability Violation Dataset: Benchmark for Automated Heuristic Evaluation in CI/CD Pipelines

Key Points

• The goal is to create a comprehensive dataset for evaluating the effectiveness of usability heuristic evaluation tools in CI/CD pipelines.
• Compiled 5,997 annotated code samples from various sources.
• Included synthetic violations, real-world violations, and negative samples.
• Categorized annotations by violation type, severity, and heuristic category.
• Dataset features 3,200 synthetic violations and 1,847 real-world violations.
• Annotations provide insights into usability issues encountered in open-source projects.
• Facilitates benchmarking of automated evaluation tools for enhanced usability detection.