From issue titles to requirements: an empirical study of large language models and prompt engineering strategies

Abstract This study empirically assesses how effectively two Large Language Models (LLMs), and, can transform terse feature-request titles from open-source software (OSS) issue trackers into well-formed software requirements. It further examines how prompt-engineering strategies shape requirement quality and evaluates a scalable “LLM-as-a-Judge” approach for automated quality assessment based on three ISO/IEC/IEEE 29148: 2018 quality attributes: Unambiguity, Verifiability, and Singularity. We extract 150 feature-request titles from five OSS repositories and pair each title with every combination of two LLMs and three prompt styles, producing 900 candidate requirements. An independent evaluator LLM rates each requirement on these three quality attributes using a Likert scale and provides textual rationales. Ordinal data are analyzed with descriptive statistics and non-parametric tests, complemented by thematic analysis of the evaluator’s explanations. A targeted human validation study with five evaluators on a stratified sample of 50 requirements assesses the reliability of the automated judge. LLMs often produce high-quality requirements; however, scores vary with input clarity and prompt design. Few-shot prompting consistently boosts Singularity, while Expert Identity prompting sometimes raises Verifiability but frequently harms Singularity. shows a modest but significant edge in Singularity. Qualitative review echoes these patterns, revealing trade-offs between added detail and focus. The LLM-as-a-Judge protocol delivers consistent, scalable evaluations whose scores correlate significantly with aggregate human judgment (p < 0. 05 for all three attributes). Modern LLMs can expedite the drafting of initial software requirements from informal OSS inputs, but their output quality hinges on careful prompt selection and the inherent clarity of the source title. Prompt effects are model-dependent and may introduce trade-offs among quality attributes, so human oversight remains indispensable for refinement. The LLM-as-a-Judge framework proves a practical, human-validated technique for large-scale evaluation, enabling rapid, reproducible insights into LLM-driven requirements engineering workflows.