The full version of this paper is accepted to the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025). A software issue is typically an online report that describes a bug or a feature request, occasionally including details like stack traces or expected/observed behavior Be08. In most cases, developers address these issues through a set of code changes (also known as a patch). In this context, an issue-reproducing test is a test accompanying the patch that fails on the unpatched code (validating the presence of the issue) and passes on the patched code (validating its resolution). An issue-reproducing test increases confidence that the issue (1) can be replicated and (2) will not be reintroduced in the future. Writing issue-reproducing tests is a demanding and time-consuming task LIH17 that is often overlooked by developers LY17, thus recent work has investigated ways to automatically generate these tests to support developers. In particular, recent techniques have proposed the use of Large Language Models (LLMs), either in a zero-shot setting Ah24 or in a multi-step workflow where the output of the previous LLM call is used as input to the next one Ah24; Na25. However, relying solely on LLMs for the generation of tests is prone to hallucinations EP24, leading for example to tests trying to import non-existent modules or use non-existent methods. In addition, recent work does not yet leverage information that could be useful for the generation of the issue-reproducing test such as the focal context or the structure of existing tests for the same class, which could hint the test setup or the available mocks.
Kitsios et al. (Thu,) studied this question.