ETD

• automated test generation
• Kubernetes
• large language models
• microservices
• OpenAPI
• prompt engineering
• system testing

This thesis experiments with and evaluates an LLM-based approach to support the automated generation of system test suites for microservice applications deployed on Kubernetes, starting from their OpenAPI specifications. The work is guided by the following research questions:

- RQ1. To what extent can a prompt based approach support the automated generation of system tests for microservice applications deployed on Kubernetes, starting from the relative OpenAPI specifications?
- RQ2. What is the impact of integrating system testing principles into the quality of the generated system test suites?
- RQ3. How well does the optimized prompt generalize to an application not used during the prompt validation phase?
- RQ4. How sensitive is the quality of the generated system test suites to variations in the model temperature?

To answer these questions, the main contribution lies in the method used to engineer and validate the LLM prompt: a leave-one-out approach over three heterogeneous microservices-based applications. Two of these compose the validation set (SockShop and KubePay), while the remaining one (Lakeside Mutual) is always held out and never used during prompt engineering, ensuring that the evaluation remains separated from the validation. If an OpenAPI specification under-described the service contract, the missing information was included in the specification supplied to the model (a description of the documented response surface and of the relevant endpoint behaviors). The validation phase consists of the following steps:

1) First, the prompt is refined until it reliably produces well-formed system tests, establishing a correct baseline before any quality improvement.
2) Next, the prompt is refined to improve the quality of the generated suites, while preserving at least the same level of well-formedness obtained from the previous step. Suites are compared across versions using two complementary measures: instruction coverage they reach on the application, and mutation score they obtain against a set of injected mutants, while preserving well-formedness obtained in the first step. The iterations over the validation set continued until no candidate edit based on a black-box system testing principle yielded further improvement.

To support experimentation, an automated testing system named WeaveSuite was designed and implemented. It operates within the Kubernetes cluster that hosts the application being tested, by first autonomously discovering the deployed microservices in the Kubernetes cluster and retrieving their OpenAPI specifications. Once an operator confirms that all the microservices have been discovered and that the specifications are complete enough for black-box generation, this verified data is supplied to a Large Language Model (Google Gemini) to generate a suite of system tests. After a human review of the generated suite, system tests are then executed, and the system statically calculates endpoint coverage. The human in the loop, therefore, ensures that the condition the automation depends on hold (complete OpenAPI specifications), while everything else is automated.

Only after the validation phase concluded, the held-out application (untouched throughout prompt engineering) was used to assess the selected prompt on an unseen application. In order to provide a direct comparison, the selected prompt was executed twenty times on each reference application of the validation set, achieving a mean instruction coverage of 60.19% with a mean mutation score of 74.07% on KubePay, and 69.69% with 62.16% on SockShop. The evaluation on the held-out application proved that the prompt generalizes beyond the validation set: each generated suite of the twenty runs was well-formed, achieving a mean instruction coverage of 44.39% and a mean mutation score of 65.16%. An analysis of the residual mutants proved that only 25 out of 32 Lakeside Mutual mutants could be killed from system tests using specifications: the mutation score normalized becomes 83.40%, which is comparable to the normalized one obtained on the validation set (71.87% on SockShop and 83.33% on KubePay). The prompt, which was engineered only on the validation set, produced well-formed suites in every run on an unseen, larger, and more complex application.