ETD

• benchmark
• chatbot
• fine-tuning
• Large Language Models (LLMs)
• SoBigData academy

This thesis presents the design, implementation, and evaluation of a domain-specific chatbot for the SoBigData Academy, based on open-source Large Language Models (LLMs). The work combines a theoretical overview of LLM architectures and fine-tuning strategies with a data-driven approach, including the creation of a custom dataset of question–answer pairs derived from all Academy courses. A comparative experimental evaluation was carried out on three families of ~3B parameter models (Ministral, Phi-3, Qwen2.5), tested in their base form and after supervised fine-tuning on both training and complete datasets. Metrics such as BLEU, METEOR, BERTScore, and Exact Match were used, alongside a mini grid search with 12 configurations to optimize training setups. Results highlight consistent improvements after fine-tuning, with Qwen2.5-3B emerging as the most robust and generalizable model. The study also identifies critical challenges—limited hardware resources, dataset heterogeneity, and library incompatibilities—and proposes future directions, including integration with APIs, adoption of Retrieval-Augmented Generation (RAG), and multimodal extensions.