• Explainable AI
• LLM
• Medical
• RAG
• Retrieval Augmented Generation
• XAI

The following study explores the integration of Large Language Models (LLMs) into the
medical domain by combining Explainable AI (XAI) techniques with Retrieval Augmented
Generation (RAG) to deliver reliable and personalized responses that account for the user’s cognitive
abilities and background knowledge. One of the key components of the system is the MedRAG
toolkit, a Retrieval-Augmented Generation (RAG) framework designed for the medical domain, used
for improving response quality by integrating information retrieved from medical documents, which
has been extended to consider patient clinical notes as contextual input. This allows the model to
provide both predictions and detailed explanations that are based on a combination of patient specific information and external medical knowledge retrieved through the RAG mechanism. The
system was validated using the MIMIC QA datasets, a subset of the MIMIC III dataset which contains
clinical notes and corresponding reference diagnoses validated by human medical experts, and
further tested on the PIMA dataset to predict diabetes. Machine learning models such as Random
Forest, Decision Tree, CatBoost, and Neural Network were trained, with explanations generated using SHAP and LIME for both local and global interpretability. The system was evaluated using
metrics such as BERT Score, BLEU and ROUGE, demonstrating that combining RAG with XAI
improves the model’s performance in providing accurate and understandable medical predictions.
This study also highlights that traditional XAI methods may not be sufficient in complex domains like
healthcare, where the added context from RAG and LLMs offers significant advantages for
interpretability and diagnostic support.