• Distributed Deep Learning

This thesis aims to study distributed deep learning by focusing on Vision Transformers, with the goal of understanding how different parallelization strategies affect the training of large-scale models. It investigates the impact of system parameters such as batch size, model size, and interconnects, using experimental evaluation to assess scalability, speedup, and efficiency.