ETD

• gradient flow
• interacting particle system
• transformer

This thesis investigates the asymptotic behavior of particle dynamics inspired by Transformer architectures, combining theoretical results with computational observations. The main goal is to develop a rigorous understanding of the mechanisms underlying the emergence of clustering phenomena and the conditions under which the system evolves towards specific equilibrium configurations. We first establish a rigorous formulation of the dynamics as a constrained gradient flow on the product of spheres. Using tools from differential geometry, we characterize the gradient flow nature of the system and derived necessary and sufficient conditions for this structure to hold. We prove that, under suitable assumptions, the particles converge towards a single-cluster configuration. This extends and generalizes the results obtained in previous literature, showing that such clustering behavior is a robust property of the underlying gradient structure. We also discussed how the introduction of additional weights in the interaction energy modifies the energy landscape, creating multiple local minima and allowing for richer asymptotic behaviors, such as the emergence of two-cluster states.