logo SBA

ETD

Digital archive of theses discussed at the University of Pisa

 

Thesis etd-11212023-165932


Thesis type
Tesi di laurea magistrale
Author
DOMENICHELLI, LUCIA
URN
etd-11212023-165932
Thesis title
On the Evolution of a Neural Language Model: emergence and organization of language skills and their impact on its abilities.
Department
FISICA
Course of study
FISICA
Supervisors
relatore Prof. Dell'Orletta, Felice
tutor Mannella, Riccardo
Keywords
  • Embedding space
  • Explainability
  • Interpretability
  • Machine Learning
  • Neural Language Models
  • NLP
  • Probing tasks
Graduation session start date
11/12/2023
Availability
Full
Summary
State-of-the-art neural language models (NLMs), characterized by intricate layers and millions, if not billions, of parameters, are often deemed "black boxes" due to their elusive interpretability. In the context of enhancing the understandability of models, our research investigates the nuanced effects of both pretraining and fine-tuning on linguistic representations. Employing linguistic probing tasks on a comprehensive sentence dataset, our analysis dissects the dynamic shifts in linguistic knowledge embedded within the model at different phases, shedding light on the intricacies of these transformations. Our study further delves into the interplay between the perplexity metric and the model's linguistic prediction errors. Additionally, we scrutinize the consequences of fine-tuning on linguistic nuances within the representations, exploring the prospect of any discernible loss. Furthermore, we explore whether the linguistic knowledge encapsulated within these representations serves as a predictive factor for the model's accuracy in downstream tasks that extend beyond linguistic objectives. In conclusion, our inquiry extends to the geometric properties of the representation space, probing into the intricate details of the degeneration phenomenon within the utilized sentence dataset across all phases of our investigation.
File