• Embedding space
• Interpretability
• Explainability
• Machine Learning
• NLP
• Neural Language Models
• Probing tasks

State-of-the-art neural language models (NLMs), characterized by intricate layers and millions, if not billions, of parameters, are often deemed "black boxes" due to their elusive interpretability. In the context of enhancing the understandability of models, our research investigates the nuanced effects of both pretraining and fine-tuning on linguistic representations. Employing linguistic probing tasks on a comprehensive sentence dataset, our analysis dissects the dynamic shifts in linguistic knowledge embedded within the model at different phases, shedding light on the intricacies of these transformations. Our study further delves into the interplay between the perplexity metric and the model's linguistic prediction errors. Additionally, we scrutinize the consequences of fine-tuning on linguistic nuances within the representations, exploring the prospect of any discernible loss. Furthermore, we explore whether the linguistic knowledge encapsulated within these representations serves as a predictive factor for the model's accuracy in downstream tasks that extend beyond linguistic objectives. In conclusion, our inquiry extends to the geometric properties of the representation space, probing into the intricate details of the degeneration phenomenon within the utilized sentence dataset across all phases of our investigation.