• Bert
• Deep Graph Networks
• embeddings regression
• Graph Attention Network
• language modeling
• node-embeddings
• symbolic knowledge
• Transformers
• word-embeddings

In recent years, Deep Graph Networks (DGNs) have proven to be one of the state-of-art for representation learning for graphs. This thesis focuses on using Graph Attention Network embeddings to regularize the Transformers model, specifically BERT. The word-embeddings learned during BERT's Masked Language Modeling are regularized to combine symbolic knowledge of Knowledge Graphs. This work evaluates the quality of graph-regularized word-embeddings concerning the baseline through word-embeddings probing tasks.