Tesi etd-09272024-182413 |
Link copiato negli appunti
Tipo di tesi
Tesi di dottorato di ricerca
Autore
TORTORELLA, DOMENICO
URN
etd-09272024-182413
Titolo
Efficient Models for Deep Learning on Graphs
Settore scientifico disciplinare
INF/01
Corso di studi
INFORMATICA
Relatori
tutor Prof. Micheli, Alessio
Parole chiave
- constructive neural networks
- deep learning
- graph convolutional networks
- graph neural networks
- graphs
- machine learning
- reservoir computing
Data inizio appello
08/10/2024
Consultabilità
Non consultabile
Data di rilascio
08/10/2027
Riassunto
Graphs are an effective representation of entities and relations of many different objects such as chemical compounds, protein interaction networks, social or citation networks, and other complex structured data.
Deep learning proved effective in tackling a wide range of learning tasks, including tasks on structured data, thanks to the ability of learning a hierarchy of representations in the several neural network layers without requiring human engineering.
However, deep learning brought several challenges, both in terms of efficiency and effectiveness.
The increase in available data --- both as large collections of graphs and as large-scale network graphs --- requires an increase in model complexity, which in turn leads to an exponential increase in hyper-parameter space to be searched during model selection; the need for efficiency both in training models and selecting models is evident.
Training effective deep neural networks is also challenging, due to difficulties encountered e.g. by gradient back-propagation through a large number of layers in end-to-end training.
In graph learning, these problems are exacerbated by other peculiar issues such as over-smoothing and over-squashing, which impair the ability of deep neural networks for graphs to learn meaningful representations.
This research has investigated two directions for building efficient and effective graph neural models: constructive approaches, which incrementally build a graph neural model, adapting its size to the task; reservoir computing approaches, which exploit random parameter initialization satisfying certain constraints to realize graph embeddings without training.
Within the first approach, a model for graph classification that automatically selects width and depth of the neural network during the incremental training is developed.
The incremental construction algorithm allows to obtain leaner and deeper neural networks for graphs without loss of accuracy compared to end-to-end trained models, with a gain in computational efficiency of up to two orders of magnitude with respect to explicit architecture selection.
In the reservoir computing (RC) research line, the theoretical understanding of the constraints for the random parameter initialization has been improved.
RC for graphs is extended to node-level tasks, demonstrating a particular effectiveness in addressing heterophilic node classification tasks.
An analysis of over-smoothing in terms of spectral centroids and entropy of node representations has been carried on, showing the importance of the role played by the Lipschtiz constant of message-passing layers.
The same tools have allowed the investigation of several dense and sparse designs of graph reservoirs for node classification.
As RC models allow to investigate the inductive bias decoupled from the training process, these have been used to evaluate graph rewiring as a mean to relieve issues connected to over-squashing.
Finally, a reservoir model for temporal graphs (DynGESN) has been developed, providing a better efficiency/accuracy trade-off with respect to both temporal graph kernels and temporal graph networks on sequence classification and next-step prediction tasks.
Deep learning proved effective in tackling a wide range of learning tasks, including tasks on structured data, thanks to the ability of learning a hierarchy of representations in the several neural network layers without requiring human engineering.
However, deep learning brought several challenges, both in terms of efficiency and effectiveness.
The increase in available data --- both as large collections of graphs and as large-scale network graphs --- requires an increase in model complexity, which in turn leads to an exponential increase in hyper-parameter space to be searched during model selection; the need for efficiency both in training models and selecting models is evident.
Training effective deep neural networks is also challenging, due to difficulties encountered e.g. by gradient back-propagation through a large number of layers in end-to-end training.
In graph learning, these problems are exacerbated by other peculiar issues such as over-smoothing and over-squashing, which impair the ability of deep neural networks for graphs to learn meaningful representations.
This research has investigated two directions for building efficient and effective graph neural models: constructive approaches, which incrementally build a graph neural model, adapting its size to the task; reservoir computing approaches, which exploit random parameter initialization satisfying certain constraints to realize graph embeddings without training.
Within the first approach, a model for graph classification that automatically selects width and depth of the neural network during the incremental training is developed.
The incremental construction algorithm allows to obtain leaner and deeper neural networks for graphs without loss of accuracy compared to end-to-end trained models, with a gain in computational efficiency of up to two orders of magnitude with respect to explicit architecture selection.
In the reservoir computing (RC) research line, the theoretical understanding of the constraints for the random parameter initialization has been improved.
RC for graphs is extended to node-level tasks, demonstrating a particular effectiveness in addressing heterophilic node classification tasks.
An analysis of over-smoothing in terms of spectral centroids and entropy of node representations has been carried on, showing the importance of the role played by the Lipschtiz constant of message-passing layers.
The same tools have allowed the investigation of several dense and sparse designs of graph reservoirs for node classification.
As RC models allow to investigate the inductive bias decoupled from the training process, these have been used to evaluate graph rewiring as a mean to relieve issues connected to over-squashing.
Finally, a reservoir model for temporal graphs (DynGESN) has been developed, providing a better efficiency/accuracy trade-off with respect to both temporal graph kernels and temporal graph networks on sequence classification and next-step prediction tasks.
File
Nome file | Dimensione |
---|---|
La tesi non è consultabile. |