• Forward-Forward for Graphs.
• FFA
• FF
• forward forward algorithm
• learning on graphs
• regularization
• entropy
• greedy layer
• classification
• neural networks
• complex data
• machine learning

In the last decade, deep learning has achieved remarkable success in a wide range of learning tasks, with increasingly larger datasets being used to train increasingly larger deep neural networks via end-to-end (E2E) backpropagation. However, training deep neural networks via E2E backpropagation presents challenges, such as the exploding/vanishing gradient problem, that can lead to slow convergence or even complete training failures. Moreover, deep training demands high amounts of computational resources, which can be limiting in resource-constrained scenarios. These factors prompted the research for alternative methods.

An alternative to E2E training is represented by greedy layer-wise (GLW) approaches. In GLW methods, the network is constructed by training one layer at a time, adding the next layer only after the previous one has been trained and frozen, avoiding deep backpropagation. However, this approach may limit the effectiveness of hidden representations since the hidden layers may not be optimally trained to effectively transfer information to subsequent layers.

The Forward-Forward Algorithm (FFA), introduced by Hinton in 2022, is a recent GLW approach that has shown promise in early experiments. In this thesis, we propose a novel approach to enhance the training of FFA layers by introducing an entropy-based regularization term in the loss function, aiming to maximize the information content within hidden layer representations. With our method, each layer is trained not only to solve the prediction task at hand, but also to transfer a high amount of useful information to subsequent layers, discouraging correlations among units in favor of richer hidden representations.

The research consists of three main parts: (i) We examine the challenge of estimating entropy from a dataset, in order to identify a method for defining the entropy regularization term in the loss function. Specifically, we evaluate selected algorithms by conducting experiments on several synthetic datasets to understand their accuracy and computational cost. (ii) We conduct experiments on vectorial datasets to assess the effectiveness of entropy regularization on the FFA, exploring different definitions of the entropy regularization term. (iii) We investigate the performance of the FFA approach both with and without entropy regularization on more complex data structures, focusing on graph data. In this context, the depth of a model plays a more crucial role compared to the vectorial case. The layers construct node embeddings by performing neighborhood-based processing. As a result, the depth of a layer is associated with the spatial scale of graph analysis, with early layers focusing on small-scale analysis and deeper layers on larger-scale one. We evaluate the performance of two models: Graph Forward-Foward (GFF), introduced by Paliotta et al. in 2023, and our own novel model, referred to as Forward-Forward for Graphs (FF4G).

Experimental results show that entropy regularization consistently improves the accuracy of FFA and GFF. Moreover, FF4G surpasses the accuracy of several E2E trained models on the considered benchmarks.