ETD

• continual learning
• knowledge distillation

This master thesis explores the application of knowledge distillation in mitigating catastrophic forgetting in a continual learning setting. Continual learning is a sub-field of machine learning in which a model is trained on a sequence of tasks, trying to avoid loss of performance as the training proceeds. However, this type of training leads to catastrophic forgetting. Here, a knowledge distillation approach is used to mitigate the forgetting phenomena, proposing a model architecture that learns from a teacher model while approaching new tasks. This method is applied to three different experimental settings: In the first the teacher if pre-trained on the full dataset (Join CIFAR-100), while in the second and third both teacher and student are trained on Split-CIFAR100.