We proposed a continual policy distillation approach which integrates knowledge from Reinforcement Learning-based control policies for In-Hand manipulation of various objects with a soft robotic hand. It learns control policies for manipulating new objects in a sequential manner without forgetting the old ones.