ETD

• constrained policy optimization
• control barrier functions
• differentiable optimization
• safe reinforcement learning

Deep Reinforcement Learning enables complex robotic control, but lacks formal safety guarantees. A common remedy places a Control Barrier Function filter downstream of the policy, projecting nominal actions onto a safe set via a Quadratic Program. While guaranteeing instantaneous safety, the policy remains unaware of these interventions. Consequently, the agent learns to "fight the shield", permanently proposing inadmissible commands and delegating safety to external correction.
This work proposes a Filter-Aware Differentiable Lagrangian Proximal Policy Optimization architecture to close this loop. The filter's intervention on the policy mean is framed as an instantaneous constraint within a Constrained Markov Decision Process. A strictly positive tolerance restores a well-posed saddle point and keeps the Lagrange multiplier finite. The constraint gradient is derived analytically via implicit differentiation of the Quadratic Program's Karush-Kuhn-Tucker conditions, giving the policy the exact pathwise direction back into the admissible set, while a two-time-scale primal-dual scheme adapts the multiplier without manual calibration.
Experiments on simulated robots show this architecture suppresses filter interventions faster than blind shielding, achieving a robust constraint internalization that no fixed-weight reward penalty can reliably reproduce.