ETD

• distributed network control system
• remote reinforcemnet learning

Reinforcement learning performs remarkably well in simulation. Agents act, environments respond, and the feedback loop remains tightly synchronized. Real-world networked systems, however, rarely operate under such ideal conditions. Observations may arrive late, actions can be delayed in transit, and the assumptions underlying standard RL formulations begin to break down. This thesis investigates the consequences of taking that reality seriously.
To study the effects of network latency in a controlled yet realistic setting, I developed a distributed reinforcement learning framework in which a Cart-Pole environment and its controller execute on separate machines, with configurable network delays introduced between them. While simple in design, this setup provides a practical platform for examining how latency affects learning and control when communication constraints cannot be ignored.
The first part of the work focuses on state-based control. Using a PPO agent trained under delayed observations, I evaluate the extent to which conventional reinforcement learning methods can tolerate communication latency. The results show that, with appropriate design choices, state-based agents remain surprisingly robust under moderate delays and can operate effectively in asynchronous settings without requiring strict synchronization.
The second part of the thesis addresses a more challenging scenario: learning directly from visual observations. I investigate several approaches, including a CNN-based PPO baseline, LSTM-augmented policies that incorporate temporal context, and a predictive latent-state model designed to anticipate future observations and compensate for communication delays. In addition, I implement a split-inference architecture that transmits compact latent embeddings rather than raw image frames, significantly reducing communication overhead in bandwidth-constrained environments.
The comparison between state-based and vision-based control revealed a more nuanced picture than initially expected. Vision-based agents introduced additional computational overhead and were more sensitive to latency during development. However, by leveraging GPU-accelerated visual encoding and efficient feature extraction, much of this performance gap could be mitigated. Under moderate network delays, vision-based policies achieved performance comparable to state-based approaches, demonstrating that robust visual control remains feasible even in distributed settings. These results suggest that the primary challenge lies not only in network latency itself, but also in the interaction between communication delays and the computational cost of perception.
Overall, the findings show that latency-awareness cannot be treated as an afterthought in distributed reinforcement learning systems. Communication delays, perception latency, bandwidth limitations, and computational resource allocation interact closely and must be considered together during system design. Efficient encoding strategies, predictive representations, and appropriate hardware acceleration can substantially improve the practicality of deploying reinforcement learning agents over real networks. By systematically examining these challenges across both state-based and vision-based settings, this thesis provides practical insights and design guidelines for building reinforcement learning systems that remain effective under realistic communication constraints.