• Zero Touch Network
• Kafka
• Telegraf
• Optical Network
• Telemetry
• Scalable

In the context of Zero Touch Networks, collecting metrics is a key component through which the network is able to autonomously apply decisions and optimize itself. The closed loop iteration time, which is the time to move from raw data to high level decisions, critically affects its efficacy. The Data collection module, which oversees gathering metrics from the available resources in the ZTN, should be able to guarantee a very low latency communication with other management functional blocks, providing good scalability and reliability.
This work proposes a Kafka-based monitoring architecture which fully meets the Data Collection module's requirements by exploiting the built-in functionalities provided by Kafka. The architecture has been used to continuously monitor an optical network scenario, using lightweight agents, and to distribute the gathered data to different consumers in a publish-subscribe manner. It also offers preprocessing capabilities by integrating Kafka Streams framework, which has been used to dynamically filter the metrics related to each active lightpath. Results show that the architecture provides effective performance, supporting more than 4000 metrics per second, with a very low end-to-end message latency of 50ms. The system is able to offer a continuous availability, even in case of failures. Moreover, the agents, enabled on the specific resources, provide a persistent and efficient monitoring data stream, with very low CPU and memory consumption.