• Transformer
• attention
• online HTR
• NLP
• explainability

In this work, the Transformer is shown to provide a framework as an end-to-end model for building mathematical expression trees or solving recognition tasks related to natural language, starting from online handwritten gestures corresponding to glyph strokes.
In particular, the attention was successfully employed to learn and enforce the underlying syntax of the target grammars, independently from the specific natural language or mathematical formalism.
The encoder is fed with tokens of spatio-temporal data forming an infinitely large input vocabulary, introducing a technique that could find application beyond that of recognition.
New strategies are investigated to optimize the architecture for tasks involving spatio-temporal dependencies, focusing on the use of pre-encoding layers and different input representations. Furthermore, the transfer learning capabilities of the encoder are analyzed to assess its segmentation and recognition power, even when transferred to new domains with no fine-tuning or adaptation. Additionally, the ability to generate representations suitable for several target syntax and semantics is evaluated. Finally, ablation studies are conducted to test robustness against incorrect inputs, together with an explainability analysis comprising novel techniques here introduced.
Our model is suitable for real-time edge inference, achieving 94% of accuracy over expression tree building and 97% for online handwritten text recognition, relying only on 2M parameters.