ETD

• Data Quality Assessment
• Diagnostic Reporting
• ETSI Data Quality Metrics
• Heterogeneous Structured Data
• IoT & Sensor data Quality

Data quality assessment is challenging when data come from heterogeneous sources with different formats, schemas, timestamp conventions, metadata fields, and measurement layouts. This is especially relevant in sensor-oriented environments, where repeated observations make manual inspection difficult to sustain. It also aligns with the 2026 Annual Union Work Programme for European Standardisation, which identifies quality data for AI and innovation as a standardisation priority. In such settings, poor-quality sensor or IoT data can affect downstream AI processing and contribute to unreliable models when sources are incomplete, inconsistent, delayed, or inaccurate. Quality metrics therefore cannot be applied reliably unless the data are first interpreted, transformed, and standardized.

This thesis designs, implements, and evaluates an automated data quality assessment framework for heterogeneous data inputs. The framework uses a source-aware workflow and a canonical long-format typed representation to convert raw structures into comparable observation-level records while preserving source-specific metadata for timestamp interpretation, consistency checks, traceability, diagnostic reporting, and referencebased comparison where applicable.

The framework operationalizes selected ETSI-based data quality dimensions: completeness, redundancy, uniqueness, timeliness, consistency, reliability, and accuracy. These dimensions are implemented as modular metric components and reported across appropriate scopes. The evaluation, based on sensor-oriented JSON data associated with the Livorno port research context and supplementary tabular inputs, shows that the framework can transform heterogeneous raw data, compute selected quality metrics, and identify both aggregate quality patterns and source-specific issues.

Beyond metric computation, the framework generates summaries, issue logs, causeanalysis outputs, and temporal summaries that link quality problems to the affected source, file, variable, timestamp, rule, or diagnostic condition where available. The thesis contributes a reproducible, source-aware, and interpretable workflow that connects heterogeneous raw inputs to canonical typed data, selected ETSI-based metric computation, and traceable diagnostic evidence.