Validation of DATEX II

Message Class Model
&
Serialized Message
Jan Vlčinský, Andreas Dis, Edgar van Wilgenburg
NAPCORE SWG4.1 - Virtual DATEX User Forum - 2023-10-04

Motivation

How to use DATEX II supported validation to:

🚀 speed up 🚀

system design and development
deployment and operation
fixing problems

📈 improve 📈

quality of data exchange

NAPCORE SWG4.1 - DATEX II

napcore-logo.png
https://napcore.eu/

https://datex2.eu

co-funded-by-eu.png

Presenters

Jan Vlčinský

jan.vlcinsky@tamtamresearch.com

Executive Researcher at TamTam Research, Czechia

featuring

Validation Concepts and Context,
Local Validators
and Daily Validations of published data (InQMS)

Andreas Dis

Andreas.Dis@austriatech.at

Data Analyst at Austriatech, Austria

featuring

On-line class model validator at RAV Centre

Edgar van Wilgenburg

edgar.van.wilgenburg@ndw.nu

Information Architect at Nationaal Dataportaal Wegverkeer (NDW), The Netherlands

featuring

On-line serialized message validator at NDW

Topics

  • Fixing the World - faster
  • DATEX II message class model validation
  • Serialized DATEX II message validation
  • Questions & Answers

Highlights

DATEX II Context and Concepts for validation

Tools and demo:

on-line DATEX II RAV Center class model validator
local message validators
on-line message validator at NDW
InQMS: daily message validations of Czech content

—

Validate to fix things faster

Introduction to Validation

by Jan Vlčinský, TamTam Research

Validation Concept

concept.svg

DATEX II Context and Concepts

d2context.png

DATEX II message class model validation

by Andreas Dis, Austriatech

—

DATEX II RAV Center (on-line) class model validator

Scope: Message class model

d2context-class-validation.png

The DATEX II RAV Centre

RAV = Reference Advice and Validation https://testcenter.datex2.eu/xmlchk/

Currently validates: profiles against Commission’s Delegated Regulations (EU) reference profile on secure truck parking information (No 885/2013) in DATEX II Version 2.3.

RAV Demo

  • find the validator
  • upload the .SEL file
  • see the report
    • when some issues are detected
    • when all things are OK

Upload a .SEL file

upload.png

Get Validation Report with Errors

rpt_err.png

Get Validation Report (all good)

rpt_ok.png

Serialized DATEX II message validation

by Jan Vlčinský, TamTam Research

Use cases and value for publisher and consumer
Publishing a schema within an hour
JSON/XML/ASN.1 schema types
XML validators: local/desktop/on-line

Scope: Serialized Message

d2context-message-validation.png

XML/JSON/ASN.1 Schema Use Cases

Message validation

Code generation

Message Validation

  • Is the sample in specs valid?
  • Test case for message serializing code
  • Is the outgoing message valid?
  • Is the ingoing message valid?
  • Test case for message deserializing code

Code generation:

  • Server side code
  • Client side code
  • Class instance to message (serialization)
  • Message to class instance (deserialization)

Use Cases in System Lifecycle

use-cases.png

Importance of accessible schema

Lack of
relevant XML/JSON/ASN.1 Schema

☠ KILLS all the benefits ☠

—

This is where NAP plays the key role.

Publishing the Schema can be easy

  • Directory Content: 

      schema/schema.xsd # mission accomplished!!!
  samples/*.xml # nice to have
  FORMAT.yaml # metadata
  sel/fname.sel # optional
  • Publish on GitHub/GitLab:
  • GitLab bonus: 🌲 multi-level repo/profile groups

Validation of: JSON

There are multiple JSON Schema variants:

  • [X] JSON Schema Draft 4
  • [ ] JSON Schema Draft 5
  • [ ] JSON Schema Draft 6
  • [ ] JSON Schema Draft 7
  • [ ] JSON Draft 2019-09
  • [ ] JSON Draft 2020-12

—

DATEX II uses Draft 4

Validation of: XML

There are multiple XML Schema variants:

  • [X] W3C XML Schema v1.0
  • [ ] W3C XML Schema v1.1
  • [ ] RELAX NG (XML and Compact Syntax)
  • [ ] Schematron

—

DATEX II uses W3C XML Schema v1.0

Local XML Schema validators

💻 Desktop: Altova XMLSpy, oXygen XML Editor, Stylus Studio, Liquid XML Studio, XMLPad, Exchanger XML Editor
🎹 CLI: xmlstarlet, xmllint, Xerces-C++ XML Parser, MSV, Saxon
📜 Code: pytest test case using lxml lib; +∞

You will find one or more for sure

CLI Validators for XML

$ xmlstarlet val \
  --xsd schema/schema.xsd \
  samples/message.xml
$ xmllint \
  --schema schema/schema.xsd \
  samples/message.xml
$ java -jar msv.jar \
  schema/schema.xsd \
  samples/message.xml
$ java -jar saxon9he.jar \
  -s:samples/message.xml \
  -xsd:schema/schema.xsd \
  -val:strict
$ XercesCParser -v -s -n \
  samples/message.xml # schema referenced within message.xml

xmlstarlet CLI for XML

Based on libxml2 (as xmllint or python lxml library)

Runs on Windows, Linux, MacOS..:

  • natively installed
  • via docker
$ # validate single file:
$ xmlstarlet val --xsd schema/schema.xsd --err sample/message.xml
$ # validate multiple files
$ xmlstarlet val --xsd schema/schema.xsd sample/*.xml
$ # list only invalid files
$ xmlstarlet val --xsd schema/schema.xsd --list-bad sample/*.xml

xmlstarlet via docker (Linux)

Pull and tag the docker image (once) 

$ export IMAGE_URL=registry.gitlab.com/datex2/presentation/2023-virtduf-validation/xmlstarlet-docker:1.0.0
$ docker pull $IMAGE_URL # pull the image
$ docker image tag $IMAGE_URL xmlstarlet # and tag it e.g. as xmlstarlet

Use it for validation 

$ docker run -it --rm -v $PWD:/data xmlstarlet \
  val --err \
  --xsd schema/schema.xsd \
  samples/accident.xml
  samples/accident.xml - valid

https://gitlab.com/datex2/presentation/2023-virtduf-validation/xmlstarlet-docker

xmlstarlet via docker (Windows)

Pull and tag the docker image (once) 

C:\DATEX2>set IMAGE_URL=registry.gitlab.com/datex2/presentation/2023-virtduf-validation/xmlstarlet-docker:1.0.0
C:\DATEX2>docker pull %IMAGE_URL%
C:\DATEX2>docker image tag %IMAGE_URL% xmlstarlet

Use it for validation 

C:\DATEX2>docker run -it --rm -v %cd%:/data xmlstarlet ^
  val --err ^
  --xsd schema/schema.xsd ^
  samples/accident.xml

  samples/accident.xml - valid

https://gitlab.com/datex2/presentation/2023-virtduf-validation/xmlstarlet-docker

pytest test case example (Python)

Conformance to a schema is crucial

Let your code do the work

Detects errors before committing the code

Makes code refactoring real fun

Example for Python

https://gitlab.com/datex2/presentation/2023-virtduf-validation/pytest-validate

tests/test_validate.py: imports

from pathlib import Path

from lxml import etree

import pytest

tests/test_validate.py: messages to test 

@pytest.fixture(
    params=[
        "samples/accident.xml",
        "samples/brokendownvehicles.xml",
        "samples/restriction-bkom.xml",
        "samples/invalid_missig_lang.xml",
    ]
)
def message(request):
    """Parsed XML message (as lxml tree instance)"""
    path = Path(request.param)
    assert path.exists()
    xml_doc = etree.parse(str(path))
    return xml_doc

tests/test_validate.py: schema to use 

@pytest.fixture
def schema():
    """W3C XML Schema (lxml XMLSchema instance)"""
    path = Path("schema/schema.xsd")
    assert path.exists()
    xsd_doc = etree.parse(path)
    xml_schema = etree.XMLSchema(xsd_doc)
    return xml_schema

tests/test_validate.py: actual test 

def test_validate(schema, message):
    """Test that the `message` conforms to `schema`"""
    assert schema.validate(message), f"Invalid: {schema.error_log}"

run the pytest test suite

call the test 

$ pdm run pytest -sv tests

and see the test report: header 

============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 -- /home/javl/devel/datex2/gitlab/presentation/2023-virtduf-validation/pytest-validate/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/javl/devel/datex2/gitlab/presentation/2023-virtduf-validation/pytest-validate
collecting ... collected 4 items
...

test report: Short summary test results 

...
collecting ... collected 4 items
tests/test_validate.py::test_validate[samples/accident.xml] PASSED
tests/test_validate.py::test_validate[samples/brokendownvehicles.xml] PASSED
tests/test_validate.py::test_validate[samples/restriction-bkom.xml] PASSED
tests/test_validate.py::test_validate[samples/invalid_missig_lang.xml] FAILED

=================================== FAILURES ===================================

test report: Detailed failure description 

...
=================================== FAILURES ===================================
________________ test_validate[samples/invalid_missig_lang.xml] ________________

schema = <lxml.etree.XMLSchema object at 0x7ffb1e93f9c0>
message = <lxml.etree._ElementTree object at 0x7ffb1e995740>

    def test_validate(schema, message):
>       assert schema.validate(message), f"Invalid: {schema.error_log}"
E       AssertionError: Invalid: samples/invalid_missig_lang.xml:11:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_4: Element '{http://datex2.eu/schema/2/2_0}payloadPublication': The attribute 'lang' is required but missing.
E       assert False
E        +  where False = <bound method _Validator.validate of <lxml.etree.XMLSchema object at 0x7ffb1e93f9c0>>(<lxml.etree._ElementTree object at 0x7ffb1e995740>)
E        +    where <bound method _Validator.validate of <lxml.etree.XMLSchema object at 0x7ffb1e93f9c0>> = <lxml.etree.XMLSchema object at 0x7ffb1e93f9c0>.validate

tests/test_validate.py:33: AssertionError
=========================== short test summary info ============================

test report: short test summary info 

=========================== short test summary info ============================
FAILED tests/test_validate.py::test_validate[samples/invalid_missig_lang.xml]
========================= 1 failed, 3 passed in 0.08s ==========================

https://gitlab.com/datex2/presentation/2023-virtduf-validation/pytest-validate

NDW Online Message Validator

by Edgar van Wilgenburg, NDW

—

On-line XML message validation
for selected DATEX II Profiles

Scope: Serialized XML Message

d2context-ndw-validator.png

The purpose of NDW validator

Stop expensive exchanges of structurally broken data

Quick and Easy (On-Line) Validation

Directory of Agreed profiles

Accessible to All types of Users

About the NDW validator

https://d2val.staging.ndw.nu/

Python + Flask web app (+ Bash + xmllint).

For DATEX II XML messages.

Accepts gzipped files.

Checks for: schema defined structure.

you-shall-not-pass.jpg

Let’s meet our on-line Gandalf

NDW validator Demo

  1. Select a Profile
  2. Upload Your XML
  3. See the validation report

Repeat for valid/invalid DATEX II Message

Summary of NDW validator

Current Status

very easy to use

very clear answer (one of):

you are fine or list of things to fix

Space for improvement

  1. simpler profile selection
  2. downloadable profile related schemas

InQMS: Validate them all

by Jan Vlčinský, TamTam Research

—

Let’s try to
validate
all the published messages
https://inqms.tamtamresearch.com/

—

Tested with Czech NDIC content.

InQMS Architecture

  • Collect and Archive
  • Evaluate (once a day)
  • Visualize (Grafana)

Validation Features

List all the published messages:

  • One validation report per message
  • The message itself

—

Makes resolving data exchange problem much simpler.

Bonus Features

Time series for:

  • sizes (plain/gzipped)
  • # of records
  • update periods
  • # of errors

Questions & Answers

Links

Slides

slides-qr.png

https://slides-datex2-presentation-2023-virtduf-validati-23493b76d5a303.gitlab.io/

GitLab presentation group

https://gitlab.com/datex2/presentation/2023-virtduf-validation:

Presented applications

RAV Centre
https://testcenter.datex2.eu/xmlchk/

NDW On-line DATEX II Validator
https://d2val.staging.ndw.nu/

InQMS
https://inqms.tamtamresearch.com/

GitLab.com/datex2

https://gitlab.com/datex2

Proof of concepts for publishing DATEX II extensions and profiles.

This is not an official DATEX II site.

Includes also materials for this presentation.

xmlstarlet

Command line utility to query, transform, validate, and edit XML documents and files using a simple set of shell commands in a way similar to how it is done with grep, sed, awk, diff, patch, join, etc.

https://xmlstar.sourceforge.net/

OS: Unix-like, Windows, CygWin, Mac OS

Wikipedia: https://en.wikipedia.org/wiki/XMLStarlet

Movie: Validation (2007)

director/writer: Kurt Kuenne

IMDb: Validation

YouTube: Validation by Kurt Kuenne (16:23)

You are Awesome!!!