SAPIENT Test Harness

At a glance

Standard

BSI Flex 335 v2.0 (ICD version string BSI Flex 335 v2.0)

Roles tested

Edge Node and Fusion Node. The harness plays whichever role you're not.

Platforms

Windows, Linux and macOS, on x86_64 and arm64. Single standalone binary, no database or runtime to install. Linux builds need glibc 2.34 or later.

Current release

v1.0.0-beta.6 (pre-release)

Output

Human-readable text, or JSON with distinct exit codes for CI

Licence

MIT or Apache-2.0

What it checks

Parsing a protobuf message tells you it is well-formed. It doesn't tell you whether a SAPIENT node behaves correctly. The harness checks three things:

• Message content. Every message it receives is checked against BSI Flex 335 v2.0: mandatory fields, UUID formats, enumerations, units, timestamps and the ICD version string.
• Session state. The harness tracks the session as it runs: whether registration has completed, which mode the Edge Node is in, which tasks and alerts are awaiting acknowledgement, and when the last status report arrived.
• Behaviour across messages. Using that state it catches problems no single message reveals, for example:
  • messages sent before the Registration / RegistrationAck handshake;
  • a TaskAck or AlertAck that doesn't correspond to an outstanding task or alert;
  • the same task acknowledged twice, or a task's acknowledgement status moving through an invalid transition;
  • duplicate mode names in a Registration, which make mode-change tasks ambiguous;
  • a StatusReport claiming a mode the Edge Node was never tasked into;
  • a mode-change task naming a mode the Edge Node never registered;
  • status reports that arrive later than the registered interval, or whose timestamps go backwards;
  • detections classified with a class or location type the active mode never declared.

SAPIENT conformance explained goes through these layers in more detail.

Install

Pre-built binaries for release v1.0.0-beta.6 are below. Extract the archive and put sapient-harness somewhere on your PATH.

Older versions, release notes and the CycloneDX SBOM are on the GitHub releases page. Linux builds need glibc 2.34 or later.

Install with a script

The install scripts download the right archive for your machine and put sapient-harness in ~/.cargo/bin.

curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/tom-mann-ironclad/sapient-test-harness/releases/download/v1.0.0-beta.6/sapient-test-harness-cli-installer.sh | sh

powershell -ExecutionPolicy Bypass -c "irm https://github.com/tom-mann-ironclad/sapient-test-harness/releases/download/v1.0.0-beta.6/sapient-test-harness-cli-installer.ps1 | iex"

Verifying a download

Every file in a release has a SHA-256 checksum, and sha256.sum lists them all. Release files also carry signed build-provenance attestations, which you can check with the GitHub CLI:

sha256sum -c sha256.sum --ignore-missing
gh attestation verify <file> --repo tom-mann-ironclad/sapient-test-harness

Building from source

You need a stable Rust toolchain:

git clone https://github.com/tom-mann-ironclad/sapient-test-harness.git
cd sapient-test-harness
cargo install --path crates/sapient-test-harness-cli

Check the installation

Before trusting a result, check that the harness itself is healthy. This needs no network access:

sapient-harness selftest

Quick start

Testing an Edge Node? Run the harness as the Fusion Node it registers with. --target is the local address to listen on, so pick one your Edge Node can reach, then point the Edge Node at it:

sapient-harness run --role fusion --target 0.0.0.0:5000

Testing a Fusion Node? Run the harness as an Edge Node that connects to it. It retries until the Fusion Node is listening or --connect-timeout-secs runs out:

sapient-harness run --role edge --target fusion-node.example:5000

Either way the harness drives one complete session (registration, status reporting, a detection, a mode-change task, an alert and a GoodBye) and prints a verdict. The quick-start guide lists every flag and what each scenario requires.

Reading the results

A run ends in one of three outcomes: passed, failed (at least one error-severity finding) or incomplete (the connection or deadline ended before the required checks finished; that is not in itself a protocol violation). Each finding has a stable rule ID, the field it concerns and an explanation:

sapient-harness run -- role=fusion suite=v2.0 target=0.0.0.0:5000

FAIL -- 1 finding(s).

Error:
  [registration.icd_version.invalid] registration.icd_version: ICD version
  specified in registration is not a valid option.

Warning-severity findings, such as relying on a legacy convention to declare the default mode, are reported but don't fail the run.

Automated and interactive testing

Conformance testing in CI

--format json writes the report as JSON on stdout, with progress on stderr. Exit codes distinguish a non-conformant node from a test that couldn't run: 0 passed, 1 findings or incomplete, 2 a harness-level failure such as a refused connection.

sapient-harness run --role edge --target fusion-node.example:5000 \
  --format json > result.json
jq '.findings[] | select(.severity == "error")' result.json

The quick-start guide has a complete GitHub Actions example.

The rule catalogue

Every finding the harness can report is listed in RULES.md, the human-readable conformance rule catalogue, with its rule ID and message. The catalogue is generated from the validation source, so it can't drift from what the harness actually checks. Rule IDs follow <message>.<field>.<violation> for message-level rules and session.… for stateful ones.

If you think a rule misreads BSI Flex 335, open an issue with the rule ID and the clause of the standard you're relying on.

Relationship to the DSTL harness

DSTL publishes the BSI Flex 335 v2 Test Harness, a Windows / .NET 6 application that uses PostgreSQL 12. This harness is a separate implementation with different trade-offs: a single cross-platform binary with no database, scripted and CI-friendly runs, and session-level checks.

Its message validators are tested against the 231 validation fixtures from the DSTL harness's own unit tests, and agree with the recorded outcome for each one.

Status and independence

The harness is independently maintained and is not an official DSTL, MOD, BSI or NATO product. A passing run is not a certification. It shows that your implementation completed the scenario without any finding the harness checks for. The current releases are pre-releases on the way to 1.0. The changelog records each change.

Bug reports and questions go to the issue tracker. Include the command you ran, the full output, and the output of sapient-harness --version.