Validation & Testing

Overview

The Validation & Testing module is the security testing management layer of the ThreatZ platform. It focuses on cybersecurity-specific testing — penetration testing, fuzz testing, robustness testing, and compliance validation — not functional or performance testing. The module provides a Test API client that connects to your existing test execution environments (MIL, SIL, HIL) to ensure full traceability and linkage of test artifacts and evidence to the continuously updated TARA and Compliance documentation modules.

Key Concepts

Test API Client

The Test API Client is a lightweight connector that integrates with your existing test execution environment. It communicates bi-directionally with ThreatZ via REST API and WebSocket, synchronizing test case assignments, receiving execution results, and uploading artifacts. The API client is tool-agnostic — it works alongside dSPACE, National Instruments, Rohde & Schwarz, Vector CANoe/CANalyzer, and custom in-house test benches. ThreatZ does not replace these tools — it connects to them as the traceability and evidence backbone.

Test Cases

Test Cases are the fundamental unit of work. Each test case includes a title, description, category, STRIDE classification, detailed steps, and configurable parameters. Test cases can be created manually, imported from external sources, or generated via AI recommendations. Every test case can be linked to one or more cybersecurity requirements, claims, or security goals — enabling full traceability from threat analysis through to validation evidence.

Campaigns

Campaigns are the execution unit that groups test cases for a specific security test run. A campaign can be configured with:

• Execution mode: Sequential or Parallel
• Failure behavior: Stop on first failure or continue
• Timeouts and retries: Per-test and per-campaign limits
• Protocol packs: Predefined sets of test vectors to import
• Attack vectors: Specific attacks with parameter overrides
• Fuzzing configuration: Mode (exploration, targeted, regression, minimization), strategy (random, coverage-guided, grammar-based, mutation, hybrid), iteration limits, duration caps, and corpus management
• Safety constraints: Maximum bus load percentage, error frame thresholds, and automatic kill conditions

Campaign types include: Penetration, Fuzz, Robustness, Compliance, System Test, and Service Pack.

Security Testing Capabilities

Penetration Testing

ThreatZ manages penetration test campaigns that target attack paths identified in the TARA module. Test cases are linked directly to threat scenarios and attack paths, so results automatically update the risk posture of the corresponding threats.

Fuzz Testing

The module supports coverage-guided, grammar-based, and mutation fuzzing strategies with built-in safety guardrails:

• Bus load monitoring: Alerts and automatic stop when CAN bus load exceeds configured thresholds
• Error frame detection: Tracks error frames and triggers safety events when thresholds are breached
• Kill conditions: Configurable rules that automatically terminate a campaign when critical safety boundaries are crossed

Robustness & Compliance Testing

Structured campaigns for robustness validation and regulatory compliance checks, with protocol packs providing predefined test vectors aligned to ISO/SAE 21434 and UN R155 requirements.

How It Works

Connecting Your Test Environment

1. Install the ThreatZ Test API client alongside your existing test infrastructure
2. Configure the API endpoint and authenticate with your project API key
3. Map your external test IDs to ThreatZ test cases (or let the API create them on import)
4. Results and artifacts flow back into ThreatZ automatically after each test run

The API client supports environments running dSPACE ControlDesk, NI TestStand, CANoe, CANalyzer, Python-based test frameworks, and any tool that can make HTTP calls.

Traceability Workflow

1. Cybersecurity requirements and claims from TARA flow into the Validation & Testing module
2. Security test cases are linked to these requirements — either manually or via bulk import
3. Your team executes security tests in their environment (MIL, SIL, HIL, or manual)
4. The Test API pushes results and artifacts back to ThreatZ
5. Coverage metrics update in real time — showing which requirements are validated
6. Evidence packages are generated for compliance audits and OEM submissions

Coverage Analysis

The module calculates coverage across multiple dimensions to identify validation gaps before they become audit findings:

• Requirement coverage: Percentage of cybersecurity requirements with at least one linked and executed test
• Claim coverage: Percentage of claims validated by test results
• Pass rate: Ratio of passed vs. failed test executions across campaigns
• Gap identification: Requirements, claims, or security goals with no linked or executed tests

Artifact Management

All test artifacts are collected centrally in ThreatZ, maintaining the link between each artifact and the requirement or claim it provides evidence for:

• Execution logs from your test framework
• Network traces (CAN, Ethernet, FlexRay)
• Tool-specific reports (dSPACE, NI, CANoe)
• Screenshots, evidence files, and signed attestations

These artifacts serve as ISO/SAE 21434 compliance evidence and can be referenced directly in regulatory submissions, OEM audits, and type-approval documentation.

TestBench Agent

For organizations that need to execute security tests directly on hardware, the ThreatZ TestBench Agent is a production-grade testing client that runs on your test bench, executing campaigns against real ECUs and vehicle networks. It ships with 36+ protocol fuzzers (CAN 2.0, CAN FD, CAN XL, LIN, FlexRay, DoIP, SOME/IP, UDS, and more), a vendor-neutral hardware abstraction layer (Vector, Peak, Kvaser, Intrepid, SocketCAN), CWE-driven attack intelligence, and a 5-level safety escalation system. All findings and artifacts flow back to the Validation & Testing module for full TARA traceability.

Read the full TestBench Agent KB article →

Security Testing — Not Functional Testing

ThreatZ focuses exclusively on cybersecurity validation: penetration testing, fuzz testing, robustness testing, and compliance verification. It does not cover functional testing, performance testing, or EMC testing. The Test API client connects to your existing test execution infrastructure — dSPACE, National Instruments, Rohde & Schwarz, CANoe, or in-house test benches — without replacing any of them. ThreatZ is the traceability backbone that ensures every security test result is linked back to the TARA, every coverage gap is visible, and every piece of compliance evidence is audit-ready.

Integration with Other Pillars