TestBench Agent

Overview

The ThreatZ TestBench Agent (ArchitTestClient v2.0.0) is a production-grade, Python-based automotive cybersecurity testing client that runs directly on your test bench hardware. It executes security campaigns against real ECUs and vehicle networks — coordinated seamlessly from the ThreatZ platform via HTTP/REST (Protocol V2). The agent receives campaign assignments through command polling, executes tests using protocol-specific fuzzers and hardware adapters, and reports findings with full artifact collection.

The TestBench Agent is part of the Validation & Testing module. While the module manages the overall traceability backbone — linking test cases to TARA requirements, tracking coverage, and collecting evidence — the TestBench Agent is the execution engine that brings testing to the metal.

Architecture

The agent follows a modular architecture with clear separation of concerns:

Supported Protocols (36+)

The TestBench Agent ships with purpose-built fuzzers for 36+ automotive communication protocols. Each fuzzer uses CWE-aware mutation strategies that target specific weakness classes — not random bytes.

Vehicle Field Buses

Diagnostic Protocols

Automotive Ethernet

Additional Protocols

CANopen, J1939, CCP/XCP, SecOC, DLT, BLE, Cellular, CHAdeMO, NFC, ISO 15118 (V2G), GNSS, CCS, NACS, GBT, E2E protection

Test Types

Fuzzing Campaigns

• Random, coverage-guided, grammar-based, and mutation-based strategies
• CWE-aware mutation shaping (20+ CWEs per protocol)
• Corpus management with minimization
• Response-guided adaptive fuzzing
• Active CWE objective rotation (10-second intervals)

Penetration Testing

• STRIDE-classified attack vectors (Spoofing, Tampering, Repudiation, Information Disclosure, DoS, Elevation of Privilege)
• CAN frame injection with arbitrary IDs and payloads
• Gateway routing table manipulation
• DoIP session hijacking and authentication bypass
• Diagnostic power mode manipulation

Robustness Testing

• Bus load stress testing (up to 90%+ load)
• Error frame generation and flood detection
• Extended payload boundary testing
• CAN/CAN-FD transition testing

UDS Diagnostic Testing

• ECU Reset (0x10), Read DID (0x22), Write DID (0x2E)
• Security Access (0x27), Routine Control (0x31)
• Write Memory Address (0x3D), I/O Control (0x2F)
• Session management (default, programming, extended)

Compliance & System-Level Testing

• Safety compliance verification (ISO 26262, SOTIF)
• Protocol compliance validation
• Multi-ECU coordination and complex attack sequences
• Vehicle-in-the-loop integration

Hardware Abstraction Layer

The agent supports multiple hardware vendors through a unified, vendor-neutral abstraction — switch hardware without changing your test configuration.

Hardware health monitoring tracks connection quality, latency per frame, frame loss detection, and performance degradation scoring with health levels: Healthy, Warning, Critical.

5-Level Safety Escalation

Built-in safety guards monitor bus load, error frame rates, and ECU heartbeat in real time. When thresholds are breached, the agent escalates automatically:

Monitoring capabilities include bus load monitoring (warning at 75%, critical kill at 90% — configurable), error frame detection (>10 errors/second default), ECU heartbeat monitoring (5-second timeout), manual kill switch via GUI, and custom kill expressions for specialized safety requirements. All safety events are recorded with type, severity, timestamp, iteration index, and detailed context.

CWE-Driven Attack Intelligence

Each campaign targets specific Common Weakness Enumerations mapped to the protocol under test:

• 20+ CWE objectives per protocol (e.g., CWE-120 Buffer Overflow, CWE-1284 Improper Validation of Specified Quantity in Input)
• Per-CWE shaping rules define how frame fields are mutated to target each weakness class
• Attack preset levels: Quick (fast scan), Standard (balanced), Full (exhaustive)
• Per-CWE BLF trace files for audit granularity — separate trace capture per weakness
• Finding correlation: Each finding maps back to the CWE that triggered it

CANoe & CAPL Integration

• CANoe Configuration Loading: Direct .cfg file loading for simulation environments
• CAPL Script Generation: The Scriptor module converts DSL test definitions into executable CAPL code
• Smart CAPL Generator: Context-aware script generation matched to ARXML signal definitions
• Report Parsing: Extracts results from CANoe test reports (.vtestreport format)
• Output Discovery: Automatic scanning of CANoe output directories for artifact collection

ARXML Parsing

The agent uses a canmatrix-based parser with multi-cluster ARXML support:

• Extracts ECU list with TX/RX frame mappings, frames with arbitration IDs and DLC, signals with types/min/max/scaling
• NM frame detection and isolation per target ECU
• Database name derivation with CAPL namespace compatibility
• Multi-bus support — handles ARXML files defining multiple CAN clusters with deduplication by frame ID

Test Execution Workflow

1. Command Polling — Agent polls the ThreatZ platform every 15 seconds for new commands
2. Campaign Initialization — Load ARXML, validate target ECU, resolve protocol type, load attack vectors
3. Exploitation — FuzzScheduler mutates inputs → Protocol fuzzer shapes for CWE targets → Hardware adapter transmits
4. Monitoring — Separate monitoring thread analyzes responses, detects anomalies, tracks timing
5. Finding Detection — Correlate monitored data with attack vector → Generate findings with severity and CWE mapping
6. Artifact Collection — BLF traces, logs, PCAP captures, reports → SHA-256 integrity hashing
7. Report Generation — FuzzReport (XML/JSON) with objective transitions, findings, coverage data, BLF references
8. Upload & Completion — S3 presigned URL upload with progress tracking → Status update to platform

Campaign states: Queued → Initializing → Running → Paused → Succeeded / Failed / Killed

Service Pack Execution

Service packs enable custom test scenarios beyond built-in protocol fuzzers:

1. CREATE_SCENARIO: Platform sends pack definition; agent creates folder structure with batch/shell stub
2. Configuration: config.json, tool_config.json, evidence_policy.json, safety_policy.json
3. RUN_SERVICE_PACK: Execute scenario, capture bus traffic (BLF/ASC/PCAP), collect logs
4. Evidence Collection: Policy-based artifact filtering and collection
5. Upload: S3 presigned URL upload with manifest and integrity verification

Artifact Collection

Every campaign produces a complete evidence package:

All artifacts are uploaded via S3 presigned URLs with multipart support for files exceeding 5 MB. Server-side verification confirms SHA-256 integrity.

GUI & Headless Operation

GUI (Flet-based)

• Real-time campaign monitoring dashboard
• CWE objective rotator with 10-second cycling
• Fuzz scope tab for target frame/signal selection
• Monitoring tab for mode, window, and tolerance configuration
• Hardware health and connection status indicators
• Project manager for multi-project switching
• Dark/light theme auto-detection

Headless Mode

• Full campaign execution without GUI
• Suitable for CI/CD integration and automated regression
• Configuration via .env and project_config.json
• All results reported to platform programmatically

Communication with ThreatZ Platform

The agent uses HTTP/REST (Protocol V2) — no WebSocket dependency:

Authentication uses JWT tokens with machine-based identifiers. Multi-tenant support via tenant ID tracking. Artifact upload uses S3 presigned URLs with SHA-256 integrity verification and multipart upload for files exceeding 5 MB.

Integration with Other Pillars