Your ECU Pen Test Expires at the Next Release: Remote Hardware-in-the-Loop Security Testing Keeps It Current

The request comes down from the OEM: send us your pen-test results. At some ECU suppliers, the person who reads it is the whole security team, one engineer who manages the entire security process and does not run pen tests. So the supplier buys one. A few weeks later a PDF goes upstream, the box is checked, and everyone gets back to shipping.

Six months later the ECU is running new firmware, and that PDF describes a product nobody is building anymore.

A pen test is worth more than its report. Every test case the testers ran, from the library cases they picked for your target to the custom ones they wrote for what they found, can run again on the next build, against the real hardware, without anyone shipping a box anywhere. Remote hardware-in-the-loop security testing makes that cheap, and it is how we think every pen test should end: with test cases you keep re-running.

The report is a photograph of one build

Penetration testing is point-in-time by nature. We made the case in Fuzz Testing vs Penetration Testing in Automotive Cybersecurity that a pen test is a snapshot of one moment and has to be repeated, during and after development, as threats change. Your product changes too. Vehicle programs run on release windows of 6 to 18 months, with patches in between, so the snapshot ages.

In practice the repeat is another engagement. Our own Regression Testing Service has been sold that way, as a bucket of hours to re-verify the findings that were fixed. The hardware itself is easy to get, since product engineers hand over units to be broken as part of the job. Getting it in front of testers is the hard part: shipping it to a lab or putting testers in the same room as the device, then rebuilding the rig and redoing the checks by hand.

A pen test should leave test cases behind

A good pen test is a mix. Part of it is a selection of pre-made test cases that fit the target; most of it is testers working by hand on what no library can reach, like the proprietary diagnostic services a discovery scan turns up and the behavior that makes your ECU yours.

Every one of those checks can be a test case. The library ones already are. What the testers found by hand can become custom test cases, written in Python and kept with the rest. Together they are everything the pen test already proved, ready to run against the next build with a test run instead of a new engagement.

The creative work of a pen test, finding what nobody thought to look for, still takes an engineer. Confirming that last release's answers still hold does not, and that part we can automate.

Yesterday a pen test from us ended in a report. Today, with what we're building in VSEC Test,  penetration tests can end with their findings and regression test cases sitting in the platform, so you re-run them after every patch.

The ECU never leaves your building

A re-run is only cheap if the hardware stays put. In VSEC Test, a Raspberry Pi sits next to the device under test and connects to the VSEC cloud over a VPN tunnel. The test cases run against the real ECU on your desk. Plug-and-play test rigs run from about $100,000 to $500,000; our VCI is a commodity computer you supply.

It reaches more than the CAN bus. In one engagement a single VCI connected to the ECU three ways: an automotive Ethernet media converter into its Ethernet port, a JTAG debugger over USB, and a USB-to-TTL adapter wired to UART. Through the UART adapter our testers got a shell on the target's own operating system, working from wherever they were. A Test Plan defined once re-runs across every bench it is pointed at.

What every release gets

Each release gets two things. The regression test cases re-run every check earlier testing proved, and VSEC Test puts the last verified run next to this one so you see every test case that flipped. Each run can export a PDF report carrying the UN R155 mitigation references (Annex 5) its test cases map to, plus detailed logs, ready for the people who assemble type-approval evidence. Where that verification sits in the development lifecycle is covered in Secure Product Development Activities in Automotive.

Your engineers spend their time on whatever this release added that nothing checks yet.

The free license of VSEC Test includes three test cases: UDS Security Access, CCP Upload and XCP Upload. Run them against one of your own targets on the build you are shipping now. Then run them again on the next one.

‍