Back to Repositories

Testing Process Fuzzing Implementation in OpenPilot

This test suite implements fuzzy testing for the OpenPilot autonomous driving system’s process components. It focuses on validating process behavior under randomized input data to uncover edge cases and potential failure modes.

Test Coverage Overview

The test suite provides comprehensive fuzzy testing coverage for OpenPilot’s process components, excluding certain processes that currently fail due to unrealistic data assumptions.

Key functionality tested includes:

Random event message generation
Process replay validation
Toyota vehicle integration testing
Message handling and processing

Implementation Analysis

The testing approach utilizes hypothesis-driven fuzzy testing to generate random but valid test data. The implementation leverages parameterized testing patterns to run tests across multiple process configurations.

Technical patterns include:

Hypothesis test strategies for data generation
Parameterized test expansion
Process replay configuration management
Event message construction and validation

Technical Details

Testing tools and libraries:

Hypothesis framework for property-based testing
Parameterized testing framework
Custom FuzzyGenerator for message creation
OpenPilot’s process_replay framework

Configuration includes timeout settings, health check suppressions, and example limits.

Best Practices Demonstrated

The test implementation demonstrates several testing best practices for complex autonomous systems. The code shows excellent organization through separation of concerns, configurable test parameters, and selective process testing.

Notable practices include:

Explicit handling of problematic test cases
Configurable test iterations via environment variables
Targeted health check suppression
Comprehensive process configuration management

commaai/openpilot

selfdrive/test/process_replay/test_fuzzy.py

            
import copy
import os
from hypothesis import given, HealthCheck, Phase, settings
import hypothesis.strategies as st
from parameterized import parameterized

from cereal import log
from opendbc.car.toyota.values import CAR as TOYOTA
from openpilot.selfdrive.test.fuzzy_generation import FuzzyGenerator
import openpilot.selfdrive.test.process_replay.process_replay as pr

# These processes currently fail because of unrealistic data breaking assumptions
# that openpilot makes causing error with NaN, inf, int size, array indexing ...
# TODO: Make each one testable
NOT_TESTED = ['selfdrived', 'controlsd', 'card', 'plannerd', 'calibrationd', 'dmonitoringd', 'paramsd', 'dmonitoringmodeld', 'modeld']

TEST_CASES = [(cfg.proc_name, copy.deepcopy(cfg)) for cfg in pr.CONFIGS if cfg.proc_name not in NOT_TESTED]
MAX_EXAMPLES = int(os.environ.get("MAX_EXAMPLES", "10"))

class TestFuzzProcesses:

  # TODO: make this faster and increase examples
  @parameterized.expand(TEST_CASES)
  @given(st.data())
  @settings(phases=[Phase.generate, Phase.target], max_examples=MAX_EXAMPLES, deadline=1000,
            suppress_health_check=[HealthCheck.too_slow, HealthCheck.data_too_large])
  def test_fuzz_process(self, proc_name, cfg, data):
    msgs = FuzzyGenerator.get_random_event_msg(data.draw, events=cfg.pubs, real_floats=True)
    lr = [log.Event.new_message(**m).as_reader() for m in msgs]
    cfg.timeout = 5
    pr.replay_process(cfg, lr, fingerprint=TOYOTA.TOYOTA_COROLLA_TSS2, disable_progress=True)