6.3 KiB

Raw Blame History

test_regress/ -- regression tests

Covers .v/.sv sources, .py drivers, and .out golden files under test_regress/. Read the repository-root AGENTS.md first. This file has two parts: Orientation explains how the harness runs a test; Before you open a PR is the test-authoring checklist.

Orientation: how the harness works

A test is a .v/.sv source plus a matching .py driver in t/. The driver calls test.compile(), test.execute(), and/or test.lint(). Without a .py the source never runs and is dead code giving false coverage confidence.
Golden output lives in .out files, compared via expect_filename. Generate them with HARNESS_UPDATE_GOLDEN=1 python3 t/<name>.py -- never hand-write them; the harness normalizes paths, version markers, and line numbers that hand edits get wrong.
Run one test from the repository root: test_regress/t/t_<name>.py. When running from a checkout, set VERILATOR_ROOT to the checkout root first so the harness compiles the generated C++ against the right headers.
test.compile() defaults are richer than they look: the vlt driver auto-injects --exe and a generated main, and assert property fires its action blocks without --assert -- try the simple form before adding flags.
The t_dist_* tests enforce repository conventions (file headers, sorted lists, warning documentation, ASCII-only) -- run the relevant ones before submitting.

Before you open a PR

Naming

Name tests t_{category}_{description} in snake_case; the first word groups the category (t_lint_unused_func_bad, not t_unused_func_lint_bad) so related tests are findable and runnable together.
Use _bad when the code is illegal SystemVerilog, _unsup when it is legal but unsupported, _off for disabled-behavior tests -- never _fail.
Keep filenames under roughly 30-35 characters.

Files and drivers

Every .v needs a matching .py driver that actually calls test.compile() and test.execute() (or test.lint() for static-analysis-only tests) -- a driver that sets up but never runs hides coverage gaps silently.
Copy the license header from an existing file: .v tests carry the CC0 notice, .py drivers carry the standard driver header -- distribution tests check headers on every committed file.
Use --binary instead of --exe --main --timing -- it implies the others.
Error tests use test.compile(fails=True, expect_filename=test.golden_filename) (or test.lint(...)) and must not call test.execute().
Use .out golden files with expect_filename instead of inline expect regex strings -- goldens are diffable and maintainable.
Use test.glob_one()/test.glob_some() and test.timeout() instead of raw glob.glob() and manual time.time() measurements.
Coverage tests run verilator_coverage and verify individual bins and points, not just aggregate percentages.
Assert optimization stats with exact expected counts: test.file_grep(test.stats, r'Optimizations, ...\s+(\d+)', N).
Add a _protect_ids variant when a feature emits user identifiers or filenames; use conservative thread counts (2 or fewer) in multithreaded tests.
Extend existing test files with related cases instead of creating many single-purpose files; keep drivers minimal -- test logic belongs in the .v.

Golden .out files

Never hand-write or hand-edit .out goldens -- regenerate with HARNESS_UPDATE_GOLDEN=1.
When a feature lands, remove its now-supported entries from t_*_unsup.v / t_*_bad.v in the same change and regenerate the goldens -- stale entries no longer error and reviewers will flag them.

Verilog style

2-space indentation, no tabs.

Declarations are flush-left with a single space between type and name; never column-align:

// WRONG: column-aligned
bit [63:0] crc    = 64'h5aef0c8d_d70a4497;
int        cyc    = 0;
// RIGHT: flush-left
bit [63:0] crc = 64'h5aef0c8d_d70a4497;
int cyc = 0;

Run nodist/verilog_format on new .v files; wrap macro definitions in // verilog_format: off/on so the formatter does not split them.
Use $display("%0d", ...) not %d -- avoids leading-space padding.
Wrap Verilator-specific test code (e.g. $c) in `ifdef VERILATOR.
Use inline // verilator lint_off WARNCODE only when that warning is itself under test -- fix root causes otherwise.
Use only IEEE 1800-compliant constructs other simulators also accept -- tests validate standard behavior, not Verilator's parser leniency.
Omit optional end labels on endmodule/endclass/endtask/endfunction.

Self-checking

Use the checkh/checkd/checks assertion macros, not manual if/$display/$stop sequences. Note checkh prints with %p, which renders integers as hex -- use checkd for integer comparisons.
Use the `stop macro rather than direct $stop.
Drive logic with runtime-varying inputs (counters, CRC/LFSR registers) so constant folding cannot pre-evaluate the logic under test; check behavior across multiple clock cycles, not just initial values.
For a test whose pass/fail depends on varying or random values, loop enough iterations that the values demonstrably differ and size the value space so a failure is probable per run, then confirm the test fails on an un-fixed tree before submitting.

Test design

Use non-power-of-2, non-word-aligned widths (7, 15, 31, 33, 63, 65, 95) -- exposes masking and word-boundary bugs that 32/64/128 hide.
Test both [high:low] and [low:high] orderings plus non-zero bounds like [3:1]; use different ranges for each axis of multidimensional arrays.
When adding type support, test all basic types (chandle, string, real) and typedef-wrapped variants.
Include the issue's own reproducer as a committed test, and verify it fails without the fix.
Assert non-blocking-assignment results in the cycle immediately after they take effect, before later overwrites, using indices that change post-NBA.

6.3 KiB Raw Blame History