From a129e3bc6846040babf005c1649ad905374f891f Mon Sep 17 00:00:00 2001 From: Zachary Snow Date: Tue, 30 May 2023 22:33:57 -0400 Subject: [PATCH] add test suite documentation --- README.md | 15 ++++-- test/README.md | 109 ++++++++++++++++++++++++++++++++++++++++++ test/relong/README.md | 30 ++++-------- 3 files changed, 129 insertions(+), 25 deletions(-) create mode 100644 test/README.md diff --git a/README.md b/README.md index fa445ae..a5c4b66 100644 --- a/README.md +++ b/README.md @@ -140,12 +140,17 @@ front end if there is significant interest. ## Testing -Once the [test dependencies](#dependencies) are installed, tests can be run with -`make test`. GitHub Actions is used to automatically test commits. +Once the [test dependencies] are installed, tests can be run with `make test`. +GitHub Actions is used to [automatically test] commits. Please review the [test +documentation] for guidance on adding, debugging, and interpreting tests. -There is also a [SystemVerilog compliance suite] being created to test -open-source tools' SystemVerilog support. Although not every test in the suite -is applicable, it has been a valuable asset in finding edge cases. +[test dependencies]: #dependencies +[test documentation]: test/README.md +[automatically test]: https://github.com/zachjs/sv2v/actions + +There is also a [SystemVerilog compliance suite] that tests open-source tools' +SystemVerilog support. Although not every test in the suite is applicable, it +has been a valuable asset in finding edge cases. [SystemVerilog compliance suite]: https://github.com/chipsalliance/sv-tests diff --git a/test/README.md b/test/README.md new file mode 100644 index 0000000..3aa2416 --- /dev/null +++ b/test/README.md @@ -0,0 +1,109 @@ +# sv2v tests + +Tests are run via `make test` or by running `run-all.sh` directly. Each +sub-folder here (except `lib`) is a suite of tests using shUnit2 as a test +framework. Every suite contains a `run.sh` that performs its tests when run from +within the suite's folder. The tests are run against the compiled `bin/sv2v` in +the root of the repository. + + +## Regular test suites + +These primary test suites collectively contain hundreds of test cases, each of +which is converted using sv2v and simulated using iverilog. The test cases in +these suites are automatically discovered. Each test case `foo` consists of +three components: + +1. **`foo.sv`** is the SystemVerilog file to be converted by sv2v. +2. **`foo.v`** is a manually-produced Verilog file that should behave + equivalently to `foo.sv`. For Verilog-compatible test cases, this file may be + omitted, in which case the runner uses the original `foo.sv` instead. +3. **`foo_tb.v`** is a Verilog testbench provided to iverilog to exercise both + `foo.v` and the converted `foo.sv`. If this file is not present, the runner + assumes both `foo.sv` and `foo.v` contain a `module top`. + +For each test, the simulation results are compared by log output (e.g., from +`$display` or `$monitor`) and by top-level VCD (excluding parameters). The test +is repeated with the conversion performed in `--verbose` mode. + +All three source files are run through sv2v repeatedly to perform the following +additional consistency checks: + +* Each file should convert with no errors or warnings. +* When the converted output is converted again, it should not change. +* `sv2v file.sv` should exactly match `sv2v --pass-through file.sv | sv2v -`. +* The converted output should not contain certain SystemVerilog-only constructs + that iverilog happens to support in Verilog-2005 mode, such as `$bits`. + +Each regular test suite has a particular focus: + +* `basic` contains Verilog-compatible tests without a `.v` +* `core` contains standard tests with at least a `.sv` and a `.v` +* `lex` contains tests for lexing and preprocessing (e.g., macros) +* `relong` contains tests provided by Reid Long in 2019 + +Rarely, a file may have an adjacent `.pat` file (e.g., [unneeded_scope.sv.pat]) +used to assert the presence or omission of specific substrings in the converted +output. This can be used to check aspects of sv2v's conversion behavior that are +not apparent in simulation. + +[unneeded_scope.sv.pat]: core/unneeded_scope.sv.pat + + +## Specialized test suites + +The `error` suite tests inputs that should cause sv2v to output an error. The +suite runs each `.sv` file in the folder and ensures that sv2v returns a +non-zero exit code and outputs something to `stderr`. Each test may contain a +`// pattern: ...` and a `// location: ...` flag at the top. When present, the +test runner checks if the `stderr` contains the given error and location +information. + +The `nosim` suite simply checks that each of the `.sv` files can be converted by +sv2v, without evaluating the converted output. This is used to provide limited +coverage of language constructs that sv2v can output but that are not supported +by `iverilog`. + +The remaining test suites have a custom `run.sh` that defines a list of test +procedures that may not correspond directly to the other files in the folder. +Many of these suites test a particular feature of the sv2v CLI. + +* `define` tests `-D`/`--define` +* `dump` tests `--dump-prefix` +* `help` ensures the `--help` output in the README is up to date +* `keyword` tests `begin_keywords` version specifiers +* `number` generates and tests short number literals +* `siloed` tests `--siloed` and default compilation unit behavior +* `truncate` tests number literal truncation and `--oversized-numbers` +* `warning` tests conversion warnings +* `write` tests `-w`/`--write` + + +## Adding new tests + +Most tests for bug fixes or new language features are added to the `core` suite, +with a `.sv` and a hand-converted `.v`. New CLI features typically get a new +specialized test suite. All test files use 4 spaces for indentation. Macros or +`.vh` files may be used to reduce duplicated code where appropriate. + + +## Debugging test failures + +* There may be incompatibilities with a specific version of iverilog. Please see + the [main GitHub Actions workflow] to determine what version of iverilog is + currently used in CI. +* It is often helpful to run a subset of the tests in the large suites (e.g., + `core` and `error`) by passing a list of file or base names to `run.sh`. For + example, `./run.sh missing_join` or `./run.sh interface_*.sv`. +* Review [functions.sh] to disambiguate test failure messages. +* Many test cases can be run outside of the test harness, e.g., + ``` + bin/sv2v test/core/inc.sv > foo.v && iverilog -g2005 foo.v && ./a.out + iverilog -g2005 test/core/inc.v && ./a.out + rm foo.v a.out # when done + ``` +* Consider diffing the output against a prior build of sv2v, e.g., `diff <(sv2v + test/core/inc.sv) <(bin/sv2v test/core/inc.sv)`. + +[functions.sh]: lib/functions.sh +[main GitHub Actions workflow]: ../.github/workflows/main.yaml diff --git a/test/relong/README.md b/test/relong/README.md index ca4e7c8..fc63e34 100644 --- a/test/relong/README.md +++ b/test/relong/README.md @@ -1,24 +1,14 @@ -# relong Tests +# Reid Long's tests -These tests are borrowed from Reid Long's [HDL Examples -repository](https://bitbucket.org/ReidLong/hdl-examples). That repository was -intended to provide examples for how the conversions in this project could be -done. sv2v does not necessarily convert code as demonstrated in the examples. -Notably, sv2v does not create `generate` blocks when converted vectors with -multiple packed dimensions, uses `localparam`s rather than macros for `enum` -conversion, and converts `struct` literals to concatenations, rather than -multiple statements. +These tests are borrowed from Reid Long's [HDL Examples repository]. Early in +sv2v's development, Reid produced these examples to demonstrate how sv2v might +perform its conversions. sv2v does not necessarily convert code in the same way, +but its output should behave equivalently. Notably, sv2v does not create +`generate` blocks when flattening vectors with multiple packed dimensions, uses +`localparam`s rather than macros to convert `enum`s, and converts `struct` +pattern literals as concatenations, rather than multiple statements. -Each test case (say, "foo") is comprised of the following files: - -1. `foo.sv`: original SystemVerilog -2. `foo.v`: hand-converted Verilog -3. `foo_tb.v`: basic testbench exercising the given modules - -The SystemVerilog source file is converted to Verilog using sv2v, and then both -the converted file and the reference Verilog are simulated using Icarus Verilog. -This produces VCD files for each which are expected to match exactly, except for -the timestamp. +[HDL Examples repository]: https://bitbucket.org/ReidLong/hdl-examples ## Modifications @@ -29,7 +19,7 @@ corresponding versions in the source repository. Though some tools allow for stray semicolons, `iverilog` does not. 2. `array.v` previously had a custom implementation of `$clog2`, which was removed. -3. `cache_request.sv` was modified to include a plain decimal literal to ensure +3. `cache_request.sv` was modified to include a plain decimal literal to provide coverage beyond the unbased-unsized literals. 4. The `cache_request2` test is omitted. It was only an example for debugging a VCS-specific issue encountered with `cache_request`.