126 lines
4.4 KiB
ReStructuredText
126 lines
4.4 KiB
ReStructuredText
|
|
The Regression Test Suite
|
|
=========================
|
|
|
|
Icarus Verilog development includes a regression test suite that is included
|
|
along with the source. The "ivtest" directory contains the regression test
|
|
suite, and this suite is used by the github actions as continuous integration
|
|
to make sure the code is always going forward.
|
|
|
|
NOTE: There are scripts written in perl to run the regression tests, but they
|
|
are being gradually replaced with a newer set of scripts. It is the newer
|
|
method that is described here.
|
|
|
|
Test Descriptions
|
|
-----------------
|
|
|
|
Regression tests are listed in the regress-vvp.list file. Each line lists the
|
|
name of the test and the path to the dest description. The list file is
|
|
therefore pretty simple, and all the description of the test is in the
|
|
description file:
|
|
|
|
.. code-block:: console
|
|
|
|
macro_str_esc vvp_tests/macro_str_esc.json
|
|
|
|
The "name" is a simple name, and the test-description-file is the path (relative
|
|
the ivtest directory) to the description file. A simple test description file
|
|
is a JSON file, like this:
|
|
|
|
.. code-block:: java
|
|
|
|
{
|
|
"type" : "normal",
|
|
"source" : "macro_str_esc.v",
|
|
"gold" : "macro_str_esc"
|
|
}
|
|
|
|
This description file contains all the information that the vvp_reg.py script
|
|
needs to run the regression test. The sections below describe the keys and
|
|
values in the description file dictionary.
|
|
|
|
source (required)
|
|
^^^^^^^^^^^^^^^^^
|
|
This specifies the name of the source file. The file is actually to be found
|
|
in the ivltests/ directory.
|
|
|
|
|
|
type (required)
|
|
^^^^^^^^^^^^^^^
|
|
|
|
This describes the kind of test to run. The valid values are:
|
|
|
|
* **normal** - Compile the source using the iverilog compiler vvp target, and if
|
|
that succeeds execute it using the vvp command. If there is no gold file
|
|
specified, then look for an output line with the "PASSED" string.
|
|
|
|
* **normal-vlog95** - This is similar to the normal case, but uses
|
|
the -tvlog95 target in a first pass to generate simplified verilog, then a
|
|
regular iverilog command with the -tvvp target to generate the actual
|
|
executable. This tests the -tvlog95 target.
|
|
|
|
* **NI** - Mark the test as not implemented. The test will be skipped without
|
|
running or reporting an error.
|
|
|
|
* **CE** - Compile, but expect the compiler to fail. This means the compiler
|
|
command process must return an error exit.
|
|
|
|
* **EF** - Compile and run, but expect the run time to fail. This means the
|
|
run time program must return an error exit.
|
|
|
|
gold (optional)
|
|
^^^^^^^^^^^^^^^
|
|
|
|
If this is specified, it replaces the "Passed" condition with a comparison of
|
|
the output with a gold file. The argument is the name of the gold file set,
|
|
which will be found in the "gold/" directory. The name here is actually the
|
|
basename of the gold files, with separate actual gold files for the iverilog
|
|
and vvp stderr and stdout. For example, if a "normal" test includes a gold
|
|
file, then the program is compiled and run, and the outputs are compared with
|
|
the gold file to make sure it ran properly.
|
|
|
|
The way the regression suite works, there are 4 log files created for each
|
|
test:
|
|
|
|
* foo-iverilog-stdout.log
|
|
* foo-iverilog-stderr.log
|
|
* foo-vvp-stdout.log
|
|
* foo-vvp-stderr.log
|
|
|
|
The "gold" value is the name of the gold file set. If the gold value is "foo",
|
|
Then the actual gold files are called:
|
|
|
|
* gold/foo-iverilog-stdout.gold
|
|
* gold/foo-iverilog-stderr.gold
|
|
* gold/foo-vvp-stdout.gold
|
|
* gold/foo/vvp-stderr.gold
|
|
|
|
If any of those files is empty, then the gold file doesn't need to be
|
|
present at all. The log files and the gold files are compared byte for
|
|
byte, so if the output you are getting is correct, then copy the log to
|
|
the corresponding gold, and you're done.
|
|
|
|
If the run type is "CE" or "RE", then the gold files still work, and can
|
|
be used to check that the error message is correct. If the gold file setting
|
|
is present, the error return is required, and also the gold files must match.
|
|
|
|
iverilog-args (optional)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
If this is specified, it is a list of strings that are passed as arguments to
|
|
the iverilog command line.
|
|
|
|
vvp-args (optional)
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
|
|
If this is specified, it is a list of strings that are passed as arguments to
|
|
the vvp command. These arguments go before the vvp input file that is to be
|
|
run.
|
|
|
|
vvp-args-extended (optional)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
If this is specified, it is a lost of strings that are passed as arguments to
|
|
the vvp command. These are extended arguments, and are placed after the vvp
|
|
input file that is being run. This is where you place things like plusargs.
|