Commentary: Test documentation, bug486. Merge from Bennett.
This commit is contained in:
Wilson Snyder
parent
2d8feabe54
commit
c361932fe2
|
|
@ -521,7 +521,9 @@ changing debug verbosity. Enabled automatically when --debug specified.
|
|||
=item --debugi-<srcfile> <level>
|
||||
|
||||
Rarely needed - for developer use. Set internal debugging level globally
|
||||
or on the specified source file to the specified level.
|
||||
to the specified debug level (1-10) or set the specified source file to the
|
||||
specified level. Higher levels produce more detailed messages (plain
|
||||
C<--debug> is equivalent to C<--debugi 4>).
|
||||
|
||||
=item +define+I<var>+I<value>
|
||||
|
||||
|
|
@ -531,9 +533,10 @@ standard across Verilog tools while -D is an alias for GCC compatibility.
|
|||
=item --dump-tree
|
||||
|
||||
Rarely needed. Enable writing .tree debug files with dumping level 3,
|
||||
which dumps the standard critical stages. --dump-tree is enabled
|
||||
automatically with --debug, so "--debug --no-dump-tree" may be useful if
|
||||
the dump files are large and not desired.
|
||||
which dumps the standard critical stages. For details on the format see
|
||||
the Verilator Internals manual. --dump-tree is enabled automatically with
|
||||
--debug, so "--debug --no-dump-tree" may be useful if the dump files are
|
||||
large and not desired.
|
||||
|
||||
=item --dump-treei
|
||||
|
||||
|
|
@ -3395,12 +3398,27 @@ follows:
|
|||
cp -p t/t_EXAMPLE.pl t/t_BUG.pl
|
||||
cp -p t/t_EXAMPLE.v t/t_BUG.v
|
||||
|
||||
There are many hits on how to write a good test in the driver.pl
|
||||
documentation which can be seen by running:
|
||||
|
||||
cd $VERILATOR_ROOT # Need the original distribution kit
|
||||
test_regress/driver.pl --help
|
||||
|
||||
Edit t/t_BUG.pl to suit your example; you can do anything you want in the
|
||||
Verilog code there; just make sure it retains the single clk input and no
|
||||
outputs. Now, the following should fail:
|
||||
|
||||
cd $VERILATOR_ROOT # Need the original distribution kit
|
||||
cd test_regress
|
||||
t/t_BUG.pl
|
||||
t/t_BUG.pl # Run on Verilator
|
||||
t/t_BUG.pl --debug # Run on Verilator, passing --debug to Verilator
|
||||
t/t_BUG.pl --vcs # Run on a commercial simulator
|
||||
t/t_BUG.pl --nc|--iv|--ghdl # Likewise on other simulators
|
||||
|
||||
The test driver accepts a number of options, many of which mirror the main
|
||||
Verilator option. For example the previous test could have been run with
|
||||
debugging enabled. The full set of test options can be seen by running
|
||||
driver.pl --help as shown above.
|
||||
|
||||
Finally, report the bug using the bug tracker at
|
||||
L<http://www.veripool.org/verilator>. The bug will become publicly
|
||||
|
|
|
|||
|
|
@ -336,10 +336,10 @@ AST.
|
|||
|
||||
=head1 TESTING
|
||||
|
||||
To write a test see notes in the forum and in the verilator.txt manual.
|
||||
To write a test see the BUGS section of the Verilator primary manual, and
|
||||
the documentation in:
|
||||
|
||||
Note you can run the regression tests in parallel; see the
|
||||
test_regress/driver.pl script -j flag.
|
||||
test_regress/t/driver.pl --help
|
||||
|
||||
=head1 DEBUGGING
|
||||
|
||||
|
|
|
|||
|
|
@ -1469,27 +1469,226 @@ The driver reports the number of tests which pass, fail, skipped (some
|
|||
resource required by the test is not available, such as SystemC), or are
|
||||
unsupported (buggy or require a feature change before will pass.)
|
||||
|
||||
=head1 ARGUMENTS
|
||||
There are hundreds of tests, and for faster completion you may want to run
|
||||
the regression tests with CCACHE enabled and in parallel on a machine with
|
||||
many cores. See the -j option.
|
||||
|
||||
=head1 TEST CONFIGURATION
|
||||
|
||||
The Perl script (e.g. C<test_regres/t/t_EXAMPLE.pl>) controls how the test
|
||||
will run by driver.pl. In general it includes a call to the C<compile>
|
||||
subroutine to compile the test with Verilator (or an alternative
|
||||
simulator), followed by a call to the C<execute> subroutine to run the
|
||||
test. Compile-only tests omit the call to C<execute>.
|
||||
|
||||
If those complete, the script calls C<ok(1)> to increment the count of
|
||||
successful tests and then returns 1 as its result.
|
||||
|
||||
Both C<compile> and C<execute> take an optional argument hash table to
|
||||
control their behavior. For example:
|
||||
|
||||
compile (
|
||||
verilator_flags2 => ["--lint-only"],
|
||||
fails => 1,
|
||||
);
|
||||
|
||||
indicates that when compiling this test, the C<--lint-only> flag should be
|
||||
passed and that the test is expected to fail.
|
||||
|
||||
The full list of arguments can be found by looking at the C<driver.pl>
|
||||
source code. Some commonly used arguments are:
|
||||
|
||||
=over 4
|
||||
|
||||
=item all_run_flags
|
||||
|
||||
A list of flags to be passed when running the simulator (Verilated model or
|
||||
one of the other simulators).
|
||||
|
||||
=item check_finished
|
||||
|
||||
Set to 1 to indicate successful completion of the test is indicated by the
|
||||
string C<*-* All Finished *-*> being printed on standard output. This is
|
||||
the normal way for successful tests to finish.
|
||||
|
||||
=item expect
|
||||
|
||||
A quoted list of strings or regular expression to be matched in the
|
||||
output. See </HINTS ON WRITING TESTS> for more detail on how this argument
|
||||
should be used.
|
||||
|
||||
=item fails
|
||||
|
||||
Set to 1 to indicate this step (C<compile> or C<execute>) is expected to
|
||||
fail. Tests that are expected to fail generally have _bad in their
|
||||
filename.
|
||||
|
||||
=item make_main
|
||||
|
||||
Set to 0 to disable the automatic creation of a C++ test wrapper (for
|
||||
example when a hand-written test wrapper is provided using C<--exe>).
|
||||
|
||||
=item make_top_shell
|
||||
|
||||
Set to 0 to disable the automatic creation of a top level shell to run the
|
||||
executable (for example when a hand-written test wrapper is provided using
|
||||
C<--exe>).
|
||||
|
||||
=item nc_flags
|
||||
|
||||
=item nc_flags2
|
||||
|
||||
=item nc_run_flags
|
||||
|
||||
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
|
||||
for use with the Cadence NC simulator.
|
||||
|
||||
=item iv_flags
|
||||
|
||||
=item iv_flags2
|
||||
|
||||
=item iv_run_flags
|
||||
|
||||
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
|
||||
for use with the Icarus Verilog simulator.
|
||||
|
||||
=item v_flags
|
||||
|
||||
A list of standard Verilog simulator flags to be passed to the simulator
|
||||
compiler (Verilator or one of the other simulators). This list is create
|
||||
by the driver and rarely changed, use v_flags2 instead.
|
||||
|
||||
=item v_flags2
|
||||
|
||||
A list of standard Verilog simulator flags to be passed to the simulator
|
||||
compiler (Verilator or one of the other simulators). Unlike v_flags, these
|
||||
options may be overridden in some simulation files.
|
||||
|
||||
Similar sets of flags exist for atsim, GHDL, Cadence NC and Synopsys VCS.
|
||||
|
||||
=item vcs_flags
|
||||
|
||||
=item vcs_flags2
|
||||
|
||||
=item vcs_run_flags
|
||||
|
||||
The equivalent of C<v_flags>, C<v_flags2> and C<all_run_flags>, but only
|
||||
for use with the Synopsys VCS simulator.
|
||||
|
||||
=item verilator_flags
|
||||
|
||||
=item verilator_flags2
|
||||
|
||||
The equivalent of C<v_flags> and C<v_flags2>, but only for use with
|
||||
Verilator. If a flag is a standard flag (+incdir for example) v_flags2
|
||||
should be used instead.
|
||||
|
||||
=back
|
||||
|
||||
=head2 HINTS ON WRITING TESTS
|
||||
|
||||
There is generally no need for the test to create its own main program or
|
||||
top level shell as the driver creates one automatically, however some tests
|
||||
require their own C++ or SystemC test harness. This is commonly given the
|
||||
same name as the test, but with .cpp as suffix
|
||||
(C<test_regress/t/t_EXAMPLE.cpp>). This can be specified as follows:
|
||||
|
||||
compile (
|
||||
make_top_shell => 0,
|
||||
make_main => 0,
|
||||
verilator_flags2 => ["--exe $Self->{t_dir}/$Self->{name}.cpp"], );
|
||||
|
||||
Tests should be self-checking, rather than producing lots of output. If a
|
||||
test succeeds it should print C<*-* All Finished *-*> to standard output
|
||||
and terminate (in Verilog C<$finish>), if not it should just stop (in
|
||||
Verilog C<$stop>) as that signals an error.
|
||||
|
||||
If termination should be triggered from the C++ wrapper, the following code
|
||||
can be used:
|
||||
|
||||
vl_fatal (__FILE__, __LINE__, "dut", "<error message goes here>");
|
||||
exit (1);
|
||||
|
||||
This can be particularly useful if checking that the Verilator model has
|
||||
not unexpectedly terminated.
|
||||
|
||||
if (Verilated::gotFinish ()) {
|
||||
vl_fatal (__FILE__, __LINE__, "dut", "<error message goes here>");
|
||||
exit (1);
|
||||
}
|
||||
|
||||
Where it might be useful for a test to produce output, it should qualify
|
||||
this with C<TEST_VERBOSE>. For example in Verilog:
|
||||
|
||||
`ifdef TEST_VERBOSE
|
||||
$write ("Conditional generate if MASK [%1d] = %d\n", g, MASK[g]);
|
||||
`endif
|
||||
|
||||
Or in a hand-written C++ wrapper:
|
||||
|
||||
#ifdef TEST_VERBOSE
|
||||
cout << "Read a = " << a << endl;
|
||||
#endif
|
||||
|
||||
The C<expect> argument should not generally be used to decide if a test has
|
||||
succeeded. However, in the case of tests that are designed to fail at
|
||||
compile time, it is the only option. For example:
|
||||
|
||||
compile (
|
||||
v_flags2 => ["--lint-only"],
|
||||
fails=>1,
|
||||
expect=>
|
||||
q{%Error: t/t_gen_cond_bitrange_bad.v:\d+: Selection index out of range: 2:2 outside 1:0
|
||||
%Error: t/t_gen_cond_bitrange_bad.v:\d+: Selection index out of range: 2:2 outside 1:0
|
||||
%Error: t/t_gen_cond_bitrange_bad.v:\d+: Selection index out of range: 2:2 outside 1:0
|
||||
%Error: t/t_gen_cond_bitrange_bad.v:\d+: Selection index out of range: 2:2 outside 1:0
|
||||
%Error: Exiting due to .*},
|
||||
);
|
||||
|
||||
The strings to match should be made general - for example the line numbers
|
||||
in the Verilog should not be critical and the total number of errors should
|
||||
not matter. This makes it easier to extend or modify the test in future.
|
||||
|
||||
=head1 DRIVER ARGUMENTS
|
||||
|
||||
=over 4
|
||||
|
||||
=item --atsim
|
||||
|
||||
Run using ATSIM.
|
||||
Run using ATSIM simulator.
|
||||
|
||||
=item --benchmark [<cycles>]
|
||||
|
||||
Show execution times of each step. If an optional number is given,
|
||||
specifies the number of simulation cycles (for tests that support it).
|
||||
|
||||
=item --debug
|
||||
|
||||
Same as C<verilator --debug>: Use the debug version of Verilator which
|
||||
enables additional assertions, debugging messages, and structure dump
|
||||
files.
|
||||
|
||||
=item --debugi(-<srcfile>) <level>
|
||||
|
||||
Same as C<verilator --debugi level>: Set Verilator internal debugging level
|
||||
globally to the specified debug level (1-10) or set the specified source
|
||||
file to the specified level. Higher levels produce more detailed messages
|
||||
(plain C<--debug> is equivalent to C<--debugi 4>).
|
||||
|
||||
=item --dump-tree
|
||||
|
||||
Same as C<verilator --dump-tree>: Enable Verilator writing .tree debug
|
||||
files with dumping level 3, which dumps the standard critical stages. For
|
||||
details on the format see the Verilator Internals manual.
|
||||
|
||||
=item --gdb
|
||||
|
||||
Run Verilator under the debugger.
|
||||
Same as C<verilator --gdb>: Run Verilator under the debugger.
|
||||
|
||||
=item --gdbbt
|
||||
|
||||
Run Verilator under the debugger, only to print backtrace information.
|
||||
Requires --debug.
|
||||
Same as C<verilator --gdbbt>: Run Verilator under the debugger, only to
|
||||
print backtrace information. Requires --debug.
|
||||
|
||||
=item --gdbsim
|
||||
|
||||
|
|
@ -1497,7 +1696,7 @@ Run Verilator generated executable under the debugger.
|
|||
|
||||
=item --ghdl
|
||||
|
||||
Run using GHDL.
|
||||
Run using GHDL simulator.
|
||||
|
||||
=item --golden
|
||||
|
||||
|
|
@ -1507,20 +1706,24 @@ Update golden files, equivalent to setting HARNESS_UPDATE_GOLDEN=1.
|
|||
|
||||
Displays this message and program version and exits.
|
||||
|
||||
=item --iverilog
|
||||
|
||||
Run using Icarus Verilog simulator.
|
||||
|
||||
=item --j #
|
||||
|
||||
Run number of parallel tests, or 0 to determine the count based on the
|
||||
number of cores installed. Requires Perl's Parallel::Forker package.
|
||||
|
||||
=item --nc
|
||||
|
||||
Run using Cadence NC-Verilog simulator.
|
||||
|
||||
=item --optimize
|
||||
|
||||
Randomly turn on/off different optimizations. With specific flags,
|
||||
use those optimization settings
|
||||
|
||||
=item --nc
|
||||
|
||||
Run using NC-Verilog.
|
||||
|
||||
=item --site
|
||||
|
||||
Run site specific tests also.
|
||||
|
|
@ -1529,17 +1732,22 @@ Run site specific tests also.
|
|||
|
||||
Stop on the first error.
|
||||
|
||||
=item --trace
|
||||
|
||||
Set the simulator specific flags to request waveform tracing.
|
||||
|
||||
=item --vcs
|
||||
|
||||
Run using VCS.
|
||||
Run using Synopsys VCS simulator.
|
||||
|
||||
=item --verbose
|
||||
|
||||
Enable test verbose messages.
|
||||
Compile and run the test in verbose mode. This means C<TEST_VERBOSE> will
|
||||
be defined for the test (Verilog and any C++/SystemC wrapper).
|
||||
|
||||
=item --vlt
|
||||
|
||||
Run using Verilator.
|
||||
Run using Verilator. Defaults set unless another simulator is requested.
|
||||
|
||||
=back
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue