223 lines
10 KiB
HTML
223 lines
10 KiB
HTML
<!DOCTYPE html>
|
||
|
||
<html lang="en" data-content_root="../">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
|
||
|
||
<title>The Regression Test Suite — Icarus Verilog documentation</title>
|
||
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=fa44fd50" />
|
||
<link rel="stylesheet" type="text/css" href="../_static/alabaster.css?v=cb25574f" />
|
||
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
|
||
<script src="../_static/doctools.js?v=888ff710"></script>
|
||
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
|
||
<link rel="icon" href="../_static/favicon.ico"/>
|
||
<link rel="index" title="Index" href="../genindex.html" />
|
||
<link rel="search" title="Search" href="../search.html" />
|
||
<link rel="next" title="Files With Version Information" href="version_stamps.html" />
|
||
<link rel="prev" title="Getting Started as a Contributor" href="getting_started.html" />
|
||
|
||
<link rel="stylesheet" href="../_static/custom.css" type="text/css" />
|
||
|
||
|
||
<meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
|
||
|
||
</head><body>
|
||
|
||
|
||
<div class="document">
|
||
<div class="documentwrapper">
|
||
<div class="bodywrapper">
|
||
|
||
|
||
<div class="body" role="main">
|
||
|
||
<section id="the-regression-test-suite">
|
||
<h1>The Regression Test Suite<a class="headerlink" href="#the-regression-test-suite" title="Link to this heading">¶</a></h1>
|
||
<p>Icarus Verilog development includes a regression test suite that is included
|
||
along with the source. The “ivtest” directory contains the regression test
|
||
suite, and this suite is used by the github actions as continuous integration
|
||
to make sure the code is always going forward.</p>
|
||
<p>NOTE: There are scripts written in perl to run the regression tests, but they
|
||
are being gradually replaced with a newer set of scripts. It is the newer
|
||
method that is described here.</p>
|
||
<section id="test-descriptions">
|
||
<h2>Test Descriptions<a class="headerlink" href="#test-descriptions" title="Link to this heading">¶</a></h2>
|
||
<p>Regression tests are listed in the regress-vvp.list file. Each line lists the
|
||
name of the test and the path to the dest description. The list file is
|
||
therefore pretty simple, and all the description of the test is in the
|
||
description file:</p>
|
||
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">macro_str_esc vvp_tests/macro_str_esc.json</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>The “name” is a simple name, and the test-description-file is the path (relative
|
||
the ivtest directory) to the description file. A simple test description file
|
||
is a JSON file, like this:</p>
|
||
<div class="highlight-java notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
|
||
<span class="w"> </span><span class="s">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s">"normal"</span><span class="p">,</span>
|
||
<span class="w"> </span><span class="s">"source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s">"macro_str_esc.v"</span><span class="p">,</span>
|
||
<span class="w"> </span><span class="s">"gold"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s">"macro_str_esc"</span>
|
||
<span class="p">}</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>This description file contains all the information that the vvp_reg.py script
|
||
needs to run the regression test. The sections below describe the keys and
|
||
values in the description file dictionary.</p>
|
||
<section id="source-required">
|
||
<h3>source (required)<a class="headerlink" href="#source-required" title="Link to this heading">¶</a></h3>
|
||
<p>This specifies the name of the source file. The file is actually to be found
|
||
in the ivltests/ directory.</p>
|
||
</section>
|
||
<section id="type-required">
|
||
<h3>type (required)<a class="headerlink" href="#type-required" title="Link to this heading">¶</a></h3>
|
||
<p>This describes the kind of test to run. The valid values are:</p>
|
||
<ul class="simple">
|
||
<li><p><strong>normal</strong> - Compile the source using the iverilog compiler vvp target, and if
|
||
that succeeds execute it using the vvp command. If there is no gold file
|
||
specified, then look for an output line with the “PASSED” string.</p></li>
|
||
<li><p><strong>normal-vlog95</strong> - This is similar to the normal case, but uses
|
||
the -tvlog95 target in a first pass to generate simplified verilog, then a
|
||
regular iverilog command with the -tvvp target to generate the actual
|
||
executable. This tests the -tvlog95 target.</p></li>
|
||
<li><p><strong>NI</strong> - Mark the test as not implemented. The test will be skipped without
|
||
running or reporting an error.</p></li>
|
||
<li><p><strong>CE</strong> - Compile, but expect the compiler to fail. This means the compiler
|
||
command process must return an error exit.</p></li>
|
||
<li><p><strong>EF</strong> - Compile and run, but expect the run time to fail. This means the
|
||
run time program must return an error exit.</p></li>
|
||
</ul>
|
||
</section>
|
||
<section id="gold-optional">
|
||
<h3>gold (optional)<a class="headerlink" href="#gold-optional" title="Link to this heading">¶</a></h3>
|
||
<p>If this is specified, it replaces the “Passed” condition with a comparison of
|
||
the output with a gold file. The argument is the name of the gold file set,
|
||
which will be found in the “gold/” directory. The name here is actually the
|
||
basename of the gold files, with separate actual gold files for the iverilog
|
||
and vvp stderr and stdout. For example, if a “normal” test includes a gold
|
||
file, then the program is compiled and run, and the outputs are compared with
|
||
the gold file to make sure it ran properly.</p>
|
||
<p>The way the regression suite works, there are 4 log files created for each
|
||
test:</p>
|
||
<ul class="simple">
|
||
<li><p>foo-iverilog-stdout.log</p></li>
|
||
<li><p>foo-iverilog-stderr.log</p></li>
|
||
<li><p>foo-vvp-stdout.log</p></li>
|
||
<li><p>foo-vvp-stderr.log</p></li>
|
||
</ul>
|
||
<p>The “gold” value is the name of the gold file set. If the gold value is “foo”,
|
||
Then the actual gold files are called:</p>
|
||
<ul class="simple">
|
||
<li><p>gold/foo-iverilog-stdout.gold</p></li>
|
||
<li><p>gold/foo-iverilog-stderr.gold</p></li>
|
||
<li><p>gold/foo-vvp-stdout.gold</p></li>
|
||
<li><p>gold/foo/vvp-stderr.gold</p></li>
|
||
</ul>
|
||
<p>If any of those files is empty, then the gold file doesn’t need to be
|
||
present at all. The log files and the gold files are compared byte for
|
||
byte, so if the output you are getting is correct, then copy the log to
|
||
the corresponding gold, and you’re done.</p>
|
||
<p>If the run type is “CE” or “RE”, then the gold files still work, and can
|
||
be used to check that the error message is correct. If the gold file setting
|
||
is present, the error return is required, and also the gold files must match.</p>
|
||
</section>
|
||
<section id="iverilog-args-optional">
|
||
<h3>iverilog-args (optional)<a class="headerlink" href="#iverilog-args-optional" title="Link to this heading">¶</a></h3>
|
||
<p>If this is specified, it is a list of strings that are passed as arguments to
|
||
the iverilog command line.</p>
|
||
</section>
|
||
<section id="vvp-args-optional">
|
||
<h3>vvp-args (optional)<a class="headerlink" href="#vvp-args-optional" title="Link to this heading">¶</a></h3>
|
||
<p>If this is specified, it is a list of strings that are passed as arguments to
|
||
the vvp command. These arguments go before the vvp input file that is to be
|
||
run.</p>
|
||
</section>
|
||
<section id="vvp-args-extended-optional">
|
||
<h3>vvp-args-extended (optional)<a class="headerlink" href="#vvp-args-extended-optional" title="Link to this heading">¶</a></h3>
|
||
<p>If this is specified, it is a lost of strings that are passed as arguments to
|
||
the vvp command. These are extended arguments, and are placed after the vvp
|
||
input file that is being run. This is where you place things like plusargs.</p>
|
||
</section>
|
||
</section>
|
||
</section>
|
||
|
||
|
||
</div>
|
||
|
||
</div>
|
||
</div>
|
||
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
|
||
<div class="sphinxsidebarwrapper">
|
||
<h1 class="logo"><a href="../index.html">Icarus Verilog</a></h1>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<h3>Navigation</h3>
|
||
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
|
||
<ul class="current">
|
||
<li class="toctree-l1"><a class="reference internal" href="../usage/index.html">Icarus Verilog Usage</a></li>
|
||
<li class="toctree-l1"><a class="reference internal" href="../targets/index.html">The Icarus Verilog Targets</a></li>
|
||
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Icarus Verilog Developer Support</a><ul class="current">
|
||
<li class="toctree-l2"><a class="reference internal" href="getting_started.html">Getting Started as a Contributor</a></li>
|
||
<li class="toctree-l2 current"><a class="current reference internal" href="#">The Regression Test Suite</a></li>
|
||
<li class="toctree-l2"><a class="reference internal" href="version_stamps.html">Files With Version Information</a></li>
|
||
<li class="toctree-l2"><a class="reference internal" href="guide/index.html">Developer Guide</a></li>
|
||
<li class="toctree-l2"><a class="reference internal" href="glossary.html">Glossary</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
|
||
<div class="relations">
|
||
<h3>Related Topics</h3>
|
||
<ul>
|
||
<li><a href="../index.html">Documentation overview</a><ul>
|
||
<li><a href="index.html">Icarus Verilog Developer Support</a><ul>
|
||
<li>Previous: <a href="getting_started.html" title="previous chapter">Getting Started as a Contributor</a></li>
|
||
<li>Next: <a href="version_stamps.html" title="next chapter">Files With Version Information</a></li>
|
||
</ul></li>
|
||
</ul></li>
|
||
</ul>
|
||
</div>
|
||
<div id="searchbox" style="display: none" role="search">
|
||
<h3 id="searchlabel">Quick search</h3>
|
||
<div class="searchformwrapper">
|
||
<form class="search" action="../search.html" method="get">
|
||
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
|
||
<input type="submit" value="Go" />
|
||
</form>
|
||
</div>
|
||
</div>
|
||
<script>document.getElementById('searchbox').style.display = "block"</script>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</div>
|
||
</div>
|
||
<div class="clearer"></div>
|
||
</div>
|
||
<div class="footer">
|
||
©2024-2025, Stephen Williams.
|
||
|
||
|
|
||
Powered by <a href="http://sphinx-doc.org/">Sphinx 7.2.6</a>
|
||
& <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.12</a>
|
||
|
||
|
|
||
<a href="../_sources/developer/regression_tests.rst.txt"
|
||
rel="nofollow">Page source</a>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
</body>
|
||
</html> |