Introduce a new DfgPrev vertex, representing the value of a variable
before any in-graph assignments. This can be used to break all remaining
cycles in the graph, so all Dfgs become acyclic after V3DfgBreakCycles.
The circular dataflow is still represented, and is taken care of by the
scheduler, it is just the DfgGraph that represents the logic that
becomes acyclic.
This makes V3DfgBreakCycles a mandatory transform, so drop the disabling
-fno-dfg-break-cycles option (still parsed, but has no effect).
Note the effect of this is small, as most cycles can be fixed up by
driver tracing, which is unchanged, but this is required for some
upcoming work.
splitCheck() cuts a function's top level statement list on node count
alone, ignoring AstVar declarations in that list. localizeVars() puts the
dynamic trigger temporaries there, which a 'wait fork' reaches, so the
declaration could land in one sub-function and its references in another.
Sub-functions are emitted as separate C++ functions, so the output failed
to compile:
error: '__Vtrigprevexpr_h5d9da2ce__0' was not declared in this scope
V3InlineCFuncs could also inline the sub-function holding the declaration
and free the AstVar while other sub-functions still referenced it, which
--debug reports as a broken link and which segfaults an -O3 build.
The existing "Can't split function with local variables" assertion only
checked AstCFunc::varsp(), not declarations among the statements.
Only allow a sub-function boundary where it does not separate a local
declaration from a reference to it. This keeps the temporaries function
local, as #6859 requires, while restoring the guarantee #5822 made that
splitting cannot orphan them.
Signed-off-by: Marco Brambilla <[email protected]>
The REPLACE_SHIFTL_SHIFTL and REPLACE_SHIFTR_SHIFTR peephole optimizations
fold '(a << b) << c' into 'a << (b + c)' (and likewise for '>>'), but computed
'b + c' in the width of the shift-amount operand. When 'b + c' overflows that
width it wraps around, producing a too-small shift amount and a wrong result.
Compute the sum one bit wider than the amounts so it cannot overflow.
Fixes#7955
PR #7902 uncovered a pre-existing bug in multi-threaded scheduling,
where we can end up with an un-ordered R-W hazard in the MTask graph,
resulting in non-deterministic runtime behaviour.
This is extremely hard to actually trigger on a small example, so using
ThreadSanitizer to flag it, which can identify the race reliably.
In this patch:
- Add configure and `verilator --get-supported TSAN` to check if the
configured compiler supports ThreadSanitizer
- Add a --tsan option to the test driver.py which builds the test with
thread sanitizer (similar idea to --gdbsim).
- Add a tests.enable_tsan() method to allow turning on TSan in the test
Python file.
- Add a suppressions file that waives TSan errors in the runtime library
- Finally add `t_sched_hybrid_hazard` that demonstrates the data race
triggered after #7902. This is currently expected failing, fix later.
With the suppression, there are 17 vltmt tests failing due races in the
generated code. (Using `driver.py --vltmt --tsan --quiet -j0`)
Recognize the common single-bit scan loop idioms in V3Unroll (before it
unrolls) and lower them to bit-reduction primitives, replacing a literal
W-iteration loop with one intrinsic-backed expression:
target=0; for (i=0;i<W;i++) if (vec[i]) target = i + 1; -> $mostsetbitp1(vec)
target=0; for (i=0;i<W;i++) if (vec[i]) target = target + 1; -> $countones(vec)
The leading-one form lowers to a new AstMostSetBitP1 node, emitted as
VL_MOSTSETBITP1_{I,Q,W}; those runtime helpers now use __builtin_clz where
available (same pattern as VL_REDXOR's __builtin_parity), with the existing
bit scan as fallback. The count-ones form reuses AstCountOnes ($countones,
popcount); as the DFG requires a 32-bit countones result it is built at 32
bits and narrowed to the accumulator width with a select.
Matching is structural to stay sound: the index must start at 0, increment
by exactly 1, and scan all W==width(vec) bits via a single 1-bit select of a
distinct vector, with the target pre-zeroed and no else branch. The loop
bound is accepted as a strict ascending 'idx < W' written either way and
signed or unsigned (Gt/GtS/Lt/LtS). Gated by -fbit-scan-loops (on at -O).
Adds t_bit_scan_loops (I/Q/W, count-ones and unsigned-index positives;
step-2, start-1, idx*2+1, vec[idx+1], target=idx and W!=width negatives, all
self-checked and asserted via --stats not to lower) plus t_bit_scan_loops_off
for the disable flag.
Motivated by a transformer inference design whose 80-bit leading-one detector
ran every cycle (~37% of runtime); the lowering is worth ~39% there.
Remove the expression combination counts from the default stats file,
and add a new `--dump-ast-patterns` option, which will dump new
`*_ast_patterns_*.txt` files. These contain the expression combinations
in a similar S-expression format as Dfg already produces with
`--dump-dfg-stats`. These dumps are not produced by just `--stats` as
they are fairly expensive to compute. Currently the new option will dump
at two points: just before we change to C types via widthMin usage, and
just before emit.
Previously V3InlineCFuncs inlined call sites but never deleted the now
dead callees. Also missed a lot of opportunities due to evaluation order.
Rewrite using a graph based algorithm, using only a single traversal of
the netlist. This is clearer, more accurate, and faster at compile time.
Also add a clean -fno-inline-cfuncs disable. Setting the limits to 0
still disables inlining, except of empty functions, which can be inlined
with 0 limits (they are no ops). It will also prune unused functions
without -fno-inline-cfuncs.
Pass now also respects `--output-split`
Extend the decoder-pattern case optimization to selectors that are too
wide for a full 2^width lookup table. A decoder-pattern case (where
every case item assigns constants to a fixed set of LHSs) is lowered to
a new AstMachMasked expression. AstMachMasked is emitted as a run-time
VL_MATCHMASKEd_* function call. It contains a packed constant pool table,
'matchp', which is a list of '(mask, bits)' pairs. At runtime, the index of the
first matching entry is returned, and is used to index a value table. This single
(albeit complicated) expression can replace large if-else trees whole, resulting
in much more compact code with fewer static hard to predict branches. It
is worth about 10% speed and 30% code size in some designs.
Example:
```systemverilog
logic [39:0] sel;
always_comb
casez (sel)
40'b???????????????????????????????????????1: out = 8'h01;
40'b??????????????????????????????????????1?: out = 8'h02;
40'b?????????????????????????????????????1??: out = 8'h03;
default: out = 8'hff;
endcase
```
is compiled to:
```c++
out = TABLE_value[VL_MATCHMASKED_Q(sel, CONST_match)];
```
Where 'CONST_match' contains 4 entries, of a 40-bit mask and 40-bit bit
pattern each, and 'TABLE_value' contains 4 entries of the corresponding
8-bit results. (Entries are aligned to word boundaries to avoid runtime
bit swizzling)
Recognize "decoder" case statements (where every case item only assigns
constants to a fixed set of left-hand sides) and replace them with a
single packed constant lookup table indexed by the case expression.
Small tables are materialized inline in the generated code, and are
always optimized. Larger ones are placed in the constant pool and only
optimized if deemed beneficial over branches.
While this slightly conflicts with V3Table, and is not worth that much
on it's own, there will be a follow up patch that converts more cases of
this form which will be much more valuable. This patch does the
necessary analysis and the simple table conversion when possible.
Split -fcase into -fcase-table (this new conversion) and -fcase-tree (the
existing bitwise branch-tree conversion); -fno-case is now an alias for
both.
Default branches, assignments preceding the case (used as default values),
casez wildcards, multiple and partial left-hand sides, and both blocking and
non-blocking assignments are handled. Cases that cannot be safely tabled (e.g.
non-exhaustive with no default, overlapping writes to one variable, or mixed
blocking/non-blocking assignments) fall back to the existing if/else lowering.
Consequently disabled re-inlining of constant pool variables in V3Const,
and rebuild the constant pool hash in V3Dead (previously we didn't
create constant pool entries early enough for this to matter)
Artur Bieniek
Geza Lore
Sunimali Rathnayake
Demin Han
Sumanth KadiyalaandSumanth Kadiyala
Wilson Snyder
Paul Campbell
Tyrone Marhguy
Marco Brambilla
JOTEGO
David Garau
Aisha
BRDR LIFE
spomatasmd
Bartosz Skorowski
Eryk Szpotański
Andrii
Jeffrey Song
Jaeuk Lee
Patrick Creighton
Jakub Michalski
Dragon-Git
Sergey Chusov
Joshua Leahy
Matthew Ballance
Wolfgang Mayerwieser
Thomas Santerre
Tom Jackson
Saksham
Igor Zaworski