This uses the environment variable `YOSYS_PLUGIN_PATH` to provide multiple colon-delimited search paths for native plugins in a similar manner to `PATH` for executables and `PYTHONPATH` for Python modules.
This addresses https://github.com/YosysHQ/yosys/issues/2545, allowing Yosys to be better packaged in non-FHS environments such as Nix.
We were performing the helper passes `abc9_ops -replace_zbufs` and
`abc9_ops -restore_zbufs` for every module, but those passes act on the
full design (and can't be applied entirely selectively due to entering
and leaving bufnorm).
This lead to an explosive creation of a lot of redundant bufnorm helper
cells that would have been cleaned up by `clean` but that never ran.
Instead we now run each helper pass once, one before and one after
iterating over the selected modules. This limits the number of bufnorm
helper cells.
For consistency.
Also trying a new thing: only rebuilding objects that use the pybind11 library. The idea is these are the only objects that include the Python/pybind headers and thus the only ones that depend on the Python ABI in any capacity, so other objects can be reused across wheel builds. This has the potential to cut down build times.
- Rewrite all Python features to use the pybind11 library instead of boost::python.
Unlike boost::python, pybind11 is a header-only library that is just included by Pyosys code, saving a lot of compile time on wheels.
- Factor out as much "translation" code from the generator into proper C++ files
- Fix running the embedded interpreter not supporting "from pyosys import libyosys as ys" like wheels
- Move Python-related elements to `pyosys` directory at the root of the repo
- Slight shift in bridging semantics:
- Containers are declared as "opaque types" and are passed by reference to Python - many methods have been implemented to make them feel right at home without the overhead/ambiguity of copying to Python and then copying back after mutation
- Monitor/Pass use "trampoline" pattern to support virual methods overridable in Python: virtual methods no longer require `py_` prefix
- Create really short test set for pyosys that just exercises basic functionality
The `portarcs` pass was already ignoring `$buf` cells when loading
timing data, but now bufnorm will also emit `$input_port` and `$connect`
helper cells, which need to be ignored as well.
Was previously the number of proposed renames, but since renames can be skipped this causes the final count to differ from the number of actually renamed objects.
Check counts in `tests/various/autoname.ys`.
This option allows you to process a design that includes unsupported
SVA. Unsupported SVA gets imported as formal cells using 'x inputs and
with the `unsupported_sva` attribute set. This allows you to get a
complete list of defined properties or to check only a supported subset
of properties. To ensure no properties are unintentionally skipped for
actual verification, even in cases where `-sva-continue-on-error` is
used by default to read and inspect a design, `hierarchy -simcheck` and
`hierarchy -smtcheck` (run by SBY) now ensure that no `unsupported_sva`
property cells remain in the design.
This code is quite confusing because there are two "is the cell known" filters
applied, one while building the cell vector and one after building the cell
vector, and they're subtly different. I'm preserving the actual behaviour here
but it looks like there is, or was, a bug here.
Currently the order of extraction can vary based on which ABC runs finish first. That's
nondeterministic, therefore bad. Instead, force the processing to happen in the same order
as `assigned_cells`, i.e. the same order we use when not using parallelism. This should
make everything deterministic.
Note that we still allow ABC runs to complete out of order. Out-of-order results are
just not extracted until all the previous runs have completed and their results
extracted.
1) Change token from ABC_DONE to YOSYS_ABC_DONE to be a bit more robust against false matches.
2) Emit the token from the sourced script so that we don't have to worry about it showing up in the echoing
of the command as it executes. It will only appear in ABC stdout when it executes, i.e. when
our script has completed.
3) `set abcout` doesn't actually switch ABC to line buffering on stdout, since HAVE_SETVBUF is not actually
set in ABC builds in general. So stop using that. ABC does the necessary flushing when
`source` has finished.
With the updated bufnorm code, buffered 'z drivers are used as anchor
points for undirected connections. These are currently not supported by
read/write_xaiger2, so we temporarily replace those by roughly
equivalent $tribuf cells which will be handled as blackboxes that
properly roundtrip through the xaiger2 front and backend.
This ensures that entering and leaving bufnorm followed by `opt_clean`
is equivalent to just running `opt_clean`.
Also make sure that 'z-$buf cells get techmapped in a compatible way.
Doing ABC runs in parallel can actually make things slower when every ABC run requires
spawning an ABC subprocess --- especially when using popen(), which on glibc does not
use vfork(). What seems to happen is that constant fork()ing keeps making the main
process data pages copy-on-write, so the main process code that is setting up each ABC
call takes a lot of minor page-faults, slowing it down.
The solution is pretty straightforward although a little tricky to implement.
We just reuse ABC subprocesses. Instead of passing the ABC script name on the command
line, we spawn an ABC REPL and pipe a command into it to source the script. When that's
done we echo an `ABC_DONE` token instead of exiting. Yosys then puts the ABC process
onto a stack which we can pull from the next time we do an ABC run.
For one of our large designs, this is an additional 5x speedup of the primary AbcPass.
It does 5155 ABC runs, all very small; runtime of the AbcPass goes from 760s to 149s
(not very scientific benchmarking but the effect size is large).
Large circuits can run hundreds or thousands of ABCs in a single AbcPass.
For some circuits, some of those ABC runs can run for hundreds of seconds.
Running ABCs in parallel with each other and in parallel with main-thread
processing (reading and writing BLIF files, copying ABC BLIF output into
the design) can give large speedups.
The code in memory_libmap expects `clk_en` to be initialized for all
`PortVariant`s but the parsing in memlib.cc didn't initialize it for
variants of kind `PortKind::Ar` (async read ports).
While this fixes the immediate CI failure, it would be best to refactor
the code so it becomes obvious if something isn't initialized.
The vast majority of ID(...) uses are in a context that is overloaded
for StaticIdString or will cause implicit conversion to an IdString
constant reference. For some sufficently overloaded contexts, implicit
conversion may fail, so it's useful to have a method to force obtaining
a `IdString const &` from an ID(...) use.
When turning all literal IdStrings of the codebase into StaticIdStrings
this was needed in exactly one place, for which this commit adds an
`id_string()` call.
When FILTERLIB is defined (attempts to compile libparse more or less standalone,) mark the `LibertyParser::error()` as weak so utilities using libparse as a library can override its behavior (the default behavior being exit(1)). As the code is quite performance-critical, I've elected to not modify it to raise an exception or have a callback or similar and simply allow for a link-time replacement.
We call 'assign_map.set()' below which wipes the `SigMap` and reconstructs it.
This operation is expensive because it scans the whole module. I think it's
better to make heavyweight operations more visible so I'm removing
the less obvious operation.
Currently `assign_map` is rebuilt from the module from scratch every time we invoke ABC.
That doesn't scale when we do thousands of ABC runs over large modules. Instead,
create it once and then maintain incrementally it as we update the module.
Still unsupported:
- wide muxes (`$_MUX16_` and friends)
Partially supported types have comments in `test_cell.cc`.
Fix `CellTypes::eval() for `$_NMUX_`.
Fix `RTLIL::Cell::fixup_parameters()` for $concat, $bwmux and $bweqx.
VS build currently failing with `error C2641: cannot deduce template arguments for 'std::array'`.
Changing to `std::array<Cone, 2>` gives `error C2027: use of undefined type` instead.
Necessary to avoid multiple drivers being inserted when a blackbox has inout ports (like when importing an unknown module with `verific`).
If any bits of an inout port have a known driver, treat the port as an input.
If there are no bits with a known driver, treat the port as an output, and mark each bit as having a driver.
- Make IdString parameter pass by const ref to avoid IdString ref counting in the constructor
- Remove extra std::string allocation to remove prefix
- Avoid using `stringf` for concatenation
`-greperr <string>` redirects stderr to 'bugpoint-case.err', and then searches that file for `<string>`.
Move `-runner` option up with the other options to reduce ambiguity (i.e. so it doesn't look like it's another design parts constraint).
Also some shuffling of `err.ys`.
Allows checking return value from crashing design. Makes it possible to only accept designs that crash with e.g. SEGFAULT.
Based on `exec -expect-return`.
`formatted_help()` introduced the override keyword, which means that the other two methods that were marked as virtual instead raised a warning about inconsistent use of override. This fixes that by bringing the synthprop (more) in line with the rest of the code-base.
Linking to optiongroups doesn't add *that* much, and is kind of a pain; meanwhile having the optiongroups adds an extra level of indentation.
Instead of options needing to be in an option group, they instead go in either the root node or nested in a usage node. Putting them in a usage node allows for more-or-less the previous behaviour but without making it the default.
Rename passes/cmds from "General passes" to "Design modification".
More `yosys.h` includes.
cmdref: Split passes/status from passes/cmds
Rename passes/cmds from "General passes" to "Design modification".
More `yosys.h` includes.
Give formal index a proper title.
Use `Pass::formatted_help()` to assign the group, but still return `false` because the help text still comes from `Pass::help()`.
Tidy up some of the affected files' includes to make use of the shared `yosys.h` includes.
Update experimental pass warnings to use a shared function. Reduces repetition, and also allows all of the warning flags to be combined (which at present is just experimental and the new internal).
Update `test_*` passes to call `internal()` in their constructors.
Content is now added to the `ContentListing` rather than the `PrettyHelp`.
`open_*` methods return the `ContentListing` that was added instead of leaving a hanging continuation.
This allows for (e.g.) options to be added directly to optiongroups, instead of requiring that groups be closed before continuation.
This also means that all `PrettyHelp`s are a listing, with the actual log being called by the default `Pass::help()`; making the mode field redundant.
Added `PrettyHelp::log_help()` which replaces the `PrettyHelp::Mode::LOG` logic.
Added `ContentListing::back()` which just returns the last element of the underlying content vector.
Some of the content tracking was made redundant and removed, in particular `PrettyHelp::_current_listing` and `ContentListing::parent`.
Converted `ContentListing` to a class instead of a struct, adjusting constructors to match.
Added `ContentListing` constructor that accepts a `source_location`.
Update `HelpPass::dump_cmds_json()` for new log_help.
Define `PrettyHelp` class with methods for declaring different parts of help message.
Currently able to produce standard help messages as expected.
Updates chformal to use (only) the new help_v2.
Currently makes use of a global static to track the current help context, allowing register.h to live in blissful ignorance and instead rely on help_v2 implementations calling `auto *help = PrettyHelp::get_current();` and `return true;` to minimise impact on rebuilds (i.e. not requiring every source file to be recompiled).
All `proc_*` passes now use the same module and process for loops, using `design->all_selected_modules()` and `mod->selected_processes()` respectively.
This simplifies the code, and makes the couple `proc_*` passes that were ignoring boxed modules stop doing that (which seems to have been erroneous rather than intentional).
Calling `throw dst_end_of_data_exception()` when the desired number of cycles has been reached means that the fst reader can't tidy up after itself and leads to memory leaks.
This doesn't happen when the `-stop` flag is used because the `Yosys::FstData` struct tracks the end time and skips the outer callback if the simulation has gone past the desired end time.
Move cycle checking into the inner callback along with the time checking means that the outer callback no longer needs to throw an exception in order to stop checking further values, while still allowing the fst reader to finish reading and deallocate memory.
libfst is no longer included in gtkwave and instead has its own repo. There has also been some refactoring, so the patches need to update to match, as does sim.cc.
When building `WITH_PYTHON`, where a global list of modules is maintained, deleting a module also erases the entry in said global list. This can lead to memory corruption if the global list is destructed before the module.
Using `on_shutdown()` instead means the module destructor is explicitly called before the global list can be destructed, preventing the issue.
Also add a comment to `Pass::~Pass()` to suggest the same for future passes that might try to use that (and see this commit in the blame if they need a reason why).
The previous commit introduced code that optimizes the activation
patterns to be able to generate smaller activation logic. The resulting
supercell was then enqueued as shareable using those optimized
activation patterns. The condition represented by the optimized patterns
is an over-approximation of the actual activiation condition. This means
using it as activiation for the supercell loses precision and pessimises
sharing of the supercell with further cells, breaking the sat/share
test.
This commit fixes that by using the optimized activiation patterns only
for the generation of activation logic and using the original patterns
for enqueuing the supercell.
In case the two sets of activation patterns are mutually exclusive
without considering the logic feeding into the activation signals, an
activation condition can only be relevant if present in both sets with
opposite polarity.
This detects pattern-only mutual exclusion by running an additional SAT
query before importing the input cone logic. If that is already UNSAT,
we remove all non-relevant condition and re-simplify the remaining
patterns.
In cases of pattern-only mutual exclusion, this will often produce much
smaller selection logic and avoid the more costly SAT query that
includes the input cones.
Partially reverts commit 9c5bffcf93.
The reasoning behind this is that setup.py is intended to strictly consume the Makefile and not be consumed by it. The attempt at using them recursively has caused a number of issues and has rendered Pyosys unusable to some users: See https://github.com/YosysHQ/yosys/issues/5012
Additionally, unlike the previous pyosys installation target, the wheel installation does not respect PREFIX=, only venvs.
For installation inside a venv, the intended method should remain a user manually executing `pip3 install .` instead of relying on the Makefile.
Use `Design::selected_modules()` directly, popping at the end instead of copying the selection.
Also default to a complete selection so that boxes work as before.
Simplify to using `RTLIL::SELECT_WHOLE_CMDERR` instead of doing it manually.
Also add tests for importing selections with boxes.
If all the (non-select) inputs of a `$_MUX{4,8,16}_` are undefined, replace it, just like we do for `$mux` and `$_MUX_`.
Add `tests/opt/opt_expr_mux_undef.ys` to verify this.
This doesn't do any const folding on the wide muxes, or shrinking to less wide muxes. It only handles the case where all inputs are 'x and the mux can be completely removed.
This adds optional in-memory caching of parsed liberty files to speed up
flows that repeatedly parse the same liberty files. To avoid increasing
the memory overhead by default, the caching is disabled by default. The
caching can be controlled globally or on a per path basis using the new
`libcache` command, which also allows purging cached data.
These were introduced by 0a6d9f4.
1) While in a paren "(", don't error on newline.
2) Don't parse an extra token when parsing vector ranges. Let the caller parse the next token as necessary.
This extends the `LibertyInputStream` added in the previous commit to
allow arbitrary lookahead. Then this uses the lookahead to find the
total length of the token within the input buffer, instead of consuming
the token byte by byte while appending to a std::string. Constructing
the std::string with the total length is known avoids any reallocations
from growing std::string's buffer.
The lexer for liberty files was using istream's `get` and `unget` which
are notorious for bad performance and that showed up during profiling.
This replaces the direct `istream` use with a custom LibertyInputStream
that does its own buffering to provide `get` and `unget` that behave the
same way but are implemented with a fast path that is easy to inline and
optimize.
If the selection stack only has one element (which it normally does), then
`design->pop_selection()` automatically resets to the default full selection.
This is a problem for `select [-none | -clear]` which were trying to replace the
current selection, but because the pop added an extra element when the `execute`
returned, the extra selection (the one we actually wanted) gets popped too. So
instead, reassign `design->selection()` in the same way as if we called `select
[selection]`.
Also adds selection stack tests, and removes the accidentally-committed
`boxes_dummy.ys`.
Instead, change the default `Design::selected_modules()` to match the behaviour (i.e. `selected_unboxed_modules_warn()`) because it's a lot of files to touch and they don't really _need_ to be updated.
Also change `Design::selected_whole_modules()` users over to `Design::selected_unboxed_whole_modules()`, except `attrmap` because I'm not convinced it should be ignoring boxes. So instead, leave the deprecation warning for that one use and come back to the pass another time.
Fixes quicklogic/pp3 problem with `dffepc` including processes.
Also means the preceding `proc` is safe to remove (and may result in some small speedup by doing so).
Used to select all modules including boxes, set when both `full` and `boxes` are true in the constructor, pulling down `full_selection`.
Add `Selection::selects_all()` method as short hand for `full_selection || complete_selection`.
Update selection operations to account for complete selections.
Add static methods to `Selection` for creating a new empty/full/complete selection to make it clearer to users when doing so.
Use said static methods to replace most instances of the `Selection` constructor.
Update `Selection::optimize` to use
The `Design::selected_*()` methods no longer unconditionally skip boxed modules. Instead, selections are now box and design aware.
The selection constructor now optionally takes a design pointer, and has a new `selects_boxes` flag. If the selection has an assigned design, then `Selection::selected_*()` will only return true for boxed modules if the selects_boxes flag is set. A warning is raised if a selection is checked and no design is set. Selections can change design via the `Selection::optimize()` method.
Most places that iterate over `Design::modules()` and check `Selection::selected_module()` should instead use `Design::selected_modules()`.
Since boxed modules should only ever be selected explicitly, and `full_selection` (now) refers to all non-boxed modules, `Selection::optimize()` will clear the `full_selection` flag if the `selects_boxes` flag is enabled, and instead explicitly selects all modules (including boxed modules). This also means that `full_selection` will only get automatically applied to a design without any boxed modules.
These changes necessitated a number of changes to `select.cc` in order to support this functionality when operating on selections, in particular when combining selections (e.g. by union or difference).
To minimize redundancy, a number of places that previously iterated over `design->modules()` now push the current selection to the design, use `design->selected_modules()`, and then pop the selection when done.
Introduce `RTLIL::NamedObject`, to allow for iterating over all members of a module with a single iterator instead of needing to iterate over wires, cells, memories, and processes separately.
Also implement `Module::selected_{memories, processes, members}()` to match wires and cells methods. The `selected_members()` method combines each of the other `selected_*()` methods into a single list.
Mohamed Gaber
Krystine Sherwin
Emil J
Miodrag Milanović
Jannis Harder
Martin Povišer
Robert O'Callahan
Emil Jiří Tywoniak
Claire Xen
Mike Inouye
clemens
suisseWalter
Anhijkt
Lofty
Fred Tombs
Drew Lewis
N. Engelhardt
N. Engelhardt
George Rennie
Gary Wong
Akash Levy
Emily Schmidt
Kelvin Chung
Sean Luchen