The previous implementation of TextAnalysisFormat contained the list
of all the existing relations that sbt knew of, and used this
information to write to and read from the disk the persisted analyses.
In this knew implementation, TextAnalysisFormat gets from the
Relations object what are the existing relations, and then persists
them to disk.
The previous situation was not optimal since it meant that, in order
to add a new dependency kind, one had to modify both the Relations
and TextAnalysisFormat.
Using this new implementation, no change to TextAnalysisFormat is
required whenever a new dependency kind is added.
Those tests use the random Analysis generator that is used in the
unit tests for the subproject `incremental-compiler`.
Random Analyses are serialized and then constructed back from this
representation.
For various reasons, we serialize sequences as:
0 -> foo
1 -> bar
...
Until now we were implicitly relying on the sequences being in order.
However external code may end up (due to bugs or otherwise) messing
with the ordering:
1 -> bar
0 -> foo
...
This change ensures that we don't get confused by that. Although
it's best if external code doesn't mess up the ordering, it's still
a good idea to be defensive about this.
Note that the sequences we serialize are short, so the extra sort
is not a performance concern.
The CompileSetup class is being used to detect changes to arguments of
incremental compiler that affect result of compilation and trigger
recompilation. Examples of such arguments include, the target (output)
directory, Scala compiler options, Scala compiler version, etc.
By adding `nameHashing` to CompileSetup we have a chance to handle change
to that flag smoothly by throwing away old Analysis object and starting
with an empty one. That's implemented in AggressiveComile by extending
the logic that was responsible for detection of changes to CompileSetup
values. Thanks to this change we fix#1081.
Analysis formats has been updated to support persisting of newly added
value in CompileSetup. We used to not store the value of `nameHashing`
flag in persisted Analysis file and infer it from contents of relations
but that leads to issue #1071 when empty relations are involved. Given
the fact that CompileSetup stores `nameHashing` value now, we can just
use it when reading relations and fix#1071. This requires reading/writing
compile setup before reading relations. I decided to make that change even
if there's a comment saying that reading/writing relations first was done
intentionally.
Serializes CompileSetup as text instead of base64-encoded
binary-serialized object.
This is necessary so that file paths in the CompileSetup can be
rebased when porting analysis files between systems.
Tracking of used names is a component needed by the name hashing
algorithm. The extraction and storage of used names is active only when
`AnalysisCallback.nameHashing` flag is enabled and it's disabled by
default.
This change constists of two parts:
1. Modification of Relations to include a new `names` relation
that allows us to track used names in Scala source files
2. Implementation of logic that extracts used names from Scala
compilation units (that correspond to Scala source files)
The first part is straightforward: add standard set of methods in
Relations (along with their implementation) and update the logic which
serializes and deserializes Relations.
The second part is implemented as tree walk that collects all symbols
associated with trees. For each symbol we extract a simple, decoded name
and add it to a set of extracted names. Check documentation of
`ExtractUsedNames` for discussion of implementation details.
The `ExtractUsedNames` comes with unit tests grouped in
`ExtractUsedNamesSpecification`. Check that class for details.
Given the fact that we fork while running tests in `compiler-interface`
subproject and tests are ran in parallel which involves allocating
multiple Scala compiler instances we had to bump the default memory limit.
This commit contains fixes for gkossakowski/sbt#3, gkossakowski/sbt#5 and
gkossakowski/sbt#6 issues.
The previous name of the flag was rather specific: it indicated
whether the new source dependency tracking is supported by given Relations
object. However, there will be more functionality added to Relations that
is specific to name hashing algorithm. Therefore it makes sense to name
the flag as just `nameHashing`.
I decided to rename Relations implementation classes to be more
consistent with the name of the flag and with the purpose they serve.
The flag in AnalysisCallback (and classes implementing it) has been
renamed as well.
Introduce an alternative source dependency tracking mechanism that is
needed by upcoming name hashing algorithm. This new mechanism is
implemented by introducing two new source dependency relations called
`memberRef` and `inheritance`.
Those relations are very similar to existing `direct` and
`publicInherited` relations in some subtle ways. Those differences
will be highlighted in the description below.
Dependencies between source files are tracked in two distinct
categories:
* dependencies introduced by inheriting from a class/trait
defined in other source file
* dependencies introduced by referring (selecting) a member
defined in other source file (that covers all other
kinds of dependencies)
Due to invalidation algorithm implementation details sbt would need to
track inheritance dependencies of public classes only. Thus, we had
relation called `publicInherited`. The name hashing algorithm which
improves invalidation logic will need more precise information about
dependencies introduced by inheritance including dependencies of non-public
classes. That's one difference between `inheritance` and `publicInherited`
relations.
One surprising (to me) thing about `publicInherited` is that it includes
all base classes of a given class and not just parents. In that sense
`publicInherited` is transitive. This is a bit irregular because
everything else in Relations doesn't include transitive dependencies.
Since we are introducing new relations we have an excellent chance to
make things more regular. Therefore `inheritance` relation is
non-transitive and includes only extracted parent classes.
The access to `direct`, `publicInherited`, `memberRef` and `inheritance`
relations is dependent upon the value of `memberRefAndInheritanceDeps`
flag. Check documentation of that flag for details.
The two alternatives for source dependency tracking are implemented by
introduction of two subclasses that implement Relations trait and one
abstract class that contains some common logic shared between those two
subclasses. The two new subclasses are needed for the time being when we
are slowly migrating to the name hashing algorithm which requires
subtle changes to dependency tracking as explained above. For some time we
plan to keep both algorithms side-by-side and have a runtime switch which
allows to pick one. So we need logic for both old and new dependency
tracking to be available. That's exactly what two subclasses of
MRelationsCommon implement. Once name hashing is proven to be stable and
reliable we'll phase out the old algorithm and the old dependency tracking
logic.
Reads/writes are a little faster with the text format,
and it's far more useful. E.g., it allows external manipulation
and inspection of the analysis.
We don't gzip the output. It does greatly shrink the files,
however it makes reads and writes 1.5x-2x slower, and we're
optimizing for speed over compactness.
We store `Seq[xsbt.api.Compilation]` in `Analysis`. Compilation are
being appended to sequence for every iteration of the incremental
compiler.
Note that `Compilation`s are never removed from the sequence unless
you start from scratch with empty `Analysis`. You can do that by using
sbt's `clean` command.
The main use-case for using `compilations` field is to determine how
many iterations it took to compilen give code. The `Compilation` object
are also stored in `Source` objects so there's an indirect way to recover
information about files being recompiled in every iteration.
Since `Analysis` is persisted you can use this mechanism to track entire
sessions spanning multiple `compile` commands.
Martin Duhem
Benjy
Eugene Yokota
Grzegorz Kossakowski
Josh Suereth
Benjy
Mark Harrah