Incremental.scala contained all three strategies of incremental
compilation:
* ant-style
* the default one (before name hashing)
* name hashing
Let's move all those classes into separate files. Also, move common code
into a separate file.
For various reasons, we serialize sequences as:
0 -> foo
1 -> bar
...
Until now we were implicitly relying on the sequences being in order.
However external code may end up (due to bugs or otherwise) messing
with the ordering:
1 -> bar
0 -> foo
...
This change ensures that we don't get confused by that. Although
it's best if external code doesn't mess up the ordering, it's still
a good idea to be defensive about this.
Note that the sequences we serialize are short, so the extra sort
is not a performance concern.
Add scala 2.11 test/build verification.
* Add 2.11 build configuratoin to travis ci
* Create command which runs `safe` unit tests
* Create command to test the scala 2.11 build
* Update scalacheck to 1.11.4
* Update specs2 to 2.3.11
* Fix various 2.11/deprecation removals
and other changes.
Fix eval test failure in scala 2.11 with XML not existing.
This commit implements an Ant-style incremental compilation mode. This mode
emulates what Ant's scalac command does. It recompiles just changed source
files and does not perform any invalidation of dependencies.
This is a very naive mode of incremental compilation that very often leads
to broken binaries.
The Ant-style mode is being introduced because Scala team needs it for
migration of Scala compiler to sbt. The name hashing algorithm doesn't
work well with Scala compiler sources due to deep inheritance chains.
There's a plan to refactor compiler's code to use more composition instead
of inheritance.
Once Scala compiler sources are refactored to work well with name hashing
algorithm, Ant-style mode will be deleted immediately.
Add an option that enables (to be implemented) Ant-style mode of
incremental compilation.
This option is unsupported and may go away at any point in the future.
NOTE: Either `antStyle` or `nameHashing` mode can be enabled. This is
being enforced with runtime assertion.
Add logging of various operations the transactional class file manager is
doing. You can pass logger to be used by the transactional class file
manager by using overloaded definition of `ClassfileManager.transactional`
method. The old overload has been deprecated.
The factory methods for class file manager in IncOptions companion object
has been deprecated in favor of using ClassfileManager companion object
directly. The code in Defaults.scala has been updated to use non-deprecated
methods. The logging is turned off by default.
The canonical way of enabling transactional class file manager in sbt
project is:
```
incOptions := incOptions.value.withNewClassfileManager(
sbt.inc.ClassfileManager.transactional(
crossTarget.value / "classes.bak",
(streams in (compile, Compile)).value.log
)
)
```
It's a bit verbose which shows that the api for this is not the best.
However, I don't expect sbt users to need this code very often.
This patch should help debug the problem described in #1184
It should have been there from the beginning because NameHashing is tied
to internals of the API subproject.
It was added to incremental subproject by mistake.
It has been reported in sbt/sbt#1237 that stack overflows may occur during the
extraction of used names (and later of dependencies between files). This
problem has been introduced by sbt/sbt#1163, which was about recording the
dependencies of macro arguments.
When a macro is expanded, the compiler attaches the tree before expansion to
the tree representing the expanded macro. As of Scala 2.11-RC3, some macros
have themselves attached as original tree, which caused the same macro to be
inspected over and over until a stack overflow.
This commit solves this problem by making sure that the original of a macro
expansion will be inspected if and only if it is different from the expanded
tree.
Fixessbt/sbt#1237
Prior to this commit, a class that inherited a macro from another
class was considered by incremental compiler as having a macro.
Now, only classes that explicitly define a macro are considered as having
a macro. This influences decision whether to invalidate (recompile)
dependencies of a file that inherits a macro upon a whitespace change.
From now on, we don't invalidate dependencies in such case which
results in much better incremental compiler experience when macros are
being involved. Check #1142 for detailed discussion.
The change to the behavior is reflected by marking the
source-dependencies/inherited-macros test as passing.
The source-dependencies/macro test covers the case of defining the macro
directly in source file. Therefore we know that the desired behavior of
invalidating dependencies of macros is preserved.
Fixes#1142
Add a unit test which checks whether we capture dependencies introduced
by arguments to macros. Those dependencies are special because macros
get expanded during type checking and arguments to macros are not visible
during regular tree walk.
It was not possible to make `ScalaCompilerForUnitTesting` compile several
files in different runs, which means that it was not possible to compile
and use a macro in a test case, since macros cannot be used in the same
compilation run that defines them.
This commit allows a test case to provide multiple grouped snippets of
code that will be compiled in separate runs.
For instance :
List(Map(<snippet A>, <snippet B>), Map(<snippet C>))
Here, <snippet A> and <snippet B> will be compiled together, and then
<snippet C> will be compiled, and will be able to use symbols defined
in <snippet A> or <snippet B>.
Macros take arguments as trees and return some other trees; both of
them have dependencies but we see trees only after expansion and
recorded only those dependencies.
This commit solves this problem by looking into the attachments of the
trees that are supposed to contain originals of macro expansions and
recording dependencies of the macro before its expansion.
The c7f435026f introduced a new parameter
to the constructor of `CompileSetup` but it turns out that this class
is being used in zinc. Introduce an overloaded variant of that constructor
that preserves backwards compatibility.
The CompileSetup class is being used to detect changes to arguments of
incremental compiler that affect result of compilation and trigger
recompilation. Examples of such arguments include, the target (output)
directory, Scala compiler options, Scala compiler version, etc.
By adding `nameHashing` to CompileSetup we have a chance to handle change
to that flag smoothly by throwing away old Analysis object and starting
with an empty one. That's implemented in AggressiveComile by extending
the logic that was responsible for detection of changes to CompileSetup
values. Thanks to this change we fix#1081.
Analysis formats has been updated to support persisting of newly added
value in CompileSetup. We used to not store the value of `nameHashing`
flag in persisted Analysis file and infer it from contents of relations
but that leads to issue #1071 when empty relations are involved. Given
the fact that CompileSetup stores `nameHashing` value now, we can just
use it when reading relations and fix#1071. This requires reading/writing
compile setup before reading relations. I decided to make that change even
if there's a comment saying that reading/writing relations first was done
intentionally.
Catch ReadException and wrap it in IOException that carries the name
of the file we failed to read in its message.
We have to catch exception and wrap them because in TextAnalysisFormat
we don't have an access to the file name (it operates using an abstract
reader).
The ae15eccd9c accidentally removed
`Incremental.incDebugProp` which broke Scala IDE build that relies on it.
We bring back that val but at the same time we deprecate it because we
have better mechanism for configuring incremental compiler now.
I also added a little comment with the history of `incDebugProp` which
explains proper migration path.
Serializes CompileSetup as text instead of base64-encoded
binary-serialized object.
This is necessary so that file paths in the CompileSetup can be
rebased when porting analysis files between systems.
Provide implementation of invalidation logic that takes computed
name hashes into account. The implementation is spread amongst two
classes:
1. `IncrementalNameHashing` which implements a variant of
incremental compilation algorithm that computes modified
names and delegates to `MemberReferenceInvalidationStrategy`
when invalidating member reference dependencies
2. `MemberReferenceInvalidationStrategy` which implements the
core logic of dealing with dependencies introduced by member
reference. See documentation of that class for details.
The name hashing optimization is applied when invalidating source files
having both internal and external dependencies (in initial iteration),
check `invalidateByExternal` and `invalidateSource` methods for details.
As seen in implementation of `MemberReferenceInvalidationStrategy`
the name hashing optimization is not applied when implicit members
change.
NOTE: All functionality introduced in this commit is enabled only
when `IncOptions.nameHashing` flag is set to true.
The `source-dependencies/transitive-memberRef` test has been changed
to test name hashing variant of incremental compilation. The change
to invalidated files reflects the difference between the old and the
new algorithm.
Also, there a few new tests added that cover issues previously found
while testing name hashing algorithm and are fixed in this commit.
Each paragraph describes a single test.
Add a test case which shows that detect properly changes to type aliases
in the name hashing algorithm. See gkossakowski/sbt#6 for details.
Add test covering bug with use of symbolic names (issue
gkossakowski/sbt#5).
Add a test which covers the case where we refer to a name that is
declared in the same file. See issue gkossakowski/sbt#3 for details.
There are two categories of places in the code that need to refer to
`nameHashing` option:
* places where Analysis object is created so it gets proper
implementation of underlying Relations object
* places with logic that is specifically designed to be enabled by
that option
This commit covers both cases.
The 39036e7c20 introduced
`recompileOnMacroDef` option to IncOptions. However, not all necessary
logic has been changed. This commit fixes that:
* `copy` method does not forget the value of the `recompileOnMacroDef`
flag
* `productArity` has been increased to match the arity of the class
* `productElement` returns the value of `recompileOnMacroDef` flag
* `hashCode` and `equals` methods take into account value of
`recompileOnMacroDef` flag
* fix the name of the key for `recompileOnMacroDef` flag
Move implementation of the following methods from IncrementalCommon
to IncrementalDefaultImpl:
* invalidatedPackageObjects
* sameAPI
* invalidateByExternal
* allDeps
* invalidateSource
These are the methods that are expected to have different implementation
in the name hashing algorithm. Hence, we make them abstract in
IncrementalCommon so they can be implemented differently in subclasses.
Refactor the `invalidateByExternal` method to take single, external
api change. Introduce `invalidateByAllExternal` that takes all APIChanges
object.
This way `invalidateByExternal` will have an access to APIChange object
that represents changed name hashes once name hashing is merged.
This way we'll be able to have a polymorphic implementation of this
method in the future. One implementation will use the old dependency
tracking mechanism and the other will use the new one (implemented
for name hashing).
In addition to `invalidateSources` we introduce `invalidateSource`
that invalidates dependencies of a single source. This is needed
for the name hashing algorithm because its invalidation logic
depends on information about API changes of each source file
individually.
The refactoring is done in `IncrementalCommon` class so it affects
the default implementation as well. However, this refactoring does
not affect the result of invalidation in the default implementation.
Introduce an abstract `IncrementalCommon class that holds the
implementation of incremental compiler that was previously done in
`Incremental` class. Also, introduce `IncrementalDefaultImpl` that
inherits from IncrementalCommon.
This is the first step to introduce a design where most of incremental
compiler's logic lives in IncrementalCommon and we have two subclasses:
1. Default, which holds implementation specific to the old algorithm
known from sbt 0.13.0
2. NameHashing, which holds implementation specific to the name
hashing algorithm
This commit is purely a refactoring and does not change any behavior.
Both Logger and IncOptions instances were passed around Incremental class
implementation unmodified. Given the fact that entire implementation of
the class uses exactly the same values for those types it makes sense
to extract them as constructor arguments so they are accessible everywhere.
This helps reducing signatures of other methods to more essential
parameters that are more specific to given method.
Move most of the functionality from `Incremental` object to its
companion class.
This commit is a preparation for making it possible to have
two different implementation of logic in `Incremental` object.
Each of Relations implementation should have specific `toString`
implementation. For example, only `MRelationsNameHashing` implementation
should be printing used names in toString representation.
A hash for given name in a source file is computed by combining
hashes of all definitions with given name. When hashing a single
definition we take into account all information about it except nested
definitions. For example, if we have following definition
class Foo[T] {
def bar(x: Int): Int = ???
}
hash sum for `Foo` will include the fact that we have a class with
a single type parameter but it won't include hash sum of `bar` method.
Computed hash sums are location-sensitive. Each definition is hashed along
with its location so we properly detect cases when definition's signature
stays the same but it's moved around in the same compilation unit.
The location is defined as sequence of selections. Each selection consists
of a name and name type. The name type is either term name or type name.
Scala specification (9.2) guarantees that each publicly visible definition
is uniquely identified by a sequence of such selectors.
For example, if we have:
object Foo {
class Bar { def abc: Int }
}
then location of `abc` is Seq((TermName, Foo), (TypeName, Bar))
It's worth mentioning that we track name-hash pairs separately for
regular (non implicit) and implicit members. That's required for name
hashing algorithm because it does not apply its heuristic when implicit
members are being modified.
Another important characteristic is that we include all inherited members
when computing name hashes.
Here comes the detailed list of changes made in this commit:
* HashAPI has new parameter `includeDefinitions` that allows
shallow hashing of Structures (where we do not compute hashes
recursively)
* HashAPI exposes `finalizeHash` method that allow one to capture
current hash at any time. This is useful if you want to hash a list of
definitions and not just whole `SourceAPI`.
* NameHashing implements actual extraction of public definitions,
grouping them by simple name and computing hash sums for each group
using HashAPI
* `Source` class (defined in interface/other file) has been extended to
include `_internalOnly_nameHashes` field. This field stores
NameHashes data structure for given source file. The NameHashes
stores two separate collections of name-hash pairs for regular and
implicit members.
The prefix `_internalOnly_` is used to indicate that this is not an
official incremental compiler's or sbt's API and it's for use by
incremental compiler internals only. We had to use such a prefix
because the `datatype` code generator doesn't support emitting access
modifiers
* `AnalysisCallback` implementation has been modified to gather all
name hashes and store them in the Source object
* TestCaseGenerators has been modified to implement generation of
NameHashes
* The NameHashingSpecification contains a few unit tests that make sure
that the basic functionality works properly
Grzegorz Kossakowski
Benjy
Josh Suereth
Eugene Yokota
Martin Duhem
Mark Harrah