In order to restore reproducibility of builds, we no longer cascade over
the possibly available versions of the compiler bridge sources (a
specific version of the bridge sources may not be available one day, but
exist on the next day), but rather let the build definition configure
which module to use.
Fixessbt/sbt#2196
Because in most cases there aren't version-specific sources, we expect
the retrieval to fail a number of times before succeeding. This
generates a lot of noise in sbt's log, so the logs will now be shown if
and only all the versions fail.
During compilation, scalac generates getters and setters for the fields
of traits, regardless of their access modifiers. Therefore, when a
field of a trait is modified, all its implementors must be recompiled to
take these changes into account.
Private fields of traits were not included in the API hash of traits,
and their implementors were thus not recompiled when modified.
This commit changes the way the API hash is computed for traits only, so
that the generated hash includes the private members of traits.
Fixessbt/sbt#2155
This commit introduces a mechanism that allows sbt to find the most
specific version of the compiler interface sources that exists using
Ivy.
For instance, when asked for a compiler interface for Scala 2.11.8-M2,
sbt will look for sources for:
- 2.11.8-M2 ;
- 2.11.8 ;
- 2.11 ;
- the default sources.
This commit also modifies the build definition by removing the
precompiled projects and configuring the compiler-interface project so
that it publishes its source artifacts in a Maven-friendly format.
sbt 0.13.1 was changed so that products were invalidated
not just when they were deleted, but also when they were
modified, however the debug message was not updated to
reflect this, causing people to think invalidated class files
had been deleted.
* IncrementalCompiler IC object now holds the actual logic to start incremental compilation (rather than AggresiveCompiler or other)
* MixedAnalyzingCompiler ONLY does anlaysis of Java/Scala code
* Moved the AnalyzingJavaCompiler into the integration library so that necessary dependencies are visible.
* Split Java analyzing compile into its own class.
* MixedAnalyzingCompiler now only does the mixing
* Start moving methods around to more-final locations
* Static analyzingCompile method now constructs a MixedAnalyzingCOmpiler and delegates to incremental compile.
* Force CompileSetup Equiv typeclass to use Equiv relations defined locally.
* Add toString methods on many of the incremental compiler datatypes.
* Remove remaining binary compatibility issues in Defaults.scala.
* Removed as many binary incompatibilities as I could find.
* Deprecating old APIs
* Attempt to construct new nomenclature that fits the design of Incremental API.
* Add as much documentation as I was comfortable writing (from my understanding of things).
This breaks the loading/saving of the incremental compiler analysis out
into separate task, thereby providing the necessary hooks for byte code
enhancement tasks to enhance bytecode and update the analysis before the
analysis gets stored to disk.
In some cases the dependency extraction may encounter a null `TypeTree`
(eg. arguments of macro annotations that are untyped). In such cases,
we simply ignore the node.
Fixes#1593, #1655.
Since `DependencyContext` is needed in the compiler interface
subproject, it has to be defined in this same subproject.
`DependencyContext` is needed in this subproject because the
`AnalysisCallback` interface uses it.
A new infrastructure to register sources and their dependencies has
been introduced in `c09a391`.
This commit brings the required modifications to `AnalysisTest`,
so that it uses it.
A new infrastructure to register sources and their dependencies has
been introduced in `c09a391`.
This commit brings the required modifications to `AnalysisCallback`,
so that it uses it.
This commit implements the abstraction over the dependency kinds and
deprecates the methods that were used to register files and their
dependencies.
Dependencies are now divided into two categories : Internal and
External dependencies. Moreover, each internal or external dependency
has a Context, which describes what introduced the dependency (for
instance, a dependency may be introduced by member reference or by
inheritance).
Dependencies must now be registered using the method `addSource` in
`Analysis` and in `Relations`. This method has the advantage of being
independent from the existing dependency contexts. That is, its
signature does not need to be modified whenever a new dependency
context is introduced.
* Move error parser into its own file.
* Add the ability to parse Windows filenames.
* Remove existence check for the file as a mandatory.
* Add specific test for the parser.
* Create a new sbt.compiler.javac package
* Create new interfaces to control running `javac` and `javadoc` whether forked or local.
* Ensure new interfaces make use of `xsbti.Reporter`.
* Create new method on `xsbti.compiler.JavaCompiler` which takes a `xsbti.Reporter`
* Create a new mechanism to parse (more accurately) Warnings + Errors, to distinguish the two.
* Ensure older xsbti.Compiler implementations still succeed via catcing NoSuchMethodError.
* Feed new toolchain through sbt.actions.Compiler API via dirty hackery until we can break things in sbt 1.0
* Added a set of unit tests for parsing errors from Javac/Javadoc
* Added a new integration test for hidden compilerReporter key, including testing threading of javac reports.
Fixes#875, Fixes#1542, Related #1178 could be looked into/cleaned up.
The fix for sbt/sbt#1237 was unfortunately not completely correct,
and infinite loops could still occur during the extraction of used
names.
In sbt/sbt#1544, a fix that was robuster and easier to understand
was applied to `/compile/interface/src/main/scala/xsbt/Dependency.scala`
in a similar situation (cyclic chains of original trees in macro
expansions).
This commit ports this fix to `ExtractUsedNames.scala`.
Closessbt/sbt#1640, sbt/sbt#1610.
When dealing with Java APIs (Java reflection in case of ClassToAPI), one
should always go through Option.apply that performs null check.
This is supposed to fix NPE reported in #1617. We don't have a reproduction
so this is just an educated guess.
This represents a sketch of the idea that we can abstract over details
of a specific dependency kind. The goal would that only a few
implementations of methods in incremental would be sensitive to specific
dependency kind:
1. Dependency extraction logic
2. Implementation of Relations which adds dependencies to specific
relations (or abstract over specific relations)
3. Invalidation algorithm (Incremental.scala) which has different
of each kind of dependency
4. TextAnalysisFormat
In particular, adding a new dependency kind would not affect signatures
of existing methods.
What needs to be done:
- finish refactoring so the code compiles again and previous semantics
are preserved
- introduce deprecated overloads that preserve old method signatures
(this is required for preserving binary compatibility)
The previous implementation of TextAnalysisFormat contained the list
of all the existing relations that sbt knew of, and used this
information to write to and read from the disk the persisted analyses.
In this knew implementation, TextAnalysisFormat gets from the
Relations object what are the existing relations, and then persists
them to disk.
The previous situation was not optimal since it meant that, in order
to add a new dependency kind, one had to modify both the Relations
and TextAnalysisFormat.
Using this new implementation, no change to TextAnalysisFormat is
required whenever a new dependency kind is added.
In some cases, expanded macros report that their original tree and
its expansion are the same, thus creating a cyclic chain. This chain
may then produce a SOE during dependencies or used names extraction.
This kind of problem was already reported in sbt/sbt#1237 and
sbt/sbt#1408. Unfortunately, the fix that was applied to the
dependencies extraction part was not sufficient.
Mark test 'source-dependencies/macro' as passing
Fixes#1544
Those tests use the random Analysis generator that is used in the
unit tests for the subproject `incremental-compiler`.
Random Analyses are serialized and then constructed back from this
representation.
Unit tests in incremental-compiler subproject use a generator to
create random Analysis objects. This generator was unfortunately
not working properly and generated only empty Analyses (it failed
to generate any non-empty Analysis because of a bug in the `unique`
generator).
Name hashing is now turned on by default, so I’m changing the value for
inc.Relations.empty, so inc.Analysis.empty functions as expected when
it’s joined with name hashing analyses.
This commit changes the default value of `IncOptions.nameHashing` to be
set to true. It means, the improved incremental compilation algorithm
known as "name hashing" will be enabled by default.
In order to disable it, users should add this to their sbt configuration:
incOptions := incOptions.value.withNameHashing(false)
Number of tests has been cleaned up as part of this change. All tests
that were marked as name hashing specific are removed. The list includes:
* constants-name-hashing
* import-class-name-hashing
* java-static-name-hashing
* macro-name-hashing
* struct-name-hashing
We'll keep just regular version of those tests. The tests will just
exercise the default algorithm: name hashing. This is the first step
towards phasing out of the old incremental compilation algorithm.
Apart from that, a few tests changed its status due to enabling name
hashing algorithm.
The `constants` test has been marked pending due to issue described in
#1543.
The `import-class` test has been marked as passing because name hashing
tracks dependencies introduced by import statements correctly, now.
The `macro` test has been marked as pending due to issue described in
#1544.
The `struct` test has been marked as pending due to issue described in
#1545.
The `java-static` has been slightly modified to exercise just static field
and not run into the same issue as with `constants` test.
There are no other known issues related to name hashing so we conclude
that name hashing is ready to be shipped to all sbt, Scala IDE and zinc
users.
In Scala 2.10.4, this macro can produce a stack overflow :
def foo(a: Any): Any = macro impl
def impl(c: Context)(a: c.Expr[Any]): c.Expr[Any] = a
Here, an application such as `foo(someVal)` will produce the expansion
`someVal`. As expected, `someVal` has `original` tree `foo(someVal)`,
but if we inspect this tree, we will find that `someVal` has an
original tree, but it shouldn't.
Moreover, in Scala 2.11, some macros have their own application as
`original` trees.
See sbt/sbt#1237 for a description of these problems.
This commit fixes these two problems.
Fixessbt/sbt#1237
Incremental.scala contained all three strategies of incremental
compilation:
* ant-style
* the default one (before name hashing)
* name hashing
Let's move all those classes into separate files. Also, move common code
into a separate file.
For various reasons, we serialize sequences as:
0 -> foo
1 -> bar
...
Until now we were implicitly relying on the sequences being in order.
However external code may end up (due to bugs or otherwise) messing
with the ordering:
1 -> bar
0 -> foo
...
This change ensures that we don't get confused by that. Although
it's best if external code doesn't mess up the ordering, it's still
a good idea to be defensive about this.
Note that the sequences we serialize are short, so the extra sort
is not a performance concern.
Add scala 2.11 test/build verification.
* Add 2.11 build configuratoin to travis ci
* Create command which runs `safe` unit tests
* Create command to test the scala 2.11 build
* Update scalacheck to 1.11.4
* Update specs2 to 2.3.11
* Fix various 2.11/deprecation removals
and other changes.
Fix eval test failure in scala 2.11 with XML not existing.
This commit implements an Ant-style incremental compilation mode. This mode
emulates what Ant's scalac command does. It recompiles just changed source
files and does not perform any invalidation of dependencies.
This is a very naive mode of incremental compilation that very often leads
to broken binaries.
The Ant-style mode is being introduced because Scala team needs it for
migration of Scala compiler to sbt. The name hashing algorithm doesn't
work well with Scala compiler sources due to deep inheritance chains.
There's a plan to refactor compiler's code to use more composition instead
of inheritance.
Once Scala compiler sources are refactored to work well with name hashing
algorithm, Ant-style mode will be deleted immediately.
Add an option that enables (to be implemented) Ant-style mode of
incremental compilation.
This option is unsupported and may go away at any point in the future.
NOTE: Either `antStyle` or `nameHashing` mode can be enabled. This is
being enforced with runtime assertion.
Add logging of various operations the transactional class file manager is
doing. You can pass logger to be used by the transactional class file
manager by using overloaded definition of `ClassfileManager.transactional`
method. The old overload has been deprecated.
The factory methods for class file manager in IncOptions companion object
has been deprecated in favor of using ClassfileManager companion object
directly. The code in Defaults.scala has been updated to use non-deprecated
methods. The logging is turned off by default.
The canonical way of enabling transactional class file manager in sbt
project is:
```
incOptions := incOptions.value.withNewClassfileManager(
sbt.inc.ClassfileManager.transactional(
crossTarget.value / "classes.bak",
(streams in (compile, Compile)).value.log
)
)
```
It's a bit verbose which shows that the api for this is not the best.
However, I don't expect sbt users to need this code very often.
This patch should help debug the problem described in #1184
It should have been there from the beginning because NameHashing is tied
to internals of the API subproject.
It was added to incremental subproject by mistake.
It has been reported in sbt/sbt#1237 that stack overflows may occur during the
extraction of used names (and later of dependencies between files). This
problem has been introduced by sbt/sbt#1163, which was about recording the
dependencies of macro arguments.
When a macro is expanded, the compiler attaches the tree before expansion to
the tree representing the expanded macro. As of Scala 2.11-RC3, some macros
have themselves attached as original tree, which caused the same macro to be
inspected over and over until a stack overflow.
This commit solves this problem by making sure that the original of a macro
expansion will be inspected if and only if it is different from the expanded
tree.
Fixessbt/sbt#1237
Prior to this commit, a class that inherited a macro from another
class was considered by incremental compiler as having a macro.
Now, only classes that explicitly define a macro are considered as having
a macro. This influences decision whether to invalidate (recompile)
dependencies of a file that inherits a macro upon a whitespace change.
From now on, we don't invalidate dependencies in such case which
results in much better incremental compiler experience when macros are
being involved. Check #1142 for detailed discussion.
The change to the behavior is reflected by marking the
source-dependencies/inherited-macros test as passing.
The source-dependencies/macro test covers the case of defining the macro
directly in source file. Therefore we know that the desired behavior of
invalidating dependencies of macros is preserved.
Fixes#1142
Add a unit test which checks whether we capture dependencies introduced
by arguments to macros. Those dependencies are special because macros
get expanded during type checking and arguments to macros are not visible
during regular tree walk.
It was not possible to make `ScalaCompilerForUnitTesting` compile several
files in different runs, which means that it was not possible to compile
and use a macro in a test case, since macros cannot be used in the same
compilation run that defines them.
This commit allows a test case to provide multiple grouped snippets of
code that will be compiled in separate runs.
For instance :
List(Map(<snippet A>, <snippet B>), Map(<snippet C>))
Here, <snippet A> and <snippet B> will be compiled together, and then
<snippet C> will be compiled, and will be able to use symbols defined
in <snippet A> or <snippet B>.
Macros take arguments as trees and return some other trees; both of
them have dependencies but we see trees only after expansion and
recorded only those dependencies.
This commit solves this problem by looking into the attachments of the
trees that are supposed to contain originals of macro expansions and
recording dependencies of the macro before its expansion.
The c7f435026f introduced a new parameter
to the constructor of `CompileSetup` but it turns out that this class
is being used in zinc. Introduce an overloaded variant of that constructor
that preserves backwards compatibility.
The CompileSetup class is being used to detect changes to arguments of
incremental compiler that affect result of compilation and trigger
recompilation. Examples of such arguments include, the target (output)
directory, Scala compiler options, Scala compiler version, etc.
By adding `nameHashing` to CompileSetup we have a chance to handle change
to that flag smoothly by throwing away old Analysis object and starting
with an empty one. That's implemented in AggressiveComile by extending
the logic that was responsible for detection of changes to CompileSetup
values. Thanks to this change we fix#1081.
Analysis formats has been updated to support persisting of newly added
value in CompileSetup. We used to not store the value of `nameHashing`
flag in persisted Analysis file and infer it from contents of relations
but that leads to issue #1071 when empty relations are involved. Given
the fact that CompileSetup stores `nameHashing` value now, we can just
use it when reading relations and fix#1071. This requires reading/writing
compile setup before reading relations. I decided to make that change even
if there's a comment saying that reading/writing relations first was done
intentionally.
Catch ReadException and wrap it in IOException that carries the name
of the file we failed to read in its message.
We have to catch exception and wrap them because in TextAnalysisFormat
we don't have an access to the file name (it operates using an abstract
reader).
The ae15eccd9c accidentally removed
`Incremental.incDebugProp` which broke Scala IDE build that relies on it.
We bring back that val but at the same time we deprecate it because we
have better mechanism for configuring incremental compiler now.
I also added a little comment with the history of `incDebugProp` which
explains proper migration path.
Serializes CompileSetup as text instead of base64-encoded
binary-serialized object.
This is necessary so that file paths in the CompileSetup can be
rebased when porting analysis files between systems.
Martin Duhem
eugene yokota
Stu Hood
Pierre DAL-PRA
Josh Suereth
fkorotkov
James Roper
Benjy
Indrajit Raychaudhuri
Jean-Rémi Desjardins
Grzegorz Kossakowski
Martin Duhem
Brian McKenna