Commit Graph

478 Commits

Author SHA1 Message Date
Grzegorz Kossakowski 2551eb2a63 Do not add source dependencies on itself.
Adding source dependency on itself doesn't really bring any value so
there's no reason to do it. We avoided recording that kind of dependencies
by performing a check in `AnalysisCallback` implementation. However, if we
have another implementation like `TestCallback` used for testing we do
not benefit from that check.

Therefore, the check has been moved to dependency phase were dependencies
are collected.
2013-11-26 18:39:23 +01:00
Grzegorz Kossakowski de1c5a4aed Add support for unit testing of extracted source dependencies.
Add `extractDependenciesFromSrcs` method to ScalaCompilerForUnitTest
class which allows us to unit test dependency extraction logic.

See the comment attached to the method that explain the details of
how it should be used.
2013-11-26 18:39:23 +01:00
Grzegorz Kossakowski 89914975e1 Refactor ScalaCompilerForUnitTesting.
Refactor ScalaCompilerForUnitTesting by introducing a new method
`extractApiFromSrc` which better describes the intent than
`compileSrc`. The `compileSrc` becomes a private, utility method.

Also, `compileSrc` method changed it's signature so it can take
multiple source code snippets as input. This functionality will
be used in future commits.
2013-11-26 18:39:23 +01:00
Grzegorz Kossakowski aac19fd02b Extract source code dependencies by tree walking.
Previously incremental compiler was extracting source code
dependencies by inspecting `CompilationUnit.depends` set. This set is
constructed by Scala compiler and it contains all symbols that given
compilation unit refers or even saw (in case of implicit search).
There are a few problems with this approach:

  * The contract for `CompilationUnit.depend` is not clearly defined
    in Scala compiler and there are no tests around it. Read: it's
    not an official, maintained API.
  * Improvements to incremental compiler require more context
    information about given dependency. For example, we want to
    distinguish between dependency on a class when you just select
    members from it or inherit from it. The other example is that
    we might want to know dependencies of a given class instead of
    the whole compilation unit to make the invalidation logic more
    precise.

That led to the idea of pushing dependency extracting logic to
incremental compiler side so it can evolve indepedently from Scala
compiler releases and can be refined as needed. We extract
dependencies of a compilation unit by walking a type-checked tree
and gathering symbols attached to them.

Specifically, the tree walk is implemented as a separate phase that
runs after pickler and extracts symbols from following tree nodes:

  * `Import` so we can track dependencies on unused imports
  * `Select` which is used for selecting all terms
  * `Ident` used for referring to local terms, package-local terms
            and top-level packages
  * `TypeTree` which is used for referring to all types

Note that we do not extract just a single symbol assigned to `TypeTree`
node because it might represent a complex type that mentions
several symbols. We collect all those symbols by traversing the type
with CollectTypeTraverser. The implementation of the traverser is inspired
by `CollectTypeCollector` from Scala 2.10. The
`source-dependencies/typeref-only` test covers a scenario where the
dependency is introduced through a TypeRef only.
2013-11-26 18:39:23 +01:00
Grzegorz Kossakowski b8371691f2 Introduce `memberRef` and `inheritance` relations.
Introduce an alternative source dependency tracking mechanism that is
needed by upcoming name hashing algorithm. This new mechanism is
implemented by introducing two new source dependency relations called
`memberRef` and `inheritance`.

Those relations are very similar to existing `direct` and
`publicInherited` relations in some subtle ways. Those differences
will be highlighted in the description below.

Dependencies between source files are tracked in two distinct
categories:

  * dependencies introduced by inheriting from a class/trait
    defined in other source file
  * dependencies introduced by referring (selecting) a member
    defined in other source file (that covers all other
    kinds of dependencies)

Due to invalidation algorithm implementation details sbt would need to
track inheritance dependencies of public classes only. Thus, we had
relation called `publicInherited`. The name hashing algorithm which
improves invalidation logic will need more precise information about
dependencies introduced by inheritance including dependencies of non-public
classes. That's one difference between `inheritance` and `publicInherited`
relations.

One surprising (to me) thing about `publicInherited` is that it includes
all base classes of a given class and not just parents. In that sense
`publicInherited` is transitive. This is a bit irregular because
everything else in Relations doesn't include transitive dependencies.

Since we are introducing new relations we have an excellent chance to
make things more regular. Therefore `inheritance` relation is
non-transitive and includes only extracted parent classes.

The access to `direct`, `publicInherited`, `memberRef` and `inheritance`
relations is dependent upon the value of `memberRefAndInheritanceDeps`
flag. Check documentation of that flag for details.

The two alternatives for source dependency tracking are implemented by
introduction of two subclasses that implement Relations trait and one
abstract class that contains some common logic shared between those two
subclasses. The two new subclasses are needed for the time being when we
are slowly migrating to the name hashing algorithm which requires
subtle changes to dependency tracking as explained above. For some time we
plan to keep both algorithms side-by-side and have a runtime switch which
allows to pick one. So we need logic for both old and new dependency
tracking to be available. That's exactly what two subclasses of
MRelationsCommon implement. Once name hashing is proven to be stable and
reliable we'll phase out the old algorithm and the old dependency tracking
logic.
2013-11-26 18:39:13 +01:00
Grzegorz Kossakowski c5317a1f72 Work harder on generating unique values in generator for Analysis.
The TestCaseGenerators uses global set for ensuring that certain generated
values are unique. This is not the best design because the more properties
you check the harder is to generate new sample inputs because of already
accumulated values. This results in:

[info] + Analysis.Simple Merge and Split: OK, proved property.
[info] ! Analysis.Complex Merge and Split: Gave up after only 8 passed tests. 93 tests were discarded.

I don't have an ambition to reduce the scope of this global set but at
least I wanted to make generators to work a bit harder on generating
samples.

Instead of using `suchThat` method for filtering out non-unique samples
we use `retryUntil` that never gives up (therefore it might not
terminate). We had to upgrade to latest (1.11.1) version of scalacheck
in order to have an access to `retryUntil` method.

Also, I overridden the `identifier` to delegate to original
`Gen.identifier` but with minimal size set to be to '3'. This means,
the generated identifier will be of size 3 or larger which is needed in
order to avoid collisions.
2013-11-25 18:50:49 +01:00
Benjy e6bf85a30b Replace binary Analysis format with a text-based one.
Reads/writes are a little faster with the text format,
and it's far more useful. E.g., it allows external manipulation
and inspection of the analysis.

We don't gzip the output. It does greatly shrink the files,
however it makes reads and writes 1.5x-2x slower, and we're
optimizing for speed over compactness.
2013-11-16 13:59:02 -08:00
Grzegorz Kossakowski 698e24da11 Mark fields storing keys in IncOptions as private.
It was an omission in the original commit that introduced them and didn't
mark them as private. They are purely an implementation detail and should
be hidden. We hiding them now.
2013-11-12 21:39:18 +01:00
Grzegorz Kossakowski 39036e7c20 Make recompilation on macro definition optional.
Introduce a new incremental compiler option that controls
incremental compiler's treatment of macro definitions and their clients.
The current strategy is that whenever a source file containing a macro
definition is touched it will cause recompilation of all direct
dependencies of that file.

That strategy has proven to be too conservative for some projects like
Scala compiler of specs2 leading to too many source files being recompiled.
We make this behavior optional by introducing a new option
`recompileOnMacroDef` in `IncOptions` class. The default value is set to
`true` which preserves the previous behavior.
2013-11-12 21:33:19 +01:00
Grzegorz Kossakowski a6f04cf53b Add specialized copy methods to IncOptions class.
Add methods that allow one to set a new value to one of the fields of
IncOptions class. These methods are meant to be an alternative to
copy method that is hard to keep binary compatible when new fields are
added to the class.

Each copying method is related to one field of the class so when new
fields are added existing methods (and their signatures) are unaffected.
2013-11-12 21:32:14 +01:00
Grzegorz Kossakowski b77b0e161e Desugar case class IncOptions in binary compatible way.
Expand case class `IncOptions` in binary compatible way so we can have
better control of methods like `unapply` when new fields are added.

Great precaution has been taken to ensure that this commit doesn't break
binary compatibility. I took a dump of javap output before and after
this change for both the class and it's companion object.
The diff is presented below:

diff -u ~/inc-options-before ~/inc-options-after
--- /Users/grek/inc-options-before	2013-11-03 14:48:45.000000000 +0100
+++ /Users/grek/inc-options-after	2013-11-03 15:53:10.000000000 +0100
@@ -9,7 +9,11 @@
     public static java.lang.String transitiveStepKey();
     public static sbt.inc.IncOptions setTransactional(sbt.inc.IncOptions, java.io.File);
     public static sbt.inc.IncOptions defaultTransactional(java.io.File);
+    public static scala.Option unapply(sbt.inc.IncOptions);
+    public static sbt.inc.IncOptions apply(int, double, boolean, boolean, int, scala.Option, scala.Function0);
     public static sbt.inc.IncOptions Default();
+    public static scala.Function1 tupled();
+    public static scala.Function1 curried();
     public int transitiveStep();
     public double recompileAllFraction();
     public boolean relationsDebug();

diff -u inc-options-module-before inc-options-module-after
--- inc-options-module-before	2013-11-03 14:48:55.000000000 +0100
+++ inc-options-module-after	2013-11-12 21:00:41.000000000 +0100
@@ -3,6 +3,9 @@
     public static final sbt.inc.IncOptions$ MODULE$;
     public static {};
     public sbt.inc.IncOptions Default();
+    public final java.lang.String toString();
+    public sbt.inc.IncOptions apply(int, double, boolean, boolean, int, scala.Option, scala.Function0);
+    public scala.Option unapply(sbt.inc.IncOptions);
     public sbt.inc.IncOptions defaultTransactional(java.io.File);
     public sbt.inc.IncOptions setTransactional(sbt.inc.IncOptions, java.io.File);
     public java.lang.String transitiveStepKey();
@@ -13,7 +16,5 @@
     public java.lang.String apiDiffContextSize();
     public sbt.inc.IncOptions fromStringMap(java.util.Map);
     public java.util.Map toStringMap(sbt.inc.IncOptions);
-    public sbt.inc.IncOptions apply(int, double, boolean, boolean, int, scala.Option, scala.Function0);
-    public scala.Option unapply(sbt.inc.IncOptions);
 }

The first diff shows that there are just more static forwarders defined
for top-level companion object and that is binary compatible change.

The second diff shows that there are just a few minor differences in
order in which `unapply`, `apply` and bridge method for `apply` are
defined. Also, there's a new `toString` declaration. All those changes are
binary compatible.

All methods that are generated for a case class are marked as deprecated
and will be removed in the future.
2013-11-12 21:31:28 +01:00
Grzegorz Kossakowski 4b43110a2c Represent api changes as values and cleanup APIChanges class.
The main motivation behind this commit is to reify information about
api changes that incremental compiler considers. We introduce a new
sealed class `APIChange` that has (at the moment) two subtypes:

  * APIChangeDueToMacroDefinition - as the name explains, this represents
    the case where incremental compiler considers an api to be changed
    just because given source file contains a macro definition
  * SourceAPIChange - this represents the case of regular api change;
    at the moment it's just a simple wrapper around value representing
    source file but in the future it will get expanded to contain more
    detailed information about API changes (e.g. collection of changed
    name hashes)

The APIChanges becomes just a collection of APIChange instances.
In particular, I removed `names` field that seems to be a dead code in
incremental compiler. The `NameChanges` class and methods that refer to
it in `SameAPI` has been deprecated.

The Incremental.scala has been adapted to changed signature of APIChanges
class. The `sameSource` method returns representation of APIChange
(if there's one) instead of just simple boolean. One notable change is
that information about APIChanges is pushed deeper into invalidation logic.
This will allow us to treat the APIChangeDueToMacroDefinition case properly
once name hashing scheme arrives.

This commit shouldn't change any behavior and is purely a refactoring.
2013-11-11 15:43:28 +01:00
Grzegorz Kossakowski 4ed8abd4fb More detailed logging of incremental compiler's invalidation logic.
The following events are logged:

  * invalidation of source file due to macro definition
  * inclusion of dependency invalidated by inheritance; we log both
    nodes of dependency edge (dependent and dependency)

The second bullet helps to understand what's going on in case of
complex inheritance hierarchies like in Scala compiler.
2013-11-11 15:27:18 +01:00
Mark Harrah 2f683ef81d Generate an error when the incremental compiler is given relative source files. Fixes #923.
Review by @gkossakowski
2013-11-08 20:43:26 -05:00
Grzegorz Kossakowski 33514ab6d7 Recover from class files and Analysis object getting out of sync.
The #958 describes a scenario where partially successful results are
produced in form of class files written to disk. However, if compilation
fails down the road we do not record any new compilation results (products)
in Analysis object. This leads to Analysis object and disk contents to get
out of sync.

One way to solve this problem is to use transactional ClassfileManager that
commits changes to class files on disk only when entire incremental
compilation session is successful. Otherwise, new class files are rolled
back to previous state.

The other way to solve this problem is to record time stamps of class files
in Analysis object. This way, incremental compiler can detect that class
files and Analysis object got out of sync and recover from that by
recompiling corresponding sources.

This commit uses latter solution which enables simpler (non-transactional)
ClassfileManager to handle scenario from #958.

Fixes #958
2013-11-08 11:29:17 +01:00
Benjy 8779409605 Allow zinc to use InternedAnalysisFormats. 2013-11-04 09:30:28 -05:00
Grzegorz Kossakowski a37d8d4770 Fix unstable existential type names bug.
Fix the problem with unstable names synthesized for existential
types (declared with underscore syntax) by renaming type variables
to a scheme that is guaranteed to be stable no matter where given
the existential type appears.

The sheme we use are De Bruijn-like indices that capture both position
of type variable declarion within single existential type and nesting
level of nested existential type. This way we properly support nested
existential types by avoiding name clashes.

In general, we can perform renamings like that because type variables
declared in existential types are scoped to those types so the renaming
operation is local.

There's a specs2 unit test covering instability of existential types.
The test is included in compiler-interface project and the build
definition has been modified to enable building and executing tests
in compiler-interface project. Some dependencies has been modified:

  * compiler-interface project depends on api project for testing
    (test makes us of SameAPI)
  * dependency on junit has been introduced because it's needed
    for `@RunWith` annotation which declares that specs2 unit
    test should be ran with JUnitRunner

SameAPI has been modified to expose a method that allows us to
compare two definitions.

This commit also adds `ScalaCompilerForUnitTesting` class that allows
to compile a piece of Scala code and inspect information recorded
callbacks defined in  `AnalysisCallback` interface. That class uses
existing ConsoleLogger for logging. I considered doing the same for
ConsoleReporter. There's LoggingReporter defined which would fit our
usecase but it's defined in compile subproject that compiler-interface
doesn't depend on so we roll our own.

ScalaCompilerForUnit testing uses TestCallback from compiler-interface
subproject for recording information passed to callbacks. In order
to be able to access TestCallback from compiler-interface
subproject I had to tweak dependencies between interface and
compiler-interface so test classes from the former are visible in the
latter. I also modified the TestCallback itself to accumulate apis in
a HashMap instead of a buffer of tuples for easier lookup.

An integration test has been added which tests scenario
mentioned in #823.

This commit fixes #823.
2013-10-29 16:39:50 +01:00
Mark Harrah 7bcc00f637 Merge remote-tracking branch 'gkossakowski/separate-dependency-phase' into 0.13 2013-10-24 16:57:32 -04:00
Grzegorz Kossakowski 59de0f00b0 Remove long comment that explains phase ordering issues.
As pointed out by @harrah in #705, we might want to merge both API
and dependency phases so we should mention that in the comment explaining
phase ordering constraints instead.

I'd still like to keep the old comment in the history (as separate commit)
because it took me a while to figure out cryptic issues related to
continuations plugin so it's valuable to keep the explanation around in
case somebody else in the future tries to mess around with dependencies
defined by sbt.
2013-10-24 16:44:46 +02:00
Grzegorz Kossakowski e8746dc0c7 Add a bit documentation to Dependency phase.
It gives some high-level overview of what this phase does.
2013-10-24 16:44:45 +02:00
Grzegorz Kossakowski 838416360a Move dependency extraction into separate compiler phase.
This is the first step towards using new mechanism for dependency
extraction that is based on tree walking.

We need dependency extraction in separate phase because the code
walking trees should run before refchecks whereas analyzer phase runs
at the very end of phase pipeline.

This change also includes a work-around for phase ordering issue with
continuations plugin. See included comment and SI-7217 for details.
2013-10-24 16:44:45 +02:00
Grzegorz Kossakowski fea18a4fbe Remove AnalysisCallback.{beginSource, endSource} methods.
As pointed out by @harrah in #705, both beginSource and endSource are
not used in sbt internally for anything meaningful.

We've discussed an option of deprecating those methods but since they
are not doing anything meaningful Mark prefers to have compile-time
error in case somebody implements or calls those methods. I agree with
that hence removal.
2013-10-24 16:44:45 +02:00
Benjy 3d5a348e1a Deprecate AnalysisFormats. 2013-10-23 11:17:51 -04:00
Benjy 8046ad7eec Added some modifiers, per code review comments. 2013-10-23 10:58:44 -04:00
Benjy c66eefcec9 Switch to using the interned serialization format. 2013-10-23 10:41:13 -04:00
Benjy 0b033337df Interned serialization format for Analysis objects. 2013-10-22 13:02:53 -04:00
Benjy 948a9cf5cc Change how test labels are combined, per code review comments. 2013-10-18 15:34:47 -07:00
Mark Harrah e3e95f902d Merge remote-tracking branch 'benjy/analysis_test' into 0.13 2013-10-17 15:17:57 -04:00
Mark Harrah 49f2f9fe1a Merge remote-tracking branch 'benjy/internpool2' into 0.13 2013-10-17 15:17:43 -04:00
Benjy da6af7c5f7 Test for Analysis split/merge.
Requires scalacheck generators for Analysis and its subobjects. These
may be useful for other tests in the future.

Also fixes a bug in RelationTest.
2013-10-17 11:56:10 -07:00
Mark Harrah 4506c8b877 remove dead APIFormats code 2013-10-16 19:26:26 -04:00
Benjy 9c380a9f1f An intern pool implementation, and associated serialization formats. 2013-10-15 13:26:20 -07:00
Benjy 30cdb1a1bb Add correct implementations of merge/groupBy to Analysis.
Copious comments to explain the non-trivial logic.
2013-10-13 21:48:02 -07:00
Benjy b208f47275 Add a merge method to Compilations.
Also fixed a cut-n-paste mishap in Relations.
2013-10-10 19:47:56 -07:00
Benjy c3aad1cd32 Deprecate Relations.groupBy.
Also add equals/hashCode implementations for MRelations.

Also add some comments to explain that ++ and -- are naively implemented.

Also fix some tabs-vs-spaces indentation nits.
2013-10-10 13:21:08 -07:00
Mark Harrah 022307c71e Merge remote-tracking branch 'benjy/apis3' into 0.13 2013-10-10 14:24:18 -04:00
Mark Harrah 89114bcb59 Merge remote-tracking branch 'benjy/stamps2' into 0.13 2013-10-10 14:24:12 -04:00
Mark Harrah 6f2b4eaa8c Merge remote-tracking branch 'benjy/sourceinfos' into 0.13 2013-10-10 14:23:38 -04:00
Benjy 95226e8639 Add equals/hashCode to APIs, and deprecate groupBy.
equals/hashCode are useful for debugging/verifying/testing,
and the groupBy implementation is naive. It'll be replaced
by a groupBy implementation in Analysis that will handle
external/internal dep transitions correction.
2013-10-10 11:17:33 -07:00
Benjy 45a3a8ae0a Add useful methods to Stamps.
- Adds equals/hashCode to Stamp and Stamps.
- Adds a static method to merge multiple Stamps instances into one.
2013-10-10 10:52:55 -07:00
Benjy 347ef1a881 Add a merge method for SourceInfos.
Also fixes a naming consistency nit.
2013-10-10 10:37:32 -07:00
Mark Harrah 8941d4f9f5 minor fixes to Relations API documentation 2013-10-10 10:43:16 -04:00
Mark Harrah b8b6426cf9 Allow main class to be non-public. Fixes #883. 2013-09-26 09:42:30 -04:00
Jason Zaugg c4efcc4df7 Make the DelegatingReporter aware of -nowarn
The test case compiles a project without and with this
setting and checks that a warning is and isn't emitted
respectively.

It's a multi-project build; this bug didn't seem to turn
up in a single-project build.
2013-09-05 13:31:04 -04:00
Grzegorz Kossakowski beea6a9b4a Move API extraction logic to a separate class.
This way we have a little bit more clear separation
between compiler phase logic and the core logic responsible for
processing each compilation unit and extracting an api for it.

As added benefit, we have a little bit less of mutable state
(e.g. sourceFile doesn't need to be a var anymore).

The API extraction logic contains some internal caches that are
required for correctness. It wasn't very clear if they have to
be maintained during entire phase run or just during single compilation
unit processing. It looks like they have to be maintained during
single compilation unit processing and refactored code both
documents that contracts and implements it in the API phase.
2013-07-24 15:18:44 -07:00
Grzegorz Kossakowski 7d4cf7b8ab Factor out class file lookup out of Analyzer class.
Move logic related to class file lookup to separate class that
can be reused outside of Analyzer class.
2013-07-23 17:11:42 -07:00
Grzegorz Kossakowski aec466cb4d Factor out compiler interface compatibility layer.
Move collection (a class `Compat`) of compatibility hacks into separate
file. This aids understanding of the code as both Analyzer and API make
use of that class and keeping it `Analyzer.scala` file suggested that
it's used only by Analyzer.
2013-07-23 15:19:24 -07:00
Mark Harrah ab5e81fc23 Merge remote-tracking branch 'cancel-bug' into 0.13 2013-07-19 18:56:01 -04:00
Grzegorz Kossakowski d77930394f Handle compilation cancellation properly.
Incremental compiler didn't have any explicit logic to handle
cancelled compilation so it would go into inconsistent state.

Specifically, what would happen is that it would treat cancelled
compilation as a compilation that finished normally and try to
produce a new Analysis object out of partial information collected
in AnalysisCallback. The most obvious outcome would be that the
new Analysis would contain latest hashes for source files. The
next time incremental compiler was asked to recompile the same files
that it didn't recompile due to cancelled compilation it would think
they were already successfully compiled and would do nothing.

We fix that problem by following the same logic that handles compilation
errors, cleans up partial results (produced class files) and makes sure
that no Analysis is created out of broken state.

We do that by introducing a new exception `CompileCancelled`
and throwing it at the same spot as an exception signalizing compilation
errors is being thrown. We also modify `IncrementalCompile` to
catch that exception and gracefully return as there was no compilation
invoked.

NOTE: In case there were compilation errors reported _before_
compilation cancellations was requested we'll still report them
using an old mechanism so partial errors are not lost in case
of cancelled compilation.
2013-07-19 14:39:26 -07:00
Mark Harrah 4b8f0f3f94 Use IMain.bindValue to bind repl values. This does a better job of getting the type to use for a bound value. 2013-07-17 14:58:53 -04:00
Mark Harrah 4bf4a80b5e Deleted unused type parsing class 2013-07-09 14:55:30 -04:00
Mark Harrah 931c297f52 make APIDiff private[inc] 2013-06-25 08:13:06 -04:00
Grzegorz Kossakowski 0d2dc75115 Make api diff context size valuable configurable.
Add `apiDiffContextSize` option to `IncOptions` which allows one
to control the size (in lines) of a context used when printing
diffs for textual API representation.

The default value for `apiDiffContextSize` is 5 which seems to be
enough for most situations. This is verified by many debugging
sessions I performed when using api diffing functionality.
2013-06-25 00:04:45 -07:00
Grzegorz Kossakowski 025eae9103 Log API diffs using ShowAPI and java-diff-utils library.
Implement displaying API changes by using textual representation
of an API (ShowAPI) and good, old textual diff algorithm. We are
using java-diff-utils library that is distributed under Apache 2.0
license.

Notice that we have only soft dependency on java-diff-utils. It means
that we'll try to lookup java-diff-utils class through reflection
and fail gracefully if none is found on the classpath. This way
sbt is not getting any new dependency. If user needs to debug
api diffs then it's matter of starting sbt with
`-Dsbt.extraClasspath=path/to/diffutils.jar` option passed
to sbt launcher.
2013-06-25 00:04:39 -07:00
xuwei-k 3f2133729b allow java varargs main
http://docs.oracle.com/javase/specs/jls/se7/html/jls-12.html#jls-12.1.4
2013-06-16 15:47:47 -04:00
Mark Harrah cd2b1edf6c IncOptions class backup directory should be per-compilation, not shared.
Otherwise, one concurrent compilation will clean the directory out from under another.
2013-05-30 15:00:26 -04:00
Mark Harrah 0a7a579f5b Merge ExtendedReporter into Reporter. 2013-05-27 19:12:39 -04:00
Mark Harrah d2d406969b Make imports from MurmurHash explicit for clarity. 2013-05-27 19:12:39 -04:00
Mark Harrah 7088a7dd5a test cases for Java inherited dependency extraction 2013-05-02 12:01:01 -04:00
Mark Harrah 626038bece Merge branch 'feature/inc-track-inherit' into 0.13 2013-05-01 19:25:01 -04:00
Mark Harrah a867d8e87c extract public inherited dependencies from Java class files 2013-05-01 17:54:10 -04:00
Mark Harrah 435bd1d587 Only invalidate package objects that inherit from invalidated files.
Originally described in cf355f1822.
2013-05-01 09:35:53 -04:00
Mark Harrah 0b876cc57d fix compiler interface compatibility with 2.11 2013-04-27 16:27:57 -04:00
Mark Harrah bedc8dbb10 Push full transitive invalidation out a step since step 3 is now relatively cheap. 2013-04-26 22:36:15 -04:00
Mark Harrah 658c3d06c4 Use public inherited dependencies in incremental compilation invalidation.
1. All parents of public/exported classes/modules/packages are tracked as
   'publicInherited' dependencies.  These are dealiased and normalized so
    that the dependency is on the actual underlying template and not the
    source enclosing the alias.
2. All CompilationUnit.depends dependencies are direct dependencies.  These
   include inherited dependencies.
3. When invalidating changed internal sources,
	a. Invalidate all inherited dependencies, transitively and include the
      originally modified sources,
	b. Invalidate all direct dependencies of these sources,
	c. Exclude any sources that were compiled in the previous step unless they
      depend on a newly invalidated source.
4. Invalidate changed external sources in the same way as #3 but remove the
   external sources from the final set.

Only public inheritance dependencies need to be considered because a template
that is not accessible outside its source file and that inherits from another
file can be handled as a normal, direct dependency.  Because the template
isn't public, changes to its API will not propagate outside of the source
file.

Several existing tests cover the correctness, especially:

1. transitive-a covers direct, transitive dependencies with inferred return
   types
2. transitive-b covers inherited, transitive dependencies with inferred return
   types

There are two new tests, one that tests that public inherited dependencies are
tracked and one that verifies the basic invalidation progression.
More tests are needed to verify the improvements that this algorithm brings:

1. Inheritance-related dependencies are processed in one step to avoid the
   otherwise unavoidable several steps.
2. Only immediate direct dependencies are ever processed, which should in many
   typical cases avoid large invalidation sets.
2013-04-26 22:36:06 -04:00
Mark Harrah 4dc75343ae Record and persist public inheritance dependencies.
Includes placeholders for adding public inherited dependencies for Java classes.
2013-04-26 22:35:27 -04:00
Mark Harrah c355bef200 Track public inherited dependencies.
There is a public inherited dependency on each (normalized) base
class of a public template (class, module, trait, structural type).
2013-04-26 22:35:27 -04:00
Mark Harrah 0e1f211fe5 move to compiler's built-in moduleSuffix method 2013-04-26 18:52:16 -04:00
Grzegorz Kossakowski 3ba9348740 Do not normalize types in the api extraction phase.
In summary this commit:

  * drops type normalization in api phase but keeps dealiasing
  * fixes #736 and marks corresponding test as passing

I discussed type normalization with @adriaanm and according to him
sbt shouldn't call that method. The purpose of this method to
convert to a form that subtyping algorithm expects. Sbt doesn't need
to call it and it's fairly expensive in some cases.

Dropping type normalization also fixes #726 by not running into
stale cache in Scala compiler problem described in SI-7361.
2013-04-27 00:31:31 +02:00
Grzegorz Kossakowski fbe4dedd11 Remove trailing whitespace in API.scala 2013-04-27 00:31:31 +02:00
Mark Harrah bef8ce05b9 Properly track 'abstract override' modifier. Ref #726. 2013-04-15 14:12:15 -04:00
Mark Harrah 77001a4259 drop canonicalization of files on classpath and other cleanup. Fixes #723. 2013-04-09 20:13:06 -04:00
Mark Harrah 6def08e029 remove resident compiler code
The infrastructure for resident compilation still exists,
but the actual scalac-side code that was backported is removed.
Future work on using a resident scalac will use that invalidation
code directly from scalac anyway.
2013-04-09 20:13:06 -04:00
Mark Harrah d7971f6857 temporary option to disable classpath lookup in incremental compiler. Ref #710. 2013-03-14 16:47:46 -04:00
Mark Harrah 6b7fbfe779 Provided details of why external binaries are detected as modified in incremental compiler debug output. 2013-03-13 12:40:03 -04:00
Mark Harrah c9c392d424 toString for Stamp types 2013-03-13 12:40:03 -04:00
Mark Harrah 008c2928b6 note that mixing RangePositions into Global isn't necessary in 2.11 2013-03-13 12:40:03 -04:00
Mark Harrah 925ec98900 Restore class files after an unsuccessful compilation.
This is useful when an error occurs in a later incremental step that
requires a fix in the originally changed files.

CC @gkossakowski
2013-03-04 07:24:31 -05:00
Mark Harrah f2d29d8678 Export approximate command lines executed for 'doc', 'compile', and 'console' 2013-02-28 17:59:38 -05:00
Mark Harrah 1870edfd27 more deprecations 2013-02-26 07:39:23 -05:00
Mark Harrah f6d73128fc deprecations 2013-02-25 09:24:04 -05:00
Mark Harrah a1c70c7462 Include jars from java.ext.dirs in incremental classpath. Fixes #678.
Ideally, we wouldn't need to construct the classpath ourselves and instead
reuse the classpath construction code from a compiler (scalac or javac).
However, we need to ensure that we only call the compiler when needed.
Because we need to construct the classpath even when compilation might
not happen, we have to duplicate classpath construction.
2013-02-22 16:31:32 -05:00
Mark Harrah 6e30bd7842 short-circuit the macro check in the incremental compiler if the source isn't a Scala source file 2013-02-22 16:31:32 -05:00
Grzegorz Kossakowski 4c261d360f Revert "Revert "Fix sbt/sbt#676: swapped declarations and members in `ClassToAPI`.""
This reverts commit 9dd5f076ea which was a revert
itself so we are back to the state before those two reverts.

The problem that caused revert that happened in 9dd5f076ea
has been fixed in 88061dd262.
2013-02-22 07:36:55 -05:00
Grzegorz Kossakowski 129df3e812 Fix problem with initialization order in `ClassToAPI.scala`.
In `ClassToAPI` both `Private` and `Protected` vals had forward
reference to `Unqualified` so they would get `null` as a result.
Fixed that by rearranging the order of vals being declared.

This fixes a problem described in 9dd5f076ea.
2013-02-22 07:36:55 -05:00
Mark Harrah 9dd5f076ea Revert "Fix sbt/sbt#676: swapped declarations and members in `ClassToAPI`."
This reverts commit beb87f2789.  It causes java/options to fail.

CC @gkossakowski
2013-02-21 20:44:54 -05:00
Grzegorz Kossakowski 5119ea2574 Mark inherited definitions in `ShowAPI`.
Mark every inherited definition with `^inherited^` marker. This helps
understanding `ShowAPI`s output.
2013-02-20 10:52:28 -05:00
Grzegorz Kossakowski f101bcd3d0 Show some of inherited members of a structure in `ShowAPI.scala`
Showing inherited members of a structure was disabled so we would not
run into cycles. To best of my knowledge, we can run into cycles only
if inherited member is `ClassLike`. We filter out those and let
anything else to be shown.
2013-02-20 10:52:27 -05:00
Grzegorz Kossakowski beb87f2789 Fix sbt/sbt#676: swapped declarations and members in `ClassToAPI`.
As described in sbt/sbt#676, arguments to `mergeMap` and `merge` methods
were swapped causing wrong structures being created for Java compiled
class files.

This commit fixes sbt/sbt#676 and makes pending test to pass now.
2013-02-20 10:52:27 -05:00
Grzegorz Kossakowski f885458f37 Do not minimize APIs if API debugging option is enabled.
That's foundation for API dumping and tests based on API representation contents.

Specific list of changes introduced by this commit:
  * `AnalysisCallback` class takes `IncOptions` as argument so it
    can determine if API should be minimized in `api` callback.
  * Introduce `Incremental.apiDebug` method that determines if api debugging
    is enabled. There are two ways to enable api debugging: through system
    property and through incremental compiler options. The system property
    method will be soon deprecated. We introduce it to make it easier to enable
    API debugging until tools (like zinc and ide) catch up with making incremental
    compiler configurable.

NOTE: The `apiDebug` method has been introduced in Incremental for two reasons:
  1. It's analogous to `incDebug` method that's already there.
  2. In other branch I need `apiDebug` to be defined in Incremental.

Once we deprecate and remove enabling debugging options through system properties
the code will be cleaned up.
2013-02-20 10:52:27 -05:00
Grzegorz Kossakowski f3d2df0f9c Use `IncOptions` for `transitiveStep` and `recompileAllFraction`
Recently introduced IncOptions allow us to make both transitiveStep
and recompileAllFraction configurable in incremental compiler.
2013-02-19 12:23:07 -05:00
Grzegorz Kossakowski 70036812ab Introduce incremental compiler options.
Introduce a way to configure incremental compiler itself instead
of underlying Java/Scala compiler.

Specific list of changes in this commit:
  * Add a method to `xsbti.compile.Setup` that returns incremental
    compiler options as a `java.util.Map<String, String>`. We considered
    statis interface instead of a `Map` but based on mailing
    list feedback we decided that it's not the best way to go because
    static interface is hard to evolve it by adding new options.
  * Since passing `java.util.Map<String, String>` not very convenient
    we convert it immediately to `sbt.inc.IncOptions`
  * Add options argument to various methods/classes that implement
    incremental compilation so in the end options reach
    `sbt.inc.IncOptions` object
  * Add `incOptions` task that allows users to configure incremental
    compiler options in their build files. Default implementation of
    that tasks returns just `IncOptions.DEFAULT`
  * Both system property `xsbt.inc.debug` and `IncOptions.relationsDebug`
    trigger debugging of relations now. In the near future, we should
    deprecate use of `xsbt.inc.debug`.
2013-02-19 12:23:07 -05:00
Grzegorz Kossakowski 4fe0c02536 Break long argument list into multiple lines.
Some methods take a lot of arguments and I'm about to add one more
which will make them too long for easy reading.

This change is changes code formatting only. That's done on purpose
to make it easier to review other changes.
2013-02-19 12:18:26 -05:00
Grzegorz Kossakowski d6f4c5ae4f Strip down trailing whitespace.
I have Eclipse configured to do that automatically when saving file.
I decided to finally commit those changes to files I touch a lot.
2013-02-19 12:18:26 -05:00
Jason Zaugg c597b48eda Stop using Predef.error
We'd like to remove this from 2.11.0.

This patch should be backported to 0.12.3.
2013-01-28 22:29:48 +01:00
Paolo G. Giarrusso 19aaaea923 Fix compile error in Eclipse
Let's consider compile/inc/src/main/scala/sbt/CompileSetup.scala.
There are multiple Output types, and according to Eclipse importing
xsbti.compile.Output within the package sbt does not work because the
import is shadowed by sbt.Output.

However, compilation proceeds just fine within SBT. Reproducing the
example however gives the same warning, if the files are in the same
project. The problem here is probably that the shadowing Output
is declared in the same package but in another project, and that seems
to give different results in Eclipse and SBT, but relying on
that looks fragile.
Reading the spec is inconclusive since it doesn't match with Scalac's
behavior — see
https://groups.google.com/d/topic/scala-internals/-Rquc2HBYLk/discussion .

ForkTests has the same behavior as CompileSetup.
2013-01-22 09:05:15 -05:00
Mark Harrah 070a87dda4 Call non-deprecated isRawType instead of isRaw 2013-01-16 10:26:32 -05:00
Mark Harrah cf355f1822 Fix incremental compilation problem with package objects inheriting from invalidated sources in a subpackage.
Resolution of https://issues.scala-lang.org/browse/SI-4695 seems to be to deprecate
inheriting from a class in a subpackage.  This commit is an alternative solution,
possibly to be reverted or restricted if resolution of SI-4695 changes or if this
proves to be too conservative in practice.

Review by @gkossakowski.  With separate inheritance/function call dependency tracking,
this probably should only pull in package objects with inheritance dependencies on
invalidated files.
2012-12-19 10:17:56 -05:00
Mark Harrah 329b42453b move Eval test to correct location 2012-12-19 10:17:56 -05:00
Mark Harrah 71f12fbcdb 2.9 compatibility for incremental compilation modules 2012-12-11 14:41:22 -05:00
Grzegorz Kossakowski 4c1c31e190 Fix for dependency on class file corresponding to a package. (#620)
While trying to determine binary dependencies sbt lookups class files
corresponding to symbols. It tried to do that for packages and most of the
time would fail because packages don't have corresponding class file
generated. However, in case of case insensitive file system, combined
with special nesting structure you could get spurious dependency.
See added test case for an example of such structure.

The remedy is to never even try to locate class files corresponding to
packages.

Fixes #620.
2012-12-10 13:53:52 -05:00
Grzegorz Kossakowski 78b0f485fa Add `compilations` field to `Analysis`.
We store `Seq[xsbt.api.Compilation]` in `Analysis`. Compilation are
being appended to sequence for every iteration of the incremental
compiler.

Note that `Compilation`s are never removed from the sequence unless
you start from scratch with empty `Analysis`. You can do that by using
sbt's `clean` command.

The main use-case for using `compilations` field is to determine how
many iterations it took to compilen give code. The `Compilation` object
are also stored in `Source` objects so there's an indirect way to recover
information about files being recompiled in every iteration.

Since `Analysis` is persisted you can use this mechanism to track entire
sessions spanning multiple `compile` commands.
2012-12-10 09:15:52 -05:00
Mark Harrah e3745540c9 More cleanup of Scala version handling, including managedScalaInstance to control automatic 'scala-tool' configuration and dependencies. 2012-12-09 20:40:41 -05:00
Mark Harrah d4fd136192 support defining Projects in .sbt files
vals of type Project are added to the Build
2012-12-09 20:40:41 -05:00
Grzegorz Kossakowski ef39aeb9c1 Follow source layout convention supported by Eclipse.
Moved source files so directory structure follow package
structure. That makes it possible to use Scala Eclipse plugin
with sbt's source code.
2012-12-07 10:27:08 -08:00
Adriaan Moors f7be122eb4 Run apiExtractor after pickler (configurable)
Extract the api after picklers, since that way we see the same symbol
information/structure irrespective of whether we were typechecking
from source / unpickling previously compiled classes.

Previously, the apiExtractor phase ran after typer.

Since this fix is hard to verify with a test (it's based on the
conceptual argument above, and anecdotal evidence of incremental
compilation of a big codebase), we're providing a way to restore the
old behaviour: run sbt with -Dsbt.api.phase=typer.

This fixes #609.
2012-12-06 19:35:51 -05:00
Adriaan Moors cb0e723923 Fix #610: represent refinement typerefs stably
goal:

a representation of a type reference to a refinement class that's stable
across compilation runs (and thus insensitive to typing from source or
unpickling from bytecode)

problem:

the current representation, which corresponds to the owner chain of the
refinement:

  1. is affected by pickling, so typing from source or using unpickled
  symbols give different results (because the unpickler "localizes"
  owners -- this could be fixed in the compiler in the long term)

  2. can't distinguish multiple refinements in the same owner (this is
  a limitation of SBT's internal representation and cannot be fixed in
  the compiler)

solution:

expand the reference to the corresponding refinement type: doing that
recursively may not terminate, but we can deal with that by
approximating recursive references (all we care about is being sound for
recompilation: recompile iff a dependency changes, and this will happen
as long as we have one unrolling of the reference to the refinement)
2012-12-06 19:11:12 -05:00
Lex Spoon b7250bb528 Fix -Yrangepos. Unlike other settings, it requires that a mixin be added to Global. 2012-12-03 20:34:11 -05:00
Mark Harrah 1c22478edc task setting macros for :=, +=, ++=
also, bump to 2.10.0-M6
2012-11-17 20:19:24 -05:00
Mark Harrah 87e406fcbd support vals/defs in build.sbt
* must start with val, lazy val, or def (no modifiers currently)
* visible only within the same .sbt file
* multiple definitions allowed without being separated by blank lines
* no blank lines allowed within a definition
2012-11-17 16:27:25 -05:00
Benjy 657d842238 Analysis.groupBy implementation. 2012-11-10 19:01:41 -05:00
Mark Harrah 52b7dccff4 Handle removed files when skipping to recompiling all files. 2012-11-10 18:18:15 -05:00
Mark Harrah 4946115521 Recompile everything when invalidated sources exceed a certain fraction of all sources 2012-11-05 10:02:33 -05:00
nau bdfb570a42 Update compile/inc/Incremental.scala
Fix for http://stackoverflow.com/questions/12972183/sbt-always-does-full-rebuild-because-of-modified-binary-dependency-rt-jar
def externalBinaryModified uses java.io.File.equals() to check if files are the same. It's better to use File.getCanonicalPath in this case.
2012-10-22 08:23:23 -04:00
Mark Harrah d6072275aa replace Symbol.nameString calls with simpleName(Symbol). Fixes #577.
nameString is only for printing and has different behavior when scalac is
given -uniqid.
2012-10-15 12:42:27 -04:00
Mark Harrah 18a03f0e25 API extraction: handle any type that is annotated, not just the spec'd simple type. Fixes #559. 2012-10-05 09:06:35 -04:00
James Roper 702ee44e0e Source position mapper support 2012-08-31 06:24:16 -04:00
James Roper 5e2fc5f005 Reported file name when an error occurs while opening a corrupt zip file in Locate 2012-08-31 06:15:33 -04:00
Mark Harrah eca9ab9bb1 2.10 compatibility 2012-08-27 10:49:13 -04:00
Mark Harrah b53104f8ff note about Scala build and compiler interface 2012-08-27 08:41:08 -04:00
Mark Harrah 42042a9d85 2.8.1 compatibility for compiler interface 2012-08-27 08:33:33 -04:00
Mark Harrah 8c0a2fbe1c require a smaller set of sources in the third (transitive) step of inc. compilation 2012-08-26 13:44:32 -04:00
Mark Harrah 089b4e284c three-stage incremental compilation 2012-08-25 07:15:23 -04:00
Eugene Vigdorchik 61dee253dd Convert println() to log.debug() 2012-08-19 22:19:39 -04:00
Eugene Vigdorchik 1b814ae8b1 Extend reporter to be used by the IDE. 2012-07-24 15:35:06 -04:00
Eugene Vigdorchik 76943e82ed Add xsbti.Reporter to required inputs instead of maxErrors. 2012-07-13 14:33:26 -04:00
Eugene Vigdorchik b5a29987e6 Changes required to use sbt as-is from Scala-IDE. 2012-07-13 14:33:26 -04:00
Eugene Vigdorchik e23df839b7 Make compiler projects compilable with 2.10 2012-07-13 14:33:26 -04:00
Eugene Vigdorchik a6c2054292 Break compiler dependency from 'collection' project. 2012-07-13 14:33:26 -04:00
Mark Harrah d8f5c39465 remove most occurrences of ScalaObject 2012-07-01 15:16:41 -04:00
Mark Harrah 3f12f2eb9f drop compilation tests
1. overlapped with integration tests
2. slower than integration tests
3. more fragile than integration tests
2012-07-01 15:16:41 -04:00
Mark Harrah b405110e7b compilation timings 2012-06-23 13:52:09 -04:00
Mark Harrah 0de4444665 sync resident compiler code 2012-06-18 08:18:39 -04:00
Mark Harrah f53d20a7a3 disable resident-compiler related code paths when it isn't being used. fixes #486.
The underlying issue with the resident compiler needs fixing, however.
2012-06-16 23:40:52 -04:00
Mark Harrah 919ac33c1a compiler interface compatibility hardening 2012-06-10 23:06:35 -04:00
Mark Harrah b7df87e2d8 compiler interface source compatibility with Scala 2.8.1/0 2012-06-02 19:03:57 -04:00
Mark Harrah 4bc993e0ba Revert "remove unneeded unlinking in compiler interface"
Unlinking is actually needed.

This reverts commit 1581d1b7e1.
2012-06-02 19:03:57 -04:00
Mark Harrah 1581d1b7e1 remove unneeded unlinking in compiler interface 2012-05-30 07:41:02 -04:00
Mark Harrah 2db0101eb8 resident compiler that passes all tests
core logic from odersky/scala/topic/inkling
2012-05-30 07:13:15 -04:00
Peter Vlugter 737b1b09bd Add scala library when compiling compiler interface sources 2012-05-25 07:21:05 -04:00
Mark Harrah 9ee30636b3 don't put scala-library.jar on the boot classpath unless it is on the classpath 2012-05-21 22:23:44 -04:00
Mark Harrah 63895a3d77 source compatibility with 2.8.1, where resident mode can't be supported 2012-05-19 18:20:20 -04:00
Mark Harrah 5ff33fad3e Second try at printing message when stack trace suppressed.
Problems:

  1. Without a message, users don't find 'last'
  2. Showing a message for every error clutters output.

This tries to address these issues by:

1. Only showing the message when other feedback has not been provided and
   'last' would not usually be helpful.  This will require ongoing tweaking.
   For now, all commands except 'compile' display the message.  'update' could
   omit the message as well, but perhaps knowing about 'last' might be
   useful there.

2. Including the exact command to show the output:
    last test:compile
   and not just
    last <task>

3. Highlighting the command in blue for visibility as an experiment.

Review by @ijuma and @retronym, please.
2012-05-19 18:20:19 -04:00
Mark Harrah 50ddd485af add a missing 'lazy' 2012-05-13 22:38:00 -04:00
Mark Harrah 208b46fd10 resident mode: package objects 2012-05-13 21:31:40 -04:00
Mark Harrah 7bed381bec cleanup compilation tests 2012-05-12 23:12:29 -04:00
Mark Harrah 864580aae1 approximate type parameters and references by name
not as accurate, but simpler.
2012-05-12 23:12:29 -04:00
Mark Harrah 00d0918714 workaround separate compilation and raw types 2012-05-12 23:12:29 -04:00
Peter Vlugter 1ed64e4d5f Use Maybe rather than Option in IC 2012-05-07 19:09:25 -04:00
Peter Vlugter fdf9be166f Add helper methods to IC for reading analysis cache files 2012-05-06 21:41:38 -04:00