Every time that the compile task is run, there are potentially a large
number of iops that must occur in order for sbt to generate the source
file list as well as for zinc to check which files have changed since
the last build. This can lead to a noticeable delay between when a build
is started (either manually or by triggered execution) and when
compilation actually begins. To reduce this latency, I am adding a
global view of the file system that will be stored in
BasicKeys.globalFileTreeView.
To make this work, I introduce the StampedFile trait, which augments the
java.io.File class with a stamp method that returns the zinc stamp for
the file. For source files, this will be a hash of the file, while for
binaries, it is just the last modified time. In order to gain access to
the sbt.internal.inc.Stamper class, I had to append addSbtZinc to the
commandProj configurations.
This view may or may not use an in-memory cache of the file system tree
to return the results. Because there is always the risk of the cache
getting out of sync with the actual file system, I both make it optional
to use a cache and provide a mechanism for flushing the cache. Moreover,
the in-memory cache implementation in sbt.io, which is backed by a
swoval FileTreeRepository, has the property that touching a monitored
directory invalidates the entire directory within the cache, so the
flush command isn't even strictly needed in general.
Because caching is optional, the global is of a FileTreeDataView, which
doesn't specify a caching strategy. Subsequent commits will make use of
this to potentially speed up incremental compilation by caching the
Stamps of the source files so that zinc does not need to compute the
hashes itself and will allow for continuous builds to use the cache to
monitor events instead of creating a new, standalone FileEventMonitor.
I had to turn off -Xfatal-warnings in commandProj because after updating
io, commandProj depends on the deprecated EventMonitor class. In #4335,
I stop using EventMonitor, but deprecate the Watched class which is both
defined and used (as an unused attribute key) in commandProj. I think we
can probably get rid of Watched in 1.4.x and certainly in a hypothetical
2.x, so hopefully we can restore -Xfatal-warnings sooner than later.
I also had to replace uses of IO.classLocationFile with
IO.classLocationPath to avoid compilation failures due to
-Xfatal-warnings.
Fixes#4241Fixes#4242
This introduces a new subproject named scripted-sbt-redux. The purpose of this new subproject is to workaround the 'sbt.test` package vs `Keys.test` key confusion (#4242) while maintaining the forward compatibility of 0.13.17's sbt cross testing ^^ (#4241).
The new subproject uses `sbt.scriptedtest` package name, and that's the one that will be used by the mothership.
Meanwhile "scripted-sbt" subproject will also be published for compatibility purpose.
This hot path was discovered by retronym using FlameGraph.
This removes intermediate creation of Array.
`filesModifiedBytes` in particular was bad as it was going through all `*.class` files, each generating an Array. This replaces that with `fileModifiedHash` that accepts `MessageDigest`.
According to the flamegraph, evalCommon footprint reduced from 4.5% to 3.6%.
Using `time sbt reload reload reload exit`, the median reduced from 41.450s to 39.467s.
previously I was using separate thread in a forked test to test the server, but that is not enough isolation to run multiple server tests.
This adds `RunFromSourceMain.fork(workingDirectory: File)`, which allows us to run a fresh sbt on the given working directory. Next, I've refactored the stateful client-side buffer to a class `TestServer`.
* 1.1.x:
Use Java's redirectInput rather than sys.process's connectInput
Re-write toolboxClasspath to use sbt-buildinfo
Cleanup generateToolboxClasspath
Upgrade to sbt-buildinfo 0.8.0
Fix how fullClasspath is defined in TestBuildInfo
delete buildinfo.BuildInfo from sbt main
Conflicts:
project/plugins.sbt
* 1.1.x:
Update mimaPreviousArtifacts/sbt.version
Introduce SBT_GLOBAL_SERVER_DIR env var to override too long paths
Handle very long socket file paths on UNIX
Conflicts:
project/build.properties
This works around the name conflict between sbt.test package and sbt.Keys.test.
1. sbt.test package is renamed to sbt.scriptedtest. This allows 1.0 plugins and builds to use `test` to mean `Keys.test`.
2. To keep binary compatibility for sbt 0.13 scripted, I am adding `sbt.test.ScriptedRunner` and `sbt.test.ScriptedTests` in `scripted-plugin` artifact.
3. Another affected user is Giter8 plugin that uses ScriptedPlugin. Since the intereactions are limited to `sbt.ScriptedPlugin.*`, we should be fine here. - https://github.com/foundweekends/giter8/blob/v0.11.0-M2/plugin/src/main/scala-sbt-1.0/giter8/SBTCompat.scala
Fixes#3538
This brings in `sbt.ScriptedPlugin` as `sbt.plugins.ScriptedPlugin` into sbt mothership.
In addition, `sbt.plugins.SbtPlugin` is added that enables the scripted plugin and `sbtPlugin := true`.
This allows plugin authors to bring in scripted plugin by writing:
```scala
lazy val root = (project in file("."))
.enablePlugins(SbtPlugin)
```
This is what it looks like now! 🎉
> sbtOn /s/t-sbtOn
[...]
[info] Running (fork) sbt.RunFromSourceMain /s/t-sbtOn
Listening for transport dt_socket at address: 5005
[warn] sbt version mismatch, current: 1.0.3, in build.properties: "1.1.0", use 'reboot' to use the new value.
[info] Loading settings from idea.sbt,global-plugins.sbt ...
[info] Loading global plugins from /Users/dnw/.dotfiles/.sbt/1.0/plugins
[info] Updating ProjectRef(uri("file:/Users/dnw/.sbt/1.0/plugins/"), "global-plugins")...
[info] Done updating.
[info] Loading settings from plugins.sbt ...
[info] Loading project definition from /s/t-sbtOn/project
[info] Updating ProjectRef(uri("file:/s/t-sbtOn/project/"), "t-sbton-build")...
[info] Done updating.
[info] Loading settings from build.sbt ...
[info] Set current project to t (in build file:/s/t-sbtOn/)
[info] sbt server started at local:///Users/dnw/.sbt/1.0/server/2c27eaf4c750902a3a41/sock
> show baseDirectory
[info] /s/t-sbtOn
> exit
[info] shutting down server
[success] Total time: 34 s, completed 16-Jan-2018 14:37:32
> Exception in thread "Thread-17" java.io.IOException: Stream closed
at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433)
at java.io.OutputStream.write(OutputStream.java:116)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at scala.sys.process.BasicIO$.$anonfun$input$1(BasicIO.scala:200)
at scala.sys.process.BasicIO$.$anonfun$input$1$adapted(BasicIO.scala:198)
at scala.sys.process.ProcessBuilderImpl$Simple.$anonfun$run$2(ProcessBuilderImpl.scala:75)
at scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:23)
> show {.}/baseDirectory
[...]
[info] ThisBuild / baseDirectory
[info] /d/sbt
Previously I was seeing the error upon the first scripted test. I thought it was because Main was somehow not early enough. It might just be because scripted technically runs as part of the build.
Ref sbt/io#110
This is an implementation of `textDocument/definition` request.
Supports types only, and only in case when type is found in Zinc Analysis. When source(s) are found then editor opens potential source(s).
This simple implementation does not use semantic data.
During the processing of `textDocument/didSave`, we will start collecting the location of Analysis files via `lspCollectAnalyses`.
Later on, when the user asked for `textDocument/definition`, sbt server will invoke a Future call to lspDefinition, which direct reads the files to locate the definition of a class.
In addition to TCP, this adds sbt server support for IPC (interprocess communication) using Unix domain socket and Windows named pipe.
The use of Unix domain socket has performance and security benefits.
This adds a new option `dev` to the `reboot` command, which deletes the only the current sbt artifacts from the boot directory. `reboot dev` reads actively from `build.properties` instead of using the current state since `reboot` can restart into another sbt version.
In general, `reboot dev` is intended for the local development of sbt.
Fixes#3590
In ca71b4b902 I went about fixing the
inexhaustive matching in Scope's resolveProjectBuild and
resolveProjectRef. Looking back the change was wrong.
For resolveProjectBuild the new implementation is less wrong, but still
not great, seeing as it doesn't actually do any build resolving.
For resolveProjectRef the new implementation now blows up instead of
lies. Which means it's less leneant, more "fail-fast".
isProjectThis is unused; remnant of the pre-AutoPlugin days when build
settings where defined in Plugin.settings.
This is the first cut for the Language Server Protocol on top of server that is still work in progress.
With this change, sbt is able to invoke `compile` task on saving files in VS Code.
Fixessbt/sbt#1812
This adds unified slash syntax for both sbt shell and the build.sbt DSL.
Instead of the current `<project-id>/config:intask::key`,
this adds `<project-id>/<config-ident>/intask/key` where <config-ident> is the Scala identifier notation for the configurations like `Compile` and `Test`.
This also adds a series of implicits called `SlashSyntax` that adds `/` operators to project refererences, configuration, and keys such that the same syntax works in build.sbt.
These examples work for both from the shell and in build.sbt.
Global / cancelable
ThisBuild / scalaVersion
Test / test
root / Compile / compile / scalacOptions
ProjectRef(uri("file:/xxx/helloworld/"),"root")/Compile/scalacOptions
Zero / Zero / name
The inspect command now outputs something that can be copy-pasted:
> inspect compile
[info] Task: sbt.inc.Analysis
[info] Description:
[info] Compiles sources.
[info] Provided by:
[info] ProjectRef(uri("file:/xxx/helloworld/"),"root")/Compile/compile
[info] Defined at:
[info] (sbt.Defaults) Defaults.scala:326
[info] Dependencies:
[info] Compile/manipulateBytecode
[info] Compile/incCompileSetup
[info] Reverse dependencies:
[info] Compile/printWarnings
[info] Compile/products
[info] Compile/discoveredSbtPlugins
[info] Compile/discoveredMainClasses
[info] Delegates:
[info] Compile/compile
[info] compile
[info] ThisBuild/Compile/compile
[info] ThisBuild/compile
[info] Zero/Compile/compile
[info] Global/compile
[info] Related:
[info] Test/compile
This implements JSON-based port file. Thoughout the lifetime of the sbt server there will be `cwd / "project" / "target" / "active.json"`, which contains `url` field.
Using this `url` the potential client, such as IDEs can find out which port number to hit.
Ref #3508
Ref #3282
We used to wrap InputStream so it will inject Thread.sleep, which then allows the thread to be cancelled, emulating a non-blocking readLine. This trick doesn't seem to work for Windows.
For non-Cygwin, actually just removing the wrapping does the job, but I couldn't get it to work for Cygwin.
To test, run some command via network, and then type `show name` into the terminal. On Cygwin, it will not respond.
The previous scalafmt plugin had two problems:
* Caching with Coursier did not work correctly
* It failed after the upgrade to 1.0 in my computer (from a clean fork):
```
[error] (testingProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (runProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (taskProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (stdTaskProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (actionsProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (protocolProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (commandProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (mainSettingsProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (mainProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (sbtProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (scriptedPluginProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (scriptedSbtProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] Total time: 19 s, completed May 24, 2017 10:44:56 AM
```
This commit replaces the previous scalafmt integration by the one
created by Lucidsoftware, big shoutout! (/cc @pauldraper)
https://github.com/lucidsoftware/neo-sbt-scalafmt
This commit moves everything behind the old util-appmacro over to
sbt/sbt.
appmacro has been renamed as core-macros. It is indeed the project
that defines the main logic to manipulate macros, and defines interfaces
that are then implemented by main-settings. Because of this strong
relationship between these two modules, I find that they have to stay
together and be able to be modified in one commit. I've realised this
while hacking on the macro changes for the usability of sbt.
The reason for this change is because the logic of this module is really
tight to the implementation of sbt and, I would even say, essentially
define the sbt DSL.
Some plugins remain commented out, for now.
sbt-doge is no longer needed because a variant of it has been folded
into sbt 1.
For some reason scripted requires src/doc jars of sbt, so switch back to
using `publishAll` rather than `publishLocalBinAll`. :(
Also, the sys.prop change in scripted is to force log4j2 to not use a
thread context classloader, and avoid the following:
ERROR StatusLogger Unable to create custom ContextSelector. Falling back to default.
java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.async.AsyncLoggerContextSelector to org.apache.logging.log4j.core.selector.ContextSelector
at java.lang.Class.cast(Class.java:3369)
at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:201)
at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOfProperty(LoaderUtil.java:226)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.createContextSelector(Log4jContextFactory.java:97)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.<init>(Log4jContextFactory.java:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:94)
at org.apache.logging.log4j.spi.ThreadContextMapFactory.createThreadContextMap(ThreadContextMapFactory.java:73)
at org.apache.logging.log4j.ThreadContext.init(ThreadContext.java:223)
at org.apache.logging.log4j.ThreadContext.<clinit>(ThreadContext.java:202)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createDefaultInjector(ContextDataInjectorFactory.java:83)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createInjector(ContextDataInjectorFactory.java:67)
at org.apache.logging.log4j.core.lookup.ContextMapLookup.<init>(ContextMapLookup.java:34)
at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:117)
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:125)
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46)
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:650)
at org.apache.logging.log4j.core.layout.PatternLayout.createDefaultLayout(PatternLayout.java:487)
at sbt.internal.util.ConsoleAppender.<init>(ConsoleAppender.scala:245)
at sbt.internal.util.ConsoleAppender$.apply(ConsoleAppender.scala:196)
at sbt.internal.util.ConsoleLogger.<init>(ConsoleAppender.scala:42)
at sbt.internal.util.ConsoleLogger$.apply(ConsoleAppender.scala:34)
at sbt.test.ScriptedRunner.run(ScriptedTests.scala:221)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at Scripted$.doScripted(Scripted.scala:125)
at $0483e89d182e9d240274$.$anonfun$scriptedTask$5(build.sbt:301)
I've tried to put together some scripted tests to remove the overhead of
compiling the whole sbt, which is around 3 minutes every time.
This new structure *should* make the scripted tests run faster.
Aside from this, we do some more tweaks:
* Increase memory.
* Fork processes to compile and run (to see if it makes a difference).
* Pass in the server flag to sbt.
* Remove MaxPermSize from another scripted opts
* Reduce memory of sbt host to 1g instead of 2g
* Add Xms java options to scripted
* Enable parallelism with 512M for sbt tests
* Parallelism + batch mode
Andrea Peruffo
Ethan Atkins
Eugene Yokota
Kazuhiro Sera
Dale Wijnand
OlegYch
Dale Wijnand
xuwei-k
wpopielarski
Antonio Cunei
Jonas Fonseca
Martin Duhem
jvican