In 64c0f0acdd, I attempted to safely close
all of the completion services when the user inputs ctrl+c. I have
noticed though that sometimes sbt crashes in CI with the
RejectedExecutionException thrown by submit. To avoid throwing when
there was no cancellation, I slightly modified the shutdown logic to not
shutdown the completion service whil still shutting down the underlying
thread pool.
Fixes https://github.com/sbt/sbt/issues/5822
Currently the entire shell gets stuck when there's a compilation error with pipelining.
This at least returns to sbt shell.
Frequently ctrl+c does not work to cancel the running tasks. This seems
to be because the signal handler is bound to a specific instance of
evaluate task but there may be multiple instances of evaluate task
running at any given time. Shutting down just one of the running engines
does not ensure that task evaluation stops. To work around this, we can
globally store all of the completion services in a weak hash map and
cancel all of them whenever a signal is received. Closing the service,
which happens at the end of task evaluation will remove the service from
the map so hopefully this shouldn't introduce a memory leak.
This adds `Def.promise` a facility that wraps `scala.concurrent.Promise`. Project layer, there's an implicit for task-that-returns-promise (`Def.Initialize[Task[PromiseWrap[A]]]`) that would inject `await` method, which returns a task. This is a special task that is tagged with `Tags.Sentinel` so that it will bypass the concurrent restrictions. Since there's no CPU- or IO-bound work, this should be ok.
The purpose of this promise for long-running task to communicate with another task midway.
To demonstrate [-Yno-lub](http://eed3si9n.com/stricter-scala-with-ynolub), this shows the code changes that removes lubing (Not all subprojects are done).
After I made the changes, I switched the Scala back to normal 2.12.10.
In some circumstances, sbt would generate a number of task progress
threads that could run concurrently. The issue was that the TaskProgress
could be shared by multiple EvaluateTaskConfigs if a dynamic task was
used. This was problematic because when a dynamic task completed, it
might call afterAllCompleted which would stop the progress thread. There
also was a race condition because multiple threads calling initial could
theoretically have created a new progress thread which would cause a
resource leak.
To fix this, we modify the shared task progress so that the `stop()`
method is a no-op. This should prevent dynamic tasks from stopping the
progress thread. We also defer the creation of the task thread until
there is at least one active task. This prevents a thread from being
created in the shell.
The motivation for this change was that I found that sometimes there was
a leaked progress thread that would make the shell not really work for
me because the progress thread would overwrite the shell prompt. This
change fixes that behavior and I was able to validate with jstack that
there was consistently either one or zero task progress threads at a
time (zero in the shell, one when tasks were actually running).
In an interactive session, it's possible for task evaluation to trigger
an OOM: Metaspace but for sbt to continue working after that failure.
Moreover, the metaspace oom can be caused by using a dependency
classloader layer. If the user changes the layering strategy, they may
be able to re-run their command successfully.
Fixes#4461
This opens up ExecuteProgress API that's been around under private[sbt].
Since the state passing mechanism hasn't been used, I got rid of it.
The build user can configure the build using two keys Boolean `taskProgress` and `State => Seq[TaskProgress]` `progressReports`. `useSuperShell` is lightweight key on/off switch for the super shell that can be used as follows:
```scala
Global / SettingKey[Boolean]("useSuperShell") := false
```
Before:
```
[warn] /Users/jz/code/zinc/internal/zinc-apiinfo/src/main/scala/xsbt/api/Visit.scala:187:19: parameter value s in method visitString is never used
[warn] def visitString(s: String): Unit = ()
[warn] ^
^C
[warn] Canceling execution...
[warn] 10 warnings found
[info] Compilation has been cancelled
[error] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@33ec70dd rejected from java.util.concurrent.ThreadPoolExecutor@4b832de6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1410]
[error] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
[error] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
[error] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
[error] at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
[error] at sbt.CompletionService$.submit(CompletionService.scala:36)
[error] at sbt.ConcurrentRestrictions$$anon$4.submitValid(ConcurrentRestrictions.scala:176)
[error] at sbt.ConcurrentRestrictions$$anon$4.submit(ConcurrentRestrictions.scala:165)
[error] at sbt.Execute.submit(Execute.scala:262)
[error] at sbt.Execute.ready(Execute.scala:242)
[error] at sbt.Execute.notifyDone(Execute.scala:181)
[error] at sbt.Execute.$anonfun$retire$2(Execute.scala:152)
[error] at sbt.Execute.$anonfun$retire$2$adapted(Execute.scala:151)
[error] at scala.collection.immutable.List.foreach(List.scala:389)
[error] at sbt.Execute.retire(Execute.scala:151)
[error] at sbt.Execute.$anonfun$work$2(Execute.scala:279)
[error] at sbt.Execute$$anon$1.process(Execute.scala:24)
[error] at sbt.Execute.next$1(Execute.scala:104)
[error] at sbt.Execute.processAll(Execute.scala:107)
[error] at sbt.Execute.runKeep(Execute.scala:84)
[error] at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:387)
[error] at sbt.EvaluateTask$.run$1(EvaluateTask.scala:386)
[error] at sbt.EvaluateTask$.runTask(EvaluateTask.scala:405)
[error] at sbt.internal.Aggregation$.$anonfun$timedRun$4(Aggregation.scala:100)
[error] at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:331)
[error] at sbt.internal.Aggregation$.timedRun(Aggregation.scala:98)
[error] at sbt.internal.Aggregation$.runTasks(Aggregation.scala:111)
[error] at sbt.internal.Aggregation$.$anonfun$applyTasks$1(Aggregation.scala:68)
[error] at sbt.Command$.$anonfun$applyEffect$2(Command.scala:130)
[error] at sbt.internal.Aggregation$.$anonfun$evaluatingParser$11(Aggregation.scala:220)
[error] at sbt.internal.Act$.$anonfun$actParser0$3(Act.scala:387)
[error] at sbt.MainLoop$.processCommand(MainLoop.scala:154)
[error] at sbt.MainLoop$.$anonfun$next$2(MainLoop.scala:137)
[error] at sbt.State$$anon$1.runCmd$1(State.scala:242)
[error] at sbt.State$$anon$1.process(State.scala:248)
[error] at sbt.MainLoop$.$anonfun$next$1(MainLoop.scala:137)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.MainLoop$.next(MainLoop.scala:137)
[error] at sbt.MainLoop$.run(MainLoop.scala:130)
[error] at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:108)
[error] at sbt.io.Using.apply(Using.scala:22)
[error] at sbt.MainLoop$.runWithNewLog(MainLoop.scala:102)
[error] at sbt.MainLoop$.runAndClearLast(MainLoop.scala:58)
[error] at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:43)
[error] at sbt.MainLoop$.runLogged(MainLoop.scala:35)
[error] at sbt.StandardMain$.runManaged(Main.scala:113)
[error] at sbt.xMain.run(Main.scala:76)
[error] at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
[error] at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
[error] at xsbt.boot.Launch$.run(Launch.scala:109)
[error] at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
[error] at xsbt.boot.Launch$.launch(Launch.scala:117)
[error] at xsbt.boot.Launch$.apply(Launch.scala:18)
[error] at xsbt.boot.Boot$.runImpl(Boot.scala:41)
[error] at xsbt.boot.Boot$.main(Boot.scala:17)
[error] at xsbt.boot.Boot.main(Boot.scala)
[error] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@33ec70dd rejected from java.util.concurrent.ThreadPoolExecutor@4b832de6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1410]
[error] Use 'last' for the full log.
```
After:
```
sbt:zinc Root> ;zincIvyIntegration/cleanClasses;zincIvyIntegration/compile
[success] Total time: 0 s, completed 03/04/2018 2:24:33 PM
[info] Compiling 5 Scala sources and 1 Java source to /Users/jz/code/zinc/internal/zinc-ivy-integration/target/scala-2.12/classes ...
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ResourceLoader.scala:14:42: parameter value classLoader in method getPropertiesFor is never used
[warn] def getPropertiesFor(resource: String, classLoader: ClassLoader): Properties = {
[warn] ^
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ZincComponentCompiler.scala:13:17: Unused import
[warn] import java.net.URLClassLoader
[warn] ^
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ZincComponentCompiler.scala:137:26: Unused import
[warn] import sbt.io.Path.toURLs
[warn] ^
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ZincComponentCompiler.scala:94:30: The outer reference in this type test cannot be checked at run time.
[warn] private final case class ScalaArtifacts(compiler: File, library: File, others: Vector[File])
[warn] ^
^C
[warn] Canceling execution...
[warn] four warnings found
[info] Compilation has been cancelled
[error] Total time: 1 s, completed 03/04/2018 2:24:35 PM
```
Best served with https://github.com/scala/scala/pull/6479Fixes#3958
Forward-port of #2303.
The checking code has bad run time characteristics and would need to be fixed
for large projects with deep task dependency chains.
The code in sbt.Execute has been in production for a long time so it seems safe
enough to drop the extra checks by default. To debug issues, you can set
`-Dsbt.execute.extrachecks=true` to revert to the old behavior.
xuwei-k
Julien Richard-Foy
Eugene Yokota
Ethan Atkins
Alex Zolotko
Ethan Atkins
Dale Wijnand
Jason Zaugg
Johannes Rudolph
Martin Duhem
Pierre DAL-PRA
Jean-Rémi Desjardins
Josh Suereth