To demonstrate [-Yno-lub](http://eed3si9n.com/stricter-scala-with-ynolub), this shows the code changes that removes lubing (Not all subprojects are done).
After I made the changes, I switched the Scala back to normal 2.12.10.
In some circumstances, sbt would generate a number of task progress
threads that could run concurrently. The issue was that the TaskProgress
could be shared by multiple EvaluateTaskConfigs if a dynamic task was
used. This was problematic because when a dynamic task completed, it
might call afterAllCompleted which would stop the progress thread. There
also was a race condition because multiple threads calling initial could
theoretically have created a new progress thread which would cause a
resource leak.
To fix this, we modify the shared task progress so that the `stop()`
method is a no-op. This should prevent dynamic tasks from stopping the
progress thread. We also defer the creation of the task thread until
there is at least one active task. This prevents a thread from being
created in the shell.
The motivation for this change was that I found that sometimes there was
a leaked progress thread that would make the shell not really work for
me because the progress thread would overwrite the shell prompt. This
change fixes that behavior and I was able to validate with jstack that
there was consistently either one or zero task progress threads at a
time (zero in the shell, one when tasks were actually running).
In an interactive session, it's possible for task evaluation to trigger
an OOM: Metaspace but for sbt to continue working after that failure.
Moreover, the metaspace oom can be caused by using a dependency
classloader layer. If the user changes the layering strategy, they may
be able to re-run their command successfully.
Fixes#4461
This opens up ExecuteProgress API that's been around under private[sbt].
Since the state passing mechanism hasn't been used, I got rid of it.
The build user can configure the build using two keys Boolean `taskProgress` and `State => Seq[TaskProgress]` `progressReports`. `useSuperShell` is lightweight key on/off switch for the super shell that can be used as follows:
```scala
Global / SettingKey[Boolean]("useSuperShell") := false
```
Before:
```
[warn] /Users/jz/code/zinc/internal/zinc-apiinfo/src/main/scala/xsbt/api/Visit.scala:187:19: parameter value s in method visitString is never used
[warn] def visitString(s: String): Unit = ()
[warn] ^
^C
[warn] Canceling execution...
[warn] 10 warnings found
[info] Compilation has been cancelled
[error] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@33ec70dd rejected from java.util.concurrent.ThreadPoolExecutor@4b832de6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1410]
[error] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
[error] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
[error] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
[error] at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
[error] at sbt.CompletionService$.submit(CompletionService.scala:36)
[error] at sbt.ConcurrentRestrictions$$anon$4.submitValid(ConcurrentRestrictions.scala:176)
[error] at sbt.ConcurrentRestrictions$$anon$4.submit(ConcurrentRestrictions.scala:165)
[error] at sbt.Execute.submit(Execute.scala:262)
[error] at sbt.Execute.ready(Execute.scala:242)
[error] at sbt.Execute.notifyDone(Execute.scala:181)
[error] at sbt.Execute.$anonfun$retire$2(Execute.scala:152)
[error] at sbt.Execute.$anonfun$retire$2$adapted(Execute.scala:151)
[error] at scala.collection.immutable.List.foreach(List.scala:389)
[error] at sbt.Execute.retire(Execute.scala:151)
[error] at sbt.Execute.$anonfun$work$2(Execute.scala:279)
[error] at sbt.Execute$$anon$1.process(Execute.scala:24)
[error] at sbt.Execute.next$1(Execute.scala:104)
[error] at sbt.Execute.processAll(Execute.scala:107)
[error] at sbt.Execute.runKeep(Execute.scala:84)
[error] at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:387)
[error] at sbt.EvaluateTask$.run$1(EvaluateTask.scala:386)
[error] at sbt.EvaluateTask$.runTask(EvaluateTask.scala:405)
[error] at sbt.internal.Aggregation$.$anonfun$timedRun$4(Aggregation.scala:100)
[error] at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:331)
[error] at sbt.internal.Aggregation$.timedRun(Aggregation.scala:98)
[error] at sbt.internal.Aggregation$.runTasks(Aggregation.scala:111)
[error] at sbt.internal.Aggregation$.$anonfun$applyTasks$1(Aggregation.scala:68)
[error] at sbt.Command$.$anonfun$applyEffect$2(Command.scala:130)
[error] at sbt.internal.Aggregation$.$anonfun$evaluatingParser$11(Aggregation.scala:220)
[error] at sbt.internal.Act$.$anonfun$actParser0$3(Act.scala:387)
[error] at sbt.MainLoop$.processCommand(MainLoop.scala:154)
[error] at sbt.MainLoop$.$anonfun$next$2(MainLoop.scala:137)
[error] at sbt.State$$anon$1.runCmd$1(State.scala:242)
[error] at sbt.State$$anon$1.process(State.scala:248)
[error] at sbt.MainLoop$.$anonfun$next$1(MainLoop.scala:137)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.MainLoop$.next(MainLoop.scala:137)
[error] at sbt.MainLoop$.run(MainLoop.scala:130)
[error] at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:108)
[error] at sbt.io.Using.apply(Using.scala:22)
[error] at sbt.MainLoop$.runWithNewLog(MainLoop.scala:102)
[error] at sbt.MainLoop$.runAndClearLast(MainLoop.scala:58)
[error] at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:43)
[error] at sbt.MainLoop$.runLogged(MainLoop.scala:35)
[error] at sbt.StandardMain$.runManaged(Main.scala:113)
[error] at sbt.xMain.run(Main.scala:76)
[error] at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
[error] at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
[error] at xsbt.boot.Launch$.run(Launch.scala:109)
[error] at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
[error] at xsbt.boot.Launch$.launch(Launch.scala:117)
[error] at xsbt.boot.Launch$.apply(Launch.scala:18)
[error] at xsbt.boot.Boot$.runImpl(Boot.scala:41)
[error] at xsbt.boot.Boot$.main(Boot.scala:17)
[error] at xsbt.boot.Boot.main(Boot.scala)
[error] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@33ec70dd rejected from java.util.concurrent.ThreadPoolExecutor@4b832de6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1410]
[error] Use 'last' for the full log.
```
After:
```
sbt:zinc Root> ;zincIvyIntegration/cleanClasses;zincIvyIntegration/compile
[success] Total time: 0 s, completed 03/04/2018 2:24:33 PM
[info] Compiling 5 Scala sources and 1 Java source to /Users/jz/code/zinc/internal/zinc-ivy-integration/target/scala-2.12/classes ...
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ResourceLoader.scala:14:42: parameter value classLoader in method getPropertiesFor is never used
[warn] def getPropertiesFor(resource: String, classLoader: ClassLoader): Properties = {
[warn] ^
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ZincComponentCompiler.scala:13:17: Unused import
[warn] import java.net.URLClassLoader
[warn] ^
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ZincComponentCompiler.scala:137:26: Unused import
[warn] import sbt.io.Path.toURLs
[warn] ^
[warn] /Users/jz/code/zinc/internal/zinc-ivy-integration/src/main/scala/sbt/internal/inc/ZincComponentCompiler.scala:94:30: The outer reference in this type test cannot be checked at run time.
[warn] private final case class ScalaArtifacts(compiler: File, library: File, others: Vector[File])
[warn] ^
^C
[warn] Canceling execution...
[warn] four warnings found
[info] Compilation has been cancelled
[error] Total time: 1 s, completed 03/04/2018 2:24:35 PM
```
Best served with https://github.com/scala/scala/pull/6479Fixes#3958
Forward-port of #2303.
The checking code has bad run time characteristics and would need to be fixed
for large projects with deep task dependency chains.
The code in sbt.Execute has been in production for a long time so it seems safe
enough to drop the extra checks by default. To debug issues, you can set
`-Dsbt.execute.extrachecks=true` to revert to the old behavior.
Set sbt.task.timings=true to print timings for tasks.
This sample progress handler shows how to get names for tasks and
deal with flatMapped tasks. There are still some tasks that make
it through as anonymous, which needs to be investigated.
A setting to provide a custom handler should come in a subsequent commit.
Alex Zolotko
Eugene Yokota
Ethan Atkins
xuwei-k
Ethan Atkins
Dale Wijnand
Jason Zaugg
Johannes Rudolph
Martin Duhem
Pierre DAL-PRA
Jean-Rémi Desjardins
Josh Suereth
Mark Harrah