Stop CI hangs in background job service shutdown

A periodic stacktrace showed that scripted tests were still hanging in ci
trying to shutdown the background job service (I had previously thought
that I'd fixed that in 16bef0cfc8). It
appears that there is a logical bug that prevents some jobs from being
removed from the jobSet even though they have finished. If that happens,
the shutdown will never exit. That is highly undesirable and can be
avoided by adding a timeout and also only trying to shutdown the job if
it is actually running.
This commit is contained in:
Ethan Atkins 2020-11-19 22:20:12 -08:00
parent 68933a628d
commit 14dee32d6b
1 changed files with 7 additions and 3 deletions

View File

@ -179,11 +179,15 @@ private[sbt] abstract class AbstractBackgroundJobService extends BackgroundJobSe
override final def close(): Unit = shutdown()
override def shutdown(): Unit = {
while (jobSet.nonEmpty) {
val deadline = 10.seconds.fromNow
while (jobSet.nonEmpty && !deadline.isOverdue) {
jobSet.headOption.foreach {
case handle: ThreadJobHandle @unchecked =>
handle.job.shutdown()
handle.job.awaitTerminationTry(10.seconds)
if (handle.job.isRunning) {
handle.job.shutdown()
handle.job.awaitTerminationTry(10.seconds)
}
jobSet = jobSet - handle
case _ => //
}
}