mirror of https://github.com/sbt/sbt.git
102e3d1969
It turns out that task progress actually introduces a fair bit of overhead. The biggest issue is that the task progress callbacks block the Execute main thread. This means that time in those callbacks delays task evaluation, slowing down sbt. This was not negligible, I was seeing a lot of the total time of a no-op compile in https://github.com/jtjeferreira/sbt-multi-module-sample was spent in TaskProgress callbacks. Prior to these changes, I ran 30 no-op compiles in that project and the average time was about 570ms. This number got worse and worse because there were memory leaks in the TaskProgress object. After these changes, it dropped to 250ms and after jit-ing, it would drop to about 200ms. I also successfully ran 5000 consecutive no-op compiles without leaking any memory. A lot of the overhead of task progress was in adding tasks to the timings map in AbstractTaskProgress. Tasks were never removed and ConcurrentHashMap insertion time is proportional to the size of the map (not sure if it's linear, quadratic or other) which was why sbt actually got slower and slower the longer it ran. Much of the time was spent adding tasks to the progress timings. To fix this, I did something similar to what I did to manage logger state in https://github.com/jtjeferreira/sbt-multi-module-sample. In MainLoop, we create a new TaskProgress instance before command evaluation and clean it up after. Earlier I made TaskProgress an object to try to ensure there was only one progress thread at a time, and that introduced the memory leak. In addition to removing the leak, I was able to improve performance by removing tasks from the timings map when they completed. Unlike TaskTimings and TaskTraceEvent, we don't care about tasks that have completed for TaskProgress so it is safe to remove them. In addition to the memory leaks, I also reworked how the background threads work. Instead of having one thread that sleeps and prints progress reports, we now use two single threaded executors. One is a scheduled executor that is used to schedule progress reports and the other is the actual thread on which the report is generated. When progress starts, we schedule a recurring report that is generated every sleep interval until task evaluation completes. Whenever we add a new task, if we have haven't previously generated a progress report, we schedule a report in threshold milliseconds. If the task completes before the threshold period has elapsed, we just cancel the schedule report. By doing things this way, we reduce the total number of reports that are generated. Because reports need to effectively lock System.out, the less we generate them, the better. I also modified the internal data structures of AbstractTaskProgress so that there is a single task map of timings instead of one map for timings and one for active tasks. |
||
|---|---|---|
| .. | ||
| util-collection | ||
| util-complete | ||
| util-control | ||
| util-interface/src/main/java/xsbti | ||
| util-logging | ||
| util-logic/src | ||
| util-position/src | ||
| util-relation/src | ||
| util-scripted/src/main | ||