Ideally, we wouldn't need to construct the classpath ourselves and instead
reuse the classpath construction code from a compiler (scalac or javac).
However, we need to ensure that we only call the compiler when needed.
Because we need to construct the classpath even when compilation might
not happen, we have to duplicate classpath construction.