Fixes https://github.com/sbt/sbt/issues/7117
Problem
-------
sbt no longer works on a Mac with M1 (aarch64) chips because it fails on
ipcsocket library, which we maintain.
This was caused by architecture detection that was added in 1.6.1,
which did not take in account for macOS universal binary.
Solution
--------
Update to ipcsocket 1.6.2.
Launcher 1.4.1 is capable of launching Scala 3 apps.
In other words, launchers prior to 1.4.1 are not capable of
launching Scala 3 apps, including tests for sbt 2.x.
This is a prepration step for Scala 3 support.
Instead of data-class, this uses
https://github.com/hamnis/dataclass-scalafix.
Since it cannot use `withX` method in the data class itself with this
approach, I created `syntax` package object to extend some methods to
the generated classes.
This removes OkHttp dependency.
This also removes an experimental feature to customize HTTP for Ivy called
CustomHttp we added in sbt 1.3.0.
Since LM got switched to Coursier in 1.3.0, I don't think we advertized
CustomHttp.
Ref https://github.com/sbt/sbt/issues/6912
Problem
-------
There's apparently a security issue with OkHttp 3.x,
which I am not really sure how applicable it is to our usage
of OkHttp but it is there.
Solution
--------
Since most of OkHttp-specic usage within LM is for Apache Ivy
downloading, I am going to drop this.
Since `sbt.librarymanagement.Http.http` is a public API,
I am substituting this with Apache HTTP backed implementation.
Migrates TreeView.scala to use Contraband from scala.util.parsing.json,
because this is now deprecated.
The TreeView logic is used in the dependencyBrowseTree task.
Fixes https://github.com/sbt/sbt/issues/6372
sbt 1.4.x includes both JLine 2 and JLine 3, and our fork of JLine 2 uses JAnsi matching with JLine 3.
I need to bump JLine 2 together, which uses the same JAnsi version as
JLine 3.
Fixes https://github.com/sbt/sbt/issues/6348
Intentionally or unintentionally the JLine 3 leaks over from sbt to
Scala REPL currently.
I think this enables things like sbtn to transmit tab completion and
other console interaction over a socket to sbt server.
Scala 2.13.5 REPL now causes some issue of displaying warning
each time the user hits tab for tab completion.
This works around the issue by upgrading to the latest JLine 3.
Fixes https://github.com/sbt/sbt/issues/6210
scodec-bits is published with pvp versionScheme (nice), this means that
we should just evaluate the version portion for pvp-ness, but I was
using `guessSecondSegment` that checks for Scala suffix. That's mistake
1.
`guessSecondSegment` assumes that the Scala suffix uses the given
ScalaModuleInfo, but with 2.13-3 sandwich we can't assume this.
In the reported case, Scala module is 3.0.0-M3 but scodec-bits uses
2.13. So that's mistake 2.
This attempts to correct both the mistakes.
1. Instead of `guessSecondSegment`, this adds a simpler `evalPvp`
function.
2. `guessSecondSegment` just looks for `_2.` or `_3` and ignores the
Scala module.
The new version provides an apple silicon arm64 jni implementation of
the ipcsocket api. It also adds a jni implementation for getting the max
socket length which is necessary because jna isn't supported on the
apple arm64 platform yet.
We no longer need to use the forked version of jline because they have
merged in our required changes. The latest version of jline does upgrade
jansi, however, and some of the apis we were relying on for windows were
removed so they had to be manually implemented. I verified that console
input still worked on my windows vm after this change.
There were a number of issues with swithcing between raw and canonical
issues that affected both the server and the thin client. These were
reported in #5863 and #5856. In both cases, there were issues with
reading input or having the input be displayed. Debugging those issues
revealed a number of issues with how we were using the jline 3 system
terminal and the hybrid interaction with the jline 2 terminal. This
commit eliminates all of our internal jline 2 usage. The only remaining
jline 2 usage is that we create and override the global terminal for the
scala console for scala versions < 2.13. By moving away from jline 2, I
was also able to fix#5828, which reported that the home, end and delete
keys were not working.
One of the big issues that this commit addresses is that the
NetworkClient was always performing blocking reads on System.in. This
was problematic because it turns out that you can't switch between raw
and canonical modes when there is a read present. To fix this, the
server now sends a message to the client when it wants to read bytes and
only then does the client create a background thread to read a single
byte.
I also figured out how to set the terminal type properly for the thin
client on windows where we had been manually setting the capabilities to
ansi, which only worked for some keys. This fix required switching to
the WindowsInputStream that I introduced in a prior commit. Before we
were using the jline 2 wrapped input stream which was converting some
system events, like home and end, to the wrong escape sequence mappings.
The remainder of the commit is mostly just converting from jline 2 apis
to jline 3 apis.
I verified that tab completions, the scala console, the ammonite console
and a run task that read from System.in all work with both the server
and the thin client on mac, linux and windows after these changes.
Fixes#5828, #5863, #5856
In order to make the console task work with scala 2.13 and the thin
client, we need to provide a way for the scala repl to use an sbt
provided jline3 terminal instead of the default terminal typically built
by the repl. We also need to put jline 3 higher up in the classloading
hierarchy to ensure that two versions of jline 3 are not loaded (which
makes it impossible to share the sbt terminal with the scala terminal).
One impact of this change is the decoupling of the version of
jline-terminal used by the in process scala console and the version
of jline-terminal specified by the scala version itself. It is possible
to override this by setting the `useScalaReplJLine` flag to true. When
that is set, the scala REPL will run in a fully isolated classloader. That
will ensure that the versions are consistent. It will, however, for sure
break the thin client and may interfere with the embedded shell ui.
As part of this work, I also discovered that jline 3 Terminal.getSize is
very slow. In jline 2, the terminal attributes were automatically cached with a
timeout of, I think, 1 second so it wasn't a big deal to call
Terminal.getAttributes. The getSize method in jline 3 is not cached and
it shells out to run a tty command. This caused a significant
performance regression in sbt because when progress is enabled, we call
Terminal.getSize whenever we log any messages. I added caching of
getSize at the TerminalImpl level to address this. The timeout is 1
second, which seems responsive enough for most use cases. We could also
move the calculation onto a background thread and have it periodically
updated, but that seems like overkill.
sbt depends on scalacache (which hasn't been updated in about a year)
and we really don't need the functionality provided by scalacache. In
fact, the java api is somewhat easier to work with for our use case. The
motivation is that scalacache uses slf4j for logging which meant that it
was implicitly loading log4j. This caused some noisy logs during
shutdown when the previously unused cache was initialized just to be
cleaned up.
This commit also upgrades caffeine and moving forward we can always
upgrade caffeine (and potentially shade it) without any conflict with
the scalacache version.
Ref https://github.com/sbt/zinc/pull/744
This implements `ThisBuild / usePipelining`, which configures subproject pipelining available from Zinc 1.4.0.
The basic idea is to start subproject compilation as soon as pickle JARs (early output) becomes available. This is in part enabled by Scala compiler's new flags `-Ypickle-java` and `-Ypickle-write`.
The other part of magic is the use of `Def.promise`:
```
earlyOutputPing := Def.promise[Boolean],
```
This notifies `compileEarly` task, which to the rest of the tasks would look like a normal task but in fact it is promise-blocked. In other words, without calling full `compile` task together, `compileEarly` will never return, forever waiting for the `earlyOutputPing`.
Ref https://github.com/sbt/sbt/issues/5710
Ref https://github.com/sbt/librarymanagement/pull/339
This adds `versionScheme` setting. When set, it is included into POM, and gets picked up on the other side as an extra attribute of ModuleID. That information in turn is used to inform the eviction warning.
This should reduce the false positives associated with SemVer'ed libraries showing up in the eviction warning.
There were a number of unused key lint warnings when loading the sbt
build. In the case of `fork in compile` and `crossVersion in update`, it
wasn't clear that these were actually used, so I removed those settings.
The others seemed to be used so I just added them to the exclude list.
To make this work with legacy versions of sbt, I redefined the
excludeLintKeys key. Once we update the build.properties to a 1.4.x
version, we can drop the `val excludeLint` definition and replace
`excludeLint` with `excludeLintKeys`.
Side note: ++= does not work with excludeLintKeys which is why I used +=
for the excludes.
This commit upgrades sbt to using jline3. The advantage to jline3 is
that it has a significantly better tab completion engine that is more
similar to what you get from zsh or fish.
The diff is bigger than I'd hoped because there are a number of
behaviors that are different in jline3 vs jline2 in how the library
consumes input streams and implements various features. I also was
unable to remove jline2 because we need it for older versions of the
scala console to work correctly with the thin client. As a result, the
changes are largely additive.
A good amount of this commit was in adding more protocol so that the
remote client can forward its jline3 terminal information to the server.
There were a number of minor changes that I made that either fixed
outstanding ui bugs from #5620 or regressions due to differences between
jline3 and jline2.
The number one thing that caused problems is that the jline3 LineReader
insists on using a NonBlockingInputStream. The implementation ofo
NonBlockingInputStream seems buggy. Moreover, sbt internally uses a
non blocking input stream for system in so jline is adding non blocking
to an already non blocking stream, which is frustrating.
A long term solution might be to consider insourcing LineReader.java
from jline3 and just adapting it to use an sbt terminal rather than
fighting with the jline3 api. This would also have the advantage of not
conflicting with other versions of jline3. Even if we don't, we may want to
shade jline3 if that is possible.
This commit makes it possible for the sbt server to render the same ui
to multiple clients. The network client ui should look nearly identical
to the console ui except for the log messages about the experimental
client.
The way that it works is that it associates a ui thread with each
terminal. Whenever a command starts or completes, callbacks are invoked
on the various channels to update their ui state. For example, if there
are two clients and one of them runs compile, then the prompt is changed
from AskUser to Running for the terminal that initiated the command
while the other client remains in the AskUser state. Whenever the client
changes uses ui states, the existing thread is terminated if it is
running and a new thread is begun.
The UITask formalizes this process. It is based on the AskUser class
from older versions of sbt. In fact, there is an AskUserTask which is
very similar. It uses jline to read input from the terminal (which could
be a network terminal). When it gets a line, it submits it to the
CommandExchange and exits. Once the next command is run (which may or
may not be the command it submitted), the ui state will be reset.
The debug, info, warn and error commands should work with the multi
client ui. When run, they set the log level globally, not just for the
client that set the level.
In order to support a multi-client sbt server ux, we need to factor
`Terminal` out into a class instead of a singleton. Each terminal provides
and outputstream and inputstream. In all of the places where we were
previously relying on the `Terminal` singleton we need to update the
code to use `Terminal.get`, which will redirect io to the terminal whose
command is currently running.
This commit does not implement the server side ui for network clients.
It is just preparatory work for the multi-client ui.
The Terminal implementations have thread safe access to the output
stream. For this reason, I had to remove the sychronization on the
ConsoleOut lockObject. There were code paths that led to deadlock when
synchronizing on the lockObject.
* Add a scripted test for "cp order"
* Forward classpathOrder into SbtUpdateReport
* Bump the target sbt version to 1.2.8
* Preserve order all through SbtUpdateReport.moduleReports
* Update scala-jars scripted test
Seems some changes in the way sbt handles the scala-library / reflect /
compiler JARs made it stale…
Co-authored-by: Alexandre Archambault <alexarchambault@users.noreply.github.com>
This commit makes it so that the scalaVersion, sbtVersion and classpath
are always passed in as parameters to any method that creates an sbt
server -- either for scripted or for the sbt server tests. By making
that change, I was able to change the implementation of scripted in the
sbt project to use publishLocalBin instead of publishLocal. This makes
the scripted tests start much faster (doc alone can easily take 30
second) with messing with the build to exclude slow tasks from
publishLocal.
As part of this change, I removed the test dependency on scriptedSbtRedux for
sbtProj and instead had scriptedSbtRedux depend on sbtProj. This allowed
me to remove some messy LocalProject logic in the resourceGenerators for
scriptedSbtReduxProj. I also had to remove a number of imports in the
scriptedSbtReduxProj because the definitions available in the sbt
package object became available.
I also removed the dependency on sbt-buildinfo and instead pass the
values from the build into test classes using scalatest properties. I
ran into a number of minor issues with the build info plugin, namely
that I couldn't get fullClasspathAsJars to reliably run as a BuildInfo
key. It also is somewhat more clear to me to just rely on the built in
scalatest functionality. The big drawback is that the scalatest
properties can only be strings, but that restriction isn't really a
problem here (strangely the TestData structure has a field configMap
which is effectively Map[String, Any] but Any is actually always String
given how the TestData is created as part of framework initialization.
Since scripted no longer publishes, scriptedUnpublished is now
effectively an alias for scripted.
To get publishLocalBin working, I had to copy private code from
IvyXml.scala into PublishBinPlugin. Once we publish a new version of
sbt, we can remove the copied code and invoke IvyXml.makeIvyXmlBefore
directly.
The main reason for having both the RunFromSourceMain and LauncherBased
scripted tests was that RunFromSourceMain would fail for any test that
ended up accessing the sbt.Package$ object. This commit fixes this bug
by reworking the classloader generated by RunFromSourceMain to invoke
sbt, switching from the classpath to jar classpath (by setting exportJars =
true) and entering sbt by calling `new xMain().run` rather than
`xMain.run`.
The reason for switching to the jar classpath is that the jvm seems to
have issues when there are two classes provided in different directories
that have the same case insensitive name, e.g. `sbt.package$` and
`sbt.Package$`. If those classes are instead provided in different
jars, the jvm seems to be able to handle it.
Exporting the jars is not enough though, I had to rework the
ClassLoader created in the launch method to have a layout that was
recognized by xMainConfiguration. I reimplemented the AppConfiguration
in java so that it could bootstrap itself in a single jar classloader
(the only needed jar is the Scripted.
If we export the jars in the build, then the NoClassDefErrors for
`sbt.Package$` go away during scripted tests using RunSourceFromMain.
This might make running tests in subprojects slightly slower but I think
its a worthy tradeoff.
In order for the sbt launcher to be able to resolve a local version of
sbt, we must publish the main jar, the sources jar, the doc jar, the pom
and an ivy.xml file. The publish and publishLocal tasks are wired in
IvyXml.scala to create an ivy.xml file before running publish. This
wasn't done with publishLocalBin which made it not work when no ivy.xml
file was already present (which was the case after running clean).
The swoval javafmt plugin uses the google java formatter (which I
believe is the only widely used java formatter) to format source files.
It does not provide an automatic javafmtOnCompile method like the
scalafmt plugin so I had to manually implement that functionality. In
general the java formatter is much faster than scalafmt so the impact of
having javafmtOnCompile set to true is very low.
Intellij has problems with older versions of scalafmt (see
https://github.com/scalameta/scalafmt/issues/1630). Not sure if this a
scalafmt issue or an intellij issue. I pinned the edition to October
2019 to avoid reformatting tons of files. Once development stabilizes,
we may wish to drop the edition but, for now, it's disruptive to change
the formatting.