In addition to ExecuteProgress, this new interface allows builds and plugins to receive events when commands start and finish, including the State before and after each command. It also makes cancellation visible to clients by making the Cancelled type top-level.
- remove unused type params
- use `withFilter` if possible
- use `collectFirst` instead of `collect` and `headOption`
- use `length` instead of `size` if `Array` or `String`
- use `foreach` instead of `map`
Problem
-------
There's a bug in ipcsocket cleanup logic that effectively
tries to wipe out /tmp.
Workaround
----------
We should fix the underlying bug, but we can start by
explicitly configuring the temp directories ipcsocket uses.
Ref https://github.com/sbt/sbt/issues/6931
The `XDG_RUNTIME_DIR` environment variable will be used to choose the base directory in which the sockets will be created by **sbt**. This can help when issues such as #6777 appear. There are other related issues, such as #6101, [Metals issue #2235](https://github.com/scalameta/metals/issues/2235), [Che issue #18394](https://github.com/eclipse/che/issues/18394). Those are closed issues, but there are some cases in which the solution does not work (such as the case in #6777). Furthermore, the solution given seems more like a workaround than an actual fix.
**What causes this issue?**
At least in my case, the **ServerAlreadyBootingException** is thrown after **sbt** tries to create a Unix domain socket and fails. In my specific environment, I was not able to create the sockets in some directories because they were mounted on an NFS. This prevented me from using any automated sbt command on any of those directories (Metals uses sbt under the hood, and it was not able to start because of the ServerAlreadyBootingException), which in turn resulted in me not being able to use Metals on a project that was created on any of those directories.
**How is the issue solved in this PR?**
If the `XDG_RUNTIME_DIR` environment variable is set, then sockets will be created in a folder with a hashed name, inside the directory `XDG_RUNTIME_DIR/.sbt/`. If this variable is not set, everything works exactly the same as always.
Let's see an example:
1. My environment variable `XDG_RUNTIME_DIR` is set to /home/alonso/.runtime-dir`
2. I create my project in `/home/alonso/workspace/hello-world`
3. I run `sbt compile`
4. The socket is created in `/home/alonso/.runtime-dir/.sbt/sbt-socket102030405060/sbt-load.sock`. The number **102030405060** is a hash that is computed from the base directory of the project (it is actually the same hash that is produced in order to create sockets in Windows) and it will be different depending on the location of the project
5. The sbt command executed correctly, which would not have happened if the socket had been created in the `home/alonso/workspace` directory or any of its subdirectories (in this example, `/home/BlackDiamond/workspace` corresponds to a network file share)
Co-authored-by: Alonso Montero <lumontero@mobilize.net>
The `XDG_RUNTIME_DIR` environment variable will be used to choose the base directory in which the sockets will be created by **sbt**. This can help when issues such as #6777 appear. There are other related issues, such as #6101, [Metals issue #2235](https://github.com/scalameta/metals/issues/2235), [Che issue #18394](https://github.com/eclipse/che/issues/18394). Those are closed issues, but there are some cases in which the solution does not work (such as the case in #6777). Furthermore, the solution given seems more like a workaround than an actual fix.
**What causes this issue?**
At least in my case, the **ServerAlreadyBootingException** is thrown after **sbt** tries to create a Unix domain socket and fails. In my specific environment, I was not able to create the sockets in some directories because they were mounted on an NFS. This prevented me from using any automated sbt command on any of those directories (Metals uses sbt under the hood, and it was not able to start because of the ServerAlreadyBootingException), which in turn resulted in me not being able to use Metals on a project that was created on any of those directories.
**How is the issue solved in this PR?**
If the `XDG_RUNTIME_DIR` environment variable is set, then sockets will be created in a folder with a hashed name, inside the directory `XDG_RUNTIME_DIR/.sbt/`. If this variable is not set, everything works exactly the same as always.
Let's see an example:
1. My environment variable `XDG_RUNTIME_DIR` is set to /home/alonso/.runtime-dir`
2. I create my project in `/home/alonso/workspace/hello-world`
3. I run `sbt compile`
4. The socket is created in `/home/alonso/.runtime-dir/.sbt/sbt-socket102030405060/sbt-load.sock`. The number **102030405060** is a hash that is computed from the base directory of the project (it is actually the same hash that is produced in order to create sockets in Windows) and it will be different depending on the location of the project
5. The sbt command executed correctly, which would not have happened if the socket had been created in the `home/alonso/workspace` directory or any of its subdirectories (in this example, `/home/BlackDiamond/workspace` corresponds to a network file share)
Co-authored-by: Alonso Montero <lumontero@mobilize.net>
In order to start the sbt server we must use the sbt script because
it is the only way to load the .sbtopts and the .jvmopts file properly.
To do so the sbt script can pass a -Dsbt.script prop to the java server.
It is used in the NetworkClient to start the server, and it is replicated
in the BuildServerConnection file (.bsp/sbt.json).
The problem was that -client was different from --client, which
makes for a confusing user experience. So, this change makes
them the same and re-names -client to --java-client. The value
of this is that hopefully -client and --client being the same
feels more logical to users.
The intellij bsp integration launches sbt with the launcher and runs
-bsp. This doesn't work if sbt is not on intellij's path. To try and
work around this, we can add an option --sbt-launch-jar that is
recognized by the network client and will make it use the launch jar
rather than the sbt script to launch a new server if needed.
Made the bspConfig task dependendant on the bspConfig value.
Changed the bspConfig setting to use a attributeKey so we can use it in the server as well.
The new version provides an apple silicon arm64 jni implementation of
the ipcsocket api. It also adds a jni implementation for getting the max
socket length which is necessary because jna isn't supported on the
apple arm64 platform yet.
There isn't yet a version of the jna available that works with the new
apple silicon using arm64. To workaround this, we can use the jni
implementation by default on arm64 macs. If the user wants to force the
jni implementation for any supported platform, they can opt in with the
`sbt.ipcsocket.jni` system property and/or by setting the serverUseJni
setting.
Network client already supports the -bsp command (since
65ab7c94d0). This commit reworks the
BspClient.run method so that it delegates to the NetworkClient. The
advantage to doing it this way is that improvements to starting up the
sbt server by the thin client will automatically propagate to the -bsp
command. The way that it is implemented, all of the output generated
during server startup will be redirected to System.err which is useful
for debugging without messing up the bsp protocol, which relies on only
bsp messages being written to System.out.
The boot server socket was not working correctly when the sbt server was
started by the thin client. This was because it is necessary for us to
create a ConsoleTerminal in order for System.out and System.err to be
properly forwarded to the clients connected over the boot server socket.
As a result, if you started a server instance of sbt with the thin
client, you wouldn't see any output util you connected to the server.
The fix is to just make sure that we create a console terminal if sbt is
run as a subprocess.
When a batch command is run with the thin client, it logs an info
message that the command completed. This is unnecessary given that
completion is implied by the success or failure method that follows. It
made the output look a little different in the thin client vs the
console.
I saw a stacktrace when exiting sbtn on windows due to an interrupted
exception being thrown during thread joining. We only want to throw this
exception if we didn't successfully join the thread. I also noticed that
we would try to join the thread forever. There was supposed to be a
timelimit so that we would eventually stop blocking even if we were
unable to join the thread. The limit was set but not respected.
With sbtn, the system properties are passed in as regular command
arguments. We need to parse them before we call Terminal.withStreams or
else system properties like -Dsbt.color=false are ignored.
EvaluateTask was holding references to SafeState that could be quite
large. This was reported as #5992. In that project, I ran the `ci` task
and observed the OOM as reported. I took a heap dump prior to OOM and
got the retained size graph from visualvm (which took hours to compute).
The lastEvaluatedState was holding a reference to SafeState that was
1.7GB. The project max heap size was set to 2GB. Instead of using the
lastEvaluatedState, we can just use StandardMain.exchange.withState.
The cached instances of state were used for task cancellation and
completions. While it is possible that early on in booting
StandardMain.exchange.withState could return a null state, in practice
this won't happen because it is set early on during the sbt boot
commands.
After this change, I successfully ran the `ci` task in the #5992 issue
project with the same memory parameters as their ci config.
The ConsoleAppender formatEnabledInEnv field was being used both as an
indicator that ansi codes were supported and that color codes are
enabled. There are cases in which general ansi codes are not supported
but color codes are and these use cases need to be handled separately.
To make things more explicit, this commit adds isColorEnabled and
isAnsiSupported to the Terminal companion object so that we can be more
specific about what the requirements are (general ansi escape codes or
just colors). There are a few cases in ConsoleAppender itself where
formatEnabledInEnv was used to set flags for both color and ansi codes.
When that is the case, we use Terminal.isAnsiSupported because when that
is true, colors should at least work but there are terminals that
support color but not general ansi escape codes.
When project/target is a symbolic link, sbt 1.4.0 crashes on startup
because Files.createDirectories will throw a FileAlreadyExistsException.
The fix is to first check if the target directory exists before trying
to create it.
When sbt is starting up and there is an error with the build loading, we
need to read input from the user to determine to restart the build or
not. What is tricky is that there are potentially two sources of input:
thin clients connected through the boot server socket and the actual sbt
console process. If there are not connected thin clients and no system
console is available, we should return -1 in System.in.read, which will
cause sbt to exit.
A user reported that no prompt was displayed when they used sbt from the
jEdit console. The reason no prompt was displayed was because
System.console was null which caused the ConsoleChannel to be
initialized with its prompt set to NoPrompt. I don't remember why it was
doing this but this bug seems worse than whatever it was trying to
address.
In the client test, the sbt server would keep open the the client
connection even after it had exited because the client was only shutting
down its side of the connection. Since in the test it wasn't exiting the
jvm, the read side of the connection was still open.
It is possible for the test cases in ClientTest to block indefinitely.
To avoid that, we can instead run them on a background thread and cancel
the thread if that happens.
CI hung in the server test that checks completions because I broke
reading from the System.in. It occurred to me that we probably shouldn't
prompt indefinitely when the user is running tab completions anyway so I
set a timeout of 5 seconds for the user to respond to input. If they
decline to start a server within the timeout, we will just exit. If they
decline to run compilation within the timeout we just skip the
compilation step (so test or main class names will not be provided for
completions).
There were a number of issues with swithcing between raw and canonical
issues that affected both the server and the thin client. These were
reported in #5863 and #5856. In both cases, there were issues with
reading input or having the input be displayed. Debugging those issues
revealed a number of issues with how we were using the jline 3 system
terminal and the hybrid interaction with the jline 2 terminal. This
commit eliminates all of our internal jline 2 usage. The only remaining
jline 2 usage is that we create and override the global terminal for the
scala console for scala versions < 2.13. By moving away from jline 2, I
was also able to fix#5828, which reported that the home, end and delete
keys were not working.
One of the big issues that this commit addresses is that the
NetworkClient was always performing blocking reads on System.in. This
was problematic because it turns out that you can't switch between raw
and canonical modes when there is a read present. To fix this, the
server now sends a message to the client when it wants to read bytes and
only then does the client create a background thread to read a single
byte.
I also figured out how to set the terminal type properly for the thin
client on windows where we had been manually setting the capabilities to
ansi, which only worked for some keys. This fix required switching to
the WindowsInputStream that I introduced in a prior commit. Before we
were using the jline 2 wrapped input stream which was converting some
system events, like home and end, to the wrong escape sequence mappings.
The remainder of the commit is mostly just converting from jline 2 apis
to jline 3 apis.
I verified that tab completions, the scala console, the ammonite console
and a run task that read from System.in all work with both the server
and the thin client on mac, linux and windows after these changes.
Fixes#5828, #5863, #5856
There are situations in windows where it is possible that a client
attempts to connect to a portfile that it did not have access to. This
can happen if, for example, the server is started in administrator mode
but the client is started as a regular user. When this happens, the
client would try to remove the portfile and start a new server. This new
server would not actually be able to start a server though becuase it
would not be able to open the named point because the other server had
it open. As a result, the client would just hang. The fix is to just
abort the thin client if it gets an access denied exception.
When a json rpc request is specified without any headers, the size of
the message may exceed the buffer that was created for reading hdeaders.
This would cause an exception to be thrown when creating a string from
the header buffer because the number of bytes requested would exceed the
capacity of the buffer. To fix this, we can expand the buffer
dynamically if needed. For the common case when the headers are
specified, this should be a no-op.
Ref https://github.com/sbt/sbt/issues/5665
This adds `--server` command that is immediately filtered out in Main.scala.
The purpose of `--server` is so we can invoke thin client from `sbt` script at some point in the future when Bash script can parse `project/build.properties`.
`sbtn` would need to call `sbt` again to start the server, and at that point the shell script would need to actually invoke the server. The intent of `--server` is to be used as the tie breaker.
Also build users may want to sometimes call `sbt --server`.
I introduced the terminalShellPrompt so that we could generate a prompt
that was colored only if the terminal supported color. Rather than
expose the terminal implementation detail, we can just use a boolean
flag that toggles whether or not color is enabled and sbt can pass in
the value of terminal.isColorEnabled into the function.
sbt 1.4.0 generates the shell prompt using the terminal properties for
the specific terminal for which the prompt is rendered. The mechanism
for doing this broke the prompt for projects that overrode the
shellPrompt key, notably the play plugin. After this change, the play
custom prompt is correctly rendered with 1.4.0-SNAPSHOT.
When starting sbt via the thin client with 1.4.0-RC1, there is no output
until sbt finishes booting up which is poor ux. The reason is that sbt
only uses virtual io when sbt.io.virtual == true or formatEnabledInEnv
== true and not ci. The default value for formatEnabledInEnv is set
based on whether color is enabled in the environment. This had copied
old logic that turned on color if ansi was enabled but it makes more
sense to check the color property (which is set by the thin client via
an environment variable when it launches sbt) and fall back to whether
or not java.lang.System.console is defined. We also can explicitly set
"-Dsbt.io.virtual=true" when the thin client launches sbt since the thin
client relies on this behavior. By doing it in both places, the sbtn
for 1.4.0-RC1 will display boot output for newer versions of sbt.
Bonus: don't call ConsoleAppender.formatEnabledInEnv which just calls
back to Terminal.formatEnabledInEnv
The sbt.log.noformat parameter should be treated very similarly to
sbt.io.virtual. When it is true, we should just use the raw io streams
for the process. This came up because of
https://github.com/sbt/sbt/issues/5784 which reported that intellij
imports were not working and that ansi control characters were being
written to the output.
There can be race conditions where we try to interrupt and join a ui
thread before it becomes interruptible by blockign on a queue. To
workaround this, we can add the JoinThread class which adds an
extension method Thread.joinFor that takes a FiniteDuration parameter.
This variant of join will repeatedly interrupt and attempt to join the
thread for up to 10 milliseconds before retrying until the limit is
reached. If the limit is reached, we print a noisy error to the console.
I'm not 100% sure if we are leaking threads in the latest sbt version
but this gives me more piece of mind that either we are always
successfully joining the threads or we will be alerted if the joining
fails.
When exiting the thin client, an interrupted exception stack trace ends
up being printed because NonFatal doesn't include interrupted
exceptions.
Fixes https://github.com/sbt/sbt/issues/5759
Fixes https://github.com/sbt/sbt/issues/5039
Fixes https://github.com/sbt/sbt/issues/4989
This is take 2 on the semicolon fix by emptying out the completion examples in the multi parser.
```
> set scalaV
```
would complete to
```
> set scalaVersion
```
and more importantly
```
coursierUseSbtCredentials := true
```
errors to
```
sbt:hello> coursierUseSbtCredentials := true
[error] Expected ID character
[error] Not a valid command: coursierUseSbtCredentials
[error] Expected project ID
[error] Expected configuration
[error] Expected ':'
[error] Expected key
[error] Not a valid key: coursierUseSbtCredentials (similar: csrExtraCredentials, credentials)
[error] coursierUseSbtCredentials := true
[error] ^
```
With the latest sbt snapshot, the ui would get stuck if the user entered
an empty command. They would be presented with an empty prompt and could
not input any commands. This was caused by the change in
d569abe70a that reset the prompt after a
line was read. I had tried to optimize line reading by ignoring empty
commands in UITask.readline so we wouldn't have to make a new thread.
This optimization wasn't really buying much since it only affects how
quickly the user is reprompted after entering an empty command. Unless a
user is spamming the <enter> key, they shouldn't notice a difference.
It was a bit tricky to reason about the state of the prompt for a
terminal. To help make things more clear, I reworked things so that the
LineReader always sets the prompt to Pending after it reads a command.
In MainLoop, we cache the prompt value and temporarily set it to Running
while the command is running, which is really how it should have always
been.
In order to make the console task work with scala 2.13 and the thin
client, we need to provide a way for the scala repl to use an sbt
provided jline3 terminal instead of the default terminal typically built
by the repl. We also need to put jline 3 higher up in the classloading
hierarchy to ensure that two versions of jline 3 are not loaded (which
makes it impossible to share the sbt terminal with the scala terminal).
One impact of this change is the decoupling of the version of
jline-terminal used by the in process scala console and the version
of jline-terminal specified by the scala version itself. It is possible
to override this by setting the `useScalaReplJLine` flag to true. When
that is set, the scala REPL will run in a fully isolated classloader. That
will ensure that the versions are consistent. It will, however, for sure
break the thin client and may interfere with the embedded shell ui.
As part of this work, I also discovered that jline 3 Terminal.getSize is
very slow. In jline 2, the terminal attributes were automatically cached with a
timeout of, I think, 1 second so it wasn't a big deal to call
Terminal.getAttributes. The getSize method in jline 3 is not cached and
it shells out to run a tty command. This caused a significant
performance regression in sbt because when progress is enabled, we call
Terminal.getSize whenever we log any messages. I added caching of
getSize at the TerminalImpl level to address this. The timeout is 1
second, which seems responsive enough for most use cases. We could also
move the calculation onto a background thread and have it periodically
updated, but that seems like overkill.
There are cases where if the ui state is changing rapidly, that an
AskUserThread can be created and cancelled in a short time windows. This
could cause problems if the AskUserThread is interrupted during
`LineReader.createReader` which I think can shell out to run some
commands so it is relatively slow. If the thread was interrupted during
the call to `LineReader.createReader` and the interruption was not
handled, then the thread would go into `LineReader.readLine`, which
wouldn't exit until the user pressed enter. This ultimately caused the
ui to break until enter because this zombie line reader would be holding
the lock on the terminal input stream.
Rather than relying on a command, I realized it makes more sense to
explicitly set the terminal for the calling channel in MainLoop. By
doing it this way, we can also ensure that we always reset it to the
previous value.
This reverts commit b1dcf031a5.
I found that b1dcf031a5 had some
unintended consequences that seemed to mess up the prompt state. The
real problem that it was trying to address was that the prompt was being
interleaved with log messages in some scenarios. There was a different
way to fix that in ProgressState that was both simpler and more
reliable.
I noticed that when reloading the build, that certain errors are logged
by sbt to System.err. These were not shown to a thin client because we
weren't forwarding System.err. This change remedies that.
System.err is handled more simply than System.out. We do not put
System.err through the progress state because generally System.err is
tends to be unbuffered. I had hesitated to add System.err to the
Terminal interface at all to give users an escape hatch but I couldn't
get project loading to work well with the thin client without it.
Iulian Dragos
xuwei-k
Julien Richard-Foy
Eugene Yokota
Matthew de Detrich
Alonso Montero
dmitrii.naumenko
Philippus
David Francoeur
Adrien Piquerez
Samuel CLARENC
Samuel CLARENC
Amina Adewusi
Ondra Pelech
Sam Halliday
João Ferreira
Guillaume Martres
Ethan Atkins
Erlend Hamnaberg