Fixessbt/sbt#2699
Before:
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * com.google.code.findbugs:jsr305:2.0.1 -> 3.0.0
[warn] Run 'evicted' to see detailed eviction warnings
After:
[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn]
[warn] * com.typesafe.akka:akka-actor_2.12:2.5.0 is selected over 2.4.17
[warn] +- de.heikoseeberger:akka-log4j_2.12:1.4.0 (depends on 2.5.0)
[warn] +- com.typesafe.akka:akka-parsing_2.12:10.0.6 (depends on 2.4.17)
[warn] +- com.typesafe.akka:akka-stream_2.12:2.4.17 () (depends on 2.4.17)
[warn]
[warn] Run 'evicted' to see detailed eviction warnings
Dotty is versioned as 0.*, but `CrossVersionUtil#binaryScalaVersion`
will return the full version instead of just `major.minor` for all
compiler versions < 2.10, add a special case for Dotty to avoid this.
`UpdateOptions` is only meant for user-provided options. Since this
option should not be exposed to users, it's moved to inline ivy
configuration, whose parameters require information about the build and
is usually defined project-wide instead of `Global`-wide.
Managed checksums tells ivy to forget about checking checksums for jar
files and just persist them in the cache.
The user that enables that option will take care of verifying they are
correct.
This is a big deal because:
1. Ivy takes *a lot of time* checking timestamps for big jars, and does
it sequentially. The user (sbt) can do better by running these checks
in parallel, speeding up the whole resolution process!
2. The fact that the sha files are not present in the cache means that
build tools cannot check if a checksum is correct without preprocessing
the jar.
Note that the user is responsible for keeping the cache consistent. If
the checksum is not correct, the user should report it and overwrite the
checksum file with the valid digest.
Sometimes, for predictability and performance, we may be interested in
specifying the concrete resolver that a `ModuleID` should use.
This patch achieves this by adding a new field to `UpdateOptions` and
then getting this information from the `SbtChainResolver`, that will
select the concrete resolver for a given dependency descriptor.
Why is this useful? Well, two reasons:
* Predictable behaviour. We have the guarantee that an artifact only
comes from a concrete resolver.
* Resolution speedup. Around 1/3 or 1/2 times faster than normal
resolution in a moderate test case scenario. If there is a lot of
latency or network connection is poor, speedups will be higher.
LOGS:
```
NORMAL RESOLUTION TIME 1790
FASTER RESOLUTION TIME 1054
```
```
NORMAL RESOLUTION TIME 2078
FASTER RESOLUTION TIME 1055
```
Lots of projects can benefit from this option, as well as organizations
and companies. This will eventually integrate with the dependency lock
file, but can be used independently of it.
To ensure that SNAPSHOTs are always the latest, we go through all the
resolved modules and check their timestamps. Good.
However, if in the process of reparsing/redownloading the module
descriptor we fail (or it's not found in that resolver at all), then we
cannot refresh the resolved module that could have been internalized or
heavily cached in memory by ivy. We do this for correctness.
This patch does two things:
1. Adds more comments.
2. Warns only when there are parsing errors in ivy files.
3. Adds debug info in the rest of the cases.
This removes the pain of seeing `Unable to parse` that could be caused by
other reasons that are not related to parsing at all and which would not
affect the algorithm at hand. For instance, if we get a URLResource,
that's totally fine -- there is no way we could parse the ivy file. Ivy
uses URLResources in several cases where the artifact origin URL is
specified.
The frozen mode is used by the dependency lock file.
It makes sure that the resolution is always intransitive and that ivy
does not check for changed dependencies and stores that information in
the resolved ivy files. Following the ivy documentation, the last change
may bring a slight performance improvement:
http://ant.apache.org/ivy/history/latest-milestone/use/resolve.html
This is the most important part of the sbt ivy management and it's
almost unreadable in the previous shape...
This attempts to make a rewrite without any semantic change.
The previous custom offline implementation was not working on 100% of
the cases and relied on the TTL of ivy. As the previous commit
enabled the native offline implementation provided by ivy as of 2.3.0,
this functionality is not useful anymore.
The current place to specify offline is `UpdateConfiguration`, and not
`InlineIvyConfiguration` that is required to instantiate sbt. With the
current approach, we can be online or offline without having to
instantiate ivy sbt twice.
I will provide a Scalafix rewrite for this change.
The following commit tries to address the well-known issue that sbt
cannot be used in offline mode. In order to enable that use case, this
commit adds support for a flag in update configuration called `offline`
that users can change as they wish (and that sbt will expose via
settings).
It adds tests to check that the resolution uses the caches instead of
trying to resolve from the Internet. Unfortunately, ivy does not expose
a way to know whether a resolution was made from the cache or the
Internet, so the test case invents a metric to check that resolution
indeed happens from cache.
In order to benefit from this 100%, we need to update to ivy 2.4.0 or
cherry-pick a commit because a major issue in `useCacheOnly` has been
fixed: https://issues.apache.org/jira/browse/IVY-1515.
In short, this is good for the dependency lock file too. Since we can
make sure that once we have downloaded and resolved all the dependencies
locally, we do resolve from the cache.
This commit reduces the code duplication in `updateEither` which was
duplicating a good deal of the resolution logic to deal with the
different resolution mechanisms: the simple one and the cached one.
It also unifies the signatures of the helpers that are invoked by
`updateEither`, removing the weirdness of the different return type
signatures and ad-hoc logic handling.
`cleanCachedResolutionCache` and `clean` were doing extra work because
`clean` does only need to clean the cache in the configuration (that is
a class parameter).
Dale Wijnand
eugene yokota
jvican
Guillaume Martres