`UpdateOptions` is only meant for user-provided options. Since this
option should not be exposed to users, it's moved to inline ivy
configuration, whose parameters require information about the build and
is usually defined project-wide instead of `Global`-wide.
Managed checksums tells ivy to forget about checking checksums for jar
files and just persist them in the cache.
The user that enables that option will take care of verifying they are
correct.
This is a big deal because:
1. Ivy takes *a lot of time* checking timestamps for big jars, and does
it sequentially. The user (sbt) can do better by running these checks
in parallel, speeding up the whole resolution process!
2. The fact that the sha files are not present in the cache means that
build tools cannot check if a checksum is correct without preprocessing
the jar.
Note that the user is responsible for keeping the cache consistent. If
the checksum is not correct, the user should report it and overwrite the
checksum file with the valid digest.
Sometimes, for predictability and performance, we may be interested in
specifying the concrete resolver that a `ModuleID` should use.
This patch achieves this by adding a new field to `UpdateOptions` and
then getting this information from the `SbtChainResolver`, that will
select the concrete resolver for a given dependency descriptor.
Why is this useful? Well, two reasons:
* Predictable behaviour. We have the guarantee that an artifact only
comes from a concrete resolver.
* Resolution speedup. Around 1/3 or 1/2 times faster than normal
resolution in a moderate test case scenario. If there is a lot of
latency or network connection is poor, speedups will be higher.
LOGS:
```
NORMAL RESOLUTION TIME 1790
FASTER RESOLUTION TIME 1054
```
```
NORMAL RESOLUTION TIME 2078
FASTER RESOLUTION TIME 1055
```
Lots of projects can benefit from this option, as well as organizations
and companies. This will eventually integrate with the dependency lock
file, but can be used independently of it.
To ensure that SNAPSHOTs are always the latest, we go through all the
resolved modules and check their timestamps. Good.
However, if in the process of reparsing/redownloading the module
descriptor we fail (or it's not found in that resolver at all), then we
cannot refresh the resolved module that could have been internalized or
heavily cached in memory by ivy. We do this for correctness.
This patch does two things:
1. Adds more comments.
2. Warns only when there are parsing errors in ivy files.
3. Adds debug info in the rest of the cases.
This removes the pain of seeing `Unable to parse` that could be caused by
other reasons that are not related to parsing at all and which would not
affect the algorithm at hand. For instance, if we get a URLResource,
that's totally fine -- there is no way we could parse the ivy file. Ivy
uses URLResources in several cases where the artifact origin URL is
specified.
The frozen mode is used by the dependency lock file.
It makes sure that the resolution is always intransitive and that ivy
does not check for changed dependencies and stores that information in
the resolved ivy files. Following the ivy documentation, the last change
may bring a slight performance improvement:
http://ant.apache.org/ivy/history/latest-milestone/use/resolve.html
This is the most important part of the sbt ivy management and it's
almost unreadable in the previous shape...
This attempts to make a rewrite without any semantic change.
The previous custom offline implementation was not working on 100% of
the cases and relied on the TTL of ivy. As the previous commit
enabled the native offline implementation provided by ivy as of 2.3.0,
this functionality is not useful anymore.
The current place to specify offline is `UpdateConfiguration`, and not
`InlineIvyConfiguration` that is required to instantiate sbt. With the
current approach, we can be online or offline without having to
instantiate ivy sbt twice.
I will provide a Scalafix rewrite for this change.
The following commit tries to address the well-known issue that sbt
cannot be used in offline mode. In order to enable that use case, this
commit adds support for a flag in update configuration called `offline`
that users can change as they wish (and that sbt will expose via
settings).
It adds tests to check that the resolution uses the caches instead of
trying to resolve from the Internet. Unfortunately, ivy does not expose
a way to know whether a resolution was made from the cache or the
Internet, so the test case invents a metric to check that resolution
indeed happens from cache.
In order to benefit from this 100%, we need to update to ivy 2.4.0 or
cherry-pick a commit because a major issue in `useCacheOnly` has been
fixed: https://issues.apache.org/jira/browse/IVY-1515.
In short, this is good for the dependency lock file too. Since we can
make sure that once we have downloaded and resolved all the dependencies
locally, we do resolve from the cache.
This commit reduces the code duplication in `updateEither` which was
duplicating a good deal of the resolution logic to deal with the
different resolution mechanisms: the simple one and the cached one.
It also unifies the signatures of the helpers that are invoked by
`updateEither`, removing the weirdness of the different return type
signatures and ad-hoc logic handling.
`cleanCachedResolutionCache` and `clean` were doing extra work because
`clean` does only need to clean the cache in the configuration (that is
a class parameter).
This is a port of https://github.com/sbt/sbt/pull/2992.
Original description of the feature:
```
Ivy downloads have traditionally been single-threaded.
Parallel downloads are a must for a modern build tool. This commit
builds upon the work done by Josh Suereth in the branch
sbt/ivy-parallel-download-artifact.
To avoid adding external dependencies, it uses the Scala parallel
collections. If maintainers consider that is worth it to use a more
modern and appropriate approach, like Task, I'm happy to reimplement
the features with it.
```
This commit does not preserve Josh's metadata in the commit since the
whole design of the repository has changed and I did not know how to
import a commit from sbt/sbt. However, it does apply several changes to
the original PR.
Co-authored-by: Josh Suereth <joshua.suereth@gmail.com>
The following PR does two things:
* Removes the unnecessary `SbtExclusionRule` that was introduced to
exclude artifacts at the project level (and not the dependency level).
This change was done in an independent class to avoid breaking
bincompat in 0.13.x series.
* Handle exclusion rules correctly, propagating the cross version to the
exclusions of the dependencies.
To fixsbt/sbt#1518, this PR takes the avenue taken in
`SbtExclusionRule`, it accepts `GroupArtifactID` which should be the
preferred way to specify dependencies from now on. Unlike
`SbtExclusionRule`, it also supports `ModuleID` for those that want to
exclude a concrete dependency.
`InclExcl` did not have any tests. The following commit also adds a
testing suite for it, showing how the issue it's fixed and how you
should use `exclude` if you're calling directly `ExclusionRule` instead
of passing in `GroupArtifactID` and `ModuleID`.
I'm now going to use `CacheStore.apply` in `JsonUtil` (used by cached resolution). This gets rid of `fileToStore` parameter from a bunch of classes and simplifies the setup around librarymanagement.
eugene yokota
jvican
Dale Wijnand