There is more we can do during the `Fetching` stage, but it will
require additional tweaks, so just keeping it simple here.
Similar to https://github.com/iterative/dvc/pull/9974
Existing logic might result in weird fs cache location like
`.dvc/cache/files/md5/fs`, while we really only want
`.dvc/cache/fs`, since it is not an odb cache.
If user uses some custom cache location, there is really no reason for us
to try to gitignore it, as we have no idea what he is planning to do with
that and overall it often might be sign of this being misused. Maybe we could
also have some warnings/errors, but that's out of scope for this PR.
Fixes https://github.com/iterative/studio/issues/7405
Currently we get odd formatting like
```
456 .devcontainer.json
139 .dvcignore
61.0 .gitattributes
96.0 .github
7.00 .gitignore
5.76k README.md
160 data
1.84k dvc.lock
519 dvc.yaml
4.00 foo
96.0 models
96.0 notebooks
225 params.yaml
46.0 requirements.txt
128 results
160 src
```
(notice stuff like 46.0)
and there doesn't seem to be a simple way to fix that in tqdm, in addition to
the general feeling that we are abusing tqdm's util for this.
So let's just use `humanize`, which gets us a more pleasing result:
```
456B .devcontainer.json
139B .dvcignore
61B .gitattributes
96B .github
7B .gitignore
5.8K README.md
160B data
1.8K dvc.lock
519B dvc.yaml
4B foo
96B models
96B notebooks
225B params.yaml
46B requirements.txt
128B results
160B src
```
Likely with removal of dvc-machine pyinstaller stopped picking up dvc_ssh automatically, so we need to declare it explicitly just like the rest of plugins.
This used to work like so but by accident, when we didn't reload config
at all when collecting other revisions.
Regression from https://github.com/iterative/dvc/pull/9758
Adds support for `--config` to all three commands. For `ls` and `get`, it is
a purely runtime option that will merge the config you provide with target
repo's configs (similar to how .dvc/config.local works). E.g.
```
$ cat myconfig
[remote "myremote"]
access_key_id = 12345
secret_access_key = 98765
$ dvc get https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
```
In case of `dvc import`, the semantics are the same, but the value is also
recorded in the resulting dvcfile. E.g.
```
$ dvc import https://github.com/efiop/mydataregistry recent-grads.csv --config myconfig
...
$ cat recent-grads.csv.dvc
md5: 85c86eeb3afa6d1f7d11beedc644d28e
frozen: true
deps:
- path: recent-grads.csv
repo:
url: https://github.com/efiop/mydataregistry
config: myconfig
rev_lock: af22e06bbfe34d62a140437abf32f48b535944a7
outs:
- md5: f447e23a170a9f25a7799f069a1ba934
size: 26872
hash: md5
path: recent-grads.csv
```
This is the most general and powerful mechanism that we can give, that works
for everything from reconfiguring your default remote to completely changing
your remote layout and beyond (will be handy in the future).
Fixes#2466
Looks like flufl-lock switched to pdm in 8.0 which triggered a
```
pkg_resources.DistributionNotFound: The 'flufl.lock>=5' distribution was not found and is required by dvc
```
error in multiple scenarious. Can't put my finger on anything specific
in flufl-lock yet, so will just limit its version for now.
Fixes https://github.com/iterative/dvc/issues/9654
Relpath might not exist (e.g. on windows between different drives),
we should just use path as is and not get too fancy with the ui,
unless we want to constantly keep ValueError in mind when using
relpath for ui needs all over the place.
These operations should use index, where we always have current
directory and relative paths, so the ui will change accordingly
anyway.
* output: save: error out on cached external outputs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
With all fetch and checkout related things that we've done in the recent past,
we now have all the mechanisms necessary to provide actual progress (no total,
but still better than spinner) during `dvc data status` operations.
* fs: callbacks: create our own TqdmCallback
We have a near identical implementation in dvc-objects, but this one is
meant to work with our Tqdm progress instance.
dvc-objects one is useful on other levels, but dvc should have itsown
that inherits from dvc-objects' Callback.
* progress: add as_callback
* checkout: use ui.progress
The build process is much more stable these days and there is no reason
to keep building these packages here. Instead every package has itsown
building process (dvc-osxpkg, dvc-exe, dvc-s3-repo) and knows best how
to build and sign correctly.
pyinstaller hooks need to go into a `__pyinstaller` later on.
For some unknown reason the mock in this test leaks into the outside tests.
I wasn't able to determine why in the reasonable amount of time, so instead just
simplifying it to not use mocker at all.
That whole file should really be migrated to dvc-data, but that needs some time.
Had the "pleasure" of finding a mocker.patch leak into other tests and one
of the suspicions is that mixing unittest.mock and mocker might be somehow
related.
In any case, we should use mocker where possible anyway.
This avoids recollecting the cache files over and over again.
For example, `dvc fetch` noop in `dvc-bench` went from ~30sec down to ~10sec
for me.
Next step is to also cache remote index, so we don't have to waste those
~10sec checking files in the remote.
Related https://github.com/iterative/dvc-data/pull/376
Ruslan Kuprieiev
