`--untracked-files` and `--unchanged` coupled with `--granular`
puts a lot of stress on `show_status`. Currently, with those options,
dvc spends ~26s on `_show_status` function.
Unhappy with the above result, I looked into improving the situation with
`console.out`. It did reduce that to 9s but it still felt like a lot.
Then I dropped to good old colorama, which dropped to 3s(term)/1.24(redirected).
I tested with buffering output which dropped this further to
~600ms(term)/~100ms(redirected).
This may seem like premature optimization, but this was preventing me
from figuring out perf issues in granular outputs, and I believe the
user should not be penalize for using feature that we already offer.
`dvc <command> | head` for example could fail with SIGPIPE and
would print a traceback with BrokenPipeError exception. Now we
redirect do /dev/null in case of error as suggested in Python's
docs: https://docs.python.org/3/library/signal.html\#note-on-sigpipe.
Currently, there is no way to find run-cache in HTTP remotes.
In the future, we could use directory-listings from supported
webservers to find files.
Temporarily, we are restricting the run-cache uploads. This is a bit
strict, we could have only restricted the downloads, but it might be
more confusing to the users, so I went to categorically restrict
stage cache uploads and downloads for http filesystems.
We were passing paths with '\\' in the filename instead of posix paths to
remote filesystems which was causing failures to upload.
Although in some filesystems, this was failing silently, so upload would seem
to have passed, but the download later would fail.
This was working for LocalFileSystem as they work for any directories/path.
It's hard to test this these days, as we don't have remote tests inside
dvc, so it was not caught.
Potentially fixes#7718.
The tests should be in dvc-objects repo npw, and will be migrated soon.
The nanotime and diskcache are specified in dvc-objects, so we don't need
those here.
This moves get_fs_config/get_fs_cloud and ODBManager to DVC
as they require DVC related utilities, and we don't need them
strictly in dvc-objects/dvc-data.
This also adds support to ReferenceHashFile to load from
any filesystem, using it's module name and filename.
Now we can get rid of DVCFileSystem dependency in
ReferenceHashFile.
This is a temporary solution, we really need a registry.
(I am not quite sure if this works in pyinstaller mode,
but I think it should, it's how fsspec does it if I understand
correctly.)
transfer raises internal TransferError, merge raises internal MergeError
and local ODB raises ODBError on unprotect.
transfer's error is reraised to FileTransferError back in DataCloud.
This commit:
* defines a `Ignore` protocol in `dvc.objects`, that expects a `find` method.
This will be used in dvc-data and dvc-objects instead of depending on dvc.ignore.
dvcignore is expected to respect this protocol, which it does.
* renames `dvcignore` kwarg to `ignore` in `dvc-data` and `dvc-objects`.
* teaches dvc.cli that `IgnoreInCollectedError` is a known exception
and is logged correctly and adds a test for the behaviour.
I am not happy that we need to hardcode `.dvcignore` in dvc-data
but I don't see any easy way, since we always try to avoid
staging if the tree contains .dvcignore file, even if .dvcignore
is not passed.
* Introduce ConfigError and AuthError in dvc.objects.fs and raise it
instead of dvc.config.ConfigError.
* Provide hints and links by catching those errors in dvc.cli.main,
i.e decouple UI messages from filesystems for AuthError and
ConfigError.
* Copy flatten/unflatten utils to dvc.objects.fs.utils.
Now HashedStreamReader will be used to calculate md5 in fobj_md5/hash_file.
Also:
* adds in `bytes_md5` that may be useful for testing.
* adds a hasher-registry with hashlib
* adds a `text=` kwarg to be able to specify to use dos2unix or not
(useful for testing)
* makes `localfs` the default filesystem when `fs` arg is not passed
(makes it easier to test in the REPL)
Note that this will change the behaviour of DVC to always show
prompt even in --quiet mode, which is what we need.
Previously it'd prompt without any message.
It turns out that diskcache.Disk saves passed arguments to the database
and uses that all the time. So this means that we are not forward
compatible as it breaks moving from newer version of DVC to an older
one. We don't promise this, but this breaks our benchmarks which
are probably not isolated enough.
Regarding the use of cache, it is a bit iffy to extend it this way.
We only need this type for error messages. We could consider removing
this, but since the error message is documented and it's not too hard
to extend this :), I decided to preserve the message.
This commit moves `HashedStreamReader`/`get_hash_file`/`file_md5`
into `dvc.objects.hash`.
The `get_hash_file` has been renamed to `hash_file`.
This also copies `relpath` and `is_exec` to the `dvc.fs.utils`.
In the spirit of fsspec.generic, it is for cross-filesystem operations.
Trying to avoid `fs.utils` as much as possible, but we have some
functions that are really utilities coming, so I am making a place
for them. :)
Tqdm supports __bool__, so we should really use `is not None` test
rather than boolean check. This is not used anywhere right now, but
I'd prefer being correct.
There is no support for chunk_size kwarg in fs.open.
Also, `upload_fobj` is now able to figure out proper
chunk size from fileobj, so I don't think we need this on _upload_file.
Before, as we were not passing `total`, the Tqdm progress bar would not
show the bar or not show any progress on the bar. As we get that value lazily,
we should force it to always show the bar.
Also updated the progress bar to say which repo is being cloned.
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
