Now HashedStreamReader will be used to calculate md5 in fobj_md5/hash_file.
Also:
* adds in `bytes_md5` that may be useful for testing.
* adds a hasher-registry with hashlib
* adds a `text=` kwarg to be able to specify to use dos2unix or not
(useful for testing)
* makes `localfs` the default filesystem when `fs` arg is not passed
(makes it easier to test in the REPL)
Note that this will change the behaviour of DVC to always show
prompt even in --quiet mode, which is what we need.
Previously it'd prompt without any message.
It turns out that diskcache.Disk saves passed arguments to the database
and uses that all the time. So this means that we are not forward
compatible as it breaks moving from newer version of DVC to an older
one. We don't promise this, but this breaks our benchmarks which
are probably not isolated enough.
Regarding the use of cache, it is a bit iffy to extend it this way.
We only need this type for error messages. We could consider removing
this, but since the error message is documented and it's not too hard
to extend this :), I decided to preserve the message.
This commit moves `HashedStreamReader`/`get_hash_file`/`file_md5`
into `dvc.objects.hash`.
The `get_hash_file` has been renamed to `hash_file`.
This also copies `relpath` and `is_exec` to the `dvc.fs.utils`.
When staging a directory, always save a "raw dir object" to the odb.
If the corresponding ".dir" object has not been added to the odb,
`stage()` calls can load the tree from the raw dir object instead of
rebuilding it by walking the directory.
This can lead to significant speed improvements when calling
`dvc status` for modified directories.
Fixes#7390
In the spirit of fsspec.generic, it is for cross-filesystem operations.
Trying to avoid `fs.utils` as much as possible, but we have some
functions that are really utilities coming, so I am making a place
for them. :)
Current naming for `DvcFileSystem` and `RepoFileSystem` is confusing, because
dvc is not only about data and it is much more natural to expect something
named `dvcfs` to include complete view for your whole repo (including git
files), which is what `repofs` is doing right now.
`DvcFileSystem` can provide a view not only for the repo data, but also for
external data (e.g. local external outputs). So it makes more sense to call
it a `DataFileSystem`, since it really represents the data management side
of dvc.
Our gitfs paths are currently tied to the local path in which the cloned
repository resides in, which results in a lot of hacky assumptions throughout
the code.
This PR detaches gitfs from local paths completely, making dvc work with a
proper fsspec gitfs implementation.
Part of #7543
Tqdm supports __bool__, so we should really use `is not None` test
rather than boolean check. This is not used anywhere right now, but
I'd prefer being correct.
This makes dvcfs/repofs much easier to work with and allows it to support
isabs/abspath/etc correctly, as previously any full path (with root_marker "")
would be impossible to distinguish from a relative one.
This also detaches external workspaces from the main repo workspace in
DvcFileSystem, which was necessary to be able to distinguish dvcfs abspaths
from local external output paths. Note that currently we still have limited
support for those (e.g. get/api supports them, but import doesn't) and this
change keeps it that way.
