In the spirit of fsspec.generic, it is for cross-filesystem operations.
Trying to avoid `fs.utils` as much as possible, but we have some
functions that are really utilities coming, so I am making a place
for them. :)
Tqdm supports __bool__, so we should really use `is not None` test
rather than boolean check. This is not used anywhere right now, but
I'd prefer being correct.
There is no support for chunk_size kwarg in fs.open.
Also, `upload_fobj` is now able to figure out proper
chunk size from fileobj, so I don't think we need this on _upload_file.
Before, as we were not passing `total`, the Tqdm progress bar would not
show the bar or not show any progress on the bar. As we get that value lazily,
we should force it to always show the bar.
Also updated the progress bar to say which repo is being cloned.
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.
Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
Also adds a kwarg `cancel_on_error` on the constructor to support
cancelling futures on `shutdown`. This is not done by default to
be backward compatible with `concurrent.futures.ThreadPool` executor.
This removes callback from internal APIs: `_list_paths`, `list_hashes`,
and `_hashes_with_limit`. These are all iterators and hence can just be
replaced by wrapping a progressbar over those iterators in the caller
side.
We still have progressbars in `hashes_exist`, `_estimate_remote_size`,
`_list_hashes_traverse` and `list_hashes_exists`, which should gradually
lifted up and replaced.
Webdav filesystem expects path to be relative
to the base url, instead of having full prefix.
So we are setting it to "" (root) through `_strip_protocol`
which converts fully-qualified name to the file-system
specific path.
- Moves workspace tree for interactive mode after prompts. Previously
this was displayed before the prompts.
- Remove the yaml contents and the confirmation prompt in an interactive
mode.
- Simplify `init_interactive()`.
By splitting dvc.cli into a module, we'll be able to share cli-related
utilities, split parser-related stuffs into smaller parts, improve
reusability and improve API for command extensions in the future.
Before it used to say 'failed to reproduce dvc.yaml', which is now changed
to print 'failed to reproduce train' or something similar.
For stages that are in dvc.yaml in the current working directory, it will
just print the stage names. For stages in other dvc.yaml files, it'll print
in <relative_path>:<stage_name>.
Rich added `vertical` kwarg in `add_column`, which linter
is now complaining (as it should). But this may keep on breaking
when rich adds/removes args/kwargs in the future.
So, this commit uses (*args, **kwargs) instead of specifying particular
parameters explicitly. This also sets `collapse` directly to the
`rich.table.Column` rather than extending through our one `Column`
implementation.
Hoping to fix `dvc[s3]` installations in pipenv. Pipenv's resolver is in loop, backtracking ruamel.yaml from 0.17.17 to 0.17.11 continuously. I think this has to do
more with `s3`'s dependency than `ruamel.yaml` as it only fails here. And, we are specifying dependency in 1.0.1 whereas aiobotocore has released 2.0 versions which is what is being installed, so it's redundant anyway. I am hopeful that this may help resolver with the simplification in constraints.
This commit removes dvc specific message hardcoded in
scm.detach_head. Now that we are decoupling scm, we are
adding `client=` kwarg to be set by the caller.
