There is no support for chunk_size kwarg in fs.open.
Also, `upload_fobj` is now able to figure out proper
chunk size from fileobj, so I don't think we need this on _upload_file.
Before, as we were not passing `total`, the Tqdm progress bar would not
show the bar or not show any progress on the bar. As we get that value lazily,
we should force it to always show the bar.
Also updated the progress bar to say which repo is being cloned.
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.
The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.
This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.
Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
Also adds a kwarg `cancel_on_error` on the constructor to support
cancelling futures on `shutdown`. This is not done by default to
be backward compatible with `concurrent.futures.ThreadPool` executor.
This removes callback from internal APIs: `_list_paths`, `list_hashes`,
and `_hashes_with_limit`. These are all iterators and hence can just be
replaced by wrapping a progressbar over those iterators in the caller
side.
We still have progressbars in `hashes_exist`, `_estimate_remote_size`,
`_list_hashes_traverse` and `list_hashes_exists`, which should gradually
lifted up and replaced.
Webdav filesystem expects path to be relative
to the base url, instead of having full prefix.
So we are setting it to "" (root) through `_strip_protocol`
which converts fully-qualified name to the file-system
specific path.
- Moves workspace tree for interactive mode after prompts. Previously
this was displayed before the prompts.
- Remove the yaml contents and the confirmation prompt in an interactive
mode.
- Simplify `init_interactive()`.
By splitting dvc.cli into a module, we'll be able to share cli-related
utilities, split parser-related stuffs into smaller parts, improve
reusability and improve API for command extensions in the future.
Before it used to say 'failed to reproduce dvc.yaml', which is now changed
to print 'failed to reproduce train' or something similar.
For stages that are in dvc.yaml in the current working directory, it will
just print the stage names. For stages in other dvc.yaml files, it'll print
in <relative_path>:<stage_name>.
Rich added `vertical` kwarg in `add_column`, which linter
is now complaining (as it should). But this may keep on breaking
when rich adds/removes args/kwargs in the future.
So, this commit uses (*args, **kwargs) instead of specifying particular
parameters explicitly. This also sets `collapse` directly to the
`rich.table.Column` rather than extending through our one `Column`
implementation.
Hoping to fix `dvc[s3]` installations in pipenv. Pipenv's resolver is in loop, backtracking ruamel.yaml from 0.17.17 to 0.17.11 continuously. I think this has to do
more with `s3`'s dependency than `ruamel.yaml` as it only fails here. And, we are specifying dependency in 1.0.1 whereas aiobotocore has released 2.0 versions which is what is being installed, so it's redundant anyway. I am hopeful that this may help resolver with the simplification in constraints.
This commit removes dvc specific message hardcoded in
scm.detach_head. Now that we are decoupling scm, we are
adding `client=` kwarg to be set by the caller.
Also adds a wrapper for DVC to use as GitFileSystem in dvc.fs.scm.
This is temporary, and will be renamed back to dvc.fs.git after the
split.
Note that this filesystem is not fully compliant, it's only
meant to be compatible with our previous implementation.
This removes dependency from dvc.utils and dvc.types from DVC.
Some utils like fix_env, is_binary have been reimplemented.
Regarding fix_env and is_binary, I think eventually this
should be a responsibility of DVC itself to set.
relpath implementation in DVC also works with paths in
different drives in Windows. I have changed this to use
os.path.relpath for now.
This adds a support for reporting progress from Git in a structured
format. The gitpython already provides somewhat structured progress
reporting, dulwich on the other hand only provides git-like line
messages.
As gitpython uses git binary, it already parse this line format
in a structured way. This PR makes dulwich use the same machinery
to unify progress reporting for both dulwich and gitpython python.
This does add a dependency on gitpython even on dulwich backend,
so I am iffy if we should do this.
The alternative is to translate gitpython's progress reports to
string, and just use string messages as progress reports on both
dulwich and gitpython.
Sometimes, `tmp_dir` fixtures take longer than a second. Using `func_only`
makes it only consider duration of test function run, excluding fixtures
duration.
