* remote: use progress bar when paginating
- add `progress_callback` parameter to `list_cache_paths()` so that
remotes can update remote cache traverse progress bar after fetching a
page
* Show remote size estimation status as no-total progress bar
* Update pbar per item rather than per page
* make tests more explicit (don't use mock.ANY)
* Apply suggestions from code review
minor fixes/tidy
* Use progress_callback in ssh/local list_cache_paths()
- keep ssh/local consistent with other remotes (even though `dvc
gc`/`RemoteBASE.all()` do not currently use progress bars)
* Fix review issues
Co-authored-by: Casper da Costa-Luis <[email protected]>
* s3/gs: do not try to make directory on top level
* tests: unittest makedirs for s3 and gs
* remove setattr and use patch.object
* Update tests/unit/remote/test_remote.py
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:
```
angles:
- 2
- 3
- 100
- 3.2
- 36.2
- 3.6
```
because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
* remote: Optimize traverse/no_traverse behavior
- estimate remote file count by fetching a single parent cache dir, then
determine whether or not to use no_traverse method for checking
remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
one parent cache dir per thread)
* Use arbitrary prefix length for traverse remote cache queries
- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)
* Add unit tests for new traverse/no_traverse behavior
* Fix deepsource warning
* Fix review issues
* fix CI
* Fix review issues
* obsolete no_traverse config option
- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
RemoteBASE.CAN_TRAVERSE (True for all remotes except http)
* review issues - use 'object exists' instead of 'no_traverse'
* fix obsolete no_traverse test issues
* fix SSHMocked func tests
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* fix formatting
* dep: param: fix schema
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* run: check if --params is specified
* dvc: rework param handling
* params: fix exc formatting
* params: support basic nested configs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: add Repo._skip_graph_checks flag
Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.
The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.
* dvc: explain skip_graphs_checks flag in a comment
* test: that skip_graph_checks works
* repo: improve hack description and test
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* Restyled by black
Co-authored-by: elgehelge <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
* checkout: implement --show-changes flag to show detailed checkout details
Show summary by default on checkout
checkout: show directory modified even when files are removed from inside it
checkout: refactor template logic
checkout: change output format
checkout: do not show summary if nothing's changed
checkout: do not show summary if show_changes is specified
checkout: show '/' at the end if it's a directory
rename variable name
test: cover stats for checkout
checkout: show relink as not modified
checkout: remove unwanted message during empty checkout
pull: show summary on checkout
import/get: change external_repo to catch CheckoutError
fix tests and check if path is url
checkout: support external deps/outputs
tests: checkout on external outputs
* fix tests
* checkout: make show changes default and introduce summary flag to show summary
* address suggestions
* exceptions: remove extra spacing on erorr
* Update dvc/command/checkout.py
Co-Authored-By: Jorge Orpinel <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* remote.http: support _upload() via HTTP POST
- uploaded files are sent as chunked encoding POST data
Fixes#3247
* remote.http: add support for HTTP basic and digest auth
- username, password, ask_password options work the same way as for
ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
remotes
- digest_auth takes precedence over basic_auth if both are enabled
* remote.http: auth_method() unit tests
* tests: Add functional tests for HTTP remotes
- HTTP remotes now tested locally using a SimpleHTTPServer instance that
allows reading/writing to a temp directory
* tests: fix http tests on py3.5, 3.6
* remote.http: add custom auth method
* s/custom_header/custom_auth_header/
* Apply suggestions from code review
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
* add: auto convert absolute_path => relative
- fixes#2955
* use path_isin
* catch-all absolute paths in repo
* unneeded leftover
* revert realtive imports for later
* output: move abs2relpath from Base to Local
* output:local:fix parse_path
* remove path_info
* test:local versus base abs2relpath
* test:fix remaining abs2relpath test
* output: local path relative to wdir
* output: local: modify path before inheritance
* output: local: minor inheritance fix
* test: local: split output test
* test: local: positively explicit
Peter Rowlands (변기호)andCasper da Costa-Luis
Saugat Pachhai
Ruslan Kuprieiev
Helge Munk JacobsenandRuslan Kuprieiev
Alexander SchepanovskiandRuslan Kuprieiev
Ramón Valles
Max Risuhin
Guro Bokum
Ivan Shcheklein
Paweł Redzyński
Fábio Santos