* redirect warnings module messages to logger to silent them in gdrive
* minor cleanup: remove unused var assignment
* dvc: don't ignore warnings manually
We use logger.captureWarnings() instead for CLI and for API this is harmful.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Makes it easier to use in shell scripts. E.g. `-z $(dvc metrics diff)`, which
has the same semantics as `git diff`. Also syncs up with the behaviour
of `dvc diff`.
Fixes#3469
* remote: ADD sock attribute and pass it to SSHConnection constructor
The sock attribute is set to a `paramiko.ProxyCommand` object
if 'ProxyCommand' exists in config to `None` other-wise
(which is the default option in `paramiko.client.SSHClient.connect()`).
FIXES#3560
* Move paramiko import to RemoteSSH __init__
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
Co-authored-by: Sergio Garcia <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* remote: base: improve exception message
* remote: use `all()` instead of `list_cache_paths()`
`list_cache_paths()` doesn't ignore paths that don't look like hashes,
which makes dvc raise an exception when it tries to convert such paths
to hashes. We should use `all()` instead.
Discord context:
https://discordapp.com/channels/485586884165107732/485596304961962003/694454282719592458
* remote: all: print debug message when skipping paths
This will allow us to see it from verbose logs coming from users.
* remote: use progress bar when paginating
- add `progress_callback` parameter to `list_cache_paths()` so that
remotes can update remote cache traverse progress bar after fetching a
page
* Show remote size estimation status as no-total progress bar
* Update pbar per item rather than per page
* make tests more explicit (don't use mock.ANY)
* Apply suggestions from code review
minor fixes/tidy
* Use progress_callback in ssh/local list_cache_paths()
- keep ssh/local consistent with other remotes (even though `dvc
gc`/`RemoteBASE.all()` do not currently use progress bars)
* Fix review issues
Co-authored-by: Casper da Costa-Luis <[email protected]>
* s3/gs: do not try to make directory on top level
* tests: unittest makedirs for s3 and gs
* remove setattr and use patch.object
* Update tests/unit/remote/test_remote.py
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:
```
angles:
- 2
- 3
- 100
- 3.2
- 36.2
- 3.6
```
because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
* remote: Optimize traverse/no_traverse behavior
- estimate remote file count by fetching a single parent cache dir, then
determine whether or not to use no_traverse method for checking
remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
one parent cache dir per thread)
* Use arbitrary prefix length for traverse remote cache queries
- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)
* Add unit tests for new traverse/no_traverse behavior
* Fix deepsource warning
* Fix review issues
* fix CI
* Fix review issues
* obsolete no_traverse config option
- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
RemoteBASE.CAN_TRAVERSE (True for all remotes except http)
* review issues - use 'object exists' instead of 'no_traverse'
* fix obsolete no_traverse test issues
* fix SSHMocked func tests
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* fix formatting
* dep: param: fix schema
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* run: check if --params is specified
* dvc: rework param handling
* params: fix exc formatting
* params: support basic nested configs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* pull: support pulling git-only imports
* pull: also add git imports from external dvc repo
* fetch: add logging
* pull: don't save link to state
When `checkout` or `pull` is done, it'll try to cleanup unused links.
So, when we add from erepo to cache, it creates a link, which DVC will try
to cleanup. Also, it'll provide unwanted checkout change details. So, skipping those
* tests: refactor/rewrite
* flake8: why it's complaining?
* tests: do not leak file descriptors
* dvc: add Repo._skip_graph_checks flag
Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.
The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.
* dvc: explain skip_graphs_checks flag in a comment
* test: that skip_graph_checks works
* repo: improve hack description and test
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: importing from bare repo
* external_repo: don't try too hard to set upstream repo especially if url is not a dvc repo.
* Update tests/func/test_import.py
* Add a comment regarding bare-repo clone
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* Restyled by black
Co-authored-by: elgehelge <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
Ivan Shcheklein
Casper da Costa-Luis
charlesbaynhamandCharles Baynham
Peter Rowlands (변기호)
Nate Gadzhibalaev
Jorge Orpinel
Max Hora
SdgJlbl
Saugat Pachhai
Paweł Redzyński
Helge Munk JacobsenandRuslan Kuprieiev
Alexander SchepanovskiandRuslan Kuprieiev
Ramón Valles
