Windows (or rather NTFS) doesn't allow deleting read-only files, which made us
try to chmod +w first. But that is not necessery on Linux and Mac, and
has led to unnecessary cache file mode changes, which was causing issues
in shared cache scenario when a user that doesn't own a file was trying
to automatically protect it.
Fixes#3618
Texttable will try to detect the column type automatically, which
results in a low precision of the float that it is showing. We need to
disable that auto-formatting and just use `str(val)` instead.
* redirect warnings module messages to logger to silent them in gdrive
* minor cleanup: remove unused var assignment
* dvc: don't ignore warnings manually
We use logger.captureWarnings() instead for CLI and for API this is harmful.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Makes it easier to use in shell scripts. E.g. `-z $(dvc metrics diff)`, which
has the same semantics as `git diff`. Also syncs up with the behaviour
of `dvc diff`.
Fixes#3469
* remote: ADD sock attribute and pass it to SSHConnection constructor
The sock attribute is set to a `paramiko.ProxyCommand` object
if 'ProxyCommand' exists in config to `None` other-wise
(which is the default option in `paramiko.client.SSHClient.connect()`).
FIXES#3560
* Move paramiko import to RemoteSSH __init__
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
Co-authored-by: Sergio Garcia <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* remote: base: improve exception message
* remote: use `all()` instead of `list_cache_paths()`
`list_cache_paths()` doesn't ignore paths that don't look like hashes,
which makes dvc raise an exception when it tries to convert such paths
to hashes. We should use `all()` instead.
Discord context:
https://discordapp.com/channels/485586884165107732/485596304961962003/694454282719592458
* remote: all: print debug message when skipping paths
This will allow us to see it from verbose logs coming from users.
* remote: use progress bar when paginating
- add `progress_callback` parameter to `list_cache_paths()` so that
remotes can update remote cache traverse progress bar after fetching a
page
* Show remote size estimation status as no-total progress bar
* Update pbar per item rather than per page
* make tests more explicit (don't use mock.ANY)
* Apply suggestions from code review
minor fixes/tidy
* Use progress_callback in ssh/local list_cache_paths()
- keep ssh/local consistent with other remotes (even though `dvc
gc`/`RemoteBASE.all()` do not currently use progress bars)
* Fix review issues
Co-authored-by: Casper da Costa-Luis <[email protected]>
* s3/gs: do not try to make directory on top level
* tests: unittest makedirs for s3 and gs
* remove setattr and use patch.object
* Update tests/unit/remote/test_remote.py
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:
```
angles:
- 2
- 3
- 100
- 3.2
- 36.2
- 3.6
```
because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
* remote: Optimize traverse/no_traverse behavior
- estimate remote file count by fetching a single parent cache dir, then
determine whether or not to use no_traverse method for checking
remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
one parent cache dir per thread)
* Use arbitrary prefix length for traverse remote cache queries
- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)
* Add unit tests for new traverse/no_traverse behavior
* Fix deepsource warning
* Fix review issues
* fix CI
* Fix review issues
* obsolete no_traverse config option
- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
RemoteBASE.CAN_TRAVERSE (True for all remotes except http)
* review issues - use 'object exists' instead of 'no_traverse'
* fix obsolete no_traverse test issues
* fix SSHMocked func tests
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* fix formatting
* dep: param: fix schema
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* run: check if --params is specified
* dvc: rework param handling
* params: fix exc formatting
* params: support basic nested configs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev
Casper da Costa-Luis
Jorge Orpinel
Peter Rowlands (변기호)
Ivan Shcheklein
charlesbaynhamandCharles Baynham
Nate Gadzhibalaev
Max Hora
SdgJlbl
Saugat Pachhai
Paweł Redzyński
Helge Munk JacobsenandRuslan Kuprieiev