* add: -f arg name from FILE to <filename>
to match dvc.org cmd ref
* dvc: cmd arg names (metavars) to match docs
for #3626
* metrics: update help output to indicate supported types
per #3409
and for #3572
* dvc: reorder subcommands to match docs
for #3608
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.
It is available via a hidden flag: -n.
Fixes#1871
PR: #3584
This is useful as it is in `dvc ls` implementation, but this is also a
prerequisite for `RepoTree` that will combine git(workspace) files and
dvc files into one tree, allowing things like `dvc metrics` and
`external_repo` to have a universal interface to access whatever
file/dir they want without cumbersome git/dvc path detection.
Using it to load dirs takes around 50 sec compared to 0.4 with simple
`replace()`. Path objects are great, but not when you need lots of them.
Related to #3635
* remote config validation
Fixes#3552
* solving config test fail
* Validation problem in config merging
sloving validation problem in config merging from different levels.
Only those config in lower levels can affect the validation process.
* reduce Cognitive Complexity
* Two request changes
1. Duplicate load current level of config
2. rewrite test to pytest style
* black it
* config: don't use all_levels
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* repo: separate dir cache and file cache in memory
- `used_cache()`/`get_used_cache()` in repo/stage/output now return
tuples of (dir_cache, file_cache) instead of one flat/merged cache
* update tests for new get_used_cache behavior
* remote: if .dir checksum exists on remote, assume contents also exists
- affects all commands which use `cache_exists()` (remote status)
* push: only upload .dir file after full file contents has been uploaded
* gc: always remove .dir checksums first
* functional tests for push/gc
* repo: support nesting dir caches in NamedCache
* remote: NamedCache updates
* Fix tests
* Fix deepsource warnings
* tests: use pytest mocker fixture
* Update dvc/cache.py
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Current logic is extremely complex and messy, hiding a lot of bugs
inside. There are pretty much only 2 scenarios for stage creation:
1) `dvc add/import/etc` when dvcfile is created beside the output file
2) `dvc run` when dvcfile is created at the place of execution
both are transparently affected by `--fname` flag. Since we've removed
`--cwd` complexity already, there is no reason to generalize stage path
resolution for those two scenarios.
Windows (or rather NTFS) doesn't allow deleting read-only files, which made us
try to chmod +w first. But that is not necessery on Linux and Mac, and
has led to unnecessary cache file mode changes, which was causing issues
in shared cache scenario when a user that doesn't own a file was trying
to automatically protect it.
Fixes#3618
Texttable will try to detect the column type automatically, which
results in a low precision of the float that it is showing. We need to
disable that auto-formatting and just use `str(val)` instead.
* redirect warnings module messages to logger to silent them in gdrive
* minor cleanup: remove unused var assignment
* dvc: don't ignore warnings manually
We use logger.captureWarnings() instead for CLI and for API this is harmful.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Makes it easier to use in shell scripts. E.g. `-z $(dvc metrics diff)`, which
has the same semantics as `git diff`. Also syncs up with the behaviour
of `dvc diff`.
Fixes#3469
* remote: ADD sock attribute and pass it to SSHConnection constructor
The sock attribute is set to a `paramiko.ProxyCommand` object
if 'ProxyCommand' exists in config to `None` other-wise
(which is the default option in `paramiko.client.SSHClient.connect()`).
FIXES#3560
* Move paramiko import to RemoteSSH __init__
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
* Update dvc/remote/ssh/__init__.py
Co-authored-by: Sergio Garcia <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* remote: base: improve exception message
* remote: use `all()` instead of `list_cache_paths()`
`list_cache_paths()` doesn't ignore paths that don't look like hashes,
which makes dvc raise an exception when it tries to convert such paths
to hashes. We should use `all()` instead.
Discord context:
https://discordapp.com/channels/485586884165107732/485596304961962003/694454282719592458
* remote: all: print debug message when skipping paths
This will allow us to see it from verbose logs coming from users.
* remote: use progress bar when paginating
- add `progress_callback` parameter to `list_cache_paths()` so that
remotes can update remote cache traverse progress bar after fetching a
page
* Show remote size estimation status as no-total progress bar
* Update pbar per item rather than per page
* make tests more explicit (don't use mock.ANY)
* Apply suggestions from code review
minor fixes/tidy
* Use progress_callback in ssh/local list_cache_paths()
- keep ssh/local consistent with other remotes (even though `dvc
gc`/`RemoteBASE.all()` do not currently use progress bars)
* Fix review issues
Co-authored-by: Casper da Costa-Luis <[email protected]>
Jorge Orpinel
Casper da Costa-Luis
Saugat Pachhai
Ruslan Kuprieiev
Marcel Ribeiro Dantas
Ivan Shcheklein
charlesbaynhamandCharles Baynham
Nate Gadzhibalaev
Max Hora
SdgJlbl