* apply: allow applying exp with multiple possible branches
* default to new checkpoint branch after applying ambiguous commit
* output shas in addition to exp names by default
* update show tests
* git: allow force checkout in detach_head context
* exp run: error when trying to resume from a non-head checkpoint without
modifications
* add test for apply/resume from intermediate checkpoint
* list some possible heads when multiple matches
* use new table data structure for diff and show
* fix params diff showing changes
* metrics show: use default precision
* skip precision format for exp diff params
* exp diff: separate tables in --quiet mode
* change precision to signify significants digits
changed f specifier to g, so that we show significant digits
instead of fractional digits. This does changes how precision
worked for metrics/params diff/show.
* stage,commit: add support for only committing data stages from
experiments
* exp run: `dvc commit` data deps when stashing exp changes
* add test for modified DVC-tracked data when generating experiment
* test with and without params change
* handle run without target
* objects: merge transfer() with stage()
* objects: drop streamfile
* objects: preserve the signature of get_file_hash
* objects: override hash_jobs if jobs= passed
* objects: support no progress bar directive
* tests: spy on _get_file_hash
* objects: update=... directive
* update: only transfer missing caches with --to-remote
* tests: remove local_remote from the primary test (due to it moves)
* objects: add move=True/False option
* objects: implement _get_file_obj on top of get_file_hash
* objects: use threading on save()
* objects: pass jobs= on to-cache as well
* objects: use in place tqdm
* objects: check for FileExistsError
* Update dvc/objects/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* exp: add util method for getting baseline HEAD during run
* exp: use baseline for dvc commands which compare to HEAD during exp run
* add test for new util method
* update plots diff test mock
This part of logic used to be only used by external repo for cloned
repos, but since we now handle local repos without cloning, we don't
always pass scm instance. This PR automatically creates scm instance
for such cases.
Fixes#5590
* params: read from ParamsDependency if possible
* params: introduce include_untracked flag
* params: include untracked upon targets
* experiments: add option to show tracked params
* fixing test
* params: rename tracked to deps
* params: roll back to default params.yaml behavior
* experiments: add option to show only used params on diff
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/params.py
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* exp show: Allow for wildcard patterns in include and exclude params/metrics
Fixes#5642
* Add test case for nested metrics/params
* match full string in _filter_name
* pre-commit
* Update dvc/command/experiments.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Added SSL verification option
* Changed param name to avoid conflicts
* verify_ssl now accepts strings
* Added simple unit tests
* Changed parameter name fron verify_ssl to ssl_verify so it's consistent with HTTP remotes
Forget to change 1 test
* add processing on ssl_verify param
typo
* revert back to bool param
* Update dvc/config_schema.py
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
This is achieved by using `lock_repo`
context manager that can be used in a recursive depth
that only resets stages when it has left any lock_repo
blocks.
Also added some ui related tests for the status which I found
missing
Before, we used to set it during initialization. But this
made it hard to test using capsys as capsys changes the
reference to sys.stdout and sys.stderr with the mocked one
during the test. But since we are using dvc.ui globally and
setting it to a real sys.stdout and sys.stderr before capsys
comes into effect, testing was not possible with it.
Now, by using property hack, we will be using the values of
sys.stdout and sys.stderr during the time of write, so the mocking
would work and would be easier to test.
This commit fixes 3 things:
1. The path printed on the terminal is properly is quoted. This was
problematic when the output is piped to `xdg-open` if the path contained
spaces or quotes or special characters.
2. The path is resolved. This is not an issue at all, but makes the
output nicer.
3. The path with which the browser is opened with uses relative path.
This is done to solve issue in WSL, that the absolute path is not
accessible outside the WSL for host programs as the linux filesystem
is available on `/wsl$` path.
* diff: clear calculated renamed files from inverted hashmap
We were trying to reuse the first encounter of file from inverted hashmap again and again, which was throwing ValueError
* refactor
* remove ordering
* add tests
* import-url: support importing empty directories on object storage services
Empty directories are basically files with a trailing slash at the end of their names,
with empty contents on object service providers (such as azure, google storage). This
patch implements the `is_empty()` method for the `FSSpecWrapper` to determining whether
a path is empty (directory or a file) and also tweaks `.isdir()` in order to see empty
directories.
The base class is only used by azure and gcsfs for now, but if we go on the route of
migrating normal providers (like proper filesystems, not just object storages), we might
need to do these tweaks on a subclass (like `ObjectStorageFSSpecWrapper` etc), but I
don't think it is neccesary as of now.
The `is_empty` method is already there, and also used in the importing part to
warn users that they are importing something empty (https://github.com/iterative/dvc/blob/e385e59a098cd661ea1ea4257c9a5735c5c32bc6/dvc/output/base.py#L252-L254)
so this is not a new method. Just for the reference, as far as I can locate there
is no variant of this method in the fsspec.
* tests: revert custom mkdir for cloud storages
* tests: allign assingments
* add some comments
* dvc: get rid of DirInfo
DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.
As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.
* objects: move Tree into tree.py
* pygit2: disable set_ref with reflog message
* git: implement is_dirty in dulwich
* pygit2: implement _stash_apply
* pygit: update to 1.5.0
* enable set_ref(message=...) for pygit2
* git: replace backend reflog_delete in favor of _stash_drop
* pygit: support stash_push with custom refs
* update tests for backend stash_drop behavior
* pygit: raise MergeConflictError on failed stash apply
* pygit: reload index from disk before stash apply
* update stash_apply checkout strategy
* mark flaky test
* repro: skip DVC deps in SCM track stage
(DVC deps will be tracked when their own stage is repro'd)
* scm: use git status when reminding to git-track files
* sort names in reminder instead of random set order
* update unit test
* live integration: get rid of dvc from live
* live: remove from api, add test for html generation during run
* stage: run: convert monitors to class context managers
* run: make monitors run in single thread
* run: move monitor logic out of run
Switch from `md5` to `etag` as the checksum parameter. Due to `etag`
is not preserved while doing server side copies, this patch drops support
for external gs outputs.
Paweł Redzyński
Peter Rowlands (변기호)
Ruslan Kuprieiev
Batuhan Taskaya
Saugat Pachhai
David de la Iglesia CastroandPeter Rowlands
dependabot[bot]andSaugat Pachhai
Alex Ford