1. update diff.py,experiments.py,metrics.py,params.py,status.py to
print prettify json
2. rewrite write_json in __init__.py to accept default function
3. add new unit test for write_json in tests/unit/ui/test_console.py
4. Fix#6712
Co-authored-by: Zhenyu Gao <[email protected]>
This was used before on opening http urls as a stream,
but we no longer use those (as we use fsspec's open method).
One thing to note however is that, compared to previous
implementation, fsspec's http filesystem's open is not very
robust to failures.
* rebased PR 6111 onto master
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use make_tmp_dir fixture in tests
Co-authored-by: U119511 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Alexander Meier <[email protected]>
Co-authored-by: meierale <[email protected]>
* machine: add `machine list`&`machine modify`
1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands
* Update dvc/config_schema.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update tests/func/machine/test_machine_config.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some reviewed problems
1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* fs.download: add callback support
* fix lint issues
* compat for gdrive, use fs to upload text for tests
* fix typing issues
* skip gdrive test
* Update tests/func/test_fs.py
* fs.upload: add support for optional callback
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Show csv format of experiments
fix#5989 and #5446
1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Clean up remotes's exps
fix#6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
* Added refresh tag
* Formatted
* Added refresh_seconds to live
* Added refresh test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add refresh_seconds to render
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update html.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Remove all of the experiments in the workspace
fix#5676
1. remove all the experiments in workspace
2. add tests for it
* Only remove baseline ones
* Add a new arguments --all
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Rename and shortcut for these commands
* Solve the issue from removing a changing list.
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* delete workspace option
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.
Kudos @skshetry
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
* Fix merging of params when empty
Recent release of python-benedict==0.24.1 started ignoring empty dictionaries
during merge. The way we expected to work was questionable and also how
benedict itself tries to handle the references to empty dictionary is
also questionable.
I consider this to be a proper way to handle this even if this gets fixed
on the benedict side.
* Revert "setup: limit python-benedict version (#6377)"
This reverts commit 15de826266.
* add comment for why passing through benedict
* objects: migrate remote status to objects.status
* data_cloud: use objects.status
* objects: migrate remote._process to objects.transfer
* data_cloud: use objects.transfer
* objects: handle memfs staging objects in transfer
* fetch/import: use unified transfer()
* update get_remote and transfer exception usage
* update tests
* objects.transfer: use src ODB verify rules after xfer
* odb: move index from remote into odb
* objects.transfer: skip src status check when possible
* update tests
* remove dvc.remote
* odb.index: migrate to diskcache
* objects: fix status/transfer optimization
* update index usage
* state: remove dead sqlite related code
* update cloud tests
* drop unnecessary remove in tests
* use abstract base class in odb index
* status: load trees from local cache instead of remote odb when possible
* handle status when dir cache is explicitly removed
Saugat Pachhai
Batuhan Taskaya
eggqq007andZhenyu Gao
Amey Kulkarni
Nikita Kodenko
David de la Iglesia Castro
GaoandPeter Rowlands
Casper da Costa-LuisandPeter Rowlands
Ruslan Kuprieiev
Paweł Redzyński
dependabot[bot]andPeter Rowlands
Batuhan Taskaya