* machine: add `machine list`&`machine modify`
1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands
* Update dvc/config_schema.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update tests/func/machine/test_machine_config.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some reviewed problems
1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* fs.download: add callback support
* fix lint issues
* compat for gdrive, use fs to upload text for tests
* fix typing issues
* skip gdrive test
* Update tests/func/test_fs.py
* fs.upload: add support for optional callback
Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Show csv format of experiments
fix#5989 and #5446
1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Clean up remotes's exps
fix#6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
* Added refresh tag
* Formatted
* Added refresh_seconds to live
* Added refresh test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add refresh_seconds to render
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update html.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Remove all of the experiments in the workspace
fix#5676
1. remove all the experiments in workspace
2. add tests for it
* Only remove baseline ones
* Add a new arguments --all
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Rename and shortcut for these commands
* Solve the issue from removing a changing list.
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* delete workspace option
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.
Kudos @skshetry
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
* Fix merging of params when empty
Recent release of python-benedict==0.24.1 started ignoring empty dictionaries
during merge. The way we expected to work was questionable and also how
benedict itself tries to handle the references to empty dictionary is
also questionable.
I consider this to be a proper way to handle this even if this gets fixed
on the benedict side.
* Revert "setup: limit python-benedict version (#6377)"
This reverts commit 15de826266.
* add comment for why passing through benedict
* objects: migrate remote status to objects.status
* data_cloud: use objects.status
* objects: migrate remote._process to objects.transfer
* data_cloud: use objects.transfer
* objects: handle memfs staging objects in transfer
* fetch/import: use unified transfer()
* update get_remote and transfer exception usage
* update tests
* objects.transfer: use src ODB verify rules after xfer
* odb: move index from remote into odb
* objects.transfer: skip src status check when possible
* update tests
* remove dvc.remote
* odb.index: migrate to diskcache
* objects: fix status/transfer optimization
* update index usage
* state: remove dead sqlite related code
* update cloud tests
* drop unnecessary remove in tests
* use abstract base class in odb index
* status: load trees from local cache instead of remote odb when possible
* handle status when dir cache is explicitly removed
* params/metrics/plots/exp show: introduce onerror
* fixup
* plots/metrics/params: removing errors
* removing more errors
* removing errors
* some more removing
* new plots ux
* new ux for metrics
* new ux for params
* fixing tests
* fixups
* minor test fixes
* minor test fixes
* commands: plots/params: fix ui imports
* working on error handlng decorator, stopped at show checkpoints
* error_handler, further dvelopment
* plots just before moving loading to collect
* appending errors to results: initial passing
* fixup
* onerror: remove onerror fixture
* fixups
* plots/metrics/params/experiments: temporarily remove error warnings
* plots: extract parsing out of class method
* metrics/experiments: provide exception serializer for show-json
* params/plots/metrics: signal errors
* rebase fixups
* fix TODOs
* fixup
* skshetry's review refactor
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/objects/db/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* bump into ossfs 2021.7.2
* Bump into ossfs 2021.7.3OC
* Switch to real_oss_server
* Switch to a real oss server
* Use the oss bucket located in US.
* Remove some emulator related code
* Add envs to ci
* remove some oss pytest XFAIL
Because we didn't use emulator, we can remove some pytest XFAILs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* fs: use fsspecwrapper in memfs
* odb: add memfs staging ODB in ODBManager
* odb: add read_only attribute for odb instances
* objects: handle memfs paths when staging trees
* odb: remove git ODB
* output: use staging ODB
* RepoDependency: use staging odb for imports
* dvcfs: handle case where dir cache is unavailable
* fix circular import check
* odb: use temp local ODB for staging
* objects: use staging ODB in stage() and tree load
* objects: add typing for add/check/load/stage
* output/dep: use new staging
* add unit tests for local odb staging
* localfs: account for string paths in move()
* include local in odb.by_scheme
* tests: count dir cache like any other object in test_pull_git_imports
* odb: clear staging on gc()
* objects: add tree.get/filter/insert
* output: use obj.filter/get for granular dir commit
* squash filter logic output
* dependency: don't filter imports at the dep level
* don't init staging ODB for external cache
* use staging ODB when computing tree hash for external outs
* remove unneeded force in test granular commit
* use unique-per-manager memfs URLs for staging
* odb: make staging per-ODBManager
* objects: move state dependent staging into ODBManager
* objects: support checking multiple odbs
- staging dependent checks can be done via ODBManager.check
* checkout: do state lookup before explicit stage() in _changed()
* output: use odbmanager based staging
* RepoDependency: use odbmanager based staging
* update obj staging unit tests
* checkout: remove check(), handle individual file exceptions on link()
* diff: use odbmanager based staging
* objects: add ObjectPermissionError for read/write perm errors
* odb: move staging out of odbmanager
* objects: handle staging ODB in objects.stage()
* objects.tree: don't stage anything in tree methods
* odb: clear staged objects on gc()
* remote: don't filter objs by scheme, objs are already grouped by ODB
* diff: update objects.stage usage
* output/dep: update stage() usage
* update ODB tests
* update func tests
* remove memfs from FS_MAP
* revert unneeded objects.check() changes
* remove unused tree.insert()
* revert unused odbmanager changes
* add separate odb error classes
* stage external outs directly in ODB
* check for obj existence in main ODB before staging
* use .dvc/tmp/staging instead of per-odb staging
* gc: only gc loal staging once
* Fixed diff summary to only show states with values.
* Removed redundant comma by using join.
* Swithced for loop to generator for better readability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Nikita Kodenko
David de la Iglesia Castro
Batuhan Taskaya
Saugat Pachhai
GaoandPeter Rowlands
Casper da Costa-LuisandPeter Rowlands
Ruslan Kuprieiev
Paweł Redzyński
dependabot[bot]andPeter Rowlands
Batuhan Taskaya
Christian Clauss
Hadas Arikandpre-commit-ci[bot]