Commit Graph
7789 Commits
Author SHA1 Message Date
Saugat Pachhai fd76e0991f cleanup index, and add test for used_objs (#6390) 2021-08-06 15:35:01 +05:45
Gao 3d604a8455 Show name of experiments in the queue (#6382)
* Show name of stashed experiments

* Add tests for it

* Reorder the stash and commit rev name
2021-08-06 11:02:16 +09:00
Saugat Pachhai 6c33fbb617 introduce index (#6300)
* introduce index

* migrate tests to use index

* migrate checkout/diff to use index.outs

* test for index

* fix typings

* add few more tests
2021-08-05 14:23:40 +03:00
Nikita KodenkoandPeter Rowlands 3be395b3a7 Unittest to pytest: moved few unittest tests to pytest (#6379)
* tests: rm TestGit in func/test_init.py

* tests: func: rm TestSCMGit in test_scm.py

* test: func: rm _check_dirs in test_init.py

* tests: func: minor fixes in test_scm.py

* Update tests/func/test_scm.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-08-04 15:11:09 +03:00
Peter Rowlands (변기호) 114a07e7a4 objects: use object IDs and references instead of naive objs in status/transfer (#6360)
* objects.status: use object IDs instead of naive objects

* objects.transfer: use object IDs instead of naive objects

* objects: add ReferenceHashFile

* odb: add ReferenceObjectDB

* repo: collect used object IDs instead of naive objs

* support basic/naive fs serialization

* objects.stage: stage file objects in addition to trees

* objects: remove save(), update transfer()

* output/dep: update for osave removal

* path_info: support both url and path attributes

* status/gc: use object IDs instead of naive objs

* diff: update staging usage

* commit: update staging usage

* tests: clean staging when needed

* tests: update for save/transfer/stage changes

* dvcfs: don't allow config serialization (repofs should be used instead)

* utils: return mtime as str, size as int

* tests: update unit tests for save/stage/transfer changes

* objects: use Repo.open when needed in ReferenceHashFile

* fix ls test staging

* objects.stage: don't skip hash check when loading from state

- individual hashes will still be loaded from state if they are still
  valid

* ReferenceHashFile: serialize repo fs paths relative to repo root

* preserve state when staging

* objects.tree: skip hash computation in digest() when we have already
loaded hash from state

* always check file existence in referencehashfile

* update repofs test since stage no longer allows nonexistent files

* odb: only protect on check() if hash was verified

* staging: use unique memfs namespace for staging per dest ODB

* tests: clean full staging namespace

* refodb: preserve repofs cache dir

* repodependency: stage imports to local ODB and strict check circular
imports

* fetch: fetch staging last

* staging: make dest odb mandatory and preserve state

* refodb: cache nonlocal fs's

* fix rebase errors

* repofs: support full instantiation from fs config

* repofs: preserve subrepo factory for erepos

* tests: clarify repofs dirty file behavior
2021-08-04 13:55:52 +03:00
Peter Rowlands (변기호) 34697ef298 exp gc: fix --queued warning message (#6381) 2021-08-03 10:37:38 +00:00
Saugat Pachhai 1eccfa9d33 benedict: cleanup code and add tests (#6380) 2021-08-03 12:28:42 +05:45
dependabot[bot]andPeter Rowlands 6c14bf39dc build(deps): bump pylint from 2.8.3 to 2.9.6 (#6369)
* build(deps): bump pylint from 2.8.3 to 2.9.6

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.8.3 to 2.9.6.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.8.3...v2.9.6)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* pylint: fix 2.9.6 issues

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Peter Rowlands <[email protected]>
2021-08-03 10:29:55 +09:00
Saugat Pachhai 7444696dcd Fix merging of params when empty (#6378)
* Fix merging of params when empty

Recent release of python-benedict==0.24.1 started ignoring empty dictionaries
during merge. The way we expected to work was questionable and also how
benedict itself tries to handle the references to empty dictionary is
also questionable.

I consider this to be a proper way to handle this even if this gets fixed
on the benedict side.

* Revert "setup: limit python-benedict version (#6377)"

This reverts commit 15de826266.

* add comment for why passing through benedict
2021-08-02 22:45:04 +03:00
pre-commit-ci[bot] 76be53bc3e [pre-commit.ci] pre-commit autoupdate (#6376)
updates:
- [github.com/timothycrosley/isort: 5.9.2 → 5.9.3](https://github.com/timothycrosley/isort/compare/5.9.2...5.9.3)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-02 21:46:35 +03:00
Ruslan Kuprieiev 8d71c3861f fs: raise RemoteNotFoundError (#6375) 2021-08-02 21:09:55 +03:00
Ruslan Kuprieiev 15de826266 setup: limit python-benedict version (#6377)
Temporary fix for #6374 Kudos @skshetry
2021-08-02 20:44:03 +03:00
Batuhan Taskaya 9634efeb84 ssh: handle concurrency issues for raw command execution (#6370)
* ssh: handle concurrency issues for raw command execution

* setup: bump sshfs
2021-07-29 14:40:49 +03:00
Ruslan Kuprieiev e3a541aa00 setup: relax aiobotocore requirements (#6367)
Fixes #6366
2021-07-28 16:19:16 +03:00
Batuhan Taskaya 9b4b6e5b31 ssh: migrate to sshfs (#6295) 2021-07-28 12:49:53 +03:00
dependabot[bot] b294bca383 build(deps): bump codecov/codecov-action from 2.0.1 to 2.0.2 (#6361)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.1...v2.0.2)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-28 10:21:10 +03:00
Batuhan Taskaya 4a843dbd61 odb: use regular upload() when the source filesystem is local (#6365)
* odb: use regular upload() when the source filesystem is local

* support download_file too
2021-07-27 15:55:49 +03:00
Ruslan Kuprieiev 1e43c303a7 output: cache tree objects (#6301) 2021-07-24 23:07:43 +03:00
meieraleandAlexander Meier 985c0328c7 remote: add excludes for AzureDefaultCredentials (#6359)
add additional excludes to DefaultAzureCredential
which are needed in a Windows Environment to
use az cli provided credentials, etc.

Co-authored-by: Alexander Meier <[email protected]>
2021-07-23 16:02:16 +03:00
Batuhan Taskaya d5722e3edf fsspec: use rm_file() (#6358) 2021-07-23 14:10:16 +03:00
Paweł Redzyński 67b42ebe4f plots: cleanup data extraction (#6355)
* plots: cleanup data extraction

* fixup
2021-07-23 10:28:36 +09:00
Batuhan Taskaya 1d7e754044 setup: bump flatten-dict to 0.4.1 (#6357) 2021-07-22 21:46:18 +03:00
Batuhan Taskaya c8739008d1 fsspec: implement callback mixins (#6322)
* fsspec: implement callback mixins

* drop redundant refresh() calls
2021-07-22 18:42:28 +03:00
Nikita KodenkoandSaugat Pachhai e731e35e6c tests: moved few unittest tests to pytest (#6339)
* tests: moved few unittest tests to pytest

* tests: unittest.mock -> pytest-mock in few tests

* tests: test_prompt: unittest.mock -> pytest-mock

* Update tests/unit/test_daemon.py

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
2021-07-22 18:06:54 +03:00
Peter Rowlands (변기호) cd7c95a785 ci: temporarily mark gdrive xfail (#6348) 2021-07-22 11:14:54 +03:00
dependabot[bot] 771b89ddf2 build(deps): bump google-cloud-storage from 1.41.0 to 1.41.1 (#6341)
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.41.0 to 1.41.1.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.41.0...v1.41.1)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-22 00:42:07 +03:00
Saugat Pachhai 64587e8b6d do not print in style by default for errors (#6344) 2021-07-21 14:42:29 +03:00
Saugat Pachhai 5faf9b3a7d update webdav4 to latest fsspec version (#6343) 2021-07-21 10:17:14 +00:00
Peter Rowlands (변기호) 12d1e3e981 objects: fix gdrive CI issue (#6338)
* fs: ensure gdrivefs.info returns integer size

* transfer: only verify on pull
2021-07-20 18:54:49 +03:00
pre-commit-ci[bot] d95bd62f8e [pre-commit.ci] pre-commit autoupdate (#6333)
updates:
- [github.com/ambv/black: 21.6b0 → 21.7b0](https://github.com/ambv/black/compare/21.6b0...21.7b0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-07-20 14:12:05 +03:00
dependabot[bot] 005bca1da6 build(deps): bump codecov/codecov-action from 1.5.2 to 2.0.1 (#6334)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1.5.2 to 2.0.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.5.2...v2.0.1)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-20 14:06:50 +03:00
Peter Rowlands (변기호) 3d84df930b objects: migrate remote push/pull to objects.transfer (#6308)
* objects: migrate remote status to objects.status

* data_cloud: use objects.status

* objects: migrate remote._process to objects.transfer

* data_cloud: use objects.transfer

* objects: handle memfs staging objects in transfer

* fetch/import: use unified transfer()

* update get_remote and transfer exception usage

* update tests

* objects.transfer: use src ODB verify rules after xfer

* odb: move index from remote into odb

* objects.transfer: skip src status check when possible

* update tests

* remove dvc.remote

* odb.index: migrate to diskcache

* objects: fix status/transfer optimization

* update index usage

* state: remove dead sqlite related code

* update cloud tests

* drop unnecessary remove in tests

* use abstract base class in odb index

* status: load trees from local cache instead of remote odb when possible

* handle status when dir cache is explicitly removed
2021-07-20 13:30:01 +03:00
Batuhan Taskaya 66d0df2d12 setup: upgrade adlfs to 2021.7.1 (#6337)
The previous release got yanked on the PyPI, due to it containing backwards incompatible changes.
2021-07-20 08:59:45 +00:00
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
dependabot[bot] 6711b647d9 build(deps): bump google-cloud-storage from 1.40.0 to 1.41.0 (#6311)
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.40.0 to 1.41.0.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.40.0...v1.41.0)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-19 00:41:23 +03:00
Batuhan Taskaya 1e81983d6d import-url: use the same chunk size for istextblock() calls (#6303) 2021-07-19 00:28:42 +03:00
Ruslan Kuprieiev 97809661bd setup: pathspec <0.9.0 (#6331) 2021-07-18 18:24:28 +03:00
Batuhan Taskaya b229588df3 azure: move to default fsspec loop (#6323) 2021-07-18 17:47:20 +03:00
Sayanta Banerjee cae117cfd4 Update README.rst (#6330)
* Update README.rst

More clean and simple

* Update README.rst

Rectification from the previous pull
2021-07-16 23:44:30 +03:00
Paweł Redzyński 21af7438db plots/metrics/params/experiments show: stop throwing exceptions (#5984)
* params/metrics/plots/exp show: introduce onerror

* fixup

* plots/metrics/params: removing errors

* removing more errors

* removing errors

* some more removing

* new plots ux

* new ux for metrics

* new ux for params

* fixing tests

* fixups

* minor test fixes

* minor test fixes

* commands: plots/params: fix ui imports

* working on error handlng decorator, stopped at show checkpoints

* error_handler, further dvelopment

* plots just before moving loading to collect

* appending errors to results: initial passing

* fixup

* onerror: remove onerror fixture

* fixups

* plots/metrics/params/experiments: temporarily remove error warnings

* plots: extract parsing out of class method

* metrics/experiments: provide exception serializer for show-json

* params/plots/metrics: signal errors

* rebase fixups

* fix TODOs

* fixup

* skshetry's review refactor
2021-07-16 15:36:04 +02:00
Batuhan Taskaya 27c1722b90 setup: bump ossfs==2021.7.5 (#6326) 2021-07-16 20:48:57 +08:00
Batuhan Taskaya 6c8673b158 setup: upgrade fsspec/s3fs/gcsfs to 2021.7.0 (#6317) 2021-07-15 11:33:07 +02:00
Batuhan Taskaya 6275f5d0d7 fsspec: create directories on upload/upload_fobj (#6309) 2021-07-14 15:42:51 +03:00
GaoandRuslan Kuprieiev 93f36d9351 oss: migrate to ossfs (#6307)
* Migrate to ossfs

fix 6177

* Add verification to object stored on oss

* Change the oss remote's dependency.

pass all of the tests

* Performance optimaztion and ossfs version update

* Remove the argument to set the resumable get/put

1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default

* Use a new api for performance optimization

* Without filesize progress bar version

* With progress bar version

* Bump and add comments

1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger

* Change the name of kwrags following parent class.

* Bump to new ossfs and rewrite upload_fobj

Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj

* Optimization on rm_files

* Some optimization follow the old version.

* Update dvc/fs/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/objects/db/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* bump into ossfs 2021.7.2

* Bump into ossfs 2021.7.3OC

* Switch to real_oss_server

* Switch to a real oss server

* Use the oss bucket located in US.

* Remove some emulator related code

* Add envs to ci

* remove some oss pytest XFAIL

Because we didn't use emulator, we can remove some pytest XFAILs

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-07-13 15:27:34 +03:00
Batuhan Taskaya 58f0911234 azure: support directory imports with azure (#6245)
* azure: support directory imports with azure

* upgrade azurite to 3.13

* tmp: use azurite 3.12?

* upgrade to azurite 3.13.1

* bump adlfs to 2021.7.0
2021-07-13 13:39:30 +03:00
pre-commit-ci[bot] f45311e658 [pre-commit.ci] pre-commit autoupdate (#6306)
updates:
- [github.com/timothycrosley/isort: 5.9.1 → 5.9.2](https://github.com/timothycrosley/isort/compare/5.9.1...5.9.2)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-07-13 08:49:27 +05:45
Peter Rowlands (변기호) 1271db6b59 exp show: display running/queued state for experiments (#6174)
* exp: allow fetching from arbitrary executor by git URL

* exp: write info for running executors to .dvc/tmp/exp/run

currently only includes PID and git fetch URL

* exp: refactor stash/queue cleanup

* exp: add ability to collect dict of currently running experiments

* exp show: display run/queue state in the table

* exp run: unlock main repo while running tmpdir executors

* add tests for new exp show behavior

* fix workspace checkpoint run handling

* fix issue where running stash revs were shown as queued

* use "running"/"queued"

* add executor (location) column

* compare: add TabularData.is_empty

* use td.is_empty to drop state/executor

* use "workspace" and "temp" as executor names
2021-07-13 00:20:38 +09:00
Saugat Pachhai beb610c3ab add: ignore duplicated entries (#6286) 2021-07-12 20:43:41 +05:45
Batuhan Taskaya 4557de420c import-url: compute the stage hash in --to-remote (#6175)
* import-url: compute the stage hash in --to-remote
* save_deps() before calculating the md5
2021-07-12 09:59:34 +03:00
dependabot[bot] 481a411a35 build(deps): bump wsgidav from 3.1.0 to 3.1.1 (#6302)
Bumps [wsgidav](https://github.com/mar10/wsgidav) from 3.1.0 to 3.1.1.
- [Release notes](https://github.com/mar10/wsgidav/releases)
- [Changelog](https://github.com/mar10/wsgidav/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mar10/wsgidav/compare/v3.1.0...v3.1.1)

---
updated-dependencies:
- dependency-name: wsgidav
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-12 11:26:14 +05:45