Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-10 07:18:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,475 Commits 7 Branches 579 Tags
66caa345b1cc1a49cf5fcd1befe72cd052dbe834
Commit Graph
20 Commits
Author SHA1 Message Date
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Batuhan TaskayaandRuslan Kuprieiev f4b14c2d4a gdrive: Implement .ls() protocol / proper tree walking with .walk_files() (#5433) 
* gdrive: .ls()/.walk()/.walk_dir()

* better walking

* proper tests

* ._ls_recursive()

* use posxipath for tests

* store ids instead of raw names

* add gdrive fixture

* add dependency

* tests: temporarily disable gdrive tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-02-13 03:10:38 +02:00
Ruslan Kuprieiev a077135d76 Consolidate trees (#4243) 
* dvc: get rid of is_walking_tree

No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.

* dvc: move GitTree to dvc/tree/

* dvc: rename *RemoteTree to *Tree
 2020-07-19 18:43:16 +03:00
Ruslan Kuprieiev 76228a2118 dvc: split remotes from trees (#4212) 
* dvc: remove tree-specific wrappers from remote

This helps combat the remote/tree/cache confusion.

* dvc: split remotes from trees

Related to #4050
 2020-07-15 23:44:09 +03:00
Saugat Pachhai ad306d7bf3 Enable most of the pylint checks (#4111) 2020-06-25 18:10:14 +03:00
Peter Rowlands (변기호) a0c229cf9b remote: continue remote/cache/tree refactoring (#4041) 
* remote: move constants into trees

* remote: replace Remote() Cache() helper functions

* output: update for remote changes

* cache/data_cloud: update for remote changes

* dependency: update for remote changes

* stage: update for remote changes

* remote: move remaining non-index methods into tree

* tests: update remote unit tests

* remote: LocalCache needs to lock remote index, not cache index

* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH

* tests: repo tree open needs state context

* remote: make Remote.gc a class method

* operates on obj.tree, so that GC can be done with both remote and
  cache instances

* bug fixes

* tests: update test_data_cloud

* bug fixes

* tests: update func tests
 2020-06-15 16:22:55 +03:00
Peter Rowlands (변기호) 74bd8137ba remote: finish separating RemoteTree methods (#3946) 
* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes
 2020-06-04 23:22:31 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805) 
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
 2020-05-15 16:48:02 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749) 
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
 2020-05-06 23:56:07 +03:00
Ivan ShchekleinandJorge Orpinel 8aefbac223 gdrive: fix multi-remote workflow, cont. cleanup (#3686) 
* remote, minor: fix parameter method name for consistency

* gdrive: cleanup, fix workflow with multiple gdrive remotes

* config: resolve gdrive cred file parth, typo fix

* grdive: address deepsource warning

* gdrive: fix tests after simplifying auth flow

* gdrive: address PR review, use backticks where appropriate

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: exception text improvements

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: fix exception message

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: fix root not found exception message

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: minor warnings/exceptions text improvement

* gdrive: add tests for the gdrive_user_credentials_file relative path

* gdrive: address review, slightly change text

* gdrive: comments -> docstrings, addressing PR review

Co-authored-by: Jorge Orpinel <[email protected]>
 2020-04-29 23:25:56 +03:00
nik123 fefbcc0b4c refactor: dvc/remotes name unification (#3684) 
Partially fixes #2089
 2020-04-26 20:55:00 +03:00
Peter Rowlands (변기호) dfe184f149 tests: convert remote unit tests to pytest (#3604) 
- update tests/unit/remote tests to use pytest style tests
- use pytest dvc repo fixtures instead of improperly mocked repo's or
  None
 2020-04-07 10:45:49 +03:00
Ivan Shcheklein f6077d06c4 gdrive: method signatures cleanup, more safeguards (#3548) 2020-03-30 09:49:34 +03:00
Peter Rowlands (변기호) 9b38aa4680 remote: Optimize traverse/no_traverse behavior (#3501) 
* remote: Optimize traverse/no_traverse behavior

- estimate remote file count by fetching a single parent cache dir, then
  determine whether or not to use no_traverse method for checking
  remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
  one parent cache dir per thread)

* Use arbitrary prefix length for traverse remote cache queries

- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)

* Add unit tests for new traverse/no_traverse behavior

* Fix deepsource warning

* Fix review issues

* fix CI

* Fix review issues

* obsolete no_traverse config option

- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
  RemoteBASE.CAN_TRAVERSE (True for all remotes except http)

* review issues - use 'object exists' instead of 'no_traverse'

* fix obsolete no_traverse test issues

* fix SSHMocked func tests
 2020-03-25 05:04:20 +02:00
Max Risuhin 3ff804b3b2 gdrive: Google Service account support (#3269) 2020-03-11 15:49:33 +02:00
Ruslan Kuprieiev e848bcb9d6 gdrive: allow configuring no_traverse 2020-01-20 23:49:49 +02:00
Guro Bokum 4918c56ca3 remote: config checksum_jobs 
Close #2920
 2020-01-15 22:23:09 +08:00
Max Risuhin 57111a7788 gdrive: add support for teamDriveId 2020-01-11 07:41:12 -08:00
Max Risuhin 4727cb7893 Improve Google Drive auth failure error reporting 2019-12-28 05:22:37 -08:00
Max Risuhin 4d4a3fc716 Standardize gdrive tests with others remotes 2019-12-15 05:31:52 -08:00