* dvc: get rid of is_walking_tree
No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.
* dvc: move GitTree to dvc/tree/
* dvc: rename *RemoteTree to *Tree
Usually we need it to parametrize tests with, so there is no
reason to duplicate the code. I've left `local_remote` fixture
just because it is so convenient and often used in a lot of tests.
* remote: move constants into trees
* remote: replace Remote() Cache() helper functions
* output: update for remote changes
* cache/data_cloud: update for remote changes
* dependency: update for remote changes
* stage: update for remote changes
* remote: move remaining non-index methods into tree
* tests: update remote unit tests
* remote: LocalCache needs to lock remote index, not cache index
* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH
* tests: repo tree open needs state context
* remote: make Remote.gc a class method
* operates on obj.tree, so that GC can be done with both remote and
cache instances
* bug fixes
* tests: update test_data_cloud
* bug fixes
* tests: update func tests
* remote: move get_file_checksum() into tree
* makes RemoteTree and RepoTree consistent with regard to checksum
calculation
* remote: save() now takes explicit tree parameter
* when tree is remote.tree, save will be a move + link operation
(same as default existing behavior)
* when saving path from a different tree, save will be a copy operation
* tests: update for moved remote/tree functions
* remote: move get_checksum into tree
* remote: separate cloud remote and cache classes
* tests: update unit tests for remote/cache separation
* remote: cloud cache should extend cloud remote
* remote: move LocalRemote.get
* tests: update func tests
* dependency: update for moved get_checksum
* remote: fix state lookup bug
* addressing: allow stage name addressing without colon
* repo: reorganize imports
* refactor/reorganize collect_granular
* add tests for stage addressing
* fix ds and cc issues
* fix unpack error in test
* tests: stage string representations
* exception: add error message
Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.
* rename NoOutputOrStage NoOutputOrStageError
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* pull/fetch/push: add stats
* fix typo: s/logging/logger
* pull: make stats detailed, with summary at the bottom
* tests: mock properly, set return_value
* add a line space
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* remote: Optimize traverse/no_traverse behavior
- estimate remote file count by fetching a single parent cache dir, then
determine whether or not to use no_traverse method for checking
remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
one parent cache dir per thread)
* Use arbitrary prefix length for traverse remote cache queries
- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)
* Add unit tests for new traverse/no_traverse behavior
* Fix deepsource warning
* Fix review issues
* fix CI
* Fix review issues
* obsolete no_traverse config option
- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
RemoteBASE.CAN_TRAVERSE (True for all remotes except http)
* review issues - use 'object exists' instead of 'no_traverse'
* fix obsolete no_traverse test issues
* fix SSHMocked func tests
* pull: support pulling git-only imports
* pull: also add git imports from external dvc repo
* fetch: add logging
* pull: don't save link to state
When `checkout` or `pull` is done, it'll try to cleanup unused links.
So, when we add from erepo to cache, it creates a link, which DVC will try
to cleanup. Also, it'll provide unwanted checkout change details. So, skipping those
* tests: refactor/rewrite
* flake8: why it's complaining?
* tests: do not leak file descriptors
* remote.http: support _upload() via HTTP POST
- uploaded files are sent as chunked encoding POST data
Fixes#3247
* remote.http: add support for HTTP basic and digest auth
- username, password, ask_password options work the same way as for
ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
remotes
- digest_auth takes precedence over basic_auth if both are enabled
* remote.http: auth_method() unit tests
* tests: Add functional tests for HTTP remotes
- HTTP remotes now tested locally using a SimpleHTTPServer instance that
allows reading/writing to a temp directory
* tests: fix http tests on py3.5, 3.6
* remote.http: add custom auth method
* s/custom_header/custom_auth_header/
* Apply suggestions from code review
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
We already have mocked ones, which are more reliable as they use their
own paramiko server instead of relying on the sshd running on the test
machine. Native tests are also quite flaky and fail sometimes due to
network issues.
Ruslan Kuprieiev
Saugat Pachhai
Peter Rowlands (변기호)
Paweł Redzyński
Casper da Costa-Luis
nik123
Ivan Shcheklein
Max Risuhin
Alexander Schepanovski
Jorge Orpinel
Fábio Santos