Outputs and dependencies don't really care about the remotes, as they
don't do any pushing or pulling. What they really need are the trees,
so let's use them directly.
Part of #4050
* remote: move get_file_checksum() into tree
* makes RemoteTree and RepoTree consistent with regard to checksum
calculation
* remote: save() now takes explicit tree parameter
* when tree is remote.tree, save will be a move + link operation
(same as default existing behavior)
* when saving path from a different tree, save will be a copy operation
* tests: update for moved remote/tree functions
* remote: move get_checksum into tree
* remote: separate cloud remote and cache classes
* tests: update unit tests for remote/cache separation
* remote: cloud cache should extend cloud remote
* remote: move LocalRemote.get
* tests: update func tests
* dependency: update for moved get_checksum
* remote: fix state lookup bug
* BaseRemote: start moving tree functions into BaseRemoteTree
* remote: add LocalRemoteTree
* remote: add cloud remote trees
* remote: move remove(), makedirs() into tree
* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree
* fix attributes for moved remote methods
* tests: update unit tests for moved remote tree methods
* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py
* fix attributes for moved remote tree functions
* tests: update func tests for moved remote tree methods
* test fixes
* fix DS warnings
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* add: auto convert absolute_path => relative
- fixes#2955
* use path_isin
* catch-all absolute paths in repo
* unneeded leftover
* revert realtive imports for later
* output: move abs2relpath from Base to Local
* output:local:fix parse_path
* remove path_info
* test:local versus base abs2relpath
* test:fix remaining abs2relpath test
* output: local path relative to wdir
* output: local: modify path before inheritance
* output: local: minor inheritance fix
* test: local: split output test
* test: local: positively explicit
Related to #1629
Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Paweł Redzyński
Ruslan Kuprieiev
Saugat Pachhai
Peter Rowlands (변기호)
Casper da Costa-Luis
nik123
Mr. Outis
Ruslan Kuprieiev
Ivan Shcheklein