Commit Graph
38 Commits
Author SHA1 Message Date
Batuhan Taskaya 4a843dbd61 odb: use regular upload() when the source filesystem is local (#6365)
* odb: use regular upload() when the source filesystem is local

* support download_file too
2021-07-27 15:55:49 +03:00
Peter Rowlands (변기호) 3d84df930b objects: migrate remote push/pull to objects.transfer (#6308)
* objects: migrate remote status to objects.status

* data_cloud: use objects.status

* objects: migrate remote._process to objects.transfer

* data_cloud: use objects.transfer

* objects: handle memfs staging objects in transfer

* fetch/import: use unified transfer()

* update get_remote and transfer exception usage

* update tests

* objects.transfer: use src ODB verify rules after xfer

* odb: move index from remote into odb

* objects.transfer: skip src status check when possible

* update tests

* remove dvc.remote

* odb.index: migrate to diskcache

* objects: fix status/transfer optimization

* update index usage

* state: remove dead sqlite related code

* update cloud tests

* drop unnecessary remove in tests

* use abstract base class in odb index

* status: load trees from local cache instead of remote odb when possible

* handle status when dir cache is explicitly removed
2021-07-20 13:30:01 +03:00
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
Peter Rowlands (변기호) 71bae7e4af objects: use separate staging ODB for staging trees (#6195)
* fs: use fsspecwrapper in memfs

* odb: add memfs staging ODB in ODBManager

* odb: add read_only attribute for odb instances

* objects: handle memfs paths when staging trees

* odb: remove git ODB

* output: use staging ODB

* RepoDependency: use staging odb for imports

* dvcfs: handle case where dir cache is unavailable

* fix circular import check

* odb: use temp local ODB for staging

* objects: use staging ODB in stage() and tree load

* objects: add typing for add/check/load/stage

* output/dep: use new staging

* add unit tests for local odb staging

* localfs: account for string paths in move()

* include local in odb.by_scheme

* tests: count dir cache like any other object in test_pull_git_imports

* odb: clear staging on gc()

* objects: add tree.get/filter/insert

* output: use obj.filter/get for granular dir commit

* squash filter logic output

* dependency: don't filter imports at the dep level

* don't init staging ODB for external cache

* use staging ODB when computing tree hash for external outs

* remove unneeded force in test granular commit

* use unique-per-manager memfs URLs for staging

* odb: make staging per-ODBManager

* objects: move state dependent staging into ODBManager

* objects: support checking multiple odbs

- staging dependent checks can be done via ODBManager.check

* checkout: do state lookup before explicit stage() in _changed()

* output: use odbmanager based staging

* RepoDependency: use odbmanager based staging

* update obj staging unit tests

* checkout: remove check(), handle individual file exceptions on link()

* diff: use odbmanager based staging

* objects: add ObjectPermissionError for read/write perm errors

* odb: move staging out of odbmanager

* objects: handle staging ODB in objects.stage()

* objects.tree: don't stage anything in tree methods

* odb: clear staged objects on gc()

* remote: don't filter objs by scheme, objs are already grouped by ODB

* diff: update objects.stage usage

* output/dep: update stage() usage

* update ODB tests

* update func tests

* remove memfs from FS_MAP

* revert unneeded objects.check() changes

* remove unused tree.insert()

* revert unused odbmanager changes

* add separate odb error classes

* stage external outs directly in ODB

* check for obj existence in main ODB before staging

* use .dvc/tmp/staging instead of per-odb staging

* gc: only gc loal staging once
2021-07-09 18:48:28 +03:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
b349b39d62 Decoupling dvcignore and fs (#5812)
* Decoupling dvcignore and fs

[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check

* Solve all of the arugments error.

* add dvcignore to dvc list

* Solve some test_ignore cases

* add dvcignore to checkout.

1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs

* Solve two tests

* Get rid of ignore_filter

* Add ignore examine to download

* Solve a test issue

* Solve dvcignore and subrepo problems

* Solve sub repo error

* Revert some changes

* pass all tests

* Remove debug codes

* Some final exams

* dvcignore trie tree only update when needed

* reuse of the dir result in path walk.

* Some change after code review.

* Recover something from false `--force` push

* add abspath trans to dvcginore

* pass pylint

* pylint related

* Update dvc/fs/repo.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Changes related to reviews.

* dvcignore call api change

from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)

* Get rid of PathInfo in test_ignore.py

* fs.walk follow the arguments of os.walk

* Dvcignore as parameters

* pass dvcignore as arguments to checkouts

* Decoupling state and repo

* Update dvc/dvcfile.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Update dvc/objects/stage.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Solve some small problems

* Add tests to verify that dependency now not ignore

fix#5256

* Update dvc/ignore.py

* Update dvc/fs/git.py

* Update dvc/repo/__init__.py

* Some small improvements

* pass black

* Refactor is_ignored

* Update dvc/output/base.py

* repofs: use dvcignore in exists/isdir/isfile

Per https://github.com/iterative/dvc/pull/5812/files#r630604524

Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-05-12 05:29:04 +03:00
Batuhan Taskaya 042b1c7f7d pull: skip isdir() checks on downloading files (#5696)
* pull: use download_file

* tree: implement filesystem.download_file
2021-03-26 15:22:45 +02:00
Ruslan Kuprieiev 4af3b3d20d dvc: refactor objects (#5607)
* dvc: get rid of DirInfo

DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.

As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.

* objects: move Tree into tree.py
2021-03-12 01:10:34 +02:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Ruslan Kuprieiev 53ba51b5fe objects: separate staging (#5470) 2021-02-16 02:21:54 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Ruslan Kuprieiev 0fdb119ed5 cache: move save() to objects (#5430) 2021-02-09 04:56:45 +02:00
Ruslan Kuprieiev 226c68f8ff dvc: introduce File/Tree cache objects (#5424) 2021-02-09 04:00:06 +02:00
Ruslan Kuprieiev c83a4ea406 erepo: refactor download/fetch logic (#5234)
* erepo: don't use _get_tree_for

* erepo: refactor download/fetch logic

* dvc: unify handling existing output
2021-01-10 10:48:36 +02:00
Ruslan Kuprieiev a6151eaf9e erepo: get rid of get_checksum (#5145) 2020-12-22 15:07:32 +02:00
Saugat Pachhai 4564e8be30 support subrepos for get/list/api/import/ls (#4465)
* support subrepos

This adds support for get/list/api/import/ls on a subrepo.

Also fixes #3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes #3369.

* Add tests

* Fix tests, started to get lock_error

* Fix tests for windows

* try to fix mac builds

* Fix comments, readd removed comments

* Fix test

* address review suggestions:

* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
2020-09-03 13:47:14 +03:00
Peter Rowlands (변기호) a4b3c6d872 erepo: --rev BRANCH: shallow clone (#4246)
* erepo: use shallow clones when rev is branch or tag name

* tests: test erepo shallow clone behavior

* erepo: fix detached head after shallow cloning a tag

* move unshallow logic into its own function

* erepo: default rev should be origin HEAD not local HEAD

* use functools.partial

* review fixes

* comment shallow clone behavior in erepo
* don't duplicate unnecessary fetch/pull call
2020-07-22 09:59:08 +09:00
Ruslan Kuprieiev b77ce02f6b dvc: separate cache and remote classes (#4245)
Finalizes code part of #4050
2020-07-19 22:12:01 +03:00
Ruslan Kuprieiev a077135d76 Consolidate trees (#4243)
* dvc: get rid of is_walking_tree

No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.

* dvc: move GitTree to dvc/tree/

* dvc: rename *RemoteTree to *Tree
2020-07-19 18:43:16 +03:00
Ruslan Kuprieiev 51a8c782ae tests: create remote fixtures (#4063)
* tests: use remote fixtures

* tests: use remote fixtures for data cloud tests

* tests: treat remotes as tmp dirs
2020-06-18 01:36:13 +03:00
Peter Rowlands (변기호) 74bd8137ba remote: finish separating RemoteTree methods (#3946)
* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes
2020-06-04 23:22:31 +03:00
Peter Rowlands (변기호) b7cd785eba external_repo: use RepoTree for fetch/import/get (#3841)
* erepo: load external repos directly from git tree

* erepo: add fetch_external

* remote/RepoTree: add callback for keeping track of download counts when
saving tree to cache

* repo: use erepo.fetch_external() in repo.fetch()

* erepo: return cache save_infos in fetch_external

* DependencyRepo: use erepo.fetch_external and cache.checkout instead of
pull_to

* erepo: remove pull_to

* DvcTree: support granularity for isdir()

* DvcTree: fix indentation bug

* RepoTree: add copytree() method

* LocalRemote: use working tree for local cache walk_files

* repo: state should be noop for GitTree based erepos

* erepo: add get_external()

* test_get: remove unneeded caplog check

* erepo: add get_rev()

* tests: fix erepo test bug

* ls: use RepoTree

* remote: support trees in _collect_dir

* remote: support trees in get_dir_checksum

* RepoTree: fix dir_cache relpaths in windows

* remote: fix duplicated fetch bug

* utils: don't skip md5 dos2unix check for git blobs

* otherwise md5 checksums will not match when comparing git version to
  cached filesystem version on windows
2020-05-22 17:23:31 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
Paweł Redzyński 451c7aac04 tests: erepo_dir: extract remote setup to separate method (#3748) 2020-05-12 02:05:22 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
nik123 fefbcc0b4c refactor: dvc/remotes name unification (#3684)
Partially fixes #2089
2020-04-26 20:55:00 +03:00
Ruslan Kuprieiev 6559b9d222 dvc: use protected mode by default (#3472)
* dvc: use protected mode by default

Fixes #3261
Fixes #2041

* unprotect: adjust help message

https://github.com/iterative/dvc.org/pull/1058/files/21aab371a487acf6f6e6201b29cd832e7c55ed23#r393324668
2020-03-17 19:16:21 +02:00
Bruno Deferrari b136eb5151 erepo: fix relative url on default remote (#2756) (#3378)
* erepo: fix relative urls for remotes (#2756)

* erepo: add test case for relative path remotes (#2756)

* Remove unused argument

* erepo: move handling of relative remotes to a separate method (#2756)

* erepo: don't manually build paths on test (#2756)

* erepo: refactor handling of missing or relative upstream (#2756)

* erepo: move handling of source repo resource to `_fix_upstream` (#2756)

* erepo: improve relative remote tests (#2756)

- use `tmp_dir` fixture
- commit config file changes

* erepo: Remove comment (#2756)

* erepo: use scm_add to commit too, no need for separate scm.commit call (#2756)

* erepo: minor refactoring in `_fix_upstream` (#2756)

* erepo: add comment explaining that old relative paths are restored (#2756)
2020-02-26 22:37:29 +02:00
Ruslan Kuprieiev d2a6c1fc62 external_repo: fix bug in path resolution (#3350) 2020-02-17 20:30:04 +02:00
Alexander Schepanovski 7df06b278a erepo: fix known sha handling (#3323)
Closes #3280.
2020-02-14 12:59:31 +02:00
Alexander Schepanovski 4e27dedeb4 erepo: make it better
- code moving during git pull won't affect any erepos
- source tag move will work as expected
- source cache dir setting will never break havoc
- cache is reused between all clones from same source
2020-01-27 18:58:30 +07:00
Alexander Schepanovski 7168a9c50a erepo: pull each time
Fixes #3062.
2020-01-23 23:09:57 +07:00
Alexander Schepanovski 033ef88a4e external: refactor external_repo() and its users
Some things are fixed along the way:
- no cache shared between incompatible funcs
- some incosistent exceptions are now consistent
- import updates for git files now works both for dvc and git repos
- "auto-generated-upstream already exists" fixed
- dvc.api.open()/read() works with git repos now

Fixes #3124, #2976. Closes #3172.
2020-01-19 21:38:06 +07:00
Ruslan Kuprieiev d831044cb6 tests: external_repo: convert to dir helpers
Related to #2896
2020-01-13 04:15:27 +02:00
Saugat Pachhai 751bdb2197 dvc: refactor startswith usage with path_isin util 2019-11-21 22:22:08 +05:45
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Alexander Schepanovski 44264857c3 ui/perf: do not clone same repo multiple times (#2583)
* ui/perf: do not clone same repo multiple times

Closes #2490

External repos are still created in /tmp. They might be reused until
main() end. If no perfect match is found then a repo with the same url
is copied (not cloned again) and adjusted to fit a request: rev and
cache_dir.

NOTE: test_download_error_pulling_imported_stage() was modified to not
rely on clone error.

* test: fix import update test

* test: fix external_repo test under Windows

* dvc: fix non-master default branch for external repos

* dvc: remove unused .unset_dir() method
2019-10-11 13:56:37 +03:00