692 Commits
Author SHA1 Message Date
Ruslan Kuprieiev b77ce02f6b dvc: separate cache and remote classes (#4245)
Finalizes code part of #4050
2020-07-19 22:12:01 +03:00
Ruslan Kuprieiev a077135d76 Consolidate trees (#4243)
* dvc: get rid of is_walking_tree

No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.

* dvc: move GitTree to dvc/tree/

* dvc: rename *RemoteTree to *Tree
2020-07-19 18:43:16 +03:00
Ruslan Kuprieiev 1c4a8cf2ca dvc: remove tree-specific wrappers from cache (#4214)
This helps combat the remote/tree/cache confusion.
2020-07-16 00:46:19 +03:00
Ruslan Kuprieiev 76228a2118 dvc: split remotes from trees (#4212)
* dvc: remove tree-specific wrappers from remote

This helps combat the remote/tree/cache confusion.

* dvc: split remotes from trees

Related to #4050
2020-07-15 23:44:09 +03:00
Ruslan Kuprieiev 375998c664 tests: hdfs: use hadoop/docker fixtures (#4183)
Fixed #4054
2020-07-10 21:38:48 +03:00
Ruslan Kuprieiev bb2cf74328 gdrive: pop prefix kwarg (#4157)
https://travis-ci.com/github/iterative/dvc/jobs/357157580#L8485
2020-07-03 03:04:26 +03:00
Peter Rowlands (변기호) fa16b56cba s3,gs: don't append "/" to prefixed path when listing objects (#4149)
* s3,gs: don't append "/" to prefixed path when listing objects

* remote: use optional prefix kwarg for walk_files

* if prefix is True, path will be walked as a prefix
* if prefix is False, path will be treated as a directory (trailing
  slash will be appended for S3-like remotes)

* tests: add unit test for BaseRemoteTree.list_paths
2020-07-02 20:04:04 +03:00
Ruslan Kuprieiev 27af048d3e azure: use knack directly instead of azure-cli-core (#4147) 2020-07-01 22:54:25 +03:00
Ruslan Kuprieiev 0275a0d8d8 azure: use CLI config (#4137)
Fixes #4129
2020-06-30 14:13:25 +03:00
Saugat Pachhai 1c0f5cdf4b pylint: enable additional checks (#4122)
Created a separate pylint plugin to disable some checks for tests.
2020-06-29 19:15:05 +03:00
Ruslan Kuprieiev 7b97d7249d hdfs: implement makedirs separately (#4128)
Clouds that have a non-noop `makedirs` should implement it in their own
method. Hdfs somehow slipped under the radar and had `mkdir` used
explicitly in `copy` and `upload` methods.
2020-06-29 10:27:02 +03:00
Saugat Pachhai ad306d7bf3 Enable most of the pylint checks (#4111) 2020-06-25 18:10:14 +03:00
Ruslan Kuprieiev d3457fb8f9 tests: data_cloud: migrate the leftovers to pytest (#4109)
Related to #1819
2020-06-25 16:13:43 +03:00
Saugat Pachhai b81ace54df Enable pylint checks (#4092)
* pylint: enable checks for imports

* ci: add github action for pylint

* enable pylint on pre-commit

* Install requirements

* use local pylint installation rather than of pre-commit's

* fix import errors

* more checks

* Silence unused imports

* Make a copy of dict
2020-06-24 18:02:39 +03:00
Ruslan Kuprieiev 2b7731e4f9 git: don't use multithreading for computhing dir hash (#4083)
GitPython is not threadsafe, which was causing issues when we were
computing hash for a directory.

Fixes #4079

Related to tree generalization #4050
2020-06-22 23:10:02 +03:00
Saugat Pachhai 4af3919062 Add flake8 extensions (#4067)
* remove unnecessary generators/comprehensions

* Use dict literals

* remove unused variables

* misc fixes

ignore .iteritems, is a false +ve
remove function call in default argument

* Flake8 config: complexity, show source on err, show err counts

* flake8: add extensions

* remove flake8-string-format

* Add flake8-debugger support

* Add flake8-string-format extension

* remove flake8-debugger

It's having some problems to install on Windows in Travis
2020-06-18 14:36:30 +03:00
Banti Gheneti 706dd05e8d remote/gs: fix url signing when using GoogleStorage on GoogleComputeEngine (#4057)
Fixes #4056
2020-06-17 13:24:02 +03:00
Ruslan Kuprieiev 613ca0bdb5 get-url/import-url: save dependency before downloading (#4049)
This allows us to properly process the errors like missing target.

Fixes #3897
2020-06-16 13:05:21 +03:00
Peter Rowlands (변기호) 0fb9870de6 remote: rename "checksum" -> "hash" (#4051)
* remote: s/get_checksum/get_hash/

* remote/tree: s/get_file_checksum/get_file_hash/

* remote: s/get_dir_checksum/get_dir_hash/

* remote: s/checksum_to_path/hash_to_path/

* remote: s/path_to_checksum/path_to_hash/

* remote: cleanup remaining checksum references

* remote.index: s/checksum/hash/

* note: does not rename "checksum" field in the index DB schema
2020-06-16 12:16:51 +03:00
Peter Rowlands (변기호) a0c229cf9b remote: continue remote/cache/tree refactoring (#4041)
* remote: move constants into trees

* remote: replace Remote() Cache() helper functions

* output: update for remote changes

* cache/data_cloud: update for remote changes

* dependency: update for remote changes

* stage: update for remote changes

* remote: move remaining non-index methods into tree

* tests: update remote unit tests

* remote: LocalCache needs to lock remote index, not cache index

* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH

* tests: repo tree open needs state context

* remote: make Remote.gc a class method

* operates on obj.tree, so that GC can be done with both remote and
  cache instances

* bug fixes

* tests: update test_data_cloud

* bug fixes

* tests: update func tests
2020-06-15 16:22:55 +03:00
Saugat Pachhai bc4bf66d17 (bugfix) get/import: fix broken progress bar (#4039)
PR #3672 (6d8499e) extended `LocalRemote::_get_plans` to return one
`checksums` too. As all of the args from `_get_plans` was passed
down to `download()`, it recognized extra arg of checksum as
`no_progress_bar` due to which it became True and stopped showing
progress bar at all.

Fix #3874
2020-06-14 20:42:53 +03:00
Casper da Costa-Luis 4d140a9944 gdrive: add open (#3916)
* gdrive: add open

Fixes #3408
Related #2865
Fixes #3897

* dependency: add gdrive

* test: api: open: gdrive

* rollback pydrive dependency

* Revert "dependency: add gdrive"

This reverts commit fd333261d2e9fe1364cc278138c79890a1f5fe7a.

* tests: remotes: GDrive: fix get_url

* tests: api: fully test GDrive

* minor typo fix

* attempt gdrive tests fix

* fix gdrive credentials config

* replace GDrivePathNotFound => FileMissingError

* fix test_open_external[GDrive]

* add ensure_dir_scm

* minor exception fix
2020-06-13 23:53:25 +03:00
Peter Rowlands (변기호) 9ead641c95 remote: separate cloud remote and cloud cache classes (#4019)
* remote: move get_file_checksum() into tree

* makes RemoteTree and RepoTree consistent with regard to checksum
  calculation

* remote: save() now takes explicit tree parameter

* when tree is remote.tree, save will be a move + link operation
  (same as default existing behavior)
* when saving path from a different tree, save will be a copy operation

* tests: update for moved remote/tree functions

* remote: move get_checksum into tree

* remote: separate cloud remote and cache classes

* tests: update unit tests for remote/cache separation

* remote: cloud cache should extend cloud remote

* remote: move LocalRemote.get

* tests: update func tests

* dependency: update for moved get_checksum

* remote: fix state lookup bug
2020-06-12 15:52:47 +03:00
Peter Rowlands (변기호) 652f5ab439 remote: use separate working tree instance for local cache (#3991)
* dvcignore: ignore paths outside of CleanTree root

* remote: local remote/cache should always use its own working tree

* repo tree (CleanTree) instance should not be relevant to local cache
  operations. When the dvcignore CleanTree is relevant, repo.tree should
  be used. For local remote/cache operations, the local remote work_tree
  should be used.

* state: tie state to local cache working tree

* tests: update tests

* state: use os.path.exists instead of WorkingTree.exists in some cases
2020-06-12 12:52:38 +03:00
Ruslan Kuprieiev 939f78948a remote: tree: don't ignore exceptions (#3977)
Moving ignore logic to the remote itself, so that tree could behave
properly.
2020-06-09 11:25:13 +03:00
Peter Rowlands (변기호) b57077af11 remote: fix remaining gdrive build issues (#3959)
* tests: fix gdrive build

* remote.gdrive: fix walk_files prefix relpath
2020-06-05 15:36:47 +09:00
Peter Rowlands (변기호) 803bb3eb61 remote: fix build errors (#3957) 2020-06-05 05:14:34 +03:00
Peter Rowlands (변기호) 74bd8137ba remote: finish separating RemoteTree methods (#3946)
* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes
2020-06-04 23:22:31 +03:00
Peter Rowlands (변기호) 2c0116ccf9 remote: separate tree functions from main remote classes (#3931)
* BaseRemote: start moving tree functions into BaseRemoteTree

* remote: add LocalRemoteTree

* remote: add cloud remote trees

* remote: move remove(), makedirs() into tree

* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree

* fix attributes for moved remote methods

* tests: update unit tests for moved remote tree methods

* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py

* fix attributes for moved remote tree functions

* tests: update func tests for moved remote tree methods

* test fixes

* fix DS warnings
2020-06-03 06:04:25 +03:00
Saugat Pachhai bb50fb8664 use higher level API selectors instead of select (#3899) 2020-05-29 09:25:07 +05:45
Nicholas A. Del Grosso 3f996048e8 Bugfix: _credentials_location (#3896)
_credentials_location  to credentials_location to removed attributeerror
2020-05-28 14:41:34 +03:00
Ruslan Kuprieiev d77af8196e optimize local status (#3867)
* remote: remove unpacked dir

We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.

* local: save us a stat call in is_protected

* local: speedup checksum_to_path ~x5.5

* tree: walk: don't use os.path.join

It is ~5.5 times slower than joining by hand.

* dvcignore: optimize matching x10

On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.

To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.

It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
2020-05-25 03:20:11 +03:00
Ruslan Kuprieiev b7ad2dc841 test: fix hdfs/azurite/oss tests (#3864)
* travis: fix typo

* tests: azure: fix expected etag format

* test: ssh: increase the number of tries

* travis: don't install docker

* dvc: update pyarrow

* tests: ssh: don't leak connections

* remote: don't fail when cache/remote doesn't have prefix subdir
2020-05-23 23:46:40 +03:00
Steffan Sluis f9fbe97509 azure: support external file dependencies (#3852)
* azure: support external dependencies and outputs

- Implement get_file_checksum with get_etag
- Implement a DependencyAzure and OutputAzure classes
- Add AzureOutput to OUTS, OUTS_MAP and CHECKSUMS_SCHEMA
- Add AzureDependency to DEPS and DEPS_MAP
- Add unit tests for AzureOutput and AzureDependency

Fixes #3540.

* Use content_md5 instead of etag

The etag is different for the same file in different locations.

* Use etag and remove external outputs for Azure

Due to inconsistencies with Azure's Content-MD5 property, it is not
stable enough to be used at the moment. Using the etag offers enough
reliability to support external dependencies. See also:
https://discordapp.com/channels/485586884165107732/565699007037571084/713462619469643796
2020-05-23 18:53:52 +03:00
Paweł Redzyński d74358eecd remote: local: protect uploaded files (#3858) 2020-05-22 22:59:48 +03:00
Ruslan Kuprieiev 58c9ff0de2 azure: drop obsoleted url format (#3856) 2020-05-22 20:36:04 +03:00
Peter Rowlands (변기호) b7cd785eba external_repo: use RepoTree for fetch/import/get (#3841)
* erepo: load external repos directly from git tree

* erepo: add fetch_external

* remote/RepoTree: add callback for keeping track of download counts when
saving tree to cache

* repo: use erepo.fetch_external() in repo.fetch()

* erepo: return cache save_infos in fetch_external

* DependencyRepo: use erepo.fetch_external and cache.checkout instead of
pull_to

* erepo: remove pull_to

* DvcTree: support granularity for isdir()

* DvcTree: fix indentation bug

* RepoTree: add copytree() method

* LocalRemote: use working tree for local cache walk_files

* repo: state should be noop for GitTree based erepos

* erepo: add get_external()

* test_get: remove unneeded caplog check

* erepo: add get_rev()

* tests: fix erepo test bug

* ls: use RepoTree

* remote: support trees in _collect_dir

* remote: support trees in get_dir_checksum

* RepoTree: fix dir_cache relpaths in windows

* remote: fix duplicated fetch bug

* utils: don't skip md5 dos2unix check for git blobs

* otherwise md5 checksums will not match when comparing git version to
  cached filesystem version on windows
2020-05-22 17:23:31 +03:00
Peter Rowlands (변기호) 75e795c491 remote: support saving RepoTree objects directly to cache (#3825)
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache

* RepoTree: support filtering out dvcfiles during walk()

* RepoTree: support recursive isdvc()

* remote: support computing checksums for tree objects

* RepoTree: pull dir contents on walk() if in `fetch` mode

* tests: add func test for DVC dir pull on RepoTree.walk()

* DvcTree: support granular open()

* RepoTree: add get_file_checksum()

* remote: revert checksum changes

* remote: add tree support for cache.save()

* tests: test saving directly to cache from RepoTree

* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree

* RepoTree: state context should be entered by open()/walk() caller

* RepoTree: only pull dir contents when cache has changed

* tree: update tree tests

* fix ds warnings, update comments

* fix windows tests
2020-05-21 06:31:28 +03:00
Ruslan Kuprieiev 324ff4be59 remote: is_dir_checksum: handle empty/None checksum (#3836)
Fixed #3434
2020-05-20 18:00:41 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
Jack WellsandRuslan Kuprieiev 7ca3a1119c Added S3 SSEKMSKeyId config option (#3762)
* Added S3 SSEKMSKeyId config option

* test: unit: s3: add SSEKMSKeyId test

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-11 22:52:55 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Casper da Costa-Luis 26e203242f gdrive: download: stream & add progress (#3722)
* wip

* gdrive: add progress

Part of #2865
See https://github.com/iterative/dvc/issues/2865#issuecomment-622984785

* gdrive: move towards next pydrive2 release

- depends on https://github.com/iterative/PyDrive2/pull/30

* update to latest pydrive>=1.4.11

* avoid unneeded API call

* progress: gdrive: ensure proper bar_format
2020-05-06 17:02:16 +03:00
Casper da Costa-Luis c873787d73 progress: add postfix info to avoid overwriting desc (#3721)
* progress: persist primary description

Move subsequent updates to a postfix.
Clear postfix on exit.
TODO: align nicely.
Fixes #3681.

* progress: move to posfix[info] for full control

It was bound to happen.

* progress: git: persist description
2020-05-02 18:07:05 +03:00
Peter Rowlands (변기호) af12f0d06c remote: adjust traverse threshold multiplier (#3705)
* Fixes #3704
2020-04-30 12:12:11 +09:00
Ivan ShchekleinandJorge Orpinel 8aefbac223 gdrive: fix multi-remote workflow, cont. cleanup (#3686)
* remote, minor: fix parameter method name for consistency

* gdrive: cleanup, fix workflow with multiple gdrive remotes

* config: resolve gdrive cred file parth, typo fix

* grdive: address deepsource warning

* gdrive: fix tests after simplifying auth flow

* gdrive: address PR review, use backticks where appropriate

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: exception text improvements

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: fix exception message

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: fix root not found exception message

Co-Authored-By: Jorge Orpinel <[email protected]>

* gdrive: minor warnings/exceptions text improvement

* gdrive: add tests for the gdrive_user_credentials_file relative path

* gdrive: address review, slightly change text

* gdrive: comments -> docstrings, addressing PR review

Co-authored-by: Jorge Orpinel <[email protected]>
2020-04-29 23:25:56 +03:00
Peter Rowlands (변기호) 31fe77848f remote.ssh: suppress paramiko logging (#3675)
* remote.ssh: suppress paramiko logging

* use SSHClient.set_log_channel
2020-04-29 18:17:53 +05:45
Peter Rowlands (변기호) 6d8499ea6d remote: use string paths over PathInfo for performance reasons (#3672)
* remote: use checksums instead of paths when filling dir statuses

* remote: prefer using str paths over PathInfo for performance reasons

* only use string paths for RemoteLOCAL

* only optimize calls made from RemoteLOCAL.cache_exists

* fix ds warning

* use os.path in local

* remote: re-add checksum_to_path() to return string paths when applicable

- cloud remotes still default to using PathInfo's

* cache fspath string

* use abspath in checksum_to_path

- if path is not relpath from cwd or abspath, posix lstat() syscall
  runtime doubles (from calculating relpath from cwd)
2020-04-28 15:19:51 +03:00
nik123 fefbcc0b4c refactor: dvc/remotes name unification (#3684)
Partially fixes #2089
2020-04-26 20:55:00 +03:00