Compare commits

..
78 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 7c6351d039 dvc: bump to 1.0.0b4 2020-06-16 15:17:19 +03:00
Ruslan Kuprieiev 613ca0bdb5 get-url/import-url: save dependency before downloading (#4049)
This allows us to properly process the errors like missing target.

Fixes #3897
2020-06-16 13:05:21 +03:00
Peter Rowlands (변기호) 0fb9870de6 remote: rename "checksum" -> "hash" (#4051)
* remote: s/get_checksum/get_hash/

* remote/tree: s/get_file_checksum/get_file_hash/

* remote: s/get_dir_checksum/get_dir_hash/

* remote: s/checksum_to_path/hash_to_path/

* remote: s/path_to_checksum/path_to_hash/

* remote: cleanup remaining checksum references

* remote.index: s/checksum/hash/

* note: does not rename "checksum" field in the index DB schema
2020-06-16 12:16:51 +03:00
Ruslan Kuprieiev 73e60a8c13 Delete pre-commit.yaml 2020-06-15 23:35:10 +03:00
Ruslan Kuprieiev 9d1dde4051 Create pre-commit.yaml 2020-06-15 23:32:39 +03:00
Ruslan Kuprieiev 38dc0f85d4 dvc: bump to 1.0.0b3 2020-06-15 22:15:57 +03:00
Paweł Redzyński d2dbe9649c plots: add smooth template (#4046) 2020-06-15 22:15:27 +03:00
Ruslan Kuprieiev ece6f2d3ee dvc: bump to 1.0.0b2 2020-06-15 21:36:05 +03:00
Ruslan Kuprieiev b672622324 plots: make sure template is using data anchor (#4045)
Fixes #3969
2020-06-15 21:14:07 +03:00
Ruslan Kuprieiev c9391df313 plots: modify: handle --unset for nonexistent options (#4044)
* tests: plots: reorginize the structure

* plots: modify: handle --unset for nonexistent options

Fixes #3933
2020-06-15 20:31:08 +03:00
Peter Rowlands (변기호) a0c229cf9b remote: continue remote/cache/tree refactoring (#4041)
* remote: move constants into trees

* remote: replace Remote() Cache() helper functions

* output: update for remote changes

* cache/data_cloud: update for remote changes

* dependency: update for remote changes

* stage: update for remote changes

* remote: move remaining non-index methods into tree

* tests: update remote unit tests

* remote: LocalCache needs to lock remote index, not cache index

* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH

* tests: repo tree open needs state context

* remote: make Remote.gc a class method

* operates on obj.tree, so that GC can be done with both remote and
  cache instances

* bug fixes

* tests: update test_data_cloud

* bug fixes

* tests: update func tests
2020-06-15 16:22:55 +03:00
karajan1001 a59f90fac9 Performance improvement of dvcignore (#3967)
fix#3869
1.Use big regex.

* Solve windows

* add rule order test

* Solve ignore order

* remove list comprehensions
2020-06-15 11:37:01 +02:00
Ruslan Kuprieiev bba502314d dvc: bump to 1.0.0b1 2020-06-14 20:44:44 +03:00
Saugat Pachhai bc4bf66d17 (bugfix) get/import: fix broken progress bar (#4039)
PR #3672 (6d8499e) extended `LocalRemote::_get_plans` to return one
`checksums` too. As all of the args from `_get_plans` was passed
down to `download()`, it recognized extra arg of checksum as
`no_progress_bar` due to which it became True and stopped showing
progress bar at all.

Fix #3874
2020-06-14 20:42:53 +03:00
f7348af1e0 metrics & plots: help output improvements (#3963)
* metrics & plots: help output improvements

* metrics & plots: ui: updated help messages

* refactor: removed redundant parentheses

* metrics: ui: updated 'targets' description in 'metrics show'

* metrics: ui: reordered params to match the order in cmd ref

* plots: ui: params modified to match cmd ref

* plots: ui: updated diff help

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
2020-06-14 20:39:33 +03:00
Casper da Costa-Luis 4d140a9944 gdrive: add open (#3916)
* gdrive: add open

Fixes #3408
Related #2865
Fixes #3897

* dependency: add gdrive

* test: api: open: gdrive

* rollback pydrive dependency

* Revert "dependency: add gdrive"

This reverts commit fd333261d2e9fe1364cc278138c79890a1f5fe7a.

* tests: remotes: GDrive: fix get_url

* tests: api: fully test GDrive

* minor typo fix

* attempt gdrive tests fix

* fix gdrive credentials config

* replace GDrivePathNotFound => FileMissingError

* fix test_open_external[GDrive]

* add ensure_dir_scm

* minor exception fix
2020-06-13 23:53:25 +03:00
Casper da Costa-Luis 2ba73daedc snap: make future major version upgrades easier (#4038)
Related: #3999
Related: #4037
2020-06-13 23:27:03 +03:00
Ruslan Kuprieiev 7a31a35cbf dvc: bump to 1.0.0b0 2020-06-13 00:56:14 +03:00
Saugat Pachhai 24a14563cc run: preserve order when -f/--force overwrite stage (#4035)
This commit preserves comments, meta and order inside a dvc.yaml
when `-f/--force` is used. But, it does not maintain comments/meta
for .dvc files.

Related: #4014
2020-06-13 00:55:15 +03:00
Ruslan Kuprieiev a1d3440f85 metrics: diff: introduce --precision (#4026)
Rounding to 5 digits by default.

Fixes #3904
2020-06-13 00:54:42 +03:00
Saugat Pachhai 45dcb7c27d Update dag.py (#4036) 2020-06-12 20:33:19 +03:00
Saugat Pachhai 9ff6fef53a bugfix: get dump folder contents in respective output folder (#4034)
The relative path for external git directory was computed relative to
the root of the repo itself, which created a relative path which is
current working directory. So, all of the contents of the erepo to download
was dumped in the pwd rather than inside of it's own directory.

Fix #4033
2020-06-12 20:31:18 +03:00
Saugat PachhaiandRuslan Kuprieiev 9e7dc2640e schema: provide a jsonschema for dvc.yaml (#4028)
* schema: provide a jsonschema for dvc.yaml

* Update scripts/schema/dvc-yaml.json

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update scripts/schema/dvc-yaml.json

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-12 20:09:59 +05:45
Paweł Redzyński 584c97a58d plots: fix handling of multiple plots from single revision (#4031) 2020-06-12 16:08:08 +03:00
Saugat Pachhai b94a18cfb9 tests: tests for dvcfile (#4032) 2020-06-12 15:54:05 +03:00
Peter Rowlands (변기호) 9ead641c95 remote: separate cloud remote and cloud cache classes (#4019)
* remote: move get_file_checksum() into tree

* makes RemoteTree and RepoTree consistent with regard to checksum
  calculation

* remote: save() now takes explicit tree parameter

* when tree is remote.tree, save will be a move + link operation
  (same as default existing behavior)
* when saving path from a different tree, save will be a copy operation

* tests: update for moved remote/tree functions

* remote: move get_checksum into tree

* remote: separate cloud remote and cache classes

* tests: update unit tests for remote/cache separation

* remote: cloud cache should extend cloud remote

* remote: move LocalRemote.get

* tests: update func tests

* dependency: update for moved get_checksum

* remote: fix state lookup bug
2020-06-12 15:52:47 +03:00
Peter Rowlands (변기호) 652f5ab439 remote: use separate working tree instance for local cache (#3991)
* dvcignore: ignore paths outside of CleanTree root

* remote: local remote/cache should always use its own working tree

* repo tree (CleanTree) instance should not be relevant to local cache
  operations. When the dvcignore CleanTree is relevant, repo.tree should
  be used. For local remote/cache operations, the local remote work_tree
  should be used.

* state: tie state to local cache working tree

* tests: update tests

* state: use os.path.exists instead of WorkingTree.exists in some cases
2020-06-12 12:52:38 +03:00
bjonnhandJonathan Bisson 28a123e727 Remove a remnant umask to solve #4021 (#4022)
Co-authored-by: Jonathan Bisson <[email protected]>
2020-06-11 19:47:25 +03:00
Ruslan Kuprieiev 566472f088 dvc: bump to 1.0.0a11 2020-06-11 19:09:51 +03:00
Saugat PachhaiandRuslan Kuprieiev 2e12a31c5a tests: cover run cache with functional tests (#4018)
* tests: cover run cache with functional tests

* run-cache: use _log_exceptions wrapper

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-11 18:44:21 +03:00
Alexander Schepanovski d7788887f0 plots: fix a couple of bugs (#4017)
- running .plots.collect() not from repo root used to skip plot files
- make Plots.render() be more convenient to be called from outside
2020-06-11 18:20:12 +07:00
Ruslan Kuprieiev 87276faef2 RepoTree: work with dirty repos (#4005)
Currently `RepoTree.open` will throw FileNotFoundError if you have a
file that is tracked by dvc, but which has no md5 or a cache for which
is missing, even though you have the file itself in your workspace.

This corresponds very nicely to the `--no-commit` or `--no-exec`
scenario, where you might not want to commit your artifacts to cache
just yet.

Fixes #3974
2020-06-11 09:50:27 +03:00
Saugat Pachhai 4042d5232d run: use -f/--force to overwrite, fix msg (#4011)
Also, makes fixes for `--overwrite` rename to `--force`
and also remove `remove_with_prompt` which was only being
used with `force\=True` anyway.

Fix #4003
2020-06-10 20:17:27 +03:00
Saugat Pachhai fda14c82d3 plots modify: do not convert output types to plot on unset (#4009)
Fix #4006
2020-06-10 15:31:17 +03:00
Saugat Pachhai 64be262d18 dvc.yaml: preserve meta (#4008) 2020-06-10 14:37:21 +03:00
Saugat Pachhai 1d6c80001d plots modify: complain if template does not exist (#3995) 2020-06-10 16:12:56 +05:45
Ruslan Kuprieiev 91a7194e4f add/run/import-url: don't use -f as --file (#4004)
If anything, `-f` is closer to `--force` than to `--file`. So let's set
this straight.
2020-06-10 05:04:55 +03:00
Ruslan Kuprieiev afe539aaba plots: rename --no-csv-header to --no-header (#4001)
* plots: rename --no-csv-header to --no-header

Fixes #3996

* fix typo
2020-06-10 03:34:25 +03:00
Ruslan Kuprieiev f259bc52cd config/remote/cache: make --system/--global/--local mutually exclusive (#4000) 2020-06-10 02:32:18 +03:00
Ruslan Kuprieiev db69def834 status: introduce --show-json (#3998)
Fixes #1406
2020-06-10 01:51:30 +03:00
Casper da Costa-Luis 5efc3bbee2 snap: build fixes (#3990)
* snap: drop sudo

Fixes #3989

* snap: ci: fix builds

* snap: ci: developer notifications

* snap: minimise size

* debug commit

* Revert "snap: minimise size"

This reverts commit 77b41a96f68b2c8df833803d0a644172b7db4779.

* fix image info

* Revert "debug commit"

This reverts commit 232baf0ad78335b0f3806456a3f20b82cdf0f24c.

* snap: v1 channels

Related: #3872
2020-06-10 00:00:15 +03:00
Alexander Schepanovski 7230d87137 Refactor plots (#3994)
- separate data collection and rendering
- provide a way to use custom template loader
- drop hard-coding datafiles in template feature
2020-06-09 22:47:17 +03:00
Paweł Redzyński 56833387ce plots: find metric data in yaml dictionary (#3992) 2020-06-09 16:42:28 +03:00
Saugat Pachhai 593e67318b do not consider stage cached if checksum/cmd has changed (#3993) 2020-06-09 18:46:49 +05:45
Saugat Pachhai afc183f4e7 tests serializing of stage to lockfile (#3988)
* tests: serialize stage for lockfile

* config: make isort/black compatible

* convert class based tests to pytest
2020-06-09 15:48:17 +03:00
Saugat Pachhai d201fad1e3 tests: serialize pipeline file (#3985)
serialize: make params ordered
2020-06-09 14:22:55 +03:00
Ruslan Kuprieiev 5be12b5aac plots: fix help message (#3983)
Per https://github.com/iterative/dvc/pull/3911#discussion_r436238442
2020-06-09 14:15:15 +03:00
Saugat Pachhai c1e5b10cbd remove unnecessary dependencies (#3987)
Fix #3971
2020-06-09 12:43:34 +03:00
Saugat Pachhai b47b2c5673 tests: update mock-ssh-server (#3986) 2020-06-09 12:39:10 +03:00
Ruslan Kuprieiev 939f78948a remote: tree: don't ignore exceptions (#3977)
Moving ignore logic to the remote itself, so that tree could behave
properly.
2020-06-09 11:25:13 +03:00
Saugat Pachhai 215ea4944c log: show modifying if entry already exists in dvcfile (#3978) 2020-06-08 18:56:00 +03:00
Saugat Pachhai 4e12e76e19 tests: unit testing for StageLoader for Pipelinefile (#3976) 2020-06-08 18:55:16 +03:00
Saugat Pachhai 33092e8614 issue template: do not auto-add labels on new issues (#3979) 2020-06-08 19:47:42 +05:45
Saugat Pachhai ae7b604531 Fix loading of params from stage cache (#3972) 2020-06-08 12:43:13 +03:00
Ruslan Kuprieiev b7c174641e tests: mark s3 test as flaky 2020-06-08 12:27:25 +03:00
Ruslan Kuprieiev cb43f9cdda dvc: bump to 1.0.0a10 2020-06-08 11:56:58 +03:00
Ruslan Kuprieiev 791ce22759 pyinstaller: add hook for google-api-python-client (#3965)
Fixes #3913
2020-06-08 11:53:12 +03:00
Casper da Costa-Luis c70adebc75 snap: restrict site-package checks (#3878)
* snap: restrict site-package checks

Fixes #3514

* add comment
2020-06-07 07:50:15 +03:00
2c9cb87679 Subcommand help (#3956)
* change DvcParser to print subcommand help

* add tests for subcommand help

* docstring refactoring

* remote: finish separating RemoteTree methods (#3946)

* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes

* Unify metrics behavior with dirs (#3935)

* run: disable directories for -m/-M flags

* run: reformatted string literals

* run: simplified metrics check

* tests: func: run: dirs as metrics outs

* tests: func: run: reformatted string literals

* tests: use pytest

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* refactor/test multistage load for params and outputs (#3949)

* refactor multistage load for params and outputs

* tests: load params

* tests: output loading from pipeline file

* fix test

* fix typo in name

* split params load

* rename params func s/inject_values/fill_values

* fix tests

* simplify loads_params and output.load_from_pipeline

* address @pared's suggestions for tests

* remote: fix build errors (#3957)

* remote: fix remaining gdrive build issues (#3959)

* tests: fix gdrive build

* remote.gdrive: fix walk_files prefix relpath

* change DvcParser to print subcommand help

* add tests for subcommand help

* docstring refactoring

* change tests to pytest-style

* remove gettext

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: nik123 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
2020-06-05 23:44:02 +03:00
Peter Rowlands (변기호) b57077af11 remote: fix remaining gdrive build issues (#3959)
* tests: fix gdrive build

* remote.gdrive: fix walk_files prefix relpath
2020-06-05 15:36:47 +09:00
Peter Rowlands (변기호) 803bb3eb61 remote: fix build errors (#3957) 2020-06-05 05:14:34 +03:00
Saugat Pachhai 99b58d3d05 refactor/test multistage load for params and outputs (#3949)
* refactor multistage load for params and outputs

* tests: load params

* tests: output loading from pipeline file

* fix test

* fix typo in name

* split params load

* rename params func s/inject_values/fill_values

* fix tests

* simplify loads_params and output.load_from_pipeline

* address @pared's suggestions for tests
2020-06-04 23:35:07 +03:00
nik123andRuslan Kuprieiev 5bbaa28cae Unify metrics behavior with dirs (#3935)
* run: disable directories for -m/-M flags

* run: reformatted string literals

* run: simplified metrics check

* tests: func: run: dirs as metrics outs

* tests: func: run: reformatted string literals

* tests: use pytest

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-04 23:32:00 +03:00
Peter Rowlands (변기호) 74bd8137ba remote: finish separating RemoteTree methods (#3946)
* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes
2020-06-04 23:22:31 +03:00
Ruslan Kuprieiev 72a576cefe dvc: bump to 1.0.0a9 2020-06-04 06:06:25 +03:00
Ruslan Kuprieiev 036ce9398e tests: move mock-ssh-server git dependency (#3953)
This one doesn't allow us to release new pre-releases, because pypi
forbids that:

https://travis-ci.com/github/iterative/dvc/jobs/343343967#L2447

Let's install it separately for now, until there is a new official
mock-ssh-server release.
2020-06-04 05:54:29 +03:00
Ruslan Kuprieiev 13012f9c0c plots: completion/spelling fixups (#3952)
* completion: don't forget --show-vega

Fixes #3890

* plots: VEGA -> Vega
2020-06-04 05:49:20 +03:00
Ruslan Kuprieiev a65c7989c8 tests: don't cache gdrive url (#3941)
This makes `GDrive.get_url` behaviour consistent with the rest of the
test remotes.

Related to #3916
2020-06-04 04:01:12 +03:00
Ruslan Kuprieiev 3590fe8414 dvc: bump to 1.0.0a8 2020-06-04 00:10:10 +03:00
Ruslan Kuprieiev 05100122e1 plots: remove print
Leftover from https://github.com/iterative/dvc/pull/3948

FIxes #3951
2020-06-04 00:09:16 +03:00
Alexander Schepanovski 9a55b909f8 Rename --xlab to --x-label, X_TITLE to X_LABEL, same for y (#3948)
Renamed:
- cli options
- dvcfile plot props
- template anchors
- variable names throughout the code
2020-06-03 20:30:17 +03:00
Ruslan Kuprieiev ff54ac1bfb plots: fix error when there are no plots in repo (#3940)
Consistent with `dvc metrics show/diff` behaviour.

Fixes #3925
2020-06-03 08:54:48 +03:00
Ruslan Kuprieiev 57e920aa39 tests: don't forget --external when adding external files 2020-06-03 08:38:51 +03:00
Ruslan Kuprieiev ac4f351daf dvc: require --external for external outputs (#3929)
Fixes #1545
2020-06-03 06:13:09 +03:00
Peter Rowlands (변기호) 2c0116ccf9 remote: separate tree functions from main remote classes (#3931)
* BaseRemote: start moving tree functions into BaseRemoteTree

* remote: add LocalRemoteTree

* remote: add cloud remote trees

* remote: move remove(), makedirs() into tree

* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree

* fix attributes for moved remote methods

* tests: update unit tests for moved remote tree methods

* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py

* fix attributes for moved remote tree functions

* tests: update func tests for moved remote tree methods

* test fixes

* fix DS warnings
2020-06-03 06:04:25 +03:00
Ruslan Kuprieiev c03ce4aa0f dag: fix help
per https://github.com/iterative/dvc.org/pull/1383#discussion_r433397776
2020-06-03 04:25:15 +03:00
Ruslan Kuprieiev 72dcbc0a9e remote: modify: fix --unset (#3939)
https://discordapp.com/channels/485586884165107732/563406153334128681/717474697721413686

Fixes https://github.com/iterative/dvc/issues/3938
2020-06-03 02:09:08 +03:00
Alexander Schepanovski fd65f44b7f Fix quadratic behavior in Repo._collect_stages() (#3932) 2020-06-02 16:56:49 +03:00
155 changed files with 5328 additions and 3322 deletions
-1
View File
@@ -1,7 +1,6 @@
---
name: "\U0001F41B Bug Report"
about: Create a bug report to help us improve DVC
labels: bug, triage
---
## Bug Report
@@ -1,6 +1,5 @@
---
name: "\U0001F680 Feature Request"
about: Suggest an idea for this project
labels: feature request, triage
---
+14 -3
View File
@@ -98,12 +98,15 @@ jobs:
- name: lxd
channel: stable
env:
- SNAPCRAFT_IMAGE_INFO: '{"build_url": "$TRAVIS_BUILD_URL"}'
- SNAPCRAFT_IMAGE_INFO: |
'{"build_url": "$TRAVIS_JOB_WEB_URL"}'
- SNAPCRAFT_BUILD_ENVIRONMENT: lxd
- SNAPCRAFT_BUILD_INFO: 1 # https://snapcraft.io/blog/introducing-developer-notifications-for-snap-security-updates
install:
- sudo usermod --append --groups lxd $USER
- sudo /snap/bin/lxd.migrate -yes
- sudo /snap/bin/lxd waitready
- sudo /snap/bin/lxd init --auto
- git fetch --tags
script:
- ./scripts/build_snap.sh
after_failure:
@@ -154,6 +157,14 @@ deploy:
script: "(echo $SNAP_TOKEN | snapcraft login --with -) && timeout 600 snapcraft push dvc_*.snap --release $SNAP_CHANNEL || echo timed out"
on:
all_branches: true
condition: "$(./scripts/ci/deploy_condition.sh dvc_*.snap) && ($TRAVIS_BRANCH = master || -n $TRAVIS_TAG)"
condition: "$(./scripts/ci/deploy_condition.sh dvc_*.snap) && ($TRAVIS_BRANCH = master || -n $TRAVIS_TAG) && -n $SNAP_CHANNEL"
repo: iterative/dvc
stage: build
- provider: script
skip_cleanup: true
script: "(echo $SNAP_TOKEN | snapcraft login --with -) && timeout 600 snapcraft push dvc_*.snap --release $SNAP_CHANNEL_MAJOR || echo timed out"
on:
all_branches: true
condition: "$(./scripts/ci/deploy_condition.sh dvc_*.snap) && ($TRAVIS_BRANCH = master || -n $TRAVIS_TAG) && -n $SNAP_CHANNEL_MAJOR"
repo: iterative/dvc
stage: build
+1 -1
View File
@@ -30,7 +30,7 @@ def get_url(path, repo=None, rev=None, remote=None):
raise UrlNotDvcRepoError(_repo.url)
out = _repo.find_out_by_relpath(path)
remote_obj = _repo.cloud.get_remote(remote)
return str(remote_obj.checksum_to_path_info(out.checksum))
return str(remote_obj.hash_to_path_info(out.checksum))
def open(path, repo=None, rev=None, remote=None, mode="r", encoding=None):
+8 -4
View File
@@ -28,13 +28,15 @@ def _make_remote_property(name):
"""
def getter(self):
from dvc.remote import Remote
from dvc.remote import get_cloud_tree
from dvc.remote.base import CloudCache
remote = self.config.get(name)
if not remote:
return None
return Remote(self.repo, name=remote)
tree = get_cloud_tree(self.repo, name=remote)
return CloudCache(tree)
getter.__name__ = name
return cached_property(getter)
@@ -50,7 +52,8 @@ class Cache:
CACHE_DIR = "cache"
def __init__(self, repo):
from dvc.remote import Remote
from dvc.remote import get_cloud_tree
from dvc.remote.local import LocalCache
self.repo = repo
self.config = config = repo.config["cache"]
@@ -62,7 +65,8 @@ class Cache:
else:
settings = {**config, "url": config["dir"]}
self.local = Remote(repo, **settings)
tree = get_cloud_tree(repo, **settings)
self.local = LocalCache(tree)
s3 = _make_remote_property("s3")
gs = _make_remote_property("gs")
+17 -2
View File
@@ -81,18 +81,33 @@ COMMANDS = [
class DvcParser(argparse.ArgumentParser):
"""Custom parser class for dvc CLI."""
def error(self, message):
def error(self, message, command=None):
"""Custom error method.
Args:
message (str): error message.
command (str): subcommand name for help message
Raises:
dvc.exceptions.DvcParser: dvc parser exception.
"""
logger.error(message)
if command is not None:
for action in self._actions:
if action.dest == "cmd" and command in action.choices:
subparser = action.choices[command]
subparser.print_help()
raise DvcParserError()
self.print_help()
raise DvcParserError()
# override this to send subcommand name to error method
def parse_args(self, args=None, namespace=None):
args, argv = self.parse_known_args(args, namespace)
if argv:
msg = "unrecognized arguments: %s"
self.error(msg % " ".join(argv), args.cmd)
return args
class VersionAction(argparse.Action): # pragma: no cover
# pylint: disable=too-few-public-methods
+7 -1
View File
@@ -18,6 +18,7 @@ class CmdAdd(CmdBase):
recursive=self.args.recursive,
no_commit=self.args.no_commit,
fname=self.args.file,
external=self.args.external,
)
except DvcException:
@@ -50,7 +51,12 @@ def add_parser(subparsers, parent_parser):
help="Don't put files/directories into cache.",
)
parser.add_argument(
"-f",
"--external",
action="store_true",
default=False,
help="Allow targets that are outside of the DVC repository.",
)
parser.add_argument(
"--file",
help="Specify name of the DVC-file this command will generate.",
metavar="<filename>",
+4 -3
View File
@@ -50,21 +50,22 @@ class CmdConfig(CmdBaseNoRepo):
parent_config_parser = argparse.ArgumentParser(add_help=False)
parent_config_parser.add_argument(
level_group = parent_config_parser.add_mutually_exclusive_group()
level_group.add_argument(
"--global",
dest="level",
action="store_const",
const="global",
help="Use global config.",
)
parent_config_parser.add_argument(
level_group.add_argument(
"--system",
dest="level",
action="store_const",
const="system",
help="Use system config.",
)
parent_config_parser.add_argument(
level_group.add_argument(
"--local",
dest="level",
action="store_const",
+3 -3
View File
@@ -73,7 +73,7 @@ class CmdDAG(CmdBase):
return 0
except DvcException:
msg = "failed to show"
msg = "failed to show "
if self.args.target:
msg += f"a pipeline for '{target}'"
else:
@@ -109,7 +109,7 @@ def add_parser(subparsers, parent_parser):
dag_parser.add_argument(
"target",
nargs="?",
help="Stage or output to show pipeline for. Optional. "
"(Finds all stages in the workspace by default.)",
help="Stage or output to show pipeline for (optional). "
"Finds all stages in the workspace by default.",
)
dag_parser.set_defaults(func=CmdDAG)
+6
View File
@@ -363,5 +363,11 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Show status of all stages in the specified directory.",
)
status_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show status in JSON format.",
)
status_parser.set_defaults(func=CmdDataStatus)
-1
View File
@@ -57,7 +57,6 @@ def add_parser(subparsers, parent_parser):
"out", nargs="?", help="Destination path to put files to."
)
import_parser.add_argument(
"-f",
"--file",
help="Specify name of the DVC-file this command will generate.",
metavar="<filename>",
+51 -21
View File
@@ -7,6 +7,9 @@ from dvc.exceptions import BadMetricError, DvcException
logger = logging.getLogger(__name__)
DEFAULT_PRECISION = 5
def show_metrics(
metrics, all_branches=False, all_tags=False, all_commits=False
):
@@ -63,11 +66,20 @@ class CmdMetricsShow(CmdBase):
return 0
def _show_diff(diff, markdown=False, no_path=False, old=False):
def _show_diff(diff, markdown=False, no_path=False, old=False, precision=None):
from collections import OrderedDict
from dvc.utils.diff import table
if precision is None:
precision = DEFAULT_PRECISION
def _round(val):
if isinstance(val, float):
return round(val, precision)
return val
rows = []
for fname, mdiff in diff.items():
sorted_mdiff = OrderedDict(sorted(mdiff.items()))
@@ -75,9 +87,9 @@ def _show_diff(diff, markdown=False, no_path=False, old=False):
row = [] if no_path else [fname]
row.append(metric)
if old:
row.append(change.get("old"))
row.append(change["new"])
row.append(change.get("diff", "diff not supported"))
row.append(_round(change.get("old")))
row.append(_round(change["new"]))
row.append(_round(change.get("diff", "diff not supported")))
rows.append(row)
header = [] if no_path else ["Path"]
@@ -108,7 +120,11 @@ class CmdMetricsDiff(CmdBase):
logger.info(json.dumps(diff))
else:
table = _show_diff(
diff, self.args.show_md, self.args.no_path, self.args.old
diff,
self.args.show_md,
self.args.no_path,
self.args.old,
precision=self.args.precision,
)
if table:
logger.info(table)
@@ -149,7 +165,10 @@ def add_parser(subparsers, parent_parser):
metrics_show_parser.add_argument(
"targets",
nargs="*",
help="Metric files or directories (see -R) to show",
help=(
"Limit command scope to these metric files. Using -R, "
"directories to search metric files in can also be given."
),
)
metrics_show_parser.add_argument(
"-a",
@@ -171,6 +190,12 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Show metrics for all commits.",
)
metrics_show_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show output in JSON format.",
)
metrics_show_parser.add_argument(
"-R",
"--recursive",
@@ -181,16 +206,12 @@ def add_parser(subparsers, parent_parser):
"metric files."
),
)
metrics_show_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show output in JSON format.",
)
metrics_show_parser.set_defaults(func=CmdMetricsShow)
METRICS_DIFF_HELP = "Show changes in metrics between commits"
" in the DVC repository, or between a commit and the workspace."
METRICS_DIFF_HELP = (
"Show changes in metrics between commits in the DVC repository, or "
"between a commit and the workspace."
)
metrics_diff_parser = metrics_subparsers.add_parser(
"diff",
parents=[parent_parser],
@@ -204,14 +225,14 @@ def add_parser(subparsers, parent_parser):
metrics_diff_parser.add_argument(
"b_rev",
nargs="?",
help=("New Git commit to compare (defaults to the current workspace)"),
help="New Git commit to compare (defaults to the current workspace)",
)
metrics_diff_parser.add_argument(
"--targets",
nargs="*",
help=(
"Metric files or directories (see -R) to show diff for. "
"Shows diff for all metric files by default."
"Limit command scope to these metric files. Using -R, "
"directories to search metric files in can also be given."
),
metavar="<paths>",
)
@@ -243,6 +264,12 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Show tabulated output in the Markdown format (GFM).",
)
metrics_diff_parser.add_argument(
"--old",
action="store_true",
default=False,
help="Show old metric value.",
)
metrics_diff_parser.add_argument(
"--no-path",
action="store_true",
@@ -250,9 +277,12 @@ def add_parser(subparsers, parent_parser):
help="Don't show metric path.",
)
metrics_diff_parser.add_argument(
"--old",
action="store_true",
default=False,
help="Show old metric value.",
"--precision",
type=int,
help=(
"Round metrics to `n` digits precision after the decimal point. "
f"Rounds to {DEFAULT_PRECISION} digits by default."
),
metavar="<n>",
)
metrics_diff_parser.set_defaults(func=CmdMetricsDiff)
+37 -18
View File
@@ -98,8 +98,8 @@ class CmdPlotsModify(CmdPlots):
def add_parser(subparsers, parent_parser):
PLOTS_HELP = (
"Generating plots for metrics stored in structured files "
"(JSON, CSV, TSV)."
"Commands to visualize and compare plot metrics in structured files "
"(JSON, YAML, CSV, TSV)"
)
plots_parser = subparsers.add_parser(
@@ -116,7 +116,7 @@ def add_parser(subparsers, parent_parser):
fix_subparsers(plots_subparsers)
SHOW_HELP = "Generate a plots image file from a metrics file."
SHOW_HELP = "Generate plots from metric files."
plots_show_parser = plots_subparsers.add_parser(
"show",
parents=[parent_parser],
@@ -134,8 +134,8 @@ def add_parser(subparsers, parent_parser):
plots_show_parser.set_defaults(func=CmdPlotsShow)
PLOTS_DIFF_HELP = (
"Plot differences in metrics between commits in the DVC "
"repository, or between the last commit and the workspace."
"Show multiple versions of plot metrics "
"by plotting them in a single image."
)
plots_diff_parser = plots_subparsers.add_parser(
"diff",
@@ -148,6 +148,7 @@ def add_parser(subparsers, parent_parser):
"--targets",
nargs="*",
help="Plots file to visualize. Shows all plots by default.",
metavar="<path>",
)
plots_diff_parser.add_argument(
"revisions", nargs="*", default=None, help="Git commits to plot from",
@@ -156,7 +157,7 @@ def add_parser(subparsers, parent_parser):
_add_output_arguments(plots_diff_parser)
plots_diff_parser.set_defaults(func=CmdPlotsDiff)
PLOTS_MODIFY_HELP = "Modify plot props associated with a target file."
PLOTS_MODIFY_HELP = "Modify display properties of plot metric files."
plots_modify_parser = plots_subparsers.add_parser(
"modify",
parents=[parent_parser],
@@ -165,11 +166,14 @@ def add_parser(subparsers, parent_parser):
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_modify_parser.add_argument(
"target", help="Plot file to set props to.",
"target", help="Metric file to set properties to",
)
_add_props_arguments(plots_modify_parser)
plots_modify_parser.add_argument(
"--unset", nargs="*", help="Props to unset.",
"--unset",
nargs="*",
metavar="<property>",
help="Unset one or more display properties.",
)
plots_modify_parser.set_defaults(func=CmdPlotsModify)
@@ -186,28 +190,43 @@ def _add_props_arguments(parser):
format_link("https://man.dvc.org/plots#plot-templates")
)
),
metavar="<path>",
)
parser.add_argument("-x", default=None, help="Field name for x axis.")
parser.add_argument("-y", default=None, help="Field name for y axis.")
parser.add_argument(
"--no-csv-header",
"-x", default=None, help="Field name for X axis.", metavar="<field>"
)
parser.add_argument(
"-y", default=None, help="Field name for Y axis.", metavar="<field>"
)
parser.add_argument(
"--no-header",
action="store_false",
dest="csv_header",
dest="header",
default=None, # Use default None to distinguish when it's not used
help="Provided CSV ot TSV datafile does not have a header.",
help="Provided CSV or TSV datafile does not have a header.",
)
parser.add_argument(
"--title", default=None, metavar="<text>", help="Plot title."
)
parser.add_argument(
"--x-label", default=None, help="X axis label", metavar="<text>"
)
parser.add_argument(
"--y-label", default=None, help="Y axis label", metavar="<text>"
)
parser.add_argument("--title", default=None, help="Plot title.")
parser.add_argument("--xlab", default=None, help="X axis title.")
parser.add_argument("--ylab", default=None, help="Y axis title.")
def _add_output_arguments(parser):
parser.add_argument(
"-o", "--out", default=None, help="Destination path to save plots to.",
"-o",
"--out",
default=None,
help="Destination path to save plots to",
metavar="<path>",
)
parser.add_argument(
"--show-vega",
action="store_true",
default=False,
help="Show output in VEGA format.",
help="Show output in Vega format.",
)
+5 -1
View File
@@ -62,7 +62,11 @@ class CmdRemoteModify(CmdRemote):
def run(self):
with self.config.edit(self.args.level) as conf:
self._check_exists(conf)
conf["remote"][self.args.name][self.args.option] = self.args.value
section = conf["remote"][self.args.name]
if self.args.unset:
section.pop(self.args.option, None)
else:
section[self.args.option] = self.args.value
return 0
+12 -6
View File
@@ -45,7 +45,7 @@ class CmdRun(CmdBase):
fname=self.args.file,
wdir=self.args.wdir,
no_exec=self.args.no_exec,
overwrite=self.args.overwrite,
force=self.args.force,
run_cache=not self.args.no_run_cache,
no_commit=self.args.no_commit,
outs_persist=self.args.outs_persist,
@@ -53,6 +53,7 @@ class CmdRun(CmdBase):
always_changed=self.args.always_changed,
name=self.args.name,
single_stage=self.args.single_stage,
external=self.args.external,
)
except DvcException:
logger.exception("")
@@ -132,7 +133,7 @@ def add_parser(subparsers, parent_parser):
"--metrics",
action="append",
default=[],
help="Declare output metric file or directory.",
help="Declare output metric file.",
metavar="<path>",
)
run_parser.add_argument(
@@ -140,8 +141,7 @@ def add_parser(subparsers, parent_parser):
"--metrics-no-cache",
action="append",
default=[],
help="Declare output metric file or directory "
"(do not put into DVC cache).",
help="Declare output metric file (do not put into DVC cache).",
metavar="<path>",
)
run_parser.add_argument(
@@ -159,7 +159,6 @@ def add_parser(subparsers, parent_parser):
metavar="<path>",
)
run_parser.add_argument(
"-f",
"--file",
help="Specify name of the DVC-file this command will generate.",
metavar="<filename>",
@@ -177,7 +176,8 @@ def add_parser(subparsers, parent_parser):
help="Only create stage file without actually running it.",
)
run_parser.add_argument(
"--overwrite",
"-f",
"--force",
action="store_true",
default=False,
help="Overwrite existing stage",
@@ -225,6 +225,12 @@ def add_parser(subparsers, parent_parser):
default=False,
help=argparse.SUPPRESS,
)
run_parser.add_argument(
"--external",
action="store_true",
default=False,
help="Allow outputs that are outside of the DVC repository.",
)
run_parser.add_argument(
"command", nargs=argparse.REMAINDER, help="Command to execute."
)
+10 -5
View File
@@ -50,11 +50,16 @@ class CmdDataStatus(CmdDataBase):
with_deps=self.args.with_deps,
recursive=self.args.recursive,
)
if st:
if self.args.quiet:
return 1
else:
self._show(st, indent)
if self.args.quiet:
return bool(st)
if self.args.show_json:
import json
logger.info(json.dumps(st))
elif st:
self._show(st, indent)
else:
logger.info(self.UP_TO_DATE_MSG)
+4 -4
View File
@@ -122,12 +122,12 @@ class CmdVersion(CmdBaseNoRepo):
@staticmethod
def get_supported_remotes():
from dvc.remote import REMOTES
from dvc.remote import TREES
supported_remotes = []
for remote in REMOTES:
if not remote.get_missing_deps():
supported_remotes.append(remote.scheme)
for tree_cls in TREES:
if not tree_cls.get_missing_deps():
supported_remotes.append(tree_cls.scheme)
return ", ".join(supported_remotes)
-1
View File
@@ -46,7 +46,6 @@ def _spawn_posix(cmd, env):
os._exit(1) # pylint: disable=protected-access
os.setsid()
os.umask(0)
try:
pid = os.fork()
+5 -5
View File
@@ -3,7 +3,7 @@
import logging
from dvc.config import NoRemoteError
from dvc.remote import Remote
from dvc.remote import get_remote
logger = logging.getLogger(__name__)
@@ -45,7 +45,7 @@ class DataCloud:
raise NoRemoteError(error_msg)
def _init_remote(self, name):
return Remote(self.repo, name=name)
return get_remote(self.repo, name=name)
def push(
self, cache, jobs=None, remote=None, show_checksums=False,
@@ -85,15 +85,15 @@ class DataCloud:
cache, jobs=jobs, remote=remote, show_checksums=show_checksums
)
if not remote.verify:
if not remote.tree.verify:
self._save_pulled_checksums(cache)
return downloaded_items_num
def _save_pulled_checksums(self, cache):
for checksum in cache.scheme_keys("local"):
cache_file = self.repo.cache.local.checksum_to_path_info(checksum)
if self.repo.cache.local.exists(cache_file):
cache_file = self.repo.cache.local.hash_to_path_info(checksum)
if self.repo.cache.local.tree.exists(cache_file):
# We can safely save here, as existing corrupted files will
# be removed upon status, while files corrupted during
# download will not be moved from tmp_file
+26 -22
View File
@@ -12,7 +12,7 @@ from dvc.dependency.param import ParamsDependency
from dvc.dependency.s3 import S3Dependency
from dvc.dependency.ssh import SSHDependency
from dvc.output.base import BaseOutput
from dvc.remote import Remote
from dvc.remote import get_remote
from dvc.scheme import Schemes
from .repo import RepoDependency
@@ -54,7 +54,7 @@ SCHEMA.update(ParamsDependency.PARAM_SCHEMA)
def _get(stage, p, info):
parsed = urlparse(p) if p else None
if parsed and parsed.scheme == "remote":
remote = Remote(stage.repo, name=parsed.netloc)
remote = get_remote(stage.repo, name=parsed.netloc)
return DEP_MAP[remote.scheme](stage, p, info, remote=remote)
if info and info.get(RepoDependency.PARAM_REPO):
@@ -80,6 +80,7 @@ def loadd_from(stage, d_list):
def loads_from(stage, s_list, erepo=None):
assert isinstance(s_list, list)
ret = []
for s in s_list:
info = {RepoDependency.PARAM_REPO: erepo} if erepo else {}
@@ -87,26 +88,29 @@ def loads_from(stage, s_list, erepo=None):
return ret
def _parse_params(path_params):
path, _, params_str = path_params.rpartition(":")
params = params_str.split(",")
return path, params
def _merge_params(s_list):
d = defaultdict(list)
default_file = ParamsDependency.DEFAULT_PARAMS_FILE
for key in s_list:
if isinstance(key, str):
d[default_file].append(key)
continue
if not isinstance(key, dict):
msg = "Only list of str/dict is supported. Got: "
msg += f"'{type(key).__name__}'."
raise ValueError(msg)
for k, params in key.items():
if not isinstance(params, list):
msg = "Expected list of params for custom params file "
msg += f"'{k}', got '{type(params).__name__}'."
raise ValueError(msg)
d[k].extend(params)
return d
def loads_params(stage, s_list):
# Creates an object for each unique file that is referenced in the list
params_by_path = defaultdict(list)
for s in s_list:
path, params = _parse_params(s)
params_by_path[path].extend(params)
d_list = []
for path, params in params_by_path.items():
d_list.append(
{
BaseOutput.PARAM_PATH: path,
ParamsDependency.PARAM_PARAMS: params,
}
)
return loadd_from(stage, d_list)
d = _merge_params(s_list)
return [
ParamsDependency(stage, path, params) for path, params in d.items()
]
+2 -2
View File
@@ -1,7 +1,7 @@
from dvc.dependency.base import BaseDependency
from dvc.output.base import BaseOutput
from dvc.remote.azure import AzureRemote
from dvc.remote.azure import AzureRemoteTree
class AzureDependency(BaseDependency, BaseOutput):
REMOTE = AzureRemote
TREE_CLS = AzureRemoteTree
+2 -2
View File
@@ -1,7 +1,7 @@
from dvc.dependency.base import BaseDependency
from dvc.output.base import BaseOutput
from dvc.remote.http import HTTPRemote
from dvc.remote.http import HTTPRemoteTree
class HTTPDependency(BaseDependency, BaseOutput):
REMOTE = HTTPRemote
TREE_CLS = HTTPRemoteTree
+2 -2
View File
@@ -1,7 +1,7 @@
from dvc.remote.https import HTTPSRemote
from dvc.remote.https import HTTPSRemoteTree
from .http import HTTPDependency
class HTTPSDependency(HTTPDependency):
REMOTE = HTTPSRemote
TREE_CLS = HTTPSRemoteTree
+1 -1
View File
@@ -40,7 +40,7 @@ class ParamsDependency(LocalDependency):
info=info,
)
def _dyn_load(self, values=None):
def fill_values(self, values=None):
"""Load params values dynamically."""
if not values:
return
+2 -4
View File
@@ -64,10 +64,8 @@ class RepoDependency(LocalDependency):
# We are polluting our repo cache with some dir listing here
if tree.isdir(path):
return self.repo.cache.local.get_dir_checksum(
path, tree=tree
)
return tree.get_file_checksum(path)
return self.repo.cache.local.get_hash(path, tree)
return tree.get_file_hash(path)
def status(self):
current_checksum = self._get_checksum(locked=True)
+9 -23
View File
@@ -4,11 +4,9 @@ import os
from voluptuous import MultipleInvalid
import dvc.prompt as prompt
from dvc import serialize
from dvc.exceptions import DvcException
from dvc.stage import serialize
from dvc.stage.exceptions import (
StageFileAlreadyExistsError,
StageFileBadNameError,
StageFileDoesNotExistError,
StageFileFormatError,
@@ -109,9 +107,6 @@ class FileMixin:
except MultipleInvalid as exc:
raise StageFileFormatError(f"'{fname}' format error: {exc}")
def remove_with_prompt(self, force=False):
raise NotImplementedError
def remove(self, force=False):
with contextlib.suppress(FileNotFoundError):
os.unlink(self.path)
@@ -148,19 +143,6 @@ class SingleStageFile(FileMixin):
dump_yaml(self.path, serialize.to_single_stage_file(stage))
self.repo.scm.track_file(self.relpath)
def remove_with_prompt(self, force=False):
if not self.exists():
return
msg = (
"'{}' already exists. Do you wish to run the command and "
"overwrite it?".format(relpath(self.path))
)
if not (force or prompt.confirm(msg)):
raise StageFileAlreadyExistsError(self.path)
self.remove()
def remove_stage(self, stage):
self.remove()
@@ -201,15 +183,19 @@ class PipelineFile(FileMixin):
data["stages"] = data.get("stages", {})
stage_data = serialize.to_pipeline_file(stage)
if data["stages"].get(stage.name):
existing_entry = stage.name in data["stages"]
action = "Modifying" if existing_entry else "Adding"
logger.info("%s stage '%s' in '%s'", action, stage.name, self.relpath)
if existing_entry:
orig_stage_data = data["stages"][stage.name]
if "meta" in orig_stage_data:
stage_data[stage.name]["meta"] = orig_stage_data["meta"]
apply_diff(stage_data[stage.name], orig_stage_data)
else:
data["stages"].update(stage_data)
logger.info(
"Adding stage '%s' to '%s'", stage.name, self.relpath,
)
dump_yaml(self.path, data)
self.repo.scm.track_file(self.relpath)
+13 -2
View File
@@ -178,6 +178,14 @@ class NoMetricsError(DvcException):
)
class NoPlotsError(DvcException):
def __init__(self):
super().__init__(
"no plots in this repository. Use `--plots/--plots-no-cache` "
"options for `dvc run` to mark stage outputs as plots."
)
class YAMLFileCorruptedError(DvcException):
def __init__(self, path):
path = relpath(path)
@@ -214,9 +222,12 @@ class ETagMismatchError(DvcException):
class FileMissingError(DvcException):
def __init__(self, path):
def __init__(self, path, hint=None):
self.path = path
super().__init__(f"Can't find '{path}' neither locally nor on remote")
hint = "" if hint is None else f". {hint}"
super().__init__(
f"Can't find '{path}' neither locally nor on remote{hint}"
)
class DvcIgnoreInCollectedDirError(DvcException):
+2 -1
View File
@@ -126,8 +126,9 @@ class BaseExternalRepo:
raise PathMissingError(path, self.url)
save_info = self.local_cache.save(
path,
self.repo_tree,
None,
tree=self.repo_tree,
save_link=False,
download_callback=download_update,
)
save_infos.append(save_info)
+26 -6
View File
@@ -1,12 +1,15 @@
import logging
import os
import re
from itertools import groupby
from funcy import cached_property
from pathspec import PathSpec
from pathspec.patterns import GitWildMatchPattern
from pathspec.util import normalize_file
from dvc.path_info import PathInfo
from dvc.scm.tree import BaseTree
from dvc.system import System
from dvc.utils import relpath
logger = logging.getLogger(__name__)
@@ -27,7 +30,16 @@ class DvcIgnorePatterns(DvcIgnore):
self.dirname = os.path.normpath(os.path.dirname(ignore_file_path))
with tree.open(ignore_file_path, encoding="utf-8") as fobj:
self.ignore_spec = PathSpec.from_lines(GitWildMatchPattern, fobj)
path_spec_lines = fobj.readlines()
regex_pattern_list = map(
GitWildMatchPattern.pattern_to_regex, path_spec_lines
)
self.ignore_spec = [
(ignore, re.compile("|".join(item[0] for item in group)))
for ignore, group in groupby(
regex_pattern_list, lambda x: x[1]
)
]
def __call__(self, root, dirs, files):
files = [f for f in files if not self.matches(root, f)]
@@ -48,7 +60,16 @@ class DvcIgnorePatterns(DvcIgnore):
else:
return False
return self.ignore_spec.match_file(path)
if not System.is_unix():
path = normalize_file(path)
return self.ignore(path)
def ignore(self, path):
result = False
for ignore, pattern in self.ignore_spec:
if pattern.match(path):
result = ignore
return result
def __hash__(self):
return hash(self.ignore_file_path)
@@ -186,8 +207,7 @@ class CleanTree(BaseTree):
if path.parent == self.tree_root or Repo.DVC_DIR in path.parts:
return True
# if path is outside of tree, assume this is a local remote/local cache
# link/move operation where we do not need to filter ignores
# paths outside of the CleanTree root should be ignored
path = relpath(path, self.tree_root)
if path.startswith("..") or (
os.name == "nt"
@@ -195,7 +215,7 @@ class CleanTree(BaseTree):
[os.path.abspath(path), self.tree_root]
)
):
return True
return False
# check if parent directories are in our ignores, starting from
# tree_root
+55 -9
View File
@@ -1,5 +1,7 @@
from collections import defaultdict
from urllib.parse import urlparse
from funcy import collecting, project
from voluptuous import And, Any, Coerce, Length, Lower, Required, SetTo
from dvc.output.base import BaseOutput
@@ -8,10 +10,10 @@ from dvc.output.hdfs import HDFSOutput
from dvc.output.local import LocalOutput
from dvc.output.s3 import S3Output
from dvc.output.ssh import SSHOutput
from dvc.remote import Remote
from dvc.remote.hdfs import HDFSRemote
from dvc.remote.local import LocalRemote
from dvc.remote.s3 import S3Remote
from dvc.remote import get_remote
from dvc.remote.hdfs import HDFSRemoteTree
from dvc.remote.local import LocalRemoteTree
from dvc.remote.s3 import S3RemoteTree
from dvc.scheme import Schemes
OUTS = [
@@ -45,9 +47,9 @@ CHECKSUM_SCHEMA = Any(
# so when a few types of outputs share the same name, we only need
# specify it once.
CHECKSUMS_SCHEMA = {
LocalRemote.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
S3Remote.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
HDFSRemote.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
LocalRemoteTree.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
S3RemoteTree.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
HDFSRemoteTree.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
}
SCHEMA = CHECKSUMS_SCHEMA.copy()
@@ -58,11 +60,13 @@ SCHEMA[BaseOutput.PARAM_PLOT] = bool
SCHEMA[BaseOutput.PARAM_PERSIST] = bool
def _get(stage, p, info, cache, metric, plot=False, persist=False):
def _get(
stage, p, info=None, cache=True, metric=False, plot=False, persist=False
):
parsed = urlparse(p)
if parsed.scheme == "remote":
remote = Remote(stage.repo, name=parsed.netloc)
remote = get_remote(stage.repo, name=parsed.netloc)
return OUTS_MAP[remote.scheme](
stage,
p,
@@ -135,3 +139,45 @@ def loads_from(
)
for s in s_list
]
def _split_dict(d, keys):
return project(d, keys), project(d, d.keys() - keys)
def _merge_data(s_list):
d = defaultdict(dict)
for key in s_list:
if isinstance(key, str):
d[key].update({})
continue
if not isinstance(key, dict):
raise ValueError(f"'{type(key).__name__}' not supported.")
for k, flags in key.items():
if not isinstance(flags, dict):
raise ValueError(
f"Expected dict for '{k}', got: '{type(flags).__name__}'"
)
d[k].update(flags)
return d
@collecting
def load_from_pipeline(stage, s_list, typ="outs"):
if typ not in (stage.PARAM_OUTS, stage.PARAM_METRICS, stage.PARAM_PLOTS):
raise ValueError(f"'{typ}' key is not allowed for pipeline files.")
metric = typ == stage.PARAM_METRICS
plot = typ == stage.PARAM_PLOTS
d = _merge_data(s_list)
for path, flags in d.items():
plt_d = {}
if plot:
from dvc.schema import PLOT_PROPS
plt_d, flags = _split_dict(flags, keys=PLOT_PROPS.keys())
extra = project(flags, ["cache", "persist"])
yield _get(stage, path, {}, plot=plt_d or plot, metric=metric, **extra)
+34 -28
View File
@@ -12,7 +12,7 @@ from dvc.exceptions import (
DvcException,
RemoteCacheRequiredError,
)
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree, Remote
logger = logging.getLogger(__name__)
@@ -47,7 +47,8 @@ class OutputIsStageFileError(DvcException):
class BaseOutput:
IS_DEPENDENCY = False
REMOTE = BaseRemote
REMOTE_CLS = Remote
TREE_CLS = BaseRemoteTree
PARAM_PATH = "path"
PARAM_CACHE = "cache"
@@ -58,10 +59,10 @@ class BaseOutput:
PARAM_PLOT_TEMPLATE = "template"
PARAM_PLOT_X = "x"
PARAM_PLOT_Y = "y"
PARAM_PLOT_XLAB = "xlab"
PARAM_PLOT_YLAB = "ylab"
PARAM_PLOT_X_LABEL = "x_label"
PARAM_PLOT_Y_LABEL = "y_label"
PARAM_PLOT_TITLE = "title"
PARAM_PLOT_CSV_HEADER = "csv_header"
PARAM_PLOT_HEADER = "header"
PARAM_PERSIST = "persist"
METRIC_SCHEMA = Any(
@@ -105,7 +106,11 @@ class BaseOutput:
self.repo = stage.repo if stage else None
self.def_path = path
self.info = info
self.remote = remote or self.REMOTE(self.repo, {})
if remote:
self.remote = remote
else:
tree = self.TREE_CLS(self.repo, {})
self.remote = self.REMOTE_CLS(tree)
self.use_cache = False if self.IS_DEPENDENCY else cache
self.metric = False if self.IS_DEPENDENCY else metric
self.plot = False if self.IS_DEPENDENCY else plot
@@ -119,7 +124,7 @@ class BaseOutput:
if remote:
parsed = urlparse(path)
return remote.path_info / parsed.path.lstrip("/")
return self.REMOTE.path_cls(path)
return self.TREE_CLS.PATH_CLS(path)
def __repr__(self):
return "{class_name}: '{def_path}'".format(
@@ -131,7 +136,7 @@ class BaseOutput:
@property
def scheme(self):
return self.REMOTE.scheme
return self.TREE_CLS.scheme
@property
def is_in_repo(self):
@@ -154,34 +159,34 @@ class BaseOutput:
@classmethod
def supported(cls, url):
return cls.REMOTE.supported(url)
return cls.TREE_CLS.supported(url)
@property
def cache_path(self):
return self.cache.checksum_to_path_info(self.checksum).url
return self.cache.hash_to_path_info(self.checksum).url
@property
def checksum_type(self):
return self.remote.PARAM_CHECKSUM
return self.remote.tree.PARAM_CHECKSUM
@property
def checksum(self):
return self.info.get(self.remote.PARAM_CHECKSUM)
return self.info.get(self.remote.tree.PARAM_CHECKSUM)
@checksum.setter
def checksum(self, checksum):
self.info[self.remote.PARAM_CHECKSUM] = checksum
self.info[self.remote.tree.PARAM_CHECKSUM] = checksum
def get_checksum(self):
return self.remote.get_checksum(self.path_info)
return self.remote.get_hash(self.path_info)
@property
def is_dir_checksum(self):
return self.remote.is_dir_checksum(self.checksum)
return self.remote.is_dir_hash(self.checksum)
@property
def exists(self):
return self.remote.exists(self.path_info)
return self.remote.tree.exists(self.path_info)
def save_info(self):
return self.remote.save_info(self.path_info)
@@ -217,13 +222,13 @@ class BaseOutput:
@property
def is_empty(self):
return self.remote.is_empty(self.path_info)
return self.remote.tree.is_empty(self.path_info)
def isdir(self):
return self.remote.isdir(self.path_info)
return self.remote.tree.isdir(self.path_info)
def isfile(self):
return self.remote.isfile(self.path_info)
return self.remote.tree.isfile(self.path_info)
def ignore(self):
if not self.use_scm_ignore:
@@ -246,10 +251,11 @@ class BaseOutput:
self.ignore()
if self.metric or self.plot:
self.verify_metric()
if not self.use_cache:
self.info = self.save_info()
if self.metric or self.plot:
self.verify_metric()
if not self.IS_DEPENDENCY:
logger.debug(
"Output '%s' doesn't use cache. Skipping saving.", self
@@ -266,7 +272,7 @@ class BaseOutput:
def commit(self):
if self.use_cache:
self.cache.save(self.path_info, self.info)
self.cache.save(self.path_info, self.cache.tree, self.info)
def dumpd(self):
ret = copy(self.info)
@@ -300,7 +306,7 @@ class BaseOutput:
raise DvcException(f"verify metric is not supported for {self.scheme}")
def download(self, to):
self.remote.download(self.path_info, to.path_info)
self.remote.tree.download(self.path_info, to.path_info)
def checkout(
self,
@@ -326,7 +332,7 @@ class BaseOutput:
)
def remove(self, ignore_remove=False):
self.remote.remove(self.path_info)
self.remote.tree.remove(self.path_info)
if self.scheme != "local":
return
@@ -337,7 +343,7 @@ class BaseOutput:
if self.scheme == "local" and self.use_scm_ignore:
self.repo.scm.ignore_remove(self.fspath)
self.remote.move(self.path_info, out.path_info)
self.remote.tree.move(self.path_info, out.path_info)
self.def_path = out.def_path
self.path_info = out.path_info
self.save()
@@ -356,7 +362,7 @@ class BaseOutput:
def unprotect(self):
if self.exists:
self.remote.unprotect(self.path_info)
self.remote.tree.unprotect(self.path_info)
def get_dir_cache(self, **kwargs):
if not self.is_dir_checksum:
@@ -416,8 +422,8 @@ class BaseOutput:
filter_path = str(filter_info) if filter_info else None
is_win = os.name == "nt"
for entry in self.dir_cache:
checksum = entry[self.remote.PARAM_CHECKSUM]
entry_relpath = entry[self.remote.PARAM_RELPATH]
checksum = entry[self.remote.tree.PARAM_CHECKSUM]
entry_relpath = entry[self.remote.tree.PARAM_RELPATH]
if is_win:
entry_relpath = entry_relpath.replace("/", os.sep)
entry_path = os.path.join(path, entry_relpath)
+2 -2
View File
@@ -1,6 +1,6 @@
from dvc.output.s3 import S3Output
from dvc.remote.gs import GSRemote
from dvc.remote.gs import GSRemoteTree
class GSOutput(S3Output):
REMOTE = GSRemote
TREE_CLS = GSRemoteTree
+2 -2
View File
@@ -1,6 +1,6 @@
from dvc.output.base import BaseOutput
from dvc.remote.hdfs import HDFSRemote
from dvc.remote.hdfs import HDFSRemoteTree
class HDFSOutput(BaseOutput):
REMOTE = HDFSRemote
TREE_CLS = HDFSRemoteTree
+8 -7
View File
@@ -5,7 +5,7 @@ from urllib.parse import urlparse
from dvc.exceptions import DvcException
from dvc.istextfile import istextfile
from dvc.output.base import BaseOutput
from dvc.remote.local import LocalRemote
from dvc.remote.local import LocalRemote, LocalRemoteTree
from dvc.utils import relpath
from dvc.utils.fs import path_isin
@@ -13,7 +13,8 @@ logger = logging.getLogger(__name__)
class LocalOutput(BaseOutput):
REMOTE = LocalRemote
REMOTE_CLS = LocalRemote
TREE_CLS = LocalRemoteTree
sep = os.sep
def __init__(self, stage, path, *args, **kwargs):
@@ -33,15 +34,15 @@ class LocalOutput(BaseOutput):
#
# FIXME: if we have Windows path containing / or posix one with \
# then we have #2059 bug and can't really handle that.
p = self.REMOTE.path_cls(path)
if not p.is_absolute():
p = self.TREE_CLS.PATH_CLS(path)
if self.stage and not p.is_absolute():
p = self.stage.wdir / p
abs_p = os.path.abspath(os.path.normpath(p))
return self.REMOTE.path_cls(abs_p)
return self.TREE_CLS.PATH_CLS(abs_p)
def __str__(self):
if not self.is_in_repo:
if not self.repo or not self.is_in_repo:
return str(self.def_path)
cur_dir = os.getcwd()
@@ -81,7 +82,7 @@ class LocalOutput(BaseOutput):
name = "metrics" if self.metric else "plot"
if os.path.isdir(path):
msg = "directory '{}' cannot be used as {}."
raise DvcException(msg.format(self.path_info, name))
raise IsADirectoryError(msg.format(self.path_info, name))
if not istextfile(path):
msg = "binary file '{}' cannot be used as {}."
+2 -2
View File
@@ -1,6 +1,6 @@
from dvc.output.base import BaseOutput
from dvc.remote.s3 import S3Remote
from dvc.remote.s3 import S3RemoteTree
class S3Output(BaseOutput):
REMOTE = S3Remote
TREE_CLS = S3RemoteTree
+3 -2
View File
@@ -1,6 +1,7 @@
from dvc.output.base import BaseOutput
from dvc.remote.ssh import SSHRemote
from dvc.remote.ssh import SSHRemote, SSHRemoteTree
class SSHOutput(BaseOutput):
REMOTE = SSHRemote
REMOTE_CLS = SSHRemote
TREE_CLS = SSHRemoteTree
+43 -29
View File
@@ -1,46 +1,46 @@
import posixpath
from urllib.parse import urlparse
from dvc.remote.azure import AzureRemote
from dvc.remote.gdrive import GDriveRemote
from dvc.remote.gs import GSRemote
from dvc.remote.hdfs import HDFSRemote
from dvc.remote.http import HTTPRemote
from dvc.remote.https import HTTPSRemote
from dvc.remote.local import LocalRemote
from dvc.remote.oss import OSSRemote
from dvc.remote.s3 import S3Remote
from dvc.remote.ssh import SSHRemote
from dvc.remote.azure import AzureRemoteTree
from dvc.remote.base import Remote
from dvc.remote.gdrive import GDriveRemoteTree
from dvc.remote.gs import GSRemoteTree
from dvc.remote.hdfs import HDFSRemoteTree
from dvc.remote.http import HTTPRemoteTree
from dvc.remote.https import HTTPSRemoteTree
from dvc.remote.local import LocalRemote, LocalRemoteTree
from dvc.remote.oss import OSSRemoteTree
from dvc.remote.s3 import S3RemoteTree
from dvc.remote.ssh import SSHRemote, SSHRemoteTree
REMOTES = [
AzureRemote,
GDriveRemote,
GSRemote,
HDFSRemote,
HTTPRemote,
HTTPSRemote,
S3Remote,
SSHRemote,
OSSRemote,
# NOTE: LocalRemote is the default
TREES = [
AzureRemoteTree,
GDriveRemoteTree,
GSRemoteTree,
HDFSRemoteTree,
HTTPRemoteTree,
HTTPSRemoteTree,
S3RemoteTree,
SSHRemoteTree,
OSSRemoteTree,
# NOTE: LocalRemoteTree is the default
]
def _get(remote_conf):
for remote in REMOTES:
if remote.supported(remote_conf):
return remote
return LocalRemote
def _get_tree(remote_conf):
for tree_cls in TREES:
if tree_cls.supported(remote_conf):
return tree_cls
return LocalRemoteTree
def Remote(repo, **kwargs):
def _get_conf(repo, **kwargs):
name = kwargs.get("name")
if name:
remote_conf = repo.config["remote"][name.lower()]
else:
remote_conf = kwargs
remote_conf = _resolve_remote_refs(repo.config, remote_conf)
return _get(remote_conf)(repo, remote_conf)
return _resolve_remote_refs(repo.config, remote_conf)
def _resolve_remote_refs(config, remote_conf):
@@ -72,3 +72,17 @@ def _resolve_remote_refs(config, remote_conf):
base = config["remote"][parsed.netloc]
url = posixpath.join(base["url"], parsed.path.lstrip("/"))
return {**base, **remote_conf, "url": url}
def get_cloud_tree(repo, **kwargs):
remote_conf = _get_conf(repo, **kwargs)
return _get_tree(remote_conf)(repo, remote_conf)
def get_remote(repo, **kwargs):
tree = get_cloud_tree(repo, **kwargs)
if tree.scheme == "local":
return LocalRemote(tree)
if tree.scheme == "ssh":
return SSHRemote(tree)
return Remote(tree)
+41 -46
View File
@@ -1,6 +1,5 @@
import logging
import os
import posixpath
import threading
from datetime import datetime, timedelta
@@ -8,15 +7,15 @@ from funcy import cached_property, wrap_prop
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
class AzureRemote(BaseRemote):
class AzureRemoteTree(BaseRemoteTree):
scheme = Schemes.AZURE
path_cls = CloudURLInfo
PATH_CLS = CloudURLInfo
REQUIRES = {"azure-storage-blob": "azure.storage.blob"}
PARAM_CHECKSUM = "etag"
COPY_POLL_SECONDS = 5
@@ -26,11 +25,11 @@ class AzureRemote(BaseRemote):
super().__init__(repo, config)
url = config.get("url", "azure://")
self.path_info = self.path_cls(url)
self.path_info = self.PATH_CLS(url)
if not self.path_info.bucket:
container = os.getenv("AZURE_STORAGE_CONTAINER_NAME")
self.path_info = self.path_cls(f"azure://{container}")
self.path_info = self.PATH_CLS(f"azure://{container}")
self.connection_string = config.get("connection_string") or os.getenv(
"AZURE_STORAGE_CONNECTION_STRING"
@@ -62,17 +61,27 @@ class AzureRemote(BaseRemote):
).properties.etag
return etag.strip('"')
def get_file_checksum(self, path_info):
return self.get_etag(path_info)
def _generate_download_url(self, path_info, expires=3600):
from azure.storage.blob import BlobPermissions
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
expires_at = datetime.utcnow() + timedelta(seconds=expires)
logger.debug(f"Removing {path_info}")
self.blob_service.delete_blob(path_info.bucket, path_info.path)
sas_token = self.blob_service.generate_blob_shared_access_signature(
path_info.bucket,
path_info.path,
permission=BlobPermissions.READ,
expiry=expires_at,
)
download_url = self.blob_service.make_blob_url(
path_info.bucket, path_info.path, sas_token=sas_token
)
return download_url
def _list_paths(self, bucket, prefix, progress_callback=None):
def exists(self, path_info):
paths = self._list_paths(path_info.bucket, path_info.path)
return any(path_info.path == path for path in paths)
def _list_paths(self, bucket, prefix):
blob_service = self.blob_service
next_marker = None
while True:
@@ -81,8 +90,6 @@ class AzureRemote(BaseRemote):
)
for blob in blobs:
if progress_callback:
progress_callback()
yield blob.name
if not blobs.next_marker:
@@ -90,16 +97,24 @@ class AzureRemote(BaseRemote):
next_marker = blobs.next_marker
def list_cache_paths(self, prefix=None, progress_callback=None):
if prefix:
prefix = posixpath.join(
self.path_info.path, prefix[:2], prefix[2:]
)
else:
prefix = self.path_info.path
return self._list_paths(
self.path_info.bucket, prefix, progress_callback
)
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(
path_info.bucket, path_info.path, **kwargs
):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
logger.debug(f"Removing {path_info}")
self.blob_service.delete_blob(path_info.bucket, path_info.path)
def get_file_hash(self, path_info):
return self.get_etag(path_info)
def _upload(
self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
@@ -122,23 +137,3 @@ class AzureRemote(BaseRemote):
to_file,
progress_callback=pbar.update_to,
)
def exists(self, path_info):
paths = self._list_paths(path_info.bucket, path_info.path)
return any(path_info.path == path for path in paths)
def _generate_download_url(self, path_info, expires=3600):
from azure.storage.blob import BlobPermissions
expires_at = datetime.utcnow() + timedelta(seconds=expires)
sas_token = self.blob_service.generate_blob_shared_access_signature(
path_info.bucket,
path_info.path,
permission=BlobPermissions.READ,
expiry=expires_at,
)
download_url = self.blob_service.make_blob_url(
path_info.bucket, path_info.path, sas_token=sas_token
)
return download_url
+856 -797
View File
File diff suppressed because it is too large Load Diff
+62 -44
View File
@@ -1,38 +1,35 @@
import io
import logging
import os
import posixpath
import re
import threading
from collections import defaultdict
from contextlib import contextmanager
from urllib.parse import urlparse
from funcy import cached_property, retry, wrap_prop, wrap_with
from funcy.py3 import cat
from dvc.exceptions import DvcException
from dvc.exceptions import DvcException, FileMissingError
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
from dvc.utils import format_link, tmp_fname
from dvc.utils.stream import IterStream
logger = logging.getLogger(__name__)
FOLDER_MIME_TYPE = "application/vnd.google-apps.folder"
class GDrivePathNotFound(DvcException):
def __init__(self, path_info, hint):
hint = "" if hint is None else f" {hint}"
super().__init__(f"GDrive path '{path_info}' not found.{hint}")
class GDriveAuthError(DvcException):
def __init__(self, cred_location):
if cred_location:
message = (
"GDrive remote auth failed with credentials in '{}'.\n"
"Backup first, remove of fix them, and run DVC again.\n"
"Backup first, remove or fix them, and run DVC again.\n"
"It should do auth again and refresh the credentials.\n\n"
"Details:".format(cred_location)
)
@@ -87,9 +84,9 @@ class GDriveURLInfo(CloudURLInfo):
self._spath = re.sub("/{2,}", "/", self._spath.rstrip("/"))
class GDriveRemote(BaseRemote):
class GDriveRemoteTree(BaseRemoteTree):
scheme = Schemes.GDRIVE
path_cls = GDriveURLInfo
PATH_CLS = GDriveURLInfo
REQUIRES = {"pydrive2": "pydrive2"}
DEFAULT_VERIFY = True
# Always prefer traverse for GDrive since API usage quotas are a concern.
@@ -104,7 +101,8 @@ class GDriveRemote(BaseRemote):
def __init__(self, repo, config):
super().__init__(repo, config)
self.path_info = self.path_cls(config["url"])
self.path_info = self.PATH_CLS(config["url"])
if not self.path_info.bucket:
raise DvcException(
@@ -131,11 +129,11 @@ class GDriveRemote(BaseRemote):
self._validate_config()
self._gdrive_user_credentials_path = (
tmp_fname(os.path.join(self.repo.tmp_dir, ""))
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA)
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA)
else config.get(
"gdrive_user_credentials_file",
os.path.join(
self.repo.tmp_dir, self.DEFAULT_USER_CREDENTIALS_FILE
self.repo.tmp_dir, self.DEFAULT_USER_CREDENTIALS_FILE,
),
)
)
@@ -173,8 +171,8 @@ class GDriveRemote(BaseRemote):
Useful for tests, exception messages, etc. Returns either env variable
name if it's set or actual path to the credentials file.
"""
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
return GDriveRemote.GDRIVE_CREDENTIALS_DATA
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
return GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA
if os.path.exists(self._gdrive_user_credentials_path):
return self._gdrive_user_credentials_path
return None
@@ -188,7 +186,7 @@ class GDriveRemote(BaseRemote):
DVC config client id or secret but forgets to remove the cached
credentials file.
"""
if not os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
if not os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
if (
settings["client_config"]["client_id"]
!= auth.credentials.client_id
@@ -211,10 +209,10 @@ class GDriveRemote(BaseRemote):
from pydrive2.auth import GoogleAuth
from pydrive2.drive import GoogleDrive
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
with open(self._gdrive_user_credentials_path, "w") as cred_file:
cred_file.write(
os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA)
os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA)
)
auth_settings = {
@@ -261,7 +259,7 @@ class GDriveRemote(BaseRemote):
gauth.ServiceAuth()
else:
gauth.CommandLineAuth()
GDriveRemote._validate_credentials(gauth, auth_settings)
GDriveRemoteTree._validate_credentials(gauth, auth_settings)
# Handle AuthenticationError, RefreshError and other auth failures
# It's hard to come up with a narrow exception, since PyDrive throws
@@ -270,7 +268,7 @@ class GDriveRemote(BaseRemote):
except Exception as exc:
raise GDriveAuthError(self.credentials_location) from exc
finally:
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
os.remove(self._gdrive_user_credentials_path)
return GoogleDrive(gauth)
@@ -393,8 +391,25 @@ class GDriveRemote(BaseRemote):
) as pbar:
gdrive_file.GetContentFile(to_file, callback=pbar.update_to)
@contextmanager
@_gdrive_retry
def _gdrive_delete_file(self, item_id):
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb"}
item_id = self._get_item_id(path_info)
param = {"id": item_id}
# it does not create a file on the remote
gdrive_file = self._drive.CreateFile(param)
fd = gdrive_file.GetContentIOBuffer()
stream = IterStream(iter(fd))
if mode != "rb":
stream = io.TextIOWrapper(stream, encoding=encoding)
yield stream
@_gdrive_retry
def gdrive_delete_file(self, item_id):
from pydrive2.files import ApiRequestError
param = {"id": item_id}
@@ -506,30 +521,17 @@ class GDriveRemote(BaseRemote):
return min(item_ids)
assert not create
raise GDrivePathNotFound(path_info, hint)
raise FileMissingError(path_info, hint)
def exists(self, path_info):
try:
self._get_item_id(path_info)
except GDrivePathNotFound:
except FileMissingError:
return False
else:
return True
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
dirname = to_info.parent
assert dirname
parent_id = self._get_item_id(dirname, True)
self._gdrive_upload_file(
parent_id, to_info.name, no_progress_bar, from_file, name
)
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
item_id = self._get_item_id(from_info)
self._gdrive_download_file(item_id, to_file, name, no_progress_bar)
def list_cache_paths(self, prefix=None, progress_callback=None):
def _list_paths(self, prefix=None):
if not self._ids_cache["ids"]:
return
@@ -545,19 +547,35 @@ class GDriveRemote(BaseRemote):
query = f"({parents_query}) and trashed=false"
for item in self._gdrive_list(query):
if progress_callback:
progress_callback()
parent_id = item["parents"][0]["id"]
yield posixpath.join(
self._ids_cache["ids"][parent_id], item["title"]
)
def walk_files(self, path_info, **kwargs):
if path_info == self.path_info:
prefix = None
else:
prefix = path_info.path
for fname in self._list_paths(prefix=prefix, **kwargs):
yield path_info.replace(fname)
def remove(self, path_info):
item_id = self._get_item_id(path_info)
self._gdrive_delete_file(item_id)
self.gdrive_delete_file(item_id)
def get_file_checksum(self, path_info):
def get_file_hash(self, path_info):
raise NotImplementedError
def walk_files(self, path_info):
raise NotImplementedError
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
dirname = to_info.parent
assert dirname
parent_id = self._get_item_id(dirname, True)
self._gdrive_upload_file(
parent_id, to_info.name, no_progress_bar, from_file, name
)
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
item_id = self._get_item_id(from_info)
self._gdrive_download_file(item_id, to_file, name, no_progress_bar)
+71 -85
View File
@@ -1,6 +1,5 @@
import logging
import os.path
import posixpath
import threading
from datetime import timedelta
from functools import wraps
@@ -10,7 +9,7 @@ from funcy import cached_property, wrap_prop
from dvc.exceptions import DvcException
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
@@ -65,9 +64,9 @@ def _upload_to_bucket(
blob.upload_from_file(wrapped)
class GSRemote(BaseRemote):
class GSRemoteTree(BaseRemoteTree):
scheme = Schemes.GS
path_cls = CloudURLInfo
PATH_CLS = CloudURLInfo
REQUIRES = {"google-cloud-storage": "google.cloud.storage"}
PARAM_CHECKSUM = "md5"
@@ -75,7 +74,7 @@ class GSRemote(BaseRemote):
super().__init__(repo, config)
url = config.get("url", "gs:///")
self.path_info = self.path_cls(url)
self.path_info = self.PATH_CLS(url)
self.projectname = config.get("projectname", None)
self.credentialpath = config.get("credentialpath")
@@ -91,74 +90,22 @@ class GSRemote(BaseRemote):
else Client(self.projectname)
)
def get_file_checksum(self, path_info):
import base64
import codecs
def _generate_download_url(self, path_info, expires=3600):
expiration = timedelta(seconds=int(expires))
bucket = path_info.bucket
path = path_info.path
blob = self.gs.bucket(bucket).get_blob(path)
if not blob:
return None
bucket = self.gs.bucket(path_info.bucket)
blob = bucket.get_blob(path_info.path)
if blob is None:
raise FileNotFoundError
return blob.generate_signed_url(expiration=expiration)
b64_md5 = blob.md5_hash
md5 = base64.b64decode(b64_md5)
return codecs.getencoder("hex")(md5)[0].decode("utf-8")
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
def copy(self, from_info, to_info):
from_bucket = self.gs.bucket(from_info.bucket)
blob = from_bucket.get_blob(from_info.path)
if not blob:
msg = f"'{from_info.path}' doesn't exist in the cloud"
raise DvcException(msg)
to_bucket = self.gs.bucket(to_info.bucket)
from_bucket.copy_blob(blob, to_bucket, new_name=to_info.path)
def remove(self, path_info):
if path_info.scheme != "gs":
raise NotImplementedError
logger.debug(f"Removing gs://{path_info}")
blob = self.gs.bucket(path_info.bucket).get_blob(path_info.path)
if not blob:
return
blob.delete()
def _list_paths(
self, path_info, max_items=None, prefix=None, progress_callback=None
):
if prefix:
prefix = posixpath.join(path_info.path, prefix[:2], prefix[2:])
else:
prefix = path_info.path
for blob in self.gs.bucket(path_info.bucket).list_blobs(
prefix=path_info.path, max_results=max_items
):
if progress_callback:
progress_callback()
yield blob.name
def list_cache_paths(self, prefix=None, progress_callback=None):
return self._list_paths(
self.path_info, prefix=prefix, progress_callback=progress_callback
)
def walk_files(self, path_info):
for fname in self._list_paths(path_info / ""):
# skip nested empty directories
if fname.endswith("/"):
continue
yield path_info.replace(fname)
def makedirs(self, path_info):
if not path_info.path:
return
self.gs.bucket(path_info.bucket).blob(
(path_info / "").path
).upload_from_string("")
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def isdir(self, path_info):
dir_path = path_info / ""
@@ -171,13 +118,61 @@ class GSRemote(BaseRemote):
blob = self.gs.bucket(path_info.bucket).blob(path_info.path)
return blob.exists()
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
def _list_paths(self, path_info, max_items=None):
for blob in self.gs.bucket(path_info.bucket).list_blobs(
prefix=path_info.path, max_results=max_items
):
yield blob.name
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(path_info / "", **kwargs):
# skip nested empty directories
if fname.endswith("/"):
continue
yield path_info.replace(fname)
def remove(self, path_info):
if path_info.scheme != "gs":
raise NotImplementedError
logger.debug(f"Removing gs://{path_info}")
blob = self.gs.bucket(path_info.bucket).get_blob(path_info.path)
if not blob:
return
blob.delete()
def makedirs(self, path_info):
if not path_info.path:
return
self.gs.bucket(path_info.bucket).blob(
(path_info / "").path
).upload_from_string("")
def copy(self, from_info, to_info):
from_bucket = self.gs.bucket(from_info.bucket)
blob = from_bucket.get_blob(from_info.path)
if not blob:
msg = f"'{from_info.path}' doesn't exist in the cloud"
raise DvcException(msg)
to_bucket = self.gs.bucket(to_info.bucket)
from_bucket.copy_blob(blob, to_bucket, new_name=to_info.path)
def get_file_hash(self, path_info):
import base64
import codecs
bucket = path_info.bucket
path = path_info.path
blob = self.gs.bucket(bucket).get_blob(path)
if not blob:
return None
b64_md5 = blob.md5_hash
md5 = base64.b64decode(b64_md5)
return codecs.getencoder("hex")(md5)[0].decode("utf-8")
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
bucket = self.gs.bucket(to_info.bucket)
@@ -201,12 +196,3 @@ class GSRemote(BaseRemote):
disable=no_progress_bar,
) as wrapped:
blob.download_to_file(wrapped)
def _generate_download_url(self, path_info, expires=3600):
expiration = timedelta(seconds=int(expires))
bucket = self.gs.bucket(path_info.bucket)
blob = bucket.get_blob(path_info.path)
if blob is None:
raise FileNotFoundError
return blob.generate_signed_url(expiration=expiration)
+82 -85
View File
@@ -11,21 +11,22 @@ from urllib.parse import urlparse
from dvc.scheme import Schemes
from dvc.utils import fix_env, tmp_fname
from .base import BaseRemote, RemoteCmdError
from .base import BaseRemoteTree, RemoteCmdError
from .pool import get_connection
logger = logging.getLogger(__name__)
class HDFSRemote(BaseRemote):
class HDFSRemoteTree(BaseRemoteTree):
scheme = Schemes.HDFS
REQUIRES = {"pyarrow": "pyarrow"}
REGEX = r"^hdfs://((?P<user>.*)@)?.*$"
PARAM_CHECKSUM = "checksum"
REQUIRES = {"pyarrow": "pyarrow"}
TRAVERSE_PREFIX_LEN = 2
def __init__(self, repo, config):
super().__init__(repo, config)
self.path_info = None
url = config.get("url")
if not url:
@@ -34,7 +35,7 @@ class HDFSRemote(BaseRemote):
parsed = urlparse(url)
user = parsed.username or config.get("user")
self.path_info = self.path_cls.from_parts(
self.path_info = self.PATH_CLS.from_parts(
scheme=self.scheme,
host=parsed.hostname,
user=user,
@@ -42,7 +43,8 @@ class HDFSRemote(BaseRemote):
path=parsed.path,
)
def hdfs(self, path_info):
@staticmethod
def hdfs(path_info):
import pyarrow
return get_connection(
@@ -52,6 +54,80 @@ class HDFSRemote(BaseRemote):
user=path_info.user,
)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb"}
try:
with self.hdfs(path_info) as hdfs, closing(
hdfs.open(path_info.path, mode="rb")
) as fd:
if mode == "rb":
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
except OSError as e:
# Empty .errno and not specific enough error class in pyarrow,
# see https://issues.apache.org/jira/browse/ARROW-6248
if "file does not exist" in str(e):
raise FileNotFoundError(*e.args)
raise
def exists(self, path_info):
assert not isinstance(path_info, list)
assert path_info.scheme == "hdfs"
with self.hdfs(path_info) as hdfs:
return hdfs.exists(path_info.path)
def walk_files(self, path_info, **kwargs):
if not self.exists(path_info):
return
root = path_info.path
dirs = deque([root])
with self.hdfs(self.path_info) as hdfs:
if not hdfs.exists(root):
return
while dirs:
try:
entries = hdfs.ls(dirs.pop(), detail=True)
for entry in entries:
if entry["kind"] == "directory":
dirs.append(urlparse(entry["name"]).path)
elif entry["kind"] == "file":
path = urlparse(entry["name"]).path
yield path_info.replace(path=path)
except OSError:
# When searching for a specific prefix pyarrow raises an
# exception if the specified cache dir does not exist
pass
def remove(self, path_info):
if path_info.scheme != "hdfs":
raise NotImplementedError
if self.exists(path_info):
logger.debug(f"Removing {path_info.path}")
with self.hdfs(path_info) as hdfs:
hdfs.rm(path_info.path)
def copy(self, from_info, to_info, **_kwargs):
dname = posixpath.dirname(to_info.path)
with self.hdfs(to_info) as hdfs:
hdfs.mkdir(dname)
# NOTE: this is how `hadoop fs -cp` works too: it copies through
# your local machine.
with hdfs.open(from_info.path, "rb") as from_fobj:
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
with hdfs.open(tmp_info.path, "wb") as tmp_fobj:
tmp_fobj.upload(from_fobj)
hdfs.rename(tmp_info.path, to_info.path)
except Exception:
self.remove(tmp_info)
raise
def hadoop_fs(self, cmd, user=None):
cmd = "hadoop fs -" + cmd
if user:
@@ -83,7 +159,7 @@ class HDFSRemote(BaseRemote):
assert match is not None
return match.group(gname)
def get_file_checksum(self, path_info):
def get_file_hash(self, path_info):
# NOTE: pyarrow doesn't support checksum, so we need to use hadoop
regex = r".*\t.*\t(?P<checksum>.*)"
stdout = self.hadoop_fs(
@@ -91,37 +167,6 @@ class HDFSRemote(BaseRemote):
)
return self._group(regex, stdout, "checksum")
def copy(self, from_info, to_info, **_kwargs):
dname = posixpath.dirname(to_info.path)
with self.hdfs(to_info) as hdfs:
hdfs.mkdir(dname)
# NOTE: this is how `hadoop fs -cp` works too: it copies through
# your local machine.
with hdfs.open(from_info.path, "rb") as from_fobj:
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
with hdfs.open(tmp_info.path, "wb") as tmp_fobj:
tmp_fobj.upload(from_fobj)
hdfs.rename(tmp_info.path, to_info.path)
except Exception:
self.remove(tmp_info)
raise
def remove(self, path_info):
if path_info.scheme != "hdfs":
raise NotImplementedError
if self.exists(path_info):
logger.debug(f"Removing {path_info.path}")
with self.hdfs(path_info) as hdfs:
hdfs.rm(path_info.path)
def exists(self, path_info):
assert not isinstance(path_info, list)
assert path_info.scheme == "hdfs"
with self.hdfs(path_info) as hdfs:
return hdfs.exists(path_info.path)
def _upload(self, from_file, to_info, **_kwargs):
with self.hdfs(to_info) as hdfs:
hdfs.mkdir(posixpath.dirname(to_info.path))
@@ -134,51 +179,3 @@ class HDFSRemote(BaseRemote):
with self.hdfs(from_info) as hdfs:
with open(to_file, "wb+") as fobj:
hdfs.download(from_info.path, fobj)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb"}
try:
with self.hdfs(path_info) as hdfs, closing(
hdfs.open(path_info.path, mode="rb")
) as fd:
if mode == "rb":
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
except OSError as e:
# Empty .errno and not specific enough error class in pyarrow,
# see https://issues.apache.org/jira/browse/ARROW-6248
if "file does not exist" in str(e):
raise FileNotFoundError(*e.args)
raise
def list_cache_paths(self, prefix=None, progress_callback=None):
if not self.exists(self.path_info):
return
if prefix:
root = posixpath.join(self.path_info.path, prefix[:2])
else:
root = self.path_info.path
dirs = deque([root])
with self.hdfs(self.path_info) as hdfs:
if prefix and not hdfs.exists(root):
return
while dirs:
try:
entries = hdfs.ls(dirs.pop(), detail=True)
for entry in entries:
if entry["kind"] == "directory":
dirs.append(urlparse(entry["name"]).path)
elif entry["kind"] == "file":
if progress_callback:
progress_callback()
yield urlparse(entry["name"]).path
except OSError as e:
# When searching for a specific prefix pyarrow raises an
# exception if the specified cache dir does not exist
if not prefix:
raise e
+77 -78
View File
@@ -8,7 +8,7 @@ import dvc.prompt as prompt
from dvc.exceptions import DvcException, HTTPError
from dvc.path_info import HTTPURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
@@ -23,22 +23,23 @@ def ask_password(host, user):
)
class HTTPRemote(BaseRemote):
class HTTPRemoteTree(BaseRemoteTree):
scheme = Schemes.HTTP
path_cls = HTTPURLInfo
PATH_CLS = HTTPURLInfo
PARAM_CHECKSUM = "etag"
CAN_TRAVERSE = False
SESSION_RETRIES = 5
SESSION_BACKOFF_FACTOR = 0.1
REQUEST_TIMEOUT = 10
CHUNK_SIZE = 2 ** 16
PARAM_CHECKSUM = "etag"
CAN_TRAVERSE = False
def __init__(self, repo, config):
super().__init__(repo, config)
url = config.get("url")
if url:
self.path_info = self.path_cls(url)
self.path_info = self.PATH_CLS(url)
user = config.get("user", None)
if user:
self.path_info.user = user
@@ -51,74 +52,7 @@ class HTTPRemote(BaseRemote):
self.ask_password = config.get("ask_password", False)
self.headers = {}
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
response = self._request("GET", from_info.url, stream=True)
if response.status_code != 200:
raise HTTPError(response.status_code, response.reason)
with open(to_file, "wb") as fd:
with Tqdm.wrapattr(
fd,
"write",
total=None
if no_progress_bar
else self._content_length(response),
leave=False,
desc=from_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):
fd_wrapped.write(chunk)
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
def chunks():
with open(from_file, "rb") as fd:
with Tqdm.wrapattr(
fd,
"read",
total=None
if no_progress_bar
else os.path.getsize(from_file),
leave=False,
desc=to_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
while True:
chunk = fd_wrapped.read(self.CHUNK_SIZE)
if not chunk:
break
yield chunk
response = self._request("POST", to_info.url, data=chunks())
if response.status_code not in (200, 201):
raise HTTPError(response.status_code, response.reason)
def exists(self, path_info):
return bool(self._request("HEAD", path_info.url))
def _content_length(self, response):
res = response.headers.get("Content-Length")
return int(res) if res else None
def get_file_checksum(self, path_info):
url = path_info.url
headers = self._request("HEAD", url).headers
etag = headers.get("ETag") or headers.get("Content-MD5")
if not etag:
raise DvcException(
"could not find an ETag or "
"Content-MD5 header for '{url}'".format(url=url)
)
if etag.startswith("W/"):
raise DvcException(
"Weak ETags are not supported."
" (Etag: '{etag}', URL: '{url}')".format(etag=etag, url=url)
)
return etag
def auth_method(self, path_info=None):
def _auth_method(self, path_info=None):
from requests.auth import HTTPBasicAuth, HTTPDigestAuth
if path_info is None:
@@ -155,7 +89,7 @@ class HTTPRemote(BaseRemote):
return session
def _request(self, method, url, **kwargs):
def request(self, method, url, **kwargs):
import requests
kwargs.setdefault("allow_redirects", True)
@@ -165,7 +99,7 @@ class HTTPRemote(BaseRemote):
res = self._session.request(
method,
url,
auth=self.auth_method(),
auth=self._auth_method(),
headers=self.headers,
**kwargs,
)
@@ -187,5 +121,70 @@ class HTTPRemote(BaseRemote):
except requests.exceptions.RequestException:
raise DvcException(f"could not perform a {method} request")
def gc(self):
raise NotImplementedError
def exists(self, path_info):
return bool(self.request("HEAD", path_info.url))
def get_file_hash(self, path_info):
url = path_info.url
headers = self.request("HEAD", url).headers
etag = headers.get("ETag") or headers.get("Content-MD5")
if not etag:
raise DvcException(
"could not find an ETag or "
"Content-MD5 header for '{url}'".format(url=url)
)
if etag.startswith("W/"):
raise DvcException(
"Weak ETags are not supported."
" (Etag: '{etag}', URL: '{url}')".format(etag=etag, url=url)
)
return etag
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
response = self.request("GET", from_info.url, stream=True)
if response.status_code != 200:
raise HTTPError(response.status_code, response.reason)
with open(to_file, "wb") as fd:
with Tqdm.wrapattr(
fd,
"write",
total=None
if no_progress_bar
else self._content_length(response),
leave=False,
desc=from_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):
fd_wrapped.write(chunk)
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
def chunks():
with open(from_file, "rb") as fd:
with Tqdm.wrapattr(
fd,
"read",
total=None
if no_progress_bar
else os.path.getsize(from_file),
leave=False,
desc=to_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
while True:
chunk = fd_wrapped.read(self.CHUNK_SIZE)
if not chunk:
break
yield chunk
response = self.request("POST", to_info.url, data=chunks())
if response.status_code not in (200, 201):
raise HTTPError(response.status_code, response.reason)
@staticmethod
def _content_length(response):
res = response.headers.get("Content-Length")
return int(res) if res else None
+2 -2
View File
@@ -1,7 +1,7 @@
from dvc.scheme import Schemes
from .http import HTTPRemote
from .http import HTTPRemoteTree
class HTTPSRemote(HTTPRemote):
class HTTPSRemoteTree(HTTPRemoteTree):
scheme = Schemes.HTTPS
+27 -31
View File
@@ -25,15 +25,15 @@ class RemoteIndexNoop:
def __iter__(self):
return iter([])
def __contains__(self, checksum):
def __contains__(self, hash_):
return False
@staticmethod
def checksums():
def hashes():
return []
@staticmethod
def dir_checksums():
def dir_hashes():
return []
def load(self):
@@ -54,13 +54,13 @@ class RemoteIndexNoop:
class RemoteIndex:
"""Class for indexing remote checksums in a sqlite3 database.
"""Class for indexing remote hashes in a sqlite3 database.
Args:
repo: repo for this remote index.
name: name for this index. Index db will be loaded from and saved to
``.dvc/tmp/index/{name}.idx``.
dir_suffix: suffix used for naming directory checksums
dir_suffix: suffix used for naming directory hashes
"""
INDEX_SUFFIX = ".idx"
@@ -79,8 +79,8 @@ class RemoteIndex:
def __iter__(self):
cmd = f"SELECT checksum FROM {self.INDEX_TABLE}"
for (checksum,) in self._execute(cmd):
yield checksum
for (hash_,) in self._execute(cmd):
yield hash_
def __enter__(self):
self.lock.acquire()
@@ -90,25 +90,25 @@ class RemoteIndex:
self.dump()
self.lock.release()
def __contains__(self, checksum):
def __contains__(self, hash_):
cmd = "SELECT checksum FROM {} WHERE checksum = (?)".format(
self.INDEX_TABLE
)
self._execute(cmd, (checksum,))
self._execute(cmd, (hash_,))
return self.cursor.fetchone() is not None
def checksums(self):
"""Iterate over checksums stored in the index."""
def hashes(self):
"""Iterate over hashes stored in the index."""
return iter(self)
def dir_checksums(self):
"""Iterate over .dir checksums stored in the index."""
def dir_hashes(self):
"""Iterate over .dir hashes stored in the index."""
cmd = f"SELECT checksum FROM {self.INDEX_TABLE} WHERE dir = 1"
for (checksum,) in self._execute(cmd):
yield checksum
for (hash_,) in self._execute(cmd):
yield hash_
def is_dir_checksum(self, checksum):
return checksum.endswith(self.dir_suffix)
def is_dir_hash(self, hash_):
return hash_.endswith(self.dir_suffix)
def _execute(self, cmd, parameters=()):
return self.cursor.execute(cmd, parameters)
@@ -185,28 +185,24 @@ class RemoteIndex:
cmd = f"DELETE FROM {self.INDEX_TABLE}"
self._execute(cmd)
def update(self, dir_checksums, file_checksums):
"""Update this index, adding the specified checksums.
def update(self, dir_hashes, file_hashes):
"""Update this index, adding the specified hashes.
Changes to the index will not committed until dump() is called.
"""
cmd = "INSERT OR IGNORE INTO {} (checksum, dir) VALUES (?, ?)".format(
self.INDEX_TABLE
)
self._executemany(
cmd, ((checksum, True) for checksum in dir_checksums)
)
self._executemany(
cmd, ((checksum, False) for checksum in file_checksums)
)
self._executemany(cmd, ((hash_, True) for hash_ in dir_hashes))
self._executemany(cmd, ((hash_, False) for hash_ in file_hashes))
def intersection(self, checksums):
"""Iterate over values from `checksums` which exist in the index."""
def intersection(self, hashes):
"""Iterate over values from `hashes` which exist in the index."""
# sqlite has a compile time limit of 999, see:
# https://www.sqlite.org/c3ref/c_limit_attached.html#sqlitelimitvariablenumber
for chunk in lchunks(999, checksums):
for chunk in lchunks(999, hashes):
cmd = "SELECT checksum FROM {} WHERE checksum IN ({})".format(
self.INDEX_TABLE, ",".join("?" for checksum in chunk)
self.INDEX_TABLE, ",".join("?" for hash_ in chunk)
)
for (checksum,) in self._execute(cmd, chunk):
yield checksum
for (hash_,) in self._execute(cmd, chunk):
yield hash_
+278 -266
View File
@@ -3,7 +3,7 @@ import logging
import os
import stat
from concurrent.futures import ThreadPoolExecutor, as_completed
from functools import partial
from functools import partial, wraps
from funcy import cached_property, concat
from shortuuid import uuid
@@ -16,7 +16,9 @@ from dvc.remote.base import (
STATUS_MAP,
STATUS_MISSING,
STATUS_NEW,
BaseRemote,
BaseRemoteTree,
CloudCache,
Remote,
index_locked,
)
from dvc.remote.index import RemoteIndexNoop
@@ -36,116 +38,65 @@ from dvc.utils.fs import (
logger = logging.getLogger(__name__)
class LocalRemote(BaseRemote):
class LocalRemoteTree(BaseRemoteTree):
scheme = Schemes.LOCAL
path_cls = PathInfo
PATH_CLS = PathInfo
PARAM_CHECKSUM = "md5"
PARAM_PATH = "path"
TRAVERSE_PREFIX_LEN = 2
INDEX_CLS = RemoteIndexNoop
UNPACKED_DIR_SUFFIX = ".unpacked"
DEFAULT_CACHE_TYPES = ["reflink", "copy"]
CACHE_MODE = 0o444
SHARED_MODE_MAP = {None: (0o644, 0o755), "group": (0o664, 0o775)}
def __init__(self, repo, config):
super().__init__(repo, config)
self.cache_dir = config.get("url")
self._dir_info = {}
url = config.get("url")
self.path_info = self.PATH_CLS(url) if url else None
@property
def state(self):
return self.repo.state
from dvc.state import StateNoop
@property
def cache_dir(self):
return self.path_info.fspath if self.path_info else None
@cache_dir.setter
def cache_dir(self, value):
self.path_info = PathInfo(value) if value else None
return self.repo.state if self.repo else StateNoop()
@cached_property
def _work_tree(self):
if self.repo:
return WorkingTree(self.repo.root_dir)
return None
@property
def work_tree(self):
# When using repo.brancher, repo.tree may change to/from WorkingTree to
# GitTree arbitarily. When repo.tree is GitTree, local cache needs to
# use its own WorkingTree instance.
if self.repo and not is_working_tree(self.repo.tree):
return self._work_tree
if self.repo:
return WorkingTree(self.repo.root_dir)
return None
@classmethod
def supported(cls, config):
return True
@cached_property
def cache_path(self):
return os.path.abspath(self.cache_dir)
def checksum_to_path(self, checksum):
# NOTE: `self.cache_path` is already normalized so we can simply use
# `os.sep` instead of `os.path.join`. This results in this helper
# being ~5.5 times faster.
return (
f"{self.cache_path}{os.sep}{checksum[0:2]}{os.sep}{checksum[2:]}"
)
def list_cache_paths(self, prefix=None, progress_callback=None):
assert self.path_info is not None
if prefix:
path_info = self.path_info / prefix[:2]
if not self.exists(path_info):
return
else:
path_info = self.path_info
if progress_callback:
for path in walk_files(path_info):
progress_callback()
yield path
else:
yield from walk_files(path_info)
def get(self, md5):
if not md5:
return None
return self.checksum_to_path_info(md5).url
@staticmethod
def open(path_info, mode="r", encoding=None):
return open(path_info, mode=mode, encoding=encoding)
def exists(self, path_info):
assert isinstance(path_info, str) or path_info.scheme == "local"
if not self.repo:
return os.path.exists(path_info)
if self.work_tree and self.work_tree.exists(path_info):
return True
return self.repo.tree.exists(path_info)
return self.work_tree.exists(path_info)
def makedirs(self, path_info):
makedirs(path_info, exist_ok=True, mode=self._dir_mode)
def isfile(self, path_info):
if not self.repo:
return os.path.isfile(path_info)
return self.work_tree.isfile(path_info)
def already_cached(self, path_info):
assert path_info.scheme in ["", "local"]
def isdir(self, path_info):
if not self.repo:
return os.path.isdir(path_info)
return self.work_tree.isdir(path_info)
current_md5 = self.get_checksum(path_info)
def iscopy(self, path_info):
return not (
System.is_symlink(path_info) or System.is_hardlink(path_info)
)
if not current_md5:
return False
return not self.changed_cache(current_md5)
def _verify_link(self, path_info, link_type):
if link_type == "hardlink" and self.getsize(path_info) == 0:
return
super()._verify_link(path_info, link_type)
def walk_files(self, path_info, **kwargs):
for fname in self.work_tree.walk_files(path_info):
yield PathInfo(fname)
def is_empty(self, path_info):
path = path_info.fspath
@@ -158,40 +109,6 @@ class LocalRemote(BaseRemote):
return False
def isfile(self, path_info):
if not self.repo:
return os.path.isfile(path_info)
if self.work_tree and self.work_tree.isfile(path_info):
return True
return self.repo.tree.isfile(path_info)
def isdir(self, path_info):
if not self.repo:
return os.path.isdir(path_info)
if self.work_tree and self.work_tree.isdir(path_info):
return True
return self.repo.tree.isdir(path_info)
def iscopy(self, path_info):
return not (
System.is_symlink(path_info) or System.is_hardlink(path_info)
)
@staticmethod
def getsize(path_info):
return os.path.getsize(path_info)
def walk_files(self, path_info):
if self.work_tree:
tree = self.work_tree
else:
tree = self.repo.tree
for fname in tree.walk_files(path_info):
yield PathInfo(fname)
def get_file_checksum(self, path_info):
return file_md5(path_info)[0]
def remove(self, path_info):
if isinstance(path_info, PathInfo):
if path_info.scheme != "local":
@@ -203,6 +120,9 @@ class LocalRemote(BaseRemote):
if self.exists(path):
remove(path)
def makedirs(self, path_info):
makedirs(path_info, exist_ok=True, mode=self.dir_mode)
def move(self, from_info, to_info, mode=None):
if from_info.scheme != "local" or to_info.scheme != "local":
raise NotImplementedError
@@ -211,9 +131,9 @@ class LocalRemote(BaseRemote):
if mode is None:
if self.isfile(from_info):
mode = self._file_mode
mode = self.file_mode
else:
mode = self._dir_mode
mode = self.dir_mode
move(from_info, to_info, mode=mode)
@@ -221,7 +141,7 @@ class LocalRemote(BaseRemote):
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
System.copy(from_info, tmp_info)
os.chmod(tmp_info, self._file_mode)
os.chmod(tmp_info, self.file_mode)
os.rename(tmp_info, to_info)
except Exception:
self.remove(tmp_info)
@@ -232,7 +152,7 @@ class LocalRemote(BaseRemote):
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
copy_fobj_to_file(fobj, tmp_info)
os.chmod(tmp_info, self._file_mode)
os.chmod(tmp_info, self.file_mode)
os.rename(tmp_info, to_info)
except Exception:
self.remove(tmp_info)
@@ -251,7 +171,7 @@ class LocalRemote(BaseRemote):
# and the cache type is `hardlink`, we might reach link limits and
# will get something like: `too many links error`
#
# This is because all those empty files will have the same checksum
# This is because all those empty files will have the same hash
# (i.e. 68b329da9893e34099c7d8ad5cb9c940), therefore, they will be
# linked to the same file in the cache.
#
@@ -281,20 +201,81 @@ class LocalRemote(BaseRemote):
System.reflink(from_info, tmp_info)
# NOTE: reflink has its own separate inode, so you can set permissions
# that are different from the source.
os.chmod(tmp_info, self._file_mode)
os.chmod(tmp_info, self.file_mode)
os.rename(tmp_info, to_info)
def cache_exists(self, checksums, jobs=None, name=None):
return [
checksum
for checksum in Tqdm(
checksums,
unit="file",
desc="Querying "
+ ("cache in " + name if name else "local cache"),
def _unprotect_file(self, path):
if System.is_symlink(path) or System.is_hardlink(path):
logger.debug(f"Unprotecting '{path}'")
tmp = os.path.join(os.path.dirname(path), "." + uuid())
# The operations order is important here - if some application
# would access the file during the process of copyfile then it
# would get only the part of file. So, at first, the file should be
# copied with the temporary name, and then original file should be
# replaced by new.
copyfile(path, tmp, name="Unprotecting '{}'".format(relpath(path)))
remove(path)
os.rename(tmp, path)
else:
logger.debug(
"Skipping copying for '{}', since it is not "
"a symlink or a hardlink.".format(path)
)
if not self.changed_cache_file(checksum)
]
os.chmod(path, self.file_mode)
def _unprotect_dir(self, path):
assert is_working_tree(self.repo.tree)
for fname in self.repo.tree.walk_files(path):
self._unprotect_file(fname)
def unprotect(self, path_info):
path = path_info.fspath
if not os.path.exists(path):
raise DvcException(f"can't unprotect non-existing data '{path}'")
if os.path.isdir(path):
self._unprotect_dir(path)
else:
self._unprotect_file(path)
def protect(self, path_info):
path = os.fspath(path_info)
mode = self.CACHE_MODE
try:
os.chmod(path, mode)
except OSError as exc:
# There is nothing we need to do in case of a read-only file system
if exc.errno == errno.EROFS:
return
# In shared cache scenario, we might not own the cache file, so we
# need to check if cache file is already protected.
if exc.errno not in [errno.EPERM, errno.EACCES]:
raise
actual = stat.S_IMODE(os.stat(path).st_mode)
if actual != mode:
raise
def is_protected(self, path_info):
try:
mode = os.stat(path_info).st_mode
except FileNotFoundError:
return False
return stat.S_IMODE(mode) == self.CACHE_MODE
def get_file_hash(self, path_info):
return file_md5(path_info)[0]
@staticmethod
def getsize(path_info):
return os.path.getsize(path_info)
def _upload(
self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
@@ -309,16 +290,118 @@ class LocalRemote(BaseRemote):
self.protect(tmp_file)
os.rename(tmp_file, to_info)
@staticmethod
def _download(
self, from_info, to_file, name=None, no_progress_bar=False, **_kwargs
from_info, to_file, name=None, no_progress_bar=False, **_kwargs
):
copyfile(
from_info, to_file, no_progress_bar=no_progress_bar, name=name
)
@staticmethod
def open(path_info, mode="r", encoding=None):
return open(path_info, mode=mode, encoding=encoding)
def list_paths(self, prefix=None, progress_callback=None):
assert self.path_info is not None
if prefix:
path_info = self.path_info / prefix[:2]
if not self.exists(path_info):
return
else:
path_info = self.path_info
# NOTE: use utils.fs walk_files since tree.walk_files will not follow
# symlinks
if progress_callback:
for path in walk_files(path_info):
progress_callback()
yield path
else:
yield from walk_files(path_info)
def _remove_unpacked_dir(self, hash_):
info = self.hash_to_path_info(hash_)
path_info = info.with_name(info.name + self.UNPACKED_DIR_SUFFIX)
self.remove(path_info)
def _log_exceptions(func, operation):
@wraps(func)
def wrapper(from_info, to_info, *args, **kwargs):
try:
func(from_info, to_info, *args, **kwargs)
return 0
except Exception as exc:
# NOTE: this means we ran out of file descriptors and there is no
# reason to try to proceed, as we will hit this error anyways.
if isinstance(exc, OSError) and exc.errno == errno.EMFILE:
raise
logger.exception(
"failed to %s '%s' to '%s'", operation, from_info, to_info
)
return 1
return wrapper
class LocalRemote(Remote):
INDEX_CLS = RemoteIndexNoop
class LocalCache(CloudCache):
DEFAULT_CACHE_TYPES = ["reflink", "copy"]
CACHE_MODE = LocalRemoteTree.CACHE_MODE
def __init__(self, tree):
super().__init__(tree)
self.cache_dir = tree.config.get("url")
@property
def cache_dir(self):
return self.tree.path_info.fspath if self.tree.path_info else None
@cache_dir.setter
def cache_dir(self, value):
self.tree.path_info = PathInfo(value) if value else None
@classmethod
def supported(cls, config):
return True
@cached_property
def cache_path(self):
return os.path.abspath(self.cache_dir)
def hash_to_path(self, hash_):
# NOTE: `self.cache_path` is already normalized so we can simply use
# `os.sep` instead of `os.path.join`. This results in this helper
# being ~5.5 times faster.
return f"{self.cache_path}{os.sep}{hash_[0:2]}{os.sep}{hash_[2:]}"
def hashes_exist(self, hashes, jobs=None, name=None):
return [
hash_
for hash_ in Tqdm(
hashes,
unit="file",
desc="Querying "
+ ("cache in " + name if name else "local cache"),
)
if not self.changed_cache_file(hash_)
]
def already_cached(self, path_info):
assert path_info.scheme in ["", "local"]
current_md5 = self.get_hash(path_info)
if not current_md5:
return False
return not self.changed_cache(current_md5)
def _verify_link(self, path_info, link_type):
if link_type == "hardlink" and self.tree.getsize(path_info) == 0:
return
super()._verify_link(path_info, link_type)
@index_locked
def status(
@@ -351,7 +434,7 @@ class LocalRemote(BaseRemote):
dir_status_info contains status for .dir files, file_status_info
contains status for all other files, and dir_contents is a dict of
{dir_checksum: set(file_checksum, ...)} which can be used to map
{dir_hash: set(file_hash, ...)} which can be used to map
a .dir file to its file contents.
"""
logger.debug(f"Preparing to collect status from {remote.path_info}")
@@ -359,7 +442,7 @@ class LocalRemote(BaseRemote):
logger.debug("Collecting information from local cache...")
local_exists = frozenset(
self.cache_exists(md5s, jobs=jobs, name=self.cache_dir)
self.hashes_exist(md5s, jobs=jobs, name=self.cache_dir)
)
# This is a performance optimization. We can safely assume that,
@@ -374,12 +457,12 @@ class LocalRemote(BaseRemote):
dir_md5s = set(named_cache.dir_keys(self.scheme))
if dir_md5s:
remote_exists.update(
self._indexed_dir_checksums(named_cache, remote, dir_md5s)
self._indexed_dir_hashes(named_cache, remote, dir_md5s)
)
md5s.difference_update(remote_exists)
if md5s:
remote_exists.update(
remote.cache_exists(
remote.hashes_exist(
md5s, jobs=jobs, name=str(remote.path_info)
)
)
@@ -390,23 +473,23 @@ class LocalRemote(BaseRemote):
def _make_status(
self, named_cache, show_checksums, local_exists, remote_exists
):
def make_names(checksum, names):
return {"name": checksum if show_checksums else " ".join(names)}
def make_names(hash_, names):
return {"name": hash_ if show_checksums else " ".join(names)}
dir_status = {}
file_status = {}
dir_contents = {}
for checksum, item in named_cache[self.scheme].items():
for hash_, item in named_cache[self.scheme].items():
if item.children:
dir_status[checksum] = make_names(checksum, item.names)
dir_contents[checksum] = set()
for child_checksum, child in item.children.items():
file_status[child_checksum] = make_names(
child_checksum, child.names
dir_status[hash_] = make_names(hash_, item.names)
dir_contents[hash_] = set()
for child_hash, child in item.children.items():
file_status[child_hash] = make_names(
child_hash, child.names
)
dir_contents[checksum].add(child_checksum)
dir_contents[hash_].add(child_hash)
else:
file_status[checksum] = make_names(checksum, item.names)
file_status[hash_] = make_names(hash_, item.names)
self._fill_statuses(dir_status, local_exists, remote_exists)
self._fill_statuses(file_status, local_exists, remote_exists)
@@ -415,50 +498,50 @@ class LocalRemote(BaseRemote):
return dir_status, file_status, dir_contents
def _indexed_dir_checksums(self, named_cache, remote, dir_md5s):
# Validate our index by verifying all indexed .dir checksums
def _indexed_dir_hashes(self, named_cache, remote, dir_md5s):
# Validate our index by verifying all indexed .dir hashes
# still exist on the remote
indexed_dirs = set(remote.index.dir_checksums())
indexed_dirs = set(remote.index.dir_hashes())
indexed_dir_exists = set()
if indexed_dirs:
indexed_dir_exists.update(
remote._cache_object_exists(indexed_dirs)
remote.tree.list_hashes_exists(indexed_dirs)
)
missing_dirs = indexed_dirs.difference(indexed_dir_exists)
if missing_dirs:
logger.debug(
"Remote cache missing indexed .dir checksums '{}', "
"Remote cache missing indexed .dir hashes '{}', "
"clearing remote index".format(", ".join(missing_dirs))
)
remote.index.clear()
# Check if non-indexed (new) dir checksums exist on remote
# Check if non-indexed (new) dir hashes exist on remote
dir_exists = dir_md5s.intersection(indexed_dir_exists)
dir_exists.update(remote._cache_object_exists(dir_md5s - dir_exists))
dir_exists.update(
remote.tree.list_hashes_exists(dir_md5s - dir_exists)
)
# If .dir checksum exists on the remote, assume directory contents
# If .dir hash exists on the remote, assume directory contents
# still exists on the remote
for dir_checksum in dir_exists:
file_checksums = list(
named_cache.child_keys(self.scheme, dir_checksum)
)
if dir_checksum not in remote.index:
for dir_hash in dir_exists:
file_hashes = list(named_cache.child_keys(self.scheme, dir_hash))
if dir_hash not in remote.index:
logger.debug(
"Indexing new .dir '{}' with '{}' nested files".format(
dir_checksum, len(file_checksums)
dir_hash, len(file_hashes)
)
)
remote.index.update([dir_checksum], file_checksums)
yield dir_checksum
yield from file_checksums
remote.index.update([dir_hash], file_hashes)
yield dir_hash
yield from file_hashes
@staticmethod
def _fill_statuses(checksum_info_dir, local_exists, remote_exists):
def _fill_statuses(hash_info_dir, local_exists, remote_exists):
# Using sets because they are way faster for lookups
local = set(local_exists)
remote = set(remote_exists)
for md5, info in checksum_info_dir.items():
for md5, info in hash_info_dir.items():
status = STATUS_MAP[(md5 in local, md5 in remote)]
info["status"] = status
@@ -466,15 +549,15 @@ class LocalRemote(BaseRemote):
cache = []
path_infos = []
names = []
checksums = []
hashes = []
for md5, info in Tqdm(
status_info.items(), desc="Analysing status", unit="file"
):
if info["status"] == status:
cache.append(self.checksum_to_path_info(md5))
path_infos.append(remote.checksum_to_path_info(md5))
cache.append(self.hash_to_path_info(md5))
path_infos.append(remote.hash_to_path_info(md5))
names.append(info["name"])
checksums.append(md5)
hashes.append(md5)
if download:
to_infos = cache
@@ -483,7 +566,7 @@ class LocalRemote(BaseRemote):
to_infos = path_infos
from_infos = cache
return from_infos, to_infos, names, checksums
return from_infos, to_infos, names, hashes
def _process(
self,
@@ -502,19 +585,19 @@ class LocalRemote(BaseRemote):
if download:
func = partial(
remote.download,
dir_mode=self._dir_mode,
file_mode=self._file_mode,
_log_exceptions(remote.tree.download, "download"),
dir_mode=self.tree.dir_mode,
file_mode=self.tree.file_mode,
)
status = STATUS_DELETED
desc = "Downloading"
else:
func = remote.upload
func = _log_exceptions(remote.tree.upload, "upload")
status = STATUS_NEW
desc = "Uploading"
if jobs is None:
jobs = remote.JOBS
jobs = remote.tree.JOBS
dir_status, file_status, dir_contents = self._status(
named_cache,
@@ -535,26 +618,28 @@ class LocalRemote(BaseRemote):
func = pbar.wrap_fn(func)
with ThreadPoolExecutor(max_workers=jobs) as executor:
if download:
fails = sum(executor.map(func, *dir_plans))
fails += sum(executor.map(func, *file_plans))
from_infos, to_infos, names, _ = (
d + f for d, f in zip(dir_plans, file_plans)
)
fails = sum(
executor.map(func, from_infos, to_infos, names)
)
else:
# for uploads, push files first, and any .dir files last
file_futures = {}
for from_info, to_info, name, checksum in zip(*file_plans):
file_futures[checksum] = executor.submit(
for from_info, to_info, name, hash_ in zip(*file_plans):
file_futures[hash_] = executor.submit(
func, from_info, to_info, name
)
dir_futures = {}
for from_info, to_info, name, dir_checksum in zip(
*dir_plans
):
for from_info, to_info, name, dir_hash in zip(*dir_plans):
wait_futures = {
future
for file_checksum, future in file_futures.items()
if file_checksum in dir_contents[dir_checksum]
for file_hash, future in file_futures.items()
if file_hash in dir_contents[dir_hash]
}
dir_futures[dir_checksum] = executor.submit(
dir_futures[dir_hash] = executor.submit(
self._dir_upload,
func,
wait_futures,
@@ -577,16 +662,14 @@ class LocalRemote(BaseRemote):
if not download:
# index successfully pushed dirs
for dir_checksum, future in dir_futures.items():
for dir_hash, future in dir_futures.items():
if future.result() == 0:
file_checksums = dir_contents[dir_checksum]
file_hashes = dir_contents[dir_hash]
logger.debug(
"Indexing pushed dir '{}' with "
"'{}' nested files".format(
dir_checksum, len(file_checksums)
)
"'{}' nested files".format(dir_hash, len(file_hashes))
)
remote.index.update([dir_checksum], file_checksums)
remote.index.update([dir_hash], file_hashes)
return len(dir_plans[0]) + len(file_plans[0])
@@ -625,10 +708,10 @@ class LocalRemote(BaseRemote):
)
@staticmethod
def _log_missing_caches(checksum_info_dict):
def _log_missing_caches(hash_info_dict):
missing_caches = [
(md5, info)
for md5, info in checksum_info_dict.items()
for md5, info in hash_info_dict.items()
if info["status"] == STATUS_MISSING
]
if missing_caches:
@@ -641,74 +724,3 @@ class LocalRemote(BaseRemote):
"nor on remote. Missing cache files: {}".format(missing_desc)
)
logger.warning(msg)
def _unprotect_file(self, path):
if System.is_symlink(path) or System.is_hardlink(path):
logger.debug(f"Unprotecting '{path}'")
tmp = os.path.join(os.path.dirname(path), "." + uuid())
# The operations order is important here - if some application
# would access the file during the process of copyfile then it
# would get only the part of file. So, at first, the file should be
# copied with the temporary name, and then original file should be
# replaced by new.
copyfile(path, tmp, name="Unprotecting '{}'".format(relpath(path)))
remove(path)
os.rename(tmp, path)
else:
logger.debug(
"Skipping copying for '{}', since it is not "
"a symlink or a hardlink.".format(path)
)
os.chmod(path, self._file_mode)
def _unprotect_dir(self, path):
assert is_working_tree(self.repo.tree)
for fname in self.repo.tree.walk_files(path):
self._unprotect_file(fname)
def unprotect(self, path_info):
path = path_info.fspath
if not os.path.exists(path):
raise DvcException(f"can't unprotect non-existing data '{path}'")
if os.path.isdir(path):
self._unprotect_dir(path)
else:
self._unprotect_file(path)
def protect(self, path_info):
path = os.fspath(path_info)
mode = self.CACHE_MODE
try:
os.chmod(path, mode)
except OSError as exc:
# There is nothing we need to do in case of a read-only file system
if exc.errno == errno.EROFS:
return
# In shared cache scenario, we might not own the cache file, so we
# need to check if cache file is already protected.
if exc.errno not in [errno.EPERM, errno.EACCES]:
raise
actual = stat.S_IMODE(os.stat(path).st_mode)
if actual != mode:
raise
def _remove_unpacked_dir(self, checksum):
info = self.checksum_to_path_info(checksum)
path_info = info.with_name(info.name + self.UNPACKED_DIR_SUFFIX)
self.remove(path_info)
def is_protected(self, path_info):
try:
mode = os.stat(path_info).st_mode
except FileNotFoundError:
return False
return stat.S_IMODE(mode) == self.CACHE_MODE
+28 -31
View File
@@ -1,19 +1,18 @@
import logging
import os
import posixpath
import threading
from funcy import cached_property, wrap_prop
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
class OSSRemote(BaseRemote):
class OSSRemoteTree(BaseRemoteTree):
"""
oss2 document:
https://www.alibabacloud.com/help/doc-detail/32026.htm
@@ -33,7 +32,7 @@ class OSSRemote(BaseRemote):
"""
scheme = Schemes.OSS
path_cls = CloudURLInfo
PATH_CLS = CloudURLInfo
REQUIRES = {"oss2": "oss2"}
PARAM_CHECKSUM = "etag"
COPY_POLL_SECONDS = 5
@@ -43,7 +42,7 @@ class OSSRemote(BaseRemote):
super().__init__(repo, config)
url = config.get("url")
self.path_info = self.path_cls(url) if url else None
self.path_info = self.PATH_CLS(url) if url else None
self.endpoint = config.get("oss_endpoint") or os.getenv("OSS_ENDPOINT")
@@ -83,6 +82,30 @@ class OSSRemote(BaseRemote):
)
return bucket
def _generate_download_url(self, path_info, expires=3600):
assert path_info.bucket == self.path_info.bucket
return self.oss_service.sign_url("GET", path_info.path, expires)
def exists(self, path_info):
paths = self._list_paths(path_info)
return any(path_info.path == path for path in paths)
def _list_paths(self, path_info):
import oss2
for blob in oss2.ObjectIterator(
self.oss_service, prefix=path_info.path
):
yield blob.key
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(path_info):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
@@ -90,23 +113,6 @@ class OSSRemote(BaseRemote):
logger.debug(f"Removing oss://{path_info}")
self.oss_service.delete_object(path_info.path)
def _list_paths(self, prefix, progress_callback=None):
import oss2
for blob in oss2.ObjectIterator(self.oss_service, prefix=prefix):
if progress_callback:
progress_callback()
yield blob.key
def list_cache_paths(self, prefix=None, progress_callback=None):
if prefix:
prefix = posixpath.join(
self.path_info.path, prefix[:2], prefix[2:]
)
else:
prefix = self.path_info.path
return self._list_paths(prefix, progress_callback)
def _upload(
self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
):
@@ -122,12 +128,3 @@ class OSSRemote(BaseRemote):
self.oss_service.get_object_to_file(
from_info.path, to_file, progress_callback=pbar.update_to
)
def _generate_download_url(self, path_info, expires=3600):
assert path_info.bucket == self.path_info.bucket
return self.oss_service.sign_url("GET", path_info.path, expires)
def exists(self, path_info):
paths = self._list_paths(path_info.path)
return any(path_info.path == path for path in paths)
+141 -159
View File
@@ -1,6 +1,5 @@
import logging
import os
import posixpath
import threading
from funcy import cached_property, wrap_prop
@@ -9,15 +8,15 @@ from dvc.config import ConfigError
from dvc.exceptions import DvcException, ETagMismatchError
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
class S3Remote(BaseRemote):
class S3RemoteTree(BaseRemoteTree):
scheme = Schemes.S3
path_cls = CloudURLInfo
PATH_CLS = CloudURLInfo
REQUIRES = {"boto3": "boto3"}
PARAM_CHECKSUM = "etag"
@@ -25,7 +24,7 @@ class S3Remote(BaseRemote):
super().__init__(repo, config)
url = config.get("url", "s3://")
self.path_info = self.path_cls(url)
self.path_info = self.PATH_CLS(url)
self.region = config.get("region")
self.profile = config.get("profile")
@@ -77,9 +76,6 @@ class S3Remote(BaseRemote):
return obj["ETag"].strip('"')
def get_file_checksum(self, path_info):
return self.get_etag(self.s3, path_info.bucket, path_info.path)
@staticmethod
def get_head_object(s3, bucket, path, *args, **kwargs):
@@ -89,6 +85,136 @@ class S3Remote(BaseRemote):
raise DvcException(f"s3://{bucket}/{path} does not exist") from exc
return obj
def _append_aws_grants_to_extra_args(self, config):
# Keys for extra_args can be one of the following list:
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS
"""
ALLOWED_UPLOAD_ARGS = [
'ACL', 'CacheControl', 'ContentDisposition', 'ContentEncoding',
'ContentLanguage', 'ContentType', 'Expires', 'GrantFullControl',
'GrantRead', 'GrantReadACP', 'GrantWriteACP', 'Metadata',
'RequestPayer', 'ServerSideEncryption', 'StorageClass',
'SSECustomerAlgorithm', 'SSECustomerKey', 'SSECustomerKeyMD5',
'SSEKMSKeyId', 'WebsiteRedirectLocation'
]
"""
grants = {
"grant_full_control": "GrantFullControl",
"grant_read": "GrantRead",
"grant_read_acp": "GrantReadACP",
"grant_write_acp": "GrantWriteACP",
}
for grant_option, extra_args_key in grants.items():
if config.get(grant_option):
if self.acl:
raise ConfigError(
"`acl` and `grant_*` AWS S3 config options "
"are mutually exclusive"
)
self.extra_args[extra_args_key] = config.get(grant_option)
def _generate_download_url(self, path_info, expires=3600):
params = {"Bucket": path_info.bucket, "Key": path_info.path}
return self.s3.generate_presigned_url(
ClientMethod="get_object", Params=params, ExpiresIn=int(expires)
)
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def isdir(self, path_info):
# S3 doesn't have a concept for directories.
#
# Using `head_object` with a path pointing to a directory
# will throw a 404 error.
#
# A reliable way to know if a given path is a directory is by
# checking if there are more files sharing the same prefix
# with a `list_objects` call.
#
# We need to make sure that the path ends with a forward slash,
# since we can end with false-positives like the following example:
#
# bucket
# └── data
# ├── alice
# └── alpha
#
# Using `data/al` as prefix will return `[data/alice, data/alpha]`,
# While `data/al/` will return nothing.
#
dir_path = path_info / ""
return bool(list(self._list_paths(dir_path, max_items=1)))
def isfile(self, path_info):
from botocore.exceptions import ClientError
if path_info.path.endswith("/"):
return False
try:
self.s3.head_object(Bucket=path_info.bucket, Key=path_info.path)
except ClientError as exc:
if exc.response["Error"]["Code"] != "404":
raise
return False
return True
def _list_objects(self, path_info, max_items=None):
""" Read config for list object api, paginate through list objects."""
kwargs = {
"Bucket": path_info.bucket,
"Prefix": path_info.path,
"PaginationConfig": {"MaxItems": max_items},
}
paginator = self.s3.get_paginator(self.list_objects_api)
for page in paginator.paginate(**kwargs):
contents = page.get("Contents", ())
yield from contents
def _list_paths(self, path_info, max_items=None):
return (
item["Key"] for item in self._list_objects(path_info, max_items)
)
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(path_info / "", **kwargs):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != "s3":
raise NotImplementedError
logger.debug(f"Removing {path_info}")
self.s3.delete_object(Bucket=path_info.bucket, Key=path_info.path)
def makedirs(self, path_info):
# We need to support creating empty directories, which means
# creating an object with an empty body and a trailing slash `/`.
#
# We are not creating directory objects for every parent prefix,
# as it is not required.
if not path_info.path:
return
dir_path = path_info / ""
self.s3.put_object(Bucket=path_info.bucket, Key=dir_path.path, Body="")
def copy(self, from_info, to_info):
self._copy(self.s3, from_info, to_info, self.extra_args)
@classmethod
def _copy_multipart(
cls, s3, from_info, to_info, size, n_parts, extra_args
@@ -101,7 +227,7 @@ class S3Remote(BaseRemote):
parts = []
byte_position = 0
for i in range(1, n_parts + 1):
obj = cls.get_head_object(
obj = S3RemoteTree.get_head_object(
s3, from_info.bucket, from_info.path, PartNumber=i
)
part_size = obj["ContentLength"]
@@ -156,7 +282,9 @@ class S3Remote(BaseRemote):
# object is transfered in the same chunks as it was originally.
from boto3.s3.transfer import TransferConfig
obj = cls.get_head_object(s3, from_info.bucket, from_info.path)
obj = S3RemoteTree.get_head_object(
s3, from_info.bucket, from_info.path
)
etag = obj["ETag"].strip('"')
size = obj["ContentLength"]
@@ -176,114 +304,12 @@ class S3Remote(BaseRemote):
Config=TransferConfig(multipart_threshold=size + 1),
)
cached_etag = cls.get_etag(s3, to_info.bucket, to_info.path)
cached_etag = S3RemoteTree.get_etag(s3, to_info.bucket, to_info.path)
if etag != cached_etag:
raise ETagMismatchError(etag, cached_etag)
def copy(self, from_info, to_info):
self._copy(self.s3, from_info, to_info, self.extra_args)
def remove(self, path_info):
if path_info.scheme != "s3":
raise NotImplementedError
logger.debug(f"Removing {path_info}")
self.s3.delete_object(Bucket=path_info.bucket, Key=path_info.path)
def _list_objects(
self, path_info, max_items=None, prefix=None, progress_callback=None
):
""" Read config for list object api, paginate through list objects."""
kwargs = {
"Bucket": path_info.bucket,
"Prefix": path_info.path,
"PaginationConfig": {"MaxItems": max_items},
}
if prefix:
kwargs["Prefix"] = posixpath.join(path_info.path, prefix[:2])
paginator = self.s3.get_paginator(self.list_objects_api)
for page in paginator.paginate(**kwargs):
contents = page.get("Contents", ())
if progress_callback:
for item in contents:
progress_callback()
yield item
else:
yield from contents
def _list_paths(
self, path_info, max_items=None, prefix=None, progress_callback=None
):
return (
item["Key"]
for item in self._list_objects(
path_info, max_items, prefix, progress_callback
)
)
def list_cache_paths(self, prefix=None, progress_callback=None):
return self._list_paths(
self.path_info, prefix=prefix, progress_callback=progress_callback
)
def isfile(self, path_info):
from botocore.exceptions import ClientError
if path_info.path.endswith("/"):
return False
try:
self.s3.head_object(Bucket=path_info.bucket, Key=path_info.path)
except ClientError as exc:
if exc.response["Error"]["Code"] != "404":
raise
return False
return True
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def makedirs(self, path_info):
# We need to support creating empty directories, which means
# creating an object with an empty body and a trailing slash `/`.
#
# We are not creating directory objects for every parent prefix,
# as it is not required.
if not path_info.path:
return
dir_path = path_info / ""
self.s3.put_object(Bucket=path_info.bucket, Key=dir_path.path, Body="")
def isdir(self, path_info):
# S3 doesn't have a concept for directories.
#
# Using `head_object` with a path pointing to a directory
# will throw a 404 error.
#
# A reliable way to know if a given path is a directory is by
# checking if there are more files sharing the same prefix
# with a `list_objects` call.
#
# We need to make sure that the path ends with a forward slash,
# since we can end with false-positives like the following example:
#
# bucket
# └── data
# ├── alice
# └── alpha
#
# Using `data/al` as prefix will return `[data/alice, data/alpha]`,
# While `data/al/` will return nothing.
#
dir_path = path_info / ""
return bool(list(self._list_paths(dir_path, max_items=1)))
def get_file_hash(self, path_info):
return self.get_etag(self.s3, path_info.bucket, path_info.path)
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
total = os.path.getsize(from_file)
@@ -311,47 +337,3 @@ class S3Remote(BaseRemote):
self.s3.download_file(
from_info.bucket, from_info.path, to_file, Callback=pbar.update
)
def _generate_download_url(self, path_info, expires=3600):
params = {"Bucket": path_info.bucket, "Key": path_info.path}
return self.s3.generate_presigned_url(
ClientMethod="get_object", Params=params, ExpiresIn=int(expires)
)
def walk_files(self, path_info, max_items=None):
for fname in self._list_paths(path_info / "", max_items):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def _append_aws_grants_to_extra_args(self, config):
# Keys for extra_args can be one of the following list:
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS
"""
ALLOWED_UPLOAD_ARGS = [
'ACL', 'CacheControl', 'ContentDisposition', 'ContentEncoding',
'ContentLanguage', 'ContentType', 'Expires', 'GrantFullControl',
'GrantRead', 'GrantReadACP', 'GrantWriteACP', 'Metadata',
'RequestPayer', 'ServerSideEncryption', 'StorageClass',
'SSECustomerAlgorithm', 'SSECustomerKey', 'SSECustomerKeyMD5',
'SSEKMSKeyId', 'WebsiteRedirectLocation'
]
"""
grants = {
"grant_full_control": "GrantFullControl",
"grant_read": "GrantRead",
"grant_read_acp": "GrantReadACP",
"grant_write_acp": "GrantWriteACP",
}
for grant_option, extra_args_key in grants.items():
if config.get(grant_option):
if self.acl:
raise ConfigError(
"`acl` and `grant_*` AWS S3 config options "
"are mutually exclusive"
)
self.extra_args[extra_args_key] = config.get(grant_option)
+61 -57
View File
@@ -14,7 +14,7 @@ from funcy import first, memoize, silent, wrap_with
import dvc.prompt as prompt
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree, Remote
from dvc.remote.pool import get_connection
from dvc.scheme import Schemes
from dvc.utils import to_chunks
@@ -33,21 +33,21 @@ def ask_password(host, user, port):
)
class SSHRemote(BaseRemote):
class SSHRemoteTree(BaseRemoteTree):
scheme = Schemes.SSH
REQUIRES = {"paramiko": "paramiko"}
JOBS = 4
PARAM_CHECKSUM = "md5"
DEFAULT_PORT = 22
TIMEOUT = 1800
# At any given time some of the connections will go over network and
# paramiko stuff, so we would ideally have it double of server processors.
# We use conservative setting of 4 instead to not exhaust max sessions.
CHECKSUM_JOBS = 4
DEFAULT_CACHE_TYPES = ["copy"]
TRAVERSE_PREFIX_LEN = 2
DEFAULT_CACHE_TYPES = ["copy"]
DEFAULT_PORT = 22
TIMEOUT = 1800
def __init__(self, repo, config):
super().__init__(repo, config)
@@ -69,7 +69,7 @@ class SSHRemote(BaseRemote):
or self._try_get_ssh_config_port(user_ssh_config)
or self.DEFAULT_PORT
)
self.path_info = self.path_cls.from_parts(
self.path_info = self.PATH_CLS.from_parts(
scheme=self.scheme,
host=host,
user=user,
@@ -103,7 +103,7 @@ class SSHRemote(BaseRemote):
def _load_user_ssh_config(hostname):
import paramiko
user_config_file = SSHRemote.ssh_config_filename()
user_config_file = SSHRemoteTree.ssh_config_filename()
user_ssh_config = {}
if hostname and os.path.exists(user_config_file):
ssh_config = paramiko.SSHConfig()
@@ -148,17 +148,22 @@ class SSHRemote(BaseRemote):
sock=self.sock,
)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb", "wb"}
with self.ssh(path_info) as ssh, closing(
ssh.sftp.open(path_info.path, mode)
) as fd:
if "b" in mode:
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
def exists(self, path_info):
with self.ssh(path_info) as ssh:
return ssh.exists(path_info.path)
def get_file_checksum(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(path_info) as ssh:
return ssh.md5(path_info.path)
def isdir(self, path_info):
with self.ssh(path_info) as ssh:
return ssh.isdir(path_info.path)
@@ -167,9 +172,29 @@ class SSHRemote(BaseRemote):
with self.ssh(path_info) as ssh:
return ssh.isfile(path_info.path)
def getsize(self, path_info):
def walk_files(self, path_info, **kwargs):
with self.ssh(path_info) as ssh:
return ssh.getsize(path_info.path)
for fname in ssh.walk_files(path_info.path):
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(path_info) as ssh:
ssh.remove(path_info.path)
def makedirs(self, path_info):
with self.ssh(path_info) as ssh:
ssh.makedirs(path_info.path)
def move(self, from_info, to_info, mode=None):
assert mode is None
if from_info.scheme != self.scheme or to_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(from_info) as ssh:
ssh.move(from_info.path, to_info.path)
def copy(self, from_info, to_info):
if not from_info.scheme == to_info.scheme == self.scheme:
@@ -212,20 +237,16 @@ class SSHRemote(BaseRemote):
with self.ssh(from_info) as ssh:
ssh.reflink(from_info.path, to_info.path)
def remove(self, path_info):
def get_file_hash(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(path_info) as ssh:
ssh.remove(path_info.path)
return ssh.md5(path_info.path)
def move(self, from_info, to_info, mode=None):
assert mode is None
if from_info.scheme != self.scheme or to_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(from_info) as ssh:
ssh.move(from_info.path, to_info.path)
def getsize(self, path_info):
with self.ssh(path_info) as ssh:
return ssh.getsize(path_info.path)
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
assert from_info.isin(self.path_info)
@@ -247,19 +268,7 @@ class SSHRemote(BaseRemote):
no_progress_bar=no_progress_bar,
)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb", "wb"}
with self.ssh(path_info) as ssh, closing(
ssh.sftp.open(path_info.path, mode)
) as fd:
if "b" in mode:
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
def list_cache_paths(self, prefix=None, progress_callback=None):
def list_paths(self, prefix=None, progress_callback=None):
if prefix:
root = posixpath.join(self.path_info.path, prefix[:2])
else:
@@ -275,15 +284,8 @@ class SSHRemote(BaseRemote):
else:
yield from ssh.walk_files(root)
def walk_files(self, path_info):
with self.ssh(path_info) as ssh:
for fname in ssh.walk_files(path_info.path):
yield path_info.replace(path=fname)
def makedirs(self, path_info):
with self.ssh(path_info) as ssh:
ssh.makedirs(path_info.path)
class SSHRemote(Remote):
def batch_exists(self, path_infos, callback):
def _exists(chunk_and_channel):
chunk, channel = chunk_and_channel
@@ -299,7 +301,7 @@ class SSHRemote(BaseRemote):
callback(path)
return ret
with self.ssh(path_infos[0]) as ssh:
with self.tree.ssh(path_infos[0]) as ssh:
channels = ssh.open_max_sftp_channels()
max_workers = len(channels)
@@ -312,32 +314,34 @@ class SSHRemote(BaseRemote):
return results
def cache_exists(self, checksums, jobs=None, name=None):
def hashes_exist(self, hashes, jobs=None, name=None):
"""This is older implementation used in remote/base.py
We are reusing it in RemoteSSH, because SSH's batch_exists proved to be
faster than current approach (relying on exists(path_info)) applied in
remote/base.
"""
if not self.CAN_TRAVERSE:
return list(set(checksums) & set(self.all()))
if not self.tree.CAN_TRAVERSE:
return list(set(hashes) & set(self.tree.all()))
# possibly prompt for credentials before "Querying" progress output
self.ensure_credentials()
self.tree.ensure_credentials()
with Tqdm(
desc="Querying "
+ ("cache in " + name if name else "remote cache"),
total=len(checksums),
total=len(hashes),
unit="file",
) as pbar:
def exists_with_progress(chunks):
return self.batch_exists(chunks, callback=pbar.update_msg)
with ThreadPoolExecutor(max_workers=jobs or self.JOBS) as executor:
path_infos = [self.checksum_to_path_info(x) for x in checksums]
chunks = to_chunks(path_infos, num_chunks=self.JOBS)
with ThreadPoolExecutor(
max_workers=jobs or self.tree.JOBS
) as executor:
path_infos = [self.hash_to_path_info(x) for x in hashes]
chunks = to_chunks(path_infos, num_chunks=self.tree.JOBS)
results = executor.map(exists_with_progress, chunks)
in_remote = itertools.chain.from_iterable(results)
ret = list(itertools.compress(checksums, in_remote))
ret = list(itertools.compress(hashes, in_remote))
return ret
+10 -11
View File
@@ -111,14 +111,14 @@ class Repo:
friendly=True,
)
self.cache = Cache(self)
self.cloud = DataCloud(self)
if not scm:
# NOTE: storing state and link_state in the repository itself to
# avoid any possible state corruption in 'shared cache dir'
# scenario.
self.state = State(self)
self.cache = Cache(self)
self.cloud = DataCloud(self)
self.state = State(self.cache.local)
self.stage_cache = StageCache(self)
@@ -187,7 +187,7 @@ class Repo:
return Repo(root_dir)
def unprotect(self, target):
return self.cache.local.unprotect(PathInfo(target))
return self.cache.local.tree.unprotect(PathInfo(target))
def _ignore(self):
flist = [self.config.files["local"], self.tmp_dir]
@@ -527,14 +527,13 @@ class Repo:
for root, dirs, files in self.tree.walk(self.root_dir):
for file_name in filter(is_valid_filename, files):
path = os.path.join(root, file_name)
stages.extend(self.get_stages(path))
new_stages = self.get_stages(os.path.join(root, file_name))
stages.extend(new_stages)
outs.update(
out.fspath
for stage in stages
for out in (
out for out in stage.outs if out.scheme == "local"
)
for stage in new_stages
for out in stage.outs
if out.scheme == "local"
)
dirs[:] = [d for d in dirs if os.path.join(root, d) not in outs]
return stages
+16 -5
View File
@@ -21,7 +21,9 @@ logger = logging.getLogger(__name__)
@locked
@scm_context
def add(repo, targets, recursive=False, no_commit=False, fname=None):
def add(
repo, targets, recursive=False, no_commit=False, fname=None, external=False
):
if recursive and fname:
raise RecursiveAddingWhileUsingFilename()
@@ -52,7 +54,9 @@ def add(repo, targets, recursive=False, no_commit=False, fname=None):
)
)
stages = _create_stages(repo, sub_targets, fname, pbar=pbar)
stages = _create_stages(
repo, sub_targets, fname, pbar=pbar, external=external
)
try:
repo.check_modified_graph(stages)
@@ -120,7 +124,7 @@ def _find_all_targets(repo, target, recursive):
return [target]
def _create_stages(repo, targets, fname, pbar=None):
def _create_stages(repo, targets, fname, pbar=None, external=False):
from dvc.stage import Stage, create_stage
stages = []
@@ -132,9 +136,16 @@ def _create_stages(repo, targets, fname, pbar=None):
unit="file",
):
path, wdir, out = resolve_paths(repo, out)
stage = create_stage(Stage, repo, fname or path, wdir=wdir, outs=[out])
stage = create_stage(
Stage,
repo,
fname or path,
wdir=wdir,
outs=[out],
external=external,
)
if stage:
Dvcfile(repo, stage.path).remove_with_prompt(force=True)
Dvcfile(repo, stage.path).remove()
repo._reset()
+1 -1
View File
@@ -37,7 +37,7 @@ def diff(self, a_rev="HEAD", b_rev=None):
def _to_checksum(output):
if on_working_tree:
return self.cache.local.get_checksum(output.path_info)
return self.cache.local.get_hash(output.path_info)
return output.checksum
def _exists(output):
+1 -1
View File
@@ -72,7 +72,7 @@ def _fetch(
if failed:
raise DownloadError(failed)
return downloaded
return downloaded + len(used_run_cache)
def _fetch_external(self, repo_url, repo_rev, files, jobs):
+12 -9
View File
@@ -8,8 +8,10 @@ from . import locked
logger = logging.getLogger(__name__)
def _do_gc(typ, func, clist, jobs=None):
removed = func(clist, jobs=jobs)
def _do_gc(typ, remote, clist, jobs=None):
from dvc.remote.base import Remote
removed = Remote.gc(clist, remote, jobs=jobs)
if not removed:
logger.info(f"No unused '{typ}' cache to remove.")
@@ -74,22 +76,23 @@ def gc(
)
)
_do_gc("local", self.cache.local.gc, used, jobs)
# treat caches as remotes for garbage collection
_do_gc("local", self.cache.local, used, jobs)
if self.cache.s3:
_do_gc("s3", self.cache.s3.gc, used, jobs)
_do_gc("s3", self.cache.s3, used, jobs)
if self.cache.gs:
_do_gc("gs", self.cache.gs.gc, used, jobs)
_do_gc("gs", self.cache.gs, used, jobs)
if self.cache.ssh:
_do_gc("ssh", self.cache.ssh.gc, used, jobs)
_do_gc("ssh", self.cache.ssh, used, jobs)
if self.cache.hdfs:
_do_gc("hdfs", self.cache.hdfs.gc, used, jobs)
_do_gc("hdfs", self.cache.hdfs, used, jobs)
if self.cache.azure:
_do_gc("azure", self.cache.azure.gc, used, jobs)
_do_gc("azure", self.cache.azure, used, jobs)
if cloud:
_do_gc("remote", self.cloud.get_remote(remote, "gc -c").gc, used, jobs)
_do_gc("remote", self.cloud.get_remote(remote, "gc -c"), used, jobs)
+2
View File
@@ -16,4 +16,6 @@ def get_url(url, out=None):
(dep,) = dependency.loads_from(None, [url])
(out,) = output.loads_from(None, [out], use_cache=False)
dep.save()
dep.download(out)
out.save()
+1 -1
View File
@@ -39,7 +39,7 @@ def imp_url(self, url, out=None, fname=None, erepo=None, frozen=True):
return None
dvcfile = Dvcfile(self, stage.path)
dvcfile.remove_with_prompt(force=True)
dvcfile.remove()
try:
self.check_modified_graph([stage])
+219 -7
View File
@@ -1,29 +1,133 @@
import logging
from funcy import first, project
from dvc.exceptions import DvcException, NoPlotsError, OutputNotFoundError
from dvc.repo.tree import RepoTree
from dvc.schema import PLOT_PROPS
from dvc.utils import relpath
logger = logging.getLogger(__name__)
class NotAPlotError(DvcException):
def __init__(self, out):
super().__init__(
f"'{out}' is not a plot. Use `dvc plots modify` to change that."
)
class PropsNotFoundError(DvcException):
pass
class Plots:
def __init__(self, repo):
self.repo = repo
def show(self, *args, **kwargs):
from .show import show
def collect(self, targets=None, revs=None):
"""Collects all props and data for plots.
return show(self.repo, *args, **kwargs)
Returns a structure like:
{rev: {plots.csv: {
props: {x: ..., "header": ..., ...},
data: "...data as a string...",
}}}
Data parsing is postponed, since it's affected by props.
"""
targets = [targets] if isinstance(targets, str) else targets or []
data = {}
for rev in self.repo.brancher(revs=revs):
# .brancher() adds unwanted workspace
if revs is not None and rev not in revs:
continue
rev = rev or "workspace"
tree = RepoTree(self.repo)
plots = _collect_plots(self.repo, targets, rev)
for path_info, props in plots.items():
datafile = relpath(path_info, self.repo.root_dir)
if rev not in data:
data[rev] = {}
data[rev].update({datafile: {"props": props}})
# Load data from git or dvc cache
try:
with tree.open(path_info) as fd:
data[rev][datafile]["data"] = fd.read()
except FileNotFoundError as e:
# This might happen simply because cache is absent
print(e)
pass
return data
@staticmethod
def render(data, revs=None, props=None, templates=None):
"""Renders plots"""
props = props or {}
# Merge data by plot file and apply overriding props
plots = _prepare_plots(data, revs, props)
return {
datafile: _render(datafile, desc["data"], desc["props"], templates)
for datafile, desc in plots.items()
}
def show(self, targets=None, revs=None, props=None):
from .data import NoMetricInHistoryError
data = self.collect(targets, revs)
# If any mentioned plot doesn't have any data then that's an error
targets = [targets] if isinstance(targets, str) else targets or []
for target in targets:
if not any("data" in d[target] for d in data.values()):
raise NoMetricInHistoryError(target)
# No data at all is a special error with a special message
if not data:
raise NoPlotsError()
return self.render(data, revs, props, self.repo.plot_templates)
def diff(self, *args, **kwargs):
from .diff import diff
return diff(self.repo, *args, **kwargs)
@staticmethod
def _unset(out, props):
missing = list(set(props) - set(out.plot.keys()))
if missing:
raise PropsNotFoundError(
f"display properties {missing} not found in plot '{out}'"
)
for prop in props:
out.plot.pop(prop)
def modify(self, path, props=None, unset=None):
from dvc.dvcfile import Dvcfile
props = props or {}
template = props.get("template")
if template:
self.repo.plot_templates.get_template(template)
(out,) = self.repo.find_outs_by_path(path)
if not out.plot and unset is not None:
raise NotAPlotError(out)
# This out will become a plot unless it is one already
if not isinstance(out.plot, dict):
out.plot = {}
for field in unset or ():
out.plot.pop(field, None)
out.plot.update(props or {})
if unset:
self._unset(out, unset)
out.plot.update(props)
# Empty dict will move it to non-plots
if not out.plot:
@@ -32,4 +136,112 @@ class Plots:
out.verify_metric()
dvcfile = Dvcfile(self.repo, out.stage.path)
dvcfile.dump(out.stage, update_pipeline=True)
dvcfile.dump(out.stage, update_pipeline=True, no_lock=True)
def _collect_plots(repo, targets=None, rev=None):
def _targets_to_outs(targets):
for t in targets:
try:
(out,) = repo.find_outs_by_path(t)
yield out
except OutputNotFoundError:
logger.warning(
"File '{}' was not found at: '{}'. It will not be "
"plotted.".format(t, rev)
)
if targets:
outs = _targets_to_outs(targets)
else:
outs = (out for stage in repo.stages for out in stage.outs if out.plot)
return {out.path_info: _plot_props(out) for out in outs}
def _plot_props(out):
if not out.plot:
raise NotAPlotError(out)
if isinstance(out.plot, list):
raise DvcException("Multiple plots per data file not supported.")
if isinstance(out.plot, bool):
return {}
return project(out.plot, PLOT_PROPS)
def _prepare_plots(data, revs, props):
"""Groups data by plot file.
Also resolves props conflicts between revs and applies global props.
"""
# we go in order revs are supplied on props conflict first ones win.
revs = iter(data) if revs is None else revs
plots, props_revs = {}, {}
for rev in revs:
# Asked for revision without data
if rev not in data:
continue
for datafile, desc in data[rev].items():
# props from command line overwrite plot props from out definition
full_props = {**desc["props"], **props}
if datafile in plots:
saved = plots[datafile]
if saved["props"] != full_props:
logger.warning(
f"Inconsistent plot props for '{datafile}' in "
f"'{props_revs[datafile]}' and '{rev}'. "
f"Going to use ones from '{props_revs[datafile]}'"
)
saved["data"][rev] = desc["data"]
else:
plots[datafile] = {
"props": full_props,
"data": {rev: desc["data"]},
}
# Save rev we got props from
props_revs[datafile] = rev
return plots
def _render(datafile, datas, props, templates):
from .data import plot_data, PlotData
# Copy it to not modify a passed value
props = props.copy()
# Add x and y to fields if set
fields = props.get("fields")
if fields is not None:
fields = {*fields, props.get("x"), props.get("y")} - {None}
template = templates.load(props.get("template") or "default")
# If x is not set add index field
if not props.get("x") and template.has_anchor("x"):
props["append_index"] = True
props["x"] = PlotData.INDEX_FIELD
# Parse all data, preprocess it and collect as a list of dicts
data = []
for rev, datablob in datas.items():
rev_data = plot_data(datafile, rev, datablob).to_datapoints(
fields=fields,
path=props.get("path"),
header=props.get("header", True),
append_index=props.get("append_index", False),
)
data.extend(rev_data)
# If y is not set then use last field not used yet
if not props.get("y") and template.has_anchor("y"):
fields = list(first(data))
skip = (PlotData.REVISION_FIELD, props.get("x"))
props["y"] = first(f for f in reversed(fields) if f not in skip)
return template.render(data, props=props)
+7 -65
View File
@@ -1,7 +1,6 @@
import csv
import io
import json
import logging
import os
from collections import OrderedDict
from copy import copy
@@ -10,9 +9,7 @@ import yaml
from funcy import first
from yaml import SafeLoader
from dvc.exceptions import DvcException, PathMissingError
logger = logging.getLogger(__name__)
from dvc.exceptions import DvcException
class PlotMetricTypeError(DvcException):
@@ -31,21 +28,6 @@ class PlotDataStructureError(DvcException):
)
class JsonParsingError(DvcException):
def __init__(self, file):
super().__init__(
"Failed to infer data structure from '{}'. Did you forget "
"to specify JSONpath?".format(file)
)
class NoMetricOnRevisionError(DvcException):
def __init__(self, path, revision):
self.path = path
self.revision = revision
super().__init__(f"Could not find '{path}' on revision '{revision}'")
class NoMetricInHistoryError(DvcException):
def __init__(self, path):
super().__init__(f"Could not find '{path}'.")
@@ -77,7 +59,7 @@ def _filter_fields(data_points, filename, revision, fields=None, **kwargs):
if keys & fields != fields:
raise DvcException(
"Could not find fields: '{}' for '{}' at '{}'.".format(
", " "".join(fields), filename, revision
", ".join(fields), filename, revision
)
)
@@ -199,10 +181,10 @@ class CSVPlotData(PlotData):
super().__init__(filename, revision, content)
self.delimiter = delimiter
def raw(self, csv_header=True, **kwargs):
def raw(self, header=True, **kwargs):
first_row = first(csv.reader(io.StringIO(self.content)))
if csv_header:
if header:
reader = csv.DictReader(
io.StringIO(self.content), delimiter=self.delimiter,
)
@@ -237,46 +219,6 @@ class YAMLPlotData(PlotData):
return yaml.load(self.content, OrderedLoader)
def _load_from_revision(repo, datafile, revision):
from dvc.repo.tree import RepoTree
tree = RepoTree(repo)
try:
with tree.open(datafile) as fobj:
datafile_content = fobj.read()
except (FileNotFoundError, PathMissingError):
raise NoMetricOnRevisionError(datafile, revision)
return plot_data(datafile, revision, datafile_content)
def _load_from_revisions(repo, datafile, revisions):
data = []
exceptions = []
for rev in repo.brancher(revs=revisions):
if rev == "workspace" and rev not in revisions:
continue
try:
data.append(_load_from_revision(repo, datafile, rev))
except NoMetricOnRevisionError as e:
exceptions.append(e)
except PlotMetricTypeError:
raise
except (yaml.error.YAMLError, json.decoder.JSONDecodeError, csv.Error):
logger.error(f"Failed to parse '{datafile}' at '{rev}'.")
raise
if not data and exceptions:
raise NoMetricInHistoryError(datafile)
else:
for e in exceptions:
logger.warning(
"File '{}' was not found at: '{}'. It will not be "
"plotted.".format(e.path, e.revision)
)
return data
def _processors(self):
parent_processors = super()._processors()
return [_find_data] + parent_processors
-210
View File
@@ -1,210 +0,0 @@
import copy
import logging
import os
from funcy import first, last, project
from dvc.exceptions import DvcException
from dvc.repo import locked
from dvc.schema import PLOT_PROPS
from .data import NoMetricInHistoryError, PlotData
from .template import NoDataForTemplateError, Template
logger = logging.getLogger(__name__)
class TooManyDataSourcesError(DvcException):
def __init__(self, datafile, template_datafiles):
super().__init__(
"Unable to infer which of possible data sources: '{}' "
"should be replaced with '{}'.".format(
", ".join(template_datafiles), datafile
)
)
class NoDataOrTemplateProvided(DvcException):
def __init__(self):
super().__init__("Datafile or template is not specified.")
class TooManyTemplatesError(DvcException):
pass
def _evaluate_templatepath(repo, template=None):
if not template:
return repo.plot_templates.default_template
if os.path.exists(template):
return template
return repo.plot_templates.get_template(template)
@locked
def fill_template(
repo, datafile, template_path, revisions, props,
):
# Copy things to not modify passed values
props = props.copy()
fields = copy.copy(props.get("fields"))
if props.get("x") and fields:
fields.add(props.get("x"))
if props.get("y") and fields:
fields.add(props.get("y"))
template_datafiles, x_anchor, y_anchor = _parse_template(
template_path, datafile
)
append_index = x_anchor and not props.get("x")
if append_index:
props["x"] = PlotData.INDEX_FIELD
template_data = {}
for template_datafile in template_datafiles:
from .data import _load_from_revisions
plot_datas = _load_from_revisions(repo, template_datafile, revisions)
tmp_data = []
for pd in plot_datas:
rev_data_points = pd.to_datapoints(
fields=fields,
path=props.get("path"),
csv_header=props.get("csv_header", True),
append_index=append_index,
)
if y_anchor and not props.get("y"):
props["y"] = _infer_y_field(rev_data_points, props.get("x"))
tmp_data.extend(rev_data_points)
template_data[template_datafile] = tmp_data
if len(template_data) == 0:
raise NoDataForTemplateError(template_path)
content = Template.fill(
template_path, template_data, priority_datafile=datafile, props=props,
)
path = datafile or ",".join(template_datafiles)
return path, content
def _infer_y_field(rev_data_points, x_field):
all_fields = list(first(rev_data_points).keys())
all_fields.remove(PlotData.REVISION_FIELD)
if x_field and x_field in all_fields:
all_fields.remove(x_field)
y_field = last(all_fields)
return y_field
def _show(repo, datafile=None, revs=None, props=None):
if revs is None:
revs = ["workspace"]
if not datafile and not props.get("template"):
raise NoDataOrTemplateProvided()
template_path = _evaluate_templatepath(repo, props.get("template"))
plot_datafile, plot_content = fill_template(
repo, datafile, template_path, revs, props
)
return plot_datafile, plot_content
def _parse_template(template_path, priority_datafile):
with open(template_path) as fobj:
tempalte_content = fobj.read()
template_datafiles = Template.parse_data_anchors(tempalte_content)
if priority_datafile:
if len(template_datafiles) > 1:
raise TooManyDataSourcesError(
priority_datafile, template_datafiles
)
template_datafiles = {priority_datafile}
return (
template_datafiles,
Template.X_ANCHOR in tempalte_content,
Template.Y_ANCHOR in tempalte_content,
)
def _collect_plots(repo, targets=None):
from dvc.exceptions import OutputNotFoundError
from contextlib import suppress
def _targets_to_outs(targets):
for t in targets:
with suppress(OutputNotFoundError):
(out,) = repo.find_outs_by_path(t)
yield out
if targets:
outs = _targets_to_outs(targets)
else:
outs = (out for stage in repo.stages for out in stage.outs if out.plot)
return {str(out): _plot_props(out) for out in outs}
def _plot_props(out):
if not out.plot:
raise DvcException(
f"'{out}' is not a plot. Use `dvc plots modify` to change that."
)
if isinstance(out.plot, list):
raise DvcException("Multiple plots per data file not supported yet.")
if isinstance(out.plot, bool):
return {}
return project(out.plot, PLOT_PROPS)
def show(repo, targets=None, revs=None, props=None) -> dict:
if isinstance(targets, str):
targets = [targets]
if props is None:
props = {}
# Collect plot data files with associated props
plots = {}
for rev in repo.brancher(revs=revs):
if revs is not None and rev not in revs:
continue
for datafile, file_props in _collect_plots(repo, targets).items():
# props from command line overwrite plot props from out definition
full_props = {**file_props, **props}
if datafile in plots:
saved_rev, saved_props = plots[datafile]
if saved_props != props:
logger.warning(
f"Inconsistent plot props for '{datafile}' in "
f"'{saved_rev}' and '{rev}'. "
f"Going to use ones from '{saved_rev}'"
)
else:
plots[datafile] = rev, full_props
if not plots:
if targets:
raise NoMetricInHistoryError(", ".join(targets))
datafile, plot = _show(repo, datafile=None, revs=revs, props=props)
return {datafile: plot}
return {
datafile: _show(repo, datafile=datafile, revs=revs, props=props)[1]
for datafile, (_, props) in plots.items()
}
+113 -122
View File
@@ -1,26 +1,19 @@
import json
import logging
import os
import re
from funcy import cached_property
from dvc.exceptions import DvcException
from dvc.utils.fs import makedirs
logger = logging.getLogger(__name__)
class TemplateNotFoundError(DvcException):
def __init__(self, path):
super().__init__(f"Template '{path}' not found.")
class NoDataForTemplateError(DvcException):
def __init__(self, template_path):
super().__init__(
"No data provided for '{}'.".format(os.path.relpath(template_path))
)
class BadTemplateError(DvcException):
pass
class NoFieldInDataError(DvcException):
@@ -34,132 +27,74 @@ class Template:
INDENT = 4
SEPARATORS = (",", ": ")
EXTENSION = ".json"
ANCHOR = '"<DVC_METRIC_{}>"'
METRIC_DATA_ANCHOR = "<DVC_METRIC_DATA>"
X_ANCHOR = "<DVC_METRIC_X>"
Y_ANCHOR = "<DVC_METRIC_Y>"
TITLE_ANCHOR = "<DVC_METRIC_TITLE>"
X_TITLE_ANCHOR = "<DVC_METRIC_X_TITLE>"
Y_TITLE_ANCHOR = "<DVC_METRIC_Y_TITLE>"
X_LABEL_ANCHOR = "<DVC_METRIC_X_LABEL>"
Y_LABEL_ANCHOR = "<DVC_METRIC_Y_LABEL>"
def __init__(self, templates_dir):
self.plot_templates_dir = templates_dir
def __init__(self, content=None, name=None):
self.content = self.DEFAULT_CONTENT if content is None else content
self.name = name or self.DEFAULT_NAME
self.filename = self.name + self.EXTENSION
def dump(self):
makedirs(self.plot_templates_dir, exist_ok=True)
with open(
os.path.join(
self.plot_templates_dir, self.TEMPLATE_NAME + self.EXTENSION
),
"w",
) as fobj:
json.dump(
self.DEFAULT_CONTENT,
fobj,
indent=self.INDENT,
separators=self.SEPARATORS,
)
fobj.write("\n")
@staticmethod
def get_data_anchor(template_content):
regex = re.compile('"<DVC_METRIC_DATA[^>"]*>"')
return regex.findall(template_content)
@staticmethod
def parse_data_anchors(template_content):
data_files = {
Template.get_datafile(m)
for m in Template.get_data_anchor(template_content)
}
return {df for df in data_files if df}
@staticmethod
def get_datafile(anchor_string):
return (
anchor_string.replace("<", "")
.replace(">", "")
.replace('"', "")
.replace("DVC_METRIC_DATA", "")
.replace(",", "")
)
@staticmethod
def fill(
template_path, data, priority_datafile=None, props=None,
):
def render(self, data, props=None):
props = props or {}
with open(template_path) as fobj:
result_content = fobj.read()
if self.METRIC_DATA_ANCHOR not in self.content:
raise BadTemplateError(
f"Template '{self.filename}' is not using "
f"'{self.METRIC_DATA_ANCHOR}' anchor"
)
if props.get("x"):
Template._check_field_exists(data, props.get("x"))
if props.get("y"):
Template._check_field_exists(data, props.get("y"))
result_content = Template._replace_data_anchors(
result_content, data, priority_datafile
)
content = self._fill_anchor(self.content, "data", data)
content = self._fill_metadata(content, props)
result_content = Template._replace_metadata_anchors(
result_content, props
)
return content
return result_content
def has_anchor(self, name):
return self._anchor(name) in self.content
@classmethod
def _fill_anchor(cls, content, name, value):
value_str = json.dumps(
value, indent=cls.INDENT, separators=cls.SEPARATORS, sort_keys=True
)
return content.replace(cls._anchor(name), value_str)
@classmethod
def _anchor(cls, name):
return cls.ANCHOR.format(name.upper())
@classmethod
def _fill_metadata(cls, content, props):
props.setdefault("title", "")
props.setdefault("x_label", props.get("x"))
props.setdefault("y_label", props.get("y"))
names = ["title", "x", "y", "x_label", "y_label"]
for name in names:
value = props.get(name)
if value is not None:
content = cls._fill_anchor(content, name, value)
return content
@staticmethod
def _check_field_exists(data, field):
for file, data_points in data.items():
if not any(
field in data_point.keys() for data_point in data_points
):
raise NoFieldInDataError(field)
@staticmethod
def _replace_metadata_anchors(result_content, props):
props.setdefault("title", "")
props.setdefault("xlab", props.get("x"))
props.setdefault("ylab", props.get("y"))
replace_pairs = [
(Template.TITLE_ANCHOR, "title"),
(Template.X_ANCHOR, "x"),
(Template.Y_ANCHOR, "y"),
(Template.X_TITLE_ANCHOR, "xlab"),
(Template.Y_TITLE_ANCHOR, "ylab"),
]
for anchor, key in replace_pairs:
value = props.get(key)
if anchor in result_content and value is not None:
result_content = result_content.replace(anchor, value)
return result_content
@staticmethod
def _replace_data_anchors(result_content, data, priority_datafile):
for anchor in Template.get_data_anchor(result_content):
file = Template.get_datafile(anchor)
if not file or priority_datafile:
key = priority_datafile
else:
key = file
result_content = result_content.replace(
anchor,
json.dumps(
data[key],
indent=Template.INDENT,
separators=Template.SEPARATORS,
sort_keys=True,
),
)
return result_content
if not any(field in row for row in data):
raise NoFieldInDataError(field)
class DefaultLinearTemplate(Template):
TEMPLATE_NAME = "default"
DEFAULT_NAME = "default"
DEFAULT_CONTENT = {
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
@@ -170,12 +105,12 @@ class DefaultLinearTemplate(Template):
"x": {
"field": Template.X_ANCHOR,
"type": "quantitative",
"title": Template.X_TITLE_ANCHOR,
"title": Template.X_LABEL_ANCHOR,
},
"y": {
"field": Template.Y_ANCHOR,
"type": "quantitative",
"title": Template.Y_TITLE_ANCHOR,
"title": Template.Y_LABEL_ANCHOR,
"scale": {"zero": False},
},
"color": {"field": "rev", "type": "nominal"},
@@ -184,7 +119,7 @@ class DefaultLinearTemplate(Template):
class DefaultConfusionTemplate(Template):
TEMPLATE_NAME = "confusion"
DEFAULT_NAME = "confusion"
DEFAULT_CONTENT = {
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
"data": {"values": Template.METRIC_DATA_ANCHOR},
@@ -195,13 +130,13 @@ class DefaultConfusionTemplate(Template):
"field": Template.X_ANCHOR,
"type": "nominal",
"sort": "ascending",
"title": Template.X_TITLE_ANCHOR,
"title": Template.X_LABEL_ANCHOR,
},
"y": {
"field": Template.Y_ANCHOR,
"type": "nominal",
"sort": "ascending",
"title": Template.Y_TITLE_ANCHOR,
"title": Template.Y_LABEL_ANCHOR,
},
"color": {"aggregate": "count", "type": "quantitative"},
"facet": {"field": "rev", "type": "nominal"},
@@ -210,7 +145,7 @@ class DefaultConfusionTemplate(Template):
class DefaultScatterTemplate(Template):
TEMPLATE_NAME = "scatter"
DEFAULT_NAME = "scatter"
DEFAULT_CONTENT = {
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
"data": {"values": Template.METRIC_DATA_ANCHOR},
@@ -220,12 +155,12 @@ class DefaultScatterTemplate(Template):
"x": {
"field": Template.X_ANCHOR,
"type": "quantitative",
"title": Template.X_TITLE_ANCHOR,
"title": Template.X_LABEL_ANCHOR,
},
"y": {
"field": Template.Y_ANCHOR,
"type": "quantitative",
"title": Template.Y_TITLE_ANCHOR,
"title": Template.Y_LABEL_ANCHOR,
"scale": {"zero": False},
},
"color": {"field": "rev", "type": "nominal"},
@@ -233,12 +168,46 @@ class DefaultScatterTemplate(Template):
}
class SmoothLinearTemplate(Template):
DEFAULT_NAME = "smooth"
DEFAULT_CONTENT = {
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
"data": {"values": Template.METRIC_DATA_ANCHOR},
"title": Template.TITLE_ANCHOR,
"mark": {"type": "line"},
"encoding": {
"x": {
"field": Template.X_ANCHOR,
"type": "quantitative",
"title": Template.X_LABEL_ANCHOR,
},
"y": {
"field": Template.Y_ANCHOR,
"type": "quantitative",
"title": Template.Y_LABEL_ANCHOR,
"scale": {"zero": False},
},
"color": {"field": "rev", "type": "nominal"},
},
"transform": [
{
"loess": Template.Y_ANCHOR,
"on": Template.X_ANCHOR,
"groupby": ["rev"],
"bandwidth": 0.3,
}
],
}
class PlotTemplates:
TEMPLATES_DIR = "plots"
TEMPLATES = [
DefaultLinearTemplate,
DefaultConfusionTemplate,
DefaultScatterTemplate,
SmoothLinearTemplate,
]
@cached_property
@@ -253,6 +222,9 @@ class PlotTemplates:
return default_plot_path
def get_template(self, path):
if os.path.exists(path):
return path
t_path = os.path.join(self.templates_dir, path)
if os.path.exists(t_path):
return t_path
@@ -279,4 +251,23 @@ class PlotTemplates:
if not os.path.exists(self.templates_dir):
makedirs(self.templates_dir, exist_ok=True)
for t in self.TEMPLATES:
t(self.templates_dir).dump()
self.dump(t())
def dump(self, template):
path = os.path.join(self.templates_dir, template.filename)
with open(path, "w") as fd:
json.dump(
template.content,
fd,
indent=template.INDENT,
separators=template.SEPARATORS,
)
fd.write("\n")
def load(self, name):
path = self.get_template(name)
with open(path) as fd:
content = fd.read()
return Template(content, name=name)
+1 -1
View File
@@ -32,4 +32,4 @@ def push(
used_run_cache=used_run_cache,
)
return self.cloud.push(used, jobs, remote=remote)
return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
+1
View File
@@ -63,6 +63,7 @@ def reproduce(
):
from dvc.utils import parse_target
assert target is None or isinstance(target, str)
if not target and not all_pipelines:
raise InvalidArgumentError(
"Neither `target` nor `--all-pipelines` are specified."
+41 -11
View File
@@ -1,8 +1,10 @@
import os
from contextlib import suppress
from funcy import concat, first
from funcy import concat, first, lfilter
from dvc.exceptions import InvalidArgumentError
from dvc.stage import PipelineStage
from dvc.stage.exceptions import (
DuplicateStageName,
InvalidStageName,
@@ -20,6 +22,19 @@ def is_valid_name(name: str):
return not INVALID_STAGENAME_CHARS & set(name)
def parse_params(path_params):
ret = []
for path_param in path_params:
path, _, params_str = path_param.rpartition(":")
# remove empty strings from params, on condition such as `-p "file1:"`
params = lfilter(bool, params_str.split(","))
if not path:
ret.extend(params)
else:
ret.append({path: params})
return ret
def _get_file_path(kwargs):
from dvc.dvcfile import DVC_FILE_SUFFIX, DVC_FILE
@@ -43,10 +58,25 @@ def _get_file_path(kwargs):
)
def _check_stage_exists(dvcfile, stage):
if not dvcfile.exists():
return
hint = "Use '--force' to overwrite."
if stage.__class__ != PipelineStage:
raise StageFileAlreadyExistsError(
f"'{stage.relpath}' already exists. {hint}"
)
elif stage.name and stage.name in dvcfile.stages:
raise DuplicateStageName(
f"Stage '{stage.name}' already exists in '{stage.relpath}'. {hint}"
)
@locked
@scm_context
def run(self, fname=None, no_exec=False, single_stage=False, **kwargs):
from dvc.stage import PipelineStage, Stage, create_stage
from dvc.stage import Stage, create_stage
from dvc.dvcfile import Dvcfile, PIPELINE_FILE
if not kwargs.get("cmd"):
@@ -72,20 +102,20 @@ def run(self, fname=None, no_exec=False, single_stage=False, **kwargs):
if not is_valid_name(stage_name):
raise InvalidStageName
stage = create_stage(stage_cls, repo=self, path=path, **kwargs)
params = parse_params(kwargs.pop("params", []))
stage = create_stage(
stage_cls, repo=self, path=path, params=params, **kwargs
)
if stage is None:
return None
dvcfile = Dvcfile(self, stage.path)
if dvcfile.exists():
if kwargs.get("overwrite", True):
dvcfile.remove_stage(stage)
elif stage_cls != PipelineStage:
raise StageFileAlreadyExistsError(dvcfile.relpath)
elif stage_name and stage_name in dvcfile.stages:
raise DuplicateStageName(stage_name, dvcfile)
try:
if kwargs.get("force", True):
with suppress(ValueError):
self.stages.remove(stage)
else:
_check_stage_exists(dvcfile, stage)
self.check_modified_graph([stage])
except OutputDuplicationError as exc:
raise OutputDuplicationError(exc.output, set(exc.stages) - {stage})
+17 -19
View File
@@ -48,11 +48,11 @@ class DvcTree(BaseTree):
raise FileNotFoundError
dir_cache = out.get_dir_cache(remote=remote)
for entry in dir_cache:
entry_relpath = entry[out.remote.PARAM_RELPATH]
entry_relpath = entry[out.remote.tree.PARAM_RELPATH]
if os.name == "nt":
entry_relpath = entry_relpath.replace("/", os.sep)
if path == out.path_info / entry_relpath:
return entry[out.remote.PARAM_CHECKSUM]
return entry[out.remote.tree.PARAM_CHECKSUM]
raise FileNotFoundError
def open(self, path, mode="r", encoding="utf-8", remote=None):
@@ -61,6 +61,10 @@ class DvcTree(BaseTree):
except OutputNotFoundError as exc:
raise FileNotFoundError from exc
# NOTE: this handles both dirty and checkout-ed out at the same time
if self.repo.tree.exists(path):
return self.repo.tree.open(path, mode=mode, encoding=encoding)
if len(outs) != 1 or (
outs[0].is_dir_checksum and path == outs[0].path_info
):
@@ -78,7 +82,7 @@ class DvcTree(BaseTree):
else:
checksum = out.checksum
try:
remote_info = remote_obj.checksum_to_path_info(checksum)
remote_info = remote_obj.hash_to_path_info(checksum)
return remote_obj.open(
remote_info, mode=mode, encoding=encoding
)
@@ -89,7 +93,7 @@ class DvcTree(BaseTree):
if out.is_dir_checksum:
checksum = self._get_granular_checksum(path, out)
cache_path = out.cache.checksum_to_path_info(checksum).url
cache_path = out.cache.hash_to_path_info(checksum).url
else:
cache_path = out.cache_path
return open(cache_path, mode=mode, encoding=encoding)
@@ -190,7 +194,7 @@ class DvcTree(BaseTree):
download_callback(downloaded)
for entry in dir_cache:
entry_relpath = entry[out.remote.PARAM_RELPATH]
entry_relpath = entry[out.remote.tree.PARAM_RELPATH]
if os.name == "nt":
entry_relpath = entry_relpath.replace("/", os.sep)
path_info = out.path_info / entry_relpath
@@ -208,7 +212,7 @@ class DvcTree(BaseTree):
def isexec(self, path):
return False
def get_file_checksum(self, path_info):
def get_file_hash(self, path_info):
outs = self._find_outs(path_info, strict=False)
if len(outs) != 1:
raise OutputNotFoundError
@@ -252,13 +256,9 @@ class RepoTree(BaseTree):
encoding = None
if self.dvctree and self.dvctree.exists(path):
try:
return self.dvctree.open(
path, mode=mode, encoding=encoding, **kwargs
)
except FileNotFoundError:
if self.isdvc(path):
raise
return self.dvctree.open(
path, mode=mode, encoding=encoding, **kwargs
)
return self.repo.tree.open(path, mode=mode, encoding=encoding)
def exists(self, path):
@@ -373,7 +373,7 @@ class RepoTree(BaseTree):
for fname in files:
yield PathInfo(root) / fname
def get_file_checksum(self, path_info):
def get_file_hash(self, path_info):
"""Return file checksum for specified path.
If path_info is a DVC out, the pre-computed checksum for the file
@@ -384,7 +384,7 @@ class RepoTree(BaseTree):
raise FileNotFoundError
if self.dvctree and self.dvctree.exists(path_info):
try:
return self.dvctree.get_file_checksum(path_info)
return self.dvctree.get_file_hash(path_info)
except OutputNotFoundError:
pass
return file_md5(path_info, self)[0]
@@ -404,10 +404,8 @@ class RepoTree(BaseTree):
for root, _, files in self.walk(top):
root = PathInfo(root)
dest_dir = root.relative_to(top)
makedirs(dest_dir, exist_ok=True)
makedirs(dest, exist_ok=True)
for fname in files:
src = root / fname
dest = dest_dir / fname
with self.open(src, mode="rb") as fobj:
copy_fobj_to_file(fobj, dest)
copy_fobj_to_file(fobj, dest / fname)
+3 -3
View File
@@ -34,10 +34,10 @@ PLOT_PROPS = {
BaseOutput.PARAM_PLOT_TEMPLATE: str,
BaseOutput.PARAM_PLOT_X: str,
BaseOutput.PARAM_PLOT_Y: str,
BaseOutput.PARAM_PLOT_XLAB: str,
BaseOutput.PARAM_PLOT_YLAB: str,
BaseOutput.PARAM_PLOT_X_LABEL: str,
BaseOutput.PARAM_PLOT_Y_LABEL: str,
BaseOutput.PARAM_PLOT_TITLE: str,
BaseOutput.PARAM_PLOT_CSV_HEADER: bool,
BaseOutput.PARAM_PLOT_HEADER: bool,
}
PLOT_PROPS_SCHEMA = {
**OUT_PSTAGE_DETAILED_SCHEMA[str],
+9 -2
View File
@@ -19,6 +19,7 @@ from .utils import (
check_circular_dependency,
check_duplicated_arguments,
check_missing_outputs,
check_no_externals,
check_stage_path,
compute_md5,
fill_stage_dependencies,
@@ -52,7 +53,7 @@ def loads_from(cls, repo, path, wdir, data):
return cls(**kw)
def create_stage(cls, repo, path, **kwargs):
def create_stage(cls, repo, path, external=False, **kwargs):
from dvc.dvcfile import check_dvc_filename
wdir = os.path.abspath(kwargs.get("wdir", None) or os.curdir)
@@ -63,6 +64,8 @@ def create_stage(cls, repo, path, **kwargs):
stage = loads_from(cls, repo, path, wdir, kwargs)
fill_stage_outputs(stage, **kwargs)
if not external:
check_no_externals(stage)
fill_stage_dependencies(
stage, **project(kwargs, ["deps", "erepo", "params"])
)
@@ -490,7 +493,11 @@ class Stage(params.StageParams):
ret.append("changed checksum")
def already_cached(self):
return self.deps_cached() and self.outs_cached()
return (
not self.changed_stage()
and self.deps_cached()
and self.outs_cached()
)
def deps_cached(self):
return all(not dep.changed() for dep in self.deps)
+16 -20
View File
@@ -6,13 +6,15 @@ import yaml
from funcy import first
from voluptuous import Invalid
from dvc.remote.local import _log_exceptions
from dvc.schema import COMPILED_LOCK_FILE_STAGE_SCHEMA
from dvc.serialize import to_single_stage_lockfile
from dvc.stage.loader import StageLoader
from dvc.utils import dict_sha256, relpath
from dvc.utils.fs import makedirs
from dvc.utils.yaml import dump_yaml
from .loader import StageLoader
from .serialize import to_single_stage_lockfile
logger = logging.getLogger(__name__)
@@ -78,20 +80,7 @@ class StageCache:
return None
def _create_stage(self, cache, wdir=None):
from dvc.stage import create_stage, PipelineStage
params = []
for param in cache.get("params", []):
if isinstance(param, str):
params.append(param)
continue
assert isinstance(param, dict)
assert len(param) == 1
path = list(param.keys())[0]
params_list = param[path]
assert isinstance(params_list, list)
params.append(f"{path}:" + ",".join(params_list))
from . import create_stage, PipelineStage
stage = create_stage(
PipelineStage,
@@ -99,9 +88,8 @@ class StageCache:
path="dvc.yaml",
cmd=cache["cmd"],
wdir=wdir,
params=params,
deps=[dep["path"] for dep in cache.get("deps", [])],
outs=[out["path"] for out in cache["outs"]],
external=True,
)
StageLoader.fill_from_lock(stage, cache)
return stage
@@ -190,11 +178,19 @@ class StageCache:
def push(self, remote):
remote = self.repo.cloud.get_remote(remote)
return self._transfer(remote.upload, self.repo.cache.local, remote)
return self._transfer(
_log_exceptions(remote.tree.upload, "upload"),
self.repo.cache.local.tree,
remote.tree,
)
def pull(self, remote):
remote = self.repo.cloud.get_remote(remote)
return self._transfer(remote.download, remote, self.repo.cache.local)
return self._transfer(
_log_exceptions(remote.tree.download, "download"),
remote.tree,
self.repo.cache.local.tree,
)
def get_used_cache(self, used_run_cache, *args, **kwargs):
from dvc.cache import NamedCache
+6 -9
View File
@@ -20,9 +20,7 @@ class StageFileDoesNotExistError(DvcException):
class StageFileAlreadyExistsError(DvcException):
def __init__(self, relpath):
msg = f"not overwriting '{relpath}'"
super().__init__(msg)
pass
class StageFileIsNotDvcFileError(DvcException):
@@ -58,6 +56,10 @@ class StageCommitError(DvcException):
pass
class StageExternalOutputsError(DvcException):
pass
class StageUpdateError(DvcException):
def __init__(self, path):
super().__init__(
@@ -96,12 +98,7 @@ class StageNameUnspecified(DvcException):
class DuplicateStageName(DvcException):
def __init__(self, name, file):
super().__init__(
"Stage '{name}' already exists in '{relpath}'.".format(
name=name, relpath=file.relpath
)
)
pass
class InvalidStageName(DvcException):
+2 -1
View File
@@ -23,7 +23,8 @@ def sync_import(stage, dry=False, force=False):
if dry:
return
if not force and not stage.changed_stage() and stage.already_cached():
if not force and stage.already_cached():
stage.outs[0].checkout()
else:
stage.save_deps()
stage.deps[0].download(stage.outs[0])
+29 -137
View File
@@ -1,28 +1,19 @@
import logging
import os
from collections import defaultdict
from collections.abc import Mapping
from copy import deepcopy
from itertools import chain
from funcy import first
from funcy import get_in, lcat, project
from dvc import dependency, output
from ..dependency import ParamsDependency
from . import PipelineStage, Stage, loads_from
from .exceptions import StageNameUnspecified, StageNotFound
from .params import StageParams
from .utils import fill_stage_dependencies, resolve_paths
logger = logging.getLogger(__name__)
DEFAULT_PARAMS_FILE = ParamsDependency.DEFAULT_PARAMS_FILE
def resolve_paths(path, wdir=None):
path = os.path.abspath(path)
wdir = wdir or os.curdir
wdir = os.path.abspath(os.path.join(os.path.dirname(path), wdir))
return path, wdir
class StageLoader(Mapping):
def __init__(self, dvcfile, stages_data, lockfile_data=None):
@@ -31,9 +22,12 @@ class StageLoader(Mapping):
self.lockfile_data = lockfile_data or {}
@staticmethod
def fill_from_lock(stage, lock_data):
from .params import StageParams
def fill_from_lock(stage, lock_data=None):
"""Fill values for params, checksums for outs and deps from lock."""
if not lock_data:
return
assert isinstance(lock_data, dict)
items = chain(
((StageParams.PARAM_DEPS, dep) for dep in stage.deps),
((StageParams.PARAM_OUTS, out) for out in stage.outs),
@@ -44,139 +38,39 @@ class StageLoader(Mapping):
for key in [StageParams.PARAM_DEPS, StageParams.PARAM_OUTS]
}
for key, item in items:
if isinstance(item, ParamsDependency):
# load the params with values inside lock dynamically
params = lock_data.get("params", {}).get(item.def_path, {})
item._dyn_load(params)
path = item.def_path
if isinstance(item, dependency.ParamsDependency):
item.fill_values(get_in(lock_data, [stage.PARAM_PARAMS, path]))
continue
item.checksum = (
checksums.get(key, {})
.get(item.def_path, {})
.get(item.checksum_type)
)
item.checksum = get_in(checksums, [key, path, item.checksum_type])
@classmethod
def _load_params(cls, stage, pipeline_params):
"""
File in pipeline file is expected to be in following format:
```
params:
- lr
- train.epochs
- params2.yaml: # notice the filename
- process.threshold
- process.bow
```
and, in lockfile, we keep it as following format:
```
params:
params.yaml:
lr: 0.0041
train.epochs: 100
params2.yaml:
process.threshold: 0.98
process.bow:
- 15000
- 123
```
In the list of `params` inside pipeline file, if any of the item is
dict-like, the key will be treated as separate params file and it's
values to be part of that params file, else, the item is considered
as part of the `params.yaml` which is a default file.
(From example above: `lr` is considered to be part of `params.yaml`
whereas `process.bow` to be part of `params2.yaml`.)
We only load the keys here, lockfile bears the values which are used
to compare between the actual params from the file in the workspace.
"""
res = defaultdict(list)
for key in pipeline_params:
if isinstance(key, str):
path = DEFAULT_PARAMS_FILE
res[path].append(key)
elif isinstance(key, dict):
path = first(key)
res[path].extend(key[path])
stage.deps.extend(
dependency.loadd_from(
stage,
[
{"path": key, "params": params}
for key, params in res.items()
],
)
)
@classmethod
def _load_outs(cls, stage, data, typ=None):
from dvc.output.base import BaseOutput
d = []
for key in data:
if isinstance(key, str):
entry = {BaseOutput.PARAM_PATH: key}
if typ:
entry[typ] = True
d.append(entry)
continue
assert isinstance(key, dict)
assert len(key) == 1
path = first(key)
extra = key[path]
if not typ:
d.append({BaseOutput.PARAM_PATH: path, **extra})
continue
entry = {BaseOutput.PARAM_PATH: path}
persist = extra.pop(BaseOutput.PARAM_PERSIST, False)
if persist:
entry[BaseOutput.PARAM_PERSIST] = persist
cache = extra.pop(BaseOutput.PARAM_CACHE, True)
if not cache:
entry[BaseOutput.PARAM_CACHE] = cache
entry[typ] = extra or True
d.append(entry)
stage.outs.extend(output.loadd_from(stage, d))
@classmethod
def _load_deps(cls, stage, data):
stage.deps.extend(dependency.loads_from(stage, data))
@classmethod
def load_stage(cls, dvcfile, name, stage_data, lock_data):
from . import PipelineStage, Stage, loads_from
def load_stage(cls, dvcfile, name, stage_data, lock_data=None):
assert all([name, dvcfile, dvcfile.repo, dvcfile.path])
assert stage_data and isinstance(stage_data, dict)
path, wdir = resolve_paths(
dvcfile.path, stage_data.get(Stage.PARAM_WDIR)
)
stage = loads_from(PipelineStage, dvcfile.repo, path, wdir, stage_data)
stage.name = name
stage.deps, stage.outs = [], []
cls._load_outs(stage, stage_data.get("outs", []))
cls._load_outs(stage, stage_data.get("metrics", []), "metric")
cls._load_outs(stage, stage_data.get("plots", []), "plot")
cls._load_deps(stage, stage_data.get("deps", []))
cls._load_params(stage, stage_data.get("params", []))
deps = project(stage_data, [stage.PARAM_DEPS, stage.PARAM_PARAMS])
fill_stage_dependencies(stage, **deps)
outs = project(
stage_data,
[stage.PARAM_OUTS, stage.PARAM_METRICS, stage.PARAM_PLOTS],
)
stage.outs = lcat(
output.load_from_pipeline(stage, data, typ=key)
for key, data in outs.items()
)
if lock_data:
stage.cmd_changed = lock_data.get(
Stage.PARAM_CMD
) != stage_data.get(Stage.PARAM_CMD)
cls.fill_from_lock(stage, lock_data)
stage.cmd_changed = lock_data.get(Stage.PARAM_CMD) != stage.cmd
cls.fill_from_lock(stage, lock_data)
return stage
def __getitem__(self, name):
@@ -229,8 +123,6 @@ class SingleStageLoader(Mapping):
@classmethod
def load_stage(cls, dvcfile, d, stage_text):
from dvc.stage import Stage, loads_from
path, wdir = resolve_paths(dvcfile.path, d.get(Stage.PARAM_WDIR))
stage = loads_from(Stage, dvcfile.repo, path, wdir, d)
stage._stage_text = stage_text
-1
View File
@@ -82,7 +82,6 @@ def _is_cached(stage):
cached = (
not stage.is_callback
and not stage.always_changed
and not stage.changed_stage()
and stage.already_cached()
)
if cached:
+73 -64
View File
@@ -3,15 +3,16 @@ from functools import partial
from operator import attrgetter
from typing import TYPE_CHECKING, List
from funcy import lsplit, rpartial
from funcy import post_processing
from dvc.dependency import ParamsDependency
from dvc.output import BaseOutput
from dvc.stage.params import StageParams
from dvc.stage.utils import resolve_wdir
from dvc.utils.collections import apply_diff
from dvc.utils.yaml import parse_yaml_for_update
from .params import StageParams
from .utils import resolve_wdir, split_params_deps
if TYPE_CHECKING:
from dvc.stage import PipelineStage, Stage
@@ -32,89 +33,96 @@ DEFAULT_PARAMS_FILE = ParamsDependency.DEFAULT_PARAMS_FILE
sort_by_path = partial(sorted, key=attrgetter("def_path"))
def _get_out(out):
res = OrderedDict()
@post_processing(OrderedDict)
def _get_flags(out):
if not out.use_cache:
res[PARAM_CACHE] = False
yield PARAM_CACHE, False
if out.persist:
res[PARAM_PERSIST] = True
yield PARAM_PERSIST, True
if out.plot and isinstance(out.plot, dict):
res.update(out.plot)
return out.def_path if not res else {out.def_path: res}
# notice `out.plot` is not sorted
# `out.plot` is in the same order as is in the file when read
# and, should be dumped as-is without any sorting
yield from out.plot.items()
def _get_outs(outs):
return [_get_out(out) for out in sort_by_path(outs)]
def _serialize_out(out):
flags = _get_flags(out)
return out.def_path if not flags else {out.def_path: flags}
def get_params_deps(stage: "PipelineStage"):
return lsplit(rpartial(isinstance, ParamsDependency), stage.deps)
def _serialize_outs(outputs: List[BaseOutput]):
outs, metrics, plots = [], [], []
for out in sort_by_path(outputs):
bucket = outs
if out.plot:
bucket = plots
elif out.metric:
bucket = metrics
bucket.append(_serialize_out(out))
return outs, metrics, plots
def _serialize_params(params: List[ParamsDependency]):
"""Return two types of values from stage:
def _serialize_params_keys(params):
"""
Returns the following format of data:
['lr', 'train', {'params2.yaml': ['lr']}]
`keys` - which is list of params without values, used in a pipeline file
which is in the shape of:
['lr', 'train', {'params2.yaml': ['lr']}]
`key_vals` - which is list of params with values, used in a lockfile
which is in the shape of:
{'params.yaml': {'lr': '1', 'train': 2}, {'params2.yaml': {'lr': '1'}}
The output is sorted, with keys of params from default params file being
at the first, and then followed by entry of other files in lexicographic
order. The keys of those custom files are also sorted in the same order.
"""
keys = []
key_vals = OrderedDict()
for param_dep in sort_by_path(params):
dump = param_dep.dumpd()
path, params = dump[PARAM_PATH], dump[PARAM_PARAMS]
assert isinstance(params, (dict, list))
# when on no_exec, params are not filled and are saved as list
k = sorted(params.keys() if isinstance(params, dict) else params)
if not k:
continue
if path == DEFAULT_PARAMS_FILE:
keys = k + keys
else:
keys.append({path: k})
return keys
def _serialize_params_values(params: List[ParamsDependency]):
"""Returns output of following format, used for lockfile:
{'params.yaml': {'lr': '1', 'train': 2}, {'params2.yaml': {'lr': '1'}}
Default params file are always kept at the start, followed by others in
alphabetical order. The param values are sorted too(not recursively though)
"""
key_vals = OrderedDict()
for param_dep in sort_by_path(params):
dump = param_dep.dumpd()
path, params = dump[PARAM_PATH], dump[PARAM_PARAMS]
if isinstance(params, dict):
k = list(params.keys())
if not k:
continue
key_vals[path] = OrderedDict(
[(key, params[key]) for key in sorted(k)]
)
else:
assert isinstance(params, list)
k = params
key_vals = OrderedDict()
# params from default file is always kept at the start of the `params:`
if path == DEFAULT_PARAMS_FILE:
keys = k + keys
if key_vals:
kv = [(key, params[key]) for key in sorted(params.keys())]
key_vals[path] = OrderedDict(kv)
if path == DEFAULT_PARAMS_FILE:
key_vals.move_to_end(path, last=False)
else:
# if it's not a default file, change the shape
# to: {path: k}
keys.append({path: k})
return keys, key_vals
return key_vals
def to_pipeline_file(stage: "PipelineStage"):
params, deps = get_params_deps(stage)
serialized_params, _ = _serialize_params(params)
wdir = resolve_wdir(stage.wdir, stage.path)
params, deps = split_params_deps(stage)
deps = sorted([d.def_path for d in deps])
params = _serialize_params_keys(params)
outs, metrics, plots = _serialize_outs(stage.outs)
res = [
(stage.PARAM_CMD, stage.cmd),
(stage.PARAM_WDIR, resolve_wdir(stage.wdir, stage.path)),
(stage.PARAM_DEPS, sorted([d.def_path for d in deps])),
(stage.PARAM_PARAMS, serialized_params),
(
PARAM_OUTS,
_get_outs(
[out for out in stage.outs if not (out.metric or out.plot)]
),
),
(
stage.PARAM_METRICS,
_get_outs([out for out in stage.outs if out.metric]),
),
(
stage.PARAM_PLOTS,
_get_outs([out for out in stage.outs if out.plot]),
),
(stage.PARAM_WDIR, wdir),
(stage.PARAM_DEPS, deps),
(stage.PARAM_PARAMS, params),
(stage.PARAM_OUTS, outs),
(stage.PARAM_METRICS, metrics),
(stage.PARAM_PLOTS, plots),
(stage.PARAM_FROZEN, stage.frozen),
(stage.PARAM_ALWAYS_CHANGED, stage.always_changed),
]
@@ -127,7 +135,7 @@ def to_single_stage_lockfile(stage: "Stage") -> dict:
assert stage.cmd
res = OrderedDict([("cmd", stage.cmd)])
params, deps = get_params_deps(stage)
params, deps = split_params_deps(stage)
deps, outs = [
[
OrderedDict(
@@ -140,10 +148,11 @@ def to_single_stage_lockfile(stage: "Stage") -> dict:
]
for items in [deps, stage.outs]
]
params = _serialize_params_values(params)
if deps:
res[PARAM_DEPS] = deps
if params:
_, res[PARAM_PARAMS] = _serialize_params(params)
res[PARAM_PARAMS] = params
if outs:
res[PARAM_OUTS] = outs
+45 -4
View File
@@ -2,13 +2,18 @@ import os
import pathlib
from itertools import product
from funcy import lsplit, rpartial
from dvc import dependency, output
from dvc.utils.fs import path_isin
from ..remote import LocalRemote, S3Remote
from ..utils import dict_md5, relpath
from ..dependency import ParamsDependency
from ..remote.local import LocalRemoteTree
from ..remote.s3 import S3RemoteTree
from ..utils import dict_md5, format_link, relpath
from .exceptions import (
MissingDataSource,
StageExternalOutputsError,
StagePathNotDirectoryError,
StagePathNotFoundError,
StagePathOutsideError,
@@ -68,6 +73,31 @@ def fill_stage_dependencies(stage, deps=None, erepo=None, params=None):
stage.deps += dependency.loads_params(stage, params or [])
def check_no_externals(stage):
from urllib.parse import urlparse
# NOTE: preventing users from accidentally using external outputs. See
# https://github.com/iterative/dvc/issues/1545 for more details.
def _is_external(out):
# NOTE: in case of `remote://` notation, the user clearly knows that
# this is an advanced feature and so we shouldn't error-out.
if out.is_in_repo or urlparse(out.def_path).scheme == "remote":
return False
return True
outs = [str(out) for out in stage.outs if _is_external(out)]
if not outs:
return
str_outs = ", ".join(outs)
link = format_link("https://dvc.org/doc/user-guide/managing-external-data")
raise StageExternalOutputsError(
f"Output(s) outside of DVC project: {str_outs}. "
f"See {link} for more info."
)
def check_circular_dependency(stage):
from dvc.exceptions import CircularDependencyError
@@ -103,8 +133,8 @@ def stage_dump_eq(stage_cls, old_d, new_d):
new_d.pop(stage_cls.PARAM_MD5, None)
outs = old_d.get(stage_cls.PARAM_OUTS, [])
for out in outs:
out.pop(LocalRemote.PARAM_CHECKSUM, None)
out.pop(S3Remote.PARAM_CHECKSUM, None)
out.pop(LocalRemoteTree.PARAM_CHECKSUM, None)
out.pop(S3RemoteTree.PARAM_CHECKSUM, None)
# outs and deps are lists of dicts. To check equality, we need to make
# them independent of the order, so, we convert them to dicts.
@@ -149,6 +179,13 @@ def resolve_wdir(wdir, path):
return pathlib.PurePath(rel_wdir).as_posix() if rel_wdir != "." else None
def resolve_paths(path, wdir=None):
path = os.path.abspath(path)
wdir = wdir or os.curdir
wdir = os.path.abspath(os.path.join(os.path.dirname(path), wdir))
return path, wdir
def get_dump(stage):
return {
key: value
@@ -163,3 +200,7 @@ def get_dump(stage):
}.items()
if value
}
def split_params_deps(stage):
return lsplit(rpartial(isinstance, ParamsDependency), stage.deps)
+8 -5
View File
@@ -7,7 +7,6 @@ import sqlite3
from urllib.parse import urlencode, urlunparse
from dvc.exceptions import DvcException
from dvc.scm.tree import WorkingTree
from dvc.utils import current_timestamp, relpath, to_chunks
from dvc.utils.fs import get_inode, get_mtime_and_size, remove
@@ -89,10 +88,11 @@ class State: # pylint: disable=too-many-instance-attributes
MAX_INT = 2 ** 63 - 1
MAX_UINT = 2 ** 64 - 2
def __init__(self, repo):
def __init__(self, local_cache):
repo = local_cache.repo
self.repo = repo
self.root_dir = repo.root_dir
self.tree = WorkingTree(self.root_dir)
self.tree = local_cache.tree.work_tree
state_config = repo.config.get("state", {})
self.row_limit = state_config.get("row_limit", self.STATE_ROW_LIMIT)
@@ -394,6 +394,9 @@ class State: # pylint: disable=too-many-instance-attributes
assert isinstance(path_info, str) or path_info.scheme == "local"
path = os.fspath(path_info)
# NOTE: use os.path.exists instead of WorkingTree.exists
# WorkingTree.exists uses lexists() and will return True for broken
# symlinks that we cannot stat() in get_mtime_and_size
if not os.path.exists(path):
return None
@@ -420,7 +423,7 @@ class State: # pylint: disable=too-many-instance-attributes
"""
assert isinstance(path_info, str) or path_info.scheme == "local"
if not os.path.exists(path_info):
if not self.tree.exists(path_info):
return
mtime, _ = get_mtime_and_size(path_info, self.tree)
@@ -446,7 +449,7 @@ class State: # pylint: disable=too-many-instance-attributes
inode = self._from_sqlite(inode)
path = os.path.join(self.root_dir, relpath)
if path in used or not os.path.exists(path):
if path in used or not self.tree.exists(path):
continue
actual_inode = get_inode(path)
+2 -44
View File
@@ -1,6 +1,8 @@
import io
from contextlib import contextmanager
from dvc.utils.stream import IterStream
@contextmanager
def open_url(url, mode="r", encoding=None):
@@ -61,47 +63,3 @@ def iter_url(url, chunk_size=io.DEFAULT_BUFFER_SIZE):
finally:
# Ensure connection is closed
it.close()
class IterStream(io.RawIOBase):
"""Wraps an iterator yielding bytes as a file object"""
def __init__(self, iterator):
self.iterator = iterator
self.leftover = None
def readable(self):
return True
# Python 3 requires only .readinto() method, it still uses other ones
# under some circumstances and falls back if those are absent. Since
# iterator already constructs byte strings for us, .readinto() is not the
# most optimal, so we provide .read1() too.
def readinto(self, b):
try:
n = len(b) # We're supposed to return at most this much
chunk = self.leftover or next(self.iterator)
output, self.leftover = chunk[:n], chunk[n:]
n_out = len(output)
b[:n_out] = output
return n_out
except StopIteration:
return 0 # indicate EOF
readinto1 = readinto
def read1(self, n=-1):
try:
chunk = self.leftover or next(self.iterator)
except StopIteration:
return b""
# Return an arbitrary number or bytes
if n <= 0:
self.leftover = None
return chunk
output, self.leftover = chunk[:n], chunk[n:]
return output
+45
View File
@@ -0,0 +1,45 @@
import io
class IterStream(io.RawIOBase):
"""Wraps an iterator yielding bytes as a file object"""
def __init__(self, iterator):
self.iterator = iterator
self.leftover = None
def readable(self):
return True
# Python 3 requires only .readinto() method, it still uses other ones
# under some circumstances and falls back if those are absent. Since
# iterator already constructs byte strings for us, .readinto() is not the
# most optimal, so we provide .read1() too.
def readinto(self, b):
try:
n = len(b) # We're supposed to return at most this much
chunk = self.leftover or next(self.iterator)
output, self.leftover = chunk[:n], chunk[n:]
n_out = len(output)
b[:n_out] = output
return n_out
except StopIteration:
return 0 # indicate EOF
readinto1 = readinto
def read1(self, n=-1):
try:
chunk = self.leftover or next(self.iterator)
except StopIteration:
return b""
# Return an arbitrary number or bytes
if n <= 0:
self.leftover = None
return chunk
output, self.leftover = chunk[:n], chunk[n:]
return output
+1 -1
View File
@@ -6,7 +6,7 @@
import os
import subprocess
_BASE_VERSION = "1.0.0a7"
_BASE_VERSION = "1.0.0b4"
def _generate_version(base_version):
+1 -1
View File
@@ -7,7 +7,7 @@ if [ ! -d "dvc" ]; then
exit 1
fi
sudo snapcraft --use-lxd
sg lxd -c snapcraft
pip uninstall -y dvc
if which dvc; then
+21 -6
View File
@@ -51,12 +51,27 @@ elif [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
ln -s -f /usr/local/bin/pip3 /usr/local/bin/pip
fi
if [[ -n "$TRAVIS_TAG" ]]; then
if [[ $(echo "$TRAVIS_TAG" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$') ]]; then
echo "export SNAP_CHANNEL=stable" >>env.sh
if [ "$TRAVIS_OS_NAME" == "linux" ]; then
# fetch tags for `git-describe`, since
# - can't rely on $TRAVIS_TAG for snapcraft `edge` (master) releases, and
# - `snapcraft` also uses `git-describe` for version detection
git fetch --tags
TAG_MAJOR="$(git describe --tags | sed -r 's/^v?([0-9]+)\.[0-9]+\.[0-9]+.*/\1/')"
[[ -n "$TAG_MAJOR" ]] || exit 1 # failed to detect major version
if [[ -n "$TRAVIS_TAG" ]]; then
if [[ $(echo "$TRAVIS_TAG" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$') ]]; then
echo "export SNAP_CHANNEL=stable" >>env.sh
echo "export SNAP_CHANNEL_MAJOR=v$TAG_MAJOR/stable" >>env.sh
else
echo "export SNAP_CHANNEL=beta" >>env.sh
echo "export SNAP_CHANNEL_MAJOR=v$TAG_MAJOR/beta" >>env.sh
fi
else
echo "export SNAP_CHANNEL=beta" >>env.sh
echo "export SNAP_CHANNEL=edge" >>env.sh
echo "export SNAP_CHANNEL_MAJOR=v$TAG_MAJOR/edge" >>env.sh
fi
else
echo "export SNAP_CHANNEL=edge" >>env.sh
# NOTE: after deprecating this major version branch, uncomment this line
# echo "unset SNAP_CHANNEL" >>env.sh
fi
+5 -4
View File
@@ -12,7 +12,7 @@ _dvc_commands='add cache checkout commit config dag destroy diff fetch freeze \
_dvc_options='-h --help -V --version'
_dvc_global_options='-h --help -q --quiet -v --verbose'
_dvc_add='-R --recursive -f --file --no-commit'
_dvc_add='-R --recursive -f --file --no-commit --external'
_dvc_add_COMPGEN=_dvc_compgen_files
_dvc_cache='dir'
_dvc_cache_dir=' --global --system --local -u --unset'
@@ -47,8 +47,9 @@ _dvc_move_COMPGEN=_dvc_compgen_files
_dvc_params='diff'
_dvc_params_diff='--all --show-json --show-md --no-path'
_dvc_plots='show diff'
_dvc_plots_show='-t --template -o --out -x -y --show-json --no-csv-header --title --xlab --ylab'
_dvc_plots_diff='-t --template --targets -o --out -x -y --show-json --no-csv-header --title --xlab --ylab'
_dvc_plots_show='-t --template -o --out -x -y --show-vega --no-csv-header --title --x-label --y-label'
_dvc_plots_diff='-t --template --targets -o --out -x -y --show-vega --no-csv-header --title --x-label --y-label'
_dvc_plots_modify='-t --template -x -y --no-csv-header --title --x-label --y-label'
_dvc_pull='-j --jobs -r --remote -a --all-branches -T --all-tags -f --force -d --with-deps -R --recursive'
_dvc_pull_COMPGEN=_dvc_compgen_DVCFiles
_dvc_push='-j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -R --recursive'
@@ -65,7 +66,7 @@ _dvc_remove_COMPGEN=_dvc_compgen_DVCFiles
_dvc_repro='-f --force -s --single-item -c --cwd -m --metrics --dry -i --interactive -p --pipeline -P --all-pipelines --no-run-cache --force-downstream --no-commit -R --recursive --downstream'
_dvc_repro_COMPGEN=_dvc_compgen_DVCFiles
_dvc_root=''
_dvc_run='--no-exec -f --file -d --deps -o --outs -O --outs-no-cache --outs-persist --outs-persist-no-cache -m --metrics -M --metrics-no-cache --overwrite-dvcfile --no-run-cache --no-commit -w --wdir'
_dvc_run='--no-exec -f --file -d --deps -o --outs -O --outs-no-cache --outs-persist --outs-persist-no-cache -m --metrics -M --metrics-no-cache --overwrite-dvcfile --no-run-cache --no-commit -w --wdir --external'
_dvc_run_COMPGEN=_dvc_compgen_DVCFiles
_dvc_status='-j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -c --cloud'
_dvc_status_COMPGEN=_dvc_compgen_DVCFiles
+3 -1
View File
@@ -66,6 +66,7 @@ _dvc_add=(
{-R,--recursive}"[Recursively add each file under the directory.]"
"--no-commit[Don't put files/directories into cache.]"
{-f,--file}"[Specify name of the DVC-file it generates.]:File:_files"
"--external[Allow targets that are outside of the DVC project.]"
"1:File:_files"
)
@@ -216,7 +217,7 @@ _dvc_push=(
)
_dvc_plots=(
"1:Sub command:(show diff)"
"1:Sub command:(show diff modify)"
)
_dvc_remote=(
@@ -267,6 +268,7 @@ _dvc_run=(
"--no-commit[Don't put files/directories into cache.]"
"--outs-persist[Declare output file or directory that will not be removed upon repro.]:Output persistent:_files"
"--outs-persist-no-cache[Declare output file or directory that will not be removed upon repro (do not put into DVC cache).]:Output persistent regular:_files"
"--external[Allow outputs that are outside of the DVC project.]"
)
_dvc_status=(
+4
View File
@@ -0,0 +1,4 @@
from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata("pydrive2")
datas += copy_metadata("google-api-python-client")
+227
View File
@@ -0,0 +1,227 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$comment": "v0.1.0",
"title": "Pipeline Schema",
"description": "A dvc.yaml pipeline definition",
"type": "object",
"definitions": {
"OutputFlags": {
"type": "object",
"description": "Output file with more flags",
"properties": {
"cache": {
"description": "Cache outputs by dvc",
"type": "boolean",
"default": true
},
"persist": {
"description": "Persist outputs among runs",
"type": "boolean",
"default": false
}
}
},
"PlotFlags": {
"description": "Plots file with more flags",
"allOf": [
{
"$ref": "#/definitions/OutputFlags"
}
],
"$comment": "Unfortunately, 'additionalProperties' only works props in immediate schema.",
"properties": {
"cache": {},
"persist": {},
"x": {
"description": "Default field name to use as x-axis data",
"type": [
"string",
"integer"
]
},
"y": {
"description": "Default field name to use as y-axis data",
"type": [
"string",
"integer"
]
},
"x_label": {
"description": "Default label for the x-axis",
"type": "string"
},
"y_label": {
"description": "Default label for the y-axis",
"type": "string"
},
"template": {
"description": "Default plot template",
"type": "string"
},
"title": {
"description": "Default plot title",
"type": "string"
},
"header": {
"description": "The target CSV or TSV has a header",
"type": "boolean",
"default": false
}
},
"additionalProperties": false
},
"Output": {
"description": "An output of the stage",
"anyOf": [
{
"type": "string",
"description": "Output will be cached by dvc"
},
{
"type": "object",
"description": "Output file",
"patternProperties": {
"^.*$": {
"$ref": "#/definitions/OutputFlags",
"additionalProperties": false
}
},
"additionalProperties": false,
"maxProperties": 1
}
]
}
},
"properties": {
"stages": {
"description": "List of stages.",
"type": "object",
"patternProperties": {
"^[A-Za-z0-9_-]+$": {
"description": "The stage",
"type": "object",
"properties": {
"cmd": {
"type": "string",
"description": "Command to run"
},
"wdir": {
"type": "string",
"description": "Working directory"
},
"deps": {
"type": "array",
"description": "Dependencies for the stage",
"items": {
"type": "string",
"description": "A dependency for the stage"
},
"minItems": 1,
"uniqueItems": true
},
"params": {
"type": "array",
"description": "Params for the stage",
"items": {
"anyOf": [
{
"type": "string",
"description": "Params from default params file"
},
{
"type": "object",
"description": "Params from custom params file",
"patternProperties": {
"^.*$": {
"type": "array",
"description": "Custom params file",
"items": {
"type": "string",
"description": "Params from custom file"
},
"minItems": 1,
"uniqueItems": true
}
},
"additionalProperties": false,
"maxProperties": 1
}
]
},
"minItems": 1,
"uniqueItems": true
},
"outs": {
"type": "array",
"description": "Outputs of the stage",
"items": {
"$ref": "#/definitions/Output"
},
"minItems": 1,
"uniqueItems": true
},
"metrics": {
"type": "array",
"description": "Metrics outputs of the stage",
"items": {
"$ref": "#/definitions/Output"
},
"minItems": 1,
"uniqueItems": true
},
"plots": {
"type": "array",
"description": "Plots outputs of the stage",
"items": {
"anyOf": [
{
"type": "string",
"description": "Plots will be cached by dvc"
},
{
"type": "object",
"description": "Output file",
"patternProperties": {
"^.*$": {
"$ref": "#/definitions/PlotFlags"
}
},
"maxProperties": 1
}
]
},
"minItems": 1,
"uniqueItems": true
},
"frozen": {
"type": "boolean",
"description": "Assume stage as unchanged",
"default": false
},
"locked": {
"type": "boolean",
"description": "Assume stage as unchanged (deprecated)",
"default": false
},
"always_changed": {
"type": "boolean",
"description": "Assume stage as always changed.",
"default": false
},
"meta": {
"description": "Additional information/metadata"
}
},
"additionalProperties": false,
"required": [
"cmd"
]
}
},
"additionalProperties": false
}
},
"required": [
"stages"
]
}
+2
View File
@@ -14,4 +14,6 @@ include_trailing_comma=true
known_first_party=dvc,tests
known_third_party=PyInstaller,RangeHTTPServer,boto3,colorama,configobj,distro,dpath,flaky,flufl,funcy,git,google,grandalf,mock,mockssh,moto,nanotime,networkx,packaging,paramiko,pathspec,pytest,requests,ruamel,setuptools,shortuuid,tqdm,voluptuous,yaml,zc
line_length=79
force_grid_wrap=0
use_parentheses=True
multi_line_output=3
+2 -5
View File
@@ -60,9 +60,6 @@ install_requires = [
"grandalf==0.6",
"distro>=1.3.0",
"appdirs>=1.4.3",
"treelib>=1.5.5",
"inflect>=2.1.0,<4",
"humanize>=0.5.1",
"PyYAML>=5.1.2,<5.4", # Compatibility with awscli
"ruamel.yaml>=0.16.1",
"funcy>=1.14",
@@ -87,7 +84,7 @@ install_requires = [
# Extra dependencies for remote integrations
gs = ["google-cloud-storage==1.19.0"]
gdrive = ["pydrive2>=1.4.13"]
gdrive = ["pydrive2>=1.4.14"]
s3 = ["boto3>=1.9.201"]
azure = ["azure-storage-blob==2.1.0"]
oss = ["oss2==2.6.1"]
@@ -121,7 +118,7 @@ tests_requirements = [
"flake8-docstrings",
"pydocstyle<4.0",
"jaraco.windows==3.9.2",
"mock-ssh-server@git+https://github.com/iterative/mock-ssh-server.git",
"mock-ssh-server>=0.8.2",
"moto==1.3.14.dev464",
"rangehttpserver==1.2.0",
"beautifulsoup4==4.4.0",
+3
View File
@@ -38,6 +38,9 @@ parts:
sed -rin 's/.*git.*diff.*--quiet.*//' $SNAPCRAFT_PART_SRC/dvc/version.py
override-build: |
snapcraftctl build
# prevent user site packages interfering with this snap - reference:
# https://github.com/snapcore/snapcraft/blob/19393ef36cd773a28131cec10cc0bfb3bf9c7e77/tools/snapcraft-override-build.sh#L18
sed -ri 's/^(ENABLE_USER_SITE = )None$/\1False/' $SNAPCRAFT_PART_INSTALL/usr/lib/python*/site.py
cp $SNAPCRAFT_PART_BUILD/scripts/completion/dvc.bash $SNAPCRAFT_PART_INSTALL/completion.sh
apps:
dvc:
+30
View File
@@ -0,0 +1,30 @@
import os
import pytest
@pytest.fixture
def run_copy_metrics(tmp_dir, run_copy):
def run(file1, file2, commit=None, tag=None, single_stage=True, **kwargs):
stage = tmp_dir.dvc.run(
cmd=f"python copy.py {file1} {file2}",
deps=[file1],
single_stage=single_stage,
**kwargs,
)
if hasattr(tmp_dir.dvc, "scm"):
files = [stage.path]
files += [
os.fspath(out.path_info)
for out in stage.outs
if not out.use_cache
]
tmp_dir.dvc.scm.add(files)
if commit:
tmp_dir.dvc.scm.commit(commit)
if tag:
tmp_dir.dvc.scm.tag(tag)
return stage
return run
-1
View File
@@ -1 +0,0 @@
from tests.func.plots import run_copy_metrics # noqa: F401
+43
View File
@@ -1,7 +1,10 @@
import json
import logging
import yaml
from dvc.main import main
def test_metrics_diff_simple(tmp_dir, scm, dvc, run_copy_metrics):
def _gen(val):
@@ -153,3 +156,43 @@ def test_no_commits(tmp_dir):
assert Git().no_commits
assert Repo.init().metrics.diff() == {}
def test_metrics_diff_dirty(tmp_dir, scm, dvc, run_copy_metrics):
def _gen(val):
tmp_dir.gen({"m_temp.yaml": str(val)})
run_copy_metrics("m_temp.yaml", "m.yaml", metrics=["m.yaml"])
dvc.scm.commit(str(val))
_gen(1)
_gen(2)
_gen(3)
tmp_dir.gen({"m.yaml": "4"})
expected = {"m.yaml": {"": {"old": 3, "new": 4, "diff": 1}}}
assert dvc.metrics.diff() == expected
def test_metrics_diff_cli(tmp_dir, scm, dvc, run_copy_metrics, caplog, capsys):
def _gen(val):
tmp_dir.gen({"m_temp.yaml": f"foo: {val}"})
run_copy_metrics("m_temp.yaml", "m.yaml", metrics=["m.yaml"])
dvc.scm.commit(str(val))
_gen(1.23456789)
_gen(2.34567891011)
_gen(3.45678910111213)
caplog.clear()
assert main(["metrics", "diff", "HEAD~2", "--old"]) == 0
(info,) = [
msg
for name, level, msg in caplog.record_tuples
if name.startswith("dvc") and level == logging.INFO
]
assert info == (
"Path Metric Old New Change\n"
"m.yaml foo 1.23457 3.45679 2.22222"
)
+14
View File
@@ -24,6 +24,20 @@ def test_diff(tmp_dir, scm, dvc):
}
def test_diff_dirty(tmp_dir, scm, dvc):
tmp_dir.gen("params.yaml", "foo: bar")
dvc.run(cmd="echo params.yaml", params=["foo"], single_stage=True)
scm.add(["params.yaml", "Dvcfile"])
scm.commit("bar")
tmp_dir.scm_gen("params.yaml", "foo: baz", commit="baz")
tmp_dir.gen("params.yaml", "foo: qux")
assert dvc.params.diff() == {
"params.yaml": {"foo": {"old": "baz", "new": "qux"}}
}
def test_diff_new(tmp_dir, scm, dvc):
tmp_dir.gen("params.yaml", "foo: bar")
dvc.run(cmd="echo params.yaml", params=["foo"], single_stage=True)
-30
View File
@@ -1,30 +0,0 @@
import os
import pytest
@pytest.fixture
def run_copy_metrics(tmp_dir, run_copy):
def run(file1, file2, commit=None, tag=None, **kwargs):
stage = tmp_dir.dvc.run(
cmd=f"python copy.py {file1} {file2}",
deps=[file1],
single_stage=True,
**kwargs,
)
if hasattr(tmp_dir.dvc, "scm"):
files = [stage.path]
files += [
os.fspath(out.path_info)
for out in stage.outs
if not out.use_cache
]
tmp_dir.dvc.scm.add(files)
if commit:
tmp_dir.dvc.scm.commit(commit)
if tag:
tmp_dir.dvc.scm.tag(tag)
return stage
return run
+12 -1
View File
@@ -1 +1,12 @@
from . import run_copy_metrics # noqa: F401
import shutil
import pytest
@pytest.fixture()
def custom_template(tmp_dir, dvc):
custom_template = tmp_dir / "custom_template.json"
shutil.copy(
tmp_dir / ".dvc" / "plots" / "default.json", custom_template,
)
return custom_template
+16 -2
View File
@@ -32,10 +32,24 @@ def test_diff_dirty(tmp_dir, scm, dvc, run_copy_metrics):
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"y": 5, PlotData.INDEX_FIELD: 0, "rev": "workspace"},
{"y": 6, PlotData.INDEX_FIELD: 1, "rev": "workspace"},
{"y": 3, PlotData.INDEX_FIELD: 0, "rev": "HEAD"},
{"y": 5, PlotData.INDEX_FIELD: 1, "rev": "HEAD"},
{"y": 5, PlotData.INDEX_FIELD: 0, "rev": "workspace"},
{"y": 6, PlotData.INDEX_FIELD: 1, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == PlotData.INDEX_FIELD
assert plot_content["encoding"]["y"]["field"] == "y"
_write_json(tmp_dir, [{"y": 7}, {"y": 8}], "metric.json")
plot_string = dvc.plots.diff(props={"fields": {"y"}})["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"y": 3, PlotData.INDEX_FIELD: 0, "rev": "HEAD"},
{"y": 5, PlotData.INDEX_FIELD: 1, "rev": "HEAD"},
{"y": 7, PlotData.INDEX_FIELD: 0, "rev": "workspace"},
{"y": 8, PlotData.INDEX_FIELD: 1, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == PlotData.INDEX_FIELD
assert plot_content["encoding"]["y"]["field"] == "y"
+70
View File
@@ -0,0 +1,70 @@
import pytest
from dvc.dvcfile import PIPELINE_LOCK
from dvc.repo.plots import PropsNotFoundError
from dvc.repo.plots.template import TemplateNotFoundError
from dvc.utils import relpath
from .test_plots import _write_json
def test_plots_modify_existing_template(
tmp_dir, dvc, run_copy_metrics, custom_template
):
metric = [{"a": 1, "b": 2}, {"a": 2, "b": 3}]
_write_json(tmp_dir, metric, "metric_t.json")
stage = run_copy_metrics(
"metric_t.json",
"metric.json",
plots_no_cache=["metric.json"],
name="copy-metrics",
single_stage=False,
)
dvc.plots.modify(
"metric.json", props={"template": relpath(custom_template)}
)
stage = stage.reload()
assert stage.outs[0].plot == {"template": relpath(custom_template)}
def test_plots_modify_should_not_change_lockfile(
tmp_dir, dvc, run_copy_metrics, custom_template
):
_write_json(tmp_dir, [{"a": 1, "b": 2}], "metric_t.json")
run_copy_metrics(
"metric_t.json",
"metric.json",
plots_no_cache=["metric.json"],
name="copy-metrics",
single_stage=False,
)
(tmp_dir / PIPELINE_LOCK).unlink()
dvc.plots.modify(
"metric.json", props={"template": relpath(custom_template)}
)
assert not (tmp_dir / PIPELINE_LOCK).exists()
def test_plots_modify_not_existing_template(dvc):
with pytest.raises(TemplateNotFoundError):
dvc.plots.modify(
"metric.json", props={"template": "not-existing-template.json"}
)
def test_unset_nonexistent(tmp_dir, dvc, run_copy_metrics, custom_template):
metric = [{"a": 1, "b": 2}, {"a": 2, "b": 3}]
_write_json(tmp_dir, metric, "metric_t.json")
run_copy_metrics(
"metric_t.json",
"metric.json",
plots_no_cache=["metric.json"],
name="copy-metrics",
single_stage=False,
)
with pytest.raises(PropsNotFoundError):
dvc.plots.modify(
"metric.json", unset=["nonexistent"],
)
+78 -87
View File
@@ -10,13 +10,14 @@ import yaml
from funcy import first
from dvc.repo.plots.data import (
JSONPlotData,
NoMetricInHistoryError,
PlotData,
PlotMetricTypeError,
YAMLPlotData,
)
from dvc.repo.plots.show import NoDataOrTemplateProvided
from dvc.repo.plots.template import (
NoDataForTemplateError,
BadTemplateError,
NoFieldInDataError,
TemplateNotFoundError,
)
@@ -50,9 +51,9 @@ def test_plot_csv_one_column(tmp_dir, scm, dvc, run_copy_metrics):
)
props = {
"csv_header": False,
"xlab": "x_title",
"ylab": "y_title",
"header": False,
"x_label": "x_title",
"y_label": "y_title",
"title": "mytitle",
}
plot_string = dvc.plots.show(props=props)["metric.csv"]
@@ -261,7 +262,11 @@ def test_plot_multiple_revs_default(tmp_dir, scm, dvc, run_copy_metrics):
def test_plot_multiple_revs(tmp_dir, scm, dvc, run_copy_metrics):
shutil.copy(tmp_dir / ".dvc" / "plots" / "default.json", "template.json")
templates_dir = dvc.plot_templates.templates_dir
shutil.copy(
os.path.join(templates_dir, "default.json"),
os.path.join(templates_dir, "template.json"),
)
metric_1 = [{"y": 2}, {"y": 3}]
_write_json(tmp_dir, metric_1, "metric_t.json")
@@ -322,13 +327,13 @@ def test_plot_even_if_metric_missing(
caplog.clear()
with caplog.at_level(logging.WARNING, "dvc"):
plot_string = dvc.plots.show(revs=["v1", "v2"])["metric.json"]
plots = dvc.plots.show(revs=["v1", "v2"], targets=["metric.json"])
assert (
"File 'metric.json' was not found at: 'v1'. "
"It will not be plotted." in caplog.text
)
plot_content = json.loads(plot_string)
plot_content = json.loads(plots["metric.json"])
assert plot_content["data"]["values"] == [
{"y": 2, PlotData.INDEX_FIELD: 0, "rev": "v2"},
{"y": 3, PlotData.INDEX_FIELD: 1, "rev": "v2"},
@@ -355,15 +360,6 @@ def test_throw_on_no_metric_at_all(tmp_dir, scm, dvc, caplog):
assert str(error.value) == "Could not find 'metric.json'."
@pytest.fixture()
def custom_template(tmp_dir, dvc):
custom_template = tmp_dir / "custom_template.json"
shutil.copy(
tmp_dir / ".dvc" / "plots" / "default.json", custom_template,
)
return custom_template
def test_custom_template(tmp_dir, scm, dvc, custom_template, run_copy_metrics):
metric = [{"a": 1, "b": 2}, {"a": 2, "b": 3}]
_write_json(tmp_dir, metric, "metric_t.json")
@@ -391,74 +387,10 @@ def _replace(path, src, dst):
path.write_text(path.read_text().replace(src, dst))
def test_custom_template_with_specified_data(
tmp_dir, scm, dvc, custom_template, run_copy_metrics
):
_replace(
custom_template, "DVC_METRIC_DATA", "DVC_METRIC_DATA,metric.json",
)
def test_no_plots(tmp_dir, dvc):
from dvc.exceptions import NoPlotsError
metric = [{"a": 1, "b": 2}, {"a": 2, "b": 3}]
_write_json(tmp_dir, metric, "metric_t.json")
run_copy_metrics(
"metric_t.json",
"metric.json",
outs_no_cache=["metric.json"],
commit="init",
tag="v1",
)
props = {"template": os.fspath(custom_template), "x": "a", "y": "b"}
plot_string = dvc.plots.show(props=props)["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"a": 1, "b": 2, "rev": "workspace"},
{"a": 2, "b": 3, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == "a"
assert plot_content["encoding"]["y"]["field"] == "b"
def test_plot_override_specified_data_source(
tmp_dir, scm, dvc, run_copy_metrics
):
shutil.copy(
tmp_dir / ".dvc" / "plots" / "default.json",
tmp_dir / "newtemplate.json",
)
_replace(
tmp_dir / "newtemplate.json",
"DVC_METRIC_DATA",
"DVC_METRIC_DATA,metric.json",
)
metric = [{"a": 1, "b": 2}, {"a": 2, "b": 3}]
_write_json(tmp_dir, metric, "metric1.json")
run_copy_metrics(
"metric1.json",
"metric2.json",
plots_no_cache=["metric2.json"],
commit="init",
tag="v1",
)
props = {"template": "newtemplate.json", "x": "a"}
plot_string = dvc.plots.show(targets=["metric2.json"], props=props)[
"metric2.json"
]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"a": 1, "b": 2, "rev": "workspace"},
{"a": 2, "b": 3, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == "a"
assert plot_content["encoding"]["y"]["field"] == "b"
def test_should_raise_on_no_template_and_datafile(tmp_dir, dvc):
with pytest.raises(NoDataOrTemplateProvided):
with pytest.raises(NoPlotsError):
dvc.plots.show()
@@ -477,9 +409,21 @@ def test_should_raise_on_no_template(tmp_dir, dvc, run_copy_metrics):
dvc.plots.show("metric.json", props=props)
def test_plot_no_data(tmp_dir, dvc):
with pytest.raises(NoDataForTemplateError):
dvc.plots.show(props={"template": "default"})
def test_bad_template(tmp_dir, dvc, run_copy_metrics):
metric = [{"val": 2}, {"val": 3}]
_write_json(tmp_dir, metric, "metric_t.json")
run_copy_metrics(
"metric_t.json",
"metric.json",
plots_no_cache=["metric.json"],
commit="first run",
)
tmp_dir.gen("template.json", json.dumps({"a": "b", "c": "d"}))
with pytest.raises(BadTemplateError):
props = {"template": "template.json"}
dvc.plots.show("metric.json", props=props)
def test_plot_wrong_metric_type(tmp_dir, scm, dvc, run_copy_metrics):
@@ -580,3 +524,50 @@ def test_raise_on_wrong_field(tmp_dir, scm, dvc, run_copy_metrics):
with pytest.raises(NoFieldInDataError):
dvc.plots.show("metric.json", props={"y": "no_val"})
def test_load_metric_from_dict_json(tmp_dir):
metric = [{"acccuracy": 1, "loss": 2}, {"accuracy": 3, "loss": 4}]
dmetric = {"train": metric}
plot_data = JSONPlotData("-", "revision", json.dumps(dmetric))
expected = metric
for d in expected:
d["rev"] = "revision"
assert list(map(dict, plot_data.to_datapoints())) == expected
def test_load_metric_from_dict_yaml(tmp_dir):
metric = [{"acccuracy": 1, "loss": 2}, {"accuracy": 3, "loss": 4}]
dmetric = {"train": metric}
plot_data = YAMLPlotData("-", "revision", yaml.dump(dmetric))
expected = metric
for d in expected:
d["rev"] = "revision"
assert list(map(dict, plot_data.to_datapoints())) == expected
def test_multiple_plots(tmp_dir, scm, dvc, run_copy_metrics):
metric1 = [
OrderedDict([("first_val", 100), ("second_val", 100), ("val", 2)]),
OrderedDict([("first_val", 200), ("second_val", 300), ("val", 3)]),
]
metric2 = [
OrderedDict([("first_val", 100), ("second_val", 100), ("val", 2)]),
OrderedDict([("first_val", 200), ("second_val", 300), ("val", 3)]),
]
_write_csv(metric1, "metric_t1.csv")
_write_json(tmp_dir, metric2, "metric_t2.json")
run_copy_metrics(
"metric_t1.csv", "metric1.csv", plots_no_cache=["metric1.csv"]
)
run_copy_metrics(
"metric_t2.json", "metric2.json", plots_no_cache=["metric2.json"]
)
assert len(dvc.plots.show().keys()) == 2
+5 -5
View File
@@ -4,19 +4,19 @@ import posixpath
import configobj
from dvc.main import main
from dvc.remote import GDriveRemote
from dvc.remote.gdrive import GDriveRemoteTree
from dvc.repo import Repo
def test_relative_user_credentials_file_config_setting(tmp_dir, dvc):
# CI sets it to test GDrive, here we want to test the work with file system
# based, regular credentials
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
del os.environ[GDriveRemote.GDRIVE_CREDENTIALS_DATA]
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
del os.environ[GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA]
credentials = os.path.join("secrets", "credentials.json")
# GDriveRemote.credentials_location helper checks for file existence,
# GDriveRemoteTree.credentials_location helper checks for file existence,
# create the file
tmp_dir.gen(credentials, "{'token': 'test'}")
@@ -50,6 +50,6 @@ def test_relative_user_credentials_file_config_setting(tmp_dir, dvc):
# Check that in the remote itself we got an absolute path
remote = repo.cloud.get_remote(remote_name)
assert os.path.normpath(remote.credentials_location) == os.path.join(
assert os.path.normpath(remote.tree.credentials_location) == os.path.join(
str_path, credentials
)
+19 -19
View File
@@ -3,9 +3,9 @@ import os
import pytest
from dvc.exceptions import DownloadError, UploadError
from dvc.remote.base import BaseRemote
from dvc.remote.base import Remote
from dvc.remote.index import RemoteIndex
from dvc.remote.local import LocalRemote
from dvc.remote.local import LocalRemote, LocalRemoteTree
from dvc.utils.fs import remove
@@ -15,14 +15,14 @@ def remote(tmp_dir, dvc, tmp_path_factory, mocker):
dvc.config["remote"]["upstream"] = {"url": url}
dvc.config["core"]["remote"] = "upstream"
# patch cache_exists since the RemoteLOCAL normally overrides
# RemoteBASE.cache_exists.
def cache_exists(self, *args, **kwargs):
return BaseRemote.cache_exists(self, *args, **kwargs)
# patch hashes_exist since the LocalRemote normally overrides
# BaseRemoteTree.hashes_exist.
def hashes_exist(self, *args, **kwargs):
return Remote.hashes_exist(self, *args, **kwargs)
mocker.patch.object(LocalRemote, "cache_exists", cache_exists)
mocker.patch.object(LocalRemote, "hashes_exist", hashes_exist)
# patch index class since RemoteLOCAL normally overrides index class
# patch index class since LocalRemote normally overrides index class
mocker.patch.object(LocalRemote, "INDEX_CLS", RemoteIndex)
return dvc.cloud.get_remote("upstream")
@@ -38,9 +38,9 @@ def test_indexed_on_status(tmp_dir, dvc, tmp_path_factory, remote):
dvc.status(cloud=True)
with remote.index:
assert {bar.checksum, baz["md5"]} == set(remote.index.checksums())
assert [bar.checksum] == list(remote.index.dir_checksums())
assert foo.checksum not in remote.index.checksums()
assert {bar.checksum, baz["md5"]} == set(remote.index.hashes())
assert [bar.checksum] == list(remote.index.dir_hashes())
assert foo.checksum not in remote.index.hashes()
def test_indexed_on_push(tmp_dir, dvc, tmp_path_factory, remote):
@@ -50,9 +50,9 @@ def test_indexed_on_push(tmp_dir, dvc, tmp_path_factory, remote):
dvc.push()
with remote.index:
assert {bar.checksum, baz["md5"]} == set(remote.index.checksums())
assert [bar.checksum] == list(remote.index.dir_checksums())
assert foo.checksum not in remote.index.checksums()
assert {bar.checksum, baz["md5"]} == set(remote.index.hashes())
assert [bar.checksum] == list(remote.index.dir_hashes())
assert foo.checksum not in remote.index.hashes()
def test_indexed_dir_missing(tmp_dir, dvc, tmp_path_factory, remote):
@@ -61,7 +61,7 @@ def test_indexed_dir_missing(tmp_dir, dvc, tmp_path_factory, remote):
remote.index.update([bar.checksum], [])
dvc.status(cloud=True)
with remote.index:
assert not list(remote.index.checksums())
assert not list(remote.index.hashes())
def test_clear_on_gc(tmp_dir, dvc, tmp_path_factory, remote, mocker):
@@ -80,7 +80,7 @@ def test_clear_on_download_err(tmp_dir, dvc, tmp_path_factory, remote, mocker):
remove(dvc.cache.local.cache_dir)
mocked_clear = mocker.patch.object(remote.INDEX_CLS, "clear")
mocker.patch.object(LocalRemote, "_download", side_effect=Exception)
mocker.patch.object(LocalRemoteTree, "_download", side_effect=Exception)
with pytest.raises(DownloadError):
dvc.pull()
mocked_clear.assert_called_once_with()
@@ -90,15 +90,15 @@ def test_partial_upload(tmp_dir, dvc, tmp_path_factory, remote, mocker):
tmp_dir.dvc_gen({"foo": "foo content"})
tmp_dir.dvc_gen({"bar": {"baz": "baz content"}})
original = LocalRemote._upload
original = LocalRemoteTree._upload
def unreliable_upload(self, from_file, to_info, name=None, **kwargs):
if "baz" in name:
raise Exception("stop baz")
return original(self, from_file, to_info, name, **kwargs)
mocker.patch.object(LocalRemote, "_upload", unreliable_upload)
mocker.patch.object(LocalRemoteTree, "_upload", unreliable_upload)
with pytest.raises(UploadError):
dvc.push()
with remote.index:
assert not list(remote.index.checksums())
assert not list(remote.index.hashes())
+17 -12
View File
@@ -21,7 +21,7 @@ from dvc.exceptions import (
)
from dvc.main import main
from dvc.output.base import OutputAlreadyTrackedError, OutputIsStageFileError
from dvc.remote import LocalRemote
from dvc.remote.local import LocalRemoteTree
from dvc.repo import Repo as DvcRepo
from dvc.stage import Stage
from dvc.system import System
@@ -187,11 +187,16 @@ def test_add_file_in_dir(tmp_dir, dvc):
class TestAddExternalLocalFile(TestDvc):
def test(self):
from dvc.stage.exceptions import StageExternalOutputsError
dname = TestDvc.mkdtemp()
fname = os.path.join(dname, "foo")
shutil.copyfile(self.FOO, fname)
stages = self.dvc.add(fname)
with self.assertRaises(StageExternalOutputsError):
self.dvc.add(fname)
stages = self.dvc.add(fname, external=True)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertNotEqual(stage, None)
@@ -335,7 +340,7 @@ class TestAddCommit(TestDvc):
def test_should_collect_dir_cache_only_once(mocker, tmp_dir, dvc):
tmp_dir.gen({"data/data": "foo"})
get_dir_checksum_counter = mocker.spy(LocalRemote, "get_dir_checksum")
get_dir_hash_counter = mocker.spy(LocalRemoteTree, "get_dir_hash")
ret = main(["add", "data"])
assert ret == 0
@@ -344,7 +349,7 @@ def test_should_collect_dir_cache_only_once(mocker, tmp_dir, dvc):
ret = main(["status"])
assert ret == 0
assert get_dir_checksum_counter.mock.call_count == 1
assert get_dir_hash_counter.mock.call_count == 1
class SymlinkAddTestBase(TestDvc):
@@ -444,20 +449,20 @@ class TestShouldThrowProperExceptionOnCorruptedStageFile(TestDvc):
class TestAddFilename(TestDvc):
def test(self):
ret = main(["add", self.FOO, self.BAR, "-f", "error.dvc"])
ret = main(["add", self.FOO, self.BAR, "--file", "error.dvc"])
self.assertNotEqual(0, ret)
ret = main(["add", "-R", self.DATA_DIR, "-f", "error.dvc"])
ret = main(["add", "-R", self.DATA_DIR, "--file", "error.dvc"])
self.assertNotEqual(0, ret)
with self.assertRaises(RecursiveAddingWhileUsingFilename):
self.dvc.add(self.DATA_DIR, recursive=True, fname="error.dvc")
ret = main(["add", self.DATA_DIR, "-f", "data_directory.dvc"])
ret = main(["add", self.DATA_DIR, "--file", "data_directory.dvc"])
self.assertEqual(0, ret)
self.assertTrue(os.path.exists("data_directory.dvc"))
ret = main(["add", self.FOO, "-f", "bar.dvc"])
ret = main(["add", self.FOO, "--file", "bar.dvc"])
self.assertEqual(0, ret)
self.assertTrue(os.path.exists("bar.dvc"))
self.assertFalse(os.path.exists("foo.dvc"))
@@ -578,7 +583,7 @@ def test_readding_dir_should_not_unprotect_all(tmp_dir, dvc, mocker):
dvc.add("dir")
tmp_dir.gen("dir/new_file", "new_file_content")
unprotect_spy = mocker.spy(LocalRemote, "unprotect")
unprotect_spy = mocker.spy(LocalRemoteTree, "unprotect")
dvc.add("dir")
assert not unprotect_spy.mock.called
@@ -592,7 +597,7 @@ def test_should_not_checkout_when_adding_cached_copy(tmp_dir, dvc, mocker):
shutil.copy("bar", "foo")
copy_spy = mocker.spy(dvc.cache.local, "copy")
copy_spy = mocker.spy(dvc.cache.local.tree, "copy")
dvc.add("foo")
@@ -618,7 +623,7 @@ def test_should_relink_on_repeated_add(
tmp_dir.dvc_gen({"foo": "foo", "bar": "bar"})
os.remove("foo")
getattr(dvc.cache.local, link)(PathInfo("bar"), PathInfo("foo"))
getattr(dvc.cache.local.tree, link)(PathInfo("bar"), PathInfo("foo"))
dvc.cache.local.cache_types = [new_link]
@@ -698,7 +703,7 @@ def test_add_empty_files(tmp_dir, dvc, link):
def test_add_optimization_for_hardlink_on_empty_files(tmp_dir, dvc, mocker):
dvc.cache.local.cache_types = ["hardlink"]
tmp_dir.gen({"foo": "", "bar": "", "lorem": "lorem", "ipsum": "ipsum"})
m = mocker.spy(LocalRemote, "is_hardlink")
m = mocker.spy(LocalRemoteTree, "is_hardlink")
stages = dvc.add(["foo", "bar", "lorem", "ipsum"])
assert m.call_count == 1
+63 -6
View File
@@ -8,9 +8,19 @@ from dvc.exceptions import FileMissingError
from dvc.main import main
from dvc.path_info import URLInfo
from dvc.utils.fs import remove
from tests.remotes import GCP, HDFS, OSS, S3, SSH, Azure, Local
from tests.remotes import (
GCP,
HDFS,
OSS,
S3,
SSH,
TEST_REMOTE,
Azure,
GDrive,
Local,
)
remote_params = [S3, GCP, Azure, OSS, SSH, HDFS]
remote_params = [S3, GCP, Azure, GDrive, OSS, SSH, HDFS]
all_remote_params = [Local] + remote_params
@@ -25,9 +35,48 @@ def run_dvc(*argv):
assert main(argv) == 0
def ensure_dir(dvc, url):
if url.startswith("gdrive://"):
GDrive.create_dir(dvc, url)
run_dvc(
"remote",
"modify",
TEST_REMOTE,
"gdrive_service_account_email",
"test",
)
run_dvc(
"remote",
"modify",
TEST_REMOTE,
"gdrive_service_account_p12_file_path",
"test.p12",
)
run_dvc(
"remote",
"modify",
TEST_REMOTE,
"gdrive_use_service_account",
"True",
)
def ensure_dir_scm(dvc, url):
if url.startswith("gdrive://"):
GDrive.create_dir(dvc, url)
with dvc.config.edit() as conf:
conf["remote"][TEST_REMOTE].update(
gdrive_service_account_email="test",
gdrive_service_account_p12_file_path="test.p12",
gdrive_use_service_account=True,
)
dvc.scm.add(dvc.config.files["repo"])
dvc.scm.commit(f"modify '{TEST_REMOTE}' remote")
@pytest.mark.parametrize("remote_url", remote_params, indirect=True)
def test_get_url(tmp_dir, dvc, remote_url):
run_dvc("remote", "add", "-d", "upstream", remote_url)
run_dvc("remote", "add", "-d", TEST_REMOTE, remote_url)
tmp_dir.dvc_gen("foo", "foo")
expected_url = URLInfo(remote_url) / "ac/bd18db4cc2f85cedef654fccc4a4d8"
@@ -58,7 +107,8 @@ def test_get_url_requires_dvc(tmp_dir, scm):
@pytest.mark.parametrize("remote_url", all_remote_params, indirect=True)
def test_open(remote_url, tmp_dir, dvc):
run_dvc("remote", "add", "-d", "upstream", remote_url)
run_dvc("remote", "add", "-d", TEST_REMOTE, remote_url)
ensure_dir(dvc, remote_url)
tmp_dir.dvc_gen("foo", "foo-text")
run_dvc("push")
@@ -72,6 +122,7 @@ def test_open(remote_url, tmp_dir, dvc):
@pytest.mark.parametrize("remote_url", all_remote_params, indirect=True)
def test_open_external(remote_url, erepo_dir, setup_remote):
setup_remote(erepo_dir.dvc, url=remote_url)
ensure_dir_scm(erepo_dir.dvc, remote_url)
with erepo_dir.chdir():
erepo_dir.dvc_gen("version", "master", commit="add version")
@@ -95,7 +146,8 @@ def test_open_external(remote_url, erepo_dir, setup_remote):
@pytest.mark.parametrize("remote_url", all_remote_params, indirect=True)
def test_open_granular(remote_url, tmp_dir, dvc):
run_dvc("remote", "add", "-d", "upstream", remote_url)
run_dvc("remote", "add", "-d", TEST_REMOTE, remote_url)
ensure_dir(dvc, remote_url)
tmp_dir.dvc_gen({"dir": {"foo": "foo-text"}})
run_dvc("push")
@@ -109,11 +161,16 @@ def test_open_granular(remote_url, tmp_dir, dvc):
@pytest.mark.parametrize("remote_url", all_remote_params, indirect=True)
def test_missing(remote_url, tmp_dir, dvc):
tmp_dir.dvc_gen("foo", "foo")
run_dvc("remote", "add", "-d", "upstream", remote_url)
run_dvc("remote", "add", "-d", TEST_REMOTE, remote_url)
ensure_dir(dvc, remote_url)
# Remove cache to make foo missing
remove(dvc.cache.local.cache_dir)
api.read("foo")
remove("foo")
with pytest.raises(FileMissingError):
api.read("foo")
+9 -9
View File
@@ -30,14 +30,14 @@ class TestCache(TestDvc):
self.create(self.cache2, "2")
def test_all(self):
md5_list = list(Cache(self.dvc).local.all())
md5_list = list(Cache(self.dvc).local.tree.all())
self.assertEqual(len(md5_list), 2)
self.assertIn(self.cache1_md5, md5_list)
self.assertIn(self.cache2_md5, md5_list)
def test_get(self):
cache = Cache(self.dvc).local.get(self.cache1_md5)
self.assertEqual(cache, self.cache1)
cache = Cache(self.dvc).local.hash_to_path_info(self.cache1_md5)
self.assertEqual(os.fspath(cache), self.cache1)
class TestCacheLoadBadDirCache(TestDvc):
@@ -46,16 +46,16 @@ class TestCacheLoadBadDirCache(TestDvc):
self.assertEqual(len(ret), 0)
def test(self):
checksum = "123.dir"
fname = self.dvc.cache.local.get(checksum)
dir_hash = "123.dir"
fname = os.fspath(self.dvc.cache.local.hash_to_path_info(dir_hash))
self.create(fname, "<clearly>not,json")
with pytest.raises(DirCacheError):
self.dvc.cache.local.load_dir_cache(checksum)
self.dvc.cache.local.load_dir_cache(dir_hash)
checksum = "234.dir"
fname = self.dvc.cache.local.get(checksum)
dir_hash = "234.dir"
fname = os.fspath(self.dvc.cache.local.hash_to_path_info(dir_hash))
self.create(fname, '{"a": "b"}')
self._do_test(self.dvc.cache.local.load_dir_cache(checksum))
self._do_test(self.dvc.cache.local.load_dir_cache(dir_hash))
class TestExternalCacheDir(TestDvc):

Some files were not shown because too many files have changed in this diff Show More