Compare commits

..
184 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 7c6351d039 dvc: bump to 1.0.0b4 2020-06-16 15:17:19 +03:00
Ruslan Kuprieiev 613ca0bdb5 get-url/import-url: save dependency before downloading (#4049)
This allows us to properly process the errors like missing target.

Fixes #3897
2020-06-16 13:05:21 +03:00
Peter Rowlands (변기호) 0fb9870de6 remote: rename "checksum" -> "hash" (#4051)
* remote: s/get_checksum/get_hash/

* remote/tree: s/get_file_checksum/get_file_hash/

* remote: s/get_dir_checksum/get_dir_hash/

* remote: s/checksum_to_path/hash_to_path/

* remote: s/path_to_checksum/path_to_hash/

* remote: cleanup remaining checksum references

* remote.index: s/checksum/hash/

* note: does not rename "checksum" field in the index DB schema
2020-06-16 12:16:51 +03:00
Ruslan Kuprieiev 73e60a8c13 Delete pre-commit.yaml 2020-06-15 23:35:10 +03:00
Ruslan Kuprieiev 9d1dde4051 Create pre-commit.yaml 2020-06-15 23:32:39 +03:00
Ruslan Kuprieiev 38dc0f85d4 dvc: bump to 1.0.0b3 2020-06-15 22:15:57 +03:00
Paweł Redzyński d2dbe9649c plots: add smooth template (#4046) 2020-06-15 22:15:27 +03:00
Ruslan Kuprieiev ece6f2d3ee dvc: bump to 1.0.0b2 2020-06-15 21:36:05 +03:00
Ruslan Kuprieiev b672622324 plots: make sure template is using data anchor (#4045)
Fixes #3969
2020-06-15 21:14:07 +03:00
Ruslan Kuprieiev c9391df313 plots: modify: handle --unset for nonexistent options (#4044)
* tests: plots: reorginize the structure

* plots: modify: handle --unset for nonexistent options

Fixes #3933
2020-06-15 20:31:08 +03:00
Peter Rowlands (변기호) a0c229cf9b remote: continue remote/cache/tree refactoring (#4041)
* remote: move constants into trees

* remote: replace Remote() Cache() helper functions

* output: update for remote changes

* cache/data_cloud: update for remote changes

* dependency: update for remote changes

* stage: update for remote changes

* remote: move remaining non-index methods into tree

* tests: update remote unit tests

* remote: LocalCache needs to lock remote index, not cache index

* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH

* tests: repo tree open needs state context

* remote: make Remote.gc a class method

* operates on obj.tree, so that GC can be done with both remote and
  cache instances

* bug fixes

* tests: update test_data_cloud

* bug fixes

* tests: update func tests
2020-06-15 16:22:55 +03:00
karajan1001 a59f90fac9 Performance improvement of dvcignore (#3967)
fix#3869
1.Use big regex.

* Solve windows

* add rule order test

* Solve ignore order

* remove list comprehensions
2020-06-15 11:37:01 +02:00
Ruslan Kuprieiev bba502314d dvc: bump to 1.0.0b1 2020-06-14 20:44:44 +03:00
Saugat Pachhai bc4bf66d17 (bugfix) get/import: fix broken progress bar (#4039)
PR #3672 (6d8499e) extended `LocalRemote::_get_plans` to return one
`checksums` too. As all of the args from `_get_plans` was passed
down to `download()`, it recognized extra arg of checksum as
`no_progress_bar` due to which it became True and stopped showing
progress bar at all.

Fix #3874
2020-06-14 20:42:53 +03:00
f7348af1e0 metrics & plots: help output improvements (#3963)
* metrics & plots: help output improvements

* metrics & plots: ui: updated help messages

* refactor: removed redundant parentheses

* metrics: ui: updated 'targets' description in 'metrics show'

* metrics: ui: reordered params to match the order in cmd ref

* plots: ui: params modified to match cmd ref

* plots: ui: updated diff help

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
2020-06-14 20:39:33 +03:00
Casper da Costa-Luis 4d140a9944 gdrive: add open (#3916)
* gdrive: add open

Fixes #3408
Related #2865
Fixes #3897

* dependency: add gdrive

* test: api: open: gdrive

* rollback pydrive dependency

* Revert "dependency: add gdrive"

This reverts commit fd333261d2e9fe1364cc278138c79890a1f5fe7a.

* tests: remotes: GDrive: fix get_url

* tests: api: fully test GDrive

* minor typo fix

* attempt gdrive tests fix

* fix gdrive credentials config

* replace GDrivePathNotFound => FileMissingError

* fix test_open_external[GDrive]

* add ensure_dir_scm

* minor exception fix
2020-06-13 23:53:25 +03:00
Casper da Costa-Luis 2ba73daedc snap: make future major version upgrades easier (#4038)
Related: #3999
Related: #4037
2020-06-13 23:27:03 +03:00
Ruslan Kuprieiev 7a31a35cbf dvc: bump to 1.0.0b0 2020-06-13 00:56:14 +03:00
Saugat Pachhai 24a14563cc run: preserve order when -f/--force overwrite stage (#4035)
This commit preserves comments, meta and order inside a dvc.yaml
when `-f/--force` is used. But, it does not maintain comments/meta
for .dvc files.

Related: #4014
2020-06-13 00:55:15 +03:00
Ruslan Kuprieiev a1d3440f85 metrics: diff: introduce --precision (#4026)
Rounding to 5 digits by default.

Fixes #3904
2020-06-13 00:54:42 +03:00
Saugat Pachhai 45dcb7c27d Update dag.py (#4036) 2020-06-12 20:33:19 +03:00
Saugat Pachhai 9ff6fef53a bugfix: get dump folder contents in respective output folder (#4034)
The relative path for external git directory was computed relative to
the root of the repo itself, which created a relative path which is
current working directory. So, all of the contents of the erepo to download
was dumped in the pwd rather than inside of it's own directory.

Fix #4033
2020-06-12 20:31:18 +03:00
Saugat PachhaiandRuslan Kuprieiev 9e7dc2640e schema: provide a jsonschema for dvc.yaml (#4028)
* schema: provide a jsonschema for dvc.yaml

* Update scripts/schema/dvc-yaml.json

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update scripts/schema/dvc-yaml.json

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-12 20:09:59 +05:45
Paweł Redzyński 584c97a58d plots: fix handling of multiple plots from single revision (#4031) 2020-06-12 16:08:08 +03:00
Saugat Pachhai b94a18cfb9 tests: tests for dvcfile (#4032) 2020-06-12 15:54:05 +03:00
Peter Rowlands (변기호) 9ead641c95 remote: separate cloud remote and cloud cache classes (#4019)
* remote: move get_file_checksum() into tree

* makes RemoteTree and RepoTree consistent with regard to checksum
  calculation

* remote: save() now takes explicit tree parameter

* when tree is remote.tree, save will be a move + link operation
  (same as default existing behavior)
* when saving path from a different tree, save will be a copy operation

* tests: update for moved remote/tree functions

* remote: move get_checksum into tree

* remote: separate cloud remote and cache classes

* tests: update unit tests for remote/cache separation

* remote: cloud cache should extend cloud remote

* remote: move LocalRemote.get

* tests: update func tests

* dependency: update for moved get_checksum

* remote: fix state lookup bug
2020-06-12 15:52:47 +03:00
Peter Rowlands (변기호) 652f5ab439 remote: use separate working tree instance for local cache (#3991)
* dvcignore: ignore paths outside of CleanTree root

* remote: local remote/cache should always use its own working tree

* repo tree (CleanTree) instance should not be relevant to local cache
  operations. When the dvcignore CleanTree is relevant, repo.tree should
  be used. For local remote/cache operations, the local remote work_tree
  should be used.

* state: tie state to local cache working tree

* tests: update tests

* state: use os.path.exists instead of WorkingTree.exists in some cases
2020-06-12 12:52:38 +03:00
bjonnhandJonathan Bisson 28a123e727 Remove a remnant umask to solve #4021 (#4022)
Co-authored-by: Jonathan Bisson <[email protected]>
2020-06-11 19:47:25 +03:00
Ruslan Kuprieiev 566472f088 dvc: bump to 1.0.0a11 2020-06-11 19:09:51 +03:00
Saugat PachhaiandRuslan Kuprieiev 2e12a31c5a tests: cover run cache with functional tests (#4018)
* tests: cover run cache with functional tests

* run-cache: use _log_exceptions wrapper

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-11 18:44:21 +03:00
Alexander Schepanovski d7788887f0 plots: fix a couple of bugs (#4017)
- running .plots.collect() not from repo root used to skip plot files
- make Plots.render() be more convenient to be called from outside
2020-06-11 18:20:12 +07:00
Ruslan Kuprieiev 87276faef2 RepoTree: work with dirty repos (#4005)
Currently `RepoTree.open` will throw FileNotFoundError if you have a
file that is tracked by dvc, but which has no md5 or a cache for which
is missing, even though you have the file itself in your workspace.

This corresponds very nicely to the `--no-commit` or `--no-exec`
scenario, where you might not want to commit your artifacts to cache
just yet.

Fixes #3974
2020-06-11 09:50:27 +03:00
Saugat Pachhai 4042d5232d run: use -f/--force to overwrite, fix msg (#4011)
Also, makes fixes for `--overwrite` rename to `--force`
and also remove `remove_with_prompt` which was only being
used with `force\=True` anyway.

Fix #4003
2020-06-10 20:17:27 +03:00
Saugat Pachhai fda14c82d3 plots modify: do not convert output types to plot on unset (#4009)
Fix #4006
2020-06-10 15:31:17 +03:00
Saugat Pachhai 64be262d18 dvc.yaml: preserve meta (#4008) 2020-06-10 14:37:21 +03:00
Saugat Pachhai 1d6c80001d plots modify: complain if template does not exist (#3995) 2020-06-10 16:12:56 +05:45
Ruslan Kuprieiev 91a7194e4f add/run/import-url: don't use -f as --file (#4004)
If anything, `-f` is closer to `--force` than to `--file`. So let's set
this straight.
2020-06-10 05:04:55 +03:00
Ruslan Kuprieiev afe539aaba plots: rename --no-csv-header to --no-header (#4001)
* plots: rename --no-csv-header to --no-header

Fixes #3996

* fix typo
2020-06-10 03:34:25 +03:00
Ruslan Kuprieiev f259bc52cd config/remote/cache: make --system/--global/--local mutually exclusive (#4000) 2020-06-10 02:32:18 +03:00
Ruslan Kuprieiev db69def834 status: introduce --show-json (#3998)
Fixes #1406
2020-06-10 01:51:30 +03:00
Casper da Costa-Luis 5efc3bbee2 snap: build fixes (#3990)
* snap: drop sudo

Fixes #3989

* snap: ci: fix builds

* snap: ci: developer notifications

* snap: minimise size

* debug commit

* Revert "snap: minimise size"

This reverts commit 77b41a96f68b2c8df833803d0a644172b7db4779.

* fix image info

* Revert "debug commit"

This reverts commit 232baf0ad78335b0f3806456a3f20b82cdf0f24c.

* snap: v1 channels

Related: #3872
2020-06-10 00:00:15 +03:00
Alexander Schepanovski 7230d87137 Refactor plots (#3994)
- separate data collection and rendering
- provide a way to use custom template loader
- drop hard-coding datafiles in template feature
2020-06-09 22:47:17 +03:00
Paweł Redzyński 56833387ce plots: find metric data in yaml dictionary (#3992) 2020-06-09 16:42:28 +03:00
Saugat Pachhai 593e67318b do not consider stage cached if checksum/cmd has changed (#3993) 2020-06-09 18:46:49 +05:45
Saugat Pachhai afc183f4e7 tests serializing of stage to lockfile (#3988)
* tests: serialize stage for lockfile

* config: make isort/black compatible

* convert class based tests to pytest
2020-06-09 15:48:17 +03:00
Saugat Pachhai d201fad1e3 tests: serialize pipeline file (#3985)
serialize: make params ordered
2020-06-09 14:22:55 +03:00
Ruslan Kuprieiev 5be12b5aac plots: fix help message (#3983)
Per https://github.com/iterative/dvc/pull/3911#discussion_r436238442
2020-06-09 14:15:15 +03:00
Saugat Pachhai c1e5b10cbd remove unnecessary dependencies (#3987)
Fix #3971
2020-06-09 12:43:34 +03:00
Saugat Pachhai b47b2c5673 tests: update mock-ssh-server (#3986) 2020-06-09 12:39:10 +03:00
Ruslan Kuprieiev 939f78948a remote: tree: don't ignore exceptions (#3977)
Moving ignore logic to the remote itself, so that tree could behave
properly.
2020-06-09 11:25:13 +03:00
Saugat Pachhai 215ea4944c log: show modifying if entry already exists in dvcfile (#3978) 2020-06-08 18:56:00 +03:00
Saugat Pachhai 4e12e76e19 tests: unit testing for StageLoader for Pipelinefile (#3976) 2020-06-08 18:55:16 +03:00
Saugat Pachhai 33092e8614 issue template: do not auto-add labels on new issues (#3979) 2020-06-08 19:47:42 +05:45
Saugat Pachhai ae7b604531 Fix loading of params from stage cache (#3972) 2020-06-08 12:43:13 +03:00
Ruslan Kuprieiev b7c174641e tests: mark s3 test as flaky 2020-06-08 12:27:25 +03:00
Ruslan Kuprieiev cb43f9cdda dvc: bump to 1.0.0a10 2020-06-08 11:56:58 +03:00
Ruslan Kuprieiev 791ce22759 pyinstaller: add hook for google-api-python-client (#3965)
Fixes #3913
2020-06-08 11:53:12 +03:00
Casper da Costa-Luis c70adebc75 snap: restrict site-package checks (#3878)
* snap: restrict site-package checks

Fixes #3514

* add comment
2020-06-07 07:50:15 +03:00
2c9cb87679 Subcommand help (#3956)
* change DvcParser to print subcommand help

* add tests for subcommand help

* docstring refactoring

* remote: finish separating RemoteTree methods (#3946)

* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes

* Unify metrics behavior with dirs (#3935)

* run: disable directories for -m/-M flags

* run: reformatted string literals

* run: simplified metrics check

* tests: func: run: dirs as metrics outs

* tests: func: run: reformatted string literals

* tests: use pytest

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* refactor/test multistage load for params and outputs (#3949)

* refactor multistage load for params and outputs

* tests: load params

* tests: output loading from pipeline file

* fix test

* fix typo in name

* split params load

* rename params func s/inject_values/fill_values

* fix tests

* simplify loads_params and output.load_from_pipeline

* address @pared's suggestions for tests

* remote: fix build errors (#3957)

* remote: fix remaining gdrive build issues (#3959)

* tests: fix gdrive build

* remote.gdrive: fix walk_files prefix relpath

* change DvcParser to print subcommand help

* add tests for subcommand help

* docstring refactoring

* change tests to pytest-style

* remove gettext

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: nik123 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
2020-06-05 23:44:02 +03:00
Peter Rowlands (변기호) b57077af11 remote: fix remaining gdrive build issues (#3959)
* tests: fix gdrive build

* remote.gdrive: fix walk_files prefix relpath
2020-06-05 15:36:47 +09:00
Peter Rowlands (변기호) 803bb3eb61 remote: fix build errors (#3957) 2020-06-05 05:14:34 +03:00
Saugat Pachhai 99b58d3d05 refactor/test multistage load for params and outputs (#3949)
* refactor multistage load for params and outputs

* tests: load params

* tests: output loading from pipeline file

* fix test

* fix typo in name

* split params load

* rename params func s/inject_values/fill_values

* fix tests

* simplify loads_params and output.load_from_pipeline

* address @pared's suggestions for tests
2020-06-04 23:35:07 +03:00
nik123andRuslan Kuprieiev 5bbaa28cae Unify metrics behavior with dirs (#3935)
* run: disable directories for -m/-M flags

* run: reformatted string literals

* run: simplified metrics check

* tests: func: run: dirs as metrics outs

* tests: func: run: reformatted string literals

* tests: use pytest

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-04 23:32:00 +03:00
Peter Rowlands (변기호) 74bd8137ba remote: finish separating RemoteTree methods (#3946)
* remote: move upload()/download() into tree

* remote: move path_info/path_cls into tree

* remote.azure: finish moving methods into tree

* remote.gs: finish moving methods into tree

* remote.hdfs: finish moving methods to tree

* remote.http: finish moving methods into tree

* remote.oss: finish moving methods into tree

* remote.s3: finish moving methods into tree

* remote.ssh: finish moving methods into tree

* remote.gdrive: finish moving methods into tree

* tests: update remote unit tests

* tests: update func tests

* remote: use walk_files() for all remotes

* list_cache_paths() now uses tree.walk_files() for all remotes except
  local/ssh

* fix DS warnings

* bugfixes
2020-06-04 23:22:31 +03:00
Ruslan Kuprieiev 72a576cefe dvc: bump to 1.0.0a9 2020-06-04 06:06:25 +03:00
Ruslan Kuprieiev 036ce9398e tests: move mock-ssh-server git dependency (#3953)
This one doesn't allow us to release new pre-releases, because pypi
forbids that:

https://travis-ci.com/github/iterative/dvc/jobs/343343967#L2447

Let's install it separately for now, until there is a new official
mock-ssh-server release.
2020-06-04 05:54:29 +03:00
Ruslan Kuprieiev 13012f9c0c plots: completion/spelling fixups (#3952)
* completion: don't forget --show-vega

Fixes #3890

* plots: VEGA -> Vega
2020-06-04 05:49:20 +03:00
Ruslan Kuprieiev a65c7989c8 tests: don't cache gdrive url (#3941)
This makes `GDrive.get_url` behaviour consistent with the rest of the
test remotes.

Related to #3916
2020-06-04 04:01:12 +03:00
Ruslan Kuprieiev 3590fe8414 dvc: bump to 1.0.0a8 2020-06-04 00:10:10 +03:00
Ruslan Kuprieiev 05100122e1 plots: remove print
Leftover from https://github.com/iterative/dvc/pull/3948

FIxes #3951
2020-06-04 00:09:16 +03:00
Alexander Schepanovski 9a55b909f8 Rename --xlab to --x-label, X_TITLE to X_LABEL, same for y (#3948)
Renamed:
- cli options
- dvcfile plot props
- template anchors
- variable names throughout the code
2020-06-03 20:30:17 +03:00
Ruslan Kuprieiev ff54ac1bfb plots: fix error when there are no plots in repo (#3940)
Consistent with `dvc metrics show/diff` behaviour.

Fixes #3925
2020-06-03 08:54:48 +03:00
Ruslan Kuprieiev 57e920aa39 tests: don't forget --external when adding external files 2020-06-03 08:38:51 +03:00
Ruslan Kuprieiev ac4f351daf dvc: require --external for external outputs (#3929)
Fixes #1545
2020-06-03 06:13:09 +03:00
Peter Rowlands (변기호) 2c0116ccf9 remote: separate tree functions from main remote classes (#3931)
* BaseRemote: start moving tree functions into BaseRemoteTree

* remote: add LocalRemoteTree

* remote: add cloud remote trees

* remote: move remove(), makedirs() into tree

* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree

* fix attributes for moved remote methods

* tests: update unit tests for moved remote tree methods

* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py

* fix attributes for moved remote tree functions

* tests: update func tests for moved remote tree methods

* test fixes

* fix DS warnings
2020-06-03 06:04:25 +03:00
Ruslan Kuprieiev c03ce4aa0f dag: fix help
per https://github.com/iterative/dvc.org/pull/1383#discussion_r433397776
2020-06-03 04:25:15 +03:00
Ruslan Kuprieiev 72dcbc0a9e remote: modify: fix --unset (#3939)
https://discordapp.com/channels/485586884165107732/563406153334128681/717474697721413686

Fixes https://github.com/iterative/dvc/issues/3938
2020-06-03 02:09:08 +03:00
Alexander Schepanovski fd65f44b7f Fix quadratic behavior in Repo._collect_stages() (#3932) 2020-06-02 16:56:49 +03:00
Ruslan Kuprieiev 7504b68c12 dvc: bump to 1.0.0a7 2020-06-02 03:34:04 +03:00
Ruslan Kuprieiev 86f3457dec dvc: remove dvc metrics add/remove (#3928)
We will be re-introducing `metrics modify` in the future and
will likely revisit `add/remove` too, so it is better to get
rid of these for now and introduce later with a clear scneario
in mind.

Fixes #3849
Fixes #3907
2020-06-02 00:17:03 +03:00
Saugat Pachhai 219662597b refactor: rename utils.stage to utils.yaml (#3923) 2020-06-01 16:49:46 +03:00
Alexander Schepanovski 536966f9d1 Fix modifying plot (#3926)
- no error when modifying plot with no props yet
- do not turn into non-plot if all props are unset
2020-06-01 16:19:11 +03:00
Saugat Pachhai 076eadceba bugfix: rerun stage when command is changed (#3922)
Previously, the stage would not rerun if the deps/outs are cached
even if the cmd has changed.

Versions affected: >0.93.0

Fixes #3918
2020-06-01 14:04:31 +05:45
Ruslan Kuprieiev d1e08b6157 dvc: replace pipeline with dag (#3905)
`pipeline list` is no more, it had some very questionable applications
(none to be precise) and was duplicating old `pipeline show` functionality.
We might introduce a different command to assist in listing (e.g. for
shell completion) in the future.

New `dvc dag` replaces `dvc pipeline show` and by-default will show
all pipelines in the project in ASCII (so there is no more --ascii)
and also `--dot` support, just like before. When given a target,
`dvc dag` (in any mode) will show DAG that leads to the target,
skipping the descendants, unless `--full` is specified.
2020-06-01 02:35:36 +03:00
Saugat Pachhai edfa1741db implement removing stage (#3881)
This should allow us to be able to remove a stage from a dvcfile,
(if it's a single stage file, it removes it), and make us able to
delete the overwrite the stage on run as well.

This commit:
* changes `run` `--overwrite-dvcfile` flag to `--overwrite`
* removes prompt in `run` for overwrite and fails hard
* changes `remove`: now it by default removes stage entry and unprotect outs
* `remove` has `--outs` to remove outputs instead of unprotecting them
2020-05-31 17:41:19 +03:00
Ruslan Kuprieiev 9ee62ef8f0 dvc: rename "working tree" to "workspace" (#3914)
Fixes #3892
2020-05-31 17:39:04 +03:00
Ruslan Kuprieiev 0bb49d2549 dvc: rename dvc lock -> dvc freeze (#3915)
Fixes #3901
2020-05-31 03:38:21 +03:00
Alexander SchepanovskiandRuslan Kuprieiev 516d82e411 Add plots props and plots modify params (#3911)
* Add plots props and `plots modify` params

* Update dvc/command/plots.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-30 14:37:11 +03:00
Saugat Pachhai ec312d8601 use fork of mockssh-server (#3910) 2020-05-30 08:31:59 +05:45
Ruslan Kuprieiev f504245f35 pre-commit: don't run dvc in parallel (#3912)
* pre-commit: use local hooks

* pre-commit: don't run dvc in parallel

By default pre-commit will try to run hooks in parallel, which results
in some dvc instances failing to acquire repo locks.

Fixes #3751
2020-05-29 21:20:46 +03:00
Saugat Pachhai bb50fb8664 use higher level API selectors instead of select (#3899) 2020-05-29 09:25:07 +05:45
Jorge Orpinel 777a9f80ae Update bug report template (#3903)
done directly from GH
2020-05-29 08:09:35 +05:45
Ruslan Kuprieiev 02a52d038a tests: make ssh_server session scoped (#3898)
Fixes #3888
2020-05-28 16:09:58 +03:00
Peter Rowlands (변기호) 7d0a06248d dvcignore: use correct root .dvcignore for subrepos (#3895)
* ignore: dvcignore root is distinct from tree_root

* for subrepos, dvcignore root should be the subrepo's .dvcignore, not
  host repo's .dvcignore

* tests: add test for subrepo dvcignore behavior
2020-05-28 20:52:48 +09:00
Ruslan Kuprieiev 2d8c6043ca plots: replace --show-json with --show-vega (#3891)
Fixes #3890
2020-05-28 14:42:41 +03:00
Nicholas A. Del Grosso 3f996048e8 Bugfix: _credentials_location (#3896)
_credentials_location  to credentials_location to removed attributeerror
2020-05-28 14:41:34 +03:00
Peter Rowlands (변기호) 8d9435e8ee brancher: only yield git revisions that contain repo_root (#3883)
* tests: add test for metrics show -T with pre-existing tags

* brancher: ignore git revisions that do not contain repo

* tests: update test to include cmd

* CleanTree: tree_root for subrepos should be the repo root, not scm root

* only force subrepo root for git trees

* fix windows shell quotes
2020-05-28 12:26:29 +09:00
Saugat Pachhai 889a17e2fc modify issue template (#3887)
* modify issue template

* change bug issue template
* auto add labels
* use emojis
* add link to Discord chat
2020-05-28 08:21:02 +05:45
Ruslan Kuprieiev 63972c3d12 dvc: bump to 1.0.0a6 2020-05-27 13:59:03 +03:00
Ruslan Kuprieiev 7027c631c5 dvcignore: fix bug in subrepo detection (#3880) 2020-05-27 12:13:53 +03:00
Saugat Pachhai e8f6cd7f5b run: raise error when command is not specified (#3850)
* run: raise error when command is not specified

* tests: adjust tests/func/plots/test_plots.py

* tests: adjust tests/func/plots/test_diff.py

* tests: add command to metrics/plots/params

* tests: fix test_remote.py

* tests: fix test_ls.py

* tests: fix test_repro* and test_run*

* tests: fix script in test_ls, was failing on Win
2020-05-27 12:13:26 +03:00
Peter Rowlands (변기호) 10f2bb93fa ignore: CleanTree should always check ignores, not just for walk() (#3876)
* tests: test that CleanTree works outside of walk()

* ignore: CleanTree should always check ignores, not just for walk()

* CleanTree: handle broken symlinks

* handle case for linking from shared local cache dirs

* Fix windows relpath/different drive issue
2020-05-27 12:07:38 +03:00
Saugat PachhaiandRuslan Kuprieiev 2fecc66cb9 addressing: allow stage name addressing without ":" (#3842)
* addressing: allow stage name addressing without colon

* repo: reorganize imports

* refactor/reorganize collect_granular

* add tests for stage addressing

* fix ds and cc issues

* fix unpack error in test

* tests: stage string representations

* exception: add error message

Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.

* rename NoOutputOrStage NoOutputOrStageError

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-26 03:51:51 +03:00
Ruslan Kuprieiev b47998463f run-cache: don't forget to set wdir (#3873)
Fixes #3848
2020-05-25 20:41:36 +03:00
Peter Rowlands (변기호) d53b740f35 state: always use working trees (#3871)
* tests: add test case for push/pull with --all-commits

* state: always use working tree
2020-05-25 15:47:30 +03:00
Saugat Pachhai 9198fa7459 log: reduce log, make it readable/easier (#3834)
* log: streamline/make it readable

* Remove 'run' from log message

* fix args passing in imports

* tests: fix test

* run: Add quotes around stage name

* run: add quotes for run-cache restore log

* address @dmitry's comments
2020-05-25 05:33:16 +03:00
Ruslan Kuprieiev 6954efc6e6 dvc: bump to 1.0.0a6 2020-05-25 03:25:37 +03:00
Ruslan Kuprieiev d77af8196e optimize local status (#3867)
* remote: remove unpacked dir

We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.

* local: save us a stat call in is_protected

* local: speedup checksum_to_path ~x5.5

* tree: walk: don't use os.path.join

It is ~5.5 times slower than joining by hand.

* dvcignore: optimize matching x10

On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.

To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.

It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
2020-05-25 03:20:11 +03:00
karajan1001 c5920b00ab Auto-completion of the dvc remote rename command (#3868)
fix #3847
2020-05-24 20:41:35 +03:00
Casper da Costa-Luis ce8036e535 snap: release non-stable-semver tags to beta (#3865)
* snap: release non-stable-semver tags to beta

- fixes #3860

* snap: document --beta
2020-05-24 20:40:53 +03:00
Ruslan Kuprieiev b7ad2dc841 test: fix hdfs/azurite/oss tests (#3864)
* travis: fix typo

* tests: azure: fix expected etag format

* test: ssh: increase the number of tries

* travis: don't install docker

* dvc: update pyarrow

* tests: ssh: don't leak connections

* remote: don't fail when cache/remote doesn't have prefix subdir
2020-05-23 23:46:40 +03:00
Ruslan Kuprieiev 5ab948f8b6 tests: unit: azure: add etag test (#3862) 2020-05-23 19:38:38 +03:00
Steffan Sluis f9fbe97509 azure: support external file dependencies (#3852)
* azure: support external dependencies and outputs

- Implement get_file_checksum with get_etag
- Implement a DependencyAzure and OutputAzure classes
- Add AzureOutput to OUTS, OUTS_MAP and CHECKSUMS_SCHEMA
- Add AzureDependency to DEPS and DEPS_MAP
- Add unit tests for AzureOutput and AzureDependency

Fixes #3540.

* Use content_md5 instead of etag

The etag is different for the same file in different locations.

* Use etag and remove external outputs for Azure

Due to inconsistencies with Azure's Content-MD5 property, it is not
stable enough to be used at the moment. Using the etag offers enough
reliability to support external dependencies. See also:
https://discordapp.com/channels/485586884165107732/565699007037571084/713462619469643796
2020-05-23 18:53:52 +03:00
Ruslan Kuprieiev 2fba2e030a dvcfile: serialize params on --no-exec (#3859)
* dvc.yaml: don't generate lockfile on --no-exec

* dvcfile: serialize params on --no-exec

Fixes #3851
2020-05-23 03:19:38 +03:00
Ruslan Kuprieiev 3e951162b8 dvc: bump to 1.0.0a4 2020-05-22 23:00:24 +03:00
Paweł Redzyński d74358eecd remote: local: protect uploaded files (#3858) 2020-05-22 22:59:48 +03:00
Ruslan Kuprieiev 58c9ff0de2 azure: drop obsoleted url format (#3856) 2020-05-22 20:36:04 +03:00
Paweł Redzyński f0e991790e plot: fixup for template help message (#3854) 2020-05-22 20:30:02 +03:00
Peter Rowlands (변기호) b7cd785eba external_repo: use RepoTree for fetch/import/get (#3841)
* erepo: load external repos directly from git tree

* erepo: add fetch_external

* remote/RepoTree: add callback for keeping track of download counts when
saving tree to cache

* repo: use erepo.fetch_external() in repo.fetch()

* erepo: return cache save_infos in fetch_external

* DependencyRepo: use erepo.fetch_external and cache.checkout instead of
pull_to

* erepo: remove pull_to

* DvcTree: support granularity for isdir()

* DvcTree: fix indentation bug

* RepoTree: add copytree() method

* LocalRemote: use working tree for local cache walk_files

* repo: state should be noop for GitTree based erepos

* erepo: add get_external()

* test_get: remove unneeded caplog check

* erepo: add get_rev()

* tests: fix erepo test bug

* ls: use RepoTree

* remote: support trees in _collect_dir

* remote: support trees in get_dir_checksum

* RepoTree: fix dir_cache relpaths in windows

* remote: fix duplicated fetch bug

* utils: don't skip md5 dos2unix check for git blobs

* otherwise md5 checksums will not match when comparing git version to
  cached filesystem version on windows
2020-05-22 17:23:31 +03:00
Ruslan Kuprieiev 21a3efe2c7 dvc: add: set top level md5 to None (#3846)
This way it won't be included into produced dvcfiles (we don't need it
for `dvc add`).
2020-05-22 17:21:43 +03:00
Ruslan Kuprieiev 865d355554 params/metrics: introduce --no-path and --old (#3845)
* metrics: diff: add --no-path

Part of #3690

* params: diff: introduce --no-path

* metrics: diff: introduce --old

Part of #3690
2020-05-22 04:35:42 +03:00
Ruslan Kuprieiev 84712c4acc dvc: don't write default values for cache/persist/metric (#3844) 2020-05-22 02:58:26 +03:00
Ruslan Kuprieiev 0f16e15453 test: add: fix typo 2020-05-22 01:11:45 +03:00
61f30ed5ba plots: point template help info to manual (#3830)
* plots: point template help info to manual

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
2020-05-22 00:53:04 +03:00
Ruslan Kuprieiev cde15e3d3e run-cache: don't forget to account for params (#3843)
Fixes #3827
2020-05-22 00:52:21 +03:00
Ruslan Kuprieiev 7672e324d2 output: base: use Git instead of git
https://github.com/iterative/dvc/pull/3808#discussion_r428853696
2020-05-21 22:44:39 +03:00
Ruslan Kuprieiev 130b76c7a5 plots: remove --select CLI option (#3840)
* plots: remove --select from CLI

It needs reconsideration, we can't release it as is right now. E.g. it
expects xpath to start with `$`, but that will get interpreted by your
shell, which is bad.

* completion: update plots
2020-05-21 17:57:27 +03:00
Puneetha PaiandPuneetha Pai 040458a726 (OutputAlreadyTrackedError): Add git suggestions to stop tracking files (#3808)
Fixes: #3678
When file has been already tracked under git, OutputAlreadyTrackedError gets raised.
This commits add git command suggestions to help user to stop tracking file from git,
so in future commits they can use DVC for the file.

Co-authored-by: Puneetha Pai <[email protected]>
2020-05-21 15:17:42 +03:00
Peter Rowlands (변기호) 75e795c491 remote: support saving RepoTree objects directly to cache (#3825)
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache

* RepoTree: support filtering out dvcfiles during walk()

* RepoTree: support recursive isdvc()

* remote: support computing checksums for tree objects

* RepoTree: pull dir contents on walk() if in `fetch` mode

* tests: add func test for DVC dir pull on RepoTree.walk()

* DvcTree: support granular open()

* RepoTree: add get_file_checksum()

* remote: revert checksum changes

* remote: add tree support for cache.save()

* tests: test saving directly to cache from RepoTree

* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree

* RepoTree: state context should be entered by open()/walk() caller

* RepoTree: only pull dir contents when cache has changed

* tree: update tree tests

* fix ds warnings, update comments

* fix windows tests
2020-05-21 06:31:28 +03:00
Saugat Pachhai 4dac2092c9 repro: fix help message for default target (#3831) 2020-05-21 00:17:30 +03:00
Shankho Boron GhoshandRestyled.io 19eaced304 dvc : renamed --outs-only to --dvc-only (#3797)
* Renamed --outs-only to --dvc-only

* Changed outs_only to dvc_only

* replace output_only with dvc_only

* Restyled by black

Co-authored-by: Restyled.io <[email protected]>
2020-05-20 18:24:16 +03:00
Casper da Costa-Luis 184028f176 logger: add trace, count --verbose/--quiet (#3806)
* logger: initial verbosity/quiet counts

Fixes #2332

* actually use TRACE

* logger: properly patch in trace

* FATAL -> CRITICAL

* only change log level if not info

Makes testing easier

* tests: logger: fix and update

* state: logger: debug => trace

* logger: filter duplicate debug messages
2020-05-20 18:23:31 +03:00
Ruslan Kuprieiev 5f63d538d2 dvc: bump to 1.0.0a3 2020-05-20 18:01:24 +03:00
Ruslan Kuprieiev 324ff4be59 remote: is_dir_checksum: handle empty/None checksum (#3836)
Fixed #3434
2020-05-20 18:00:41 +03:00
karajan1001andPeter Rowlands 3f846a1794 Add a new rename command (#3794)
* fixed #3599

1. add new command `dvc remote rename <old> <new>`.
2. add tests for this new command.

* Update dvc/command/remote.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Change about code review

* For deepsource

* Update dvc/command/remote.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2020-05-20 17:53:25 +03:00
Paweł Redzyński b52c925d5f lock: make froze message clickable (#3833) 2020-05-20 16:05:09 +03:00
Ruslan Kuprieiev 4a4f07968c dvc: bump to 1.0.0a2 2020-05-20 15:21:07 +03:00
Ruslan Kuprieiev 09136f2378 plots: add plot markers to DVC files (#3807)
* run: add --plot/--plot-no-cache

Part of #3409

* plot: move show/diff to API

* plots: show marked plots by default

Involves some refactoring to make plots comply with the rest of the code
base.

* dvc.yaml: use top-level metrics and plots

* dvc.yaml: add support for plot template

* plots: add support for templates in dvc.yaml

* dvc.yaml: allow metrics/plots to have persist/cache

* plots: render single page by-default and introduce --show-json

* plot: use "working tree" instead of "workspace"

Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.

* plots: rename -f|--file to -o|--out

Better corresponds to other commands like `import`.

* plots: use RepoTree and brancher instead of api

* stage: get rid of OutputParams

* tests: plots: add simple diff test

* plots: use `--targets` instead of `--datafile`

Makes it comply with `dvc metrics/params` CLI options.

* tests: unit: plots: add --show-json test

* stage: cleanup exception for format error

Based on feedback from the users.

* lockfile: use relpath in corruption error
2020-05-20 04:48:53 +03:00
nik123 aaca1401ea Fixes #3814 (#3828) 2020-05-20 04:14:04 +03:00
Saugat Pachhai 2aaf065ea3 default target: reduce loglevel to debug (#3822) 2020-05-19 16:35:13 +03:00
Saugat Pachhai d4ec871a76 repro: do not log when stage is locked and is repo import (#3823) 2020-05-19 16:23:28 +03:00
Paweł Redzyński f44ddcf64e plot: dont zero y axis (#3813) 2020-05-19 16:20:21 +03:00
Peter Rowlands (변기호) b88b3b81c6 utils: support use of (GitTree) tree file objects in utils (#3819)
* GitTree: add stat() function

* BaseTree: make walk_files a base tree method

* WorkingTree: add stat() method (wrapper for os.stat)

* utils.fs: support non-working tree for get_mtime_and_size

* utils.fs: add copy_fobj_to_file

* utils: support using tree.open in file_md5
2020-05-19 16:20:04 +03:00
Ruslan Kuprieiev 81d04f887a stage: fix commit (#3816) 2020-05-19 03:30:52 +03:00
Peter Rowlands (변기호) afbb3a0add repo: Support streaming and pulling files on RepoTree/DvcTree.open() (#3810)
* DvcTree: support stream or fetch on open()

* tests: test DvcTree dir cache support for walk()

* RepoTree: finish implementation of BaseTree functions

* repo: use RepoTree/DvcTree.open() for repo.open_by_relpath

* repo: only use RepoTree when needed for now

* fix docstring

* RepoTree: only use DvcTree.open for DVC outs
2020-05-18 15:16:11 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Saugat Pachhai 9897f7c470 test: metric/cache/persist keys (#3785) 2020-05-15 13:58:50 +03:00
Saugat Pachhai cf7f125c14 refactor: simplify checkouts and run, get rid of check_can_commit (#3800)
* stage: split checkouts

* stage: simplify can check commit

* split shell executable expansion to separate function

* refactor changed_entries for better message on pipeline file

* fix commit msg entry for stage

* reduce complexity of run_stage

* simplify prompt for commit

* fixes

* refactor commit

* refactor run_stage

* commit refactor
2020-05-15 10:08:32 +05:45
Ruslan Kuprieiev cec4b4e003 dvc: rename plot to plots (#3802)
* dvc: rename plot to plots

Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.

Part of #3409

* completion: rename plot -> plots
2020-05-14 22:28:22 +03:00
Ruslan Kuprieiev ff4fcc937b stage: fix --dry-run (#3799)
* tests: unit: move stage tests into a separate dir

* stage: fix --dry-run

Part of #3777
2020-05-14 20:19:34 +05:45
Saugat Pachhai 1676a7a1cd stage: moving things around, refactor (#3793)
* stage: extract unnecessary methods out of Stage class

* stage: change naming of functions

* stage: extract dump equal to utils

* stage: extract computation of md5 to utils

* stage: extract command run related logic in run.py

* stage: reorganize run()

* stage: boolean cosmetic changes

* stage: reorder save() and extract some into func

* stage: refactor status() into smaller functions

* extract resolve_wdir into utils

* merge _changed() and changed()

* cosmetic changes

* extract dump to utils

* extract import updates/sync to imports.py

* remove meta from PipelineStage

* fix imports, params load

* make run_stage a function, fix mocks in tests

* use original cmd

* rollback imports on decorators

* fix ds and cc
2020-05-14 16:43:02 +03:00
Ruslan Kuprieiev 974cf097dd run/repro: rename "build cache" -> "run cache" (#3788)
Part of #3777
2020-05-14 02:27:18 +03:00
Ruslan Kuprieiev 597ab98d0e diff: hanle repo with no commits (#3784)
`git diff` doesn't throw errors, so we could do the same.

Fixes #3398
2020-05-14 02:14:37 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
Peter Rowlands (변기호) 3916d4fa07 tests: mark test_data_cloud::test_pull_git_imports as flaky (#3790) 2020-05-14 00:16:37 +09:00
Paweł Redzyński d81b56978f plot: add doctype at the beggining of html embedding (#3787) 2020-05-13 00:17:05 +03:00
Jorge Orpinel 1e7efffe42 plot: update some output strings (#3759)
* plot: update datafile desc.

* plot: link to cmd ref in output

* plot: update --stdout desc
per https://github.com/iterative/dvc/pull/3577#pullrequestreview-407036181

* plot: a few more output string updates
per https://github.com/iterative/dvc/pull/3759#pullrequestreview-409011291
and https://github.com/iterative/dvc.org/pull/1283
2020-05-12 16:52:06 +03:00
Saugat Pachhai cc16b28a26 diff: autocompletion script for params, and show-md (#3786) 2020-05-12 16:41:09 +03:00
Paweł Redzyński 451c7aac04 tests: erepo_dir: extract remote setup to separate method (#3748) 2020-05-12 02:05:22 +03:00
Ruslan Kuprieiev 7b4466c982 run-cache: support uncached files (#3778)
Currently we only restore `--outs`, but `--outs-no-cache` are also
useful, especially when they are metrics files
(`-M|--metrics-no-cache`). So this PR makes run-cache save uncached
files to our cache and also tries to checkout them back during the
`restore` phase.
2020-05-12 02:02:53 +03:00
Jack WellsandRuslan Kuprieiev 7ca3a1119c Added S3 SSEKMSKeyId config option (#3762)
* Added S3 SSEKMSKeyId config option

* test: unit: s3: add SSEKMSKeyId test

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-11 22:52:55 +03:00
Saugat Pachhai 2c026066db pipeline file: disallow punctuation characters in stage name (#3767)
except hyphen and underscore
2020-05-11 19:11:47 +03:00
nik123 7f17c6aa03 refactor: check if targets is instances of str (#3779) 2020-05-11 19:10:14 +03:00
Ned Letcher 9f015fac1d import: check erepo is None before modifying url - fixes #3780 (#3781)
* import: check erepo is None before modifying url - fixes #3780

* import: add test to cover this local import
2020-05-11 18:29:35 +03:00
Saugat PachhaiandRuslan Kuprieiev 666e203bbb params/metrics: diff: implement tabulated markdown output (--show-md) (#3757)
* params/metrics: diff: implement tabulated markdown output

Via `--show-md` on `diff`s.

* table: show None on missing val

* tests: use `dedent`, add tests for markdown

* diff: add --show-md

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-10 18:39:51 +03:00
Karthikeyan Singaravelan ce946d1160 Import ABC from collections.abc for Python 3 compatibility. (#3772) 2020-05-10 10:49:32 +05:45
karajan1001 76b31ac282 fix #3588 (#3770)
* fix #3588

remove improper edges from output DAG.

* less code modification

* deepsource required

* deepsource required

* refactoring for reducing cognitive complexity.
2020-05-09 21:00:46 +03:00
6093a1fe97 Dvc remote default in list validation (#3715)
* fix #3470

Add validation to restrict default remote repo in list of remote repos.

* fix #3470

1. add two tests
2. add one validation
3. modified remote remove to satithe validation change

* Pass DeepSource

* deepsource required

* Redesigned.

* fix typo

* Revert "deepsource required"

This reverts commit 1ae458c34f49e8d7b35c39c1e6c592b423493a97.

* Revert "Pass DeepSource"

This reverts commit b254d104df65295e0a1156f3979b365be9d38a3d.

Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-09 20:59:17 +03:00
Ruslan Kuprieiev d62a54c7b0 push/pull: properly collect run cache (#3768)
Fixes #1234
2020-05-08 23:34:35 +03:00
nik123 c922da4f3f update: --recursive flag (#3760)
* update: --recursive flag

* refactor: applied isort changes to dvc/repo/update.py

* update: removed default values, check for str

* update: fixed code climate cognitive complexity issues
2020-05-08 23:23:48 +03:00
Ruslan Kuprieiev 2a16b021ad stage: hide unwanted warnings (#3763)
These messages all correpond to expected behaviour, so there is no need
to polute the output with unnecessary warnings.
2020-05-08 15:52:09 +03:00
Ruslan Kuprieiev fc42ca721c dvc: bump to 1.0.0a1 2020-05-07 19:33:30 +03:00
Ruslan Kuprieiev 785dcfeef0 push/pull: naively transfer run cache (#3746)
This is the most simple (aka dumb) implementation for this
functionality that is needed to unblock CICD development.
2020-05-07 16:55:10 +03:00
Paweł Redzyński 9714b14456 plot: fix help messages (#3734) 2020-05-07 13:22:55 +03:00
Ivan Shcheklein 6a406abde8 trying to enable the sponsorship button 2020-05-06 21:09:45 -07:00
Ivan Shcheklein 7ca055e868 fix: bump PyDrive2 to fix Gdrive remote download 2020-05-06 20:11:33 -07:00
Ruslan Kuprieiev 6a64f86178 pre-commit: use isort seeder 2020-05-07 00:25:41 +03:00
Ruslan Kuprieiev 227dbf44d9 isort: fix config 2020-05-07 00:17:38 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Saugat Pachhai e3831066de repro: use dvc.yaml by default (#3747)
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
2020-05-06 18:33:42 +03:00
Paweł Redzyński aa99cc9934 plot: check field exists (#3742) 2020-05-06 18:33:10 +03:00
Casper da Costa-Luis 26e203242f gdrive: download: stream & add progress (#3722)
* wip

* gdrive: add progress

Part of #2865
See https://github.com/iterative/dvc/issues/2865#issuecomment-622984785

* gdrive: move towards next pydrive2 release

- depends on https://github.com/iterative/PyDrive2/pull/30

* update to latest pydrive>=1.4.11

* avoid unneeded API call

* progress: gdrive: ensure proper bar_format
2020-05-06 17:02:16 +03:00
Saugat Pachhai d00bd4dfe5 pull/fetch/push: add stats (#3735)
* pull/fetch/push: add stats

* fix typo: s/logging/logger

* pull: make stats detailed, with summary at the bottom

* tests: mock properly, set return_value

* add a line space
2020-05-05 16:10:50 +03:00
Casper da Costa-Luis 795f001929 Snap version (#3741)
* snap: potential build fix

* snap: fix travis
2020-05-05 12:48:19 +03:00
304 changed files with 12409 additions and 8009 deletions
-7
View File
@@ -1,7 +0,0 @@
[flake8]
ignore =
E203, # Whitespace before ':'
E266, # Too many leading '#' for block comment
W503, # Line break occurred before a binary operator
max-line-length = 79
select = B,C,E,F,W,T4,B9
+3
View File
@@ -0,0 +1,3 @@
# These are supported funding model platforms
patreon: DVCorg
+13 -5
View File
@@ -1,10 +1,18 @@
---
name: Bug report
about: Create a report to help us improve
name: "\U0001F41B Bug Report"
about: Create a bug report to help us improve DVC
---
## Bug Report
**Please provide information about your setup**
DVC version(i.e. `dvc --version`), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))
### Please provide information about your setup
**Output of `dvc version`:**
```console
$ dvc version
```
**Additional Information (if any):**
If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
+5
View File
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: "🤗 Need help?"
url: https://dvc.org/chat
about: If you have a question, ask us on Discord. Please join with this invite 👉
+1 -2
View File
@@ -1,6 +1,5 @@
---
name: Feature request
name: "\U0001F680 Feature Request"
about: Suggest an idea for this project
---
+35 -7
View File
@@ -4,6 +4,15 @@ repos:
language_version: python3
repo: https://github.com/ambv/black
rev: 19.10b0
- repo: https://github.com/asottile/seed-isort-config
rev: v2.1.1
hooks:
- id: seed-isort-config
- hooks:
- id: isort
language_version: python3
repo: https://github.com/timothycrosley/isort
rev: 4.3.21
- hooks:
- id: flake8
language_version: python3
@@ -18,18 +27,37 @@ repos:
repo: https://github.com/lovesegfault/beautysh
rev: master
- hooks:
- id: dvc-pre-commit
language_version: python3
- args:
- git-hook
- pre-commit
entry: dvc
id: dvc-pre-commit
language: system
name: DVC pre-commit
stages:
- commit
- id: dvc-pre-push
language_version: python3
verbose: true
require_serial: true
- args:
- git-hook
- pre-push
entry: dvc
id: dvc-pre-push
language: system
name: DVC pre-push
stages:
- push
require_serial: true
- always_run: true
args:
- git-hook
- post-checkout
entry: dvc
id: dvc-post-checkout
language_version: python3
language: system
minimum_pre_commit_version: 2.2.0
name: DVC post-checkout
stages:
- post-checkout
repo: https://github.com/iterative/dvc
rev: master
require_serial: true
repo: local
+3
View File
@@ -6,6 +6,7 @@
language: python
language_version: python3
name: DVC pre-commit
require_serial: true
stages:
- commit
verbose: true
@@ -17,6 +18,7 @@
language: python
language_version: python3
name: DVC pre-push
require_serial: true
stages:
- push
- always_run: true
@@ -29,5 +31,6 @@
language_version: python3
minimum_pre_commit_version: 2.2.0
name: DVC post-checkout
require_serial: true
stages:
- post-checkout
+9
View File
@@ -9,3 +9,12 @@ restylers:
- "**/*.py"
interpreters:
- python
- name: isort
image: restyled/restyler-isort:v4.3.21
command:
- isort
arguments: []
include:
- "**/*.py"
interpreters:
- python
+14 -5
View File
@@ -46,9 +46,6 @@ jobs:
env:
- PYTHON_VERSION=3.8.0
- PATH=/c/Python38:/c/Python38/Scripts:$PATH
- name: "3.5 on Linux"
language: python
python: 3.5
- name: "3.6 on Linux"
language: python
python: 3.6
@@ -101,8 +98,12 @@ jobs:
- name: lxd
channel: stable
env:
- SNAPCRAFT_IMAGE_INFO: '{"build_url": "$TRAVIS_BUILD_URL"}'
- SNAPCRAFT_IMAGE_INFO: |
'{"build_url": "$TRAVIS_JOB_WEB_URL"}'
- SNAPCRAFT_BUILD_ENVIRONMENT: lxd
- SNAPCRAFT_BUILD_INFO: 1 # https://snapcraft.io/blog/introducing-developer-notifications-for-snap-security-updates
install:
- sudo usermod --append --groups lxd $USER
- sudo /snap/bin/lxd.migrate -yes
- sudo /snap/bin/lxd waitready
- sudo /snap/bin/lxd init --auto
@@ -156,6 +157,14 @@ deploy:
script: "(echo $SNAP_TOKEN | snapcraft login --with -) && timeout 600 snapcraft push dvc_*.snap --release $SNAP_CHANNEL || echo timed out"
on:
all_branches: true
condition: "$(./scripts/ci/deploy_condition.sh dvc_*.snap) && ($TRAVIS_BRANCH = master || -n $TRAVIS_TAG)"
condition: "$(./scripts/ci/deploy_condition.sh dvc_*.snap) && ($TRAVIS_BRANCH = master || -n $TRAVIS_TAG) && -n $SNAP_CHANNEL"
repo: iterative/dvc
stage: build
- provider: script
skip_cleanup: true
script: "(echo $SNAP_TOKEN | snapcraft login --with -) && timeout 600 snapcraft push dvc_*.snap --release $SNAP_CHANNEL_MAJOR || echo timed out"
on:
all_branches: true
condition: "$(./scripts/ci/deploy_condition.sh dvc_*.snap) && ($TRAVIS_BRANCH = master || -n $TRAVIS_TAG) && -n $SNAP_CHANNEL_MAJOR"
repo: iterative/dvc
stage: build
+2 -1
View File
@@ -101,7 +101,8 @@ Snap (Snapcraft/Linux)
snap install dvc --classic
This corresponds to the latest tagged release.
Add ``--edge`` for the latest ``master`` version.
Add ``--beta`` for the latest tagged release candidate,
or ``--edge`` for the latest ``master`` version.
Choco (Chocolatey/Windows)
--------------------------
-1
View File
@@ -6,5 +6,4 @@ Make your data science projects reproducible and shareable.
import dvc.logger
from dvc.version import __version__ # noqa: F401
dvc.logger.setup()
+4 -7
View File
@@ -2,12 +2,12 @@ import json
import logging
import os
import platform
import requests
import sys
import tempfile
import uuid
import distro
import requests
from dvc import __version__
from dvc.config import Config, to_bool
@@ -20,7 +20,6 @@ from dvc.scm.base import SCMError
from dvc.utils import env2bool, is_binary
from dvc.utils.fs import makedirs
logger = logging.getLogger(__name__)
@@ -73,7 +72,7 @@ def send(path):
url = "https://analytics.dvc.org"
headers = {"content-type": "application/json"}
with open(path, "r") as fobj:
with open(path) as fobj:
report = json.load(fobj)
report.update(_runtime_info())
@@ -159,7 +158,7 @@ def _find_or_create_user_id():
try:
with Lock(lockfile):
try:
with open(fname, "r") as fobj:
with open(fname) as fobj:
user_id = json.load(fobj)["user_id"]
except (FileNotFoundError, ValueError, KeyError):
@@ -171,6 +170,4 @@ def _find_or_create_user_id():
return user_id
except LockError:
logger.debug(
"Failed to acquire '{lockfile}'".format(lockfile=lockfile)
)
logger.debug(f"Failed to acquire '{lockfile}'")
+5 -4
View File
@@ -1,16 +1,17 @@
import os
from contextlib import contextmanager, _GeneratorContextManager as GCM
from contextlib import _GeneratorContextManager as GCM
from contextlib import contextmanager
from dvc.repo import Repo
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.external_repo import external_repo
from dvc.repo import Repo
class UrlNotDvcRepoError(DvcException):
"""Thrown if the given URL is not a DVC repository."""
def __init__(self, url):
super().__init__("'{}' is not a DVC repository.".format(url))
super().__init__(f"'{url}' is not a DVC repository.")
def get_url(path, repo=None, rev=None, remote=None):
@@ -29,7 +30,7 @@ def get_url(path, repo=None, rev=None, remote=None):
raise UrlNotDvcRepoError(_repo.url)
out = _repo.find_out_by_relpath(path)
remote_obj = _repo.cloud.get_remote(remote)
return str(remote_obj.checksum_to_path_info(out.checksum))
return str(remote_obj.hash_to_path_info(out.checksum))
def open(path, repo=None, rev=None, remote=None, mode="r", encoding=None):
+10 -6
View File
@@ -28,19 +28,21 @@ def _make_remote_property(name):
"""
def getter(self):
from dvc.remote import Remote
from dvc.remote import get_cloud_tree
from dvc.remote.base import CloudCache
remote = self.config.get(name)
if not remote:
return None
return Remote(self.repo, name=remote)
tree = get_cloud_tree(self.repo, name=remote)
return CloudCache(tree)
getter.__name__ = name
return cached_property(getter)
class Cache(object):
class Cache:
"""Class that manages cache locations of a DVC repo.
Args:
@@ -50,7 +52,8 @@ class Cache(object):
CACHE_DIR = "cache"
def __init__(self, repo):
from dvc.remote import Remote
from dvc.remote import get_cloud_tree
from dvc.remote.local import LocalCache
self.repo = repo
self.config = config = repo.config["cache"]
@@ -62,7 +65,8 @@ class Cache(object):
else:
settings = {**config, "url": config["dir"]}
self.local = Remote(repo, **settings)
tree = get_cloud_tree(repo, **settings)
self.local = LocalCache(tree)
s3 = _make_remote_property("s3")
gs = _make_remote_property("gs")
@@ -101,7 +105,7 @@ class NamedCacheItem:
self.children[checksum].update(child_item)
class NamedCache(object):
class NamedCache:
def __init__(self):
self._items = defaultdict(lambda: defaultdict(NamedCacheItem))
self.external = defaultdict(set)
+27 -17
View File
@@ -10,22 +10,24 @@ from .command import (
commit,
config,
daemon,
dag,
data_sync,
destroy,
diff,
freeze,
gc,
get,
get_url,
git_hook,
imp,
imp_url,
init,
install,
ls,
lock,
metrics,
params,
move,
pipeline,
params,
plots,
remote,
remove,
repro,
@@ -34,13 +36,10 @@ from .command import (
unprotect,
update,
version,
git_hook,
plot,
)
from .command.base import fix_subparsers
from .exceptions import DvcParserError
logger = logging.getLogger(__name__)
COMMANDS = [
@@ -67,33 +66,48 @@ COMMANDS = [
install,
root,
ls,
lock,
pipeline,
freeze,
dag,
daemon,
commit,
diff,
version,
update,
git_hook,
plot,
plots,
]
class DvcParser(argparse.ArgumentParser):
"""Custom parser class for dvc CLI."""
def error(self, message):
def error(self, message, command=None):
"""Custom error method.
Args:
message (str): error message.
command (str): subcommand name for help message
Raises:
dvc.exceptions.DvcParser: dvc parser exception.
"""
logger.error(message)
if command is not None:
for action in self._actions:
if action.dest == "cmd" and command in action.choices:
subparser = action.choices[command]
subparser.print_help()
raise DvcParserError()
self.print_help()
raise DvcParserError()
# override this to send subcommand name to error method
def parse_args(self, args=None, namespace=None):
args, argv = self.parse_known_args(args, namespace)
if argv:
msg = "unrecognized arguments: %s"
self.error(msg % " ".join(argv), args.cmd)
return args
class VersionAction(argparse.Action): # pragma: no cover
# pylint: disable=too-few-public-methods
@@ -117,14 +131,10 @@ def get_parent_parser():
log_level_group = parent_parser.add_mutually_exclusive_group()
log_level_group.add_argument(
"-q", "--quiet", action="store_true", default=False, help="Be quiet."
"-q", "--quiet", action="count", default=0, help="Be quiet."
)
log_level_group.add_argument(
"-v",
"--verbose",
action="store_true",
default=False,
help="Be verbose.",
"-v", "--verbose", action="count", default=0, help="Be verbose."
)
return parent_parser
+8 -2
View File
@@ -1,7 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link, CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException, RecursiveAddingWhileUsingFilename
logger = logging.getLogger(__name__)
@@ -18,6 +18,7 @@ class CmdAdd(CmdBase):
recursive=self.args.recursive,
no_commit=self.args.no_commit,
fname=self.args.file,
external=self.args.external,
)
except DvcException:
@@ -50,7 +51,12 @@ def add_parser(subparsers, parent_parser):
help="Don't put files/directories into cache.",
)
parser.add_argument(
"-f",
"--external",
action="store_true",
default=False,
help="Allow targets that are outside of the DVC repository.",
)
parser.add_argument(
"--file",
help="Specify name of the DVC-file this command will generate.",
metavar="<filename>",
+5 -7
View File
@@ -1,6 +1,5 @@
import logging
logger = logging.getLogger(__name__)
@@ -27,7 +26,7 @@ def append_doc_link(help_message, path):
)
class CmdBase(object):
class CmdBase:
def __init__(self, args):
from dvc.repo import Repo
from dvc.updater import Updater
@@ -41,12 +40,11 @@ class CmdBase(object):
@property
def default_targets(self):
"""Default targets for `dvc repro` and `dvc pipeline`."""
from dvc.dvcfile import DVC_FILE
"""Default targets for `dvc repro`."""
from dvc.dvcfile import PIPELINE_FILE
msg = "assuming default target '{}'.".format(DVC_FILE)
logger.warning(msg)
return [DVC_FILE]
logger.debug(f"assuming default target '{PIPELINE_FILE}'.")
return [PIPELINE_FILE]
# Abstract methods that have to be implemented by any inheritance class
def run(self):
+8 -27
View File
@@ -1,39 +1,16 @@
import argparse
import logging
import operator
import colorama
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import CheckoutError
from dvc.utils.humanize import get_summary
logger = logging.getLogger(__name__)
def _human_join(words):
words = list(words)
if not words:
return ""
return (
"{before} and {after}".format(
before=", ".join(words[:-1]), after=words[-1],
)
if len(words) > 1
else words[0]
)
def log_summary(stats):
states = ("added", "deleted", "modified")
summary = (
"{} {}".format(len(stats[state]), state)
for state in states
if stats.get(state)
)
logger.info(_human_join(summary) or "No changes.")
def log_changes(stats):
colors = [
("modified", colorama.Fore.YELLOW,),
@@ -75,7 +52,11 @@ class CmdCheckout(CmdBase):
stats = exc.stats
if self.args.summary:
log_summary(stats)
default_message = "No changes."
msg = get_summary(
sorted(stats.items(), key=operator.itemgetter(0))
)
logger.info(msg or default_message)
else:
log_changes(stats)
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+5 -4
View File
@@ -1,7 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link, CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.config import Config, ConfigError
logger = logging.getLogger(__name__)
@@ -50,21 +50,22 @@ class CmdConfig(CmdBaseNoRepo):
parent_config_parser = argparse.ArgumentParser(add_help=False)
parent_config_parser.add_argument(
level_group = parent_config_parser.add_mutually_exclusive_group()
level_group.add_argument(
"--global",
dest="level",
action="store_const",
const="global",
help="Use global config.",
)
parent_config_parser.add_argument(
level_group.add_argument(
"--system",
dest="level",
action="store_const",
const="system",
help="Use system config.",
)
parent_config_parser.add_argument(
level_group.add_argument(
"--local",
dest="level",
action="store_const",
+1 -2
View File
@@ -1,5 +1,4 @@
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import fix_subparsers
from dvc.command.base import CmdBaseNoRepo, fix_subparsers
class CmdDaemonBase(CmdBaseNoRepo):
+115
View File
@@ -0,0 +1,115 @@
import argparse
import logging
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
def _show_ascii(G):
from dvc.repo.graph import get_pipelines
from dvc.dagascii import draw
pipelines = get_pipelines(G)
ret = []
for pipeline in pipelines:
ret.append(draw(pipeline.nodes, pipeline.edges))
return "\n".join(ret)
def _show_dot(G):
import io
from networkx.drawing.nx_pydot import write_dot
dot_file = io.StringIO()
write_dot(G, dot_file)
return dot_file.getvalue()
def _build(G, target=None, full=False):
import networkx as nx
from dvc.repo.graph import get_pipeline, get_pipelines
if target:
H = get_pipeline(get_pipelines(G), target)
if not full:
descendants = nx.descendants(G, target)
descendants.add(target)
H.remove_nodes_from(set(G.nodes()) - descendants)
else:
H = G
def _relabel(stage):
return stage.addressing
return nx.relabel_nodes(H, _relabel, copy=False)
class CmdDAG(CmdBase):
def run(self):
try:
target = None
if self.args.target:
stages = self.repo.collect(self.args.target)
if len(stages) > 1:
logger.error(
f"'{self.args.target}' contains more than one stage "
"{stages}, please specify one stage"
)
return 1
target = stages[0]
G = _build(self.repo.graph, target=target, full=self.args.full,)
if self.args.dot:
logger.info(_show_dot(G))
else:
from dvc.utils.pager import pager
pager(_show_ascii(G))
return 0
except DvcException:
msg = "failed to show "
if self.args.target:
msg += f"a pipeline for '{target}'"
else:
msg += "pipelines"
logger.exception(msg)
return 1
def add_parser(subparsers, parent_parser):
DAG_HELP = "Visualize DVC project DAG."
dag_parser = subparsers.add_parser(
"dag",
parents=[parent_parser],
description=append_doc_link(DAG_HELP, "dag"),
help=DAG_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
dag_parser.add_argument(
"--dot",
action="store_true",
default=False,
help="Print DAG with .dot format.",
)
dag_parser.add_argument(
"--full",
action="store_true",
default=False,
help=(
"Show full DAG that the target belongs too, instead of "
"showing DAG consisting only of ancestors."
),
)
dag_parser.add_argument(
"target",
nargs="?",
help="Stage or output to show pipeline for (optional). "
"Finds all stages in the workspace by default.",
)
dag_parser.set_defaults(func=CmdDAG)
+46 -18
View File
@@ -1,23 +1,25 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.checkout import log_summary
from dvc.exceptions import DvcException, CheckoutError
from dvc.command.base import CmdBase, append_doc_link
from dvc.command.checkout import log_changes
from dvc.exceptions import CheckoutError, DvcException
from dvc.utils.humanize import get_summary
logger = logging.getLogger(__name__)
class CmdDataBase(CmdBase):
@classmethod
def check_up_to_date(cls, processed_files_count):
if processed_files_count == 0:
logger.info("Everything is up to date.")
def log_summary(self, stats):
default_msg = "Everything is up to date."
logger.info(get_summary(stats.items()) or default_msg)
class CmdDataPull(CmdDataBase):
def log_summary(self, stats):
log_changes(stats)
super().log_summary(stats)
def run(self):
try:
stats = self.repo.pull(
@@ -30,14 +32,14 @@ class CmdDataPull(CmdDataBase):
with_deps=self.args.with_deps,
force=self.args.force,
recursive=self.args.recursive,
run_cache=self.args.run_cache,
)
log_summary(stats)
self.log_summary(stats)
except (CheckoutError, DvcException) as exc:
log_summary(getattr(exc, "stats", {}))
self.log_summary(getattr(exc, "stats", {}))
logger.exception("failed to pull data from the cloud")
return 1
self.check_up_to_date(stats["downloaded"])
return 0
@@ -53,11 +55,12 @@ class CmdDataPush(CmdDataBase):
all_commits=self.args.all_commits,
with_deps=self.args.with_deps,
recursive=self.args.recursive,
run_cache=self.args.run_cache,
)
self.log_summary({"pushed": processed_files_count})
except DvcException:
logger.exception("failed to push data to the cloud")
return 1
self.check_up_to_date(processed_files_count)
return 0
@@ -73,11 +76,12 @@ class CmdDataFetch(CmdDataBase):
all_commits=self.args.all_commits,
with_deps=self.args.with_deps,
recursive=self.args.recursive,
run_cache=self.args.run_cache,
)
self.log_summary({"fetched": processed_files_count})
except DvcException:
logger.exception("failed to fetch data from the cloud")
return 1
self.check_up_to_date(processed_files_count)
return 0
@@ -85,24 +89,24 @@ def shared_parent_parser():
from dvc.cli import get_parent_parser
# Parent parser used in pull/push/status
shared_parent_parser = argparse.ArgumentParser(
parent_parser = argparse.ArgumentParser(
add_help=False, parents=[get_parent_parser()]
)
shared_parent_parser.add_argument(
parent_parser.add_argument(
"-j",
"--jobs",
type=int,
help="Number of jobs to run simultaneously.",
metavar="<number>",
)
shared_parent_parser.add_argument(
parent_parser.add_argument(
"targets",
nargs="*",
help="Limit command scope to these DVC-files. "
"Using -R, directories to search DVC-files in can also be given.",
)
return shared_parent_parser
return parent_parser
def add_parser(subparsers, _parent_parser):
@@ -162,6 +166,12 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Pull cache for subdirectories of the specified directory.",
)
pull_parser.add_argument(
"--run-cache",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
pull_parser.set_defaults(func=CmdDataPull)
# Push
@@ -211,6 +221,12 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Push cache for subdirectories of specified directory.",
)
push_parser.add_argument(
"--run-cache",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
push_parser.set_defaults(func=CmdDataPush)
# Fetch
@@ -266,6 +282,12 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Fetch cache for subdirectories of specified directory.",
)
fetch_parser.add_argument(
"--run-cache",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
fetch_parser.set_defaults(func=CmdDataFetch)
# Status
@@ -341,5 +363,11 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Show status of all stages in the specified directory.",
)
status_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show status in JSON format.",
)
status_parser.set_defaults(func=CmdDataStatus)
+1 -3
View File
@@ -2,11 +2,9 @@ import argparse
import logging
import dvc.prompt as prompt
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+23 -1
View File
@@ -8,10 +8,24 @@ import colorama
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
def _show_md(diff):
from dvc.utils.diff import table
rows = []
for status in ["added", "deleted", "modified"]:
entries = diff.get(status, [])
if not entries:
continue
paths = sorted([entry["path"] for entry in entries])
for path in paths:
rows.append([status, path])
return table(["Status", "Path"], rows, True)
class CmdDiff(CmdBase):
@staticmethod
def _format(diff):
@@ -104,6 +118,8 @@ class CmdDiff(CmdBase):
if self.args.show_json:
logger.info(json.dumps(diff))
elif self.args.show_md:
logger.info(_show_md(diff))
elif diff:
logger.info(self._format(diff))
@@ -148,4 +164,10 @@ def add_parser(subparsers, parent_parser):
action="store_true",
default=False,
)
diff_parser.add_argument(
"--show-md",
help="Show tabulated output in the Markdown format (GFM).",
action="store_true",
default=False,
)
diff_parser.set_defaults(func=CmdDiff)
+57
View File
@@ -0,0 +1,57 @@
import argparse
import logging
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdFreezeBase(CmdBase):
def _run(self, func, name):
ret = 0
for target in self.args.targets:
try:
func(target)
except DvcException:
logger.exception(f"failed to {name} '{target}'")
ret = 1
return ret
class CmdFreeze(CmdFreezeBase):
def run(self):
return self._run(self.repo.freeze, "freeze")
class CmdUnfreeze(CmdFreezeBase):
def run(self):
return self._run(self.repo.unfreeze, "unfreeze")
def add_parser(subparsers, parent_parser):
FREEZE_HELP = "Freeze DVC-files."
freeze_parser = subparsers.add_parser(
"freeze",
parents=[parent_parser],
description=append_doc_link(FREEZE_HELP, "freeze"),
help=FREEZE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
freeze_parser.add_argument(
"targets", nargs="+", help="DVC-files to freeze."
)
freeze_parser.set_defaults(func=CmdFreeze)
UNFREEZE_HELP = "Unfreeze DVC-files."
unfreeze_parser = subparsers.add_parser(
"unfreeze",
parents=[parent_parser],
description=append_doc_link(UNFREEZE_HELP, "unfreeze"),
help=UNFREEZE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
unfreeze_parser.add_argument(
"targets", nargs="+", help="DVC-files to unfreeze."
)
unfreeze_parser.set_defaults(func=CmdUnfreeze)
+3 -5
View File
@@ -3,9 +3,7 @@ import logging
import os
import dvc.prompt as prompt
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
logger = logging.getLogger(__name__)
@@ -23,7 +21,7 @@ class CmdGC(CmdBase):
msg = "This will remove all cache except items used in "
msg += "the working tree"
msg += "the workspace"
if self.args.all_commits:
msg += " and all git commits"
elif self.args.all_branches and self.args.all_tags:
@@ -79,7 +77,7 @@ def add_parser(subparsers, parent_parser):
"--workspace",
action="store_true",
default=False,
help="Keep data files used in the current workspace (working tree).",
help="Keep data files used in the current workspace.",
)
gc_parser.add_argument(
"-a",
+1 -2
View File
@@ -1,10 +1,9 @@
import argparse
import logging
from .base import append_doc_link
from .base import CmdBaseNoRepo
from dvc.exceptions import DvcException
from .base import CmdBaseNoRepo, append_doc_link
logger = logging.getLogger(__name__)
+2 -3
View File
@@ -1,10 +1,9 @@
import argparse
import logging
from .base import append_doc_link
from .base import CmdBaseNoRepo
from dvc.exceptions import DvcException
from .base import CmdBaseNoRepo, append_doc_link
logger = logging.getLogger(__name__)
@@ -17,7 +16,7 @@ class CmdGetUrl(CmdBaseNoRepo):
Repo.get_url(self.args.url, out=self.args.out)
return 0
except DvcException:
logger.exception("failed to get '{}'".format(self.args.url))
logger.exception(f"failed to get '{self.args.url}'")
return 1
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+1 -4
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -59,7 +57,6 @@ def add_parser(subparsers, parent_parser):
"out", nargs="?", help="Destination path to put files to."
)
import_parser.add_argument(
"-f",
"--file",
help="Specify name of the DVC-file this command will generate.",
metavar="<filename>",
+1 -3
View File
@@ -1,9 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
logger = logging.getLogger(__name__)
+1 -3
View File
@@ -1,9 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
logger = logging.getLogger(__name__)
-61
View File
@@ -1,61 +0,0 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdLockBase(CmdBase):
def _run(self, unlock):
for target in self.args.targets:
try:
self.repo.lock_stage(target, unlock=unlock)
except DvcException:
logger.exception(
"failed to {}lock '{}'".format(
"un" if unlock else "", target
)
)
return 1
return 0
class CmdLock(CmdLockBase):
def run(self):
return self._run(False)
class CmdUnlock(CmdLockBase):
def run(self):
return self._run(True)
def add_parser(subparsers, parent_parser):
LOCK_HELP = "Lock DVC-files."
lock_parser = subparsers.add_parser(
"lock",
parents=[parent_parser],
description=append_doc_link(LOCK_HELP, "lock"),
help=LOCK_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
lock_parser.add_argument("targets", nargs="+", help="DVC-files to lock.")
lock_parser.set_defaults(func=CmdLock)
UNLOCK_HELP = "Unlock DVC-files."
unlock_parser = subparsers.add_parser(
"unlock",
parents=[parent_parser],
description=append_doc_link(UNLOCK_HELP, "unlock"),
help=UNLOCK_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
unlock_parser.add_argument(
"targets", nargs="+", help="DVC-files to unlock."
)
unlock_parser.set_defaults(func=CmdUnlock)
+4 -6
View File
@@ -2,12 +2,10 @@ import argparse
import logging
import sys
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.command.ls.ls_colors import LsColors
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -33,14 +31,14 @@ class CmdList(CmdBaseNoRepo):
self.args.path,
rev=self.args.rev,
recursive=self.args.recursive,
outs_only=self.args.outs_only,
dvc_only=self.args.dvc_only,
)
if entries:
entries = _prettify(entries, sys.stdout.isatty())
logger.info("\n".join(entries))
return 0
except DvcException:
logger.exception("failed to list '{}'".format(self.args.url))
logger.exception(f"failed to list '{self.args.url}'")
return 1
@@ -64,7 +62,7 @@ def add_parser(subparsers, parent_parser):
help="Recursively list files.",
)
list_parser.add_argument(
"--outs-only", action="store_true", help="Show only DVC outputs."
"--dvc-only", action="store_true", help="Show only DVC outputs."
)
list_parser.add_argument(
"--rev",
+2 -2
View File
@@ -1,7 +1,7 @@
import os
class LsColors(object):
class LsColors:
default = "rs=0:di=01;34:ex=01;32"
def __init__(self, lscolors=None):
@@ -45,4 +45,4 @@ class LsColors(object):
if not val:
return text
rs = self._codes.get("rs", 0)
return "\033[{}m{}\033[{}m".format(val, text, rs)
return f"\033[{val}m{text}\033[{rs}m"
+86 -79
View File
@@ -1,16 +1,15 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import fix_subparsers
from dvc.exceptions import BadMetricError
from dvc.exceptions import DvcException
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import BadMetricError, DvcException
logger = logging.getLogger(__name__)
DEFAULT_PRECISION = 5
def show_metrics(
metrics, all_branches=False, all_tags=False, all_commits=False
):
@@ -23,16 +22,16 @@ def show_metrics(
for branch, val in metrics.items():
if all_branches or all_tags or all_commits:
logger.info("{branch}:".format(branch=branch))
logger.info(f"{branch}:")
for fname, metric in val.items():
if not isinstance(metric, dict):
logger.info("\t{}: {}".format(fname, str(metric)))
continue
logger.info("\t{}:".format(fname))
logger.info(f"\t{fname}:")
for key, value in flatten(format_dict(metric), ".").items():
logger.info("\t\t{}: {}".format(key, value))
logger.info(f"\t\t{key}: {value}")
if missing:
raise BadMetricError(missing)
@@ -67,49 +66,41 @@ class CmdMetricsShow(CmdBase):
return 0
class CmdMetricsAdd(CmdBase):
def run(self):
try:
self.repo.metrics.add(self.args.path)
except DvcException:
msg = "failed to add metric file '{}'".format(self.args.path)
logger.exception(msg)
return 1
return 0
class CmdMetricsRemove(CmdBase):
def run(self):
try:
self.repo.metrics.remove(self.args.path)
except DvcException:
msg = "failed to remove metric file '{}'".format(self.args.path)
logger.exception(msg)
return 1
return 0
def _show_diff(diff):
def _show_diff(diff, markdown=False, no_path=False, old=False, precision=None):
from collections import OrderedDict
from dvc.utils.diff import table
if precision is None:
precision = DEFAULT_PRECISION
def _round(val):
if isinstance(val, float):
return round(val, precision)
return val
rows = []
for fname, mdiff in diff.items():
sorted_mdiff = OrderedDict(sorted(mdiff.items()))
for metric, change in sorted_mdiff.items():
rows.append(
[
fname,
metric,
change["new"],
change.get("diff", "diff not supported"),
]
)
row = [] if no_path else [fname]
row.append(metric)
if old:
row.append(_round(change.get("old")))
row.append(_round(change["new"]))
row.append(_round(change.get("diff", "diff not supported")))
rows.append(row)
return table(["Path", "Metric", "Value", "Change"], rows)
header = [] if no_path else ["Path"]
header.append("Metric")
if old:
header.extend(["Old", "New"])
else:
header.append("Value")
header.append("Change")
return table(header, rows, markdown)
class CmdMetricsDiff(CmdBase):
@@ -128,7 +119,13 @@ class CmdMetricsDiff(CmdBase):
logger.info(json.dumps(diff))
else:
table = _show_diff(diff)
table = _show_diff(
diff,
self.args.show_md,
self.args.no_path,
self.args.old,
precision=self.args.precision,
)
if table:
logger.info(table)
@@ -140,7 +137,7 @@ class CmdMetricsDiff(CmdBase):
def add_parser(subparsers, parent_parser):
METRICS_HELP = "Commands to add, manage, collect, and display metrics."
METRICS_HELP = "Commands to display and compare metrics."
metrics_parser = subparsers.add_parser(
"metrics",
@@ -157,17 +154,6 @@ def add_parser(subparsers, parent_parser):
fix_subparsers(metrics_subparsers)
METRICS_ADD_HELP = "Mark a DVC-tracked file as a metric."
metrics_add_parser = metrics_subparsers.add_parser(
"add",
parents=[parent_parser],
description=append_doc_link(METRICS_ADD_HELP, "metrics/add"),
help=METRICS_ADD_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
metrics_add_parser.add_argument("path", help="Path to a metric file.")
metrics_add_parser.set_defaults(func=CmdMetricsAdd)
METRICS_SHOW_HELP = "Print metrics, with optional formatting."
metrics_show_parser = metrics_subparsers.add_parser(
"show",
@@ -179,7 +165,10 @@ def add_parser(subparsers, parent_parser):
metrics_show_parser.add_argument(
"targets",
nargs="*",
help="Metric files or directories (see -R) to show",
help=(
"Limit command scope to these metric files. Using -R, "
"directories to search metric files in can also be given."
),
)
metrics_show_parser.add_argument(
"-a",
@@ -201,6 +190,12 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Show metrics for all commits.",
)
metrics_show_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show output in JSON format.",
)
metrics_show_parser.add_argument(
"-R",
"--recursive",
@@ -211,16 +206,12 @@ def add_parser(subparsers, parent_parser):
"metric files."
),
)
metrics_show_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show output in JSON format.",
)
metrics_show_parser.set_defaults(func=CmdMetricsShow)
METRICS_DIFF_HELP = "Show changes in metrics between commits"
" in the DVC repository, or between a commit and the workspace."
METRICS_DIFF_HELP = (
"Show changes in metrics between commits in the DVC repository, or "
"between a commit and the workspace."
)
metrics_diff_parser = metrics_subparsers.add_parser(
"diff",
parents=[parent_parser],
@@ -234,14 +225,14 @@ def add_parser(subparsers, parent_parser):
metrics_diff_parser.add_argument(
"b_rev",
nargs="?",
help=("New Git commit to compare (defaults to the current workspace)"),
help="New Git commit to compare (defaults to the current workspace)",
)
metrics_diff_parser.add_argument(
"--targets",
nargs="*",
help=(
"Metric files or directories (see -R) to show diff for. "
"Shows diff for all metric files by default."
"Limit command scope to these metric files. Using -R, "
"directories to search metric files in can also be given."
),
metavar="<paths>",
)
@@ -267,15 +258,31 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Show output in JSON format.",
)
metrics_diff_parser.set_defaults(func=CmdMetricsDiff)
METRICS_REMOVE_HELP = "Remove metric mark on a DVC-tracked file."
metrics_remove_parser = metrics_subparsers.add_parser(
"remove",
parents=[parent_parser],
description=append_doc_link(METRICS_REMOVE_HELP, "metrics/remove"),
help=METRICS_REMOVE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
metrics_diff_parser.add_argument(
"--show-md",
action="store_true",
default=False,
help="Show tabulated output in the Markdown format (GFM).",
)
metrics_remove_parser.add_argument("path", help="Path to a metric file.")
metrics_remove_parser.set_defaults(func=CmdMetricsRemove)
metrics_diff_parser.add_argument(
"--old",
action="store_true",
default=False,
help="Show old metric value.",
)
metrics_diff_parser.add_argument(
"--no-path",
action="store_true",
default=False,
help="Don't show metric path.",
)
metrics_diff_parser.add_argument(
"--precision",
type=int,
help=(
"Round metrics to `n` digits precision after the decimal point. "
f"Rounds to {DEFAULT_PRECISION} digits by default."
),
metavar="<n>",
)
metrics_diff_parser.set_defaults(func=CmdMetricsDiff)
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+26 -9
View File
@@ -1,27 +1,32 @@
import argparse
import logging
from collections import OrderedDict
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import fix_subparsers
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
def _show_diff(diff):
def _show_diff(diff, markdown=False, no_path=False):
from dvc.utils.diff import table
rows = []
for fname, pdiff in diff.items():
sorted_pdiff = OrderedDict(sorted(pdiff.items()))
for param, change in sorted_pdiff.items():
rows.append([fname, param, change["old"], change["new"]])
row = [] if no_path else [fname]
row.append(param)
row.append(change["old"])
row.append(change["new"])
rows.append(row)
return table(["Path", "Param", "Old", "New"], rows)
header = [] if no_path else ["Path"]
header.append("Param")
header.append("Old")
header.append("New")
return table(header, rows, markdown)
class CmdParamsDiff(CmdBase):
@@ -38,7 +43,7 @@ class CmdParamsDiff(CmdBase):
logger.info(json.dumps(diff))
else:
table = _show_diff(diff)
table = _show_diff(diff, self.args.show_md, self.args.no_path)
if table:
logger.info(table)
@@ -98,4 +103,16 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Show output in JSON format.",
)
params_diff_parser.add_argument(
"--show-md",
action="store_true",
default=False,
help="Show tabulated output in the Markdown format (GFM).",
)
params_diff_parser.add_argument(
"--no-path",
action="store_true",
default=False,
help="Don't show params path.",
)
params_diff_parser.set_defaults(func=CmdParamsDiff)
-245
View File
@@ -1,245 +0,0 @@
import argparse
import logging
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdPipelineShow(CmdBase):
def _show(self, target, commands, outs, locked):
import networkx
from dvc import dvcfile
from dvc.utils import parse_target
path, name = parse_target(target)
stage = dvcfile.Dvcfile(self.repo, path).stages[name]
G = self.repo.graph
stages = networkx.dfs_postorder_nodes(G, stage)
if locked:
stages = [s for s in stages if s.locked]
for stage in stages:
if commands:
if stage.cmd is None:
continue
logger.info(stage.cmd)
elif outs:
for out in stage.outs:
logger.info(str(out))
else:
logger.info(stage.addressing)
def _build_graph(self, target, commands=False, outs=False):
import networkx
from dvc import dvcfile
from dvc.repo.graph import get_pipeline
from dvc.utils import parse_target
path, name = parse_target(target)
target_stage = dvcfile.Dvcfile(self.repo, path).stages[name]
G = get_pipeline(self.repo.pipelines, target_stage)
nodes = set()
for stage in networkx.dfs_preorder_nodes(G, target_stage):
if commands:
if stage.cmd is None:
continue
nodes.add(stage.cmd)
elif outs:
for out in stage.outs:
nodes.add(str(out))
else:
nodes.add(stage.addressing)
edges = []
for from_stage, to_stage in networkx.edge_dfs(G, target_stage):
if commands:
if to_stage.cmd is None:
continue
edges.append((from_stage.cmd, to_stage.cmd))
elif outs:
for from_out in from_stage.outs:
for to_out in to_stage.outs:
edges.append((str(from_out), str(to_out)))
else:
edges.append((from_stage.addressing, to_stage.addressing))
return list(nodes), edges, networkx.is_tree(G)
def _show_ascii(self, target, commands, outs):
from dvc.dagascii import draw
nodes, edges, _ = self._build_graph(target, commands, outs)
if not nodes:
return
draw(nodes, edges)
def _show_dependencies_tree(self, target, commands, outs):
from treelib import Tree
nodes, edges, is_tree = self._build_graph(target, commands, outs)
if not nodes:
return
if not is_tree:
raise DvcException(
"DAG is not a tree, can not print it in tree-structure way, "
"please use --ascii instead"
)
tree = Tree()
tree.create_node(target, target) # Root node
observe_list = [target]
while len(observe_list) > 0:
current_root = observe_list[0]
for edge in edges:
if edge[0] == current_root:
tree.create_node(edge[1], edge[1], parent=current_root)
observe_list.append(edge[1])
observe_list.pop(0)
tree.show()
def _write_dot(self, target, commands, outs):
import io
import networkx
from networkx.drawing.nx_pydot import write_dot
_, edges, _ = self._build_graph(target, commands, outs)
edges = [edge[::-1] for edge in edges]
simple_g = networkx.DiGraph()
simple_g.add_edges_from(edges)
dot_file = io.StringIO()
write_dot(simple_g, dot_file)
logger.info(dot_file.getvalue())
def run(self):
if not self.args.targets:
self.args.targets = self.default_targets
for target in self.args.targets:
try:
if self.args.ascii:
self._show_ascii(
target, self.args.commands, self.args.outs
)
elif self.args.dot:
self._write_dot(target, self.args.commands, self.args.outs)
elif self.args.tree:
self._show_dependencies_tree(
target, self.args.commands, self.args.outs
)
else:
self._show(
target,
self.args.commands,
self.args.outs,
self.args.locked,
)
except DvcException:
msg = "failed to show pipeline for '{}'".format(target)
logger.exception(msg)
return 1
return 0
class CmdPipelineList(CmdBase):
def run(self):
pipelines = self.repo.pipelines
for pipeline in pipelines:
for stage in pipeline:
logger.info(stage.addressing)
if len(pipeline) != 0:
logger.info("=" * 80)
logger.info("{} pipelines total".format(len(pipelines)))
return 0
def add_parser(subparsers, parent_parser):
PIPELINE_HELP = "Manage pipelines."
pipeline_parser = subparsers.add_parser(
"pipeline",
parents=[parent_parser],
description=append_doc_link(PIPELINE_HELP, "pipeline"),
help=PIPELINE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
pipeline_subparsers = pipeline_parser.add_subparsers(
dest="cmd",
help="Use `dvc pipeline CMD --help` for command-specific help.",
)
fix_subparsers(pipeline_subparsers)
PIPELINE_LIST_HELP = "List connected groups of stages (pipelines)."
pipeline_list_parser = pipeline_subparsers.add_parser(
"list",
parents=[parent_parser],
description=append_doc_link(PIPELINE_LIST_HELP, "pipeline/list"),
help=PIPELINE_LIST_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
pipeline_list_parser.set_defaults(func=CmdPipelineList)
PIPELINE_SHOW_HELP = "Show stages in a pipeline."
pipeline_show_parser = pipeline_subparsers.add_parser(
"show",
parents=[parent_parser],
description=append_doc_link(PIPELINE_SHOW_HELP, "pipeline/show"),
help=PIPELINE_SHOW_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
pipeline_show_group = pipeline_show_parser.add_mutually_exclusive_group()
pipeline_show_group.add_argument(
"-c",
"--commands",
action="store_true",
default=False,
help="Print commands instead of paths to DVC-files.",
)
pipeline_show_group.add_argument(
"-o",
"--outs",
action="store_true",
default=False,
help="Print output files instead of paths to DVC-files.",
)
pipeline_show_parser.add_argument(
"-l",
"--locked",
action="store_true",
default=False,
help="Print locked DVC stages",
)
pipeline_show_parser.add_argument(
"--ascii",
action="store_true",
default=False,
help="Output DAG as ASCII.",
)
pipeline_show_parser.add_argument(
"--dot",
action="store_true",
default=False,
help="Print DAG with .dot format.",
)
pipeline_show_parser.add_argument(
"--tree",
action="store_true",
default=False,
help="Output DAG as Dependencies Tree.",
)
pipeline_show_parser.add_argument(
"targets",
nargs="*",
help="DVC-files to show pipeline for. Optional. "
"(Finds all DVC-files in the workspace by default.)",
)
pipeline_show_parser.set_defaults(func=CmdPipelineShow)
-243
View File
@@ -1,243 +0,0 @@
import argparse
import logging
import os
from dvc.command.base import append_doc_link, CmdBase, fix_subparsers
from dvc.exceptions import DvcException
from dvc.repo.plot.data import WORKSPACE_REVISION_NAME
logger = logging.getLogger(__name__)
class CmdPLot(CmdBase):
def _revisions(self):
raise NotImplementedError
def _result_file(self):
if self.args.file:
return self.args.file
extension = self._result_extension()
base = self._result_basename()
result_file = base + extension
return result_file
def _result_basename(self):
if self.args.datafile:
return self.args.datafile
return "plot"
def _result_extension(self):
if not self.args.no_html:
return ".html"
elif self.args.template:
return os.path.splitext(self.args.template)[-1]
return ".json"
def run(self):
fields = None
jsonpath = None
if self.args.select:
if self.args.select.startswith("$"):
jsonpath = self.args.select
else:
fields = set(self.args.select.split(","))
try:
plot_string = self.repo.plot(
datafile=self.args.datafile,
template=self.args.template,
revisions=self._revisions(),
fields=fields,
x_field=self.args.x,
y_field=self.args.y,
path=jsonpath,
embed=not self.args.no_html,
csv_header=not self.args.no_csv_header,
title=self.args.title,
x_title=self.args.xlab,
y_title=self.args.ylab,
)
if self.args.stdout:
logger.info(plot_string)
else:
result_path = self._result_file()
with open(result_path, "w") as fobj:
fobj.write(plot_string)
logger.info(
"file://{}".format(
os.path.join(self.repo.root_dir, result_path)
)
)
except DvcException:
logger.exception("")
return 1
return 0
class CmdPlotShow(CmdPLot):
def _revisions(self):
return None
class CmdPlotDiff(CmdPLot):
def _revisions(self):
revisions = self.args.revisions or []
if len(revisions) <= 1:
if len(revisions) == 0 and self.repo.scm.is_dirty():
revisions.append("HEAD")
revisions.append(WORKSPACE_REVISION_NAME)
return revisions
def add_parser(subparsers, parent_parser):
PLOT_HELP = (
"Generating plots for continuous metrics stored in structured files "
"(JSON, CSV, TSV)."
)
plot_parser = subparsers.add_parser(
"plot",
parents=[parent_parser],
description=append_doc_link(PLOT_HELP, "plot"),
help=PLOT_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plot_subparsers = plot_parser.add_subparsers(
dest="cmd",
help="Use `dvc plot CMD --help` to display command-specific help.",
)
fix_subparsers(plot_subparsers)
SHOW_HELP = "Generate a plot image file from a continuous metrics file."
plot_show_parser = plot_subparsers.add_parser(
"show",
parents=[parent_parser],
description=append_doc_link(SHOW_HELP, "plot/show"),
help=SHOW_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plot_show_parser.add_argument(
"-t",
"--template",
nargs="?",
default=None,
help="File to be injected with data.",
)
plot_show_parser.add_argument(
"-f", "--file", default=None, help="Name of the generated file."
)
plot_show_parser.add_argument(
"-s",
"--select",
default=None,
help="Choose which field(s) or JSONPath to include in the plot.",
)
plot_show_parser.add_argument(
"-x", default=None, help="Field name for x axis."
)
plot_show_parser.add_argument(
"-y", default=None, help="Field name for y axis."
)
plot_show_parser.add_argument(
"--stdout",
action="store_true",
default=False,
help="Print plot specification to stdout.",
)
plot_show_parser.add_argument(
"--no-csv-header",
action="store_true",
default=False,
help="Required when CSV or TSV datafile does not have a header.",
)
plot_show_parser.add_argument(
"--no-html",
action="store_true",
default=False,
help="Do not wrap Vega plot JSON with HTML.",
)
plot_show_parser.add_argument("--title", default=None, help="Plot title.")
plot_show_parser.add_argument("--xlab", default=None, help="X axis title.")
plot_show_parser.add_argument("--ylab", default=None, help="Y axis title.")
plot_show_parser.add_argument(
"datafile",
nargs="?",
default=None,
help="Continuous metrics file to visualize.",
)
plot_show_parser.set_defaults(func=CmdPlotShow)
PLOT_DIFF_HELP = (
"Plot continuous metrics differences between commits in the DVC "
"repository, or between the last commit and the workspace."
)
plot_diff_parser = plot_subparsers.add_parser(
"diff",
parents=[parent_parser],
description=append_doc_link(PLOT_DIFF_HELP, "plot/diff"),
help=PLOT_DIFF_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plot_diff_parser.add_argument(
"-t",
"--template",
nargs="?",
default=None,
help="File to be injected with data.",
)
plot_diff_parser.add_argument(
"-d",
"--datafile",
nargs="?",
default=None,
help="Continuous metrics file to visualize.",
)
plot_diff_parser.add_argument(
"-f", "--file", default=None, help="Name of the generated file."
)
plot_diff_parser.add_argument(
"-s",
"--select",
default=None,
help="Choose which field(s) or JSONPath to include in the plot.",
)
plot_diff_parser.add_argument(
"-x", default=None, help="Field name for x axis."
)
plot_diff_parser.add_argument(
"-y", default=None, help="Field name for y axis."
)
plot_diff_parser.add_argument(
"--stdout",
action="store_true",
default=False,
help="Print plot specification to stdout.",
)
plot_diff_parser.add_argument(
"--no-csv-header",
action="store_true",
default=False,
help="Provided CSV ot TSV datafile does not have a header.",
)
plot_diff_parser.add_argument(
"--no-html",
action="store_true",
default=False,
help="Do not wrap Vega plot JSON with HTML.",
)
plot_diff_parser.add_argument("--title", default=None, help="Plot title.")
plot_diff_parser.add_argument("--xlab", default=None, help="X axis title.")
plot_diff_parser.add_argument("--ylab", default=None, help="Y axis title.")
plot_diff_parser.add_argument(
"revisions",
nargs="*",
default=None,
help="Git revisions to plot from",
)
plot_diff_parser.set_defaults(func=CmdPlotDiff)
+232
View File
@@ -0,0 +1,232 @@
import argparse
import logging
import os
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
from dvc.schema import PLOT_PROPS
from dvc.utils import format_link
logger = logging.getLogger(__name__)
PAGE_HTML = """<!DOCTYPE html>
<html>
<head>
<title>DVC Plot</title>
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
</head>
<body>
{divs}
</body>
</html>"""
DIV_HTML = """<div id = "{id}"></div>
<script type = "text/javascript">
var spec = {vega_json};
vegaEmbed('#{id}', spec);
</script>"""
class CmdPlots(CmdBase):
def _func(self, *args, **kwargs):
raise NotImplementedError
def _props(self):
# Pass only props specified by user, to not shadow ones from plot def
props = {p: getattr(self.args, p) for p in PLOT_PROPS}
return {k: v for k, v in props.items() if v is not None}
def run(self):
if self.args.show_vega:
if not self.args.targets:
logger.error("please specify a target for `--show-vega`")
return 1
if len(self.args.targets) > 1:
logger.error(
"you can only specify one target for `--show-vega`"
)
return 1
try:
plots = self._func(targets=self.args.targets, props=self._props())
if self.args.show_vega:
target = self.args.targets[0]
logger.info(plots[target])
return 0
divs = [
DIV_HTML.format(id=f"plot{i}", vega_json=plot)
for i, plot in enumerate(plots.values())
]
html = PAGE_HTML.format(divs="\n".join(divs))
path = self.args.out or "plots.html"
with open(path, "w") as fobj:
fobj.write(html)
logger.info(
"file://{}".format(os.path.join(self.repo.root_dir, path))
)
except DvcException:
logger.exception("")
return 1
return 0
class CmdPlotsShow(CmdPlots):
def _func(self, *args, **kwargs):
return self.repo.plots.show(*args, **kwargs)
class CmdPlotsDiff(CmdPlots):
def _func(self, *args, **kwargs):
return self.repo.plots.diff(*args, revs=self.args.revisions, **kwargs)
class CmdPlotsModify(CmdPlots):
def run(self):
self.repo.plots.modify(
self.args.target, props=self._props(), unset=self.args.unset,
)
return 0
def add_parser(subparsers, parent_parser):
PLOTS_HELP = (
"Commands to visualize and compare plot metrics in structured files "
"(JSON, YAML, CSV, TSV)"
)
plots_parser = subparsers.add_parser(
"plots",
parents=[parent_parser],
description=append_doc_link(PLOTS_HELP, "plots"),
help=PLOTS_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_subparsers = plots_parser.add_subparsers(
dest="cmd",
help="Use `dvc plots CMD --help` to display command-specific help.",
)
fix_subparsers(plots_subparsers)
SHOW_HELP = "Generate plots from metric files."
plots_show_parser = plots_subparsers.add_parser(
"show",
parents=[parent_parser],
description=append_doc_link(SHOW_HELP, "plots/show"),
help=SHOW_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_show_parser.add_argument(
"targets",
nargs="*",
help="Plots files to visualize. Shows all plots by default.",
)
_add_props_arguments(plots_show_parser)
_add_output_arguments(plots_show_parser)
plots_show_parser.set_defaults(func=CmdPlotsShow)
PLOTS_DIFF_HELP = (
"Show multiple versions of plot metrics "
"by plotting them in a single image."
)
plots_diff_parser = plots_subparsers.add_parser(
"diff",
parents=[parent_parser],
description=append_doc_link(PLOTS_DIFF_HELP, "plots/diff"),
help=PLOTS_DIFF_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_diff_parser.add_argument(
"--targets",
nargs="*",
help="Plots file to visualize. Shows all plots by default.",
metavar="<path>",
)
plots_diff_parser.add_argument(
"revisions", nargs="*", default=None, help="Git commits to plot from",
)
_add_props_arguments(plots_diff_parser)
_add_output_arguments(plots_diff_parser)
plots_diff_parser.set_defaults(func=CmdPlotsDiff)
PLOTS_MODIFY_HELP = "Modify display properties of plot metric files."
plots_modify_parser = plots_subparsers.add_parser(
"modify",
parents=[parent_parser],
description=append_doc_link(PLOTS_MODIFY_HELP, "plots/modify"),
help=PLOTS_MODIFY_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_modify_parser.add_argument(
"target", help="Metric file to set properties to",
)
_add_props_arguments(plots_modify_parser)
plots_modify_parser.add_argument(
"--unset",
nargs="*",
metavar="<property>",
help="Unset one or more display properties.",
)
plots_modify_parser.set_defaults(func=CmdPlotsModify)
def _add_props_arguments(parser):
parser.add_argument(
"-t",
"--template",
nargs="?",
default=None,
help=(
"Special JSON or HTML schema file to inject with the data. "
"See {}".format(
format_link("https://man.dvc.org/plots#plot-templates")
)
),
metavar="<path>",
)
parser.add_argument(
"-x", default=None, help="Field name for X axis.", metavar="<field>"
)
parser.add_argument(
"-y", default=None, help="Field name for Y axis.", metavar="<field>"
)
parser.add_argument(
"--no-header",
action="store_false",
dest="header",
default=None, # Use default None to distinguish when it's not used
help="Provided CSV or TSV datafile does not have a header.",
)
parser.add_argument(
"--title", default=None, metavar="<text>", help="Plot title."
)
parser.add_argument(
"--x-label", default=None, help="X axis label", metavar="<text>"
)
parser.add_argument(
"--y-label", default=None, help="Y axis label", metavar="<text>"
)
def _add_output_arguments(parser):
parser.add_argument(
"-o",
"--out",
default=None,
help="Destination path to save plots to",
metavar="<path>",
)
parser.add_argument(
"--show-vega",
action="store_true",
default=False,
help="Show output in Vega format.",
)
+60 -9
View File
@@ -1,9 +1,9 @@
import argparse
import logging
from dvc.config import ConfigError
from dvc.command.base import append_doc_link, fix_subparsers
from dvc.command.config import CmdConfig
from dvc.config import ConfigError
from dvc.utils import format_link
logger = logging.getLogger(__name__)
@@ -18,17 +18,13 @@ class CmdRemote(CmdConfig):
def _check_exists(self, conf):
if self.args.name not in conf["remote"]:
raise ConfigError(
"remote '{}' doesn't exists.".format(self.args.name)
)
raise ConfigError(f"remote '{self.args.name}' doesn't exists.")
class CmdRemoteAdd(CmdRemote):
def run(self):
if self.args.default:
logger.info(
"Setting '{}' as a default remote.".format(self.args.name)
)
logger.info(f"Setting '{self.args.name}' as a default remote.")
with self.config.edit(self.args.level) as conf:
if self.args.name in conf["remote"] and not self.args.force:
@@ -66,7 +62,11 @@ class CmdRemoteModify(CmdRemote):
def run(self):
with self.config.edit(self.args.level) as conf:
self._check_exists(conf)
conf["remote"][self.args.name][self.args.option] = self.args.value
section = conf["remote"][self.args.name]
if self.args.unset:
section.pop(self.args.option, None)
else:
section[self.args.option] = self.args.value
return 0
@@ -85,7 +85,18 @@ class CmdRemoteDefault(CmdRemote):
if self.args.unset:
conf["core"].pop("remote", None)
else:
conf["core"]["remote"] = self.args.name
merged_conf = self.config.load_config_to_level(
self.args.level
)
if (
self.args.name in conf["remote"]
or self.args.name in merged_conf["remote"]
):
conf["core"]["remote"] = self.args.name
else:
raise ConfigError(
"default remote must be present in remote list."
)
return 0
@@ -97,6 +108,35 @@ class CmdRemoteList(CmdRemote):
return 0
class CmdRemoteRename(CmdRemote):
def _rename_default(self, conf):
if conf["core"].get("remote") == self.args.name:
conf["core"]["remote"] = self.args.new
def run(self):
all_config = self.config.load_config_to_level(None)
if self.args.new in all_config.get("remote", {}):
raise ConfigError(
"Rename failed. Remote name '{}' already exists.".format(
{self.args.new}
)
)
with self.config.edit(self.args.level) as conf:
self._check_exists(conf)
conf["remote"][self.args.new] = conf["remote"][self.args.name]
del conf["remote"][self.args.name]
self._rename_default(conf)
for level in reversed(self.config.LEVELS):
if level == self.args.level:
break
with self.config.edit(level) as level_conf:
self._rename_default(level_conf)
return 0
def add_parser(subparsers, parent_parser):
from dvc.command.config import parent_config_parser
@@ -213,3 +253,14 @@ def add_parser(subparsers, parent_parser):
"name", help="Name of the remote to remove."
)
remote_remove_parser.set_defaults(func=CmdRemoteRemove)
REMOTE_RENAME_HELP = "Rename a DVC remote"
remote_rename_parser = remote_subparsers.add_parser(
"rename",
parents=[parent_config_parser, parent_parser],
description=append_doc_link(REMOTE_RENAME_HELP, "remote/rename"),
help=REMOTE_RENAME_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
remote_rename_parser.add_argument("name", help="Remote to be renamed")
remote_rename_parser.add_argument("new", help="New name of the remote")
remote_rename_parser.set_defaults(func=CmdRemoteRename)
+6 -45
View File
@@ -1,48 +1,25 @@
import argparse
import logging
import dvc.prompt as prompt
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdRemove(CmdBase):
def _is_outs_only(self, target):
if not self.args.purge:
return True
if self.args.force:
return False
msg = "Are you sure you want to remove '{}' with its outputs?".format(
target
)
if prompt.confirm(msg):
return False
raise DvcException(
"Cannot purge without a confirmation from the user."
" Use `-f` to force."
)
def run(self):
for target in self.args.targets:
try:
outs_only = self._is_outs_only(target)
self.repo.remove(target, outs_only=outs_only)
self.repo.remove(target, outs=self.args.outs)
except DvcException:
logger.exception("failed to remove '{}'".format(target))
logger.exception(f"failed to remove '{target}'")
return 1
return 0
def add_parser(subparsers, parent_parser):
REMOVE_HELP = "Remove DVC-tracked files or directories."
REMOVE_HELP = "Remove stage entry and unprotect outputs"
remove_parser = subparsers.add_parser(
"remove",
parents=[parent_parser],
@@ -50,27 +27,11 @@ def add_parser(subparsers, parent_parser):
help=REMOVE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
remove_parser_group = remove_parser.add_mutually_exclusive_group()
remove_parser_group.add_argument(
"-o",
remove_parser.add_argument(
"--outs",
action="store_true",
default=True,
help="Only remove DVC-file outputs. (Default)",
)
remove_parser_group.add_argument(
"-p",
"--purge",
action="store_true",
default=False,
help="Remove DVC-file and all its outputs.",
)
remove_parser.add_argument(
"-f",
"--force",
action="store_true",
default=False,
help="Force purge.",
help="Remove outputs as well.",
)
remove_parser.add_argument(
"targets", nargs="+", help="DVC-files to remove."
+15 -6
View File
@@ -2,13 +2,12 @@ import argparse
import logging
import os
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.command.metrics import show_metrics
from dvc.command.status import CmdDataStatus
from dvc.dvcfile import PIPELINE_FILE
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -35,10 +34,11 @@ class CmdRepro(CmdBase):
interactive=self.args.interactive,
pipeline=self.args.pipeline,
all_pipelines=self.args.all_pipelines,
ignore_build_cache=self.args.ignore_build_cache,
run_cache=not self.args.no_run_cache,
no_commit=self.args.no_commit,
downstream=self.args.downstream,
recursive=self.args.recursive,
force_downstream=self.args.force_downstream,
)
if len(stages) == 0:
@@ -70,7 +70,7 @@ def add_parser(subparsers, parent_parser):
repro_parser.add_argument(
"targets",
nargs="*",
help="DVC-file to reproduce. 'Dvcfile' by default.",
help=f"Stages to reproduce. '{PIPELINE_FILE}' by default.",
)
repro_parser.add_argument(
"-f",
@@ -138,7 +138,16 @@ def add_parser(subparsers, parent_parser):
help="Reproduce all stages in the specified directory.",
)
repro_parser.add_argument(
"--ignore-build-cache",
"--no-run-cache",
action="store_true",
default=False,
help=(
"Execute stage commands even if they have already been run with "
"the same command/dependencies/outputs/etc before."
),
)
repro_parser.add_argument(
"--force-downstream",
action="store_true",
default=False,
help="Reproduce all descendants of a changed stage even if their "
+1 -2
View File
@@ -1,8 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.utils import relpath
logger = logging.getLogger(__name__)
+41 -17
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -18,6 +16,8 @@ class CmdRun(CmdBase):
self.args.outs_no_cache,
self.args.metrics,
self.args.metrics_no_cache,
self.args.plots,
self.args.plots_no_cache,
self.args.outs_persist,
self.args.outs_persist_no_cache,
self.args.params,
@@ -26,8 +26,8 @@ class CmdRun(CmdBase):
): # pragma: no cover
logger.error(
"too few arguments. Specify at least one: `-d`, `-o`, `-O`, "
"`-m`, `-M`, `-p`, `--outs-persist`, "
"`--outs-persist-no-cache`, `command`."
"`-m`, `-M`, `-p`, `--plots`, `--plots-no-cache`, "
"`--outs-persist`, `--outs-persist-no-cache`, `command`."
)
return 1
@@ -38,19 +38,22 @@ class CmdRun(CmdBase):
outs_no_cache=self.args.outs_no_cache,
metrics=self.args.metrics,
metrics_no_cache=self.args.metrics_no_cache,
plots=self.args.plots,
plots_no_cache=self.args.plots_no_cache,
deps=self.args.deps,
params=self.args.params,
fname=self.args.file,
wdir=self.args.wdir,
no_exec=self.args.no_exec,
overwrite=self.args.overwrite_dvcfile,
ignore_build_cache=self.args.ignore_build_cache,
force=self.args.force,
run_cache=not self.args.no_run_cache,
no_commit=self.args.no_commit,
outs_persist=self.args.outs_persist,
outs_persist_no_cache=self.args.outs_persist_no_cache,
always_changed=self.args.always_changed,
name=self.args.name,
single_stage=self.args.single_stage,
external=self.args.external,
)
except DvcException:
logger.exception("")
@@ -77,7 +80,7 @@ class CmdRun(CmdBase):
if " " not in argument or '"' in argument:
return argument
return '"{}"'.format(argument)
return f'"{argument}"'
def add_parser(subparsers, parent_parser):
@@ -130,7 +133,7 @@ def add_parser(subparsers, parent_parser):
"--metrics",
action="append",
default=[],
help="Declare output metric file or directory.",
help="Declare output metric file.",
metavar="<path>",
)
run_parser.add_argument(
@@ -138,12 +141,24 @@ def add_parser(subparsers, parent_parser):
"--metrics-no-cache",
action="append",
default=[],
help="Declare output metric file or directory "
"(do not put into DVC cache).",
help="Declare output metric file (do not put into DVC cache).",
metavar="<path>",
)
run_parser.add_argument(
"--plots",
action="append",
default=[],
help="Declare output plot file.",
metavar="<path>",
)
run_parser.add_argument(
"--plots-no-cache",
action="append",
default=[],
help="Declare output plot file (do not put into DVC cache).",
metavar="<path>",
)
run_parser.add_argument(
"-f",
"--file",
help="Specify name of the DVC-file this command will generate.",
metavar="<filename>",
@@ -161,17 +176,20 @@ def add_parser(subparsers, parent_parser):
help="Only create stage file without actually running it.",
)
run_parser.add_argument(
"--overwrite-dvcfile",
"-f",
"--force",
action="store_true",
default=False,
help="Overwrite existing DVC-file without asking for confirmation.",
help="Overwrite existing stage",
)
run_parser.add_argument(
"--ignore-build-cache",
"--no-run-cache",
action="store_true",
default=False,
help="Run this stage even if it has been already ran with the same "
"command/dependencies/outputs/etc before.",
help=(
"Execute the command even if this stage has already been run "
"with the same command/dependencies/outputs/etc before."
),
)
run_parser.add_argument(
"--no-commit",
@@ -207,6 +225,12 @@ def add_parser(subparsers, parent_parser):
default=False,
help=argparse.SUPPRESS,
)
run_parser.add_argument(
"--external",
action="store_true",
default=False,
help="Allow outputs that are outside of the DVC repository.",
)
run_parser.add_argument(
"command", nargs=argparse.REMAINDER, help="Command to execute."
)
+12 -8
View File
@@ -2,7 +2,6 @@ import logging
from dvc.command.data_sync import CmdDataBase
logger = logging.getLogger(__name__)
@@ -20,7 +19,7 @@ class CmdDataStatus(CmdDataBase):
ind = indent * self.STATUS_INDENT
if isinstance(status, str):
logger.info("{}{}".format(ind, status))
logger.info(f"{ind}{status}")
return
if isinstance(status, list):
@@ -34,7 +33,7 @@ class CmdDataStatus(CmdDataBase):
if isinstance(value, str):
logger.info("{}{}{}".format(ind, self._normalize(value), key))
elif value:
logger.info("{}{}:".format(ind, key))
logger.info(f"{ind}{key}:")
self._show(value, indent + 1)
def run(self):
@@ -51,11 +50,16 @@ class CmdDataStatus(CmdDataBase):
with_deps=self.args.with_deps,
recursive=self.args.recursive,
)
if st:
if self.args.quiet:
return 1
else:
self._show(st, indent)
if self.args.quiet:
return bool(st)
if self.args.show_json:
import json
logger.info(json.dumps(st))
elif st:
self._show(st, indent)
else:
logger.info(self.UP_TO_DATE_MSG)
+2 -4
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -15,7 +13,7 @@ class CmdUnprotect(CmdBase):
try:
self.repo.unprotect(target)
except DvcException:
msg = "failed to unprotect '{}'".format(target)
msg = f"failed to unprotect '{target}'"
logger.exception(msg)
return 1
return 0
+17 -9
View File
@@ -1,23 +1,24 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdUpdate(CmdBase):
def run(self):
ret = 0
for target in self.args.targets:
try:
self.repo.update(target, self.args.rev)
except DvcException:
logger.exception("failed to update '{}'.".format(target))
ret = 1
try:
self.repo.update(
targets=self.args.targets,
rev=self.args.rev,
recursive=self.args.recursive,
)
except DvcException:
logger.exception("failed update data")
ret = 1
return ret
@@ -39,4 +40,11 @@ def add_parser(subparsers, parent_parser):
help="Git revision (e.g. SHA, branch, tag)",
metavar="<commit>",
)
update_parser.add_argument(
"-R",
"--recursive",
action="store_true",
default=False,
help="Update all stages in the specified directory.",
)
update_parser.set_defaults(func=CmdUpdate)
+22 -21
View File
@@ -2,22 +2,23 @@ import argparse
import itertools
import logging
import os
import pathlib
import platform
import uuid
import pathlib
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.scm.base import SCMError
from dvc.system import System
from dvc.utils import is_binary, relpath
from dvc.utils.pkg import PKG
from dvc.version import __version__
try:
import psutil
except ImportError:
psutil = None
from dvc.utils import is_binary, relpath
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.version import __version__
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.scm.base import SCMError
from dvc.system import System
from dvc.utils.pkg import PKG
logger = logging.getLogger(__name__)
@@ -27,12 +28,12 @@ class CmdVersion(CmdBaseNoRepo):
from dvc.repo import Repo
info = [
"DVC version: {}".format(__version__),
"Python version: {}".format(platform.python_version()),
"Platform: {}".format(platform.platform()),
"Binary: {}".format(is_binary()),
"Package: {}".format(PKG),
"Supported remotes: {}".format(self.get_supported_remotes()),
f"DVC version: {__version__}",
f"Python version: {platform.python_version()}",
f"Platform: {platform.platform()}",
f"Binary: {is_binary()}",
f"Package: {PKG}",
f"Supported remotes: {self.get_supported_remotes()}",
]
try:
@@ -50,7 +51,7 @@ class CmdVersion(CmdBaseNoRepo):
if psutil:
fs_type = self.get_fs_type(repo.cache.local.cache_dir)
info.append(
"Filesystem type (cache directory): {}".format(fs_type)
f"Filesystem type (cache directory): {fs_type}"
)
else:
logger.warning(
@@ -71,7 +72,7 @@ class CmdVersion(CmdBaseNoRepo):
if psutil:
fs_root = self.get_fs_type(os.path.abspath(root_directory))
info.append("Filesystem type (workspace): {}".format(fs_root))
info.append(f"Filesystem type (workspace): {fs_root}")
logger.info("\n".join(info))
return 0
@@ -114,19 +115,19 @@ class CmdVersion(CmdBaseNoRepo):
os.unlink(dst)
except DvcException:
status = "not supported"
cache.append("{name} - {status}".format(name=name, status=status))
cache.append(f"{name} - {status}")
os.remove(src)
return ", ".join(cache)
@staticmethod
def get_supported_remotes():
from dvc.remote import REMOTES
from dvc.remote import TREES
supported_remotes = []
for remote in REMOTES:
if not remote.get_missing_deps():
supported_remotes.append(remote.scheme)
for tree_cls in TREES:
if not tree_cls.get_missing_deps():
supported_remotes.append(tree_cls.scheme)
return ", ".join(supported_remotes)
-42
View File
@@ -1,42 +0,0 @@
"""Python version compatibility code"""
# Backport os.fspath() from Python 3.6
try:
from os import fspath # noqa: F821
fspath_py35 = lambda s: s # noqa: E731
except ImportError:
def fspath(path):
"""Return the path representation of a path-like object.
If str or bytes is passed in, it is returned unchanged. Otherwise the
os.PathLike interface is used to get the path representation. If the
path representation is not str or bytes, TypeError is raised. If the
provided path is not str, bytes, or os.PathLike, TypeError is raised.
"""
if isinstance(path, (str, bytes)):
return path
# Work from the object's type to match method resolution of other magic
# methods.
path_type = type(path)
try:
path_repr = path_type.__fspath__(path)
except AttributeError:
if hasattr(path_type, "__fspath__"):
raise
else:
raise TypeError(
"expected str, bytes or os.PathLike object, "
"not " + path_type.__name__
)
if isinstance(path_repr, (str, bytes)):
return path_repr
else:
raise TypeError(
"expected {}.__fspath__() to return str or bytes, "
"not {}".format(path_type.__name__, type(path_repr).__name__)
)
fspath_py35 = fspath
+8 -7
View File
@@ -31,7 +31,7 @@ class ConfigError(DvcException):
"""DVC config exception."""
def __init__(self, msg):
super().__init__("config file error: {}".format(msg))
super().__init__(f"config file error: {msg}")
class NoRemoteError(ConfigError):
@@ -89,7 +89,7 @@ def ByUrl(mapping):
if os.name == "nt" and len(parsed.scheme) == 1 and parsed.netloc == "":
return schemas[""](data)
if parsed.scheme not in schemas:
raise Invalid("Unsupported URL type {}://".format(parsed.scheme))
raise Invalid(f"Unsupported URL type {parsed.scheme}://")
return schemas[parsed.scheme](data)
@@ -151,6 +151,7 @@ SCHEMA = {
Optional("listobjects", default=False): Bool,
Optional("use_ssl", default=True): Bool,
"sse": str,
"sse_kms_key_id": str,
"acl": str,
"grant_read": str,
"grant_read_acp": str,
@@ -289,7 +290,7 @@ class Config(dict):
Raises:
ConfigError: thrown if config has an invalid format.
"""
conf = self._load_config_to_level()
conf = self.load_config_to_level()
if validate:
conf = self.validate(conf)
@@ -349,7 +350,7 @@ class Config(dict):
}
return Schema(dirs_schema, extra=ALLOW_EXTRA)(conf)
def _load_config_to_level(self, level=None):
def load_config_to_level(self, level=None):
merged_conf = {}
for merge_level in self.LEVELS:
if merge_level == level:
@@ -368,7 +369,7 @@ class Config(dict):
conf = self._save_paths(conf, self.files[level])
merged_conf = self._load_config_to_level(level)
merged_conf = self.load_config_to_level(level)
_merge(merged_conf, conf)
self.validate(merged_conf)
@@ -389,7 +390,7 @@ def _load_config(filename):
def _save_config(filename, conf_dict):
logger.debug("Writing '{}'.".format(filename))
logger.debug(f"Writing '{filename}'.")
os.makedirs(os.path.dirname(filename), exist_ok=True)
config = configobj.ConfigObj(_pack_remotes(conf_dict))
@@ -416,7 +417,7 @@ def _pack_remotes(conf):
# Transform remote.name -> 'remote "name"'
for name, val in conf["remote"].items():
result['remote "{}"'.format(name)] = val
result[f'remote "{name}"'] = val
result.pop("remote", None)
return result
+3 -6
View File
@@ -7,9 +7,7 @@ import sys
from subprocess import Popen
from dvc.env import DVC_DAEMON
from dvc.utils import fix_env
from dvc.utils import is_binary
from dvc.utils import fix_env, is_binary
logger = logging.getLogger(__name__)
@@ -48,7 +46,6 @@ def _spawn_posix(cmd, env):
os._exit(1) # pylint: disable=protected-access
os.setsid()
os.umask(0)
try:
pid = os.fork()
@@ -68,7 +65,7 @@ def _spawn_posix(cmd, env):
def _spawn(cmd, env):
logger.debug("Trying to spawn '{}'".format(cmd))
logger.debug(f"Trying to spawn '{cmd}'")
if os.name == "nt":
_spawn_windows(cmd, env)
@@ -77,7 +74,7 @@ def _spawn(cmd, env):
else:
raise NotImplementedError
logger.debug("Spawned '{}'".format(cmd))
logger.debug(f"Spawned '{cmd}'")
def daemon(args):
+11 -15
View File
@@ -6,12 +6,9 @@ import os
import pydoc
import sys
from grandalf.graphs import Edge
from grandalf.graphs import Graph
from grandalf.graphs import Vertex
from grandalf.graphs import Edge, Graph, Vertex
from grandalf.layouts import SugiyamaLayout
from grandalf.routing import EdgeViewer
from grandalf.routing import route_with_lines
from grandalf.routing import EdgeViewer, route_with_lines
from dvc.env import DVC_PAGER
from dvc.utils import format_link
@@ -40,7 +37,7 @@ def find_pager():
if env_pager:
return make_pager(env_pager)
if os.system("({}) 2>{}".format(DEFAULT_PAGER, os.devnull)) == 0:
if os.system(f"({DEFAULT_PAGER}) 2>{os.devnull}") == 0:
return make_pager(DEFAULT_PAGER_FORMATTED)
logger.warning(
@@ -52,7 +49,7 @@ def find_pager():
return pydoc.plainpager
class VertexViewer(object):
class VertexViewer:
"""Class to define vertex box boundaries that will be accounted for during
graph building by grandalf.
@@ -78,7 +75,7 @@ class VertexViewer(object):
return self._w
class AsciiCanvas(object):
class AsciiCanvas:
"""Class for drawing in ASCII.
Args:
@@ -99,10 +96,9 @@ class AsciiCanvas(object):
def draw(self):
"""Draws ASCII canvas on the screen."""
pager = find_pager()
lines = map("".join, self.canvas)
joined_lines = os.linesep.join(lines)
pager(joined_lines)
return joined_lines
def point(self, x, y, char):
"""Create a point on ASCII canvas.
@@ -148,21 +144,21 @@ class AsciiCanvas(object):
if dx == 0:
y = y0
else:
y = y0 + int(round((x - x0) * dy / float((dx))))
y = y0 + int(round((x - x0) * dy / float(dx)))
self.point(x, y, char)
elif y0 < y1:
for y in range(y0, y1 + 1):
if dy == 0:
x = x0
else:
x = x0 + int(round((y - y0) * dx / float((dy))))
x = x0 + int(round((y - y0) * dx / float(dy)))
self.point(x, y, char)
else:
for y in range(y1, y0 + 1):
if dy == 0:
x = x0
else:
x = x1 + int(round((y - y1) * dx / float((dy))))
x = x1 + int(round((y - y1) * dx / float(dy)))
self.point(x, y, char)
def text(self, x, y, text):
@@ -216,7 +212,7 @@ def _build_sugiyama_layout(vertexes, edges):
# Y
#
vertexes = {v: Vertex(" {} ".format(v)) for v in vertexes}
vertexes = {v: Vertex(f" {v} ") for v in vertexes}
# NOTE: reverting edges to correctly orientate the graph
edges = [Edge(vertexes[e], vertexes[s]) for s, e in edges]
vertexes = vertexes.values()
@@ -319,4 +315,4 @@ def draw(vertexes, edges):
int(round(x - minx)) + 1, int(round(y - miny)) + 1, vertex.data
)
canvas.draw()
return canvas.draw()
+16 -13
View File
@@ -3,13 +3,12 @@
import logging
from dvc.config import NoRemoteError
from dvc.remote import Remote
from dvc.remote import get_remote
logger = logging.getLogger(__name__)
class DataCloud(object):
class DataCloud:
"""Class that manages dvc remotes.
Args:
@@ -46,9 +45,11 @@ class DataCloud(object):
raise NoRemoteError(error_msg)
def _init_remote(self, name):
return Remote(self.repo, name=name)
return get_remote(self.repo, name=name)
def push(self, cache, jobs=None, remote=None, show_checksums=False):
def push(
self, cache, jobs=None, remote=None, show_checksums=False,
):
"""Push data items in a cloud-agnostic way.
Args:
@@ -59,14 +60,15 @@ class DataCloud(object):
show_checksums (bool): show checksums instead of file names in
information messages.
"""
remote = self.get_remote(remote, "push")
return self.repo.cache.local.push(
cache,
jobs=jobs,
remote=self.get_remote(remote, "push"),
show_checksums=show_checksums,
cache, jobs=jobs, remote=remote, show_checksums=show_checksums,
)
def pull(self, cache, jobs=None, remote=None, show_checksums=False):
def pull(
self, cache, jobs=None, remote=None, show_checksums=False,
):
"""Pull data items in a cloud-agnostic way.
Args:
@@ -78,19 +80,20 @@ class DataCloud(object):
information messages.
"""
remote = self.get_remote(remote, "pull")
downloaded_items_num = self.repo.cache.local.pull(
cache, jobs=jobs, remote=remote, show_checksums=show_checksums
)
if not remote.verify:
if not remote.tree.verify:
self._save_pulled_checksums(cache)
return downloaded_items_num
def _save_pulled_checksums(self, cache):
for checksum in cache.scheme_keys("local"):
cache_file = self.repo.cache.local.checksum_to_path_info(checksum)
if self.repo.cache.local.exists(cache_file):
cache_file = self.repo.cache.local.hash_to_path_info(checksum)
if self.repo.cache.local.tree.exists(cache_file):
# We can safely save here, as existing corrupted files will
# be removed upon status, while files corrupted during
# download will not be moved from tmp_file
+32 -25
View File
@@ -1,22 +1,24 @@
from urllib.parse import urlparse
from collections import defaultdict
from urllib.parse import urlparse
import dvc.output as output
from dvc.dependency.azure import AzureDependency
from dvc.dependency.gs import GSDependency
from dvc.dependency.hdfs import HDFSDependency
from dvc.dependency.http import HTTPDependency
from dvc.dependency.https import HTTPSDependency
from dvc.dependency.local import LocalDependency
from dvc.dependency.param import ParamsDependency
from dvc.dependency.s3 import S3Dependency
from dvc.dependency.ssh import SSHDependency
from dvc.dependency.param import ParamsDependency
from dvc.output.base import BaseOutput
from dvc.remote import Remote
from dvc.remote import get_remote
from dvc.scheme import Schemes
from .repo import RepoDependency
DEPS = [
AzureDependency,
GSDependency,
HDFSDependency,
HTTPDependency,
@@ -30,6 +32,7 @@ DEP_MAP = {
Schemes.LOCAL: LocalDependency,
Schemes.SSH: SSHDependency,
Schemes.S3: S3Dependency,
Schemes.AZURE: AzureDependency,
Schemes.GS: GSDependency,
Schemes.HDFS: HDFSDependency,
Schemes.HTTP: HTTPDependency,
@@ -51,7 +54,7 @@ SCHEMA.update(ParamsDependency.PARAM_SCHEMA)
def _get(stage, p, info):
parsed = urlparse(p) if p else None
if parsed and parsed.scheme == "remote":
remote = Remote(stage.repo, name=parsed.netloc)
remote = get_remote(stage.repo, name=parsed.netloc)
return DEP_MAP[remote.scheme](stage, p, info, remote=remote)
if info and info.get(RepoDependency.PARAM_REPO):
@@ -77,6 +80,7 @@ def loadd_from(stage, d_list):
def loads_from(stage, s_list, erepo=None):
assert isinstance(s_list, list)
ret = []
for s in s_list:
info = {RepoDependency.PARAM_REPO: erepo} if erepo else {}
@@ -84,26 +88,29 @@ def loads_from(stage, s_list, erepo=None):
return ret
def _parse_params(path_params):
path, _, params_str = path_params.rpartition(":")
params = params_str.split(",")
return path, params
def _merge_params(s_list):
d = defaultdict(list)
default_file = ParamsDependency.DEFAULT_PARAMS_FILE
for key in s_list:
if isinstance(key, str):
d[default_file].append(key)
continue
if not isinstance(key, dict):
msg = "Only list of str/dict is supported. Got: "
msg += f"'{type(key).__name__}'."
raise ValueError(msg)
for k, params in key.items():
if not isinstance(params, list):
msg = "Expected list of params for custom params file "
msg += f"'{k}', got '{type(params).__name__}'."
raise ValueError(msg)
d[k].extend(params)
return d
def loads_params(stage, s_list):
# Creates an object for each unique file that is referenced in the list
params_by_path = defaultdict(list)
for s in s_list:
path, params = _parse_params(s)
params_by_path[path].extend(params)
d_list = []
for path, params in params_by_path.items():
d_list.append(
{
BaseOutput.PARAM_PATH: path,
ParamsDependency.PARAM_PARAMS: params,
}
)
return loadd_from(stage, d_list)
d = _merge_params(s_list)
return [
ParamsDependency(stage, path, params) for path, params in d.items()
]
+7
View File
@@ -0,0 +1,7 @@
from dvc.dependency.base import BaseDependency
from dvc.output.base import BaseOutput
from dvc.remote.azure import AzureRemoteTree
class AzureDependency(BaseDependency, BaseOutput):
TREE_CLS = AzureRemoteTree
+4 -6
View File
@@ -3,24 +3,22 @@ from dvc.exceptions import DvcException
class DependencyDoesNotExistError(DvcException):
def __init__(self, path):
msg = "dependency '{}' does not exist".format(path)
msg = f"dependency '{path}' does not exist"
super().__init__(msg)
class DependencyIsNotFileOrDirError(DvcException):
def __init__(self, path):
msg = "dependency '{}' is not a file or directory".format(path)
msg = f"dependency '{path}' is not a file or directory"
super().__init__(msg)
class DependencyIsStageFileError(DvcException):
def __init__(self, path):
super().__init__(
"Stage file '{}' cannot be a dependency.".format(path)
)
super().__init__(f"Stage file '{path}' cannot be a dependency.")
class BaseDependency(object):
class BaseDependency:
IS_DEPENDENCY = True
DoesNotExistError = DependencyDoesNotExistError
+2 -2
View File
@@ -1,7 +1,7 @@
from dvc.dependency.base import BaseDependency
from dvc.output.base import BaseOutput
from dvc.remote.http import HTTPRemote
from dvc.remote.http import HTTPRemoteTree
class HTTPDependency(BaseDependency, BaseOutput):
REMOTE = HTTPRemote
TREE_CLS = HTTPRemoteTree
+3 -2
View File
@@ -1,6 +1,7 @@
from dvc.remote.https import HTTPSRemoteTree
from .http import HTTPDependency
from dvc.remote.https import HTTPSRemote
class HTTPSDependency(HTTPDependency):
REMOTE = HTTPSRemote
TREE_CLS = HTTPSRemoteTree
+4 -5
View File
@@ -1,11 +1,10 @@
import os
import yaml
from collections import defaultdict
import dpath.util
import yaml
from voluptuous import Any
from dvc.compat import fspath_py35
from dvc.dependency.local import LocalDependency
from dvc.exceptions import DvcException
@@ -41,7 +40,7 @@ class ParamsDependency(LocalDependency):
info=info,
)
def _dyn_load(self, values=None):
def fill_values(self, values=None):
"""Load params values dynamically."""
if not values:
return
@@ -87,12 +86,12 @@ class ParamsDependency(LocalDependency):
if not self.exists:
return {}
with self.repo.tree.open(fspath_py35(self.path_info), "r") as fobj:
with self.repo.tree.open(self.path_info, "r") as fobj:
try:
config = yaml.safe_load(fobj)
except yaml.YAMLError as exc:
raise BadParamFileError(
"Unable to read parameters from '{}'".format(self)
f"Unable to read parameters from '{self}'"
) from exc
ret = {}
+18 -8
View File
@@ -2,10 +2,11 @@ import os
from voluptuous import Required
from .local import LocalDependency
from dvc.exceptions import OutputNotFoundError
from dvc.path_info import PathInfo
from .local import LocalDependency
class RepoDependency(LocalDependency):
PARAM_REPO = "repo"
@@ -49,13 +50,22 @@ class RepoDependency(LocalDependency):
return external_repo(d["url"], rev=rev)
def _get_checksum(self, locked=True):
from dvc.repo.tree import RepoTree
with self._make_repo(locked=locked) as repo:
try:
return repo.find_out_by_relpath(self.def_path).info["md5"]
except OutputNotFoundError:
path = PathInfo(os.path.join(repo.root_dir, self.def_path))
# we want stream but not fetch, so DVC out directories are
# walked, but dir contents is not fetched
tree = RepoTree(repo, stream=True)
# We are polluting our repo cache with some dir listing here
return self.repo.cache.local.get_checksum(path)
if tree.isdir(path):
return self.repo.cache.local.get_hash(path, tree)
return tree.get_file_hash(path)
def status(self):
current_checksum = self._get_checksum(locked=True)
@@ -75,16 +85,16 @@ class RepoDependency(LocalDependency):
def download(self, to):
with self._make_repo() as repo:
if self.def_repo.get(self.PARAM_REV_LOCK) is None:
self.def_repo[self.PARAM_REV_LOCK] = repo.scm.get_rev()
self.def_repo[self.PARAM_REV_LOCK] = repo.get_rev()
if hasattr(repo, "cache"):
repo.cache.local.cache_dir = self.repo.cache.local.cache_dir
repo.pull_to(self.def_path, to.path_info)
cache = self.repo.cache.local
with repo.use_cache(cache):
_, _, cache_infos = repo.fetch_external([self.def_path])
cache.checkout(to.path_info, cache_infos[0])
def update(self, rev=None):
if rev:
self.def_repo[self.PARAM_REV] = rev
with self._make_repo(locked=False) as repo:
self.def_repo[self.PARAM_REV_LOCK] = repo.scm.get_rev()
self.def_repo[self.PARAM_REV_LOCK] = repo.get_rev()
+79 -47
View File
@@ -1,27 +1,21 @@
import contextlib
import os
import logging
import dvc.prompt as prompt
import os
from voluptuous import MultipleInvalid
from dvc import serialize
from dvc.exceptions import DvcException
from dvc.stage.loader import SingleStageLoader, StageLoader
from dvc.stage import serialize
from dvc.stage.exceptions import (
StageFileBadNameError,
StageFileDoesNotExistError,
StageFileIsNotDvcFileError,
StageFileFormatError,
StageFileAlreadyExistsError,
StageFileIsNotDvcFileError,
)
from dvc.stage.loader import SingleStageLoader, StageLoader
from dvc.utils import relpath
from dvc.utils.collections import apply_diff
from dvc.utils.stage import (
dump_stage_file,
parse_stage,
parse_stage_for_update,
)
from dvc.utils.yaml import dump_yaml, parse_yaml, parse_yaml_for_update
logger = logging.getLogger(__name__)
@@ -32,8 +26,7 @@ PIPELINE_LOCK = "dvc.lock"
class LockfileCorruptedError(DvcException):
def __init__(self, path):
super().__init__("Lockfile '{}' is corrupted.".format(path))
pass
def is_valid_filename(path):
@@ -80,7 +73,7 @@ class FileMixin:
) == os.path.abspath(other.path)
def __str__(self):
return "{}: {}".format(self.__class__.__name__, self.relpath)
return f"{self.__class__.__name__}: {self.relpath}"
@property
def relpath(self):
@@ -102,8 +95,8 @@ class FileMixin:
with self.repo.tree.open(self.path) as fd:
stage_text = fd.read()
d = parse_stage(stage_text, self.path)
self.validate(d, self.path)
d = parse_yaml(stage_text, self.path)
self.validate(d, self.relpath)
return d, stage_text
@classmethod
@@ -112,10 +105,7 @@ class FileMixin:
try:
cls.SCHEMA(d)
except MultipleInvalid as exc:
raise StageFileFormatError(fname, exc)
def remove_with_prompt(self, force=False):
raise NotImplementedError
raise StageFileFormatError(f"'{fname}' format error: {exc}")
def remove(self, force=False):
with contextlib.suppress(FileNotFoundError):
@@ -150,20 +140,10 @@ class SingleStageFile(FileMixin):
logger.debug(
"Saving information to '{file}'.".format(file=relpath(self.path))
)
dump_stage_file(self.path, serialize.to_single_stage_file(stage))
self.repo.scm.track_file(relpath(self.path))
def remove_with_prompt(self, force=False):
if not self.exists():
return
msg = (
"'{}' already exists. Do you wish to run the command and "
"overwrite it?".format(relpath(self.path))
)
if not (force or prompt.confirm(msg)):
raise StageFileAlreadyExistsError(self.path)
dump_yaml(self.path, serialize.to_single_stage_file(stage))
self.repo.scm.track_file(self.relpath)
def remove_stage(self, stage):
self.remove()
@@ -176,37 +156,48 @@ class PipelineFile(FileMixin):
def _lockfile(self):
return Lockfile(self.repo, os.path.splitext(self.path)[0] + ".lock")
def dump(self, stage, update_pipeline=False, **kwargs):
def dump(self, stage, update_pipeline=False, no_lock=False):
"""Dumps given stage appropriately in the dvcfile."""
from dvc.stage import PipelineStage
assert isinstance(stage, PipelineStage)
check_dvc_filename(self.path)
self._dump_lockfile(stage)
if update_pipeline and not stage.is_data_source:
self._dump_pipeline_file(stage)
if not no_lock:
self._dump_lockfile(stage)
def _dump_lockfile(self, stage):
self._lockfile.dump(stage)
def _dump_pipeline_file(self, stage):
data = {}
if self.exists():
with open(self.path, "r") as fd:
data = parse_stage_for_update(fd.read(), self.path)
with open(self.path) as fd:
data = parse_yaml_for_update(fd.read(), self.path)
else:
logger.info("Creating '%s'", self.relpath)
open(self.path, "w+").close()
data["stages"] = data.get("stages", {})
stage_data = serialize.to_pipeline_file(stage)
if data["stages"].get(stage.name):
existing_entry = stage.name in data["stages"]
action = "Modifying" if existing_entry else "Adding"
logger.info("%s stage '%s' in '%s'", action, stage.name, self.relpath)
if existing_entry:
orig_stage_data = data["stages"][stage.name]
if "meta" in orig_stage_data:
stage_data[stage.name]["meta"] = orig_stage_data["meta"]
apply_diff(stage_data[stage.name], orig_stage_data)
else:
data["stages"].update(stage_data)
dump_stage_file(self.path, data)
self.repo.scm.track_file(relpath(self.path))
dump_yaml(self.path, data)
self.repo.scm.track_file(self.relpath)
@property
def stage(self):
@@ -228,6 +219,22 @@ class PipelineFile(FileMixin):
super().remove()
self._lockfile.remove()
def remove_stage(self, stage):
self._lockfile.remove_stage(stage)
if not self.exists():
return
with open(self.path, "r") as f:
d = parse_yaml_for_update(f.read(), self.path)
self.validate(d, self.path)
if stage.name not in d.get("stages", {}):
return
logger.debug("Removing '%s' from '%s'", stage.name, self.path)
del d["stages"][stage.name]
dump_yaml(self.path, d)
class Lockfile(FileMixin):
from dvc.schema import COMPILED_LOCKFILE_SCHEMA as SCHEMA
@@ -236,25 +243,50 @@ class Lockfile(FileMixin):
if not self.exists():
return {}
with self.repo.tree.open(self.path) as fd:
data = parse_stage(fd.read(), self.path)
data = parse_yaml(fd.read(), self.path)
try:
self.validate(data, fname=self.path)
self.validate(data, fname=self.relpath)
except StageFileFormatError:
raise LockfileCorruptedError(self.path)
raise LockfileCorruptedError(
f"Lockfile '{self.relpath}' is corrupted."
)
return data
def dump(self, stage, **kwargs):
stage_data = serialize.to_lockfile(stage)
if not self.exists():
modified = True
logger.info("Generating lock file '%s'", self.relpath)
data = stage_data
open(self.path, "w+").close()
else:
with self.repo.tree.open(self.path, "r") as fd:
data = parse_stage_for_update(fd.read(), self.path)
data = parse_yaml_for_update(fd.read(), self.path)
modified = data.get(stage.name, {}) != stage_data.get(
stage.name, {}
)
if modified:
logger.info("Updating lock file '%s'", self.relpath)
data.update(stage_data)
dump_yaml(self.path, data)
if modified:
self.repo.scm.track_file(self.relpath)
dump_stage_file(self.path, data)
self.repo.scm.track_file(relpath(self.path))
def remove_stage(self, stage):
if not self.exists():
return
with open(self.path) as f:
d = parse_yaml_for_update(f.read(), self.path)
self.validate(d, self.path)
if stage.name not in d:
return
logger.debug("Removing '%s' from '%s'", stage.name, self.path)
del d[stage.name]
dump_yaml(self.path, d)
class Dvcfile:
+41 -30
View File
@@ -1,7 +1,7 @@
"""Exceptions raised by the dvc."""
from funcy import first
from dvc.utils import relpath, format_link
from dvc.utils import format_link, relpath
class DvcException(Exception):
@@ -34,8 +34,7 @@ class OutputDuplicationError(DvcException):
)
else:
msg = "output '{}' is already specified in stages:\n{}".format(
output,
"\n".join("\t- {}".format(s.addressing) for s in stages),
output, "\n".join(f"\t- {s.addressing}" for s in stages),
)
super().__init__(msg)
self.stages = stages
@@ -102,7 +101,7 @@ class ArgumentDuplicationError(DvcException):
def __init__(self, path):
assert isinstance(path, str)
super().__init__("file '{}' is specified more than once.".format(path))
super().__init__(f"file '{path}' is specified more than once.")
class MoveNotDataSourceError(DvcException):
@@ -158,7 +157,7 @@ class InitError(DvcException):
class ReproductionError(DvcException):
def __init__(self, dvc_file_name):
self.path = dvc_file_name
super().__init__("failed to reproduce '{}'".format(dvc_file_name))
super().__init__(f"failed to reproduce '{dvc_file_name}'")
class BadMetricError(DvcException):
@@ -166,7 +165,7 @@ class BadMetricError(DvcException):
super().__init__(
"the following metrics do not exist, "
"are not metric files or are malformed: {paths}".format(
paths=", ".join("'{}'".format(path) for path in paths)
paths=", ".join(f"'{path}'" for path in paths)
)
)
@@ -174,17 +173,24 @@ class BadMetricError(DvcException):
class NoMetricsError(DvcException):
def __init__(self):
super().__init__(
"no metric files in this repository. "
"Use `dvc metrics add` to add a metric file to track."
"no metric files in this repository. Use `-m/-M` options for "
"`dvc run` to mark stage outputs as metrics."
)
class StageFileCorruptedError(DvcException):
class NoPlotsError(DvcException):
def __init__(self):
super().__init__(
"no plots in this repository. Use `--plots/--plots-no-cache` "
"options for `dvc run` to mark stage outputs as plots."
)
class YAMLFileCorruptedError(DvcException):
def __init__(self, path):
path = relpath(path)
super().__init__(
"unable to read DVC-file: {} "
"YAML file structure is corrupted".format(path)
f"unable to read: '{path}', YAML file structure is corrupted"
)
@@ -204,7 +210,7 @@ class OverlappingOutputPathsError(DvcException):
class CheckoutErrorSuggestGit(DvcException):
def __init__(self, target):
super().__init__("Did you mean `git checkout {}`?".format(target))
super().__init__(f"Did you mean `git checkout {target}`?")
class ETagMismatchError(DvcException):
@@ -216,10 +222,11 @@ class ETagMismatchError(DvcException):
class FileMissingError(DvcException):
def __init__(self, path):
def __init__(self, path, hint=None):
self.path = path
hint = "" if hint is None else f". {hint}"
super().__init__(
"Can't find '{}' neither locally nor on remote".format(path)
f"Can't find '{path}' neither locally nor on remote{hint}"
)
@@ -245,18 +252,14 @@ class DownloadError(DvcException):
def __init__(self, amount):
self.amount = amount
super().__init__(
"{amount} files failed to download".format(amount=amount)
)
super().__init__(f"{amount} files failed to download")
class UploadError(DvcException):
def __init__(self, amount):
self.amount = amount
super().__init__(
"{amount} files failed to upload".format(amount=amount)
)
super().__init__(f"{amount} files failed to upload")
class CheckoutError(DvcException):
@@ -278,7 +281,7 @@ class CollectCacheError(DvcException):
class NoRemoteInExternalRepoError(DvcException):
def __init__(self, url):
super().__init__(
"No DVC remote is specified in target repository '{}'.".format(url)
f"No DVC remote is specified in target repository '{url}'."
)
@@ -293,7 +296,7 @@ class NoOutputInExternalRepoError(DvcException):
class HTTPError(DvcException):
def __init__(self, code, reason):
super().__init__("'{} {}'".format(code, reason))
super().__init__(f"'{code} {reason}'")
class PathMissingError(DvcException):
@@ -301,19 +304,15 @@ class PathMissingError(DvcException):
"The path '{}' does not exist in the target repository '{}'"
" neither as a DVC output nor as a Git-tracked file."
)
default_msg_output_only = (
default_msg_dvc_only = (
"The path '{}' does not exist in the target repository '{}'"
" as an DVC output."
)
def __init__(self, path, repo, output_only=False):
msg = (
self.default_msg
if not output_only
else self.default_msg_output_only
)
def __init__(self, path, repo, dvc_only=False):
msg = self.default_msg if not dvc_only else self.default_msg_dvc_only
super().__init__(msg.format(path, repo))
self.output_only = output_only
self.dvc_only = dvc_only
class RemoteCacheRequiredError(DvcException):
@@ -333,3 +332,15 @@ class RemoteCacheRequiredError(DvcException):
class IsADirectoryError(DvcException):
"""Raised when a file operation is requested on a directory."""
class NoOutputOrStageError(DvcException):
"""
Raised when the target is neither an output nor a stage name in dvc.yaml
"""
def __init__(self, target, file):
super().__init__(
f"'{target}' "
f"does not exist as an output or a stage name in '{file}'"
)
+126 -88
View File
@@ -1,23 +1,27 @@
import logging
import os
import tempfile
import threading
from contextlib import contextmanager
from distutils.dir_util import copy_tree
import threading
from typing import Iterable
from funcy import retry, suppress, wrap_with, cached_property
from funcy import cached_property, retry, wrap_with
from dvc.path_info import PathInfo
from dvc.compat import fspath
from dvc.repo import Repo
from dvc.config import NoRemoteError, NotDvcRepoError
from dvc.exceptions import NoRemoteInExternalRepoError, CheckoutError
from dvc.exceptions import OutputNotFoundError, NoOutputInExternalRepoError
from dvc.exceptions import FileMissingError, PathMissingError
from dvc.utils.fs import remove, fs_copy, move
from dvc.utils import tmp_fname
from dvc.exceptions import (
FileMissingError,
NoOutputInExternalRepoError,
NoRemoteInExternalRepoError,
OutputNotFoundError,
PathMissingError,
)
from dvc.path_info import PathInfo
from dvc.repo import Repo
from dvc.repo.tree import RepoTree
from dvc.scm.git import Git
from dvc.scm.tree import is_working_tree
from dvc.utils.fs import remove
logger = logging.getLogger(__name__)
@@ -26,10 +30,12 @@ logger = logging.getLogger(__name__)
def external_repo(url, rev=None, for_write=False):
logger.debug("Creating external repo %s@%s", url, rev)
path = _cached_clone(url, rev, for_write=for_write)
if not rev:
rev = "HEAD"
try:
repo = ExternalRepo(path, url)
repo = ExternalRepo(path, url, rev, for_write=for_write)
except NotDvcRepoError:
repo = ExternalGitRepo(path, url)
repo = ExternalGitRepo(path, url, rev)
try:
yield repo
@@ -61,59 +67,100 @@ def clean_repos():
_remove(path)
class ExternalRepo(Repo):
def __init__(self, root_dir, url):
super().__init__(root_dir)
class BaseExternalRepo:
_local_cache = None
@property
def local_cache(self):
if hasattr(self, "cache"):
return self.cache.local
return self._local_cache
@contextmanager
def use_cache(self, cache):
"""Use the specified cache in place of default tmpdir cache for
download operations.
"""
if hasattr(self, "cache"):
save_cache = self.cache.local
self.cache.local = cache
self._local_cache = cache
yield
if hasattr(self, "cache"):
self.cache.local = save_cache
self._local_cache = None
@cached_property
def repo_tree(self):
return RepoTree(self, fetch=True)
def get_rev(self):
if is_working_tree(self.tree):
return self.scm.get_rev()
if hasattr(self.tree, "tree"):
return self.tree.tree.rev
return self.tree.rev
def fetch_external(self, paths: Iterable, **kwargs):
"""Fetch specified external repo paths into cache.
Returns 3-tuple in the form
(downloaded, failed, list(cache_infos))
where cache_infos can be used as checkout targets for the
fetched paths.
"""
download_results = []
failed = 0
paths = [PathInfo(self.root_dir) / path for path in paths]
def download_update(result):
download_results.append(result)
save_infos = []
for path in paths:
if not self.repo_tree.exists(path):
raise PathMissingError(path, self.url)
save_info = self.local_cache.save(
path,
self.repo_tree,
None,
save_link=False,
download_callback=download_update,
)
save_infos.append(save_info)
return sum(download_results), failed, save_infos
def get_external(self, path, dest):
"""Convenience wrapper for fetch_external and checkout."""
if self.local_cache:
# fetch DVC and git files to tmpdir cache, then checkout
_, _, save_infos = self.fetch_external([path])
self.local_cache.checkout(PathInfo(dest), save_infos[0])
else:
# git-only erepo with no cache, just copy files directly
# to dest
path = PathInfo(self.root_dir) / path
if not self.repo_tree.exists(path):
raise PathMissingError(path, self.url)
self.repo_tree.copytree(path, dest)
class ExternalRepo(Repo, BaseExternalRepo):
def __init__(self, root_dir, url, rev, for_write=False):
if for_write:
super().__init__(root_dir)
else:
root_dir = os.path.realpath(root_dir)
super().__init__(root_dir, scm=Git(root_dir), rev=rev)
self.url = url
self._set_cache_dir()
self._fix_upstream()
def pull_to(self, path, to_info):
"""
Pull the corresponding file or directory specified by `path` and
checkout it into `to_info`.
It works with files tracked by Git and DVC, and also local files
outside the repository.
"""
out = None
path_info = PathInfo(self.root_dir) / path
with suppress(OutputNotFoundError):
(out,) = self.find_outs_by_path(fspath(path_info), strict=False)
try:
if out and out.use_cache:
self._pull_cached(out, path_info, to_info)
return
# Check if it is handled by Git (it can't have an absolute path)
if os.path.isabs(path):
raise FileNotFoundError
fs_copy(fspath(path_info), fspath(to_info))
except FileNotFoundError:
raise PathMissingError(path, self.url)
def _pull_cached(self, out, path_info, dest):
with self.state:
tmp = PathInfo(tmp_fname(dest))
src = tmp / path_info.relative_to(out.path_info)
out.path_info = tmp
# Only pull unless all needed cache is present
if out.changed_cache(filter_info=src):
self.cloud.pull(out.get_used_cache(filter_info=src))
try:
out.checkout(filter_info=src)
except CheckoutError:
raise FileNotFoundError
move(src, dest)
remove(tmp)
@wrap_with(threading.Lock())
def _set_cache_dir(self):
try:
@@ -122,6 +169,7 @@ class ExternalRepo(Repo):
cache_dir = CACHE_DIRS[self.url] = tempfile.mkdtemp("dvc-cache")
self.cache.local.cache_dir = cache_dir
self._local_cache = self.cache.local
def _fix_upstream(self):
if not os.path.isdir(self.url):
@@ -162,10 +210,11 @@ class ExternalRepo(Repo):
self.config["core"]["remote"] = "auto-generated-upstream"
class ExternalGitRepo:
def __init__(self, root_dir, url):
self.root_dir = root_dir
class ExternalGitRepo(BaseExternalRepo):
def __init__(self, root_dir, url, rev):
self.root_dir = os.path.realpath(root_dir)
self.url = url
self.tree = self.scm.get_tree(rev)
@cached_property
def scm(self):
@@ -178,23 +227,15 @@ class ExternalGitRepo:
def find_out_by_relpath(self, path):
raise OutputNotFoundError(path, self)
def pull_to(self, path, to_info):
try:
# Git handled files can't have absolute path
if os.path.isabs(path):
raise FileNotFoundError
fs_copy(os.path.join(self.root_dir, path), fspath(to_info))
except FileNotFoundError:
raise PathMissingError(path, self.url)
@contextmanager
def open_by_relpath(self, path, mode="r", encoding=None, **kwargs):
"""Opens a specified resource as a file object."""
tree = RepoTree(self)
try:
abs_path = os.path.join(self.root_dir, path)
with open(abs_path, mode, encoding=encoding) as fd:
yield fd
with tree.open(
path, mode=mode, encoding=encoding, **kwargs
) as fobj:
yield fobj
except FileNotFoundError:
raise PathMissingError(path, self.url)
@@ -206,14 +247,12 @@ def _cached_clone(url, rev, for_write=False):
revision checked out. If for_write is set prevents reusing this dir via
cache.
"""
if not for_write and Git.is_sha(rev) and (url, rev) in CLONES:
return CLONES[url, rev]
# even if we have already cloned this repo, we may need to
# fetch/fast-forward to get specified rev
clone_path = _clone_default_branch(url, rev)
rev_sha = Git(clone_path).resolve_rev(rev or "HEAD")
if not for_write and (url, rev_sha) in CLONES:
return CLONES[url, rev_sha]
if not for_write and (url) in CLONES:
return CLONES[url]
# Copy to a new dir to keep the clone clean
repo_path = tempfile.mkdtemp("dvc-erepo")
@@ -221,11 +260,10 @@ def _cached_clone(url, rev, for_write=False):
copy_tree(clone_path, repo_path)
# Check out the specified revision
if rev is not None:
if for_write:
_git_checkout(repo_path, rev)
if not for_write:
CLONES[url, rev_sha] = repo_path
else:
CLONES[url] = repo_path
return repo_path
+123 -25
View File
@@ -1,17 +1,21 @@
import logging
import os
import re
from itertools import groupby
from funcy import cached_property
from pathspec import PathSpec
from pathspec.patterns import GitWildMatchPattern
from pathspec.util import normalize_file
from dvc.path_info import PathInfo
from dvc.scm.tree import BaseTree
from dvc.system import System
from dvc.utils import relpath
logger = logging.getLogger(__name__)
class DvcIgnore(object):
class DvcIgnore:
DVCIGNORE_FILE = ".dvcignore"
def __call__(self, root, dirs, files):
@@ -26,7 +30,16 @@ class DvcIgnorePatterns(DvcIgnore):
self.dirname = os.path.normpath(os.path.dirname(ignore_file_path))
with tree.open(ignore_file_path, encoding="utf-8") as fobj:
self.ignore_spec = PathSpec.from_lines(GitWildMatchPattern, fobj)
path_spec_lines = fobj.readlines()
regex_pattern_list = map(
GitWildMatchPattern.pattern_to_regex, path_spec_lines
)
self.ignore_spec = [
(ignore, re.compile("|".join(item[0] for item in group)))
for ignore, group in groupby(
regex_pattern_list, lambda x: x[1]
)
]
def __call__(self, root, dirs, files):
files = [f for f in files if not self.matches(root, f)]
@@ -35,12 +48,28 @@ class DvcIgnorePatterns(DvcIgnore):
return dirs, files
def matches(self, dirname, basename):
abs_path = os.path.join(dirname, basename)
rel_path = relpath(abs_path, self.dirname)
if os.pardir + os.sep in rel_path:
# NOTE: `relpath` is too slow, so we have to assume that both
# `dirname` and `self.dirname` are relative or absolute together.
prefix = self.dirname + os.sep
if dirname == self.dirname:
path = basename
elif dirname.startswith(prefix):
rel = dirname[len(prefix) :]
# NOTE: `os.path.join` is ~x5.5 slower
path = f"{rel}{os.sep}{basename}"
else:
return False
return self.ignore_spec.match_file(rel_path)
if not System.is_unix():
path = normalize_file(path)
return self.ignore(path)
def ignore(self, path):
result = False
for ignore, pattern in self.ignore_spec:
if pattern.match(path):
result = ignore
return result
def __hash__(self):
return hash(self.ignore_file_path)
@@ -76,21 +105,22 @@ class DvcIgnoreRepo(DvcIgnore):
def is_dvc_repo(directory):
from dvc.repo import Repo
return os.path.isdir(os.path.join(directory, Repo.DVC_DIR))
return os.path.isdir(os.path.join(root, directory, Repo.DVC_DIR))
dirs = [d for d in dirs if not is_dvc_repo(d)]
return dirs, files
class DvcIgnoreFilter(object):
def __init__(self, tree):
class DvcIgnoreFilter:
def __init__(self, tree, root_dir):
self.tree = tree
self.root_dir = root_dir
self.ignores = {
DvcIgnoreDirs([".git", ".hg", ".dvc"]),
DvcIgnoreRepo(),
}
for root, dirs, files in self.tree.walk(self.tree.tree_root):
for root, dirs, files in self.tree.walk(self.root_dir):
self._update(root)
dirs[:], files[:] = self(root, dirs, files)
@@ -107,39 +137,107 @@ class DvcIgnoreFilter(object):
class CleanTree(BaseTree):
def __init__(self, tree):
def __init__(self, tree, tree_root=None):
self.tree = tree
if tree_root:
self._tree_root = tree_root
else:
self._tree_root = self.tree.tree_root
@cached_property
def dvcignore(self):
return DvcIgnoreFilter(self.tree)
return DvcIgnoreFilter(self.tree, self.tree_root)
@property
def tree_root(self):
return self.tree.tree_root
return self._tree_root
def open(self, path, mode="r", encoding="utf-8"):
return self.tree.open(path, mode, encoding)
if self.isfile(path):
return self.tree.open(path, mode, encoding)
raise FileNotFoundError
def exists(self, path):
return self.tree.exists(path)
if self.tree.exists(path) and self._parents_exist(path):
if self.tree.isdir(path):
return self._valid_dirname(path)
return self._valid_filename(path)
return False
def isdir(self, path):
return self.tree.isdir(path)
return (
self.tree.isdir(path)
and self._parents_exist(path)
and self._valid_dirname(path)
)
def _valid_dirname(self, path):
path = os.path.abspath(path)
if path == self.tree_root:
return True
dirname, basename = os.path.split(path)
dirs, _ = self.dvcignore(dirname, [basename], [])
if dirs:
return True
return False
def isfile(self, path):
return self.tree.isfile(path)
return (
self.tree.isfile(path)
and self._parents_exist(path)
and self._valid_filename(path)
)
def _valid_filename(self, path):
dirname, basename = os.path.split(os.path.normpath(path))
_, files = self.dvcignore(os.path.abspath(dirname), [], [basename])
if files:
return True
return False
def isexec(self, path):
return self.tree.isexec(path)
return self.exists(path) and self.tree.isexec(path)
def _parents_exist(self, path):
from dvc.repo import Repo
path = PathInfo(path)
# if parent is tree_root or inside a .dvc dir we can skip this check
if path.parent == self.tree_root or Repo.DVC_DIR in path.parts:
return True
# paths outside of the CleanTree root should be ignored
path = relpath(path, self.tree_root)
if path.startswith("..") or (
os.name == "nt"
and not os.path.commonprefix(
[os.path.abspath(path), self.tree_root]
)
):
return False
# check if parent directories are in our ignores, starting from
# tree_root
for parent_dir in reversed(PathInfo(path).parents):
dirname, basename = os.path.split(parent_dir)
if basename == ".":
# parent_dir == tree_root
continue
dirs, _ = self.dvcignore(os.path.abspath(dirname), [basename], [])
if not dirs:
return False
return True
def walk(self, top, topdown=True):
for root, dirs, files in self.tree.walk(top, topdown):
dirs[:], files[:] = self.dvcignore(root, dirs, files)
dirs[:], files[:] = self.dvcignore(
os.path.abspath(root), dirs, files
)
yield root, dirs, files
def walk_files(self, top):
for root, _, files in self.walk(top):
for file in files:
yield os.path.join(root, file)
def stat(self, path):
if self.exists(path):
return self.tree.stat(path)
raise FileNotFoundError
+6 -2
View File
@@ -7,13 +7,17 @@
TEXT_CHARS = bytes(range(32, 127)) + b"\n\r\t\f\b"
def istextfile(fname, blocksize=512):
def istextfile(fname, blocksize=512, tree=None):
""" Uses heuristics to guess whether the given file is text or binary,
by reading a single block of bytes from the file.
If more than 30% of the chars in the block are non-text, or there
are NUL ('\x00') bytes in the block, assume this is a binary file.
"""
with open(fname, "rb") as fobj:
if tree:
open_func = tree.open
else:
open_func = open
with open_func(fname, "rb") as fobj:
block = fobj.read(blocksize)
if not block:
+4 -4
View File
@@ -5,12 +5,12 @@ import os
import time
from datetime import timedelta
import zc.lockfile
import flufl.lock
import zc.lockfile
from dvc.exceptions import DvcException
from dvc.utils import format_link
from dvc.progress import Tqdm
from dvc.utils import format_link
DEFAULT_TIMEOUT = 5
@@ -25,7 +25,7 @@ class LockError(DvcException):
"""Thrown when unable to acquire the lock for DVC repo."""
class Lock(object):
class Lock:
"""Class for DVC repo lock.
Uses zc.lockfile as backend.
@@ -47,7 +47,7 @@ class Lock(object):
disable=not self._friendly,
desc=(
"If DVC froze, see `hardlink_lock` in {}".format(
format_link("man.dvc.org/config#core")
format_link("https://man.dvc.org/config#core")
)
),
):
+50 -11
View File
@@ -7,7 +7,6 @@ import colorama
from dvc.progress import Tqdm
FOOTER = (
"\n{yellow}Having any troubles?{nc}"
" Hit us up at {blue}https://dvc.org/support{nc},"
@@ -19,20 +18,45 @@ FOOTER = (
)
def addLoggingLevel(levelName, levelNum, methodName=None):
"""
Adds a new logging level to the `logging` module and the
currently configured logging class.
Based on https://stackoverflow.com/questions/2183233
"""
if methodName is None:
methodName = levelName.lower()
assert not hasattr(logging, levelName)
assert not hasattr(logging, methodName)
assert not hasattr(logging.getLoggerClass(), methodName)
def logForLevel(self, message, *args, **kwargs):
if self.isEnabledFor(levelNum):
self._log(levelNum, message, args, **kwargs)
def logToRoot(message, *args, **kwargs):
logging.log(levelNum, message, *args, **kwargs)
logging.addLevelName(levelNum, levelName)
setattr(logging, levelName, levelNum)
setattr(logging.getLoggerClass(), methodName, logForLevel)
setattr(logging, methodName, logToRoot)
class LoggingException(Exception):
def __init__(self, record):
msg = "failed to log {}".format(str(record))
super().__init__(msg)
class ExcludeErrorsFilter(logging.Filter):
def filter(self, record):
return record.levelno < logging.WARNING
def excludeFilter(level):
class ExcludeLevelFilter(logging.Filter):
def filter(self, record):
return record.levelno < level
class ExcludeInfoFilter(logging.Filter):
def filter(self, record):
return record.levelno < logging.INFO
return ExcludeLevelFilter
class ColorFormatter(logging.Formatter):
@@ -48,6 +72,7 @@ class ColorFormatter(logging.Formatter):
"""
color_code = {
"TRACE": colorama.Fore.GREEN,
"DEBUG": colorama.Fore.BLUE,
"WARNING": colorama.Fore.YELLOW,
"ERROR": colorama.Fore.RED,
@@ -117,7 +142,11 @@ class LoggerHandler(logging.StreamHandler):
def _is_verbose():
return logging.getLogger("dvc").getEffectiveLevel() == logging.DEBUG
return (
logging.NOTSET
< logging.getLogger("dvc").getEffectiveLevel()
<= logging.DEBUG
)
def _iter_causes(exc):
@@ -153,12 +182,14 @@ def disable_other_loggers():
def setup(level=logging.INFO):
colorama.init()
addLoggingLevel("TRACE", logging.DEBUG - 5)
logging.config.dictConfig(
{
"version": 1,
"filters": {
"exclude_errors": {"()": ExcludeErrorsFilter},
"exclude_info": {"()": ExcludeInfoFilter},
"exclude_errors": {"()": excludeFilter(logging.WARNING)},
"exclude_info": {"()": excludeFilter(logging.INFO)},
"exclude_debug": {"()": excludeFilter(logging.DEBUG)},
},
"formatters": {"color": {"()": ColorFormatter}},
"handlers": {
@@ -176,6 +207,13 @@ def setup(level=logging.INFO):
"stream": "ext://sys.stdout",
"filters": ["exclude_info"],
},
"console_trace": {
"class": "dvc.logger.LoggerHandler",
"level": "TRACE",
"formatter": "color",
"stream": "ext://sys.stdout",
"filters": ["exclude_debug"],
},
"console_errors": {
"class": "dvc.logger.LoggerHandler",
"level": "WARNING",
@@ -189,6 +227,7 @@ def setup(level=logging.INFO):
"handlers": [
"console_info",
"console_debug",
"console_trace",
"console_errors",
],
},
+14 -10
View File
@@ -6,12 +6,11 @@ import logging
from dvc import analytics
from dvc.cli import parse_args
from dvc.config import ConfigError
from dvc.exceptions import DvcParserError, DvcException
from dvc.exceptions import NotDvcRepoError
from dvc.exceptions import DvcException, DvcParserError, NotDvcRepoError
from dvc.external_repo import clean_repos
from dvc.logger import disable_other_loggers, FOOTER
from dvc.utils import format_link
from dvc.logger import FOOTER, disable_other_loggers
from dvc.remote.pool import close_pools
from dvc.utils import format_link
# Workaround for CPython bug. See [1] and [2] for more info.
# [1] https://github.com/aws/aws-cli/blob/1.16.277/awscli/clidriver.py#L55
@@ -19,7 +18,6 @@ from dvc.remote.pool import close_pools
"".encode("idna")
logger = logging.getLogger("dvc")
@@ -39,11 +37,17 @@ def main(argv=None):
try:
args = parse_args(argv)
if args.quiet:
logger.setLevel(logging.CRITICAL)
elif args.verbose:
logger.setLevel(logging.DEBUG)
verbosity = args.verbose - args.quiet
if verbosity:
logger.setLevel(
{
-2: logging.CRITICAL,
-1: logging.ERROR,
1: logging.DEBUG,
2: logging.TRACE,
}[max(-2, min(verbosity, 2))]
)
logger.trace(args)
cmd = args.func(args)
ret = cmd.run()
+83 -24
View File
@@ -1,5 +1,8 @@
from collections import defaultdict
from urllib.parse import urlparse
from voluptuous import Any, Required, Lower, Length, Coerce, And, SetTo
from funcy import collecting, project
from voluptuous import And, Any, Coerce, Length, Lower, Required, SetTo
from dvc.output.base import BaseOutput
from dvc.output.gs import GSOutput
@@ -7,10 +10,10 @@ from dvc.output.hdfs import HDFSOutput
from dvc.output.local import LocalOutput
from dvc.output.s3 import S3Output
from dvc.output.ssh import SSHOutput
from dvc.remote import Remote
from dvc.remote.hdfs import HDFSRemote
from dvc.remote.local import LocalRemote
from dvc.remote.s3 import S3Remote
from dvc.remote import get_remote
from dvc.remote.hdfs import HDFSRemoteTree
from dvc.remote.local import LocalRemoteTree
from dvc.remote.s3 import S3RemoteTree
from dvc.scheme import Schemes
OUTS = [
@@ -44,23 +47,26 @@ CHECKSUM_SCHEMA = Any(
# so when a few types of outputs share the same name, we only need
# specify it once.
CHECKSUMS_SCHEMA = {
LocalRemote.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
S3Remote.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
HDFSRemote.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
LocalRemoteTree.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
S3RemoteTree.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
HDFSRemoteTree.PARAM_CHECKSUM: CHECKSUM_SCHEMA,
}
SCHEMA = CHECKSUMS_SCHEMA.copy()
SCHEMA[Required(BaseOutput.PARAM_PATH)] = str
SCHEMA[BaseOutput.PARAM_CACHE] = bool
SCHEMA[BaseOutput.PARAM_METRIC] = BaseOutput.METRIC_SCHEMA
SCHEMA[BaseOutput.PARAM_PLOT] = bool
SCHEMA[BaseOutput.PARAM_PERSIST] = bool
def _get(stage, p, info, cache, metric, persist=False):
def _get(
stage, p, info=None, cache=True, metric=False, plot=False, persist=False
):
parsed = urlparse(p)
if parsed.scheme == "remote":
remote = Remote(stage.repo, name=parsed.netloc)
remote = get_remote(stage.repo, name=parsed.netloc)
return OUTS_MAP[remote.scheme](
stage,
p,
@@ -68,6 +74,7 @@ def _get(stage, p, info, cache, metric, persist=False):
cache=cache,
remote=remote,
metric=metric,
plot=plot,
persist=persist,
)
@@ -80,6 +87,7 @@ def _get(stage, p, info, cache, metric, persist=False):
cache=cache,
remote=None,
metric=metric,
plot=plot,
persist=persist,
)
return LocalOutput(
@@ -89,6 +97,7 @@ def _get(stage, p, info, cache, metric, persist=False):
cache=cache,
remote=None,
metric=metric,
plot=plot,
persist=persist,
)
@@ -99,26 +108,76 @@ def loadd_from(stage, d_list):
p = d.pop(BaseOutput.PARAM_PATH)
cache = d.pop(BaseOutput.PARAM_CACHE, True)
metric = d.pop(BaseOutput.PARAM_METRIC, False)
plot = d.pop(BaseOutput.PARAM_PLOT, False)
persist = d.pop(BaseOutput.PARAM_PERSIST, False)
ret.append(
_get(
stage, p, info=d, cache=cache, metric=metric, persist=persist,
)
)
return ret
def loads_from(stage, s_list, use_cache=True, metric=False, persist=False):
ret = []
for s in s_list:
ret.append(
_get(
stage,
s,
info={},
cache=use_cache,
p,
info=d,
cache=cache,
metric=metric,
plot=plot,
persist=persist,
)
)
return ret
def loads_from(
stage, s_list, use_cache=True, metric=False, plot=False, persist=False
):
return [
_get(
stage,
s,
info={},
cache=use_cache,
metric=metric,
plot=plot,
persist=persist,
)
for s in s_list
]
def _split_dict(d, keys):
return project(d, keys), project(d, d.keys() - keys)
def _merge_data(s_list):
d = defaultdict(dict)
for key in s_list:
if isinstance(key, str):
d[key].update({})
continue
if not isinstance(key, dict):
raise ValueError(f"'{type(key).__name__}' not supported.")
for k, flags in key.items():
if not isinstance(flags, dict):
raise ValueError(
f"Expected dict for '{k}', got: '{type(flags).__name__}'"
)
d[k].update(flags)
return d
@collecting
def load_from_pipeline(stage, s_list, typ="outs"):
if typ not in (stage.PARAM_OUTS, stage.PARAM_METRICS, stage.PARAM_PLOTS):
raise ValueError(f"'{typ}' key is not allowed for pipeline files.")
metric = typ == stage.PARAM_METRICS
plot = typ == stage.PARAM_PLOTS
d = _merge_data(s_list)
for path, flags in d.items():
plt_d = {}
if plot:
from dvc.schema import PLOT_PROPS
plt_d, flags = _split_dict(flags, keys=PLOT_PROPS.keys())
extra = project(flags, ["cache", "persist"])
yield _get(stage, path, {}, plot=plt_d or plot, metric=metric, **extra)
+88 -60
View File
@@ -1,53 +1,68 @@
import logging
import os
from urllib.parse import urlparse
from copy import copy
from urllib.parse import urlparse
from voluptuous import Any
import dvc.prompt as prompt
from dvc.cache import NamedCache
from dvc.exceptions import CollectCacheError, RemoteCacheRequiredError
from dvc.exceptions import DvcException
from dvc.remote.base import BaseRemote
from dvc.exceptions import (
CollectCacheError,
DvcException,
RemoteCacheRequiredError,
)
from dvc.remote.base import BaseRemoteTree, Remote
logger = logging.getLogger(__name__)
class OutputDoesNotExistError(DvcException):
def __init__(self, path):
msg = "output '{}' does not exist".format(path)
msg = f"output '{path}' does not exist"
super().__init__(msg)
class OutputIsNotFileOrDirError(DvcException):
def __init__(self, path):
msg = "output '{}' is not a file or directory".format(path)
msg = f"output '{path}' is not a file or directory"
super().__init__(msg)
class OutputAlreadyTrackedError(DvcException):
def __init__(self, path):
msg = "output '{}' is already tracked by SCM (e.g. Git)".format(path)
msg = f""" output '{path}' is already tracked by SCM (e.g. Git).
You can remove it from Git, then add to DVC.
To stop tracking from Git:
git rm -r --cached '{path}'
git commit -m "stop tracking {path}" """
super().__init__(msg)
class OutputIsStageFileError(DvcException):
def __init__(self, path):
super().__init__("Stage file '{}' cannot be an output.".format(path))
super().__init__(f"Stage file '{path}' cannot be an output.")
class BaseOutput(object):
class BaseOutput:
IS_DEPENDENCY = False
REMOTE = BaseRemote
REMOTE_CLS = Remote
TREE_CLS = BaseRemoteTree
PARAM_PATH = "path"
PARAM_CACHE = "cache"
PARAM_METRIC = "metric"
PARAM_METRIC_TYPE = "type"
PARAM_METRIC_XPATH = "xpath"
PARAM_PLOT = "plot"
PARAM_PLOT_TEMPLATE = "template"
PARAM_PLOT_X = "x"
PARAM_PLOT_Y = "y"
PARAM_PLOT_X_LABEL = "x_label"
PARAM_PLOT_Y_LABEL = "y_label"
PARAM_PLOT_TITLE = "title"
PARAM_PLOT_HEADER = "header"
PARAM_PERSIST = "persist"
METRIC_SCHEMA = Any(
@@ -73,6 +88,7 @@ class BaseOutput(object):
remote=None,
cache=True,
metric=False,
plot=False,
persist=False,
):
self._validate_output_path(path)
@@ -90,9 +106,14 @@ class BaseOutput(object):
self.repo = stage.repo if stage else None
self.def_path = path
self.info = info
self.remote = remote or self.REMOTE(self.repo, {})
if remote:
self.remote = remote
else:
tree = self.TREE_CLS(self.repo, {})
self.remote = self.REMOTE_CLS(tree)
self.use_cache = False if self.IS_DEPENDENCY else cache
self.metric = False if self.IS_DEPENDENCY else metric
self.plot = False if self.IS_DEPENDENCY else plot
self.persist = persist
self.path_info = self._parse_path(remote, path)
@@ -103,7 +124,7 @@ class BaseOutput(object):
if remote:
parsed = urlparse(path)
return remote.path_info / parsed.path.lstrip("/")
return self.REMOTE.path_cls(path)
return self.TREE_CLS.PATH_CLS(path)
def __repr__(self):
return "{class_name}: '{def_path}'".format(
@@ -115,7 +136,7 @@ class BaseOutput(object):
@property
def scheme(self):
return self.REMOTE.scheme
return self.TREE_CLS.scheme
@property
def is_in_repo(self):
@@ -138,34 +159,34 @@ class BaseOutput(object):
@classmethod
def supported(cls, url):
return cls.REMOTE.supported(url)
return cls.TREE_CLS.supported(url)
@property
def cache_path(self):
return self.cache.checksum_to_path_info(self.checksum).url
return self.cache.hash_to_path_info(self.checksum).url
@property
def checksum_type(self):
return self.remote.PARAM_CHECKSUM
return self.remote.tree.PARAM_CHECKSUM
@property
def checksum(self):
return self.info.get(self.remote.PARAM_CHECKSUM)
return self.info.get(self.remote.tree.PARAM_CHECKSUM)
@checksum.setter
def checksum(self, checksum):
self.info[self.remote.PARAM_CHECKSUM] = checksum
self.info[self.remote.tree.PARAM_CHECKSUM] = checksum
def get_checksum(self):
return self.remote.get_checksum(self.path_info)
return self.remote.get_hash(self.path_info)
@property
def is_dir_checksum(self):
return self.remote.is_dir_checksum(self.checksum)
return self.remote.is_dir_hash(self.checksum)
@property
def exists(self):
return self.remote.exists(self.path_info)
return self.remote.tree.exists(self.path_info)
def save_info(self):
return self.remote.save_info(self.path_info)
@@ -201,13 +222,13 @@ class BaseOutput(object):
@property
def is_empty(self):
return self.remote.is_empty(self.path_info)
return self.remote.tree.is_empty(self.path_info)
def isdir(self):
return self.remote.isdir(self.path_info)
return self.remote.tree.isdir(self.path_info)
def isfile(self):
return self.remote.isfile(self.path_info)
return self.remote.tree.isfile(self.path_info)
def ignore(self):
if not self.use_scm_ignore:
@@ -226,35 +247,32 @@ class BaseOutput(object):
raise self.IsNotFileOrDirError(self)
if self.is_empty:
logger.warning("'{}' is empty.".format(self))
logger.warning(f"'{self}' is empty.")
self.ignore()
if self.metric or self.plot:
self.verify_metric()
if not self.use_cache:
self.info = self.save_info()
if self.metric:
self.verify_metric()
if not self.IS_DEPENDENCY:
logger.info(
"Output '{}' doesn't use cache. Skipping saving.".format(
self
)
logger.debug(
"Output '%s' doesn't use cache. Skipping saving.", self
)
return
assert not self.IS_DEPENDENCY
if not self.changed():
logger.info(
"Output '{}' didn't change. Skipping saving.".format(self)
)
logger.debug("Output '%s' didn't change. Skipping saving.", self)
return
self.info = self.save_info()
def commit(self):
if self.use_cache:
self.cache.save(self.path_info, self.info)
self.cache.save(self.path_info, self.cache.tree, self.info)
def dumpd(self):
ret = copy(self.info)
@@ -263,7 +281,8 @@ class BaseOutput(object):
if self.IS_DEPENDENCY:
return ret
ret[self.PARAM_CACHE] = self.use_cache
if not self.use_cache:
ret[self.PARAM_CACHE] = self.use_cache
if isinstance(self.metric, dict):
if (
@@ -272,18 +291,22 @@ class BaseOutput(object):
):
del self.metric[self.PARAM_METRIC_XPATH]
ret[self.PARAM_METRIC] = self.metric
ret[self.PARAM_PERSIST] = self.persist
if self.metric:
ret[self.PARAM_METRIC] = self.metric
if self.plot:
ret[self.PARAM_PLOT] = self.plot
if self.persist:
ret[self.PARAM_PERSIST] = self.persist
return ret
def verify_metric(self):
raise DvcException(
"verify metric is not supported for {}".format(self.scheme)
)
raise DvcException(f"verify metric is not supported for {self.scheme}")
def download(self, to):
self.remote.download(self.path_info, to.path_info)
self.remote.tree.download(self.path_info, to.path_info)
def checkout(
self,
@@ -309,7 +332,7 @@ class BaseOutput(object):
)
def remove(self, ignore_remove=False):
self.remote.remove(self.path_info)
self.remote.tree.remove(self.path_info)
if self.scheme != "local":
return
@@ -320,7 +343,7 @@ class BaseOutput(object):
if self.scheme == "local" and self.use_scm_ignore:
self.repo.scm.ignore_remove(self.fspath)
self.remote.move(self.path_info, out.path_info)
self.remote.tree.move(self.path_info, out.path_info)
self.def_path = out.def_path
self.path_info = out.path_info
self.save()
@@ -339,9 +362,20 @@ class BaseOutput(object):
def unprotect(self):
if self.exists:
self.remote.unprotect(self.path_info)
self.remote.tree.unprotect(self.path_info)
def _collect_used_dir_cache(
def get_dir_cache(self, **kwargs):
if not self.is_dir_checksum:
raise DvcException("cannot get dir cache for file checksum")
if self.cache.changed_cache_file(self.checksum):
self.repo.cloud.pull(
NamedCache.make("local", self.checksum, str(self)),
show_checksums=False,
**kwargs,
)
return self.dir_cache
def collect_used_dir_cache(
self, remote=None, force=False, jobs=None, filter_info=None
):
"""Get a list of `info`s related to the given directory.
@@ -365,16 +399,10 @@ class BaseOutput(object):
cache = NamedCache()
if self.cache.changed_cache_file(self.checksum):
try:
self.repo.cloud.pull(
NamedCache.make("local", self.checksum, str(self)),
jobs=jobs,
remote=remote,
show_checksums=False,
)
except DvcException:
logger.debug("failed to pull cache for '{}'".format(self))
try:
self.get_dir_cache(jobs=jobs, remote=remote)
except DvcException:
logger.debug(f"failed to pull cache for '{self}'")
if self.cache.changed_cache_file(self.checksum):
msg = (
@@ -394,8 +422,8 @@ class BaseOutput(object):
filter_path = str(filter_info) if filter_info else None
is_win = os.name == "nt"
for entry in self.dir_cache:
checksum = entry[self.remote.PARAM_CHECKSUM]
entry_relpath = entry[self.remote.PARAM_RELPATH]
checksum = entry[self.remote.tree.PARAM_CHECKSUM]
entry_relpath = entry[self.remote.tree.PARAM_RELPATH]
if is_win:
entry_relpath = entry_relpath.replace("/", os.sep)
entry_path = os.path.join(path, entry_relpath)
@@ -451,7 +479,7 @@ class BaseOutput(object):
return ret
ret.add_child_cache(
self.checksum, self._collect_used_dir_cache(**kwargs),
self.checksum, self.collect_used_dir_cache(**kwargs),
)
return ret
+2 -2
View File
@@ -1,6 +1,6 @@
from dvc.output.s3 import S3Output
from dvc.remote.gs import GSRemote
from dvc.remote.gs import GSRemoteTree
class GSOutput(S3Output):
REMOTE = GSRemote
TREE_CLS = GSRemoteTree
+2 -2
View File
@@ -1,6 +1,6 @@
from dvc.output.base import BaseOutput
from dvc.remote.hdfs import HDFSRemote
from dvc.remote.hdfs import HDFSRemoteTree
class HDFSOutput(BaseOutput):
REMOTE = HDFSRemote
TREE_CLS = HDFSRemoteTree
+15 -15
View File
@@ -5,17 +5,16 @@ from urllib.parse import urlparse
from dvc.exceptions import DvcException
from dvc.istextfile import istextfile
from dvc.output.base import BaseOutput
from dvc.remote.local import LocalRemote
from dvc.remote.local import LocalRemote, LocalRemoteTree
from dvc.utils import relpath
from dvc.compat import fspath_py35
from dvc.utils.fs import path_isin
logger = logging.getLogger(__name__)
class LocalOutput(BaseOutput):
REMOTE = LocalRemote
REMOTE_CLS = LocalRemote
TREE_CLS = LocalRemoteTree
sep = os.sep
def __init__(self, stage, path, *args, **kwargs):
@@ -35,15 +34,15 @@ class LocalOutput(BaseOutput):
#
# FIXME: if we have Windows path containing / or posix one with \
# then we have #2059 bug and can't really handle that.
p = self.REMOTE.path_cls(path)
if not p.is_absolute():
p = self.TREE_CLS.PATH_CLS(path)
if self.stage and not p.is_absolute():
p = self.stage.wdir / p
abs_p = os.path.abspath(os.path.normpath(fspath_py35(p)))
return self.REMOTE.path_cls(abs_p)
abs_p = os.path.abspath(os.path.normpath(p))
return self.TREE_CLS.PATH_CLS(abs_p)
def __str__(self):
if not self.is_in_repo:
if not self.repo or not self.is_in_repo:
return str(self.def_path)
cur_dir = os.getcwd()
@@ -73,17 +72,18 @@ class LocalOutput(BaseOutput):
return ret
def verify_metric(self):
if not self.metric:
if not self.metric or self.plot:
return
path = fspath_py35(self.path_info)
path = os.fspath(self.path_info)
if not os.path.exists(path):
return
name = "metrics" if self.metric else "plot"
if os.path.isdir(path):
msg = "directory '{}' cannot be used as metrics."
raise DvcException(msg.format(self.path_info))
msg = "directory '{}' cannot be used as {}."
raise IsADirectoryError(msg.format(self.path_info, name))
if not istextfile(path):
msg = "binary file '{}' cannot be used as metrics."
raise DvcException(msg.format(self.path_info))
msg = "binary file '{}' cannot be used as {}."
raise DvcException(msg.format(self.path_info, name))
+2 -2
View File
@@ -1,6 +1,6 @@
from dvc.output.base import BaseOutput
from dvc.remote.s3 import S3Remote
from dvc.remote.s3 import S3RemoteTree
class S3Output(BaseOutput):
REMOTE = S3Remote
TREE_CLS = S3RemoteTree
+3 -2
View File
@@ -1,6 +1,7 @@
from dvc.output.base import BaseOutput
from dvc.remote.ssh import SSHRemote
from dvc.remote.ssh import SSHRemote, SSHRemoteTree
class SSHOutput(BaseOutput):
REMOTE = SSHRemote
REMOTE_CLS = SSHRemote
TREE_CLS = SSHRemoteTree
+8 -9
View File
@@ -1,6 +1,6 @@
import os
import posixpath
import pathlib
import posixpath
from urllib.parse import urlparse
from funcy import cached_property
@@ -8,7 +8,7 @@ from funcy import cached_property
from dvc.utils import relpath
class _BasePath(object):
class _BasePath:
def overlaps(self, other):
if isinstance(other, (str, bytes)):
other = self.__class__(other)
@@ -50,7 +50,6 @@ class PathInfo(pathlib.PurePath, _BasePath):
return "{}: '{}'".format(type(self).__name__, self)
# This permits passing it to file utils directly in Python 3.6+
# With Python 2.7, Python 3.5+ we are stuck with path_info.fspath for now
def __fspath__(self):
return pathlib.PurePath.__str__(self)
@@ -88,7 +87,7 @@ class _URLPathInfo(PosixPathInfo):
__unicode__ = __str__
class _URLPathParents(object):
class _URLPathParents:
def __init__(self, src):
self.src = src
self._parents = self.src._path.parents
@@ -100,7 +99,7 @@ class _URLPathParents(object):
return self.src.replace(path=self._parents[idx])
def __repr__(self):
return "<{}.parents>".format(self.src)
return f"<{self.src}.parents>"
class URLInfo(_BasePath):
@@ -120,7 +119,7 @@ class URLInfo(_BasePath):
assert bool(host) ^ bool(netloc)
if netloc is not None:
return cls("{}://{}{}".format(scheme, netloc, path))
return cls(f"{scheme}://{netloc}{path}")
obj = cls.__new__(cls)
obj._fill_parts(scheme, host, user, port, path)
@@ -154,7 +153,7 @@ class URLInfo(_BasePath):
@cached_property
def url(self):
return "{}://{}{}".format(self.scheme, self.netloc, self._spath)
return f"{self.scheme}://{self.netloc}{self._spath}"
def __str__(self):
return self.url
@@ -216,10 +215,10 @@ class URLInfo(_BasePath):
if isinstance(other, (str, bytes)):
other = self.__class__(other)
if self.__class__ != other.__class__:
msg = "'{}' has incompatible class with '{}'".format(self, other)
msg = f"'{self}' has incompatible class with '{other}'"
raise ValueError(msg)
if self._base_parts != other._base_parts:
msg = "'{}' does not start with '{}'".format(self, other)
msg = f"'{self}' does not start with '{other}'"
raise ValueError(msg)
return self._path.relative_to(other._path)
+2 -2
View File
@@ -38,7 +38,7 @@ def confirm(statement):
Returns:
bool: whether or not specified statement was confirmed.
"""
prompt = "{statement} [y/n]".format(statement=statement)
prompt = f"{statement} [y/n]"
answer = _ask(prompt, limited_to=["yes", "no", "y", "n"])
return answer and answer.startswith("y")
@@ -52,5 +52,5 @@ def password(statement):
Returns:
str: password entered by the user.
"""
logger.info("{statement}: ".format(statement=statement))
logger.info(f"{statement}: ")
return getpass("")
+43 -30
View File
@@ -1,47 +1,46 @@
import posixpath
from urllib.parse import urlparse
from dvc.remote.azure import AzureRemote
from dvc.remote.gdrive import GDriveRemote
from dvc.remote.gs import GSRemote
from dvc.remote.hdfs import HDFSRemote
from dvc.remote.http import HTTPRemote
from dvc.remote.https import HTTPSRemote
from dvc.remote.local import LocalRemote
from dvc.remote.oss import OSSRemote
from dvc.remote.s3 import S3Remote
from dvc.remote.ssh import SSHRemote
from dvc.remote.azure import AzureRemoteTree
from dvc.remote.base import Remote
from dvc.remote.gdrive import GDriveRemoteTree
from dvc.remote.gs import GSRemoteTree
from dvc.remote.hdfs import HDFSRemoteTree
from dvc.remote.http import HTTPRemoteTree
from dvc.remote.https import HTTPSRemoteTree
from dvc.remote.local import LocalRemote, LocalRemoteTree
from dvc.remote.oss import OSSRemoteTree
from dvc.remote.s3 import S3RemoteTree
from dvc.remote.ssh import SSHRemote, SSHRemoteTree
REMOTES = [
AzureRemote,
GDriveRemote,
GSRemote,
HDFSRemote,
HTTPRemote,
HTTPSRemote,
S3Remote,
SSHRemote,
OSSRemote,
# NOTE: LocalRemote is the default
TREES = [
AzureRemoteTree,
GDriveRemoteTree,
GSRemoteTree,
HDFSRemoteTree,
HTTPRemoteTree,
HTTPSRemoteTree,
S3RemoteTree,
SSHRemoteTree,
OSSRemoteTree,
# NOTE: LocalRemoteTree is the default
]
def _get(remote_conf):
for remote in REMOTES:
if remote.supported(remote_conf):
return remote
return LocalRemote
def _get_tree(remote_conf):
for tree_cls in TREES:
if tree_cls.supported(remote_conf):
return tree_cls
return LocalRemoteTree
def Remote(repo, **kwargs):
def _get_conf(repo, **kwargs):
name = kwargs.get("name")
if name:
remote_conf = repo.config["remote"][name.lower()]
else:
remote_conf = kwargs
remote_conf = _resolve_remote_refs(repo.config, remote_conf)
return _get(remote_conf)(repo, remote_conf)
return _resolve_remote_refs(repo.config, remote_conf)
def _resolve_remote_refs(config, remote_conf):
@@ -73,3 +72,17 @@ def _resolve_remote_refs(config, remote_conf):
base = config["remote"][parsed.netloc]
url = posixpath.join(base["url"], parsed.path.lstrip("/"))
return {**base, **remote_conf, "url": url}
def get_cloud_tree(repo, **kwargs):
remote_conf = _get_conf(repo, **kwargs)
return _get_tree(remote_conf)(repo, remote_conf)
def get_remote(repo, **kwargs):
tree = get_cloud_tree(repo, **kwargs)
if tree.scheme == "local":
return LocalRemote(tree)
if tree.scheme == "ssh":
return SSHRemote(tree)
return Remote(tree)
+56 -78
View File
@@ -1,31 +1,21 @@
import logging
import os
import posixpath
import re
from datetime import datetime, timedelta
from urllib.parse import urlparse
import threading
from datetime import datetime, timedelta
from funcy import cached_property, wrap_prop
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
class AzureRemote(BaseRemote):
class AzureRemoteTree(BaseRemoteTree):
scheme = Schemes.AZURE
path_cls = CloudURLInfo
REGEX = (
r"azure://((?P<path>[^=;]*)?|("
# backward compatibility
r"(ContainerName=(?P<container_name>[^;]+);?)?"
r"(?P<connection_string>.+)?)?)$"
)
PATH_CLS = CloudURLInfo
REQUIRES = {"azure-storage-blob": "azure.storage.blob"}
PARAM_CHECKSUM = "etag"
COPY_POLL_SECONDS = 5
@@ -35,31 +25,14 @@ class AzureRemote(BaseRemote):
super().__init__(repo, config)
url = config.get("url", "azure://")
self.path_info = self.PATH_CLS(url)
match = re.match(self.REGEX, url) # backward compatibility
path = match.group("path")
bucket = (
urlparse(url if path else "").netloc
or match.group("container_name") # backward compatibility
or os.getenv("AZURE_STORAGE_CONTAINER_NAME")
)
if not self.path_info.bucket:
container = os.getenv("AZURE_STORAGE_CONTAINER_NAME")
self.path_info = self.PATH_CLS(f"azure://{container}")
self.connection_string = (
config.get("connection_string")
or match.group("connection_string") # backward compatibility
or os.getenv("AZURE_STORAGE_CONNECTION_STRING")
)
if not bucket:
raise ValueError("azure storage container name missing")
if not self.connection_string:
raise ValueError("azure storage connection string missing")
self.path_info = (
self.path_cls(url)
if path
else self.path_cls.from_parts(scheme=self.scheme, netloc=bucket)
self.connection_string = config.get("connection_string") or os.getenv(
"AZURE_STORAGE_CONNECTION_STRING"
)
@wrap_prop(threading.Lock())
@@ -68,12 +41,12 @@ class AzureRemote(BaseRemote):
from azure.storage.blob import BlockBlobService
from azure.common import AzureMissingResourceHttpError
logger.debug("URL {}".format(self.path_info))
logger.debug("Connection string {}".format(self.connection_string))
logger.debug(f"URL {self.path_info}")
logger.debug(f"Connection string {self.connection_string}")
blob_service = BlockBlobService(
connection_string=self.connection_string
)
logger.debug("Container name {}".format(self.path_info.bucket))
logger.debug(f"Container name {self.path_info.bucket}")
try: # verify that container exists
blob_service.list_blobs(
self.path_info.bucket, delimiter="/", num_results=1
@@ -82,14 +55,33 @@ class AzureRemote(BaseRemote):
blob_service.create_container(self.path_info.bucket)
return blob_service
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
def get_etag(self, path_info):
etag = self.blob_service.get_blob_properties(
path_info.bucket, path_info.path
).properties.etag
return etag.strip('"')
logger.debug("Removing {}".format(path_info))
self.blob_service.delete_blob(path_info.bucket, path_info.path)
def _generate_download_url(self, path_info, expires=3600):
from azure.storage.blob import BlobPermissions
def _list_paths(self, bucket, prefix, progress_callback=None):
expires_at = datetime.utcnow() + timedelta(seconds=expires)
sas_token = self.blob_service.generate_blob_shared_access_signature(
path_info.bucket,
path_info.path,
permission=BlobPermissions.READ,
expiry=expires_at,
)
download_url = self.blob_service.make_blob_url(
path_info.bucket, path_info.path, sas_token=sas_token
)
return download_url
def exists(self, path_info):
paths = self._list_paths(path_info.bucket, path_info.path)
return any(path_info.path == path for path in paths)
def _list_paths(self, bucket, prefix):
blob_service = self.blob_service
next_marker = None
while True:
@@ -98,8 +90,6 @@ class AzureRemote(BaseRemote):
)
for blob in blobs:
if progress_callback:
progress_callback()
yield blob.name
if not blobs.next_marker:
@@ -107,16 +97,24 @@ class AzureRemote(BaseRemote):
next_marker = blobs.next_marker
def list_cache_paths(self, prefix=None, progress_callback=None):
if prefix:
prefix = posixpath.join(
self.path_info.path, prefix[:2], prefix[2:]
)
else:
prefix = self.path_info.path
return self._list_paths(
self.path_info.bucket, prefix, progress_callback
)
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(
path_info.bucket, path_info.path, **kwargs
):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
logger.debug(f"Removing {path_info}")
self.blob_service.delete_blob(path_info.bucket, path_info.path)
def get_file_hash(self, path_info):
return self.get_etag(path_info)
def _upload(
self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
@@ -139,23 +137,3 @@ class AzureRemote(BaseRemote):
to_file,
progress_callback=pbar.update_to,
)
def exists(self, path_info):
paths = self._list_paths(path_info.bucket, path_info.path)
return any(path_info.path == path for path in paths)
def _generate_download_url(self, path_info, expires=3600):
from azure.storage.blob import BlobPermissions
expires_at = datetime.utcnow() + timedelta(seconds=expires)
sas_token = self.blob_service.generate_blob_shared_access_signature(
path_info.bucket,
path_info.path,
permission=BlobPermissions.READ,
expiry=expires_at,
)
download_url = self.blob_service.make_blob_url(
path_info.bucket, path_info.path, sas_token=sas_token
)
return download_url
+865 -758
View File
File diff suppressed because it is too large Load Diff
+94 -88
View File
@@ -1,40 +1,35 @@
from collections import defaultdict
import io
import logging
import os
import posixpath
import logging
import re
import threading
from collections import defaultdict
from contextlib import contextmanager
from urllib.parse import urlparse
from funcy import retry, wrap_with, wrap_prop, cached_property
from funcy import cached_property, retry, wrap_prop, wrap_with
from funcy.py3 import cat
from dvc.progress import Tqdm
from dvc.scheme import Schemes
from dvc.exceptions import DvcException, FileMissingError
from dvc.path_info import CloudURLInfo
from dvc.remote.base import BaseRemote
from dvc.exceptions import DvcException
from dvc.utils import tmp_fname, format_link
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
from dvc.utils import format_link, tmp_fname
from dvc.utils.stream import IterStream
logger = logging.getLogger(__name__)
FOLDER_MIME_TYPE = "application/vnd.google-apps.folder"
class GDrivePathNotFound(DvcException):
def __init__(self, path_info, hint):
hint = "" if hint is None else " {}".format(hint)
super().__init__(
"GDrive path '{}' not found.{}".format(path_info, hint)
)
class GDriveAuthError(DvcException):
def __init__(self, cred_location):
if cred_location:
message = (
"GDrive remote auth failed with credentials in '{}'.\n"
"Backup first, remove of fix them, and run DVC again.\n"
"Backup first, remove or fix them, and run DVC again.\n"
"It should do auth again and refresh the credentials.\n\n"
"Details:".format(cred_location)
)
@@ -44,17 +39,6 @@ class GDriveAuthError(DvcException):
super().__init__(message)
def _extract(exc, field):
from pydrive2.files import ApiRequestError
assert isinstance(exc, ApiRequestError)
# https://cloud.google.com/storage/docs/json_api/v1/status-codes#errorformat
return (
exc.error["errors"][0].get(field, "") if "errors" in exc.error else ""
)
def _gdrive_retry(func):
def should_retry(exc):
from pydrive2.files import ApiRequestError
@@ -68,12 +52,12 @@ def _gdrive_retry(func):
result = True
if error_code == 403:
result = _extract(exc, "reason") in [
result = exc.GetField("reason") in [
"userRateLimitExceeded",
"rateLimitExceeded",
]
if result:
logger.debug("Retrying GDrive API call, error: {}.".format(exc))
logger.debug(f"Retrying GDrive API call, error: {exc}.")
return result
@@ -100,9 +84,9 @@ class GDriveURLInfo(CloudURLInfo):
self._spath = re.sub("/{2,}", "/", self._spath.rstrip("/"))
class GDriveRemote(BaseRemote):
class GDriveRemoteTree(BaseRemoteTree):
scheme = Schemes.GDRIVE
path_cls = GDriveURLInfo
PATH_CLS = GDriveURLInfo
REQUIRES = {"pydrive2": "pydrive2"}
DEFAULT_VERIFY = True
# Always prefer traverse for GDrive since API usage quotas are a concern.
@@ -117,7 +101,8 @@ class GDriveRemote(BaseRemote):
def __init__(self, repo, config):
super().__init__(repo, config)
self.path_info = self.path_cls(config["url"])
self.path_info = self.PATH_CLS(config["url"])
if not self.path_info.bucket:
raise DvcException(
@@ -144,11 +129,11 @@ class GDriveRemote(BaseRemote):
self._validate_config()
self._gdrive_user_credentials_path = (
tmp_fname(os.path.join(self.repo.tmp_dir, ""))
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA)
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA)
else config.get(
"gdrive_user_credentials_file",
os.path.join(
self.repo.tmp_dir, self.DEFAULT_USER_CREDENTIALS_FILE
self.repo.tmp_dir, self.DEFAULT_USER_CREDENTIALS_FILE,
),
)
)
@@ -186,8 +171,8 @@ class GDriveRemote(BaseRemote):
Useful for tests, exception messages, etc. Returns either env variable
name if it's set or actual path to the credentials file.
"""
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
return GDriveRemote.GDRIVE_CREDENTIALS_DATA
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
return GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA
if os.path.exists(self._gdrive_user_credentials_path):
return self._gdrive_user_credentials_path
return None
@@ -201,7 +186,7 @@ class GDriveRemote(BaseRemote):
DVC config client id or secret but forgets to remove the cached
credentials file.
"""
if not os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
if not os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
if (
settings["client_config"]["client_id"]
!= auth.credentials.client_id
@@ -224,10 +209,10 @@ class GDriveRemote(BaseRemote):
from pydrive2.auth import GoogleAuth
from pydrive2.drive import GoogleDrive
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
with open(self._gdrive_user_credentials_path, "w") as cred_file:
cred_file.write(
os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA)
os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA)
)
auth_settings = {
@@ -274,7 +259,7 @@ class GDriveRemote(BaseRemote):
gauth.ServiceAuth()
else:
gauth.CommandLineAuth()
GDriveRemote._validate_credentials(gauth, auth_settings)
GDriveRemoteTree._validate_credentials(gauth, auth_settings)
# Handle AuthenticationError, RefreshError and other auth failures
# It's hard to come up with a narrow exception, since PyDrive throws
@@ -283,7 +268,7 @@ class GDriveRemote(BaseRemote):
except Exception as exc:
raise GDriveAuthError(self.credentials_location) from exc
finally:
if os.getenv(GDriveRemote.GDRIVE_CREDENTIALS_DATA):
if os.getenv(GDriveRemoteTree.GDRIVE_CREDENTIALS_DATA):
os.remove(self._gdrive_user_credentials_path)
return GoogleDrive(gauth)
@@ -353,7 +338,7 @@ class GDriveRemote(BaseRemote):
"use the `gdrive_user_credentials_file` config\n"
" option if you use multiple GDrive remotes with "
"different email accounts.\n\nDetails".format(
item_id, self.path_info, self._credentials_location
item_id, self.path_info, self.credentials_location
)
) from exc
raise
@@ -396,17 +381,35 @@ class GDriveRemote(BaseRemote):
param = {"id": item_id}
# it does not create a file on the remote
gdrive_file = self._drive.CreateFile(param)
bar_format = (
"Downloading {desc:{ncols_desc}.{ncols_desc}}... "
+ Tqdm.format_sizeof(int(gdrive_file["fileSize"]), "B", 1024)
)
with Tqdm(
bar_format=bar_format, desc=progress_desc, disable=no_progress_bar
):
gdrive_file.GetContentFile(to_file)
desc=progress_desc,
disable=no_progress_bar,
bytes=True,
# explicit `bar_format` as `total` will be set by `update_to`
bar_format=Tqdm.BAR_FMT_DEFAULT,
) as pbar:
gdrive_file.GetContentFile(to_file, callback=pbar.update_to)
@contextmanager
@_gdrive_retry
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb"}
item_id = self._get_item_id(path_info)
param = {"id": item_id}
# it does not create a file on the remote
gdrive_file = self._drive.CreateFile(param)
fd = gdrive_file.GetContentIOBuffer()
stream = IterStream(iter(fd))
if mode != "rb":
stream = io.TextIOWrapper(stream, encoding=encoding)
yield stream
@_gdrive_retry
def _gdrive_delete_file(self, item_id):
def gdrive_delete_file(self, item_id):
from pydrive2.files import ApiRequestError
param = {"id": item_id}
@@ -420,7 +423,7 @@ class GDriveRemote(BaseRemote):
if (
http_error_code == 403
and self._list_params["corpora"] == "drive"
and _extract(exc, "location") == "file.permissions"
and exc.GetField("location") == "file.permissions"
):
raise DvcException(
"Insufficient permissions to {}. You should have {} "
@@ -475,7 +478,7 @@ class GDriveRemote(BaseRemote):
return None
query = "trashed=false and ({})".format(
" or ".join(
"'{}' in parents".format(parent_id) for parent_id in parent_ids
f"'{parent_id}' in parents" for parent_id in parent_ids
)
)
query += " and title='{}'".format(title.replace("'", "\\'"))
@@ -518,16 +521,52 @@ class GDriveRemote(BaseRemote):
return min(item_ids)
assert not create
raise GDrivePathNotFound(path_info, hint)
raise FileMissingError(path_info, hint)
def exists(self, path_info):
try:
self._get_item_id(path_info)
except GDrivePathNotFound:
except FileMissingError:
return False
else:
return True
def _list_paths(self, prefix=None):
if not self._ids_cache["ids"]:
return
if prefix:
dir_ids = self._ids_cache["dirs"].get(prefix[:2])
if not dir_ids:
return
else:
dir_ids = self._ids_cache["ids"]
parents_query = " or ".join(
f"'{dir_id}' in parents" for dir_id in dir_ids
)
query = f"({parents_query}) and trashed=false"
for item in self._gdrive_list(query):
parent_id = item["parents"][0]["id"]
yield posixpath.join(
self._ids_cache["ids"][parent_id], item["title"]
)
def walk_files(self, path_info, **kwargs):
if path_info == self.path_info:
prefix = None
else:
prefix = path_info.path
for fname in self._list_paths(prefix=prefix, **kwargs):
yield path_info.replace(fname)
def remove(self, path_info):
item_id = self._get_item_id(path_info)
self.gdrive_delete_file(item_id)
def get_file_hash(self, path_info):
raise NotImplementedError
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
dirname = to_info.parent
assert dirname
@@ -540,36 +579,3 @@ class GDriveRemote(BaseRemote):
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
item_id = self._get_item_id(from_info)
self._gdrive_download_file(item_id, to_file, name, no_progress_bar)
def list_cache_paths(self, prefix=None, progress_callback=None):
if not self._ids_cache["ids"]:
return
if prefix:
dir_ids = self._ids_cache["dirs"].get(prefix[:2])
if not dir_ids:
return
else:
dir_ids = self._ids_cache["ids"]
parents_query = " or ".join(
"'{}' in parents".format(dir_id) for dir_id in dir_ids
)
query = "({}) and trashed=false".format(parents_query)
for item in self._gdrive_list(query):
if progress_callback:
progress_callback()
parent_id = item["parents"][0]["id"]
yield posixpath.join(
self._ids_cache["ids"][parent_id], item["title"]
)
def remove(self, path_info):
item_id = self._get_item_id(path_info)
self._gdrive_delete_file(item_id)
def get_file_checksum(self, path_info):
raise NotImplementedError
def walk_files(self, path_info):
raise NotImplementedError
+75 -90
View File
@@ -1,17 +1,15 @@
import logging
import os.path
import threading
from datetime import timedelta
from functools import wraps
import io
import os.path
import posixpath
import threading
from funcy import cached_property, wrap_prop
from dvc.exceptions import DvcException
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
@@ -55,7 +53,7 @@ def _upload_to_bucket(
no_progress_bar=False,
):
blob = bucket.blob(to_info.path, chunk_size=chunk_size)
with io.open(from_file, mode="rb") as fobj:
with open(from_file, mode="rb") as fobj:
with Tqdm.wrapattr(
fobj,
"read",
@@ -66,9 +64,9 @@ def _upload_to_bucket(
blob.upload_from_file(wrapped)
class GSRemote(BaseRemote):
class GSRemoteTree(BaseRemoteTree):
scheme = Schemes.GS
path_cls = CloudURLInfo
PATH_CLS = CloudURLInfo
REQUIRES = {"google-cloud-storage": "google.cloud.storage"}
PARAM_CHECKSUM = "md5"
@@ -76,7 +74,7 @@ class GSRemote(BaseRemote):
super().__init__(repo, config)
url = config.get("url", "gs:///")
self.path_info = self.path_cls(url)
self.path_info = self.PATH_CLS(url)
self.projectname = config.get("projectname", None)
self.credentialpath = config.get("credentialpath")
@@ -92,74 +90,22 @@ class GSRemote(BaseRemote):
else Client(self.projectname)
)
def get_file_checksum(self, path_info):
import base64
import codecs
def _generate_download_url(self, path_info, expires=3600):
expiration = timedelta(seconds=int(expires))
bucket = path_info.bucket
path = path_info.path
blob = self.gs.bucket(bucket).get_blob(path)
if not blob:
return None
bucket = self.gs.bucket(path_info.bucket)
blob = bucket.get_blob(path_info.path)
if blob is None:
raise FileNotFoundError
return blob.generate_signed_url(expiration=expiration)
b64_md5 = blob.md5_hash
md5 = base64.b64decode(b64_md5)
return codecs.getencoder("hex")(md5)[0].decode("utf-8")
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
def copy(self, from_info, to_info):
from_bucket = self.gs.bucket(from_info.bucket)
blob = from_bucket.get_blob(from_info.path)
if not blob:
msg = "'{}' doesn't exist in the cloud".format(from_info.path)
raise DvcException(msg)
to_bucket = self.gs.bucket(to_info.bucket)
from_bucket.copy_blob(blob, to_bucket, new_name=to_info.path)
def remove(self, path_info):
if path_info.scheme != "gs":
raise NotImplementedError
logger.debug("Removing gs://{}".format(path_info))
blob = self.gs.bucket(path_info.bucket).get_blob(path_info.path)
if not blob:
return
blob.delete()
def _list_paths(
self, path_info, max_items=None, prefix=None, progress_callback=None
):
if prefix:
prefix = posixpath.join(path_info.path, prefix[:2], prefix[2:])
else:
prefix = path_info.path
for blob in self.gs.bucket(path_info.bucket).list_blobs(
prefix=path_info.path, max_results=max_items
):
if progress_callback:
progress_callback()
yield blob.name
def list_cache_paths(self, prefix=None, progress_callback=None):
return self._list_paths(
self.path_info, prefix=prefix, progress_callback=progress_callback
)
def walk_files(self, path_info):
for fname in self._list_paths(path_info / ""):
# skip nested empty directories
if fname.endswith("/"):
continue
yield path_info.replace(fname)
def makedirs(self, path_info):
if not path_info.path:
return
self.gs.bucket(path_info.bucket).blob(
(path_info / "").path
).upload_from_string("")
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def isdir(self, path_info):
dir_path = path_info / ""
@@ -172,13 +118,61 @@ class GSRemote(BaseRemote):
blob = self.gs.bucket(path_info.bucket).blob(path_info.path)
return blob.exists()
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
def _list_paths(self, path_info, max_items=None):
for blob in self.gs.bucket(path_info.bucket).list_blobs(
prefix=path_info.path, max_results=max_items
):
yield blob.name
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(path_info / "", **kwargs):
# skip nested empty directories
if fname.endswith("/"):
continue
yield path_info.replace(fname)
def remove(self, path_info):
if path_info.scheme != "gs":
raise NotImplementedError
logger.debug(f"Removing gs://{path_info}")
blob = self.gs.bucket(path_info.bucket).get_blob(path_info.path)
if not blob:
return
blob.delete()
def makedirs(self, path_info):
if not path_info.path:
return
self.gs.bucket(path_info.bucket).blob(
(path_info / "").path
).upload_from_string("")
def copy(self, from_info, to_info):
from_bucket = self.gs.bucket(from_info.bucket)
blob = from_bucket.get_blob(from_info.path)
if not blob:
msg = f"'{from_info.path}' doesn't exist in the cloud"
raise DvcException(msg)
to_bucket = self.gs.bucket(to_info.bucket)
from_bucket.copy_blob(blob, to_bucket, new_name=to_info.path)
def get_file_hash(self, path_info):
import base64
import codecs
bucket = path_info.bucket
path = path_info.path
blob = self.gs.bucket(bucket).get_blob(path)
if not blob:
return None
b64_md5 = blob.md5_hash
md5 = base64.b64decode(b64_md5)
return codecs.getencoder("hex")(md5)[0].decode("utf-8")
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
bucket = self.gs.bucket(to_info.bucket)
@@ -193,7 +187,7 @@ class GSRemote(BaseRemote):
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
bucket = self.gs.bucket(from_info.bucket)
blob = bucket.get_blob(from_info.path)
with io.open(to_file, mode="wb") as fobj:
with open(to_file, mode="wb") as fobj:
with Tqdm.wrapattr(
fobj,
"write",
@@ -202,12 +196,3 @@ class GSRemote(BaseRemote):
disable=no_progress_bar,
) as wrapped:
blob.download_to_file(wrapped)
def _generate_download_url(self, path_info, expires=3600):
expiration = timedelta(seconds=int(expires))
bucket = self.gs.bucket(path_info.bucket)
blob = bucket.get_blob(path_info.path)
if blob is None:
raise FileNotFoundError
return blob.generate_signed_url(expiration=expiration)
+86 -86
View File
@@ -8,23 +8,25 @@ from collections import deque
from contextlib import closing, contextmanager
from urllib.parse import urlparse
from .base import BaseRemote, RemoteCmdError
from .pool import get_connection
from dvc.scheme import Schemes
from dvc.utils import fix_env, tmp_fname
from .base import BaseRemoteTree, RemoteCmdError
from .pool import get_connection
logger = logging.getLogger(__name__)
class HDFSRemote(BaseRemote):
class HDFSRemoteTree(BaseRemoteTree):
scheme = Schemes.HDFS
REQUIRES = {"pyarrow": "pyarrow"}
REGEX = r"^hdfs://((?P<user>.*)@)?.*$"
PARAM_CHECKSUM = "checksum"
REQUIRES = {"pyarrow": "pyarrow"}
TRAVERSE_PREFIX_LEN = 2
def __init__(self, repo, config):
super().__init__(repo, config)
self.path_info = None
url = config.get("url")
if not url:
@@ -33,7 +35,7 @@ class HDFSRemote(BaseRemote):
parsed = urlparse(url)
user = parsed.username or config.get("user")
self.path_info = self.path_cls.from_parts(
self.path_info = self.PATH_CLS.from_parts(
scheme=self.scheme,
host=parsed.hostname,
user=user,
@@ -41,7 +43,8 @@ class HDFSRemote(BaseRemote):
path=parsed.path,
)
def hdfs(self, path_info):
@staticmethod
def hdfs(path_info):
import pyarrow
return get_connection(
@@ -51,10 +54,84 @@ class HDFSRemote(BaseRemote):
user=path_info.user,
)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb"}
try:
with self.hdfs(path_info) as hdfs, closing(
hdfs.open(path_info.path, mode="rb")
) as fd:
if mode == "rb":
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
except OSError as e:
# Empty .errno and not specific enough error class in pyarrow,
# see https://issues.apache.org/jira/browse/ARROW-6248
if "file does not exist" in str(e):
raise FileNotFoundError(*e.args)
raise
def exists(self, path_info):
assert not isinstance(path_info, list)
assert path_info.scheme == "hdfs"
with self.hdfs(path_info) as hdfs:
return hdfs.exists(path_info.path)
def walk_files(self, path_info, **kwargs):
if not self.exists(path_info):
return
root = path_info.path
dirs = deque([root])
with self.hdfs(self.path_info) as hdfs:
if not hdfs.exists(root):
return
while dirs:
try:
entries = hdfs.ls(dirs.pop(), detail=True)
for entry in entries:
if entry["kind"] == "directory":
dirs.append(urlparse(entry["name"]).path)
elif entry["kind"] == "file":
path = urlparse(entry["name"]).path
yield path_info.replace(path=path)
except OSError:
# When searching for a specific prefix pyarrow raises an
# exception if the specified cache dir does not exist
pass
def remove(self, path_info):
if path_info.scheme != "hdfs":
raise NotImplementedError
if self.exists(path_info):
logger.debug(f"Removing {path_info.path}")
with self.hdfs(path_info) as hdfs:
hdfs.rm(path_info.path)
def copy(self, from_info, to_info, **_kwargs):
dname = posixpath.dirname(to_info.path)
with self.hdfs(to_info) as hdfs:
hdfs.mkdir(dname)
# NOTE: this is how `hadoop fs -cp` works too: it copies through
# your local machine.
with hdfs.open(from_info.path, "rb") as from_fobj:
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
with hdfs.open(tmp_info.path, "wb") as tmp_fobj:
tmp_fobj.upload(from_fobj)
hdfs.rename(tmp_info.path, to_info.path)
except Exception:
self.remove(tmp_info)
raise
def hadoop_fs(self, cmd, user=None):
cmd = "hadoop fs -" + cmd
if user:
cmd = "HADOOP_USER_NAME={} ".format(user) + cmd
cmd = f"HADOOP_USER_NAME={user} " + cmd
# NOTE: close_fds doesn't work with redirected stdin/stdout/stderr.
# See https://github.com/iterative/dvc/issues/1197.
@@ -82,45 +159,14 @@ class HDFSRemote(BaseRemote):
assert match is not None
return match.group(gname)
def get_file_checksum(self, path_info):
def get_file_hash(self, path_info):
# NOTE: pyarrow doesn't support checksum, so we need to use hadoop
regex = r".*\t.*\t(?P<checksum>.*)"
stdout = self.hadoop_fs(
"checksum {}".format(path_info.path), user=path_info.user
f"checksum {path_info.path}", user=path_info.user
)
return self._group(regex, stdout, "checksum")
def copy(self, from_info, to_info, **_kwargs):
dname = posixpath.dirname(to_info.path)
with self.hdfs(to_info) as hdfs:
hdfs.mkdir(dname)
# NOTE: this is how `hadoop fs -cp` works too: it copies through
# your local machine.
with hdfs.open(from_info.path, "rb") as from_fobj:
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
with hdfs.open(tmp_info.path, "wb") as tmp_fobj:
tmp_fobj.upload(from_fobj)
hdfs.rename(tmp_info.path, to_info.path)
except Exception:
self.remove(tmp_info)
raise
def remove(self, path_info):
if path_info.scheme != "hdfs":
raise NotImplementedError
if self.exists(path_info):
logger.debug("Removing {}".format(path_info.path))
with self.hdfs(path_info) as hdfs:
hdfs.rm(path_info.path)
def exists(self, path_info):
assert not isinstance(path_info, list)
assert path_info.scheme == "hdfs"
with self.hdfs(path_info) as hdfs:
return hdfs.exists(path_info.path)
def _upload(self, from_file, to_info, **_kwargs):
with self.hdfs(to_info) as hdfs:
hdfs.mkdir(posixpath.dirname(to_info.path))
@@ -133,49 +179,3 @@ class HDFSRemote(BaseRemote):
with self.hdfs(from_info) as hdfs:
with open(to_file, "wb+") as fobj:
hdfs.download(from_info.path, fobj)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb"}
try:
with self.hdfs(path_info) as hdfs, closing(
hdfs.open(path_info.path, mode="rb")
) as fd:
if mode == "rb":
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
except IOError as e:
# Empty .errno and not specific enough error class in pyarrow,
# see https://issues.apache.org/jira/browse/ARROW-6248
if "file does not exist" in str(e):
raise FileNotFoundError(*e.args)
raise
def list_cache_paths(self, prefix=None, progress_callback=None):
if not self.exists(self.path_info):
return
if prefix:
root = posixpath.join(self.path_info.path, prefix[:2])
else:
root = self.path_info.path
dirs = deque([root])
with self.hdfs(self.path_info) as hdfs:
while dirs:
try:
entries = hdfs.ls(dirs.pop(), detail=True)
for entry in entries:
if entry["kind"] == "directory":
dirs.append(urlparse(entry["name"]).path)
elif entry["kind"] == "file":
if progress_callback:
progress_callback()
yield urlparse(entry["name"]).path
except IOError as e:
# When searching for a specific prefix pyarrow raises an
# exception if the specified cache dir does not exist
if not prefix:
raise e
+79 -80
View File
@@ -4,11 +4,11 @@ import threading
from funcy import cached_property, memoize, wrap_prop, wrap_with
from dvc.path_info import HTTPURLInfo
import dvc.prompt as prompt
from dvc.exceptions import DvcException, HTTPError
from dvc.path_info import HTTPURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
@@ -23,22 +23,23 @@ def ask_password(host, user):
)
class HTTPRemote(BaseRemote):
class HTTPRemoteTree(BaseRemoteTree):
scheme = Schemes.HTTP
path_cls = HTTPURLInfo
PATH_CLS = HTTPURLInfo
PARAM_CHECKSUM = "etag"
CAN_TRAVERSE = False
SESSION_RETRIES = 5
SESSION_BACKOFF_FACTOR = 0.1
REQUEST_TIMEOUT = 10
CHUNK_SIZE = 2 ** 16
PARAM_CHECKSUM = "etag"
CAN_TRAVERSE = False
def __init__(self, repo, config):
super().__init__(repo, config)
url = config.get("url")
if url:
self.path_info = self.path_cls(url)
self.path_info = self.PATH_CLS(url)
user = config.get("user", None)
if user:
self.path_info.user = user
@@ -51,74 +52,7 @@ class HTTPRemote(BaseRemote):
self.ask_password = config.get("ask_password", False)
self.headers = {}
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
response = self._request("GET", from_info.url, stream=True)
if response.status_code != 200:
raise HTTPError(response.status_code, response.reason)
with open(to_file, "wb") as fd:
with Tqdm.wrapattr(
fd,
"write",
total=None
if no_progress_bar
else self._content_length(response),
leave=False,
desc=from_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):
fd_wrapped.write(chunk)
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
def chunks():
with open(from_file, "rb") as fd:
with Tqdm.wrapattr(
fd,
"read",
total=None
if no_progress_bar
else os.path.getsize(from_file),
leave=False,
desc=to_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
while True:
chunk = fd_wrapped.read(self.CHUNK_SIZE)
if not chunk:
break
yield chunk
response = self._request("POST", to_info.url, data=chunks())
if response.status_code not in (200, 201):
raise HTTPError(response.status_code, response.reason)
def exists(self, path_info):
return bool(self._request("HEAD", path_info.url))
def _content_length(self, response):
res = response.headers.get("Content-Length")
return int(res) if res else None
def get_file_checksum(self, path_info):
url = path_info.url
headers = self._request("HEAD", url).headers
etag = headers.get("ETag") or headers.get("Content-MD5")
if not etag:
raise DvcException(
"could not find an ETag or "
"Content-MD5 header for '{url}'".format(url=url)
)
if etag.startswith("W/"):
raise DvcException(
"Weak ETags are not supported."
" (Etag: '{etag}', URL: '{url}')".format(etag=etag, url=url)
)
return etag
def auth_method(self, path_info=None):
def _auth_method(self, path_info=None):
from requests.auth import HTTPBasicAuth, HTTPDigestAuth
if path_info is None:
@@ -155,7 +89,7 @@ class HTTPRemote(BaseRemote):
return session
def _request(self, method, url, **kwargs):
def request(self, method, url, **kwargs):
import requests
kwargs.setdefault("allow_redirects", True)
@@ -165,7 +99,7 @@ class HTTPRemote(BaseRemote):
res = self._session.request(
method,
url,
auth=self.auth_method(),
auth=self._auth_method(),
headers=self.headers,
**kwargs,
)
@@ -185,7 +119,72 @@ class HTTPRemote(BaseRemote):
return res
except requests.exceptions.RequestException:
raise DvcException("could not perform a {} request".format(method))
raise DvcException(f"could not perform a {method} request")
def gc(self):
raise NotImplementedError
def exists(self, path_info):
return bool(self.request("HEAD", path_info.url))
def get_file_hash(self, path_info):
url = path_info.url
headers = self.request("HEAD", url).headers
etag = headers.get("ETag") or headers.get("Content-MD5")
if not etag:
raise DvcException(
"could not find an ETag or "
"Content-MD5 header for '{url}'".format(url=url)
)
if etag.startswith("W/"):
raise DvcException(
"Weak ETags are not supported."
" (Etag: '{etag}', URL: '{url}')".format(etag=etag, url=url)
)
return etag
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
response = self.request("GET", from_info.url, stream=True)
if response.status_code != 200:
raise HTTPError(response.status_code, response.reason)
with open(to_file, "wb") as fd:
with Tqdm.wrapattr(
fd,
"write",
total=None
if no_progress_bar
else self._content_length(response),
leave=False,
desc=from_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
for chunk in response.iter_content(chunk_size=self.CHUNK_SIZE):
fd_wrapped.write(chunk)
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
def chunks():
with open(from_file, "rb") as fd:
with Tqdm.wrapattr(
fd,
"read",
total=None
if no_progress_bar
else os.path.getsize(from_file),
leave=False,
desc=to_info.url if name is None else name,
disable=no_progress_bar,
) as fd_wrapped:
while True:
chunk = fd_wrapped.read(self.CHUNK_SIZE)
if not chunk:
break
yield chunk
response = self.request("POST", to_info.url, data=chunks())
if response.status_code not in (200, 201):
raise HTTPError(response.status_code, response.reason)
@staticmethod
def _content_length(response):
res = response.headers.get("Content-Length")
return int(res) if res else None
+3 -2
View File
@@ -1,6 +1,7 @@
from .http import HTTPRemote
from dvc.scheme import Schemes
from .http import HTTPRemoteTree
class HTTPSRemote(HTTPRemote):
class HTTPSRemoteTree(HTTPRemoteTree):
scheme = Schemes.HTTPS
+31 -37
View File
@@ -25,15 +25,15 @@ class RemoteIndexNoop:
def __iter__(self):
return iter([])
def __contains__(self, checksum):
def __contains__(self, hash_):
return False
@staticmethod
def checksums():
def hashes():
return []
@staticmethod
def dir_checksums():
def dir_hashes():
return []
def load(self):
@@ -54,13 +54,13 @@ class RemoteIndexNoop:
class RemoteIndex:
"""Class for indexing remote checksums in a sqlite3 database.
"""Class for indexing remote hashes in a sqlite3 database.
Args:
repo: repo for this remote index.
name: name for this index. Index db will be loaded from and saved to
``.dvc/tmp/index/{name}.idx``.
dir_suffix: suffix used for naming directory checksums
dir_suffix: suffix used for naming directory hashes
"""
INDEX_SUFFIX = ".idx"
@@ -69,9 +69,7 @@ class RemoteIndex:
INDEX_TABLE_LAYOUT = "checksum TEXT PRIMARY KEY, " "dir INTEGER NOT NULL"
def __init__(self, repo, name, dir_suffix=".dir"):
self.path = os.path.join(
repo.index_dir, "{}{}".format(name, self.INDEX_SUFFIX)
)
self.path = os.path.join(repo.index_dir, f"{name}{self.INDEX_SUFFIX}")
self.dir_suffix = dir_suffix
self.database = None
@@ -80,9 +78,9 @@ class RemoteIndex:
self.lock = threading.Lock()
def __iter__(self):
cmd = "SELECT checksum FROM {}".format(self.INDEX_TABLE)
for (checksum,) in self._execute(cmd):
yield checksum
cmd = f"SELECT checksum FROM {self.INDEX_TABLE}"
for (hash_,) in self._execute(cmd):
yield hash_
def __enter__(self):
self.lock.acquire()
@@ -92,25 +90,25 @@ class RemoteIndex:
self.dump()
self.lock.release()
def __contains__(self, checksum):
def __contains__(self, hash_):
cmd = "SELECT checksum FROM {} WHERE checksum = (?)".format(
self.INDEX_TABLE
)
self._execute(cmd, (checksum,))
self._execute(cmd, (hash_,))
return self.cursor.fetchone() is not None
def checksums(self):
"""Iterate over checksums stored in the index."""
def hashes(self):
"""Iterate over hashes stored in the index."""
return iter(self)
def dir_checksums(self):
"""Iterate over .dir checksums stored in the index."""
cmd = "SELECT checksum FROM {} WHERE dir = 1".format(self.INDEX_TABLE)
for (checksum,) in self._execute(cmd):
yield checksum
def dir_hashes(self):
"""Iterate over .dir hashes stored in the index."""
cmd = f"SELECT checksum FROM {self.INDEX_TABLE} WHERE dir = 1"
for (hash_,) in self._execute(cmd):
yield hash_
def is_dir_checksum(self, checksum):
return checksum.endswith(self.dir_suffix)
def is_dir_hash(self, hash_):
return hash_.endswith(self.dir_suffix)
def _execute(self, cmd, parameters=()):
return self.cursor.execute(cmd, parameters)
@@ -184,31 +182,27 @@ class RemoteIndex:
Changes to the index will not committed until dump() is called.
"""
cmd = "DELETE FROM {}".format(self.INDEX_TABLE)
cmd = f"DELETE FROM {self.INDEX_TABLE}"
self._execute(cmd)
def update(self, dir_checksums, file_checksums):
"""Update this index, adding the specified checksums.
def update(self, dir_hashes, file_hashes):
"""Update this index, adding the specified hashes.
Changes to the index will not committed until dump() is called.
"""
cmd = "INSERT OR IGNORE INTO {} (checksum, dir) VALUES (?, ?)".format(
self.INDEX_TABLE
)
self._executemany(
cmd, ((checksum, True) for checksum in dir_checksums)
)
self._executemany(
cmd, ((checksum, False) for checksum in file_checksums)
)
self._executemany(cmd, ((hash_, True) for hash_ in dir_hashes))
self._executemany(cmd, ((hash_, False) for hash_ in file_hashes))
def intersection(self, checksums):
"""Iterate over values from `checksums` which exist in the index."""
def intersection(self, hashes):
"""Iterate over values from `hashes` which exist in the index."""
# sqlite has a compile time limit of 999, see:
# https://www.sqlite.org/c3ref/c_limit_attached.html#sqlitelimitvariablenumber
for chunk in lchunks(999, checksums):
for chunk in lchunks(999, hashes):
cmd = "SELECT checksum FROM {} WHERE checksum IN ({})".format(
self.INDEX_TABLE, ",".join("?" for checksum in chunk)
self.INDEX_TABLE, ",".join("?" for hash_ in chunk)
)
for (checksum,) in self._execute(cmd, chunk):
yield checksum
for (hash_,) in self._execute(cmd, chunk):
yield hash_
+315 -313
View File
@@ -2,120 +2,101 @@ import errno
import logging
import os
import stat
from concurrent.futures import as_completed, ThreadPoolExecutor
from functools import partial
from concurrent.futures import ThreadPoolExecutor, as_completed
from functools import partial, wraps
from funcy import cached_property, concat
from shortuuid import uuid
from dvc.compat import fspath_py35
from dvc.exceptions import DvcException, DownloadError, UploadError
from dvc.exceptions import DownloadError, DvcException, UploadError
from dvc.path_info import PathInfo
from dvc.progress import Tqdm
from dvc.remote.base import (
index_locked,
BaseRemote,
STATUS_MAP,
STATUS_DELETED,
STATUS_MAP,
STATUS_MISSING,
STATUS_NEW,
BaseRemoteTree,
CloudCache,
Remote,
index_locked,
)
from dvc.remote.index import RemoteIndexNoop
from dvc.scheme import Schemes
from dvc.scm.tree import is_working_tree
from dvc.scm.tree import WorkingTree, is_working_tree
from dvc.system import System
from dvc.utils import file_md5, relpath, tmp_fname
from dvc.utils.fs import copyfile, move, makedirs, remove, walk_files
from dvc.utils.fs import (
copy_fobj_to_file,
copyfile,
makedirs,
move,
remove,
walk_files,
)
logger = logging.getLogger(__name__)
class LocalRemote(BaseRemote):
class LocalRemoteTree(BaseRemoteTree):
scheme = Schemes.LOCAL
path_cls = PathInfo
PATH_CLS = PathInfo
PARAM_CHECKSUM = "md5"
PARAM_PATH = "path"
TRAVERSE_PREFIX_LEN = 2
INDEX_CLS = RemoteIndexNoop
UNPACKED_DIR_SUFFIX = ".unpacked"
DEFAULT_CACHE_TYPES = ["reflink", "copy"]
CACHE_MODE = 0o444
SHARED_MODE_MAP = {None: (0o644, 0o755), "group": (0o664, 0o775)}
def __init__(self, repo, config):
super().__init__(repo, config)
self.cache_dir = config.get("url")
self._dir_info = {}
url = config.get("url")
self.path_info = self.PATH_CLS(url) if url else None
@property
def state(self):
return self.repo.state
from dvc.state import StateNoop
@property
def cache_dir(self):
return self.path_info.fspath if self.path_info else None
@cache_dir.setter
def cache_dir(self, value):
self.path_info = PathInfo(value) if value else None
@classmethod
def supported(cls, config):
return True
return self.repo.state if self.repo else StateNoop()
@cached_property
def cache_path(self):
return os.path.abspath(self.cache_dir)
def work_tree(self):
# When using repo.brancher, repo.tree may change to/from WorkingTree to
# GitTree arbitarily. When repo.tree is GitTree, local cache needs to
# use its own WorkingTree instance.
if self.repo:
return WorkingTree(self.repo.root_dir)
return None
def checksum_to_path(self, checksum):
return os.path.join(self.cache_path, checksum[0:2], checksum[2:])
def list_cache_paths(self, prefix=None, progress_callback=None):
assert self.path_info is not None
if prefix:
path_info = self.path_info / prefix[:2]
else:
path_info = self.path_info
if progress_callback:
for path in walk_files(path_info):
progress_callback()
yield path
else:
yield from walk_files(path_info)
def get(self, md5):
if not md5:
return None
return self.checksum_to_path_info(md5).url
@staticmethod
def open(path_info, mode="r", encoding=None):
return open(path_info, mode=mode, encoding=encoding)
def exists(self, path_info):
assert is_working_tree(self.repo.tree)
assert isinstance(path_info, str) or path_info.scheme == "local"
return self.repo.tree.exists(fspath_py35(path_info))
if not self.repo:
return os.path.exists(path_info)
return self.work_tree.exists(path_info)
def makedirs(self, path_info):
makedirs(path_info, exist_ok=True, mode=self._dir_mode)
def isfile(self, path_info):
if not self.repo:
return os.path.isfile(path_info)
return self.work_tree.isfile(path_info)
def already_cached(self, path_info):
assert path_info.scheme in ["", "local"]
def isdir(self, path_info):
if not self.repo:
return os.path.isdir(path_info)
return self.work_tree.isdir(path_info)
current_md5 = self.get_checksum(path_info)
def iscopy(self, path_info):
return not (
System.is_symlink(path_info) or System.is_hardlink(path_info)
)
if not current_md5:
return False
return not self.changed_cache(current_md5)
def _verify_link(self, path_info, link_type):
if link_type == "hardlink" and self.getsize(path_info) == 0:
return
super()._verify_link(path_info, link_type)
def walk_files(self, path_info, **kwargs):
for fname in self.work_tree.walk_files(path_info):
yield PathInfo(fname)
def is_empty(self, path_info):
path = path_info.fspath
@@ -128,32 +109,6 @@ class LocalRemote(BaseRemote):
return False
@staticmethod
def isfile(path_info):
return os.path.isfile(fspath_py35(path_info))
@staticmethod
def isdir(path_info):
return os.path.isdir(fspath_py35(path_info))
def iscopy(self, path_info):
return not (
System.is_symlink(path_info) or System.is_hardlink(path_info)
)
@staticmethod
def getsize(path_info):
return os.path.getsize(fspath_py35(path_info))
def walk_files(self, path_info):
assert is_working_tree(self.repo.tree)
for fname in self.repo.tree.walk_files(path_info):
yield PathInfo(fname)
def get_file_checksum(self, path_info):
return file_md5(path_info)[0]
def remove(self, path_info):
if isinstance(path_info, PathInfo):
if path_info.scheme != "local":
@@ -165,6 +120,9 @@ class LocalRemote(BaseRemote):
if self.exists(path):
remove(path)
def makedirs(self, path_info):
makedirs(path_info, exist_ok=True, mode=self.dir_mode)
def move(self, from_info, to_info, mode=None):
if from_info.scheme != "local" or to_info.scheme != "local":
raise NotImplementedError
@@ -173,9 +131,9 @@ class LocalRemote(BaseRemote):
if mode is None:
if self.isfile(from_info):
mode = self._file_mode
mode = self.file_mode
else:
mode = self._dir_mode
mode = self.dir_mode
move(from_info, to_info, mode=mode)
@@ -183,8 +141,19 @@ class LocalRemote(BaseRemote):
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
System.copy(from_info, tmp_info)
os.chmod(fspath_py35(tmp_info), self._file_mode)
os.rename(fspath_py35(tmp_info), fspath_py35(to_info))
os.chmod(tmp_info, self.file_mode)
os.rename(tmp_info, to_info)
except Exception:
self.remove(tmp_info)
raise
def copy_fobj(self, fobj, to_info):
self.makedirs(to_info.parent)
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
copy_fobj_to_file(fobj, tmp_info)
os.chmod(tmp_info, self.file_mode)
os.rename(tmp_info, to_info)
except Exception:
self.remove(tmp_info)
raise
@@ -202,7 +171,7 @@ class LocalRemote(BaseRemote):
# and the cache type is `hardlink`, we might reach link limits and
# will get something like: `too many links error`
#
# This is because all those empty files will have the same checksum
# This is because all those empty files will have the same hash
# (i.e. 68b329da9893e34099c7d8ad5cb9c940), therefore, they will be
# linked to the same file in the cache.
#
@@ -232,20 +201,81 @@ class LocalRemote(BaseRemote):
System.reflink(from_info, tmp_info)
# NOTE: reflink has its own separate inode, so you can set permissions
# that are different from the source.
os.chmod(fspath_py35(tmp_info), self._file_mode)
os.rename(fspath_py35(tmp_info), fspath_py35(to_info))
os.chmod(tmp_info, self.file_mode)
os.rename(tmp_info, to_info)
def cache_exists(self, checksums, jobs=None, name=None):
return [
checksum
for checksum in Tqdm(
checksums,
unit="file",
desc="Querying "
+ ("cache in " + name if name else "local cache"),
def _unprotect_file(self, path):
if System.is_symlink(path) or System.is_hardlink(path):
logger.debug(f"Unprotecting '{path}'")
tmp = os.path.join(os.path.dirname(path), "." + uuid())
# The operations order is important here - if some application
# would access the file during the process of copyfile then it
# would get only the part of file. So, at first, the file should be
# copied with the temporary name, and then original file should be
# replaced by new.
copyfile(path, tmp, name="Unprotecting '{}'".format(relpath(path)))
remove(path)
os.rename(tmp, path)
else:
logger.debug(
"Skipping copying for '{}', since it is not "
"a symlink or a hardlink.".format(path)
)
if not self.changed_cache_file(checksum)
]
os.chmod(path, self.file_mode)
def _unprotect_dir(self, path):
assert is_working_tree(self.repo.tree)
for fname in self.repo.tree.walk_files(path):
self._unprotect_file(fname)
def unprotect(self, path_info):
path = path_info.fspath
if not os.path.exists(path):
raise DvcException(f"can't unprotect non-existing data '{path}'")
if os.path.isdir(path):
self._unprotect_dir(path)
else:
self._unprotect_file(path)
def protect(self, path_info):
path = os.fspath(path_info)
mode = self.CACHE_MODE
try:
os.chmod(path, mode)
except OSError as exc:
# There is nothing we need to do in case of a read-only file system
if exc.errno == errno.EROFS:
return
# In shared cache scenario, we might not own the cache file, so we
# need to check if cache file is already protected.
if exc.errno not in [errno.EPERM, errno.EACCES]:
raise
actual = stat.S_IMODE(os.stat(path).st_mode)
if actual != mode:
raise
def is_protected(self, path_info):
try:
mode = os.stat(path_info).st_mode
except FileNotFoundError:
return False
return stat.S_IMODE(mode) == self.CACHE_MODE
def get_file_hash(self, path_info):
return file_md5(path_info)[0]
@staticmethod
def getsize(path_info):
return os.path.getsize(path_info)
def _upload(
self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
@@ -256,18 +286,122 @@ class LocalRemote(BaseRemote):
copyfile(
from_file, tmp_file, name=name, no_progress_bar=no_progress_bar
)
os.rename(tmp_file, fspath_py35(to_info))
self.protect(tmp_file)
os.rename(tmp_file, to_info)
@staticmethod
def _download(
self, from_info, to_file, name=None, no_progress_bar=False, **_kwargs
from_info, to_file, name=None, no_progress_bar=False, **_kwargs
):
copyfile(
from_info, to_file, no_progress_bar=no_progress_bar, name=name
)
@staticmethod
def open(path_info, mode="r", encoding=None):
return open(fspath_py35(path_info), mode=mode, encoding=encoding)
def list_paths(self, prefix=None, progress_callback=None):
assert self.path_info is not None
if prefix:
path_info = self.path_info / prefix[:2]
if not self.exists(path_info):
return
else:
path_info = self.path_info
# NOTE: use utils.fs walk_files since tree.walk_files will not follow
# symlinks
if progress_callback:
for path in walk_files(path_info):
progress_callback()
yield path
else:
yield from walk_files(path_info)
def _remove_unpacked_dir(self, hash_):
info = self.hash_to_path_info(hash_)
path_info = info.with_name(info.name + self.UNPACKED_DIR_SUFFIX)
self.remove(path_info)
def _log_exceptions(func, operation):
@wraps(func)
def wrapper(from_info, to_info, *args, **kwargs):
try:
func(from_info, to_info, *args, **kwargs)
return 0
except Exception as exc:
# NOTE: this means we ran out of file descriptors and there is no
# reason to try to proceed, as we will hit this error anyways.
if isinstance(exc, OSError) and exc.errno == errno.EMFILE:
raise
logger.exception(
"failed to %s '%s' to '%s'", operation, from_info, to_info
)
return 1
return wrapper
class LocalRemote(Remote):
INDEX_CLS = RemoteIndexNoop
class LocalCache(CloudCache):
DEFAULT_CACHE_TYPES = ["reflink", "copy"]
CACHE_MODE = LocalRemoteTree.CACHE_MODE
def __init__(self, tree):
super().__init__(tree)
self.cache_dir = tree.config.get("url")
@property
def cache_dir(self):
return self.tree.path_info.fspath if self.tree.path_info else None
@cache_dir.setter
def cache_dir(self, value):
self.tree.path_info = PathInfo(value) if value else None
@classmethod
def supported(cls, config):
return True
@cached_property
def cache_path(self):
return os.path.abspath(self.cache_dir)
def hash_to_path(self, hash_):
# NOTE: `self.cache_path` is already normalized so we can simply use
# `os.sep` instead of `os.path.join`. This results in this helper
# being ~5.5 times faster.
return f"{self.cache_path}{os.sep}{hash_[0:2]}{os.sep}{hash_[2:]}"
def hashes_exist(self, hashes, jobs=None, name=None):
return [
hash_
for hash_ in Tqdm(
hashes,
unit="file",
desc="Querying "
+ ("cache in " + name if name else "local cache"),
)
if not self.changed_cache_file(hash_)
]
def already_cached(self, path_info):
assert path_info.scheme in ["", "local"]
current_md5 = self.get_hash(path_info)
if not current_md5:
return False
return not self.changed_cache(current_md5)
def _verify_link(self, path_info, link_type):
if link_type == "hardlink" and self.tree.getsize(path_info) == 0:
return
super()._verify_link(path_info, link_type)
@index_locked
def status(
@@ -300,17 +434,15 @@ class LocalRemote(BaseRemote):
dir_status_info contains status for .dir files, file_status_info
contains status for all other files, and dir_contents is a dict of
{dir_checksum: set(file_checksum, ...)} which can be used to map
{dir_hash: set(file_hash, ...)} which can be used to map
a .dir file to its file contents.
"""
logger.debug(
"Preparing to collect status from {}".format(remote.path_info)
)
logger.debug(f"Preparing to collect status from {remote.path_info}")
md5s = set(named_cache.scheme_keys(self.scheme))
logger.debug("Collecting information from local cache...")
local_exists = frozenset(
self.cache_exists(md5s, jobs=jobs, name=self.cache_dir)
self.hashes_exist(md5s, jobs=jobs, name=self.cache_dir)
)
# This is a performance optimization. We can safely assume that,
@@ -325,12 +457,12 @@ class LocalRemote(BaseRemote):
dir_md5s = set(named_cache.dir_keys(self.scheme))
if dir_md5s:
remote_exists.update(
self._indexed_dir_checksums(named_cache, remote, dir_md5s)
self._indexed_dir_hashes(named_cache, remote, dir_md5s)
)
md5s.difference_update(remote_exists)
if md5s:
remote_exists.update(
remote.cache_exists(
remote.hashes_exist(
md5s, jobs=jobs, name=str(remote.path_info)
)
)
@@ -341,23 +473,23 @@ class LocalRemote(BaseRemote):
def _make_status(
self, named_cache, show_checksums, local_exists, remote_exists
):
def make_names(checksum, names):
return {"name": checksum if show_checksums else " ".join(names)}
def make_names(hash_, names):
return {"name": hash_ if show_checksums else " ".join(names)}
dir_status = {}
file_status = {}
dir_contents = {}
for checksum, item in named_cache[self.scheme].items():
for hash_, item in named_cache[self.scheme].items():
if item.children:
dir_status[checksum] = make_names(checksum, item.names)
dir_contents[checksum] = set()
for child_checksum, child in item.children.items():
file_status[child_checksum] = make_names(
child_checksum, child.names
dir_status[hash_] = make_names(hash_, item.names)
dir_contents[hash_] = set()
for child_hash, child in item.children.items():
file_status[child_hash] = make_names(
child_hash, child.names
)
dir_contents[checksum].add(child_checksum)
dir_contents[hash_].add(child_hash)
else:
file_status[checksum] = make_names(checksum, item.names)
file_status[hash_] = make_names(hash_, item.names)
self._fill_statuses(dir_status, local_exists, remote_exists)
self._fill_statuses(file_status, local_exists, remote_exists)
@@ -366,50 +498,50 @@ class LocalRemote(BaseRemote):
return dir_status, file_status, dir_contents
def _indexed_dir_checksums(self, named_cache, remote, dir_md5s):
# Validate our index by verifying all indexed .dir checksums
def _indexed_dir_hashes(self, named_cache, remote, dir_md5s):
# Validate our index by verifying all indexed .dir hashes
# still exist on the remote
indexed_dirs = set(remote.index.dir_checksums())
indexed_dirs = set(remote.index.dir_hashes())
indexed_dir_exists = set()
if indexed_dirs:
indexed_dir_exists.update(
remote._cache_object_exists(indexed_dirs)
remote.tree.list_hashes_exists(indexed_dirs)
)
missing_dirs = indexed_dirs.difference(indexed_dir_exists)
if missing_dirs:
logger.debug(
"Remote cache missing indexed .dir checksums '{}', "
"Remote cache missing indexed .dir hashes '{}', "
"clearing remote index".format(", ".join(missing_dirs))
)
remote.index.clear()
# Check if non-indexed (new) dir checksums exist on remote
# Check if non-indexed (new) dir hashes exist on remote
dir_exists = dir_md5s.intersection(indexed_dir_exists)
dir_exists.update(remote._cache_object_exists(dir_md5s - dir_exists))
dir_exists.update(
remote.tree.list_hashes_exists(dir_md5s - dir_exists)
)
# If .dir checksum exists on the remote, assume directory contents
# If .dir hash exists on the remote, assume directory contents
# still exists on the remote
for dir_checksum in dir_exists:
file_checksums = list(
named_cache.child_keys(self.scheme, dir_checksum)
)
if dir_checksum not in remote.index:
for dir_hash in dir_exists:
file_hashes = list(named_cache.child_keys(self.scheme, dir_hash))
if dir_hash not in remote.index:
logger.debug(
"Indexing new .dir '{}' with '{}' nested files".format(
dir_checksum, len(file_checksums)
dir_hash, len(file_hashes)
)
)
remote.index.update([dir_checksum], file_checksums)
yield dir_checksum
yield from file_checksums
remote.index.update([dir_hash], file_hashes)
yield dir_hash
yield from file_hashes
@staticmethod
def _fill_statuses(checksum_info_dir, local_exists, remote_exists):
def _fill_statuses(hash_info_dir, local_exists, remote_exists):
# Using sets because they are way faster for lookups
local = set(local_exists)
remote = set(remote_exists)
for md5, info in checksum_info_dir.items():
for md5, info in hash_info_dir.items():
status = STATUS_MAP[(md5 in local, md5 in remote)]
info["status"] = status
@@ -417,15 +549,15 @@ class LocalRemote(BaseRemote):
cache = []
path_infos = []
names = []
checksums = []
hashes = []
for md5, info in Tqdm(
status_info.items(), desc="Analysing status", unit="file"
):
if info["status"] == status:
cache.append(self.checksum_to_path_info(md5))
path_infos.append(remote.checksum_to_path_info(md5))
cache.append(self.hash_to_path_info(md5))
path_infos.append(remote.hash_to_path_info(md5))
names.append(info["name"])
checksums.append(md5)
hashes.append(md5)
if download:
to_infos = cache
@@ -434,7 +566,7 @@ class LocalRemote(BaseRemote):
to_infos = path_infos
from_infos = cache
return from_infos, to_infos, names, checksums
return from_infos, to_infos, names, hashes
def _process(
self,
@@ -453,19 +585,19 @@ class LocalRemote(BaseRemote):
if download:
func = partial(
remote.download,
dir_mode=self._dir_mode,
file_mode=self._file_mode,
_log_exceptions(remote.tree.download, "download"),
dir_mode=self.tree.dir_mode,
file_mode=self.tree.file_mode,
)
status = STATUS_DELETED
desc = "Downloading"
else:
func = remote.upload
func = _log_exceptions(remote.tree.upload, "upload")
status = STATUS_NEW
desc = "Uploading"
if jobs is None:
jobs = remote.JOBS
jobs = remote.tree.JOBS
dir_status, file_status, dir_contents = self._status(
named_cache,
@@ -486,26 +618,28 @@ class LocalRemote(BaseRemote):
func = pbar.wrap_fn(func)
with ThreadPoolExecutor(max_workers=jobs) as executor:
if download:
fails = sum(executor.map(func, *dir_plans))
fails += sum(executor.map(func, *file_plans))
from_infos, to_infos, names, _ = (
d + f for d, f in zip(dir_plans, file_plans)
)
fails = sum(
executor.map(func, from_infos, to_infos, names)
)
else:
# for uploads, push files first, and any .dir files last
file_futures = {}
for from_info, to_info, name, checksum in zip(*file_plans):
file_futures[checksum] = executor.submit(
for from_info, to_info, name, hash_ in zip(*file_plans):
file_futures[hash_] = executor.submit(
func, from_info, to_info, name
)
dir_futures = {}
for from_info, to_info, name, dir_checksum in zip(
*dir_plans
):
for from_info, to_info, name, dir_hash in zip(*dir_plans):
wait_futures = {
future
for file_checksum, future in file_futures.items()
if file_checksum in dir_contents[dir_checksum]
for file_hash, future in file_futures.items()
if file_hash in dir_contents[dir_hash]
}
dir_futures[dir_checksum] = executor.submit(
dir_futures[dir_hash] = executor.submit(
self._dir_upload,
func,
wait_futures,
@@ -528,16 +662,14 @@ class LocalRemote(BaseRemote):
if not download:
# index successfully pushed dirs
for dir_checksum, future in dir_futures.items():
for dir_hash, future in dir_futures.items():
if future.result() == 0:
file_checksums = dir_contents[dir_checksum]
file_hashes = dir_contents[dir_hash]
logger.debug(
"Indexing pushed dir '{}' with "
"'{}' nested files".format(
dir_checksum, len(file_checksums)
)
"'{}' nested files".format(dir_hash, len(file_hashes))
)
remote.index.update([dir_checksum], file_checksums)
remote.index.update([dir_hash], file_hashes)
return len(dir_plans[0]) + len(file_plans[0])
@@ -551,9 +683,7 @@ class LocalRemote(BaseRemote):
"failed to upload full contents of '{}', "
"aborting .dir file upload".format(name)
)
logger.error(
"failed to upload '{}' to '{}'".format(from_info, to_info)
)
logger.error(f"failed to upload '{from_info}' to '{to_info}'")
return 1
return func(from_info, to_info, name)
@@ -578,10 +708,10 @@ class LocalRemote(BaseRemote):
)
@staticmethod
def _log_missing_caches(checksum_info_dict):
def _log_missing_caches(hash_info_dict):
missing_caches = [
(md5, info)
for md5, info in checksum_info_dict.items()
for md5, info in hash_info_dict.items()
if info["status"] == STATUS_MISSING
]
if missing_caches:
@@ -594,131 +724,3 @@ class LocalRemote(BaseRemote):
"nor on remote. Missing cache files: {}".format(missing_desc)
)
logger.warning(msg)
def _unprotect_file(self, path):
if System.is_symlink(path) or System.is_hardlink(path):
logger.debug("Unprotecting '{}'".format(path))
tmp = os.path.join(os.path.dirname(path), "." + uuid())
# The operations order is important here - if some application
# would access the file during the process of copyfile then it
# would get only the part of file. So, at first, the file should be
# copied with the temporary name, and then original file should be
# replaced by new.
copyfile(path, tmp, name="Unprotecting '{}'".format(relpath(path)))
remove(path)
os.rename(tmp, path)
else:
logger.debug(
"Skipping copying for '{}', since it is not "
"a symlink or a hardlink.".format(path)
)
os.chmod(path, self._file_mode)
def _unprotect_dir(self, path):
assert is_working_tree(self.repo.tree)
for fname in self.repo.tree.walk_files(path):
self._unprotect_file(fname)
def unprotect(self, path_info):
path = path_info.fspath
if not os.path.exists(path):
raise DvcException(
"can't unprotect non-existing data '{}'".format(path)
)
if os.path.isdir(path):
self._unprotect_dir(path)
else:
self._unprotect_file(path)
def protect(self, path_info):
path = fspath_py35(path_info)
mode = self.CACHE_MODE
try:
os.chmod(path, mode)
except OSError as exc:
# There is nothing we need to do in case of a read-only file system
if exc.errno == errno.EROFS:
return
# In shared cache scenario, we might not own the cache file, so we
# need to check if cache file is already protected.
if exc.errno not in [errno.EPERM, errno.EACCES]:
raise
actual = stat.S_IMODE(os.stat(path).st_mode)
if actual != mode:
raise
def _get_unpacked_dir_path_info(self, checksum):
info = self.checksum_to_path_info(checksum)
return info.with_name(info.name + self.UNPACKED_DIR_SUFFIX)
def _remove_unpacked_dir(self, checksum):
path_info = self._get_unpacked_dir_path_info(checksum)
self.remove(path_info)
def _path_info_changed(self, path_info):
if self.exists(path_info) and self.state.get(path_info):
return False
return True
def _update_unpacked_dir(self, checksum):
unpacked_dir_info = self._get_unpacked_dir_path_info(checksum)
if not self._path_info_changed(unpacked_dir_info):
return
self.remove(unpacked_dir_info)
try:
dir_info = self.get_dir_cache(checksum)
self._create_unpacked_dir(checksum, dir_info, unpacked_dir_info)
except DvcException:
logger.warning("Could not create '{}'".format(unpacked_dir_info))
self.remove(unpacked_dir_info)
def _create_unpacked_dir(self, checksum, dir_info, unpacked_dir_info):
self.makedirs(unpacked_dir_info)
for entry in Tqdm(dir_info, desc="Creating unpacked dir", unit="file"):
entry_cache_info = self.checksum_to_path_info(
entry[self.PARAM_CHECKSUM]
)
relative_path = entry[self.PARAM_RELPATH]
# In shared cache mode some cache files might not be owned by the
# user, so we need to use symlinks because, unless
# /proc/sys/fs/protected_hardlinks is disabled, the user is not
# allowed to create hardlinks to files that he doesn't own.
link_types = ["hardlink", "symlink"]
self._link(
entry_cache_info, unpacked_dir_info / relative_path, link_types
)
self.state.save(unpacked_dir_info, checksum)
def _changed_unpacked_dir(self, checksum):
status_unpacked_dir_info = self._get_unpacked_dir_path_info(checksum)
return not self.state.get(status_unpacked_dir_info)
def _get_unpacked_dir_names(self, checksums):
unpacked = set()
for c in checksums:
if self.is_dir_checksum(c):
unpacked.add(c + self.UNPACKED_DIR_SUFFIX)
return unpacked
def is_protected(self, path_info):
if not self.exists(path_info):
return False
mode = os.stat(fspath_py35(path_info)).st_mode
return stat.S_IMODE(mode) == self.CACHE_MODE
+32 -36
View File
@@ -1,20 +1,18 @@
import logging
import os
import posixpath
import threading
from funcy import cached_property, wrap_prop
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
class OSSRemote(BaseRemote):
class OSSRemoteTree(BaseRemoteTree):
"""
oss2 document:
https://www.alibabacloud.com/help/doc-detail/32026.htm
@@ -34,7 +32,7 @@ class OSSRemote(BaseRemote):
"""
scheme = Schemes.OSS
path_cls = CloudURLInfo
PATH_CLS = CloudURLInfo
REQUIRES = {"oss2": "oss2"}
PARAM_CHECKSUM = "etag"
COPY_POLL_SECONDS = 5
@@ -44,7 +42,7 @@ class OSSRemote(BaseRemote):
super().__init__(repo, config)
url = config.get("url")
self.path_info = self.path_cls(url) if url else None
self.path_info = self.PATH_CLS(url) if url else None
self.endpoint = config.get("oss_endpoint") or os.getenv("OSS_ENDPOINT")
@@ -65,9 +63,9 @@ class OSSRemote(BaseRemote):
def oss_service(self):
import oss2
logger.debug("URL: {}".format(self.path_info))
logger.debug("key id: {}".format(self.key_id))
logger.debug("key secret: {}".format(self.key_secret))
logger.debug(f"URL: {self.path_info}")
logger.debug(f"key id: {self.key_id}")
logger.debug(f"key secret: {self.key_secret}")
auth = oss2.Auth(self.key_id, self.key_secret)
bucket = oss2.Bucket(auth, self.endpoint, self.path_info.bucket)
@@ -84,30 +82,37 @@ class OSSRemote(BaseRemote):
)
return bucket
def _generate_download_url(self, path_info, expires=3600):
assert path_info.bucket == self.path_info.bucket
return self.oss_service.sign_url("GET", path_info.path, expires)
def exists(self, path_info):
paths = self._list_paths(path_info)
return any(path_info.path == path for path in paths)
def _list_paths(self, path_info):
import oss2
for blob in oss2.ObjectIterator(
self.oss_service, prefix=path_info.path
):
yield blob.key
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(path_info):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
logger.debug("Removing oss://{}".format(path_info))
logger.debug(f"Removing oss://{path_info}")
self.oss_service.delete_object(path_info.path)
def _list_paths(self, prefix, progress_callback=None):
import oss2
for blob in oss2.ObjectIterator(self.oss_service, prefix=prefix):
if progress_callback:
progress_callback()
yield blob.key
def list_cache_paths(self, prefix=None, progress_callback=None):
if prefix:
prefix = posixpath.join(
self.path_info.path, prefix[:2], prefix[2:]
)
else:
prefix = self.path_info.path
return self._list_paths(prefix, progress_callback)
def _upload(
self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
):
@@ -123,12 +128,3 @@ class OSSRemote(BaseRemote):
self.oss_service.get_object_to_file(
from_info.path, to_file, progress_callback=pbar.update_to
)
def _generate_download_url(self, path_info, expires=3600):
assert path_info.bucket == self.path_info.bucket
return self.oss_service.sign_url("GET", path_info.path, expires)
def exists(self, path_info):
paths = self._list_paths(path_info.path)
return any(path_info.path == path for path in paths)
+2 -2
View File
@@ -1,6 +1,6 @@
import threading
from collections import deque
from contextlib import contextmanager
import threading
from funcy import memoize, wrap_with
@@ -30,7 +30,7 @@ def close_pools():
get_pool.memory.clear()
class Pool(object):
class Pool:
def __init__(self, conn_func, *conn_args, **conn_kwargs):
self._conn_func = conn_func
self._conn_args = conn_args
+148 -167
View File
@@ -1,26 +1,22 @@
# -*- coding: utf-8 -*-
import logging
import os
import posixpath
import threading
from funcy import cached_property, wrap_prop
from dvc.config import ConfigError
from dvc.exceptions import DvcException
from dvc.exceptions import ETagMismatchError
from dvc.exceptions import DvcException, ETagMismatchError
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
class S3Remote(BaseRemote):
class S3RemoteTree(BaseRemoteTree):
scheme = Schemes.S3
path_cls = CloudURLInfo
PATH_CLS = CloudURLInfo
REQUIRES = {"boto3": "boto3"}
PARAM_CHECKSUM = "etag"
@@ -28,7 +24,7 @@ class S3Remote(BaseRemote):
super().__init__(repo, config)
url = config.get("url", "s3://")
self.path_info = self.path_cls(url)
self.path_info = self.PATH_CLS(url)
self.region = config.get("region")
self.profile = config.get("profile")
@@ -47,6 +43,10 @@ class S3Remote(BaseRemote):
if self.sse:
self.extra_args["ServerSideEncryption"] = self.sse
self.sse_kms_key_id = config.get("sse_kms_key_id")
if self.sse_kms_key_id:
self.extra_args["SSEKMSKeyId"] = self.sse_kms_key_id
self.acl = config.get("acl")
if self.acl:
self.extra_args["ACL"] = self.acl
@@ -76,20 +76,145 @@ class S3Remote(BaseRemote):
return obj["ETag"].strip('"')
def get_file_checksum(self, path_info):
return self.get_etag(self.s3, path_info.bucket, path_info.path)
@staticmethod
def get_head_object(s3, bucket, path, *args, **kwargs):
try:
obj = s3.head_object(Bucket=bucket, Key=path, *args, **kwargs)
except Exception as exc:
raise DvcException(
"s3://{}/{} does not exist".format(bucket, path)
) from exc
raise DvcException(f"s3://{bucket}/{path} does not exist") from exc
return obj
def _append_aws_grants_to_extra_args(self, config):
# Keys for extra_args can be one of the following list:
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS
"""
ALLOWED_UPLOAD_ARGS = [
'ACL', 'CacheControl', 'ContentDisposition', 'ContentEncoding',
'ContentLanguage', 'ContentType', 'Expires', 'GrantFullControl',
'GrantRead', 'GrantReadACP', 'GrantWriteACP', 'Metadata',
'RequestPayer', 'ServerSideEncryption', 'StorageClass',
'SSECustomerAlgorithm', 'SSECustomerKey', 'SSECustomerKeyMD5',
'SSEKMSKeyId', 'WebsiteRedirectLocation'
]
"""
grants = {
"grant_full_control": "GrantFullControl",
"grant_read": "GrantRead",
"grant_read_acp": "GrantReadACP",
"grant_write_acp": "GrantWriteACP",
}
for grant_option, extra_args_key in grants.items():
if config.get(grant_option):
if self.acl:
raise ConfigError(
"`acl` and `grant_*` AWS S3 config options "
"are mutually exclusive"
)
self.extra_args[extra_args_key] = config.get(grant_option)
def _generate_download_url(self, path_info, expires=3600):
params = {"Bucket": path_info.bucket, "Key": path_info.path}
return self.s3.generate_presigned_url(
ClientMethod="get_object", Params=params, ExpiresIn=int(expires)
)
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def isdir(self, path_info):
# S3 doesn't have a concept for directories.
#
# Using `head_object` with a path pointing to a directory
# will throw a 404 error.
#
# A reliable way to know if a given path is a directory is by
# checking if there are more files sharing the same prefix
# with a `list_objects` call.
#
# We need to make sure that the path ends with a forward slash,
# since we can end with false-positives like the following example:
#
# bucket
# └── data
# ├── alice
# └── alpha
#
# Using `data/al` as prefix will return `[data/alice, data/alpha]`,
# While `data/al/` will return nothing.
#
dir_path = path_info / ""
return bool(list(self._list_paths(dir_path, max_items=1)))
def isfile(self, path_info):
from botocore.exceptions import ClientError
if path_info.path.endswith("/"):
return False
try:
self.s3.head_object(Bucket=path_info.bucket, Key=path_info.path)
except ClientError as exc:
if exc.response["Error"]["Code"] != "404":
raise
return False
return True
def _list_objects(self, path_info, max_items=None):
""" Read config for list object api, paginate through list objects."""
kwargs = {
"Bucket": path_info.bucket,
"Prefix": path_info.path,
"PaginationConfig": {"MaxItems": max_items},
}
paginator = self.s3.get_paginator(self.list_objects_api)
for page in paginator.paginate(**kwargs):
contents = page.get("Contents", ())
yield from contents
def _list_paths(self, path_info, max_items=None):
return (
item["Key"] for item in self._list_objects(path_info, max_items)
)
def walk_files(self, path_info, **kwargs):
for fname in self._list_paths(path_info / "", **kwargs):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != "s3":
raise NotImplementedError
logger.debug(f"Removing {path_info}")
self.s3.delete_object(Bucket=path_info.bucket, Key=path_info.path)
def makedirs(self, path_info):
# We need to support creating empty directories, which means
# creating an object with an empty body and a trailing slash `/`.
#
# We are not creating directory objects for every parent prefix,
# as it is not required.
if not path_info.path:
return
dir_path = path_info / ""
self.s3.put_object(Bucket=path_info.bucket, Key=dir_path.path, Body="")
def copy(self, from_info, to_info):
self._copy(self.s3, from_info, to_info, self.extra_args)
@classmethod
def _copy_multipart(
cls, s3, from_info, to_info, size, n_parts, extra_args
@@ -102,7 +227,7 @@ class S3Remote(BaseRemote):
parts = []
byte_position = 0
for i in range(1, n_parts + 1):
obj = cls.get_head_object(
obj = S3RemoteTree.get_head_object(
s3, from_info.bucket, from_info.path, PartNumber=i
)
part_size = obj["ContentLength"]
@@ -110,7 +235,7 @@ class S3Remote(BaseRemote):
if lastbyte > size:
lastbyte = size - 1
srange = "bytes={}-{}".format(byte_position, lastbyte)
srange = f"bytes={byte_position}-{lastbyte}"
part = s3.upload_part_copy(
Bucket=to_info.bucket,
@@ -157,7 +282,9 @@ class S3Remote(BaseRemote):
# object is transfered in the same chunks as it was originally.
from boto3.s3.transfer import TransferConfig
obj = cls.get_head_object(s3, from_info.bucket, from_info.path)
obj = S3RemoteTree.get_head_object(
s3, from_info.bucket, from_info.path
)
etag = obj["ETag"].strip('"')
size = obj["ContentLength"]
@@ -177,114 +304,12 @@ class S3Remote(BaseRemote):
Config=TransferConfig(multipart_threshold=size + 1),
)
cached_etag = cls.get_etag(s3, to_info.bucket, to_info.path)
cached_etag = S3RemoteTree.get_etag(s3, to_info.bucket, to_info.path)
if etag != cached_etag:
raise ETagMismatchError(etag, cached_etag)
def copy(self, from_info, to_info):
self._copy(self.s3, from_info, to_info, self.extra_args)
def remove(self, path_info):
if path_info.scheme != "s3":
raise NotImplementedError
logger.debug("Removing {}".format(path_info))
self.s3.delete_object(Bucket=path_info.bucket, Key=path_info.path)
def _list_objects(
self, path_info, max_items=None, prefix=None, progress_callback=None
):
""" Read config for list object api, paginate through list objects."""
kwargs = {
"Bucket": path_info.bucket,
"Prefix": path_info.path,
"PaginationConfig": {"MaxItems": max_items},
}
if prefix:
kwargs["Prefix"] = posixpath.join(path_info.path, prefix[:2])
paginator = self.s3.get_paginator(self.list_objects_api)
for page in paginator.paginate(**kwargs):
contents = page.get("Contents", ())
if progress_callback:
for item in contents:
progress_callback()
yield item
else:
yield from contents
def _list_paths(
self, path_info, max_items=None, prefix=None, progress_callback=None
):
return (
item["Key"]
for item in self._list_objects(
path_info, max_items, prefix, progress_callback
)
)
def list_cache_paths(self, prefix=None, progress_callback=None):
return self._list_paths(
self.path_info, prefix=prefix, progress_callback=progress_callback
)
def isfile(self, path_info):
from botocore.exceptions import ClientError
if path_info.path.endswith("/"):
return False
try:
self.s3.head_object(Bucket=path_info.bucket, Key=path_info.path)
except ClientError as exc:
if exc.response["Error"]["Code"] != "404":
raise
return False
return True
def exists(self, path_info):
"""Check if the blob exists. If it does not exist,
it could be a part of a directory path.
eg: if `data/file.txt` exists, check for `data` should return True
"""
return self.isfile(path_info) or self.isdir(path_info)
def makedirs(self, path_info):
# We need to support creating empty directories, which means
# creating an object with an empty body and a trailing slash `/`.
#
# We are not creating directory objects for every parent prefix,
# as it is not required.
if not path_info.path:
return
dir_path = path_info / ""
self.s3.put_object(Bucket=path_info.bucket, Key=dir_path.path, Body="")
def isdir(self, path_info):
# S3 doesn't have a concept for directories.
#
# Using `head_object` with a path pointing to a directory
# will throw a 404 error.
#
# A reliable way to know if a given path is a directory is by
# checking if there are more files sharing the same prefix
# with a `list_objects` call.
#
# We need to make sure that the path ends with a forward slash,
# since we can end with false-positives like the following example:
#
# bucket
# └── data
# ├── alice
# └── alpha
#
# Using `data/al` as prefix will return `[data/alice, data/alpha]`,
# While `data/al/` will return nothing.
#
dir_path = path_info / ""
return bool(list(self._list_paths(dir_path, max_items=1)))
def get_file_hash(self, path_info):
return self.get_etag(self.s3, path_info.bucket, path_info.path)
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
total = os.path.getsize(from_file)
@@ -312,47 +337,3 @@ class S3Remote(BaseRemote):
self.s3.download_file(
from_info.bucket, from_info.path, to_file, Callback=pbar.update
)
def _generate_download_url(self, path_info, expires=3600):
params = {"Bucket": path_info.bucket, "Key": path_info.path}
return self.s3.generate_presigned_url(
ClientMethod="get_object", Params=params, ExpiresIn=int(expires)
)
def walk_files(self, path_info, max_items=None):
for fname in self._list_paths(path_info / "", max_items):
if fname.endswith("/"):
continue
yield path_info.replace(path=fname)
def _append_aws_grants_to_extra_args(self, config):
# Keys for extra_args can be one of the following list:
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS
"""
ALLOWED_UPLOAD_ARGS = [
'ACL', 'CacheControl', 'ContentDisposition', 'ContentEncoding',
'ContentLanguage', 'ContentType', 'Expires', 'GrantFullControl',
'GrantRead', 'GrantReadACP', 'GrantWriteACP', 'Metadata',
'RequestPayer', 'ServerSideEncryption', 'StorageClass',
'SSECustomerAlgorithm', 'SSECustomerKey', 'SSECustomerKeyMD5',
'SSEKMSKeyId', 'WebsiteRedirectLocation'
]
"""
grants = {
"grant_full_control": "GrantFullControl",
"grant_read": "GrantRead",
"grant_read_acp": "GrantReadACP",
"grant_write_acp": "GrantWriteACP",
}
for grant_option, extra_args_key in grants.items():
if config.get(grant_option):
if self.acl:
raise ConfigError(
"`acl` and `grant_*` AWS S3 config options "
"are mutually exclusive"
)
self.extra_args[extra_args_key] = config.get(grant_option)
+1 -2
View File
@@ -1,10 +1,9 @@
import logging
import sys
import time
from functools import wraps
import colorama
import colorama
logger = logging.getLogger(__name__)
this = sys.modules[__name__]
+65 -59
View File
@@ -10,11 +10,11 @@ from concurrent.futures import ThreadPoolExecutor
from contextlib import closing, contextmanager
from urllib.parse import urlparse
from funcy import memoize, wrap_with, silent, first
from funcy import first, memoize, silent, wrap_with
import dvc.prompt as prompt
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.remote.base import BaseRemoteTree, Remote
from dvc.remote.pool import get_connection
from dvc.scheme import Schemes
from dvc.utils import to_chunks
@@ -33,21 +33,21 @@ def ask_password(host, user, port):
)
class SSHRemote(BaseRemote):
class SSHRemoteTree(BaseRemoteTree):
scheme = Schemes.SSH
REQUIRES = {"paramiko": "paramiko"}
JOBS = 4
PARAM_CHECKSUM = "md5"
DEFAULT_PORT = 22
TIMEOUT = 1800
# At any given time some of the connections will go over network and
# paramiko stuff, so we would ideally have it double of server processors.
# We use conservative setting of 4 instead to not exhaust max sessions.
CHECKSUM_JOBS = 4
DEFAULT_CACHE_TYPES = ["copy"]
TRAVERSE_PREFIX_LEN = 2
DEFAULT_CACHE_TYPES = ["copy"]
DEFAULT_PORT = 22
TIMEOUT = 1800
def __init__(self, repo, config):
super().__init__(repo, config)
@@ -69,7 +69,7 @@ class SSHRemote(BaseRemote):
or self._try_get_ssh_config_port(user_ssh_config)
or self.DEFAULT_PORT
)
self.path_info = self.path_cls.from_parts(
self.path_info = self.PATH_CLS.from_parts(
scheme=self.scheme,
host=host,
user=user,
@@ -103,7 +103,7 @@ class SSHRemote(BaseRemote):
def _load_user_ssh_config(hostname):
import paramiko
user_config_file = SSHRemote.ssh_config_filename()
user_config_file = SSHRemoteTree.ssh_config_filename()
user_ssh_config = {}
if hostname and os.path.exists(user_config_file):
ssh_config = paramiko.SSHConfig()
@@ -148,17 +148,22 @@ class SSHRemote(BaseRemote):
sock=self.sock,
)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb", "wb"}
with self.ssh(path_info) as ssh, closing(
ssh.sftp.open(path_info.path, mode)
) as fd:
if "b" in mode:
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
def exists(self, path_info):
with self.ssh(path_info) as ssh:
return ssh.exists(path_info.path)
def get_file_checksum(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(path_info) as ssh:
return ssh.md5(path_info.path)
def isdir(self, path_info):
with self.ssh(path_info) as ssh:
return ssh.isdir(path_info.path)
@@ -167,9 +172,29 @@ class SSHRemote(BaseRemote):
with self.ssh(path_info) as ssh:
return ssh.isfile(path_info.path)
def getsize(self, path_info):
def walk_files(self, path_info, **kwargs):
with self.ssh(path_info) as ssh:
return ssh.getsize(path_info.path)
for fname in ssh.walk_files(path_info.path):
yield path_info.replace(path=fname)
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(path_info) as ssh:
ssh.remove(path_info.path)
def makedirs(self, path_info):
with self.ssh(path_info) as ssh:
ssh.makedirs(path_info.path)
def move(self, from_info, to_info, mode=None):
assert mode is None
if from_info.scheme != self.scheme or to_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(from_info) as ssh:
ssh.move(from_info.path, to_info.path)
def copy(self, from_info, to_info):
if not from_info.scheme == to_info.scheme == self.scheme:
@@ -212,20 +237,16 @@ class SSHRemote(BaseRemote):
with self.ssh(from_info) as ssh:
ssh.reflink(from_info.path, to_info.path)
def remove(self, path_info):
def get_file_hash(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(path_info) as ssh:
ssh.remove(path_info.path)
return ssh.md5(path_info.path)
def move(self, from_info, to_info, mode=None):
assert mode is None
if from_info.scheme != self.scheme or to_info.scheme != self.scheme:
raise NotImplementedError
with self.ssh(from_info) as ssh:
ssh.move(from_info.path, to_info.path)
def getsize(self, path_info):
with self.ssh(path_info) as ssh:
return ssh.getsize(path_info.path)
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
assert from_info.isin(self.path_info)
@@ -247,24 +268,14 @@ class SSHRemote(BaseRemote):
no_progress_bar=no_progress_bar,
)
@contextmanager
def open(self, path_info, mode="r", encoding=None):
assert mode in {"r", "rt", "rb", "wb"}
with self.ssh(path_info) as ssh, closing(
ssh.sftp.open(path_info.path, mode)
) as fd:
if "b" in mode:
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
def list_cache_paths(self, prefix=None, progress_callback=None):
def list_paths(self, prefix=None, progress_callback=None):
if prefix:
root = posixpath.join(self.path_info.path, prefix[:2])
else:
root = self.path_info.path
with self.ssh(self.path_info) as ssh:
if prefix and not ssh.exists(root):
return
# If we simply return an iterator then with above closes instantly
if progress_callback:
for path in ssh.walk_files(root):
@@ -273,15 +284,8 @@ class SSHRemote(BaseRemote):
else:
yield from ssh.walk_files(root)
def walk_files(self, path_info):
with self.ssh(path_info) as ssh:
for fname in ssh.walk_files(path_info.path):
yield path_info.replace(path=fname)
def makedirs(self, path_info):
with self.ssh(path_info) as ssh:
ssh.makedirs(path_info.path)
class SSHRemote(Remote):
def batch_exists(self, path_infos, callback):
def _exists(chunk_and_channel):
chunk, channel = chunk_and_channel
@@ -290,14 +294,14 @@ class SSHRemote(BaseRemote):
try:
channel.stat(path)
ret.append(True)
except IOError as exc:
except OSError as exc:
if exc.errno != errno.ENOENT:
raise
ret.append(False)
callback(path)
return ret
with self.ssh(path_infos[0]) as ssh:
with self.tree.ssh(path_infos[0]) as ssh:
channels = ssh.open_max_sftp_channels()
max_workers = len(channels)
@@ -310,32 +314,34 @@ class SSHRemote(BaseRemote):
return results
def cache_exists(self, checksums, jobs=None, name=None):
def hashes_exist(self, hashes, jobs=None, name=None):
"""This is older implementation used in remote/base.py
We are reusing it in RemoteSSH, because SSH's batch_exists proved to be
faster than current approach (relying on exists(path_info)) applied in
remote/base.
"""
if not self.CAN_TRAVERSE:
return list(set(checksums) & set(self.all()))
if not self.tree.CAN_TRAVERSE:
return list(set(hashes) & set(self.tree.all()))
# possibly prompt for credentials before "Querying" progress output
self.ensure_credentials()
self.tree.ensure_credentials()
with Tqdm(
desc="Querying "
+ ("cache in " + name if name else "remote cache"),
total=len(checksums),
total=len(hashes),
unit="file",
) as pbar:
def exists_with_progress(chunks):
return self.batch_exists(chunks, callback=pbar.update_msg)
with ThreadPoolExecutor(max_workers=jobs or self.JOBS) as executor:
path_infos = [self.checksum_to_path_info(x) for x in checksums]
chunks = to_chunks(path_infos, num_chunks=self.JOBS)
with ThreadPoolExecutor(
max_workers=jobs or self.tree.JOBS
) as executor:
path_infos = [self.hash_to_path_info(x) for x in hashes]
chunks = to_chunks(path_infos, num_chunks=self.tree.JOBS)
results = executor.map(exists_with_progress, chunks)
in_remote = itertools.chain.from_iterable(results)
ret = list(itertools.compress(checksums, in_remote))
ret = list(itertools.compress(hashes, in_remote))
return ret
+24 -22
View File
@@ -7,16 +7,16 @@ from contextlib import suppress
from funcy import cached_property
from dvc.exceptions import DvcException
from dvc.progress import Tqdm
from dvc.remote.base import RemoteCmdError
from dvc.utils import tmp_fname
try:
import paramiko
except ImportError:
paramiko = None
from dvc.utils import tmp_fname
from dvc.progress import Tqdm
from dvc.exceptions import DvcException
from dvc.remote.base import RemoteCmdError
logger = logging.getLogger(__name__)
@@ -25,9 +25,9 @@ def sizeof_fmt(num, suffix="B"):
""" Convert number of bytes to human-readable string """
for unit in ["", "K", "M", "G", "T", "P", "E", "Z"]:
if abs(num) < 1024.0:
return "%3.1f%s%s" % (num, unit, suffix)
return f"{num:3.1f}{unit}{suffix}"
num /= 1024.0
return "%.1f%s%s" % (num, "Y", suffix)
return "{:.1f}{}{}".format(num, "Y", suffix)
class SSHConnection:
@@ -100,7 +100,7 @@ class SSHConnection:
if stat.S_ISREG(st_mode) or stat.S_ISLNK(st_mode):
raise DvcException(
"a file with the same name '{}' already exists".format(path)
f"a file with the same name '{path}' already exists"
)
head, tail = posixpath.split(path)
@@ -111,12 +111,12 @@ class SSHConnection:
if tail:
try:
self.sftp.mkdir(path)
except IOError as exc:
except OSError as exc:
# Since paramiko errors are very vague we need to recheck
# whether it's because path already exists or something else
if exc.errno == errno.EACCES or not self.exists(path):
raise DvcException(
"unable to create remote directory '{}'".format(path)
f"unable to create remote directory '{path}'"
) from exc
def walk(self, directory, topdown=True):
@@ -126,7 +126,7 @@ class SSHConnection:
# [1] https://github.com/python/cpython/blob/master/Lib/os.py
try:
dir_entries = self.sftp.listdir_attr(directory)
except IOError as exc:
except OSError as exc:
raise DvcException(
"couldn't get the '{}' remote directory files list".format(
directory
@@ -237,11 +237,15 @@ class SSHConnection:
or channel.recv_ready()
or channel.recv_stderr_ready()
):
import select
import selectors
selector = selectors.DefaultSelector()
selector.register(stdout.channel, selectors.EVENT_READ)
got_chunk = False
readq, _, _ = select.select([stdout.channel], [], [], self.timeout)
for c in readq:
events = selector.select(self.timeout)
for key, _ in events:
c = key.fileobj
if c.recv_ready():
stdout_chunks.append(stdout.channel.recv(len(c.in_buffer)))
got_chunk = True
@@ -292,14 +296,14 @@ class SSHConnection:
md5 = self.execute("md5 " + path).split()[-1]
else:
raise DvcException(
"'{}' is not supported as a SSH remote".format(self.uname)
f"'{self.uname}' is not supported as a SSH remote"
)
assert len(md5) == 32
return md5
def copy(self, src, dest):
self.execute("cp {} {}".format(src, dest))
self.execute(f"cp {src} {dest}")
def open_max_sftp_channels(self):
# If there are more than 1 it means we've already opened max amount
@@ -321,14 +325,12 @@ class SSHConnection:
def reflink(self, src, dest):
if self.uname == "Linux":
return self.execute("cp --reflink {} {}".format(src, dest))
return self.execute(f"cp --reflink {src} {dest}")
if self.uname == "Darwin":
return self.execute("cp -c {} {}".format(src, dest))
return self.execute(f"cp -c {src} {dest}")
raise DvcException(
"'{}' is not supported as a SSH remote".format(self.uname)
)
raise DvcException(f"'{self.uname}' is not supported as a SSH remote")
def hardlink(self, src, dest):
self.execute("ln {} {}".format(src, dest))
self.execute(f"ln {src} {dest}")
+168 -104
View File
@@ -3,23 +3,28 @@ import os
from contextlib import contextmanager
from functools import wraps
from dvc.ignore import CleanTree
from dvc.compat import fspath_py35
from funcy import cached_property, cat, first
from dvc.config import Config
from dvc.dvcfile import PIPELINE_FILE, Dvcfile, is_valid_filename
from dvc.exceptions import FileMissingError
from dvc.exceptions import IsADirectoryError as DvcIsADirectoryError
from dvc.exceptions import (
FileMissingError,
IsADirectoryError,
NoOutputOrStageError,
NotDvcRepoError,
OutputNotFoundError,
)
from dvc.ignore import CleanTree
from dvc.path_info import PathInfo
from dvc.remote.base import RemoteActionNotImplemented
from dvc.repo.tree import RepoTree
from dvc.scm.tree import is_working_tree
from dvc.utils.fs import path_isin
from .graph import check_acyclic, get_pipeline, get_pipelines
from ..stage.exceptions import StageFileDoesNotExistError, StageNotFound
from ..utils import parse_target
from .graph import check_acyclic, get_pipeline, get_pipelines
logger = logging.getLogger(__name__)
def locked(f):
@@ -35,7 +40,7 @@ def locked(f):
return wrapper
class Repo(object):
class Repo:
DVC_DIR = ".dvc"
from dvc.repo.destroy import destroy
@@ -43,7 +48,7 @@ class Repo(object):
from dvc.repo.add import add
from dvc.repo.remove import remove
from dvc.repo.ls import ls
from dvc.repo.lock import lock as lock_stage
from dvc.repo.freeze import freeze, unfreeze
from dvc.repo.move import move
from dvc.repo.run import run
from dvc.repo.imp import imp
@@ -61,31 +66,38 @@ class Repo(object):
from dvc.repo.get import get
from dvc.repo.get_url import get_url
from dvc.repo.update import update
from dvc.repo.plot import plot
def __init__(self, root_dir=None):
from dvc.state import State
def __init__(self, root_dir=None, scm=None, rev=None):
from dvc.state import State, StateNoop
from dvc.lock import make_lock
from dvc.scm import SCM
from dvc.cache import Cache
from dvc.data_cloud import DataCloud
from dvc.repo.metrics import Metrics
from dvc.repo.plots import Plots
from dvc.repo.params import Params
from dvc.scm.tree import WorkingTree
from dvc.utils.fs import makedirs
from dvc.stage.cache import StageCache
root_dir = self.find_root(root_dir)
if scm:
# use GitTree instead of WorkingTree as default repo tree instance
tree = scm.get_tree(rev)
self.root_dir = self.find_root(root_dir, tree)
self.scm = scm
self.tree = tree
self.state = StateNoop()
else:
root_dir = self.find_root(root_dir)
self.root_dir = os.path.abspath(os.path.realpath(root_dir))
self.root_dir = os.path.abspath(os.path.realpath(root_dir))
self.dvc_dir = os.path.join(self.root_dir, self.DVC_DIR)
self.config = Config(self.dvc_dir)
no_scm = self.config["core"].get("no_scm", False)
self.scm = SCM(self.root_dir, no_scm=no_scm)
self.tree = WorkingTree(self.root_dir)
if not scm:
no_scm = self.config["core"].get("no_scm", False)
self.scm = SCM(self.root_dir, no_scm=no_scm)
self.tree = WorkingTree(self.root_dir)
self.tmp_dir = os.path.join(self.dvc_dir, "tmp")
self.index_dir = os.path.join(self.tmp_dir, "index")
@@ -99,16 +111,19 @@ class Repo(object):
friendly=True,
)
# NOTE: storing state and link_state in the repository itself to avoid
# any possible state corruption in 'shared cache dir' scenario.
self.state = State(self)
self.cache = Cache(self)
self.cloud = DataCloud(self)
self.stage_cache = StageCache(self.cache.local.cache_dir)
if not scm:
# NOTE: storing state and link_state in the repository itself to
# avoid any possible state corruption in 'shared cache dir'
# scenario.
self.state = State(self.cache.local)
self.stage_cache = StageCache(self)
self.metrics = Metrics(self)
self.plots = Plots(self)
self.params = Params(self)
self._ignore()
@@ -119,20 +134,31 @@ class Repo(object):
@tree.setter
def tree(self, tree):
self._tree = tree if isinstance(tree, CleanTree) else CleanTree(tree)
if is_working_tree(tree) or tree.tree_root == self.root_dir:
root = None
else:
root = self.root_dir
self._tree = (
tree if isinstance(tree, CleanTree) else CleanTree(tree, root)
)
# Our graph cache is no longer valid, as it was based on the previous
# tree.
self._reset()
def __repr__(self):
return "{}: '{}'".format(self.__class__.__name__, self.root_dir)
return f"{self.__class__.__name__}: '{self.root_dir}'"
@classmethod
def find_root(cls, root=None):
def find_root(cls, root=None, tree=None):
root_dir = os.path.realpath(root or os.curdir)
if tree:
if tree.isdir(os.path.join(root_dir, cls.DVC_DIR)):
return root_dir
raise NotDvcRepoError(f"'{root}' does not contain DVC directory")
if not os.path.isdir(root_dir):
raise NotDvcRepoError("directory '{}' does not exist".format(root))
raise NotDvcRepoError(f"directory '{root}' does not exist")
while True:
dvc_dir = os.path.join(root_dir, cls.DVC_DIR)
@@ -161,7 +187,7 @@ class Repo(object):
return Repo(root_dir)
def unprotect(self, target):
return self.cache.local.unprotect(PathInfo(target))
return self.cache.local.tree.unprotect(PathInfo(target))
def _ignore(self):
flist = [self.config.files["local"], self.tmp_dir]
@@ -171,6 +197,25 @@ class Repo(object):
self.scm.ignore_list(flist)
def get_stage(self, path=None, name=None):
if not path:
path = PIPELINE_FILE
logger.debug("Assuming '%s' to be a stage inside '%s'", name, path)
dvcfile = Dvcfile(self, path)
return dvcfile.stages[name]
def get_stages(self, path=None, name=None):
if not path:
path = PIPELINE_FILE
logger.debug("Assuming '%s' to be a stage inside '%s'", name, path)
if name:
return [self.get_stage(path, name)]
dvcfile = Dvcfile(self, path)
return list(dvcfile.stages.values())
def check_modified_graph(self, new_stages):
"""Generate graph including the new stage to check for errors"""
# Building graph might be costly for the ones with many DVC-files,
@@ -194,10 +239,9 @@ class Repo(object):
stages = nx.dfs_postorder_nodes(graph)
return [stage for stage in stages if path_isin(stage.path, path)]
def collect(self, target, with_deps=False, recursive=False, graph=None):
import networkx as nx
from ..dvcfile import Dvcfile
def collect(
self, target=None, with_deps=False, recursive=False, graph=None
):
if not target:
return list(graph) if graph else self.stages
@@ -207,36 +251,81 @@ class Repo(object):
)
path, name = parse_target(target)
dvcfile = Dvcfile(self, path)
stages = list(dvcfile.stages.filter(name).values())
stages = self.get_stages(path, name)
if not with_deps:
return stages
res = set()
for stage in stages:
pipeline = get_pipeline(get_pipelines(graph or self.graph), stage)
res.update(nx.dfs_postorder_nodes(pipeline, stage))
res.update(self._collect_pipeline(stage, graph=graph))
return res
def collect_granular(self, target, *args, **kwargs):
from ..dvcfile import Dvcfile, is_valid_filename
def _collect_pipeline(self, stage, graph=None):
import networkx as nx
pipeline = get_pipeline(get_pipelines(graph or self.graph), stage)
return nx.dfs_postorder_nodes(pipeline, stage)
def _collect_from_default_dvcfile(self, target):
dvcfile = Dvcfile(self, PIPELINE_FILE)
if dvcfile.exists():
return dvcfile.stages.get(target)
def collect_granular(
self, target=None, with_deps=False, recursive=False, graph=None
):
"""
Priority is in the order of following in case of ambiguity:
- .dvc file or .yaml file
- dir if recursive and directory exists
- stage_name
- output file
"""
if not target:
return [(stage, None) for stage in self.stages]
file, name = parse_target(target)
if is_valid_filename(file) and not kwargs.get("with_deps"):
# Optimization: do not collect the graph for a specific .dvc target
stages = Dvcfile(self, file).stages.filter(name)
return [(stage, None) for stage in stages.values()]
stages = []
try:
(out,) = self.find_outs_by_path(file, strict=False)
filter_info = PathInfo(os.path.abspath(file))
return [(out.stage, filter_info)]
except OutputNotFoundError:
stages = self.collect(target, *args, **kwargs)
return [(stage, None) for stage in stages]
# Optimization: do not collect the graph for a specific target
if not file:
# parsing is ambiguous when it does not have a colon
# or if it's not a dvcfile, as it can be a stage name
# in `dvc.yaml` or, an output in a stage.
logger.debug(
"Checking if stage '%s' is in '%s'", target, PIPELINE_FILE
)
if not (recursive and os.path.isdir(target)):
stage = self._collect_from_default_dvcfile(target)
if stage:
stages = (
self._collect_pipeline(stage) if with_deps else [stage]
)
elif not with_deps and is_valid_filename(file):
stages = self.get_stages(file, name)
if not stages:
if not (recursive and os.path.isdir(target)):
try:
(out,) = self.find_outs_by_path(target, strict=False)
filter_info = PathInfo(os.path.abspath(target))
return [(out.stage, filter_info)]
except OutputNotFoundError:
pass
try:
stages = self.collect(target, with_deps, recursive, graph)
except StageFileDoesNotExistError as exc:
# collect() might try to use `target` as a stage name
# and throw error that dvc.yaml does not exist, whereas it
# should say that both stage name and file does not exist.
if file and is_valid_filename(file):
raise
raise NoOutputOrStageError(target, exc.file) from exc
except StageNotFound as exc:
raise NoOutputOrStageError(target, exc.file) from exc
return [(stage, None) for stage in stages]
def used_cache(
self,
@@ -249,6 +338,7 @@ class Repo(object):
force=False,
jobs=None,
recursive=False,
used_run_cache=None,
):
"""Get the stages related to the given target and collect
the `info` of its outputs.
@@ -282,7 +372,7 @@ class Repo(object):
for target in targets
)
suffix = "({})".format(branch) if branch else ""
suffix = f"({branch})" if branch else ""
for stage, filter_info in pairs:
used_cache = stage.get_used_cache(
remote=remote,
@@ -292,6 +382,12 @@ class Repo(object):
)
cache.update(used_cache, suffix=suffix)
if used_run_cache:
used_cache = self.stage_cache.get_used_cache(
used_run_cache, remote=remote, force=force, jobs=jobs,
)
cache.update(used_cache)
return cache
def _collect_graph(self, stages):
@@ -345,7 +441,7 @@ class Repo(object):
stages = stages or self.stages
outs = Trie() # Use trie to efficiently find overlapping outs and deps
for stage in filter(bool, stages):
for stage in filter(bool, stages): # bug? not using it later
for out in stage.outs:
out_key = out.path_info.parts
@@ -421,28 +517,23 @@ class Repo(object):
@cached_property
def plot_templates(self):
from dvc.repo.plot.template import PlotTemplates
from .plots.template import PlotTemplates
return PlotTemplates(self.dvc_dir)
def _collect_stages(self):
from dvc.dvcfile import Dvcfile, is_valid_filename
stages = []
outs = set()
for root, dirs, files in self.tree.walk(self.root_dir):
for file_name in filter(is_valid_filename, files):
path = os.path.join(root, file_name)
stage_loader = Dvcfile(self, path).stages
with stage_loader.log_level(at=logging.DEBUG):
stages.extend(stage_loader.values())
new_stages = self.get_stages(os.path.join(root, file_name))
stages.extend(new_stages)
outs.update(
out.fspath
for stage in stages
for out in (
out for out in stage.outs if out.scheme == "local"
)
for stage in new_stages
for out in stage.outs
if out.scheme == "local"
)
dirs[:] = [d for d in dirs if os.path.join(root, d) not in outs]
return stages
@@ -482,49 +573,22 @@ class Repo(object):
@contextmanager
def open_by_relpath(self, path, remote=None, mode="r", encoding=None):
"""Opens a specified resource as a file descriptor"""
cause = None
tree = RepoTree(self, stream=True)
path = os.path.join(self.root_dir, path)
try:
out = self.find_out_by_relpath(path)
except OutputNotFoundError as exc:
out = None
cause = exc
if out and out.use_cache:
try:
with self._open_cached(out, remote, mode, encoding) as fd:
yield fd
return
except FileNotFoundError as exc:
raise FileMissingError(path) from exc
abs_path = os.path.join(self.root_dir, path)
if os.path.exists(abs_path):
with open(abs_path, mode=mode, encoding=encoding) as fd:
yield fd
return
raise FileMissingError(path) from cause
def _open_cached(self, out, remote=None, mode="r", encoding=None):
if out.isdir():
raise IsADirectoryError("Can't open a dir")
cache_file = self.cache.local.checksum_to_path_info(out.checksum)
cache_file = fspath_py35(cache_file)
if os.path.exists(cache_file):
return open(cache_file, mode=mode, encoding=encoding)
try:
remote_obj = self.cloud.get_remote(remote)
remote_info = remote_obj.checksum_to_path_info(out.checksum)
return remote_obj.open(remote_info, mode=mode, encoding=encoding)
except RemoteActionNotImplemented:
with self.state:
cache_info = out.get_used_cache(remote=remote)
self.cloud.pull(cache_info, remote=remote)
return open(cache_file, mode=mode, encoding=encoding)
with tree.open(
os.path.join(self.root_dir, path),
mode=mode,
encoding=encoding,
remote=remote,
) as fobj:
yield fobj
except FileNotFoundError as exc:
raise FileMissingError(path) from exc
except IsADirectoryError as exc:
raise DvcIsADirectoryError(f"'{path}' is a directory") from exc
def close(self):
self.scm.close()
+20 -8
View File
@@ -3,24 +3,27 @@ import os
import colorama
from . import locked
from dvc.dvcfile import Dvcfile, is_dvc_file
from ..exceptions import (
RecursiveAddingWhileUsingFilename,
OverlappingOutputPathsError,
OutputDuplicationError,
OverlappingOutputPathsError,
RecursiveAddingWhileUsingFilename,
)
from ..output.base import OutputDoesNotExistError
from ..progress import Tqdm
from ..repo.scm_context import scm_context
from ..utils import LARGE_DIR_SIZE, resolve_paths
from . import locked
logger = logging.getLogger(__name__)
@locked
@scm_context
def add(repo, targets, recursive=False, no_commit=False, fname=None):
def add(
repo, targets, recursive=False, no_commit=False, fname=None, external=False
):
if recursive and fname:
raise RecursiveAddingWhileUsingFilename()
@@ -51,7 +54,9 @@ def add(repo, targets, recursive=False, no_commit=False, fname=None):
)
)
stages = _create_stages(repo, sub_targets, fname, pbar=pbar)
stages = _create_stages(
repo, sub_targets, fname, pbar=pbar, external=external
)
try:
repo.check_modified_graph(stages)
@@ -119,7 +124,7 @@ def _find_all_targets(repo, target, recursive):
return [target]
def _create_stages(repo, targets, fname, pbar=None):
def _create_stages(repo, targets, fname, pbar=None, external=False):
from dvc.stage import Stage, create_stage
stages = []
@@ -131,9 +136,16 @@ def _create_stages(repo, targets, fname, pbar=None):
unit="file",
):
path, wdir, out = resolve_paths(repo, out)
stage = create_stage(Stage, repo, fname or path, wdir=wdir, outs=[out])
stage = create_stage(
Stage,
repo,
fname or path,
wdir=wdir,
outs=[out],
external=external,
)
if stage:
Dvcfile(repo, stage.path).remove_with_prompt(force=True)
Dvcfile(repo, stage.path).remove()
repo._reset()
+10 -4
View File
@@ -18,19 +18,22 @@ def brancher( # noqa: E302
str: the display name for the currently selected tree, it could be:
- a git revision identifier
- empty string it there is no branches to iterate over
- "Working Tree" if there are uncommitted changes in the SCM repo
- "workspace" if there are uncommitted changes in the SCM repo
"""
if not any([revs, all_branches, all_tags, all_commits]):
yield ""
return
saved_tree = self.tree
revs = revs or []
revs = revs.copy() if revs else []
scm = self.scm
self.tree = WorkingTree(self.root_dir)
yield "working tree"
yield "workspace"
if revs and "workspace" in revs:
revs.remove("workspace")
if all_commits:
revs = scm.list_all_commits()
@@ -45,6 +48,9 @@ def brancher( # noqa: E302
if revs:
for sha, names in group_by(scm.resolve_rev, revs).items():
self.tree = scm.get_tree(sha)
yield ", ".join(names)
# ignore revs that don't contain repo root
# (i.e. revs from before a subdir=True repo was init'ed)
if self.tree.exists(self.root_dir):
yield ", ".join(names)
finally:
self.tree = saved_tree
+18 -10
View File
@@ -1,8 +1,11 @@
import logging
import os
from dvc.compat import fspath
from dvc.exceptions import CheckoutError, CheckoutErrorSuggestGit
from dvc.exceptions import (
CheckoutError,
CheckoutErrorSuggestGit,
NoOutputOrStageError,
)
from dvc.progress import Tqdm
from dvc.utils import relpath
@@ -19,11 +22,11 @@ def _get_unused_links(repo):
return repo.state.get_unused_links(used)
def _fspath_dir(path, root):
def _fspath_dir(path):
if not os.path.exists(str(path)):
return str(path)
path = relpath(fspath(path), root)
path = relpath(path)
return os.path.join(path, "") if os.path.isdir(path) else path
@@ -57,9 +60,12 @@ def _checkout(
targets = [None]
unused = _get_unused_links(self)
stats["deleted"] = [_fspath_dir(u, self.root_dir) for u in unused]
stats["deleted"] = [_fspath_dir(u) for u in unused]
self.state.remove_links(unused)
if isinstance(targets, str):
targets = [targets]
pairs = set()
for target in targets:
try:
@@ -68,7 +74,11 @@ def _checkout(
target, with_deps=with_deps, recursive=recursive
)
)
except (StageFileDoesNotExistError, StageFileBadNameError) as exc:
except (
StageFileDoesNotExistError,
StageFileBadNameError,
NoOutputOrStageError,
) as exc:
if not target:
raise
raise CheckoutErrorSuggestGit(target) from exc
@@ -84,10 +94,8 @@ def _checkout(
relink=relink,
filter_info=filter_info,
)
for data in ["failed", "added", "modified"]:
stats[data].extend(
_fspath_dir(path, self.root_dir) for path in result[data]
)
for key, items in result.items():
stats[key].extend(_fspath_dir(path) for path in items)
if stats.get("failed"):
raise CheckoutError(stats["failed"], stats)

Some files were not shown because too many files have changed in this diff Show More