Compare commits

..
108 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 63972c3d12 dvc: bump to 1.0.0a6 2020-05-27 13:59:03 +03:00
Ruslan Kuprieiev 7027c631c5 dvcignore: fix bug in subrepo detection (#3880) 2020-05-27 12:13:53 +03:00
Saugat Pachhai e8f6cd7f5b run: raise error when command is not specified (#3850)
* run: raise error when command is not specified

* tests: adjust tests/func/plots/test_plots.py

* tests: adjust tests/func/plots/test_diff.py

* tests: add command to metrics/plots/params

* tests: fix test_remote.py

* tests: fix test_ls.py

* tests: fix test_repro* and test_run*

* tests: fix script in test_ls, was failing on Win
2020-05-27 12:13:26 +03:00
Peter Rowlands (변기호) 10f2bb93fa ignore: CleanTree should always check ignores, not just for walk() (#3876)
* tests: test that CleanTree works outside of walk()

* ignore: CleanTree should always check ignores, not just for walk()

* CleanTree: handle broken symlinks

* handle case for linking from shared local cache dirs

* Fix windows relpath/different drive issue
2020-05-27 12:07:38 +03:00
Saugat PachhaiandRuslan Kuprieiev 2fecc66cb9 addressing: allow stage name addressing without ":" (#3842)
* addressing: allow stage name addressing without colon

* repo: reorganize imports

* refactor/reorganize collect_granular

* add tests for stage addressing

* fix ds and cc issues

* fix unpack error in test

* tests: stage string representations

* exception: add error message

Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.

* rename NoOutputOrStage NoOutputOrStageError

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-26 03:51:51 +03:00
Ruslan Kuprieiev b47998463f run-cache: don't forget to set wdir (#3873)
Fixes #3848
2020-05-25 20:41:36 +03:00
Peter Rowlands (변기호) d53b740f35 state: always use working trees (#3871)
* tests: add test case for push/pull with --all-commits

* state: always use working tree
2020-05-25 15:47:30 +03:00
Saugat Pachhai 9198fa7459 log: reduce log, make it readable/easier (#3834)
* log: streamline/make it readable

* Remove 'run' from log message

* fix args passing in imports

* tests: fix test

* run: Add quotes around stage name

* run: add quotes for run-cache restore log

* address @dmitry's comments
2020-05-25 05:33:16 +03:00
Ruslan Kuprieiev 6954efc6e6 dvc: bump to 1.0.0a6 2020-05-25 03:25:37 +03:00
Ruslan Kuprieiev d77af8196e optimize local status (#3867)
* remote: remove unpacked dir

We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.

* local: save us a stat call in is_protected

* local: speedup checksum_to_path ~x5.5

* tree: walk: don't use os.path.join

It is ~5.5 times slower than joining by hand.

* dvcignore: optimize matching x10

On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.

To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.

It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
2020-05-25 03:20:11 +03:00
karajan1001 c5920b00ab Auto-completion of the dvc remote rename command (#3868)
fix #3847
2020-05-24 20:41:35 +03:00
Casper da Costa-Luis ce8036e535 snap: release non-stable-semver tags to beta (#3865)
* snap: release non-stable-semver tags to beta

- fixes #3860

* snap: document --beta
2020-05-24 20:40:53 +03:00
Ruslan Kuprieiev b7ad2dc841 test: fix hdfs/azurite/oss tests (#3864)
* travis: fix typo

* tests: azure: fix expected etag format

* test: ssh: increase the number of tries

* travis: don't install docker

* dvc: update pyarrow

* tests: ssh: don't leak connections

* remote: don't fail when cache/remote doesn't have prefix subdir
2020-05-23 23:46:40 +03:00
Ruslan Kuprieiev 5ab948f8b6 tests: unit: azure: add etag test (#3862) 2020-05-23 19:38:38 +03:00
Steffan Sluis f9fbe97509 azure: support external file dependencies (#3852)
* azure: support external dependencies and outputs

- Implement get_file_checksum with get_etag
- Implement a DependencyAzure and OutputAzure classes
- Add AzureOutput to OUTS, OUTS_MAP and CHECKSUMS_SCHEMA
- Add AzureDependency to DEPS and DEPS_MAP
- Add unit tests for AzureOutput and AzureDependency

Fixes #3540.

* Use content_md5 instead of etag

The etag is different for the same file in different locations.

* Use etag and remove external outputs for Azure

Due to inconsistencies with Azure's Content-MD5 property, it is not
stable enough to be used at the moment. Using the etag offers enough
reliability to support external dependencies. See also:
https://discordapp.com/channels/485586884165107732/565699007037571084/713462619469643796
2020-05-23 18:53:52 +03:00
Ruslan Kuprieiev 2fba2e030a dvcfile: serialize params on --no-exec (#3859)
* dvc.yaml: don't generate lockfile on --no-exec

* dvcfile: serialize params on --no-exec

Fixes #3851
2020-05-23 03:19:38 +03:00
Ruslan Kuprieiev 3e951162b8 dvc: bump to 1.0.0a4 2020-05-22 23:00:24 +03:00
Paweł Redzyński d74358eecd remote: local: protect uploaded files (#3858) 2020-05-22 22:59:48 +03:00
Ruslan Kuprieiev 58c9ff0de2 azure: drop obsoleted url format (#3856) 2020-05-22 20:36:04 +03:00
Paweł Redzyński f0e991790e plot: fixup for template help message (#3854) 2020-05-22 20:30:02 +03:00
Peter Rowlands (변기호) b7cd785eba external_repo: use RepoTree for fetch/import/get (#3841)
* erepo: load external repos directly from git tree

* erepo: add fetch_external

* remote/RepoTree: add callback for keeping track of download counts when
saving tree to cache

* repo: use erepo.fetch_external() in repo.fetch()

* erepo: return cache save_infos in fetch_external

* DependencyRepo: use erepo.fetch_external and cache.checkout instead of
pull_to

* erepo: remove pull_to

* DvcTree: support granularity for isdir()

* DvcTree: fix indentation bug

* RepoTree: add copytree() method

* LocalRemote: use working tree for local cache walk_files

* repo: state should be noop for GitTree based erepos

* erepo: add get_external()

* test_get: remove unneeded caplog check

* erepo: add get_rev()

* tests: fix erepo test bug

* ls: use RepoTree

* remote: support trees in _collect_dir

* remote: support trees in get_dir_checksum

* RepoTree: fix dir_cache relpaths in windows

* remote: fix duplicated fetch bug

* utils: don't skip md5 dos2unix check for git blobs

* otherwise md5 checksums will not match when comparing git version to
  cached filesystem version on windows
2020-05-22 17:23:31 +03:00
Ruslan Kuprieiev 21a3efe2c7 dvc: add: set top level md5 to None (#3846)
This way it won't be included into produced dvcfiles (we don't need it
for `dvc add`).
2020-05-22 17:21:43 +03:00
Ruslan Kuprieiev 865d355554 params/metrics: introduce --no-path and --old (#3845)
* metrics: diff: add --no-path

Part of #3690

* params: diff: introduce --no-path

* metrics: diff: introduce --old

Part of #3690
2020-05-22 04:35:42 +03:00
Ruslan Kuprieiev 84712c4acc dvc: don't write default values for cache/persist/metric (#3844) 2020-05-22 02:58:26 +03:00
Ruslan Kuprieiev 0f16e15453 test: add: fix typo 2020-05-22 01:11:45 +03:00
61f30ed5ba plots: point template help info to manual (#3830)
* plots: point template help info to manual

* Update dvc/command/plots.py

Co-authored-by: Jorge Orpinel <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
2020-05-22 00:53:04 +03:00
Ruslan Kuprieiev cde15e3d3e run-cache: don't forget to account for params (#3843)
Fixes #3827
2020-05-22 00:52:21 +03:00
Ruslan Kuprieiev 7672e324d2 output: base: use Git instead of git
https://github.com/iterative/dvc/pull/3808#discussion_r428853696
2020-05-21 22:44:39 +03:00
Ruslan Kuprieiev 130b76c7a5 plots: remove --select CLI option (#3840)
* plots: remove --select from CLI

It needs reconsideration, we can't release it as is right now. E.g. it
expects xpath to start with `$`, but that will get interpreted by your
shell, which is bad.

* completion: update plots
2020-05-21 17:57:27 +03:00
Puneetha PaiandPuneetha Pai 040458a726 (OutputAlreadyTrackedError): Add git suggestions to stop tracking files (#3808)
Fixes: #3678
When file has been already tracked under git, OutputAlreadyTrackedError gets raised.
This commits add git command suggestions to help user to stop tracking file from git,
so in future commits they can use DVC for the file.

Co-authored-by: Puneetha Pai <[email protected]>
2020-05-21 15:17:42 +03:00
Peter Rowlands (변기호) 75e795c491 remote: support saving RepoTree objects directly to cache (#3825)
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache

* RepoTree: support filtering out dvcfiles during walk()

* RepoTree: support recursive isdvc()

* remote: support computing checksums for tree objects

* RepoTree: pull dir contents on walk() if in `fetch` mode

* tests: add func test for DVC dir pull on RepoTree.walk()

* DvcTree: support granular open()

* RepoTree: add get_file_checksum()

* remote: revert checksum changes

* remote: add tree support for cache.save()

* tests: test saving directly to cache from RepoTree

* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree

* RepoTree: state context should be entered by open()/walk() caller

* RepoTree: only pull dir contents when cache has changed

* tree: update tree tests

* fix ds warnings, update comments

* fix windows tests
2020-05-21 06:31:28 +03:00
Saugat Pachhai 4dac2092c9 repro: fix help message for default target (#3831) 2020-05-21 00:17:30 +03:00
Shankho Boron GhoshandRestyled.io 19eaced304 dvc : renamed --outs-only to --dvc-only (#3797)
* Renamed --outs-only to --dvc-only

* Changed outs_only to dvc_only

* replace output_only with dvc_only

* Restyled by black

Co-authored-by: Restyled.io <[email protected]>
2020-05-20 18:24:16 +03:00
Casper da Costa-Luis 184028f176 logger: add trace, count --verbose/--quiet (#3806)
* logger: initial verbosity/quiet counts

Fixes #2332

* actually use TRACE

* logger: properly patch in trace

* FATAL -> CRITICAL

* only change log level if not info

Makes testing easier

* tests: logger: fix and update

* state: logger: debug => trace

* logger: filter duplicate debug messages
2020-05-20 18:23:31 +03:00
Ruslan Kuprieiev 5f63d538d2 dvc: bump to 1.0.0a3 2020-05-20 18:01:24 +03:00
Ruslan Kuprieiev 324ff4be59 remote: is_dir_checksum: handle empty/None checksum (#3836)
Fixed #3434
2020-05-20 18:00:41 +03:00
karajan1001andPeter Rowlands 3f846a1794 Add a new rename command (#3794)
* fixed #3599

1. add new command `dvc remote rename <old> <new>`.
2. add tests for this new command.

* Update dvc/command/remote.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Change about code review

* For deepsource

* Update dvc/command/remote.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2020-05-20 17:53:25 +03:00
Paweł Redzyński b52c925d5f lock: make froze message clickable (#3833) 2020-05-20 16:05:09 +03:00
Ruslan Kuprieiev 4a4f07968c dvc: bump to 1.0.0a2 2020-05-20 15:21:07 +03:00
Ruslan Kuprieiev 09136f2378 plots: add plot markers to DVC files (#3807)
* run: add --plot/--plot-no-cache

Part of #3409

* plot: move show/diff to API

* plots: show marked plots by default

Involves some refactoring to make plots comply with the rest of the code
base.

* dvc.yaml: use top-level metrics and plots

* dvc.yaml: add support for plot template

* plots: add support for templates in dvc.yaml

* dvc.yaml: allow metrics/plots to have persist/cache

* plots: render single page by-default and introduce --show-json

* plot: use "working tree" instead of "workspace"

Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.

* plots: rename -f|--file to -o|--out

Better corresponds to other commands like `import`.

* plots: use RepoTree and brancher instead of api

* stage: get rid of OutputParams

* tests: plots: add simple diff test

* plots: use `--targets` instead of `--datafile`

Makes it comply with `dvc metrics/params` CLI options.

* tests: unit: plots: add --show-json test

* stage: cleanup exception for format error

Based on feedback from the users.

* lockfile: use relpath in corruption error
2020-05-20 04:48:53 +03:00
nik123 aaca1401ea Fixes #3814 (#3828) 2020-05-20 04:14:04 +03:00
Saugat Pachhai 2aaf065ea3 default target: reduce loglevel to debug (#3822) 2020-05-19 16:35:13 +03:00
Saugat Pachhai d4ec871a76 repro: do not log when stage is locked and is repo import (#3823) 2020-05-19 16:23:28 +03:00
Paweł Redzyński f44ddcf64e plot: dont zero y axis (#3813) 2020-05-19 16:20:21 +03:00
Peter Rowlands (변기호) b88b3b81c6 utils: support use of (GitTree) tree file objects in utils (#3819)
* GitTree: add stat() function

* BaseTree: make walk_files a base tree method

* WorkingTree: add stat() method (wrapper for os.stat)

* utils.fs: support non-working tree for get_mtime_and_size

* utils.fs: add copy_fobj_to_file

* utils: support using tree.open in file_md5
2020-05-19 16:20:04 +03:00
Ruslan Kuprieiev 81d04f887a stage: fix commit (#3816) 2020-05-19 03:30:52 +03:00
Peter Rowlands (변기호) afbb3a0add repo: Support streaming and pulling files on RepoTree/DvcTree.open() (#3810)
* DvcTree: support stream or fetch on open()

* tests: test DvcTree dir cache support for walk()

* RepoTree: finish implementation of BaseTree functions

* repo: use RepoTree/DvcTree.open() for repo.open_by_relpath

* repo: only use RepoTree when needed for now

* fix docstring

* RepoTree: only use DvcTree.open for DVC outs
2020-05-18 15:16:11 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Saugat Pachhai 9897f7c470 test: metric/cache/persist keys (#3785) 2020-05-15 13:58:50 +03:00
Saugat Pachhai cf7f125c14 refactor: simplify checkouts and run, get rid of check_can_commit (#3800)
* stage: split checkouts

* stage: simplify can check commit

* split shell executable expansion to separate function

* refactor changed_entries for better message on pipeline file

* fix commit msg entry for stage

* reduce complexity of run_stage

* simplify prompt for commit

* fixes

* refactor commit

* refactor run_stage

* commit refactor
2020-05-15 10:08:32 +05:45
Ruslan Kuprieiev cec4b4e003 dvc: rename plot to plots (#3802)
* dvc: rename plot to plots

Follows `dvc params/metrics` convention and is needed in preparation for new
commands and some refactoring.

Part of #3409

* completion: rename plot -> plots
2020-05-14 22:28:22 +03:00
Ruslan Kuprieiev ff4fcc937b stage: fix --dry-run (#3799)
* tests: unit: move stage tests into a separate dir

* stage: fix --dry-run

Part of #3777
2020-05-14 20:19:34 +05:45
Saugat Pachhai 1676a7a1cd stage: moving things around, refactor (#3793)
* stage: extract unnecessary methods out of Stage class

* stage: change naming of functions

* stage: extract dump equal to utils

* stage: extract computation of md5 to utils

* stage: extract command run related logic in run.py

* stage: reorganize run()

* stage: boolean cosmetic changes

* stage: reorder save() and extract some into func

* stage: refactor status() into smaller functions

* extract resolve_wdir into utils

* merge _changed() and changed()

* cosmetic changes

* extract dump to utils

* extract import updates/sync to imports.py

* remove meta from PipelineStage

* fix imports, params load

* make run_stage a function, fix mocks in tests

* use original cmd

* rollback imports on decorators

* fix ds and cc
2020-05-14 16:43:02 +03:00
Ruslan Kuprieiev 974cf097dd run/repro: rename "build cache" -> "run cache" (#3788)
Part of #3777
2020-05-14 02:27:18 +03:00
Ruslan Kuprieiev 597ab98d0e diff: hanle repo with no commits (#3784)
`git diff` doesn't throw errors, so we could do the same.

Fixes #3398
2020-05-14 02:14:37 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
Peter Rowlands (변기호) 3916d4fa07 tests: mark test_data_cloud::test_pull_git_imports as flaky (#3790) 2020-05-14 00:16:37 +09:00
Paweł Redzyński d81b56978f plot: add doctype at the beggining of html embedding (#3787) 2020-05-13 00:17:05 +03:00
Jorge Orpinel 1e7efffe42 plot: update some output strings (#3759)
* plot: update datafile desc.

* plot: link to cmd ref in output

* plot: update --stdout desc
per https://github.com/iterative/dvc/pull/3577#pullrequestreview-407036181

* plot: a few more output string updates
per https://github.com/iterative/dvc/pull/3759#pullrequestreview-409011291
and https://github.com/iterative/dvc.org/pull/1283
2020-05-12 16:52:06 +03:00
Saugat Pachhai cc16b28a26 diff: autocompletion script for params, and show-md (#3786) 2020-05-12 16:41:09 +03:00
Paweł Redzyński 451c7aac04 tests: erepo_dir: extract remote setup to separate method (#3748) 2020-05-12 02:05:22 +03:00
Ruslan Kuprieiev 7b4466c982 run-cache: support uncached files (#3778)
Currently we only restore `--outs`, but `--outs-no-cache` are also
useful, especially when they are metrics files
(`-M|--metrics-no-cache`). So this PR makes run-cache save uncached
files to our cache and also tries to checkout them back during the
`restore` phase.
2020-05-12 02:02:53 +03:00
Jack WellsandRuslan Kuprieiev 7ca3a1119c Added S3 SSEKMSKeyId config option (#3762)
* Added S3 SSEKMSKeyId config option

* test: unit: s3: add SSEKMSKeyId test

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-11 22:52:55 +03:00
Saugat Pachhai 2c026066db pipeline file: disallow punctuation characters in stage name (#3767)
except hyphen and underscore
2020-05-11 19:11:47 +03:00
nik123 7f17c6aa03 refactor: check if targets is instances of str (#3779) 2020-05-11 19:10:14 +03:00
Ned Letcher 9f015fac1d import: check erepo is None before modifying url - fixes #3780 (#3781)
* import: check erepo is None before modifying url - fixes #3780

* import: add test to cover this local import
2020-05-11 18:29:35 +03:00
Saugat PachhaiandRuslan Kuprieiev 666e203bbb params/metrics: diff: implement tabulated markdown output (--show-md) (#3757)
* params/metrics: diff: implement tabulated markdown output

Via `--show-md` on `diff`s.

* table: show None on missing val

* tests: use `dedent`, add tests for markdown

* diff: add --show-md

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-10 18:39:51 +03:00
Karthikeyan Singaravelan ce946d1160 Import ABC from collections.abc for Python 3 compatibility. (#3772) 2020-05-10 10:49:32 +05:45
karajan1001 76b31ac282 fix #3588 (#3770)
* fix #3588

remove improper edges from output DAG.

* less code modification

* deepsource required

* deepsource required

* refactoring for reducing cognitive complexity.
2020-05-09 21:00:46 +03:00
6093a1fe97 Dvc remote default in list validation (#3715)
* fix #3470

Add validation to restrict default remote repo in list of remote repos.

* fix #3470

1. add two tests
2. add one validation
3. modified remote remove to satithe validation change

* Pass DeepSource

* deepsource required

* Redesigned.

* fix typo

* Revert "deepsource required"

This reverts commit 1ae458c34f49e8d7b35c39c1e6c592b423493a97.

* Revert "Pass DeepSource"

This reverts commit b254d104df65295e0a1156f3979b365be9d38a3d.

Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-09 20:59:17 +03:00
Ruslan Kuprieiev d62a54c7b0 push/pull: properly collect run cache (#3768)
Fixes #1234
2020-05-08 23:34:35 +03:00
nik123 c922da4f3f update: --recursive flag (#3760)
* update: --recursive flag

* refactor: applied isort changes to dvc/repo/update.py

* update: removed default values, check for str

* update: fixed code climate cognitive complexity issues
2020-05-08 23:23:48 +03:00
Ruslan Kuprieiev 2a16b021ad stage: hide unwanted warnings (#3763)
These messages all correpond to expected behaviour, so there is no need
to polute the output with unnecessary warnings.
2020-05-08 15:52:09 +03:00
Ruslan Kuprieiev fc42ca721c dvc: bump to 1.0.0a1 2020-05-07 19:33:30 +03:00
Ruslan Kuprieiev 785dcfeef0 push/pull: naively transfer run cache (#3746)
This is the most simple (aka dumb) implementation for this
functionality that is needed to unblock CICD development.
2020-05-07 16:55:10 +03:00
Paweł Redzyński 9714b14456 plot: fix help messages (#3734) 2020-05-07 13:22:55 +03:00
Ivan Shcheklein 6a406abde8 trying to enable the sponsorship button 2020-05-06 21:09:45 -07:00
Ivan Shcheklein 7ca055e868 fix: bump PyDrive2 to fix Gdrive remote download 2020-05-06 20:11:33 -07:00
Ruslan Kuprieiev 6a64f86178 pre-commit: use isort seeder 2020-05-07 00:25:41 +03:00
Ruslan Kuprieiev 227dbf44d9 isort: fix config 2020-05-07 00:17:38 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Saugat Pachhai e3831066de repro: use dvc.yaml by default (#3747)
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
2020-05-06 18:33:42 +03:00
Paweł Redzyński aa99cc9934 plot: check field exists (#3742) 2020-05-06 18:33:10 +03:00
Casper da Costa-Luis 26e203242f gdrive: download: stream & add progress (#3722)
* wip

* gdrive: add progress

Part of #2865
See https://github.com/iterative/dvc/issues/2865#issuecomment-622984785

* gdrive: move towards next pydrive2 release

- depends on https://github.com/iterative/PyDrive2/pull/30

* update to latest pydrive>=1.4.11

* avoid unneeded API call

* progress: gdrive: ensure proper bar_format
2020-05-06 17:02:16 +03:00
Saugat Pachhai d00bd4dfe5 pull/fetch/push: add stats (#3735)
* pull/fetch/push: add stats

* fix typo: s/logging/logger

* pull: make stats detailed, with summary at the bottom

* tests: mock properly, set return_value

* add a line space
2020-05-05 16:10:50 +03:00
Casper da Costa-Luis 795f001929 Snap version (#3741)
* snap: potential build fix

* snap: fix travis
2020-05-05 12:48:19 +03:00
Ruslan Kuprieiev 93d7b9173b tests: use --single-stage 2020-05-04 20:23:49 +03:00
Ruslan Kuprieiev bc7f177c7a run: use multistage dvcfiles by default (#3740)
Adding hidden `--single-stage` to simplify the test migration.
2020-05-04 19:55:39 +03:00
Ruslan Kuprieiev 6f73a67c60 dvc: bump to 1.0.0a0 2020-05-04 19:46:49 +03:00
Ruslan Kuprieiev 68a0c39ad1 run: remove deprecated -y|--yes (#3737) 2020-05-04 17:51:07 +03:00
Saugat Pachhai c186a8ea8f ui: simplify/shorten error messages (#3730)
* ui: simplify/shorten error messages

* address efiop's review
2020-05-04 17:04:28 +03:00
Paweł Redzyński af8ac006ad plot: completion for zsh (#3733) 2020-05-04 15:01:29 +03:00
Ruslan Kuprieiev b2ea279c9e cache: rename .dvc/cache/run to .dvc/cache/runs (#3732)
To be consistent with how git does it (e.g. objects) as well as with
future cache structure where we plan on adding `dirs`/`files`/`objects`
subdirs in the future.
2020-05-04 13:26:52 +03:00
Saugat Pachhai 2dda540db6 Make comparisons in is_cached independent of order (#3731)
On is_cached, we used to compare two dicts: one of current stage in memory
created by `run` and one that's already written to the file.
As `outs` and `dicts` are lists (which is generate by `dumpd`), the
comparison was dependent on the order of outs and dicts.

So, the `run` before and now, can be in different order and would fail
the comparisons.
2020-05-04 11:40:42 +03:00
Ruslan Kuprieiev 76e6ba24f7 run: add to .gitignore on --no-exec (#3729) 2020-05-04 02:19:56 +03:00
Ruslan Kuprieiev 505b88afae diff: make output consistent with other diff commands (#3727)
* diff: make output consistent with other diff commands

Fixes #3435

* tests: plot: fix flaky test
2020-05-04 00:25:55 +03:00
nik123 415a85c6be status: --recursive flag (#3726)
* status: --recursive flag

* tests: fixed test_status_recursive for Windows platform
2020-05-03 21:12:17 +03:00
Ruslan Kuprieiev 4e9b38970a dvc: move temporary files to .dvc/tmp (#3725)
Fixes #3666
2020-05-03 18:42:12 +03:00
Ruslan Kuprieiev 3e1eb6d1e7 stage: rename .dvc/cache/stages -> .dvc/cache/run (#3724) 2020-05-03 17:19:37 +03:00
Casper da Costa-Luis c873787d73 progress: add postfix info to avoid overwriting desc (#3721)
* progress: persist primary description

Move subsequent updates to a postfix.
Clear postfix on exit.
TODO: align nicely.
Fixes #3681.

* progress: move to posfix[info] for full control

It was bound to happen.

* progress: git: persist description
2020-05-02 18:07:05 +03:00
Paweł RedzyńskiandRuslan Kuprieiev e553511085 Metrics - plotting for multiple revisions initial (#3577)
* init

* rename to plot data insertion basig on dicts update

* revision support

* roll back revision

* plot makedirs for backward compatibility

* log path

* pretty plot link to visualization page

* make target default title

* efiop review

* efiop review

* plot multiple initial

* add some missing metric file tests

* proper id generation

* proper id generation

* add confusion matrix template

* refactor tests

* plot from dvct file

* plot from dvct

* brush up commands

* fix confusion matrix multiple plot

* plot: change confusion matrix data schema

* should be working as intended

* support for src file in dvct files

* minor fixes

* plot: support json templates

* plot: rename confusion template

* plot: polish command behaviour

* fix test for json

* plot: test command

* some minor fixes for tests

* plot: unit test loading

* plot: unit test loading

* plot: handle TODOS

* cleanup

* use mocker

* plot: support tsv

* plot: command refactoring

* plot: fix windows issues with tests

* plot: test: some more windows fixes

* plot: _load_from_revisions complexity fix

* plot: reduce complexity

* plot: complexity reduction

* plot: deepsource suggestions

* plot: move template path evaluation

* fixup

* fixup

* exception on no datafile and no template

* json metric load with OrderedDict

* plot: improve handling non-existing files on revisions

* plot: improve handling non-existing files on revisions

* change default plot path

* some exceptions and fixes

* add yaml metrics support

* fixup

* some more suggestions

* default filename fix

* efiop review requests

* log exception on failur

* move revisions deduction to commands

* json templates

* extract template filling to separate method

* some parsing improvements

* add columns functionality

* extract default data transformation to separate method

* plot: initial support for jsonpath

* plot: rename columns to filters, tests are dict based

* plot: fixups

* plot: refactoring

* repo: plot: convert to package

* plot: data loading refactor, support searching for data

* plot: raise if wrong fields provided

* plot: command description

* plot: default: pass y axis info for default plot

* plot: get rid of fieldnames, expect ordered data

* plot: handle default plot in separate method

* plot: fix default

* plot: command option names fixes

* refactoring

* fixes

* plot: provide option for stdout redirection

* plot: rename show-json to no-html

* plot: add no-csv-header option

* plot: improve error message for wrongly structured metric

* plot: match template name exactly, whit suffix appended only

* plot: dmpetrov and ivan review

* plot: refactor --stdout help message

* plot: move template to repo/plot

* plot: add -x and -y options

* plot: add -x and -y options

* plot: command: order change

* plot: scatter

* plot: rename confusion matrix template, new name generation format

* plot: add title anchor

* plot: review from jorgeorpinel

* plot: rename filter and result options to select and file

* plot: add --title, --x-title, --y-title

* plot: xlab ylab

* Update dvc/repo/plot/template.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plot/template.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* efiop review

* plot: bash completion

* plot: static code analysis fixes

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-01 22:45:03 +03:00
Ruslan KuprieievandSaugat Pachhai 827c994e74 stage: cache: use lockfiles (#3713)
* stage: cache: use lockfiles

* reorganize

* Reorganize, use fill_from_lock to load stage cache

* load params from build cache

* adjust tests

Co-authored-by: Saugat Pachhai <[email protected]>
2020-05-01 15:23:11 +03:00
Saugat PachhaiandRuslan Kuprieiev 22c60dd608 dump: deterministic lockfile dump (#3711)
* dump: lockfile is dumped deterministically

The dump is no longer deterministic/dependent on the pipeline file,
but is sorted based on file names in outs, deps or params.
Also, the params inside each files are also sorted based on name.
However, the objects inside params are not sorted deterministically
as I think it's too much to sort that, and is not easy (considering
the types of objects it might hold, eg: lists, objects, etc).

This will also provide ordered dumps for Python3.5

* fix windows 3.8 test

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-04-30 22:34:06 +03:00
Ruslan Kuprieiev f9088e1c67 dvc: rename pipelines.yaml -> dvc.yaml (#3710) 2020-04-30 18:19:51 +03:00
Ruslan Kuprieiev 71d156ac49 run: try to save deps before running the command (#3709)
Unlike old `_check_missing_deps`, this also verifies that we are able to
save more complex dependencies such as parameters, where we not only
care about the config file, but also about the parameters in it.

Fixes #3707
2020-04-30 17:16:16 +03:00
Ruslan Kuprieiev d998c0a536 serialize: use checksums that are already saved (#3708)
`get_checksum()` recomputes the checksum which might not match the
pre-recorded one. `checksum` is the one that was `save()`ed during run
and it is the one that should be used in the lockfile.
2020-04-30 16:21:14 +03:00
Peter Rowlands (변기호) af12f0d06c remote: adjust traverse threshold multiplier (#3705)
* Fixes #3704
2020-04-30 12:12:11 +09:00
Ruslan Kuprieiev 18e8f07572 dvc: introduce local build cache (#3603)
This patch introduces `.dvc/cache/stages` that is used to store previous
runs and their results, which could then be reused later when we stumble
upon the same command with the same deps and outs.

Format of build cache entries is single-line json, which is readable by
humans and might also be used for lock files discussed in #1871.

Related to #1871
Local part of #1234
2020-04-30 00:22:32 +03:00
263 changed files with 8285 additions and 3311 deletions
-7
View File
@@ -1,7 +0,0 @@
[flake8]
ignore =
E203, # Whitespace before ':'
E266, # Too many leading '#' for block comment
W503, # Line break occurred before a binary operator
max-line-length = 79
select = B,C,E,F,W,T4,B9
+3
View File
@@ -0,0 +1,3 @@
# These are supported funding model platforms
patreon: DVCorg
+9
View File
@@ -4,6 +4,15 @@ repos:
language_version: python3
repo: https://github.com/ambv/black
rev: 19.10b0
- repo: https://github.com/asottile/seed-isort-config
rev: v2.1.1
hooks:
- id: seed-isort-config
- hooks:
- id: isort
language_version: python3
repo: https://github.com/timothycrosley/isort
rev: 4.3.21
- hooks:
- id: flake8
language_version: python3
+9
View File
@@ -9,3 +9,12 @@ restylers:
- "**/*.py"
interpreters:
- python
- name: isort
image: restyled/restyler-isort:v4.3.21
command:
- isort
arguments: []
include:
- "**/*.py"
interpreters:
- python
+1 -3
View File
@@ -46,9 +46,6 @@ jobs:
env:
- PYTHON_VERSION=3.8.0
- PATH=/c/Python38:/c/Python38/Scripts:$PATH
- name: "3.5 on Linux"
language: python
python: 3.5
- name: "3.6 on Linux"
language: python
python: 3.6
@@ -106,6 +103,7 @@ jobs:
- sudo /snap/bin/lxd.migrate -yes
- sudo /snap/bin/lxd waitready
- sudo /snap/bin/lxd init --auto
- git fetch --tags
script:
- ./scripts/build_snap.sh
after_failure:
+2 -1
View File
@@ -101,7 +101,8 @@ Snap (Snapcraft/Linux)
snap install dvc --classic
This corresponds to the latest tagged release.
Add ``--edge`` for the latest ``master`` version.
Add ``--beta`` for the latest tagged release candidate,
or ``--edge`` for the latest ``master`` version.
Choco (Chocolatey/Windows)
--------------------------
-1
View File
@@ -6,5 +6,4 @@ Make your data science projects reproducible and shareable.
import dvc.logger
from dvc.version import __version__ # noqa: F401
dvc.logger.setup()
+4 -7
View File
@@ -2,12 +2,12 @@ import json
import logging
import os
import platform
import requests
import sys
import tempfile
import uuid
import distro
import requests
from dvc import __version__
from dvc.config import Config, to_bool
@@ -20,7 +20,6 @@ from dvc.scm.base import SCMError
from dvc.utils import env2bool, is_binary
from dvc.utils.fs import makedirs
logger = logging.getLogger(__name__)
@@ -73,7 +72,7 @@ def send(path):
url = "https://analytics.dvc.org"
headers = {"content-type": "application/json"}
with open(path, "r") as fobj:
with open(path) as fobj:
report = json.load(fobj)
report.update(_runtime_info())
@@ -159,7 +158,7 @@ def _find_or_create_user_id():
try:
with Lock(lockfile):
try:
with open(fname, "r") as fobj:
with open(fname) as fobj:
user_id = json.load(fobj)["user_id"]
except (FileNotFoundError, ValueError, KeyError):
@@ -171,6 +170,4 @@ def _find_or_create_user_id():
return user_id
except LockError:
logger.debug(
"Failed to acquire '{lockfile}'".format(lockfile=lockfile)
)
logger.debug(f"Failed to acquire '{lockfile}'")
+4 -3
View File
@@ -1,16 +1,17 @@
import os
from contextlib import contextmanager, _GeneratorContextManager as GCM
from contextlib import _GeneratorContextManager as GCM
from contextlib import contextmanager
from dvc.repo import Repo
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.external_repo import external_repo
from dvc.repo import Repo
class UrlNotDvcRepoError(DvcException):
"""Thrown if the given URL is not a DVC repository."""
def __init__(self, url):
super().__init__("'{}' is not a DVC repository.".format(url))
super().__init__(f"'{url}' is not a DVC repository.")
def get_url(path, repo=None, rev=None, remote=None):
+2 -2
View File
@@ -40,7 +40,7 @@ def _make_remote_property(name):
return cached_property(getter)
class Cache(object):
class Cache:
"""Class that manages cache locations of a DVC repo.
Args:
@@ -101,7 +101,7 @@ class NamedCacheItem:
self.children[checksum].update(child_item)
class NamedCache(object):
class NamedCache:
def __init__(self):
self._items = defaultdict(lambda: defaultdict(NamedCacheItem))
self.external = defaultdict(set)
+7 -10
View File
@@ -16,16 +16,18 @@ from .command import (
gc,
get,
get_url,
git_hook,
imp,
imp_url,
init,
install,
ls,
lock,
ls,
metrics,
params,
move,
params,
pipeline,
plots,
remote,
remove,
repro,
@@ -34,12 +36,10 @@ from .command import (
unprotect,
update,
version,
git_hook,
)
from .command.base import fix_subparsers
from .exceptions import DvcParserError
logger = logging.getLogger(__name__)
COMMANDS = [
@@ -74,6 +74,7 @@ COMMANDS = [
version,
update,
git_hook,
plots,
]
@@ -115,14 +116,10 @@ def get_parent_parser():
log_level_group = parent_parser.add_mutually_exclusive_group()
log_level_group.add_argument(
"-q", "--quiet", action="store_true", default=False, help="Be quiet."
"-q", "--quiet", action="count", default=0, help="Be quiet."
)
log_level_group.add_argument(
"-v",
"--verbose",
action="store_true",
default=False,
help="Be verbose.",
"-v", "--verbose", action="count", default=0, help="Be verbose."
)
return parent_parser
+1 -1
View File
@@ -1,7 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link, CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException, RecursiveAddingWhileUsingFilename
logger = logging.getLogger(__name__)
+6 -8
View File
@@ -1,6 +1,5 @@
import logging
logger = logging.getLogger(__name__)
@@ -27,7 +26,7 @@ def append_doc_link(help_message, path):
)
class CmdBase(object):
class CmdBase:
def __init__(self, args):
from dvc.repo import Repo
from dvc.updater import Updater
@@ -36,17 +35,16 @@ class CmdBase(object):
self.config = self.repo.config
self.args = args
hardlink_lock = self.config["core"].get("hardlink_lock", False)
updater = Updater(self.repo.dvc_dir, hardlink_lock=hardlink_lock)
updater = Updater(self.repo.tmp_dir, hardlink_lock=hardlink_lock)
updater.check()
@property
def default_targets(self):
"""Default targets for `dvc repro` and `dvc pipeline`."""
from dvc.dvcfile import DVC_FILE
"""Default targets for `dvc repro`."""
from dvc.dvcfile import PIPELINE_FILE
msg = "assuming default target '{}'.".format(DVC_FILE)
logger.warning(msg)
return [DVC_FILE]
logger.debug(f"assuming default target '{PIPELINE_FILE}'.")
return [PIPELINE_FILE]
# Abstract methods that have to be implemented by any inheritance class
def run(self):
+8 -27
View File
@@ -1,39 +1,16 @@
import argparse
import logging
import operator
import colorama
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import CheckoutError
from dvc.utils.humanize import get_summary
logger = logging.getLogger(__name__)
def _human_join(words):
words = list(words)
if not words:
return ""
return (
"{before} and {after}".format(
before=", ".join(words[:-1]), after=words[-1],
)
if len(words) > 1
else words[0]
)
def log_summary(stats):
states = ("added", "deleted", "modified")
summary = (
"{} {}".format(len(stats[state]), state)
for state in states
if stats.get(state)
)
logger.info(_human_join(summary) or "No changes.")
def log_changes(stats):
colors = [
("modified", colorama.Fore.YELLOW,),
@@ -75,7 +52,11 @@ class CmdCheckout(CmdBase):
stats = exc.stats
if self.args.summary:
log_summary(stats)
default_message = "No changes."
msg = get_summary(
sorted(stats.items(), key=operator.itemgetter(0))
)
logger.info(msg or default_message)
else:
log_changes(stats)
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+1 -1
View File
@@ -1,7 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link, CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.config import Config, ConfigError
logger = logging.getLogger(__name__)
+3 -3
View File
@@ -1,5 +1,4 @@
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import fix_subparsers
from dvc.command.base import CmdBaseNoRepo, fix_subparsers
class CmdDaemonBase(CmdBaseNoRepo):
@@ -15,9 +14,10 @@ class CmdDaemonUpdater(CmdDaemonBase):
root_dir = Repo.find_root()
dvc_dir = os.path.join(root_dir, Repo.DVC_DIR)
tmp_dir = os.path.join(dvc_dir, "tmp")
config = Config(dvc_dir, validate=False)
hardlink_lock = config.get("core", {}).get("hardlink_lock", False)
updater = Updater(dvc_dir, hardlink_lock=hardlink_lock)
updater = Updater(tmp_dir, hardlink_lock=hardlink_lock)
updater.fetch(detach=False)
return 0
+48 -18
View File
@@ -1,23 +1,25 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.checkout import log_summary
from dvc.exceptions import DvcException, CheckoutError
from dvc.command.base import CmdBase, append_doc_link
from dvc.command.checkout import log_changes
from dvc.exceptions import CheckoutError, DvcException
from dvc.utils.humanize import get_summary
logger = logging.getLogger(__name__)
class CmdDataBase(CmdBase):
@classmethod
def check_up_to_date(cls, processed_files_count):
if processed_files_count == 0:
logger.info("Everything is up to date.")
def log_summary(self, stats):
default_msg = "Everything is up to date."
logger.info(get_summary(stats.items()) or default_msg)
class CmdDataPull(CmdDataBase):
def log_summary(self, stats):
log_changes(stats)
super().log_summary(stats)
def run(self):
try:
stats = self.repo.pull(
@@ -30,14 +32,14 @@ class CmdDataPull(CmdDataBase):
with_deps=self.args.with_deps,
force=self.args.force,
recursive=self.args.recursive,
run_cache=self.args.run_cache,
)
log_summary(stats)
self.log_summary(stats)
except (CheckoutError, DvcException) as exc:
log_summary(getattr(exc, "stats", {}))
self.log_summary(getattr(exc, "stats", {}))
logger.exception("failed to pull data from the cloud")
return 1
self.check_up_to_date(stats["downloaded"])
return 0
@@ -53,11 +55,12 @@ class CmdDataPush(CmdDataBase):
all_commits=self.args.all_commits,
with_deps=self.args.with_deps,
recursive=self.args.recursive,
run_cache=self.args.run_cache,
)
self.log_summary({"pushed": processed_files_count})
except DvcException:
logger.exception("failed to push data to the cloud")
return 1
self.check_up_to_date(processed_files_count)
return 0
@@ -73,11 +76,12 @@ class CmdDataFetch(CmdDataBase):
all_commits=self.args.all_commits,
with_deps=self.args.with_deps,
recursive=self.args.recursive,
run_cache=self.args.run_cache,
)
self.log_summary({"fetched": processed_files_count})
except DvcException:
logger.exception("failed to fetch data from the cloud")
return 1
self.check_up_to_date(processed_files_count)
return 0
@@ -85,24 +89,24 @@ def shared_parent_parser():
from dvc.cli import get_parent_parser
# Parent parser used in pull/push/status
shared_parent_parser = argparse.ArgumentParser(
parent_parser = argparse.ArgumentParser(
add_help=False, parents=[get_parent_parser()]
)
shared_parent_parser.add_argument(
parent_parser.add_argument(
"-j",
"--jobs",
type=int,
help="Number of jobs to run simultaneously.",
metavar="<number>",
)
shared_parent_parser.add_argument(
parent_parser.add_argument(
"targets",
nargs="*",
help="Limit command scope to these DVC-files. "
"Using -R, directories to search DVC-files in can also be given.",
)
return shared_parent_parser
return parent_parser
def add_parser(subparsers, _parent_parser):
@@ -162,6 +166,12 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Pull cache for subdirectories of the specified directory.",
)
pull_parser.add_argument(
"--run-cache",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
pull_parser.set_defaults(func=CmdDataPull)
# Push
@@ -211,6 +221,12 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Push cache for subdirectories of specified directory.",
)
push_parser.add_argument(
"--run-cache",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
push_parser.set_defaults(func=CmdDataPush)
# Fetch
@@ -266,6 +282,12 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Fetch cache for subdirectories of specified directory.",
)
fetch_parser.add_argument(
"--run-cache",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
fetch_parser.set_defaults(func=CmdDataFetch)
# Status
@@ -334,4 +356,12 @@ def add_parser(subparsers, _parent_parser):
default=False,
help="Show status for all dependencies of the specified target.",
)
status_parser.add_argument(
"-R",
"--recursive",
action="store_true",
default=False,
help="Show status of all stages in the specified directory.",
)
status_parser.set_defaults(func=CmdDataStatus)
+1 -3
View File
@@ -2,11 +2,9 @@ import argparse
import logging
import dvc.prompt as prompt
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+26 -9
View File
@@ -8,10 +8,24 @@ import colorama
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
def _show_md(diff):
from dvc.utils.diff import table
rows = []
for status in ["added", "deleted", "modified"]:
entries = diff.get(status, [])
if not entries:
continue
paths = sorted([entry["path"] for entry in entries])
for path in paths:
rows.append([status, path])
return table(["Status", "Path"], rows, True)
class CmdDiff(CmdBase):
@staticmethod
def _format(diff):
@@ -97,20 +111,17 @@ class CmdDiff(CmdBase):
try:
diff = self.repo.diff(self.args.a_rev, self.args.b_rev)
if not any(diff.values()):
return 0
if not self.args.show_hash:
for _, entries in diff.items():
for entry in entries:
del entry["hash"]
if self.args.show_json:
res = json.dumps(diff)
else:
res = self._format(diff)
logger.info(res)
logger.info(json.dumps(diff))
elif self.args.show_md:
logger.info(_show_md(diff))
elif diff:
logger.info(self._format(diff))
except DvcException:
logger.exception("failed to get diff")
@@ -153,4 +164,10 @@ def add_parser(subparsers, parent_parser):
action="store_true",
default=False,
)
diff_parser.add_argument(
"--show-md",
help="Show tabulated output in the Markdown format (GFM).",
action="store_true",
default=False,
)
diff_parser.set_defaults(func=CmdDiff)
+1 -3
View File
@@ -3,9 +3,7 @@ import logging
import os
import dvc.prompt as prompt
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
logger = logging.getLogger(__name__)
+1 -2
View File
@@ -1,10 +1,9 @@
import argparse
import logging
from .base import append_doc_link
from .base import CmdBaseNoRepo
from dvc.exceptions import DvcException
from .base import CmdBaseNoRepo, append_doc_link
logger = logging.getLogger(__name__)
+2 -3
View File
@@ -1,10 +1,9 @@
import argparse
import logging
from .base import append_doc_link
from .base import CmdBaseNoRepo
from dvc.exceptions import DvcException
from .base import CmdBaseNoRepo, append_doc_link
logger = logging.getLogger(__name__)
@@ -17,7 +16,7 @@ class CmdGetUrl(CmdBaseNoRepo):
Repo.get_url(self.args.url, out=self.args.out)
return 0
except DvcException:
logger.exception("failed to get '{}'".format(self.args.url))
logger.exception(f"failed to get '{self.args.url}'")
return 1
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+1 -3
View File
@@ -1,9 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
logger = logging.getLogger(__name__)
+1 -3
View File
@@ -1,9 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
logger = logging.getLogger(__name__)
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+4 -6
View File
@@ -2,12 +2,10 @@ import argparse
import logging
import sys
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.command.ls.ls_colors import LsColors
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -33,14 +31,14 @@ class CmdList(CmdBaseNoRepo):
self.args.path,
rev=self.args.rev,
recursive=self.args.recursive,
outs_only=self.args.outs_only,
dvc_only=self.args.dvc_only,
)
if entries:
entries = _prettify(entries, sys.stdout.isatty())
logger.info("\n".join(entries))
return 0
except DvcException:
logger.exception("failed to list '{}'".format(self.args.url))
logger.exception(f"failed to list '{self.args.url}'")
return 1
@@ -64,7 +62,7 @@ def add_parser(subparsers, parent_parser):
help="Recursively list files.",
)
list_parser.add_argument(
"--outs-only", action="store_true", help="Show only DVC outputs."
"--dvc-only", action="store_true", help="Show only DVC outputs."
)
list_parser.add_argument(
"--rev",
+2 -2
View File
@@ -1,7 +1,7 @@
import os
class LsColors(object):
class LsColors:
default = "rs=0:di=01;34:ex=01;32"
def __init__(self, lscolors=None):
@@ -45,4 +45,4 @@ class LsColors(object):
if not val:
return text
rs = self._codes.get("rs", 0)
return "\033[{}m{}\033[{}m".format(val, text, rs)
return f"\033[{val}m{text}\033[{rs}m"
+45 -22
View File
@@ -1,12 +1,8 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import fix_subparsers
from dvc.exceptions import BadMetricError
from dvc.exceptions import DvcException
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import BadMetricError, DvcException
logger = logging.getLogger(__name__)
@@ -23,16 +19,16 @@ def show_metrics(
for branch, val in metrics.items():
if all_branches or all_tags or all_commits:
logger.info("{branch}:".format(branch=branch))
logger.info(f"{branch}:")
for fname, metric in val.items():
if not isinstance(metric, dict):
logger.info("\t{}: {}".format(fname, str(metric)))
continue
logger.info("\t{}:".format(fname))
logger.info(f"\t{fname}:")
for key, value in flatten(format_dict(metric), ".").items():
logger.info("\t\t{}: {}".format(key, value))
logger.info(f"\t\t{key}: {value}")
if missing:
raise BadMetricError(missing)
@@ -72,7 +68,7 @@ class CmdMetricsAdd(CmdBase):
try:
self.repo.metrics.add(self.args.path)
except DvcException:
msg = "failed to add metric file '{}'".format(self.args.path)
msg = f"failed to add metric file '{self.args.path}'"
logger.exception(msg)
return 1
@@ -84,14 +80,14 @@ class CmdMetricsRemove(CmdBase):
try:
self.repo.metrics.remove(self.args.path)
except DvcException:
msg = "failed to remove metric file '{}'".format(self.args.path)
msg = f"failed to remove metric file '{self.args.path}'"
logger.exception(msg)
return 1
return 0
def _show_diff(diff):
def _show_diff(diff, markdown=False, no_path=False, old=False):
from collections import OrderedDict
from dvc.utils.diff import table
@@ -100,16 +96,23 @@ def _show_diff(diff):
for fname, mdiff in diff.items():
sorted_mdiff = OrderedDict(sorted(mdiff.items()))
for metric, change in sorted_mdiff.items():
rows.append(
[
fname,
metric,
change["new"],
change.get("diff", "diff not supported"),
]
)
row = [] if no_path else [fname]
row.append(metric)
if old:
row.append(change.get("old"))
row.append(change["new"])
row.append(change.get("diff", "diff not supported"))
rows.append(row)
return table(["Path", "Metric", "Value", "Change"], rows)
header = [] if no_path else ["Path"]
header.append("Metric")
if old:
header.extend(["Old", "New"])
else:
header.append("Value")
header.append("Change")
return table(header, rows, markdown)
class CmdMetricsDiff(CmdBase):
@@ -128,7 +131,9 @@ class CmdMetricsDiff(CmdBase):
logger.info(json.dumps(diff))
else:
table = _show_diff(diff)
table = _show_diff(
diff, self.args.show_md, self.args.no_path, self.args.old
)
if table:
logger.info(table)
@@ -267,6 +272,24 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Show output in JSON format.",
)
metrics_diff_parser.add_argument(
"--show-md",
action="store_true",
default=False,
help="Show tabulated output in the Markdown format (GFM).",
)
metrics_diff_parser.add_argument(
"--no-path",
action="store_true",
default=False,
help="Don't show metric path.",
)
metrics_diff_parser.add_argument(
"--old",
action="store_true",
default=False,
help="Show old metric value.",
)
metrics_diff_parser.set_defaults(func=CmdMetricsDiff)
METRICS_REMOVE_HELP = "Remove metric mark on a DVC-tracked file."
+1 -3
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
+26 -9
View File
@@ -1,27 +1,32 @@
import argparse
import logging
from collections import OrderedDict
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import fix_subparsers
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
def _show_diff(diff):
def _show_diff(diff, markdown=False, no_path=False):
from dvc.utils.diff import table
rows = []
for fname, pdiff in diff.items():
sorted_pdiff = OrderedDict(sorted(pdiff.items()))
for param, change in sorted_pdiff.items():
rows.append([fname, param, change["old"], change["new"]])
row = [] if no_path else [fname]
row.append(param)
row.append(change["old"])
row.append(change["new"])
rows.append(row)
return table(["Path", "Param", "Old", "New"], rows)
header = [] if no_path else ["Path"]
header.append("Param")
header.append("Old")
header.append("New")
return table(header, rows, markdown)
class CmdParamsDiff(CmdBase):
@@ -38,7 +43,7 @@ class CmdParamsDiff(CmdBase):
logger.info(json.dumps(diff))
else:
table = _show_diff(diff)
table = _show_diff(diff, self.args.show_md, self.args.no_path)
if table:
logger.info(table)
@@ -98,4 +103,16 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Show output in JSON format.",
)
params_diff_parser.add_argument(
"--show-md",
action="store_true",
default=False,
help="Show tabulated output in the Markdown format (GFM).",
)
params_diff_parser.add_argument(
"--no-path",
action="store_true",
default=False,
help="Don't show params path.",
)
params_diff_parser.set_defaults(func=CmdParamsDiff)
+36 -16
View File
@@ -4,18 +4,16 @@ import logging
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdPipelineShow(CmdBase):
def _show(self, target, commands, outs, locked):
import networkx
from dvc import dvcfile
from dvc.utils import parse_target
path, name = parse_target(target)
stage = dvcfile.Dvcfile(self.repo, path).stages[name]
stage = self.repo.get_stage(path, name)
G = self.repo.graph
stages = networkx.dfs_postorder_nodes(G, stage)
if locked:
@@ -32,14 +30,38 @@ class CmdPipelineShow(CmdBase):
else:
logger.info(stage.addressing)
@staticmethod
def _build_output_graph(G, target_stage):
import networkx
from itertools import product
nodes = {str(out) for out in target_stage.outs}
edges = []
for from_stage, to_stage in networkx.edge_dfs(G, target_stage):
from_stage_deps = {dep.path_info.parts for dep in from_stage.deps}
to_outs = {
to_out
for to_out in to_stage.outs
if to_out.path_info.parts in from_stage_deps
}
from_outs = {
from_out
for from_out in from_stage.outs
if str(from_out) in nodes
}
nodes |= {str(to_out) for to_out in to_outs}
for from_out, to_out in product(from_outs, to_outs):
edges.append((str(from_out), str(to_out)))
return nodes, edges
def _build_graph(self, target, commands=False, outs=False):
import networkx
from dvc import dvcfile
from dvc.repo.graph import get_pipeline
from dvc.utils import parse_target
path, name = parse_target(target)
target_stage = dvcfile.Dvcfile(self.repo, path).stages[name]
target_stage = self.repo.get_stage(path, name)
G = get_pipeline(self.repo.pipelines, target_stage)
nodes = set()
@@ -48,10 +70,7 @@ class CmdPipelineShow(CmdBase):
if stage.cmd is None:
continue
nodes.add(stage.cmd)
elif outs:
for out in stage.outs:
nodes.add(str(out))
else:
elif not outs:
nodes.add(stage.addressing)
edges = []
@@ -60,13 +79,12 @@ class CmdPipelineShow(CmdBase):
if to_stage.cmd is None:
continue
edges.append((from_stage.cmd, to_stage.cmd))
elif outs:
for from_out in from_stage.outs:
for to_out in to_stage.outs:
edges.append((str(from_out), str(to_out)))
else:
elif not outs:
edges.append((from_stage.addressing, to_stage.addressing))
if outs:
nodes, edges = self._build_output_graph(G, target_stage)
return list(nodes), edges, networkx.is_tree(G)
def _show_ascii(self, target, commands, outs):
@@ -119,8 +137,10 @@ class CmdPipelineShow(CmdBase):
logger.info(dot_file.getvalue())
def run(self):
from dvc.dvcfile import DVC_FILE
if not self.args.targets:
self.args.targets = self.default_targets
self.args.targets = [DVC_FILE]
for target in self.args.targets:
try:
@@ -142,7 +162,7 @@ class CmdPipelineShow(CmdBase):
self.args.locked,
)
except DvcException:
msg = "failed to show pipeline for '{}'".format(target)
msg = f"failed to show pipeline for '{target}'"
logger.exception(msg)
return 1
return 0
+219
View File
@@ -0,0 +1,219 @@
import argparse
import logging
import os
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
from dvc.utils import format_link
logger = logging.getLogger(__name__)
PAGE_HTML = """<!DOCTYPE html>
<html>
<head>
<title>DVC Plot</title>
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
</head>
<body>
{divs}
</body>
</html>"""
DIV_HTML = """<div id = "{id}"></div>
<script type = "text/javascript">
var spec = {vega_json};
vegaEmbed('#{id}', spec);
</script>"""
class CmdPlots(CmdBase):
def _func(self, *args, **kwargs):
raise NotImplementedError
def run(self):
try:
plots = self._func(
targets=self.args.targets,
template=self.args.template,
x_field=self.args.x,
y_field=self.args.y,
csv_header=not self.args.no_csv_header,
title=self.args.title,
x_title=self.args.xlab,
y_title=self.args.ylab,
)
if self.args.show_json:
import json
logger.info(json.dumps(plots))
return 0
divs = [
DIV_HTML.format(id=f"plot{i}", vega_json=plot)
for i, plot in enumerate(plots.values())
]
html = PAGE_HTML.format(divs="\n".join(divs))
path = self.args.out or "plots.html"
with open(path, "w") as fobj:
fobj.write(html)
logger.info(
"file://{}".format(os.path.join(self.repo.root_dir, path))
)
except DvcException:
logger.exception("")
return 1
return 0
class CmdPlotsShow(CmdPlots):
def _func(self, *args, **kwargs):
return self.repo.plots.show(*args, **kwargs)
class CmdPlotsDiff(CmdPlots):
def _func(self, *args, **kwargs):
return self.repo.plots.diff(*args, revs=self.args.revisions, **kwargs)
def add_parser(subparsers, parent_parser):
PLOTS_HELP = (
"Generating plots for metrics stored in structured files "
"(JSON, CSV, TSV)."
)
plots_parser = subparsers.add_parser(
"plots",
parents=[parent_parser],
description=append_doc_link(PLOTS_HELP, "plots"),
help=PLOTS_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_subparsers = plots_parser.add_subparsers(
dest="cmd",
help="Use `dvc plots CMD --help` to display command-specific help.",
)
fix_subparsers(plots_subparsers)
SHOW_HELP = "Generate a plots image file from a metrics file."
plots_show_parser = plots_subparsers.add_parser(
"show",
parents=[parent_parser],
description=append_doc_link(SHOW_HELP, "plots/show"),
help=SHOW_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_show_parser.add_argument(
"-t",
"--template",
nargs="?",
default=None,
help=(
"Special JSON or HTML schema file to inject with the data. "
"See {}".format(
format_link("https://man.dvc.org/plots#plot-templates")
)
),
)
plots_show_parser.add_argument(
"-o", "--out", default=None, help="Destination path to save plots to.",
)
plots_show_parser.add_argument(
"-x", default=None, help="Field name for x axis."
)
plots_show_parser.add_argument(
"-y", default=None, help="Field name for y axis."
)
plots_show_parser.add_argument(
"--no-csv-header",
action="store_true",
default=False,
help="Required when CSV or TSV datafile does not have a header.",
)
plots_show_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show output in JSON format.",
)
plots_show_parser.add_argument("--title", default=None, help="Plot title.")
plots_show_parser.add_argument(
"--xlab", default=None, help="X axis title."
)
plots_show_parser.add_argument(
"--ylab", default=None, help="Y axis title."
)
plots_show_parser.add_argument(
"targets",
nargs="*",
help="Metrics files to visualize. Shows all plots by default.",
)
plots_show_parser.set_defaults(func=CmdPlotsShow)
PLOTS_DIFF_HELP = (
"Plot differences in metrics between commits in the DVC "
"repository, or between the last commit and the workspace."
)
plots_diff_parser = plots_subparsers.add_parser(
"diff",
parents=[parent_parser],
description=append_doc_link(PLOTS_DIFF_HELP, "plots/diff"),
help=PLOTS_DIFF_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_diff_parser.add_argument(
"-t",
"--template",
nargs="?",
default=None,
help=(
"Special JSON or HTML schema file to inject with the data. "
"See {}".format(
format_link("https://man.dvc.org/plots#plot-templates")
)
),
)
plots_diff_parser.add_argument(
"--targets",
nargs="*",
help="Metrics file to visualize. Shows all plots by default.",
)
plots_diff_parser.add_argument(
"-o", "--out", default=None, help="Destination path to save plots to.",
)
plots_diff_parser.add_argument(
"-x", default=None, help="Field name for x axis."
)
plots_diff_parser.add_argument(
"-y", default=None, help="Field name for y axis."
)
plots_diff_parser.add_argument(
"--no-csv-header",
action="store_true",
default=False,
help="Provided CSV ot TSV datafile does not have a header.",
)
plots_diff_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show output in JSON format.",
)
plots_diff_parser.add_argument("--title", default=None, help="Plot title.")
plots_diff_parser.add_argument(
"--xlab", default=None, help="X axis title."
)
plots_diff_parser.add_argument(
"--ylab", default=None, help="Y axis title."
)
plots_diff_parser.add_argument(
"revisions", nargs="*", default=None, help="Git commits to plot from",
)
plots_diff_parser.set_defaults(func=CmdPlotsDiff)
+55 -8
View File
@@ -1,9 +1,9 @@
import argparse
import logging
from dvc.config import ConfigError
from dvc.command.base import append_doc_link, fix_subparsers
from dvc.command.config import CmdConfig
from dvc.config import ConfigError
from dvc.utils import format_link
logger = logging.getLogger(__name__)
@@ -18,17 +18,13 @@ class CmdRemote(CmdConfig):
def _check_exists(self, conf):
if self.args.name not in conf["remote"]:
raise ConfigError(
"remote '{}' doesn't exists.".format(self.args.name)
)
raise ConfigError(f"remote '{self.args.name}' doesn't exists.")
class CmdRemoteAdd(CmdRemote):
def run(self):
if self.args.default:
logger.info(
"Setting '{}' as a default remote.".format(self.args.name)
)
logger.info(f"Setting '{self.args.name}' as a default remote.")
with self.config.edit(self.args.level) as conf:
if self.args.name in conf["remote"] and not self.args.force:
@@ -85,7 +81,18 @@ class CmdRemoteDefault(CmdRemote):
if self.args.unset:
conf["core"].pop("remote", None)
else:
conf["core"]["remote"] = self.args.name
merged_conf = self.config.load_config_to_level(
self.args.level
)
if (
self.args.name in conf["remote"]
or self.args.name in merged_conf["remote"]
):
conf["core"]["remote"] = self.args.name
else:
raise ConfigError(
"default remote must be present in remote list."
)
return 0
@@ -97,6 +104,35 @@ class CmdRemoteList(CmdRemote):
return 0
class CmdRemoteRename(CmdRemote):
def _rename_default(self, conf):
if conf["core"].get("remote") == self.args.name:
conf["core"]["remote"] = self.args.new
def run(self):
all_config = self.config.load_config_to_level(None)
if self.args.new in all_config.get("remote", {}):
raise ConfigError(
"Rename failed. Remote name '{}' already exists.".format(
{self.args.new}
)
)
with self.config.edit(self.args.level) as conf:
self._check_exists(conf)
conf["remote"][self.args.new] = conf["remote"][self.args.name]
del conf["remote"][self.args.name]
self._rename_default(conf)
for level in reversed(self.config.LEVELS):
if level == self.args.level:
break
with self.config.edit(level) as level_conf:
self._rename_default(level_conf)
return 0
def add_parser(subparsers, parent_parser):
from dvc.command.config import parent_config_parser
@@ -213,3 +249,14 @@ def add_parser(subparsers, parent_parser):
"name", help="Name of the remote to remove."
)
remote_remove_parser.set_defaults(func=CmdRemoteRemove)
REMOTE_RENAME_HELP = "Rename a DVC remote"
remote_rename_parser = remote_subparsers.add_parser(
"rename",
parents=[parent_config_parser, parent_parser],
description=append_doc_link(REMOTE_RENAME_HELP, "remote/rename"),
help=REMOTE_RENAME_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
remote_rename_parser.add_argument("name", help="Remote to be renamed")
remote_rename_parser.add_argument("new", help="New name of the remote")
remote_rename_parser.set_defaults(func=CmdRemoteRename)
+5 -7
View File
@@ -2,16 +2,14 @@ import argparse
import logging
import dvc.prompt as prompt
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdRemove(CmdBase):
def _is_outs_only(self, target):
def _is_dvc_only(self, target):
if not self.args.purge:
return True
@@ -33,10 +31,10 @@ class CmdRemove(CmdBase):
def run(self):
for target in self.args.targets:
try:
outs_only = self._is_outs_only(target)
self.repo.remove(target, outs_only=outs_only)
dvc_only = self._is_dvc_only(target)
self.repo.remove(target, dvc_only=dvc_only)
except DvcException:
logger.exception("failed to remove '{}'".format(target))
logger.exception(f"failed to remove '{target}'")
return 1
return 0
+15 -6
View File
@@ -2,13 +2,12 @@ import argparse
import logging
import os
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.command.metrics import show_metrics
from dvc.command.status import CmdDataStatus
from dvc.dvcfile import PIPELINE_FILE
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -35,10 +34,11 @@ class CmdRepro(CmdBase):
interactive=self.args.interactive,
pipeline=self.args.pipeline,
all_pipelines=self.args.all_pipelines,
ignore_build_cache=self.args.ignore_build_cache,
run_cache=not self.args.no_run_cache,
no_commit=self.args.no_commit,
downstream=self.args.downstream,
recursive=self.args.recursive,
force_downstream=self.args.force_downstream,
)
if len(stages) == 0:
@@ -70,7 +70,7 @@ def add_parser(subparsers, parent_parser):
repro_parser.add_argument(
"targets",
nargs="*",
help="DVC-file to reproduce. 'Dvcfile' by default.",
help=f"Stages to reproduce. '{PIPELINE_FILE}' by default.",
)
repro_parser.add_argument(
"-f",
@@ -138,7 +138,16 @@ def add_parser(subparsers, parent_parser):
help="Reproduce all stages in the specified directory.",
)
repro_parser.add_argument(
"--ignore-build-cache",
"--no-run-cache",
action="store_true",
default=False,
help=(
"Execute stage commands even if they have already been run with "
"the same command/dependencies/outputs/etc before."
),
)
repro_parser.add_argument(
"--force-downstream",
action="store_true",
default=False,
help="Reproduce all descendants of a changed stage even if their "
+1 -2
View File
@@ -1,8 +1,7 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBaseNoRepo
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.utils import relpath
logger = logging.getLogger(__name__)
+40 -22
View File
@@ -1,18 +1,14 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdRun(CmdBase):
def run(self):
overwrite = self.args.yes or self.args.overwrite_dvcfile
if not any(
[
self.args.deps,
@@ -20,6 +16,8 @@ class CmdRun(CmdBase):
self.args.outs_no_cache,
self.args.metrics,
self.args.metrics_no_cache,
self.args.plots,
self.args.plots_no_cache,
self.args.outs_persist,
self.args.outs_persist_no_cache,
self.args.params,
@@ -28,8 +26,8 @@ class CmdRun(CmdBase):
): # pragma: no cover
logger.error(
"too few arguments. Specify at least one: `-d`, `-o`, `-O`, "
"`-m`, `-M`, `-p`, `--outs-persist`, "
"`--outs-persist-no-cache`, `command`."
"`-m`, `-M`, `-p`, `--plots`, `--plots-no-cache`, "
"`--outs-persist`, `--outs-persist-no-cache`, `command`."
)
return 1
@@ -40,21 +38,24 @@ class CmdRun(CmdBase):
outs_no_cache=self.args.outs_no_cache,
metrics=self.args.metrics,
metrics_no_cache=self.args.metrics_no_cache,
plots=self.args.plots,
plots_no_cache=self.args.plots_no_cache,
deps=self.args.deps,
params=self.args.params,
fname=self.args.file,
wdir=self.args.wdir,
no_exec=self.args.no_exec,
overwrite=overwrite,
ignore_build_cache=self.args.ignore_build_cache,
overwrite=self.args.overwrite_dvcfile,
run_cache=not self.args.no_run_cache,
no_commit=self.args.no_commit,
outs_persist=self.args.outs_persist,
outs_persist_no_cache=self.args.outs_persist_no_cache,
always_changed=self.args.always_changed,
name=self.args.name,
single_stage=self.args.single_stage,
)
except DvcException:
logger.exception("failed to run command")
logger.exception("")
return 1
return 0
@@ -78,7 +79,7 @@ class CmdRun(CmdBase):
if " " not in argument or '"' in argument:
return argument
return '"{}"'.format(argument)
return f'"{argument}"'
def add_parser(subparsers, parent_parser):
@@ -98,7 +99,9 @@ def add_parser(subparsers, parent_parser):
help="Declare dependencies for reproducible cmd.",
metavar="<path>",
)
run_parser.add_argument("-n", "--name", help=argparse.SUPPRESS)
run_parser.add_argument(
"-n", "--name", help="Stage name.",
)
run_parser.add_argument(
"-o",
"--outs",
@@ -141,6 +144,20 @@ def add_parser(subparsers, parent_parser):
"(do not put into DVC cache).",
metavar="<path>",
)
run_parser.add_argument(
"--plots",
action="append",
default=[],
help="Declare output plot file.",
metavar="<path>",
)
run_parser.add_argument(
"--plots-no-cache",
action="append",
default=[],
help="Declare output plot file (do not put into DVC cache).",
metavar="<path>",
)
run_parser.add_argument(
"-f",
"--file",
@@ -159,13 +176,6 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Only create stage file without actually running it.",
)
run_parser.add_argument(
"-y",
"--yes",
action="store_true",
default=False,
help="Deprecated, use --overwrite-dvcfile instead",
)
run_parser.add_argument(
"--overwrite-dvcfile",
action="store_true",
@@ -173,11 +183,13 @@ def add_parser(subparsers, parent_parser):
help="Overwrite existing DVC-file without asking for confirmation.",
)
run_parser.add_argument(
"--ignore-build-cache",
"--no-run-cache",
action="store_true",
default=False,
help="Run this stage even if it has been already ran with the same "
"command/dependencies/outputs/etc before.",
help=(
"Execute the command even if this stage has already been run "
"with the same command/dependencies/outputs/etc before."
),
)
run_parser.add_argument(
"--no-commit",
@@ -207,6 +219,12 @@ def add_parser(subparsers, parent_parser):
default=False,
help="Always consider this DVC-file as changed.",
)
run_parser.add_argument(
"--single-stage",
action="store_true",
default=False,
help=argparse.SUPPRESS,
)
run_parser.add_argument(
"command", nargs=argparse.REMAINDER, help="Command to execute."
)
+3 -3
View File
@@ -2,7 +2,6 @@ import logging
from dvc.command.data_sync import CmdDataBase
logger = logging.getLogger(__name__)
@@ -20,7 +19,7 @@ class CmdDataStatus(CmdDataBase):
ind = indent * self.STATUS_INDENT
if isinstance(status, str):
logger.info("{}{}".format(ind, status))
logger.info(f"{ind}{status}")
return
if isinstance(status, list):
@@ -34,7 +33,7 @@ class CmdDataStatus(CmdDataBase):
if isinstance(value, str):
logger.info("{}{}{}".format(ind, self._normalize(value), key))
elif value:
logger.info("{}{}:".format(ind, key))
logger.info(f"{ind}{key}:")
self._show(value, indent + 1)
def run(self):
@@ -49,6 +48,7 @@ class CmdDataStatus(CmdDataBase):
all_tags=self.args.all_tags,
all_commits=self.args.all_commits,
with_deps=self.args.with_deps,
recursive=self.args.recursive,
)
if st:
if self.args.quiet:
+2 -4
View File
@@ -1,11 +1,9 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -15,7 +13,7 @@ class CmdUnprotect(CmdBase):
try:
self.repo.unprotect(target)
except DvcException:
msg = "failed to unprotect '{}'".format(target)
msg = f"failed to unprotect '{target}'"
logger.exception(msg)
return 1
return 0
+17 -9
View File
@@ -1,23 +1,24 @@
import argparse
import logging
from dvc.command.base import append_doc_link
from dvc.command.base import CmdBase
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdUpdate(CmdBase):
def run(self):
ret = 0
for target in self.args.targets:
try:
self.repo.update(target, self.args.rev)
except DvcException:
logger.exception("failed to update '{}'.".format(target))
ret = 1
try:
self.repo.update(
targets=self.args.targets,
rev=self.args.rev,
recursive=self.args.recursive,
)
except DvcException:
logger.exception("failed update data")
ret = 1
return ret
@@ -39,4 +40,11 @@ def add_parser(subparsers, parent_parser):
help="Git revision (e.g. SHA, branch, tag)",
metavar="<commit>",
)
update_parser.add_argument(
"-R",
"--recursive",
action="store_true",
default=False,
help="Update all stages in the specified directory.",
)
update_parser.set_defaults(func=CmdUpdate)
+18 -17
View File
@@ -2,22 +2,23 @@ import argparse
import itertools
import logging
import os
import pathlib
import platform
import uuid
import pathlib
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.scm.base import SCMError
from dvc.system import System
from dvc.utils import is_binary, relpath
from dvc.utils.pkg import PKG
from dvc.version import __version__
try:
import psutil
except ImportError:
psutil = None
from dvc.utils import is_binary, relpath
from dvc.command.base import CmdBaseNoRepo, append_doc_link
from dvc.version import __version__
from dvc.exceptions import DvcException, NotDvcRepoError
from dvc.scm.base import SCMError
from dvc.system import System
from dvc.utils.pkg import PKG
logger = logging.getLogger(__name__)
@@ -27,12 +28,12 @@ class CmdVersion(CmdBaseNoRepo):
from dvc.repo import Repo
info = [
"DVC version: {}".format(__version__),
"Python version: {}".format(platform.python_version()),
"Platform: {}".format(platform.platform()),
"Binary: {}".format(is_binary()),
"Package: {}".format(PKG),
"Supported remotes: {}".format(self.get_supported_remotes()),
f"DVC version: {__version__}",
f"Python version: {platform.python_version()}",
f"Platform: {platform.platform()}",
f"Binary: {is_binary()}",
f"Package: {PKG}",
f"Supported remotes: {self.get_supported_remotes()}",
]
try:
@@ -50,7 +51,7 @@ class CmdVersion(CmdBaseNoRepo):
if psutil:
fs_type = self.get_fs_type(repo.cache.local.cache_dir)
info.append(
"Filesystem type (cache directory): {}".format(fs_type)
f"Filesystem type (cache directory): {fs_type}"
)
else:
logger.warning(
@@ -71,7 +72,7 @@ class CmdVersion(CmdBaseNoRepo):
if psutil:
fs_root = self.get_fs_type(os.path.abspath(root_directory))
info.append("Filesystem type (workspace): {}".format(fs_root))
info.append(f"Filesystem type (workspace): {fs_root}")
logger.info("\n".join(info))
return 0
@@ -114,7 +115,7 @@ class CmdVersion(CmdBaseNoRepo):
os.unlink(dst)
except DvcException:
status = "not supported"
cache.append("{name} - {status}".format(name=name, status=status))
cache.append(f"{name} - {status}")
os.remove(src)
return ", ".join(cache)
-42
View File
@@ -1,42 +0,0 @@
"""Python version compatibility code"""
# Backport os.fspath() from Python 3.6
try:
from os import fspath # noqa: F821
fspath_py35 = lambda s: s # noqa: E731
except ImportError:
def fspath(path):
"""Return the path representation of a path-like object.
If str or bytes is passed in, it is returned unchanged. Otherwise the
os.PathLike interface is used to get the path representation. If the
path representation is not str or bytes, TypeError is raised. If the
provided path is not str, bytes, or os.PathLike, TypeError is raised.
"""
if isinstance(path, (str, bytes)):
return path
# Work from the object's type to match method resolution of other magic
# methods.
path_type = type(path)
try:
path_repr = path_type.__fspath__(path)
except AttributeError:
if hasattr(path_type, "__fspath__"):
raise
else:
raise TypeError(
"expected str, bytes or os.PathLike object, "
"not " + path_type.__name__
)
if isinstance(path_repr, (str, bytes)):
return path_repr
else:
raise TypeError(
"expected {}.__fspath__() to return str or bytes, "
"not {}".format(path_type.__name__, type(path_repr).__name__)
)
fspath_py35 = fspath
+8 -7
View File
@@ -31,7 +31,7 @@ class ConfigError(DvcException):
"""DVC config exception."""
def __init__(self, msg):
super().__init__("config file error: {}".format(msg))
super().__init__(f"config file error: {msg}")
class NoRemoteError(ConfigError):
@@ -89,7 +89,7 @@ def ByUrl(mapping):
if os.name == "nt" and len(parsed.scheme) == 1 and parsed.netloc == "":
return schemas[""](data)
if parsed.scheme not in schemas:
raise Invalid("Unsupported URL type {}://".format(parsed.scheme))
raise Invalid(f"Unsupported URL type {parsed.scheme}://")
return schemas[parsed.scheme](data)
@@ -151,6 +151,7 @@ SCHEMA = {
Optional("listobjects", default=False): Bool,
Optional("use_ssl", default=True): Bool,
"sse": str,
"sse_kms_key_id": str,
"acl": str,
"grant_read": str,
"grant_read_acp": str,
@@ -289,7 +290,7 @@ class Config(dict):
Raises:
ConfigError: thrown if config has an invalid format.
"""
conf = self._load_config_to_level()
conf = self.load_config_to_level()
if validate:
conf = self.validate(conf)
@@ -349,7 +350,7 @@ class Config(dict):
}
return Schema(dirs_schema, extra=ALLOW_EXTRA)(conf)
def _load_config_to_level(self, level=None):
def load_config_to_level(self, level=None):
merged_conf = {}
for merge_level in self.LEVELS:
if merge_level == level:
@@ -368,7 +369,7 @@ class Config(dict):
conf = self._save_paths(conf, self.files[level])
merged_conf = self._load_config_to_level(level)
merged_conf = self.load_config_to_level(level)
_merge(merged_conf, conf)
self.validate(merged_conf)
@@ -389,7 +390,7 @@ def _load_config(filename):
def _save_config(filename, conf_dict):
logger.debug("Writing '{}'.".format(filename))
logger.debug(f"Writing '{filename}'.")
os.makedirs(os.path.dirname(filename), exist_ok=True)
config = configobj.ConfigObj(_pack_remotes(conf_dict))
@@ -416,7 +417,7 @@ def _pack_remotes(conf):
# Transform remote.name -> 'remote "name"'
for name, val in conf["remote"].items():
result['remote "{}"'.format(name)] = val
result[f'remote "{name}"'] = val
result.pop("remote", None)
return result
+3 -5
View File
@@ -7,9 +7,7 @@ import sys
from subprocess import Popen
from dvc.env import DVC_DAEMON
from dvc.utils import fix_env
from dvc.utils import is_binary
from dvc.utils import fix_env, is_binary
logger = logging.getLogger(__name__)
@@ -68,7 +66,7 @@ def _spawn_posix(cmd, env):
def _spawn(cmd, env):
logger.debug("Trying to spawn '{}'".format(cmd))
logger.debug(f"Trying to spawn '{cmd}'")
if os.name == "nt":
_spawn_windows(cmd, env)
@@ -77,7 +75,7 @@ def _spawn(cmd, env):
else:
raise NotImplementedError
logger.debug("Spawned '{}'".format(cmd))
logger.debug(f"Spawned '{cmd}'")
def daemon(args):
+9 -12
View File
@@ -6,12 +6,9 @@ import os
import pydoc
import sys
from grandalf.graphs import Edge
from grandalf.graphs import Graph
from grandalf.graphs import Vertex
from grandalf.graphs import Edge, Graph, Vertex
from grandalf.layouts import SugiyamaLayout
from grandalf.routing import EdgeViewer
from grandalf.routing import route_with_lines
from grandalf.routing import EdgeViewer, route_with_lines
from dvc.env import DVC_PAGER
from dvc.utils import format_link
@@ -40,7 +37,7 @@ def find_pager():
if env_pager:
return make_pager(env_pager)
if os.system("({}) 2>{}".format(DEFAULT_PAGER, os.devnull)) == 0:
if os.system(f"({DEFAULT_PAGER}) 2>{os.devnull}") == 0:
return make_pager(DEFAULT_PAGER_FORMATTED)
logger.warning(
@@ -52,7 +49,7 @@ def find_pager():
return pydoc.plainpager
class VertexViewer(object):
class VertexViewer:
"""Class to define vertex box boundaries that will be accounted for during
graph building by grandalf.
@@ -78,7 +75,7 @@ class VertexViewer(object):
return self._w
class AsciiCanvas(object):
class AsciiCanvas:
"""Class for drawing in ASCII.
Args:
@@ -148,21 +145,21 @@ class AsciiCanvas(object):
if dx == 0:
y = y0
else:
y = y0 + int(round((x - x0) * dy / float((dx))))
y = y0 + int(round((x - x0) * dy / float(dx)))
self.point(x, y, char)
elif y0 < y1:
for y in range(y0, y1 + 1):
if dy == 0:
x = x0
else:
x = x0 + int(round((y - y0) * dx / float((dy))))
x = x0 + int(round((y - y0) * dx / float(dy)))
self.point(x, y, char)
else:
for y in range(y1, y0 + 1):
if dy == 0:
x = x0
else:
x = x1 + int(round((y - y1) * dx / float((dy))))
x = x1 + int(round((y - y1) * dx / float(dy)))
self.point(x, y, char)
def text(self, x, y, text):
@@ -216,7 +213,7 @@ def _build_sugiyama_layout(vertexes, edges):
# Y
#
vertexes = {v: Vertex(" {} ".format(v)) for v in vertexes}
vertexes = {v: Vertex(f" {v} ") for v in vertexes}
# NOTE: reverting edges to correctly orientate the graph
edges = [Edge(vertexes[e], vertexes[s]) for s, e in edges]
vertexes = vertexes.values()
+11 -8
View File
@@ -5,11 +5,10 @@ import logging
from dvc.config import NoRemoteError
from dvc.remote import Remote
logger = logging.getLogger(__name__)
class DataCloud(object):
class DataCloud:
"""Class that manages dvc remotes.
Args:
@@ -48,7 +47,9 @@ class DataCloud(object):
def _init_remote(self, name):
return Remote(self.repo, name=name)
def push(self, cache, jobs=None, remote=None, show_checksums=False):
def push(
self, cache, jobs=None, remote=None, show_checksums=False,
):
"""Push data items in a cloud-agnostic way.
Args:
@@ -59,14 +60,15 @@ class DataCloud(object):
show_checksums (bool): show checksums instead of file names in
information messages.
"""
remote = self.get_remote(remote, "push")
return self.repo.cache.local.push(
cache,
jobs=jobs,
remote=self.get_remote(remote, "push"),
show_checksums=show_checksums,
cache, jobs=jobs, remote=remote, show_checksums=show_checksums,
)
def pull(self, cache, jobs=None, remote=None, show_checksums=False):
def pull(
self, cache, jobs=None, remote=None, show_checksums=False,
):
"""Pull data items in a cloud-agnostic way.
Args:
@@ -78,6 +80,7 @@ class DataCloud(object):
information messages.
"""
remote = self.get_remote(remote, "pull")
downloaded_items_num = self.repo.cache.local.pull(
cache, jobs=jobs, remote=remote, show_checksums=show_checksums
)
+6 -3
View File
@@ -1,22 +1,24 @@
from urllib.parse import urlparse
from collections import defaultdict
from urllib.parse import urlparse
import dvc.output as output
from dvc.dependency.azure import AzureDependency
from dvc.dependency.gs import GSDependency
from dvc.dependency.hdfs import HDFSDependency
from dvc.dependency.http import HTTPDependency
from dvc.dependency.https import HTTPSDependency
from dvc.dependency.local import LocalDependency
from dvc.dependency.param import ParamsDependency
from dvc.dependency.s3 import S3Dependency
from dvc.dependency.ssh import SSHDependency
from dvc.dependency.param import ParamsDependency
from dvc.output.base import BaseOutput
from dvc.remote import Remote
from dvc.scheme import Schemes
from .repo import RepoDependency
DEPS = [
AzureDependency,
GSDependency,
HDFSDependency,
HTTPDependency,
@@ -30,6 +32,7 @@ DEP_MAP = {
Schemes.LOCAL: LocalDependency,
Schemes.SSH: SSHDependency,
Schemes.S3: S3Dependency,
Schemes.AZURE: AzureDependency,
Schemes.GS: GSDependency,
Schemes.HDFS: HDFSDependency,
Schemes.HTTP: HTTPDependency,
+7
View File
@@ -0,0 +1,7 @@
from dvc.dependency.base import BaseDependency
from dvc.output.base import BaseOutput
from dvc.remote.azure import AzureRemote
class AzureDependency(BaseDependency, BaseOutput):
REMOTE = AzureRemote
+4 -6
View File
@@ -3,24 +3,22 @@ from dvc.exceptions import DvcException
class DependencyDoesNotExistError(DvcException):
def __init__(self, path):
msg = "dependency '{}' does not exist".format(path)
msg = f"dependency '{path}' does not exist"
super().__init__(msg)
class DependencyIsNotFileOrDirError(DvcException):
def __init__(self, path):
msg = "dependency '{}' is not a file or directory".format(path)
msg = f"dependency '{path}' is not a file or directory"
super().__init__(msg)
class DependencyIsStageFileError(DvcException):
def __init__(self, path):
super().__init__(
"Stage file '{}' cannot be a dependency.".format(path)
)
super().__init__(f"Stage file '{path}' cannot be a dependency.")
class BaseDependency(object):
class BaseDependency:
IS_DEPENDENCY = True
DoesNotExistError = DependencyDoesNotExistError
+2 -1
View File
@@ -1,6 +1,7 @@
from .http import HTTPDependency
from dvc.remote.https import HTTPSRemote
from .http import HTTPDependency
class HTTPSDependency(HTTPDependency):
REMOTE = HTTPSRemote
+12 -4
View File
@@ -1,11 +1,10 @@
import os
import yaml
from collections import defaultdict
import dpath.util
import yaml
from voluptuous import Any
from dvc.compat import fspath_py35
from dvc.dependency.local import LocalDependency
from dvc.exceptions import DvcException
@@ -41,6 +40,15 @@ class ParamsDependency(LocalDependency):
info=info,
)
def _dyn_load(self, values=None):
"""Load params values dynamically."""
if not values:
return
for param in self.params:
value = values.get(param)
if value:
self.info[param] = value
def save(self):
super().save()
self.info = self.save_info()
@@ -78,12 +86,12 @@ class ParamsDependency(LocalDependency):
if not self.exists:
return {}
with self.repo.tree.open(fspath_py35(self.path_info), "r") as fobj:
with self.repo.tree.open(self.path_info, "r") as fobj:
try:
config = yaml.safe_load(fobj)
except yaml.YAMLError as exc:
raise BadParamFileError(
"Unable to read parameters from '{}'".format(self)
f"Unable to read parameters from '{self}'"
) from exc
ret = {}
+20 -8
View File
@@ -2,10 +2,11 @@ import os
from voluptuous import Required
from .local import LocalDependency
from dvc.exceptions import OutputNotFoundError
from dvc.path_info import PathInfo
from .local import LocalDependency
class RepoDependency(LocalDependency):
PARAM_REPO = "repo"
@@ -49,13 +50,24 @@ class RepoDependency(LocalDependency):
return external_repo(d["url"], rev=rev)
def _get_checksum(self, locked=True):
from dvc.repo.tree import RepoTree
with self._make_repo(locked=locked) as repo:
try:
return repo.find_out_by_relpath(self.def_path).info["md5"]
except OutputNotFoundError:
path = PathInfo(os.path.join(repo.root_dir, self.def_path))
# we want stream but not fetch, so DVC out directories are
# walked, but dir contents is not fetched
tree = RepoTree(repo, stream=True)
# We are polluting our repo cache with some dir listing here
return self.repo.cache.local.get_checksum(path)
if tree.isdir(path):
return self.repo.cache.local.get_dir_checksum(
path, tree=tree
)
return tree.get_file_checksum(path)
def status(self):
current_checksum = self._get_checksum(locked=True)
@@ -75,16 +87,16 @@ class RepoDependency(LocalDependency):
def download(self, to):
with self._make_repo() as repo:
if self.def_repo.get(self.PARAM_REV_LOCK) is None:
self.def_repo[self.PARAM_REV_LOCK] = repo.scm.get_rev()
self.def_repo[self.PARAM_REV_LOCK] = repo.get_rev()
if hasattr(repo, "cache"):
repo.cache.local.cache_dir = self.repo.cache.local.cache_dir
repo.pull_to(self.def_path, to.path_info)
cache = self.repo.cache.local
with repo.use_cache(cache):
_, _, cache_infos = repo.fetch_external([self.def_path])
cache.checkout(to.path_info, cache_infos[0])
def update(self, rev=None):
if rev:
self.def_repo[self.PARAM_REV] = rev
with self._make_repo(locked=False) as repo:
self.def_repo[self.PARAM_REV_LOCK] = repo.scm.get_rev()
self.def_repo[self.PARAM_REV_LOCK] = repo.get_rev()
+35 -20
View File
@@ -1,20 +1,20 @@
import contextlib
import os
import logging
import dvc.prompt as prompt
import os
from voluptuous import MultipleInvalid
import dvc.prompt as prompt
from dvc import serialize
from dvc.exceptions import DvcException
from dvc.stage.loader import SingleStageLoader, StageLoader
from dvc.stage.exceptions import (
StageFileAlreadyExistsError,
StageFileBadNameError,
StageFileDoesNotExistError,
StageFileIsNotDvcFileError,
StageFileFormatError,
StageFileAlreadyExistsError,
StageFileIsNotDvcFileError,
)
from dvc.stage.loader import SingleStageLoader, StageLoader
from dvc.utils import relpath
from dvc.utils.collections import apply_diff
from dvc.utils.stage import (
@@ -27,13 +27,12 @@ logger = logging.getLogger(__name__)
DVC_FILE = "Dvcfile"
DVC_FILE_SUFFIX = ".dvc"
PIPELINE_FILE = "pipelines.yaml"
PIPELINE_LOCK = "pipelines.lock"
PIPELINE_FILE = "dvc.yaml"
PIPELINE_LOCK = "dvc.lock"
class LockfileCorruptedError(DvcException):
def __init__(self, path):
super().__init__("Lockfile '{}' is corrupted.".format(path))
pass
def is_valid_filename(path):
@@ -80,7 +79,7 @@ class FileMixin:
) == os.path.abspath(other.path)
def __str__(self):
return "{}: {}".format(self.__class__.__name__, self.relpath)
return f"{self.__class__.__name__}: {self.relpath}"
@property
def relpath(self):
@@ -103,7 +102,7 @@ class FileMixin:
with self.repo.tree.open(self.path) as fd:
stage_text = fd.read()
d = parse_stage(stage_text, self.path)
self.validate(d, self.path)
self.validate(d, self.relpath)
return d, stage_text
@classmethod
@@ -112,7 +111,7 @@ class FileMixin:
try:
cls.SCHEMA(d)
except MultipleInvalid as exc:
raise StageFileFormatError(fname, exc)
raise StageFileFormatError(f"'{fname}' format error: {exc}")
def remove_with_prompt(self, force=False):
raise NotImplementedError
@@ -176,25 +175,29 @@ class PipelineFile(FileMixin):
def _lockfile(self):
return Lockfile(self.repo, os.path.splitext(self.path)[0] + ".lock")
def dump(self, stage, update_pipeline=False, **kwargs):
def dump(self, stage, update_pipeline=False, no_lock=False):
"""Dumps given stage appropriately in the dvcfile."""
from dvc.stage import PipelineStage
assert isinstance(stage, PipelineStage)
check_dvc_filename(self.path)
self._dump_lockfile(stage)
if update_pipeline and not stage.is_data_source:
self._dump_pipeline_file(stage)
if not no_lock:
self._dump_lockfile(stage)
def _dump_lockfile(self, stage):
self._lockfile.dump(stage)
def _dump_pipeline_file(self, stage):
data = {}
if self.exists():
with open(self.path, "r") as fd:
with open(self.path) as fd:
data = parse_stage_for_update(fd.read(), self.path)
else:
logger.info("Creating '%s'", self.relpath)
open(self.path, "w+").close()
data["stages"] = data.get("stages", {})
@@ -205,6 +208,9 @@ class PipelineFile(FileMixin):
else:
data["stages"].update(stage_data)
logger.info(
"Adding stage '%s' to '%s'", stage.name, self.relpath,
)
dump_stage_file(self.path, data)
self.repo.scm.track_file(relpath(self.path))
@@ -238,23 +244,32 @@ class Lockfile(FileMixin):
with self.repo.tree.open(self.path) as fd:
data = parse_stage(fd.read(), self.path)
try:
self.validate(data, fname=self.path)
self.validate(data, fname=self.relpath)
except StageFileFormatError:
raise LockfileCorruptedError(self.path)
raise LockfileCorruptedError(
f"Lockfile '{self.relpath}' is corrupted."
)
return data
def dump(self, stage, **kwargs):
stage_data = serialize.to_lockfile(stage)
if not self.exists():
modified = True
logger.info("Generating lock file '%s'", self.relpath)
data = stage_data
open(self.path, "w+").close()
else:
with self.repo.tree.open(self.path, "r") as fd:
data = parse_stage_for_update(fd.read(), self.path)
modified = data.get(stage.name, {}) != stage_data.get(
stage.name, {}
)
if modified:
logger.info("Updating lock file '%s'", self.relpath)
data.update(stage_data)
dump_stage_file(self.path, data)
self.repo.scm.track_file(relpath(self.path))
if modified:
self.repo.scm.track_file(self.relpath)
class Dvcfile:
+43 -44
View File
@@ -1,11 +1,16 @@
"""Exceptions raised by the dvc."""
from funcy import first
from dvc.utils import relpath, format_link
from dvc.utils import format_link, relpath
class DvcException(Exception):
"""Base class for all dvc exceptions."""
def __init__(self, msg, *args):
assert msg
super().__init__(msg, *args)
class InvalidArgumentError(ValueError, DvcException):
"""Thrown if arguments are invalid."""
@@ -23,13 +28,14 @@ class OutputDuplicationError(DvcException):
def __init__(self, output, stages):
assert isinstance(output, str)
assert all(hasattr(stage, "relpath") for stage in stages)
msg = (
"file/directory '{}' is specified as an output in more than one "
"stage: \n{}\n"
"This is not allowed. Consider using a different output name."
).format(
output, "\n".join("\t{}".format(s.addressing) for s in stages)
)
if len(stages) == 1:
msg = "output '{}' is already specified in {}.".format(
output, first(stages)
)
else:
msg = "output '{}' is already specified in stages:\n{}".format(
output, "\n".join(f"\t- {s.addressing}" for s in stages),
)
super().__init__(msg)
self.stages = stages
self.output = output
@@ -81,10 +87,7 @@ class CircularDependencyError(DvcException):
def __init__(self, dependency):
assert isinstance(dependency, str)
msg = (
"file/directory '{}' is specified as an output and as a "
"dependency."
)
msg = "'{}' is specified as an output and as a dependency."
super().__init__(msg.format(dependency))
@@ -98,7 +101,7 @@ class ArgumentDuplicationError(DvcException):
def __init__(self, path):
assert isinstance(path, str)
super().__init__("file '{}' is specified more than once.".format(path))
super().__init__(f"file '{path}' is specified more than once.")
class MoveNotDataSourceError(DvcException):
@@ -133,14 +136,8 @@ class DvcParserError(DvcException):
class CyclicGraphError(DvcException):
def __init__(self, stages):
assert isinstance(stages, list)
stages = "\n".join(
"\t- {}".format(stage.addressing) for stage in stages
)
msg = (
"you've introduced a cycle in your pipeline that involves "
"the following stages:"
"\n"
"{stages}".format(stages=stages)
msg = "Pipeline has a cycle involving: {}.".format(
", ".join(s.addressing for s in stages)
)
super().__init__(msg)
@@ -148,7 +145,7 @@ class CyclicGraphError(DvcException):
class ConfirmRemoveError(DvcException):
def __init__(self, path):
super().__init__(
"unable to remove '{}' without a confirmation from the user. Use "
"unable to remove '{}' without a confirmation. Use "
"`-f` to force.".format(path)
)
@@ -160,7 +157,7 @@ class InitError(DvcException):
class ReproductionError(DvcException):
def __init__(self, dvc_file_name):
self.path = dvc_file_name
super().__init__("failed to reproduce '{}'".format(dvc_file_name))
super().__init__(f"failed to reproduce '{dvc_file_name}'")
class BadMetricError(DvcException):
@@ -168,7 +165,7 @@ class BadMetricError(DvcException):
super().__init__(
"the following metrics do not exist, "
"are not metric files or are malformed: {paths}".format(
paths=", ".join("'{}'".format(path) for path in paths)
paths=", ".join(f"'{path}'" for path in paths)
)
)
@@ -206,7 +203,7 @@ class OverlappingOutputPathsError(DvcException):
class CheckoutErrorSuggestGit(DvcException):
def __init__(self, target):
super().__init__("Did you mean `git checkout {}`?".format(target))
super().__init__(f"Did you mean `git checkout {target}`?")
class ETagMismatchError(DvcException):
@@ -220,9 +217,7 @@ class ETagMismatchError(DvcException):
class FileMissingError(DvcException):
def __init__(self, path):
self.path = path
super().__init__(
"Can't find '{}' neither locally nor on remote".format(path)
)
super().__init__(f"Can't find '{path}' neither locally nor on remote")
class DvcIgnoreInCollectedDirError(DvcException):
@@ -247,18 +242,14 @@ class DownloadError(DvcException):
def __init__(self, amount):
self.amount = amount
super().__init__(
"{amount} files failed to download".format(amount=amount)
)
super().__init__(f"{amount} files failed to download")
class UploadError(DvcException):
def __init__(self, amount):
self.amount = amount
super().__init__(
"{amount} files failed to upload".format(amount=amount)
)
super().__init__(f"{amount} files failed to upload")
class CheckoutError(DvcException):
@@ -280,7 +271,7 @@ class CollectCacheError(DvcException):
class NoRemoteInExternalRepoError(DvcException):
def __init__(self, url):
super().__init__(
"No DVC remote is specified in target repository '{}'.".format(url)
f"No DVC remote is specified in target repository '{url}'."
)
@@ -295,7 +286,7 @@ class NoOutputInExternalRepoError(DvcException):
class HTTPError(DvcException):
def __init__(self, code, reason):
super().__init__("'{} {}'".format(code, reason))
super().__init__(f"'{code} {reason}'")
class PathMissingError(DvcException):
@@ -303,19 +294,15 @@ class PathMissingError(DvcException):
"The path '{}' does not exist in the target repository '{}'"
" neither as a DVC output nor as a Git-tracked file."
)
default_msg_output_only = (
default_msg_dvc_only = (
"The path '{}' does not exist in the target repository '{}'"
" as an DVC output."
)
def __init__(self, path, repo, output_only=False):
msg = (
self.default_msg
if not output_only
else self.default_msg_output_only
)
def __init__(self, path, repo, dvc_only=False):
msg = self.default_msg if not dvc_only else self.default_msg_dvc_only
super().__init__(msg.format(path, repo))
self.output_only = output_only
self.dvc_only = dvc_only
class RemoteCacheRequiredError(DvcException):
@@ -335,3 +322,15 @@ class RemoteCacheRequiredError(DvcException):
class IsADirectoryError(DvcException):
"""Raised when a file operation is requested on a directory."""
class NoOutputOrStageError(DvcException):
"""
Raised when the target is neither an output nor a stage name in dvc.yaml
"""
def __init__(self, target, file):
super().__init__(
f"'{target}' "
f"does not exist as an output or a stage name in '{file}'"
)
+125 -88
View File
@@ -1,23 +1,27 @@
import logging
import os
import tempfile
import threading
from contextlib import contextmanager
from distutils.dir_util import copy_tree
import threading
from typing import Iterable
from funcy import retry, suppress, wrap_with, cached_property
from funcy import cached_property, retry, wrap_with
from dvc.path_info import PathInfo
from dvc.compat import fspath
from dvc.repo import Repo
from dvc.config import NoRemoteError, NotDvcRepoError
from dvc.exceptions import NoRemoteInExternalRepoError, CheckoutError
from dvc.exceptions import OutputNotFoundError, NoOutputInExternalRepoError
from dvc.exceptions import FileMissingError, PathMissingError
from dvc.utils.fs import remove, fs_copy, move
from dvc.utils import tmp_fname
from dvc.exceptions import (
FileMissingError,
NoOutputInExternalRepoError,
NoRemoteInExternalRepoError,
OutputNotFoundError,
PathMissingError,
)
from dvc.path_info import PathInfo
from dvc.repo import Repo
from dvc.repo.tree import RepoTree
from dvc.scm.git import Git
from dvc.scm.tree import is_working_tree
from dvc.utils.fs import remove
logger = logging.getLogger(__name__)
@@ -26,10 +30,12 @@ logger = logging.getLogger(__name__)
def external_repo(url, rev=None, for_write=False):
logger.debug("Creating external repo %s@%s", url, rev)
path = _cached_clone(url, rev, for_write=for_write)
if not rev:
rev = "HEAD"
try:
repo = ExternalRepo(path, url)
repo = ExternalRepo(path, url, rev, for_write=for_write)
except NotDvcRepoError:
repo = ExternalGitRepo(path, url)
repo = ExternalGitRepo(path, url, rev)
try:
yield repo
@@ -61,59 +67,99 @@ def clean_repos():
_remove(path)
class ExternalRepo(Repo):
def __init__(self, root_dir, url):
super().__init__(root_dir)
class BaseExternalRepo:
_local_cache = None
@property
def local_cache(self):
if hasattr(self, "cache"):
return self.cache.local
return self._local_cache
@contextmanager
def use_cache(self, cache):
"""Use the specified cache in place of default tmpdir cache for
download operations.
"""
if hasattr(self, "cache"):
save_cache = self.cache.local
self.cache.local = cache
self._local_cache = cache
yield
if hasattr(self, "cache"):
self.cache.local = save_cache
self._local_cache = None
@cached_property
def repo_tree(self):
return RepoTree(self, fetch=True)
def get_rev(self):
if is_working_tree(self.tree):
return self.scm.get_rev()
if hasattr(self.tree, "tree"):
return self.tree.tree.rev
return self.tree.rev
def fetch_external(self, paths: Iterable, **kwargs):
"""Fetch specified external repo paths into cache.
Returns 3-tuple in the form
(downloaded, failed, list(cache_infos))
where cache_infos can be used as checkout targets for the
fetched paths.
"""
download_results = []
failed = 0
paths = [PathInfo(self.root_dir) / path for path in paths]
def download_update(result):
download_results.append(result)
save_infos = []
for path in paths:
if not self.repo_tree.exists(path):
raise PathMissingError(path, self.url)
save_info = self.local_cache.save(
path,
None,
tree=self.repo_tree,
download_callback=download_update,
)
save_infos.append(save_info)
return sum(download_results), failed, save_infos
def get_external(self, path, dest):
"""Convenience wrapper for fetch_external and checkout."""
if self.local_cache:
# fetch DVC and git files to tmpdir cache, then checkout
_, _, save_infos = self.fetch_external([path])
self.local_cache.checkout(PathInfo(dest), save_infos[0])
else:
# git-only erepo with no cache, just copy files directly
# to dest
path = PathInfo(self.root_dir) / path
if not self.repo_tree.exists(path):
raise PathMissingError(path, self.url)
self.repo_tree.copytree(path, dest)
class ExternalRepo(Repo, BaseExternalRepo):
def __init__(self, root_dir, url, rev, for_write=False):
if for_write:
super().__init__(root_dir)
else:
root_dir = os.path.realpath(root_dir)
super().__init__(root_dir, scm=Git(root_dir), rev=rev)
self.url = url
self._set_cache_dir()
self._fix_upstream()
def pull_to(self, path, to_info):
"""
Pull the corresponding file or directory specified by `path` and
checkout it into `to_info`.
It works with files tracked by Git and DVC, and also local files
outside the repository.
"""
out = None
path_info = PathInfo(self.root_dir) / path
with suppress(OutputNotFoundError):
(out,) = self.find_outs_by_path(fspath(path_info), strict=False)
try:
if out and out.use_cache:
self._pull_cached(out, path_info, to_info)
return
# Check if it is handled by Git (it can't have an absolute path)
if os.path.isabs(path):
raise FileNotFoundError
fs_copy(fspath(path_info), fspath(to_info))
except FileNotFoundError:
raise PathMissingError(path, self.url)
def _pull_cached(self, out, path_info, dest):
with self.state:
tmp = PathInfo(tmp_fname(dest))
src = tmp / path_info.relative_to(out.path_info)
out.path_info = tmp
# Only pull unless all needed cache is present
if out.changed_cache(filter_info=src):
self.cloud.pull(out.get_used_cache(filter_info=src))
try:
out.checkout(filter_info=src)
except CheckoutError:
raise FileNotFoundError
move(src, dest)
remove(tmp)
@wrap_with(threading.Lock())
def _set_cache_dir(self):
try:
@@ -122,6 +168,7 @@ class ExternalRepo(Repo):
cache_dir = CACHE_DIRS[self.url] = tempfile.mkdtemp("dvc-cache")
self.cache.local.cache_dir = cache_dir
self._local_cache = self.cache.local
def _fix_upstream(self):
if not os.path.isdir(self.url):
@@ -162,10 +209,11 @@ class ExternalRepo(Repo):
self.config["core"]["remote"] = "auto-generated-upstream"
class ExternalGitRepo:
def __init__(self, root_dir, url):
self.root_dir = root_dir
class ExternalGitRepo(BaseExternalRepo):
def __init__(self, root_dir, url, rev):
self.root_dir = os.path.realpath(root_dir)
self.url = url
self.tree = self.scm.get_tree(rev)
@cached_property
def scm(self):
@@ -178,23 +226,15 @@ class ExternalGitRepo:
def find_out_by_relpath(self, path):
raise OutputNotFoundError(path, self)
def pull_to(self, path, to_info):
try:
# Git handled files can't have absolute path
if os.path.isabs(path):
raise FileNotFoundError
fs_copy(os.path.join(self.root_dir, path), fspath(to_info))
except FileNotFoundError:
raise PathMissingError(path, self.url)
@contextmanager
def open_by_relpath(self, path, mode="r", encoding=None, **kwargs):
"""Opens a specified resource as a file object."""
tree = RepoTree(self)
try:
abs_path = os.path.join(self.root_dir, path)
with open(abs_path, mode, encoding=encoding) as fd:
yield fd
with tree.open(
path, mode=mode, encoding=encoding, **kwargs
) as fobj:
yield fobj
except FileNotFoundError:
raise PathMissingError(path, self.url)
@@ -206,14 +246,12 @@ def _cached_clone(url, rev, for_write=False):
revision checked out. If for_write is set prevents reusing this dir via
cache.
"""
if not for_write and Git.is_sha(rev) and (url, rev) in CLONES:
return CLONES[url, rev]
# even if we have already cloned this repo, we may need to
# fetch/fast-forward to get specified rev
clone_path = _clone_default_branch(url, rev)
rev_sha = Git(clone_path).resolve_rev(rev or "HEAD")
if not for_write and (url, rev_sha) in CLONES:
return CLONES[url, rev_sha]
if not for_write and (url) in CLONES:
return CLONES[url]
# Copy to a new dir to keep the clone clean
repo_path = tempfile.mkdtemp("dvc-erepo")
@@ -221,11 +259,10 @@ def _cached_clone(url, rev, for_write=False):
copy_tree(clone_path, repo_path)
# Check out the specified revision
if rev is not None:
if for_write:
_git_checkout(repo_path, rev)
if not for_write:
CLONES[url, rev_sha] = repo_path
else:
CLONES[url] = repo_path
return repo_path
+88 -18
View File
@@ -5,13 +5,14 @@ from funcy import cached_property
from pathspec import PathSpec
from pathspec.patterns import GitWildMatchPattern
from dvc.path_info import PathInfo
from dvc.scm.tree import BaseTree
from dvc.utils import relpath
logger = logging.getLogger(__name__)
class DvcIgnore(object):
class DvcIgnore:
DVCIGNORE_FILE = ".dvcignore"
def __call__(self, root, dirs, files):
@@ -35,12 +36,19 @@ class DvcIgnorePatterns(DvcIgnore):
return dirs, files
def matches(self, dirname, basename):
abs_path = os.path.join(dirname, basename)
rel_path = relpath(abs_path, self.dirname)
if os.pardir + os.sep in rel_path:
# NOTE: `relpath` is too slow, so we have to assume that both
# `dirname` and `self.dirname` are relative or absolute together.
prefix = self.dirname + os.sep
if dirname == self.dirname:
path = basename
elif dirname.startswith(prefix):
rel = dirname[len(prefix) :]
# NOTE: `os.path.join` is ~x5.5 slower
path = f"{rel}{os.sep}{basename}"
else:
return False
return self.ignore_spec.match_file(rel_path)
return self.ignore_spec.match_file(path)
def __hash__(self):
return hash(self.ignore_file_path)
@@ -76,14 +84,14 @@ class DvcIgnoreRepo(DvcIgnore):
def is_dvc_repo(directory):
from dvc.repo import Repo
return os.path.isdir(os.path.join(directory, Repo.DVC_DIR))
return os.path.isdir(os.path.join(root, directory, Repo.DVC_DIR))
dirs = [d for d in dirs if not is_dvc_repo(d)]
return dirs, files
class DvcIgnoreFilter(object):
class DvcIgnoreFilter:
def __init__(self, tree):
self.tree = tree
self.ignores = {
@@ -119,27 +127,89 @@ class CleanTree(BaseTree):
return self.tree.tree_root
def open(self, path, mode="r", encoding="utf-8"):
return self.tree.open(path, mode, encoding)
if self.isfile(path):
return self.tree.open(path, mode, encoding)
raise FileNotFoundError
def exists(self, path):
return self.tree.exists(path)
if self.tree.exists(path) and self._parents_exist(path):
if self.tree.isdir(path):
return self._valid_dirname(path)
return self._valid_filename(path)
return False
def isdir(self, path):
return self.tree.isdir(path)
return (
self.tree.isdir(path)
and self._parents_exist(path)
and self._valid_dirname(path)
)
def _valid_dirname(self, path):
dirname, basename = os.path.split(os.path.normpath(path))
dirs, _ = self.dvcignore(os.path.abspath(dirname), [basename], [])
if dirs:
return True
return False
def isfile(self, path):
return self.tree.isfile(path)
return (
self.tree.isfile(path)
and self._parents_exist(path)
and self._valid_filename(path)
)
def _valid_filename(self, path):
dirname, basename = os.path.split(os.path.normpath(path))
_, files = self.dvcignore(os.path.abspath(dirname), [], [basename])
if files:
return True
return False
def isexec(self, path):
return self.tree.isexec(path)
return self.exists(path) and self.tree.isexec(path)
def _parents_exist(self, path):
from dvc.repo import Repo
path = PathInfo(path)
# if parent is tree_root or inside a .dvc dir we can skip this check
if path.parent == self.tree_root or Repo.DVC_DIR in path.parts:
return True
# if path is outside of tree, assume this is a local remote/local cache
# link/move operation where we do not need to filter ignores
path = relpath(path, self.tree_root)
if path.startswith("..") or (
os.name == "nt"
and not os.path.commonprefix(
[os.path.abspath(path), self.tree_root]
)
):
return True
# check if parent directories are in our ignores, starting from
# tree_root
for parent_dir in reversed(PathInfo(path).parents):
dirname, basename = os.path.split(parent_dir)
if basename == ".":
# parent_dir == tree_root
continue
dirs, _ = self.dvcignore(os.path.abspath(dirname), [basename], [])
if not dirs:
return False
return True
def walk(self, top, topdown=True):
for root, dirs, files in self.tree.walk(top, topdown):
dirs[:], files[:] = self.dvcignore(root, dirs, files)
dirs[:], files[:] = self.dvcignore(
os.path.abspath(root), dirs, files
)
yield root, dirs, files
def walk_files(self, top):
for root, _, files in self.walk(top):
for file in files:
yield os.path.join(root, file)
def stat(self, path):
if self.exists(path):
return self.tree.stat(path)
raise FileNotFoundError
+6 -2
View File
@@ -7,13 +7,17 @@
TEXT_CHARS = bytes(range(32, 127)) + b"\n\r\t\f\b"
def istextfile(fname, blocksize=512):
def istextfile(fname, blocksize=512, tree=None):
""" Uses heuristics to guess whether the given file is text or binary,
by reading a single block of bytes from the file.
If more than 30% of the chars in the block are non-text, or there
are NUL ('\x00') bytes in the block, assume this is a binary file.
"""
with open(fname, "rb") as fobj:
if tree:
open_func = tree.open
else:
open_func = open
with open_func(fname, "rb") as fobj:
block = fobj.read(blocksize)
if not block:
+5 -5
View File
@@ -5,19 +5,19 @@ import os
import time
from datetime import timedelta
import zc.lockfile
import flufl.lock
import zc.lockfile
from dvc.exceptions import DvcException
from dvc.utils import format_link
from dvc.progress import Tqdm
from dvc.utils import format_link
DEFAULT_TIMEOUT = 5
FAILED_TO_LOCK_MESSAGE = (
"cannot perform the command because another DVC process seems to be "
"running on this project. If that is not the case, manually remove "
"`.dvc/lock` and try again."
"`.dvc/tmp/lock` and try again."
)
@@ -25,7 +25,7 @@ class LockError(DvcException):
"""Thrown when unable to acquire the lock for DVC repo."""
class Lock(object):
class Lock:
"""Class for DVC repo lock.
Uses zc.lockfile as backend.
@@ -47,7 +47,7 @@ class Lock(object):
disable=not self._friendly,
desc=(
"If DVC froze, see `hardlink_lock` in {}".format(
format_link("man.dvc.org/config#core")
format_link("https://man.dvc.org/config#core")
)
),
):
+50 -11
View File
@@ -7,7 +7,6 @@ import colorama
from dvc.progress import Tqdm
FOOTER = (
"\n{yellow}Having any troubles?{nc}"
" Hit us up at {blue}https://dvc.org/support{nc},"
@@ -19,20 +18,45 @@ FOOTER = (
)
def addLoggingLevel(levelName, levelNum, methodName=None):
"""
Adds a new logging level to the `logging` module and the
currently configured logging class.
Based on https://stackoverflow.com/questions/2183233
"""
if methodName is None:
methodName = levelName.lower()
assert not hasattr(logging, levelName)
assert not hasattr(logging, methodName)
assert not hasattr(logging.getLoggerClass(), methodName)
def logForLevel(self, message, *args, **kwargs):
if self.isEnabledFor(levelNum):
self._log(levelNum, message, args, **kwargs)
def logToRoot(message, *args, **kwargs):
logging.log(levelNum, message, *args, **kwargs)
logging.addLevelName(levelNum, levelName)
setattr(logging, levelName, levelNum)
setattr(logging.getLoggerClass(), methodName, logForLevel)
setattr(logging, methodName, logToRoot)
class LoggingException(Exception):
def __init__(self, record):
msg = "failed to log {}".format(str(record))
super().__init__(msg)
class ExcludeErrorsFilter(logging.Filter):
def filter(self, record):
return record.levelno < logging.WARNING
def excludeFilter(level):
class ExcludeLevelFilter(logging.Filter):
def filter(self, record):
return record.levelno < level
class ExcludeInfoFilter(logging.Filter):
def filter(self, record):
return record.levelno < logging.INFO
return ExcludeLevelFilter
class ColorFormatter(logging.Formatter):
@@ -48,6 +72,7 @@ class ColorFormatter(logging.Formatter):
"""
color_code = {
"TRACE": colorama.Fore.GREEN,
"DEBUG": colorama.Fore.BLUE,
"WARNING": colorama.Fore.YELLOW,
"ERROR": colorama.Fore.RED,
@@ -117,7 +142,11 @@ class LoggerHandler(logging.StreamHandler):
def _is_verbose():
return logging.getLogger("dvc").getEffectiveLevel() == logging.DEBUG
return (
logging.NOTSET
< logging.getLogger("dvc").getEffectiveLevel()
<= logging.DEBUG
)
def _iter_causes(exc):
@@ -153,12 +182,14 @@ def disable_other_loggers():
def setup(level=logging.INFO):
colorama.init()
addLoggingLevel("TRACE", logging.DEBUG - 5)
logging.config.dictConfig(
{
"version": 1,
"filters": {
"exclude_errors": {"()": ExcludeErrorsFilter},
"exclude_info": {"()": ExcludeInfoFilter},
"exclude_errors": {"()": excludeFilter(logging.WARNING)},
"exclude_info": {"()": excludeFilter(logging.INFO)},
"exclude_debug": {"()": excludeFilter(logging.DEBUG)},
},
"formatters": {"color": {"()": ColorFormatter}},
"handlers": {
@@ -176,6 +207,13 @@ def setup(level=logging.INFO):
"stream": "ext://sys.stdout",
"filters": ["exclude_info"],
},
"console_trace": {
"class": "dvc.logger.LoggerHandler",
"level": "TRACE",
"formatter": "color",
"stream": "ext://sys.stdout",
"filters": ["exclude_debug"],
},
"console_errors": {
"class": "dvc.logger.LoggerHandler",
"level": "WARNING",
@@ -189,6 +227,7 @@ def setup(level=logging.INFO):
"handlers": [
"console_info",
"console_debug",
"console_trace",
"console_errors",
],
},
+17 -10
View File
@@ -6,12 +6,11 @@ import logging
from dvc import analytics
from dvc.cli import parse_args
from dvc.config import ConfigError
from dvc.exceptions import DvcParserError
from dvc.exceptions import NotDvcRepoError
from dvc.exceptions import DvcException, DvcParserError, NotDvcRepoError
from dvc.external_repo import clean_repos
from dvc.logger import disable_other_loggers, FOOTER
from dvc.utils import format_link
from dvc.logger import FOOTER, disable_other_loggers
from dvc.remote.pool import close_pools
from dvc.utils import format_link
# Workaround for CPython bug. See [1] and [2] for more info.
# [1] https://github.com/aws/aws-cli/blob/1.16.277/awscli/clidriver.py#L55
@@ -19,7 +18,6 @@ from dvc.remote.pool import close_pools
"".encode("idna")
logger = logging.getLogger("dvc")
@@ -39,11 +37,17 @@ def main(argv=None):
try:
args = parse_args(argv)
if args.quiet:
logger.setLevel(logging.CRITICAL)
elif args.verbose:
logger.setLevel(logging.DEBUG)
verbosity = args.verbose - args.quiet
if verbosity:
logger.setLevel(
{
-2: logging.CRITICAL,
-1: logging.ERROR,
1: logging.DEBUG,
2: logging.TRACE,
}[max(-2, min(verbosity, 2))]
)
logger.trace(args)
cmd = args.func(args)
ret = cmd.run()
@@ -58,6 +62,9 @@ def main(argv=None):
ret = 253
except DvcParserError:
ret = 254
except DvcException:
ret = 255
logger.exception("")
except Exception as exc: # pylint: disable=broad-except
if isinstance(exc, OSError) and exc.errno == errno.EMFILE:
logger.exception(
+29 -16
View File
@@ -1,5 +1,6 @@
from urllib.parse import urlparse
from voluptuous import Any, Required, Lower, Length, Coerce, And, SetTo
from voluptuous import And, Any, Coerce, Length, Lower, Required, SetTo
from dvc.output.base import BaseOutput
from dvc.output.gs import GSOutput
@@ -53,10 +54,11 @@ SCHEMA = CHECKSUMS_SCHEMA.copy()
SCHEMA[Required(BaseOutput.PARAM_PATH)] = str
SCHEMA[BaseOutput.PARAM_CACHE] = bool
SCHEMA[BaseOutput.PARAM_METRIC] = BaseOutput.METRIC_SCHEMA
SCHEMA[BaseOutput.PARAM_PLOT] = bool
SCHEMA[BaseOutput.PARAM_PERSIST] = bool
def _get(stage, p, info, cache, metric, persist=False):
def _get(stage, p, info, cache, metric, plot=False, persist=False):
parsed = urlparse(p)
if parsed.scheme == "remote":
@@ -68,6 +70,7 @@ def _get(stage, p, info, cache, metric, persist=False):
cache=cache,
remote=remote,
metric=metric,
plot=plot,
persist=persist,
)
@@ -80,6 +83,7 @@ def _get(stage, p, info, cache, metric, persist=False):
cache=cache,
remote=None,
metric=metric,
plot=plot,
persist=persist,
)
return LocalOutput(
@@ -89,6 +93,7 @@ def _get(stage, p, info, cache, metric, persist=False):
cache=cache,
remote=None,
metric=metric,
plot=plot,
persist=persist,
)
@@ -99,26 +104,34 @@ def loadd_from(stage, d_list):
p = d.pop(BaseOutput.PARAM_PATH)
cache = d.pop(BaseOutput.PARAM_CACHE, True)
metric = d.pop(BaseOutput.PARAM_METRIC, False)
plot = d.pop(BaseOutput.PARAM_PLOT, False)
persist = d.pop(BaseOutput.PARAM_PERSIST, False)
ret.append(
_get(
stage, p, info=d, cache=cache, metric=metric, persist=persist,
)
)
return ret
def loads_from(stage, s_list, use_cache=True, metric=False, persist=False):
ret = []
for s in s_list:
ret.append(
_get(
stage,
s,
info={},
cache=use_cache,
p,
info=d,
cache=cache,
metric=metric,
plot=plot,
persist=persist,
)
)
return ret
def loads_from(
stage, s_list, use_cache=True, metric=False, plot=False, persist=False
):
return [
_get(
stage,
s,
info={},
cache=use_cache,
metric=metric,
plot=plot,
persist=persist,
)
for s in s_list
]
+62 -41
View File
@@ -1,44 +1,50 @@
import logging
import os
from urllib.parse import urlparse
from copy import copy
from urllib.parse import urlparse
from voluptuous import Any
import dvc.prompt as prompt
from dvc.cache import NamedCache
from dvc.exceptions import CollectCacheError, RemoteCacheRequiredError
from dvc.exceptions import DvcException
from dvc.exceptions import (
CollectCacheError,
DvcException,
RemoteCacheRequiredError,
)
from dvc.remote.base import BaseRemote
logger = logging.getLogger(__name__)
class OutputDoesNotExistError(DvcException):
def __init__(self, path):
msg = "output '{}' does not exist".format(path)
msg = f"output '{path}' does not exist"
super().__init__(msg)
class OutputIsNotFileOrDirError(DvcException):
def __init__(self, path):
msg = "output '{}' is not a file or directory".format(path)
msg = f"output '{path}' is not a file or directory"
super().__init__(msg)
class OutputAlreadyTrackedError(DvcException):
def __init__(self, path):
msg = "output '{}' is already tracked by SCM (e.g. Git)".format(path)
msg = f""" output '{path}' is already tracked by SCM (e.g. Git).
You can remove it from Git, then add to DVC.
To stop tracking from Git:
git rm -r --cached '{path}'
git commit -m "stop tracking {path}" """
super().__init__(msg)
class OutputIsStageFileError(DvcException):
def __init__(self, path):
super().__init__("Stage file '{}' cannot be an output.".format(path))
super().__init__(f"Stage file '{path}' cannot be an output.")
class BaseOutput(object):
class BaseOutput:
IS_DEPENDENCY = False
REMOTE = BaseRemote
@@ -48,6 +54,8 @@ class BaseOutput(object):
PARAM_METRIC = "metric"
PARAM_METRIC_TYPE = "type"
PARAM_METRIC_XPATH = "xpath"
PARAM_PLOT = "plot"
PARAM_PLOT_TEMPLATE = "template"
PARAM_PERSIST = "persist"
METRIC_SCHEMA = Any(
@@ -73,6 +81,7 @@ class BaseOutput(object):
remote=None,
cache=True,
metric=False,
plot=False,
persist=False,
):
self._validate_output_path(path)
@@ -93,6 +102,7 @@ class BaseOutput(object):
self.remote = remote or self.REMOTE(self.repo, {})
self.use_cache = False if self.IS_DEPENDENCY else cache
self.metric = False if self.IS_DEPENDENCY else metric
self.plot = False if self.IS_DEPENDENCY else plot
self.persist = persist
self.path_info = self._parse_path(remote, path)
@@ -209,6 +219,15 @@ class BaseOutput(object):
def isfile(self):
return self.remote.isfile(self.path_info)
def ignore(self):
if not self.use_scm_ignore:
return
if self.repo.scm.is_tracked(self.fspath):
raise OutputAlreadyTrackedError(self)
self.repo.scm.ignore(self.fspath)
def save(self):
if not self.exists:
raise self.DoesNotExistError(self)
@@ -217,32 +236,24 @@ class BaseOutput(object):
raise self.IsNotFileOrDirError(self)
if self.is_empty:
logger.warning("'{}' is empty.".format(self))
logger.warning(f"'{self}' is empty.")
if self.use_scm_ignore:
if self.repo.scm.is_tracked(self.fspath):
raise OutputAlreadyTrackedError(self)
self.repo.scm.ignore(self.fspath)
self.ignore()
if not self.use_cache:
self.info = self.save_info()
if self.metric:
if self.metric or self.plot:
self.verify_metric()
if not self.IS_DEPENDENCY:
logger.info(
"Output '{}' doesn't use cache. Skipping saving.".format(
self
)
logger.debug(
"Output '%s' doesn't use cache. Skipping saving.", self
)
return
assert not self.IS_DEPENDENCY
if not self.changed():
logger.info(
"Output '{}' didn't change. Skipping saving.".format(self)
)
logger.debug("Output '%s' didn't change. Skipping saving.", self)
return
self.info = self.save_info()
@@ -258,7 +269,8 @@ class BaseOutput(object):
if self.IS_DEPENDENCY:
return ret
ret[self.PARAM_CACHE] = self.use_cache
if not self.use_cache:
ret[self.PARAM_CACHE] = self.use_cache
if isinstance(self.metric, dict):
if (
@@ -267,15 +279,19 @@ class BaseOutput(object):
):
del self.metric[self.PARAM_METRIC_XPATH]
ret[self.PARAM_METRIC] = self.metric
ret[self.PARAM_PERSIST] = self.persist
if self.metric:
ret[self.PARAM_METRIC] = self.metric
if self.plot:
ret[self.PARAM_PLOT] = self.plot
if self.persist:
ret[self.PARAM_PERSIST] = self.persist
return ret
def verify_metric(self):
raise DvcException(
"verify metric is not supported for {}".format(self.scheme)
)
raise DvcException(f"verify metric is not supported for {self.scheme}")
def download(self, to):
self.remote.download(self.path_info, to.path_info)
@@ -336,7 +352,18 @@ class BaseOutput(object):
if self.exists:
self.remote.unprotect(self.path_info)
def _collect_used_dir_cache(
def get_dir_cache(self, **kwargs):
if not self.is_dir_checksum:
raise DvcException("cannot get dir cache for file checksum")
if self.cache.changed_cache_file(self.checksum):
self.repo.cloud.pull(
NamedCache.make("local", self.checksum, str(self)),
show_checksums=False,
**kwargs,
)
return self.dir_cache
def collect_used_dir_cache(
self, remote=None, force=False, jobs=None, filter_info=None
):
"""Get a list of `info`s related to the given directory.
@@ -360,16 +387,10 @@ class BaseOutput(object):
cache = NamedCache()
if self.cache.changed_cache_file(self.checksum):
try:
self.repo.cloud.pull(
NamedCache.make("local", self.checksum, str(self)),
jobs=jobs,
remote=remote,
show_checksums=False,
)
except DvcException:
logger.debug("failed to pull cache for '{}'".format(self))
try:
self.get_dir_cache(jobs=jobs, remote=remote)
except DvcException:
logger.debug(f"failed to pull cache for '{self}'")
if self.cache.changed_cache_file(self.checksum):
msg = (
@@ -446,7 +467,7 @@ class BaseOutput(object):
return ret
ret.add_child_cache(
self.checksum, self._collect_used_dir_cache(**kwargs),
self.checksum, self.collect_used_dir_cache(**kwargs),
)
return ret
+8 -9
View File
@@ -7,10 +7,8 @@ from dvc.istextfile import istextfile
from dvc.output.base import BaseOutput
from dvc.remote.local import LocalRemote
from dvc.utils import relpath
from dvc.compat import fspath_py35
from dvc.utils.fs import path_isin
logger = logging.getLogger(__name__)
@@ -39,7 +37,7 @@ class LocalOutput(BaseOutput):
if not p.is_absolute():
p = self.stage.wdir / p
abs_p = os.path.abspath(os.path.normpath(fspath_py35(p)))
abs_p = os.path.abspath(os.path.normpath(p))
return self.REMOTE.path_cls(abs_p)
def __str__(self):
@@ -73,17 +71,18 @@ class LocalOutput(BaseOutput):
return ret
def verify_metric(self):
if not self.metric:
if not self.metric or self.plot:
return
path = fspath_py35(self.path_info)
path = os.fspath(self.path_info)
if not os.path.exists(path):
return
name = "metrics" if self.metric else "plot"
if os.path.isdir(path):
msg = "directory '{}' cannot be used as metrics."
raise DvcException(msg.format(self.path_info))
msg = "directory '{}' cannot be used as {}."
raise DvcException(msg.format(self.path_info, name))
if not istextfile(path):
msg = "binary file '{}' cannot be used as metrics."
raise DvcException(msg.format(self.path_info))
msg = "binary file '{}' cannot be used as {}."
raise DvcException(msg.format(self.path_info, name))
+8 -9
View File
@@ -1,6 +1,6 @@
import os
import posixpath
import pathlib
import posixpath
from urllib.parse import urlparse
from funcy import cached_property
@@ -8,7 +8,7 @@ from funcy import cached_property
from dvc.utils import relpath
class _BasePath(object):
class _BasePath:
def overlaps(self, other):
if isinstance(other, (str, bytes)):
other = self.__class__(other)
@@ -50,7 +50,6 @@ class PathInfo(pathlib.PurePath, _BasePath):
return "{}: '{}'".format(type(self).__name__, self)
# This permits passing it to file utils directly in Python 3.6+
# With Python 2.7, Python 3.5+ we are stuck with path_info.fspath for now
def __fspath__(self):
return pathlib.PurePath.__str__(self)
@@ -88,7 +87,7 @@ class _URLPathInfo(PosixPathInfo):
__unicode__ = __str__
class _URLPathParents(object):
class _URLPathParents:
def __init__(self, src):
self.src = src
self._parents = self.src._path.parents
@@ -100,7 +99,7 @@ class _URLPathParents(object):
return self.src.replace(path=self._parents[idx])
def __repr__(self):
return "<{}.parents>".format(self.src)
return f"<{self.src}.parents>"
class URLInfo(_BasePath):
@@ -120,7 +119,7 @@ class URLInfo(_BasePath):
assert bool(host) ^ bool(netloc)
if netloc is not None:
return cls("{}://{}{}".format(scheme, netloc, path))
return cls(f"{scheme}://{netloc}{path}")
obj = cls.__new__(cls)
obj._fill_parts(scheme, host, user, port, path)
@@ -154,7 +153,7 @@ class URLInfo(_BasePath):
@cached_property
def url(self):
return "{}://{}{}".format(self.scheme, self.netloc, self._spath)
return f"{self.scheme}://{self.netloc}{self._spath}"
def __str__(self):
return self.url
@@ -216,10 +215,10 @@ class URLInfo(_BasePath):
if isinstance(other, (str, bytes)):
other = self.__class__(other)
if self.__class__ != other.__class__:
msg = "'{}' has incompatible class with '{}'".format(self, other)
msg = f"'{self}' has incompatible class with '{other}'"
raise ValueError(msg)
if self._base_parts != other._base_parts:
msg = "'{}' does not start with '{}'".format(self, other)
msg = f"'{self}' does not start with '{other}'"
raise ValueError(msg)
return self._path.relative_to(other._path)
+23 -21
View File
@@ -19,18 +19,17 @@ class Tqdm(tqdm):
BAR_FMT_DEFAULT = (
"{percentage:3.0f}% {desc}|{bar}|"
"{n_fmt}/{total_fmt}"
" [{elapsed}<{remaining}, {rate_fmt:>11}{postfix}]"
"{postfix[info]}{n_fmt}/{total_fmt}"
" [{elapsed}<{remaining}, {rate_fmt:>11}]"
)
# nested bars should have fixed bar widths to align nicely
BAR_FMT_DEFAULT_NESTED = (
"{percentage:3.0f}%|{bar:10}|{desc:{ncols_desc}.{ncols_desc}}"
"{n_fmt}/{total_fmt}"
" [{elapsed}<{remaining}, {rate_fmt:>11}{postfix}]"
"{postfix[info]}{n_fmt}/{total_fmt}"
" [{elapsed}<{remaining}, {rate_fmt:>11}]"
)
BAR_FMT_NOTOTAL = (
"{desc:{ncols_desc}.{ncols_desc}}{n_fmt}"
" [{elapsed}, {rate_fmt:>11}{postfix}]"
"{desc}{bar:b}|{postfix[info]}{n_fmt} [{elapsed}, {rate_fmt:>11}]"
)
BYTES_DEFAULTS = dict(
unit="B", unit_scale=True, unit_divisor=1024, miniters=1
@@ -47,6 +46,7 @@ class Tqdm(tqdm):
bytes=False, # pylint: disable=W0622
file=None,
total=None,
postfix=None,
**kwargs
):
"""
@@ -68,7 +68,6 @@ class Tqdm(tqdm):
kwargs.setdefault("unit_scale", total > 999 if total else True)
if file is None:
file = sys.stderr
self.desc_persist = desc
# auto-disable based on `logger.level`
if not disable:
disable = logger.getEffectiveLevel() > level
@@ -89,6 +88,7 @@ class Tqdm(tqdm):
total=total,
**kwargs
)
self.postfix = postfix or {"info": ""}
if bar_format is None:
if self.__len__():
self.bar_format = (
@@ -102,11 +102,11 @@ class Tqdm(tqdm):
self.bar_format = bar_format
self.refresh()
def update_desc(self, desc, n=1):
def update_msg(self, msg, n=1):
"""
Calls `set_description_str(desc)` and `update(n)`
Sets `msg` as a postfix and calls `update(n)`.
"""
self.set_description_str(desc, refresh=False)
self.postfix["info"] = " %s |" % msg
self.update(n)
def update_to(self, current, total=None):
@@ -130,12 +130,11 @@ class Tqdm(tqdm):
return wrapped
def close(self):
if self.desc_persist is not None:
self.set_description_str(self.desc_persist, refresh=False)
# unknown/zero ETA
self.bar_format = self.bar_format.replace("<{remaining}", "")
# remove completed bar
self.bar_format = self.bar_format.replace("|{bar:10}|", " ")
self.postfix["info"] = ""
# remove ETA (either unknown or zero); remove completed bar
self.bar_format = self.bar_format.replace("<{remaining}", "").replace(
"|{bar:10}|", " "
)
super().close()
@property
@@ -143,12 +142,15 @@ class Tqdm(tqdm):
"""inject `ncols_desc` to fill the display width (`ncols`)"""
d = super().format_dict
ncols = d["ncols"] or 80
ncols_desc = ncols - len(self.format_meter(ncols_desc=1, **d)) + 1
ncols_desc = max(ncols_desc, 0)
if ncols_desc:
d["ncols_desc"] = ncols_desc
# assumes `bar_format` has max one of ("ncols_desc" & "ncols_info")
ncols_left = (
ncols - len(self.format_meter(ncols_desc=1, ncols_info=1, **d)) + 1
)
ncols_left = max(ncols_left, 0)
if ncols_left:
d["ncols_desc"] = d["ncols_info"] = ncols_left
else:
# work-around for zero-width description
d["ncols_desc"] = 1
d["ncols_desc"] = d["ncols_info"] = 1
d["prefix"] = ""
return d
+2 -2
View File
@@ -38,7 +38,7 @@ def confirm(statement):
Returns:
bool: whether or not specified statement was confirmed.
"""
prompt = "{statement} [y/n]".format(statement=statement)
prompt = f"{statement} [y/n]"
answer = _ask(prompt, limited_to=["yes", "no", "y", "n"])
return answer and answer.startswith("y")
@@ -52,5 +52,5 @@ def password(statement):
Returns:
str: password entered by the user.
"""
logger.info("{statement}: ".format(statement=statement))
logger.info(f"{statement}: ")
return getpass("")
-1
View File
@@ -12,7 +12,6 @@ from dvc.remote.oss import OSSRemote
from dvc.remote.s3 import S3Remote
from dvc.remote.ssh import SSHRemote
REMOTES = [
AzureRemote,
GDriveRemote,
+20 -37
View File
@@ -1,10 +1,8 @@
import logging
import os
import posixpath
import re
from datetime import datetime, timedelta
from urllib.parse import urlparse
import threading
from datetime import datetime, timedelta
from funcy import cached_property, wrap_prop
@@ -13,19 +11,12 @@ from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
class AzureRemote(BaseRemote):
scheme = Schemes.AZURE
path_cls = CloudURLInfo
REGEX = (
r"azure://((?P<path>[^=;]*)?|("
# backward compatibility
r"(ContainerName=(?P<container_name>[^;]+);?)?"
r"(?P<connection_string>.+)?)?)$"
)
REQUIRES = {"azure-storage-blob": "azure.storage.blob"}
PARAM_CHECKSUM = "etag"
COPY_POLL_SECONDS = 5
@@ -35,31 +26,14 @@ class AzureRemote(BaseRemote):
super().__init__(repo, config)
url = config.get("url", "azure://")
self.path_info = self.path_cls(url)
match = re.match(self.REGEX, url) # backward compatibility
path = match.group("path")
bucket = (
urlparse(url if path else "").netloc
or match.group("container_name") # backward compatibility
or os.getenv("AZURE_STORAGE_CONTAINER_NAME")
)
if not self.path_info.bucket:
container = os.getenv("AZURE_STORAGE_CONTAINER_NAME")
self.path_info = self.path_cls(f"azure://{container}")
self.connection_string = (
config.get("connection_string")
or match.group("connection_string") # backward compatibility
or os.getenv("AZURE_STORAGE_CONNECTION_STRING")
)
if not bucket:
raise ValueError("azure storage container name missing")
if not self.connection_string:
raise ValueError("azure storage connection string missing")
self.path_info = (
self.path_cls(url)
if path
else self.path_cls.from_parts(scheme=self.scheme, netloc=bucket)
self.connection_string = config.get("connection_string") or os.getenv(
"AZURE_STORAGE_CONNECTION_STRING"
)
@wrap_prop(threading.Lock())
@@ -68,12 +42,12 @@ class AzureRemote(BaseRemote):
from azure.storage.blob import BlockBlobService
from azure.common import AzureMissingResourceHttpError
logger.debug("URL {}".format(self.path_info))
logger.debug("Connection string {}".format(self.connection_string))
logger.debug(f"URL {self.path_info}")
logger.debug(f"Connection string {self.connection_string}")
blob_service = BlockBlobService(
connection_string=self.connection_string
)
logger.debug("Container name {}".format(self.path_info.bucket))
logger.debug(f"Container name {self.path_info.bucket}")
try: # verify that container exists
blob_service.list_blobs(
self.path_info.bucket, delimiter="/", num_results=1
@@ -82,11 +56,20 @@ class AzureRemote(BaseRemote):
blob_service.create_container(self.path_info.bucket)
return blob_service
def get_etag(self, path_info):
etag = self.blob_service.get_blob_properties(
path_info.bucket, path_info.path
).properties.etag
return etag.strip('"')
def get_file_checksum(self, path_info):
return self.get_etag(path_info)
def remove(self, path_info):
if path_info.scheme != self.scheme:
raise NotImplementedError
logger.debug("Removing {}".format(path_info))
logger.debug(f"Removing {path_info}")
self.blob_service.delete_blob(path_info.bucket, path_info.path)
def _list_paths(self, bucket, prefix, progress_callback=None):
+138 -90
View File
@@ -4,20 +4,20 @@ import itertools
import json
import logging
import tempfile
from urllib.parse import urlparse
from concurrent.futures import ThreadPoolExecutor
from copy import copy
from functools import partial, wraps
from multiprocessing import cpu_count
from operator import itemgetter
from urllib.parse import urlparse
from shortuuid import uuid
import dvc.prompt as prompt
from dvc.exceptions import (
CheckoutError,
DvcException,
ConfirmRemoveError,
DvcException,
DvcIgnoreInCollectedDirError,
RemoteCacheRequiredError,
)
@@ -26,9 +26,10 @@ from dvc.path_info import PathInfo, URLInfo, WindowsPathInfo
from dvc.progress import Tqdm
from dvc.remote.index import RemoteIndex, RemoteIndexNoop
from dvc.remote.slow_link_detection import slow_link_guard
from dvc.scm.tree import is_working_tree
from dvc.state import StateNoop
from dvc.utils import tmp_fname
from dvc.utils.fs import move, makedirs
from dvc.utils.fs import makedirs, move
from dvc.utils.http import open_url
logger = logging.getLogger(__name__)
@@ -57,7 +58,7 @@ class RemoteCmdError(DvcException):
class RemoteActionNotImplemented(DvcException):
def __init__(self, action, scheme):
m = "{} is not supported for {} remotes".format(action, scheme)
m = f"{action} is not supported for {scheme} remotes"
super().__init__(m)
@@ -68,7 +69,7 @@ class RemoteMissingDepsError(DvcException):
class DirCacheError(DvcException):
def __init__(self, checksum):
super().__init__(
"Failed to load dir cache for hash value: '{}'.".format(checksum)
f"Failed to load dir cache for hash value: '{checksum}'."
)
@@ -82,7 +83,7 @@ def index_locked(f):
return wrapper
class BaseRemote(object):
class BaseRemote:
scheme = "base"
path_cls = URLInfo
REQUIRES = {}
@@ -95,7 +96,7 @@ class BaseRemote(object):
DEFAULT_CACHE_TYPES = ["copy"]
DEFAULT_VERIFY = False
LIST_OBJECT_PAGE_SIZE = 1000
TRAVERSE_WEIGHT_MULTIPLIER = 20
TRAVERSE_WEIGHT_MULTIPLIER = 5
TRAVERSE_PREFIX_LEN = 3
TRAVERSE_THRESHOLD_SIZE = 500000
CAN_TRAVERSE = True
@@ -151,7 +152,7 @@ class BaseRemote(object):
if not missing:
return
url = config.get("url", "{}://".format(self.scheme))
url = config.get("url", f"{self.scheme}://")
msg = (
"URL '{}' is supported but requires these missing "
"dependencies: {}. If you have installed dvc using pip, "
@@ -211,23 +212,33 @@ class BaseRemote(object):
checksums = dict(zip(file_infos, tasks))
return checksums
def _collect_dir(self, path_info):
def _collect_dir(self, path_info, tree=None, save_tree=False, **kwargs):
file_infos = set()
for fname in self.walk_files(path_info):
if tree:
walk_files = tree.walk_files
else:
walk_files = self.walk_files
for fname in walk_files(path_info, **kwargs):
if DvcIgnore.DVCIGNORE_FILE == fname.name:
raise DvcIgnoreInCollectedDirError(fname.parent)
file_infos.add(fname)
checksums = {fi: self.state.get(fi) for fi in file_infos}
not_in_state = {
fi for fi, checksum in checksums.items() if checksum is None
}
if tree:
checksums = {fi: tree.get_file_checksum(fi) for fi in file_infos}
if save_tree:
for fi, checksum in checksums.items():
self._save_file(fi, checksum, tree=tree, **kwargs)
else:
checksums = {fi: self.state.get(fi) for fi in file_infos}
not_in_state = {
fi for fi, checksum in checksums.items() if checksum is None
}
new_checksums = self._calculate_checksums(not_in_state)
checksums.update(new_checksums)
new_checksums = self._calculate_checksums(not_in_state)
checksums.update(new_checksums)
result = [
{
@@ -248,18 +259,25 @@ class BaseRemote(object):
# Sorting the list by path to ensure reproducibility
return sorted(result, key=itemgetter(self.PARAM_RELPATH))
def get_dir_checksum(self, path_info):
def get_dir_checksum(self, path_info, tree=None):
if not self.cache:
raise RemoteCacheRequiredError(path_info)
dir_info = self._collect_dir(path_info)
dir_info = self._collect_dir(path_info, tree=None)
if tree:
# don't save state entry for path_info if it is a tree path
path_info = None
return self._save_dir_info(dir_info, path_info)
def _save_dir_info(self, dir_info, path_info=None):
checksum, tmp_info = self._get_dir_info_checksum(dir_info)
new_info = self.cache.checksum_to_path_info(checksum)
if self.cache.changed_cache_file(checksum):
self.cache.makedirs(new_info.parent)
self.cache.move(tmp_info, new_info, mode=self.CACHE_MODE)
self.state.save(path_info, checksum)
if path_info:
self.state.save(path_info, checksum)
self.state.save(new_info, checksum)
return checksum
@@ -319,6 +337,8 @@ class BaseRemote(object):
@classmethod
def is_dir_checksum(cls, checksum):
if not checksum:
return False
return checksum.endswith(cls.CHECKSUM_DIR_SUFFIX)
def get_checksum(self, path_info):
@@ -418,10 +438,10 @@ class BaseRemote(object):
if self.cache_type_confirmed:
return
is_link = getattr(self, "is_{}".format(link_type), None)
is_link = getattr(self, f"is_{link_type}", None)
if is_link and not is_link(path_info):
self.remove(path_info)
raise DvcException("failed to verify {}".format(link_type))
raise DvcException(f"failed to verify {link_type}")
self.cache_type_confirmed = True
@@ -444,7 +464,7 @@ class BaseRemote(object):
def _do_link(self, from_info, to_info, link_method):
if self.exists(to_info):
raise DvcException("Link '{}' already exists!".format(to_info))
raise DvcException(f"Link '{to_info}' already exists!")
link_method(from_info, to_info)
@@ -452,28 +472,45 @@ class BaseRemote(object):
"Created '%s': %s -> %s", self.cache_types[0], from_info, to_info,
)
def _save_file(self, path_info, checksum, save_link=True):
def _save_file(
self, path_info, checksum, save_link=True, tree=None, **kwargs
):
assert checksum
cache_info = self.checksum_to_path_info(checksum)
if self.changed_cache(checksum):
self.move(path_info, cache_info, mode=self.CACHE_MODE)
self.link(cache_info, path_info)
elif self.iscopy(path_info) and self._cache_is_copy(path_info):
# Default relink procedure involves unneeded copy
self.unprotect(path_info)
if tree:
if self.changed_cache(checksum):
with tree.open(path_info, mode="rb") as fobj:
# if tree has fetch enabled, DVC out will be fetched on
# open and we do not need to read/copy any data
if not (
tree.isdvc(path_info, strict=False) and tree.fetch
):
self.copy_fobj(fobj, cache_info)
callback = kwargs.get("download_callback")
if callback:
callback(1)
else:
self.remove(path_info)
self.link(cache_info, path_info)
if self.changed_cache(checksum):
self.move(path_info, cache_info, mode=self.CACHE_MODE)
self.link(cache_info, path_info)
elif self.iscopy(path_info) and self._cache_is_copy(path_info):
# Default relink procedure involves unneeded copy
self.unprotect(path_info)
else:
self.remove(path_info)
self.link(cache_info, path_info)
if save_link:
self.state.save_link(path_info)
if save_link:
self.state.save_link(path_info)
# we need to update path and cache, since in case of reflink,
# or copy cache type moving original file results in updates on
# next executed command, which causes md5 recalculation
self.state.save(path_info, checksum)
if not tree or is_working_tree(tree):
self.state.save(path_info, checksum)
self.state.save(cache_info, checksum)
return {self.PARAM_CHECKSUM: checksum}
def _cache_is_copy(self, path_info):
"""Checks whether cache uses copies."""
@@ -497,22 +534,32 @@ class BaseRemote(object):
self.cache_type_confirmed = True
return self.cache_types[0] == "copy"
def _save_dir(self, path_info, checksum, save_link=True):
def _save_dir(
self, path_info, checksum, save_link=True, tree=None, **kwargs
):
if tree:
dir_info = self._collect_dir(
path_info, tree=tree, save_tree=True, **kwargs
)
checksum = self._save_dir_info(dir_info)
else:
dir_info = self.get_dir_cache(checksum)
for entry in Tqdm(
dir_info, desc="Saving " + path_info.name, unit="file"
):
entry_info = path_info / entry[self.PARAM_RELPATH]
entry_checksum = entry[self.PARAM_CHECKSUM]
self._save_file(entry_info, entry_checksum, save_link=False)
if save_link:
self.state.save_link(path_info)
cache_info = self.checksum_to_path_info(checksum)
dir_info = self.get_dir_cache(checksum)
for entry in Tqdm(
dir_info, desc="Saving " + path_info.name, unit="file"
):
entry_info = path_info / entry[self.PARAM_RELPATH]
entry_checksum = entry[self.PARAM_CHECKSUM]
self._save_file(entry_info, entry_checksum, save_link=False)
if save_link:
self.state.save_link(path_info)
self.state.save(cache_info, checksum)
self.state.save(path_info, checksum)
if not tree or is_working_tree(tree):
self.state.save(path_info, checksum)
return {self.PARAM_CHECKSUM: checksum}
def is_empty(self, path_info):
return False
@@ -541,23 +588,42 @@ class BaseRemote(object):
def protect(path_info):
pass
def save(self, path_info, checksum_info, save_link=True):
def save(
self, path_info, checksum_info, save_link=True, tree=None, **kwargs
):
if path_info.scheme != self.scheme:
raise RemoteActionNotImplemented(
"save {} -> {}".format(path_info.scheme, self.scheme),
self.scheme,
f"save {path_info.scheme} -> {self.scheme}", self.scheme,
)
checksum = checksum_info[self.PARAM_CHECKSUM]
self._save(path_info, checksum, save_link)
if tree:
if tree.isdir(path_info):
# save checksum will be computed during tree walk
checksum = None
else:
checksum = tree.get_file_checksum(path_info)
else:
checksum = checksum_info[self.PARAM_CHECKSUM]
return self._save(path_info, checksum, save_link, tree, **kwargs)
def _save(self, path_info, checksum, save_link=True):
to_info = self.checksum_to_path_info(checksum)
logger.debug("Saving '%s' to '%s'.", path_info, to_info)
if self.isdir(path_info):
self._save_dir(path_info, checksum, save_link)
return
self._save_file(path_info, checksum, save_link)
def _save(self, path_info, checksum, save_link=True, tree=None, **kwargs):
if tree:
logger.debug("Saving tree path '%s' to cache.", path_info)
else:
to_info = self.checksum_to_path_info(checksum)
logger.debug("Saving '%s' to '%s'.", path_info, to_info)
if tree:
isdir = tree.isdir
save_link = False
else:
isdir = self.isdir
if isdir(path_info):
return self._save_dir(
path_info, checksum, save_link, tree, **kwargs
)
return self._save_file(path_info, checksum, save_link, tree, **kwargs)
def _handle_transfer_exception(
self, from_info, to_info, exception, operation
@@ -696,6 +762,9 @@ class BaseRemote(object):
def copy(self, from_info, to_info):
raise RemoteActionNotImplemented("copy", self.scheme)
def copy_fobj(self, fobj, to_info):
raise RemoteActionNotImplemented("copy_fobj", self.scheme)
def symlink(self, from_info, to_info):
raise RemoteActionNotImplemented("symlink", self.scheme)
@@ -712,7 +781,7 @@ class BaseRemote(object):
parts = self.path_cls(path).parts[-2:]
if not (len(parts) == 2 and parts[0] and len(parts[0]) == 2):
raise ValueError("Bad cache file path '{}'".format(path))
raise ValueError(f"Bad cache file path '{path}'")
return "".join(parts)
@@ -762,7 +831,7 @@ class BaseRemote(object):
@index_locked
def gc(self, named_cache, jobs=None):
used = self.extract_used_local_checksums(named_cache)
used = set(named_cache.scheme_keys("local"))
if self.scheme != "":
used.update(named_cache.scheme_keys(self.scheme))
@@ -778,6 +847,7 @@ class BaseRemote(object):
continue
path_info = self.checksum_to_path_info(checksum)
if self.is_dir_checksum(checksum):
# backward compatibility
self._remove_unpacked_dir(checksum)
self.remove(path_info)
removed = True
@@ -836,11 +906,6 @@ class BaseRemote(object):
if self.changed_cache_file(checksum):
return True
if not (path_info and filter_info) and not self._changed_unpacked_dir(
checksum
):
return False
for entry in self.get_dir_cache(checksum):
entry_checksum = entry[self.PARAM_CHECKSUM]
@@ -852,9 +917,6 @@ class BaseRemote(object):
if self.changed_cache_file(entry_checksum):
return True
if not (path_info and filter_info):
self._update_unpacked_dir(checksum)
return False
def changed_cache(self, checksum, path_info=None, filter_info=None):
@@ -991,7 +1053,7 @@ class BaseRemote(object):
with Tqdm(
desc="Estimating size of "
+ ("cache in '{}'".format(name) if name else "remote cache"),
+ (f"cache in '{name}'" if name else "remote cache"),
unit="file",
) as pbar:
@@ -1010,7 +1072,7 @@ class BaseRemote(object):
remote_size = total_prefixes * len(remote_checksums)
else:
remote_size = total_prefixes
logger.debug("Estimated remote size: {} files".format(remote_size))
logger.debug(f"Estimated remote size: {remote_size} files")
return remote_size, remote_checksums
def _cache_checksums_traverse(
@@ -1043,7 +1105,7 @@ class BaseRemote(object):
else:
yield from remote_checksums
initial = len(remote_checksums)
traverse_prefixes = ["{:02x}".format(i) for i in range(1, 256)]
traverse_prefixes = [f"{i:02x}" for i in range(1, 256)]
if self.TRAVERSE_PREFIX_LEN > 2:
traverse_prefixes += [
"{0:0{1}x}".format(i, self.TRAVERSE_PREFIX_LEN)
@@ -1051,7 +1113,7 @@ class BaseRemote(object):
]
with Tqdm(
desc="Querying "
+ ("cache in '{}'".format(name) if name else "remote cache"),
+ (f"cache in '{name}'" if name else "remote cache"),
total=remote_size,
initial=initial,
unit="file",
@@ -1081,7 +1143,7 @@ class BaseRemote(object):
def exists_with_progress(path_info):
ret = self.exists(path_info)
pbar.update_desc(str(path_info))
pbar.update_msg(str(path_info))
return ret
with ThreadPoolExecutor(max_workers=jobs or self.JOBS) as executor:
@@ -1293,19 +1355,5 @@ class BaseRemote(object):
def unprotect(path_info):
pass
def _get_unpacked_dir_names(self, checksums):
return set()
def extract_used_local_checksums(self, named_cache):
used = set(named_cache.scheme_keys("local"))
unpacked = self._get_unpacked_dir_names(used)
return used | unpacked
def _changed_unpacked_dir(self, checksum):
return True
def _update_unpacked_dir(self, checksum):
pass
def _remove_unpacked_dir(self, checksum):
pass
+24 -36
View File
@@ -1,20 +1,20 @@
from collections import defaultdict
import logging
import os
import posixpath
import logging
import re
import threading
from collections import defaultdict
from urllib.parse import urlparse
from funcy import retry, wrap_with, wrap_prop, cached_property
from funcy import cached_property, retry, wrap_prop, wrap_with
from funcy.py3 import cat
from dvc.progress import Tqdm
from dvc.scheme import Schemes
from dvc.path_info import CloudURLInfo
from dvc.remote.base import BaseRemote
from dvc.exceptions import DvcException
from dvc.utils import tmp_fname, format_link
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.scheme import Schemes
from dvc.utils import format_link, tmp_fname
logger = logging.getLogger(__name__)
FOLDER_MIME_TYPE = "application/vnd.google-apps.folder"
@@ -22,10 +22,8 @@ FOLDER_MIME_TYPE = "application/vnd.google-apps.folder"
class GDrivePathNotFound(DvcException):
def __init__(self, path_info, hint):
hint = "" if hint is None else " {}".format(hint)
super().__init__(
"GDrive path '{}' not found.{}".format(path_info, hint)
)
hint = "" if hint is None else f" {hint}"
super().__init__(f"GDrive path '{path_info}' not found.{hint}")
class GDriveAuthError(DvcException):
@@ -44,17 +42,6 @@ class GDriveAuthError(DvcException):
super().__init__(message)
def _extract(exc, field):
from pydrive2.files import ApiRequestError
assert isinstance(exc, ApiRequestError)
# https://cloud.google.com/storage/docs/json_api/v1/status-codes#errorformat
return (
exc.error["errors"][0].get(field, "") if "errors" in exc.error else ""
)
def _gdrive_retry(func):
def should_retry(exc):
from pydrive2.files import ApiRequestError
@@ -68,12 +55,12 @@ def _gdrive_retry(func):
result = True
if error_code == 403:
result = _extract(exc, "reason") in [
result = exc.GetField("reason") in [
"userRateLimitExceeded",
"rateLimitExceeded",
]
if result:
logger.debug("Retrying GDrive API call, error: {}.".format(exc))
logger.debug(f"Retrying GDrive API call, error: {exc}.")
return result
@@ -396,14 +383,15 @@ class GDriveRemote(BaseRemote):
param = {"id": item_id}
# it does not create a file on the remote
gdrive_file = self._drive.CreateFile(param)
bar_format = (
"Downloading {desc:{ncols_desc}.{ncols_desc}}... "
+ Tqdm.format_sizeof(int(gdrive_file["fileSize"]), "B", 1024)
)
with Tqdm(
bar_format=bar_format, desc=progress_desc, disable=no_progress_bar
):
gdrive_file.GetContentFile(to_file)
desc=progress_desc,
disable=no_progress_bar,
bytes=True,
# explicit `bar_format` as `total` will be set by `update_to`
bar_format=Tqdm.BAR_FMT_DEFAULT,
) as pbar:
gdrive_file.GetContentFile(to_file, callback=pbar.update_to)
@_gdrive_retry
def _gdrive_delete_file(self, item_id):
@@ -420,7 +408,7 @@ class GDriveRemote(BaseRemote):
if (
http_error_code == 403
and self._list_params["corpora"] == "drive"
and _extract(exc, "location") == "file.permissions"
and exc.GetField("location") == "file.permissions"
):
raise DvcException(
"Insufficient permissions to {}. You should have {} "
@@ -475,7 +463,7 @@ class GDriveRemote(BaseRemote):
return None
query = "trashed=false and ({})".format(
" or ".join(
"'{}' in parents".format(parent_id) for parent_id in parent_ids
f"'{parent_id}' in parents" for parent_id in parent_ids
)
)
query += " and title='{}'".format(title.replace("'", "\\'"))
@@ -552,9 +540,9 @@ class GDriveRemote(BaseRemote):
else:
dir_ids = self._ids_cache["ids"]
parents_query = " or ".join(
"'{}' in parents".format(dir_id) for dir_id in dir_ids
f"'{dir_id}' in parents" for dir_id in dir_ids
)
query = "({}) and trashed=false".format(parents_query)
query = f"({parents_query}) and trashed=false"
for item in self._gdrive_list(query):
if progress_callback:
+6 -7
View File
@@ -1,10 +1,9 @@
import logging
from datetime import timedelta
from functools import wraps
import io
import os.path
import posixpath
import threading
from datetime import timedelta
from functools import wraps
from funcy import cached_property, wrap_prop
@@ -55,7 +54,7 @@ def _upload_to_bucket(
no_progress_bar=False,
):
blob = bucket.blob(to_info.path, chunk_size=chunk_size)
with io.open(from_file, mode="rb") as fobj:
with open(from_file, mode="rb") as fobj:
with Tqdm.wrapattr(
fobj,
"read",
@@ -110,7 +109,7 @@ class GSRemote(BaseRemote):
from_bucket = self.gs.bucket(from_info.bucket)
blob = from_bucket.get_blob(from_info.path)
if not blob:
msg = "'{}' doesn't exist in the cloud".format(from_info.path)
msg = f"'{from_info.path}' doesn't exist in the cloud"
raise DvcException(msg)
to_bucket = self.gs.bucket(to_info.bucket)
@@ -120,7 +119,7 @@ class GSRemote(BaseRemote):
if path_info.scheme != "gs":
raise NotImplementedError
logger.debug("Removing gs://{}".format(path_info))
logger.debug(f"Removing gs://{path_info}")
blob = self.gs.bucket(path_info.bucket).get_blob(path_info.path)
if not blob:
return
@@ -193,7 +192,7 @@ class GSRemote(BaseRemote):
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
bucket = self.gs.bucket(from_info.bucket)
blob = bucket.get_blob(from_info.path)
with io.open(to_file, mode="wb") as fobj:
with open(to_file, mode="wb") as fobj:
with Tqdm.wrapattr(
fobj,
"write",
+10 -7
View File
@@ -8,11 +8,12 @@ from collections import deque
from contextlib import closing, contextmanager
from urllib.parse import urlparse
from .base import BaseRemote, RemoteCmdError
from .pool import get_connection
from dvc.scheme import Schemes
from dvc.utils import fix_env, tmp_fname
from .base import BaseRemote, RemoteCmdError
from .pool import get_connection
logger = logging.getLogger(__name__)
@@ -54,7 +55,7 @@ class HDFSRemote(BaseRemote):
def hadoop_fs(self, cmd, user=None):
cmd = "hadoop fs -" + cmd
if user:
cmd = "HADOOP_USER_NAME={} ".format(user) + cmd
cmd = f"HADOOP_USER_NAME={user} " + cmd
# NOTE: close_fds doesn't work with redirected stdin/stdout/stderr.
# See https://github.com/iterative/dvc/issues/1197.
@@ -86,7 +87,7 @@ class HDFSRemote(BaseRemote):
# NOTE: pyarrow doesn't support checksum, so we need to use hadoop
regex = r".*\t.*\t(?P<checksum>.*)"
stdout = self.hadoop_fs(
"checksum {}".format(path_info.path), user=path_info.user
f"checksum {path_info.path}", user=path_info.user
)
return self._group(regex, stdout, "checksum")
@@ -111,7 +112,7 @@ class HDFSRemote(BaseRemote):
raise NotImplementedError
if self.exists(path_info):
logger.debug("Removing {}".format(path_info.path))
logger.debug(f"Removing {path_info.path}")
with self.hdfs(path_info) as hdfs:
hdfs.rm(path_info.path)
@@ -146,7 +147,7 @@ class HDFSRemote(BaseRemote):
yield fd
else:
yield io.TextIOWrapper(fd, encoding=encoding)
except IOError as e:
except OSError as e:
# Empty .errno and not specific enough error class in pyarrow,
# see https://issues.apache.org/jira/browse/ARROW-6248
if "file does not exist" in str(e):
@@ -164,6 +165,8 @@ class HDFSRemote(BaseRemote):
dirs = deque([root])
with self.hdfs(self.path_info) as hdfs:
if prefix and not hdfs.exists(root):
return
while dirs:
try:
entries = hdfs.ls(dirs.pop(), detail=True)
@@ -174,7 +177,7 @@ class HDFSRemote(BaseRemote):
if progress_callback:
progress_callback()
yield urlparse(entry["name"]).path
except IOError as e:
except OSError as e:
# When searching for a specific prefix pyarrow raises an
# exception if the specified cache dir does not exist
if not prefix:
+2 -2
View File
@@ -4,9 +4,9 @@ import threading
from funcy import cached_property, memoize, wrap_prop, wrap_with
from dvc.path_info import HTTPURLInfo
import dvc.prompt as prompt
from dvc.exceptions import DvcException, HTTPError
from dvc.path_info import HTTPURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.scheme import Schemes
@@ -185,7 +185,7 @@ class HTTPRemote(BaseRemote):
return res
except requests.exceptions.RequestException:
raise DvcException("could not perform a {} request".format(method))
raise DvcException(f"could not perform a {method} request")
def gc(self):
raise NotImplementedError
+2 -1
View File
@@ -1,6 +1,7 @@
from .http import HTTPRemote
from dvc.scheme import Schemes
from .http import HTTPRemote
class HTTPSRemote(HTTPRemote):
scheme = Schemes.HTTPS
+4 -6
View File
@@ -69,9 +69,7 @@ class RemoteIndex:
INDEX_TABLE_LAYOUT = "checksum TEXT PRIMARY KEY, " "dir INTEGER NOT NULL"
def __init__(self, repo, name, dir_suffix=".dir"):
self.path = os.path.join(
repo.index_dir, "{}{}".format(name, self.INDEX_SUFFIX)
)
self.path = os.path.join(repo.index_dir, f"{name}{self.INDEX_SUFFIX}")
self.dir_suffix = dir_suffix
self.database = None
@@ -80,7 +78,7 @@ class RemoteIndex:
self.lock = threading.Lock()
def __iter__(self):
cmd = "SELECT checksum FROM {}".format(self.INDEX_TABLE)
cmd = f"SELECT checksum FROM {self.INDEX_TABLE}"
for (checksum,) in self._execute(cmd):
yield checksum
@@ -105,7 +103,7 @@ class RemoteIndex:
def dir_checksums(self):
"""Iterate over .dir checksums stored in the index."""
cmd = "SELECT checksum FROM {} WHERE dir = 1".format(self.INDEX_TABLE)
cmd = f"SELECT checksum FROM {self.INDEX_TABLE} WHERE dir = 1"
for (checksum,) in self._execute(cmd):
yield checksum
@@ -184,7 +182,7 @@ class RemoteIndex:
Changes to the index will not committed until dump() is called.
"""
cmd = "DELETE FROM {}".format(self.INDEX_TABLE)
cmd = f"DELETE FROM {self.INDEX_TABLE}"
self._execute(cmd)
def update(self, dir_checksums, file_checksums):
+89 -99
View File
@@ -2,31 +2,36 @@ import errno
import logging
import os
import stat
from concurrent.futures import as_completed, ThreadPoolExecutor
from concurrent.futures import ThreadPoolExecutor, as_completed
from functools import partial
from funcy import cached_property, concat
from shortuuid import uuid
from dvc.compat import fspath_py35
from dvc.exceptions import DvcException, DownloadError, UploadError
from dvc.exceptions import DownloadError, DvcException, UploadError
from dvc.path_info import PathInfo
from dvc.progress import Tqdm
from dvc.remote.base import (
index_locked,
BaseRemote,
STATUS_MAP,
STATUS_DELETED,
STATUS_MAP,
STATUS_MISSING,
STATUS_NEW,
BaseRemote,
index_locked,
)
from dvc.remote.index import RemoteIndexNoop
from dvc.scheme import Schemes
from dvc.scm.tree import is_working_tree
from dvc.scm.tree import WorkingTree, is_working_tree
from dvc.system import System
from dvc.utils import file_md5, relpath, tmp_fname
from dvc.utils.fs import copyfile, move, makedirs, remove, walk_files
from dvc.utils.fs import (
copy_fobj_to_file,
copyfile,
makedirs,
move,
remove,
walk_files,
)
logger = logging.getLogger(__name__)
@@ -63,6 +68,21 @@ class LocalRemote(BaseRemote):
def cache_dir(self, value):
self.path_info = PathInfo(value) if value else None
@cached_property
def _work_tree(self):
if self.repo:
return WorkingTree(self.repo.root_dir)
return None
@property
def work_tree(self):
# When using repo.brancher, repo.tree may change to/from WorkingTree to
# GitTree arbitarily. When repo.tree is GitTree, local cache needs to
# use its own WorkingTree instance.
if self.repo and not is_working_tree(self.repo.tree):
return self._work_tree
return None
@classmethod
def supported(cls, config):
return True
@@ -72,12 +92,19 @@ class LocalRemote(BaseRemote):
return os.path.abspath(self.cache_dir)
def checksum_to_path(self, checksum):
return os.path.join(self.cache_path, checksum[0:2], checksum[2:])
# NOTE: `self.cache_path` is already normalized so we can simply use
# `os.sep` instead of `os.path.join`. This results in this helper
# being ~5.5 times faster.
return (
f"{self.cache_path}{os.sep}{checksum[0:2]}{os.sep}{checksum[2:]}"
)
def list_cache_paths(self, prefix=None, progress_callback=None):
assert self.path_info is not None
if prefix:
path_info = self.path_info / prefix[:2]
if not self.exists(path_info):
return
else:
path_info = self.path_info
if progress_callback:
@@ -94,9 +121,12 @@ class LocalRemote(BaseRemote):
return self.checksum_to_path_info(md5).url
def exists(self, path_info):
assert is_working_tree(self.repo.tree)
assert isinstance(path_info, str) or path_info.scheme == "local"
return self.repo.tree.exists(fspath_py35(path_info))
if not self.repo:
return os.path.exists(path_info)
if self.work_tree and self.work_tree.exists(path_info):
return True
return self.repo.tree.exists(path_info)
def makedirs(self, path_info):
makedirs(path_info, exist_ok=True, mode=self._dir_mode)
@@ -128,13 +158,19 @@ class LocalRemote(BaseRemote):
return False
@staticmethod
def isfile(path_info):
return os.path.isfile(fspath_py35(path_info))
def isfile(self, path_info):
if not self.repo:
return os.path.isfile(path_info)
if self.work_tree and self.work_tree.isfile(path_info):
return True
return self.repo.tree.isfile(path_info)
@staticmethod
def isdir(path_info):
return os.path.isdir(fspath_py35(path_info))
def isdir(self, path_info):
if not self.repo:
return os.path.isdir(path_info)
if self.work_tree and self.work_tree.isdir(path_info):
return True
return self.repo.tree.isdir(path_info)
def iscopy(self, path_info):
return not (
@@ -143,12 +179,14 @@ class LocalRemote(BaseRemote):
@staticmethod
def getsize(path_info):
return os.path.getsize(fspath_py35(path_info))
return os.path.getsize(path_info)
def walk_files(self, path_info):
assert is_working_tree(self.repo.tree)
for fname in self.repo.tree.walk_files(path_info):
if self.work_tree:
tree = self.work_tree
else:
tree = self.repo.tree
for fname in tree.walk_files(path_info):
yield PathInfo(fname)
def get_file_checksum(self, path_info):
@@ -183,8 +221,19 @@ class LocalRemote(BaseRemote):
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
System.copy(from_info, tmp_info)
os.chmod(fspath_py35(tmp_info), self._file_mode)
os.rename(fspath_py35(tmp_info), fspath_py35(to_info))
os.chmod(tmp_info, self._file_mode)
os.rename(tmp_info, to_info)
except Exception:
self.remove(tmp_info)
raise
def copy_fobj(self, fobj, to_info):
self.makedirs(to_info.parent)
tmp_info = to_info.parent / tmp_fname(to_info.name)
try:
copy_fobj_to_file(fobj, tmp_info)
os.chmod(tmp_info, self._file_mode)
os.rename(tmp_info, to_info)
except Exception:
self.remove(tmp_info)
raise
@@ -232,8 +281,8 @@ class LocalRemote(BaseRemote):
System.reflink(from_info, tmp_info)
# NOTE: reflink has its own separate inode, so you can set permissions
# that are different from the source.
os.chmod(fspath_py35(tmp_info), self._file_mode)
os.rename(fspath_py35(tmp_info), fspath_py35(to_info))
os.chmod(tmp_info, self._file_mode)
os.rename(tmp_info, to_info)
def cache_exists(self, checksums, jobs=None, name=None):
return [
@@ -256,7 +305,9 @@ class LocalRemote(BaseRemote):
copyfile(
from_file, tmp_file, name=name, no_progress_bar=no_progress_bar
)
os.rename(tmp_file, fspath_py35(to_info))
self.protect(tmp_file)
os.rename(tmp_file, to_info)
def _download(
self, from_info, to_file, name=None, no_progress_bar=False, **_kwargs
@@ -267,7 +318,7 @@ class LocalRemote(BaseRemote):
@staticmethod
def open(path_info, mode="r", encoding=None):
return open(fspath_py35(path_info), mode=mode, encoding=encoding)
return open(path_info, mode=mode, encoding=encoding)
@index_locked
def status(
@@ -303,9 +354,7 @@ class LocalRemote(BaseRemote):
{dir_checksum: set(file_checksum, ...)} which can be used to map
a .dir file to its file contents.
"""
logger.debug(
"Preparing to collect status from {}".format(remote.path_info)
)
logger.debug(f"Preparing to collect status from {remote.path_info}")
md5s = set(named_cache.scheme_keys(self.scheme))
logger.debug("Collecting information from local cache...")
@@ -551,9 +600,7 @@ class LocalRemote(BaseRemote):
"failed to upload full contents of '{}', "
"aborting .dir file upload".format(name)
)
logger.error(
"failed to upload '{}' to '{}'".format(from_info, to_info)
)
logger.error(f"failed to upload '{from_info}' to '{to_info}'")
return 1
return func(from_info, to_info, name)
@@ -597,7 +644,7 @@ class LocalRemote(BaseRemote):
def _unprotect_file(self, path):
if System.is_symlink(path) or System.is_hardlink(path):
logger.debug("Unprotecting '{}'".format(path))
logger.debug(f"Unprotecting '{path}'")
tmp = os.path.join(os.path.dirname(path), "." + uuid())
# The operations order is important here - if some application
@@ -626,9 +673,7 @@ class LocalRemote(BaseRemote):
def unprotect(self, path_info):
path = path_info.fspath
if not os.path.exists(path):
raise DvcException(
"can't unprotect non-existing data '{}'".format(path)
)
raise DvcException(f"can't unprotect non-existing data '{path}'")
if os.path.isdir(path):
self._unprotect_dir(path)
@@ -636,7 +681,7 @@ class LocalRemote(BaseRemote):
self._unprotect_file(path)
def protect(self, path_info):
path = fspath_py35(path_info)
path = os.fspath(path_info)
mode = self.CACHE_MODE
try:
@@ -655,70 +700,15 @@ class LocalRemote(BaseRemote):
if actual != mode:
raise
def _get_unpacked_dir_path_info(self, checksum):
info = self.checksum_to_path_info(checksum)
return info.with_name(info.name + self.UNPACKED_DIR_SUFFIX)
def _remove_unpacked_dir(self, checksum):
path_info = self._get_unpacked_dir_path_info(checksum)
info = self.checksum_to_path_info(checksum)
path_info = info.with_name(info.name + self.UNPACKED_DIR_SUFFIX)
self.remove(path_info)
def _path_info_changed(self, path_info):
if self.exists(path_info) and self.state.get(path_info):
return False
return True
def _update_unpacked_dir(self, checksum):
unpacked_dir_info = self._get_unpacked_dir_path_info(checksum)
if not self._path_info_changed(unpacked_dir_info):
return
self.remove(unpacked_dir_info)
try:
dir_info = self.get_dir_cache(checksum)
self._create_unpacked_dir(checksum, dir_info, unpacked_dir_info)
except DvcException:
logger.warning("Could not create '{}'".format(unpacked_dir_info))
self.remove(unpacked_dir_info)
def _create_unpacked_dir(self, checksum, dir_info, unpacked_dir_info):
self.makedirs(unpacked_dir_info)
for entry in Tqdm(dir_info, desc="Creating unpacked dir", unit="file"):
entry_cache_info = self.checksum_to_path_info(
entry[self.PARAM_CHECKSUM]
)
relative_path = entry[self.PARAM_RELPATH]
# In shared cache mode some cache files might not be owned by the
# user, so we need to use symlinks because, unless
# /proc/sys/fs/protected_hardlinks is disabled, the user is not
# allowed to create hardlinks to files that he doesn't own.
link_types = ["hardlink", "symlink"]
self._link(
entry_cache_info, unpacked_dir_info / relative_path, link_types
)
self.state.save(unpacked_dir_info, checksum)
def _changed_unpacked_dir(self, checksum):
status_unpacked_dir_info = self._get_unpacked_dir_path_info(checksum)
return not self.state.get(status_unpacked_dir_info)
def _get_unpacked_dir_names(self, checksums):
unpacked = set()
for c in checksums:
if self.is_dir_checksum(c):
unpacked.add(c + self.UNPACKED_DIR_SUFFIX)
return unpacked
def is_protected(self, path_info):
if not self.exists(path_info):
try:
mode = os.stat(path_info).st_mode
except FileNotFoundError:
return False
mode = os.stat(fspath_py35(path_info)).st_mode
return stat.S_IMODE(mode) == self.CACHE_MODE
+4 -5
View File
@@ -10,7 +10,6 @@ from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
from dvc.scheme import Schemes
logger = logging.getLogger(__name__)
@@ -65,9 +64,9 @@ class OSSRemote(BaseRemote):
def oss_service(self):
import oss2
logger.debug("URL: {}".format(self.path_info))
logger.debug("key id: {}".format(self.key_id))
logger.debug("key secret: {}".format(self.key_secret))
logger.debug(f"URL: {self.path_info}")
logger.debug(f"key id: {self.key_id}")
logger.debug(f"key secret: {self.key_secret}")
auth = oss2.Auth(self.key_id, self.key_secret)
bucket = oss2.Bucket(auth, self.endpoint, self.path_info.bucket)
@@ -88,7 +87,7 @@ class OSSRemote(BaseRemote):
if path_info.scheme != self.scheme:
raise NotImplementedError
logger.debug("Removing oss://{}".format(path_info))
logger.debug(f"Removing oss://{path_info}")
self.oss_service.delete_object(path_info.path)
def _list_paths(self, prefix, progress_callback=None):
+2 -2
View File
@@ -1,6 +1,6 @@
import threading
from collections import deque
from contextlib import contextmanager
import threading
from funcy import memoize, wrap_with
@@ -30,7 +30,7 @@ def close_pools():
get_pool.memory.clear()
class Pool(object):
class Pool:
def __init__(self, conn_func, *conn_args, **conn_kwargs):
self._conn_func = conn_func
self._conn_args = conn_args
+8 -9
View File
@@ -1,5 +1,3 @@
# -*- coding: utf-8 -*-
import logging
import os
import posixpath
@@ -8,8 +6,7 @@ import threading
from funcy import cached_property, wrap_prop
from dvc.config import ConfigError
from dvc.exceptions import DvcException
from dvc.exceptions import ETagMismatchError
from dvc.exceptions import DvcException, ETagMismatchError
from dvc.path_info import CloudURLInfo
from dvc.progress import Tqdm
from dvc.remote.base import BaseRemote
@@ -47,6 +44,10 @@ class S3Remote(BaseRemote):
if self.sse:
self.extra_args["ServerSideEncryption"] = self.sse
self.sse_kms_key_id = config.get("sse_kms_key_id")
if self.sse_kms_key_id:
self.extra_args["SSEKMSKeyId"] = self.sse_kms_key_id
self.acl = config.get("acl")
if self.acl:
self.extra_args["ACL"] = self.acl
@@ -85,9 +86,7 @@ class S3Remote(BaseRemote):
try:
obj = s3.head_object(Bucket=bucket, Key=path, *args, **kwargs)
except Exception as exc:
raise DvcException(
"s3://{}/{} does not exist".format(bucket, path)
) from exc
raise DvcException(f"s3://{bucket}/{path} does not exist") from exc
return obj
@classmethod
@@ -110,7 +109,7 @@ class S3Remote(BaseRemote):
if lastbyte > size:
lastbyte = size - 1
srange = "bytes={}-{}".format(byte_position, lastbyte)
srange = f"bytes={byte_position}-{lastbyte}"
part = s3.upload_part_copy(
Bucket=to_info.bucket,
@@ -188,7 +187,7 @@ class S3Remote(BaseRemote):
if path_info.scheme != "s3":
raise NotImplementedError
logger.debug("Removing {}".format(path_info))
logger.debug(f"Removing {path_info}")
self.s3.delete_object(Bucket=path_info.bucket, Key=path_info.path)
def _list_objects(
+1 -2
View File
@@ -1,10 +1,9 @@
import logging
import sys
import time
from functools import wraps
import colorama
import colorama
logger = logging.getLogger(__name__)
this = sys.modules[__name__]
+5 -3
View File
@@ -10,7 +10,7 @@ from concurrent.futures import ThreadPoolExecutor
from contextlib import closing, contextmanager
from urllib.parse import urlparse
from funcy import memoize, wrap_with, silent, first
from funcy import first, memoize, silent, wrap_with
import dvc.prompt as prompt
from dvc.progress import Tqdm
@@ -265,6 +265,8 @@ class SSHRemote(BaseRemote):
else:
root = self.path_info.path
with self.ssh(self.path_info) as ssh:
if prefix and not ssh.exists(root):
return
# If we simply return an iterator then with above closes instantly
if progress_callback:
for path in ssh.walk_files(root):
@@ -290,7 +292,7 @@ class SSHRemote(BaseRemote):
try:
channel.stat(path)
ret.append(True)
except IOError as exc:
except OSError as exc:
if exc.errno != errno.ENOENT:
raise
ret.append(False)
@@ -330,7 +332,7 @@ class SSHRemote(BaseRemote):
) as pbar:
def exists_with_progress(chunks):
return self.batch_exists(chunks, callback=pbar.update_desc)
return self.batch_exists(chunks, callback=pbar.update_msg)
with ThreadPoolExecutor(max_workers=jobs or self.JOBS) as executor:
path_infos = [self.checksum_to_path_info(x) for x in checksums]
+17 -19
View File
@@ -7,16 +7,16 @@ from contextlib import suppress
from funcy import cached_property
from dvc.exceptions import DvcException
from dvc.progress import Tqdm
from dvc.remote.base import RemoteCmdError
from dvc.utils import tmp_fname
try:
import paramiko
except ImportError:
paramiko = None
from dvc.utils import tmp_fname
from dvc.progress import Tqdm
from dvc.exceptions import DvcException
from dvc.remote.base import RemoteCmdError
logger = logging.getLogger(__name__)
@@ -25,9 +25,9 @@ def sizeof_fmt(num, suffix="B"):
""" Convert number of bytes to human-readable string """
for unit in ["", "K", "M", "G", "T", "P", "E", "Z"]:
if abs(num) < 1024.0:
return "%3.1f%s%s" % (num, unit, suffix)
return f"{num:3.1f}{unit}{suffix}"
num /= 1024.0
return "%.1f%s%s" % (num, "Y", suffix)
return "{:.1f}{}{}".format(num, "Y", suffix)
class SSHConnection:
@@ -100,7 +100,7 @@ class SSHConnection:
if stat.S_ISREG(st_mode) or stat.S_ISLNK(st_mode):
raise DvcException(
"a file with the same name '{}' already exists".format(path)
f"a file with the same name '{path}' already exists"
)
head, tail = posixpath.split(path)
@@ -111,12 +111,12 @@ class SSHConnection:
if tail:
try:
self.sftp.mkdir(path)
except IOError as exc:
except OSError as exc:
# Since paramiko errors are very vague we need to recheck
# whether it's because path already exists or something else
if exc.errno == errno.EACCES or not self.exists(path):
raise DvcException(
"unable to create remote directory '{}'".format(path)
f"unable to create remote directory '{path}'"
) from exc
def walk(self, directory, topdown=True):
@@ -126,7 +126,7 @@ class SSHConnection:
# [1] https://github.com/python/cpython/blob/master/Lib/os.py
try:
dir_entries = self.sftp.listdir_attr(directory)
except IOError as exc:
except OSError as exc:
raise DvcException(
"couldn't get the '{}' remote directory files list".format(
directory
@@ -292,14 +292,14 @@ class SSHConnection:
md5 = self.execute("md5 " + path).split()[-1]
else:
raise DvcException(
"'{}' is not supported as a SSH remote".format(self.uname)
f"'{self.uname}' is not supported as a SSH remote"
)
assert len(md5) == 32
return md5
def copy(self, src, dest):
self.execute("cp {} {}".format(src, dest))
self.execute(f"cp {src} {dest}")
def open_max_sftp_channels(self):
# If there are more than 1 it means we've already opened max amount
@@ -321,14 +321,12 @@ class SSHConnection:
def reflink(self, src, dest):
if self.uname == "Linux":
return self.execute("cp --reflink {} {}".format(src, dest))
return self.execute(f"cp --reflink {src} {dest}")
if self.uname == "Darwin":
return self.execute("cp -c {} {}".format(src, dest))
return self.execute(f"cp -c {src} {dest}")
raise DvcException(
"'{}' is not supported as a SSH remote".format(self.uname)
)
raise DvcException(f"'{self.uname}' is not supported as a SSH remote")
def hardlink(self, src, dest):
self.execute("ln {} {}".format(src, dest))
self.execute(f"ln {src} {dest}")
+162 -102
View File
@@ -3,23 +3,27 @@ import os
from contextlib import contextmanager
from functools import wraps
from dvc.ignore import CleanTree
from dvc.compat import fspath_py35
from funcy import cached_property, cat, first
from dvc.config import Config
from dvc.dvcfile import PIPELINE_FILE, Dvcfile, is_valid_filename
from dvc.exceptions import FileMissingError
from dvc.exceptions import IsADirectoryError as DvcIsADirectoryError
from dvc.exceptions import (
FileMissingError,
IsADirectoryError,
NoOutputOrStageError,
NotDvcRepoError,
OutputNotFoundError,
)
from dvc.ignore import CleanTree
from dvc.path_info import PathInfo
from dvc.remote.base import RemoteActionNotImplemented
from dvc.repo.tree import RepoTree
from dvc.utils.fs import path_isin
from .graph import check_acyclic, get_pipeline, get_pipelines
from ..stage.exceptions import StageFileDoesNotExistError, StageNotFound
from ..utils import parse_target
from .graph import check_acyclic, get_pipeline, get_pipelines
logger = logging.getLogger(__name__)
def locked(f):
@@ -35,7 +39,7 @@ def locked(f):
return wrapper
class Repo(object):
class Repo:
DVC_DIR = ".dvc"
from dvc.repo.destroy import destroy
@@ -62,28 +66,37 @@ class Repo(object):
from dvc.repo.get_url import get_url
from dvc.repo.update import update
def __init__(self, root_dir=None):
from dvc.state import State
def __init__(self, root_dir=None, scm=None, rev=None):
from dvc.state import State, StateNoop
from dvc.lock import make_lock
from dvc.scm import SCM
from dvc.cache import Cache
from dvc.data_cloud import DataCloud
from dvc.repo.metrics import Metrics
from dvc.repo.plots import Plots
from dvc.repo.params import Params
from dvc.scm.tree import WorkingTree
from dvc.utils.fs import makedirs
from dvc.stage.cache import StageCache
root_dir = self.find_root(root_dir)
if scm:
# use GitTree instead of WorkingTree as default repo tree instance
tree = scm.get_tree(rev)
self.root_dir = self.find_root(root_dir, tree)
self.scm = scm
self.tree = tree
self.state = StateNoop()
else:
root_dir = self.find_root(root_dir)
self.root_dir = os.path.abspath(os.path.realpath(root_dir))
self.root_dir = os.path.abspath(os.path.realpath(root_dir))
self.dvc_dir = os.path.join(self.root_dir, self.DVC_DIR)
self.config = Config(self.dvc_dir)
no_scm = self.config["core"].get("no_scm", False)
self.scm = SCM(self.root_dir, no_scm=no_scm)
self.tree = WorkingTree(self.root_dir)
if not scm:
no_scm = self.config["core"].get("no_scm", False)
self.scm = SCM(self.root_dir, no_scm=no_scm)
self.tree = WorkingTree(self.root_dir)
self.tmp_dir = os.path.join(self.dvc_dir, "tmp")
self.index_dir = os.path.join(self.tmp_dir, "index")
@@ -91,20 +104,25 @@ class Repo(object):
hardlink_lock = self.config["core"].get("hardlink_lock", False)
self.lock = make_lock(
os.path.join(self.dvc_dir, "lock"),
tmp_dir=os.path.join(self.dvc_dir, "tmp"),
os.path.join(self.tmp_dir, "lock"),
tmp_dir=self.tmp_dir,
hardlink_lock=hardlink_lock,
friendly=True,
)
# NOTE: storing state and link_state in the repository itself to avoid
# any possible state corruption in 'shared cache dir' scenario.
self.state = State(self)
if not scm:
# NOTE: storing state and link_state in the repository itself to
# avoid any possible state corruption in 'shared cache dir'
# scenario.
self.state = State(self)
self.cache = Cache(self)
self.cloud = DataCloud(self)
self.stage_cache = StageCache(self)
self.metrics = Metrics(self)
self.plots = Plots(self)
self.params = Params(self)
self._ignore()
@@ -121,14 +139,19 @@ class Repo(object):
self._reset()
def __repr__(self):
return "{}: '{}'".format(self.__class__.__name__, self.root_dir)
return f"{self.__class__.__name__}: '{self.root_dir}'"
@classmethod
def find_root(cls, root=None):
def find_root(cls, root=None, tree=None):
root_dir = os.path.realpath(root or os.curdir)
if tree:
if tree.isdir(os.path.join(root_dir, cls.DVC_DIR)):
return root_dir
raise NotDvcRepoError(f"'{root}' does not contain DVC directory")
if not os.path.isdir(root_dir):
raise NotDvcRepoError("directory '{}' does not exist".format(root))
raise NotDvcRepoError(f"directory '{root}' does not exist")
while True:
dvc_dir = os.path.join(root_dir, cls.DVC_DIR)
@@ -160,21 +183,32 @@ class Repo(object):
return self.cache.local.unprotect(PathInfo(target))
def _ignore(self):
from dvc.updater import Updater
updater = Updater(self.dvc_dir)
flist = (
[self.config.files["local"], updater.updater_file]
+ [self.lock.lockfile, updater.lock.lockfile, self.tmp_dir]
+ self.state.files
)
flist = [self.config.files["local"], self.tmp_dir]
if path_isin(self.cache.local.cache_dir, self.root_dir):
flist += [self.cache.local.cache_dir]
self.scm.ignore_list(flist)
def get_stage(self, path=None, name=None):
if not path:
path = PIPELINE_FILE
logger.debug("Assuming '%s' to be a stage inside '%s'", name, path)
dvcfile = Dvcfile(self, path)
return dvcfile.stages[name]
def get_stages(self, path=None, name=None):
if not path:
path = PIPELINE_FILE
logger.debug("Assuming '%s' to be a stage inside '%s'", name, path)
if name:
return [self.get_stage(path, name)]
dvcfile = Dvcfile(self, path)
return list(dvcfile.stages.values())
def check_modified_graph(self, new_stages):
"""Generate graph including the new stage to check for errors"""
# Building graph might be costly for the ones with many DVC-files,
@@ -198,10 +232,9 @@ class Repo(object):
stages = nx.dfs_postorder_nodes(graph)
return [stage for stage in stages if path_isin(stage.path, path)]
def collect(self, target, with_deps=False, recursive=False, graph=None):
import networkx as nx
from ..dvcfile import Dvcfile
def collect(
self, target=None, with_deps=False, recursive=False, graph=None
):
if not target:
return list(graph) if graph else self.stages
@@ -211,36 +244,81 @@ class Repo(object):
)
path, name = parse_target(target)
dvcfile = Dvcfile(self, path)
stages = list(dvcfile.stages.filter(name).values())
stages = self.get_stages(path, name)
if not with_deps:
return stages
res = set()
for stage in stages:
pipeline = get_pipeline(get_pipelines(graph or self.graph), stage)
res.update(nx.dfs_postorder_nodes(pipeline, stage))
res.update(self._collect_pipeline(stage, graph=graph))
return res
def collect_granular(self, target, *args, **kwargs):
from ..dvcfile import Dvcfile, is_valid_filename
def _collect_pipeline(self, stage, graph=None):
import networkx as nx
pipeline = get_pipeline(get_pipelines(graph or self.graph), stage)
return nx.dfs_postorder_nodes(pipeline, stage)
def _collect_from_default_dvcfile(self, target):
dvcfile = Dvcfile(self, PIPELINE_FILE)
if dvcfile.exists():
return dvcfile.stages.get(target)
def collect_granular(
self, target=None, with_deps=False, recursive=False, graph=None
):
"""
Priority is in the order of following in case of ambiguity:
- .dvc file or .yaml file
- dir if recursive and directory exists
- stage_name
- output file
"""
if not target:
return [(stage, None) for stage in self.stages]
file, name = parse_target(target)
if is_valid_filename(file) and not kwargs.get("with_deps"):
# Optimization: do not collect the graph for a specific .dvc target
stages = Dvcfile(self, file).stages.filter(name)
return [(stage, None) for stage in stages.values()]
stages = []
try:
(out,) = self.find_outs_by_path(file, strict=False)
filter_info = PathInfo(os.path.abspath(file))
return [(out.stage, filter_info)]
except OutputNotFoundError:
stages = self.collect(target, *args, **kwargs)
return [(stage, None) for stage in stages]
# Optimization: do not collect the graph for a specific target
if not file:
# parsing is ambiguous when it does not have a colon
# or if it's not a dvcfile, as it can be a stage name
# in `dvc.yaml` or, an output in a stage.
logger.debug(
"Checking if stage '%s' is in '%s'", target, PIPELINE_FILE
)
if not (recursive and os.path.isdir(target)):
stage = self._collect_from_default_dvcfile(target)
if stage:
stages = (
self._collect_pipeline(stage) if with_deps else [stage]
)
elif not with_deps and is_valid_filename(file):
stages = self.get_stages(file, name)
if not stages:
if not (recursive and os.path.isdir(target)):
try:
(out,) = self.find_outs_by_path(target, strict=False)
filter_info = PathInfo(os.path.abspath(target))
return [(out.stage, filter_info)]
except OutputNotFoundError:
pass
try:
stages = self.collect(target, with_deps, recursive, graph)
except StageFileDoesNotExistError as exc:
# collect() might try to use `target` as a stage name
# and throw error that dvc.yaml does not exist, whereas it
# should say that both stage name and file does not exist.
if file and is_valid_filename(file):
raise
raise NoOutputOrStageError(target, exc.file) from exc
except StageNotFound as exc:
raise NoOutputOrStageError(target, exc.file) from exc
return [(stage, None) for stage in stages]
def used_cache(
self,
@@ -253,6 +331,7 @@ class Repo(object):
force=False,
jobs=None,
recursive=False,
used_run_cache=None,
):
"""Get the stages related to the given target and collect
the `info` of its outputs.
@@ -286,7 +365,7 @@ class Repo(object):
for target in targets
)
suffix = "({})".format(branch) if branch else ""
suffix = f"({branch})" if branch else ""
for stage, filter_info in pairs:
used_cache = stage.get_used_cache(
remote=remote,
@@ -296,6 +375,12 @@ class Repo(object):
)
cache.update(used_cache, suffix=suffix)
if used_run_cache:
used_cache = self.stage_cache.get_used_cache(
used_run_cache, remote=remote, force=force, jobs=jobs,
)
cache.update(used_cache)
return cache
def _collect_graph(self, stages):
@@ -423,18 +508,20 @@ class Repo(object):
"""
return self._collect_stages()
def _collect_stages(self):
from dvc.dvcfile import Dvcfile, is_valid_filename
@cached_property
def plot_templates(self):
from .plots.template import PlotTemplates
return PlotTemplates(self.dvc_dir)
def _collect_stages(self):
stages = []
outs = set()
for root, dirs, files in self.tree.walk(self.root_dir):
for file_name in filter(is_valid_filename, files):
path = os.path.join(root, file_name)
stage_loader = Dvcfile(self, path).stages
with stage_loader.log_level(at=logging.DEBUG):
stages.extend(stage_loader.values())
stages.extend(self.get_stages(path))
outs.update(
out.fspath
for stage in stages
@@ -480,49 +567,22 @@ class Repo(object):
@contextmanager
def open_by_relpath(self, path, remote=None, mode="r", encoding=None):
"""Opens a specified resource as a file descriptor"""
cause = None
tree = RepoTree(self, stream=True)
path = os.path.join(self.root_dir, path)
try:
out = self.find_out_by_relpath(path)
except OutputNotFoundError as exc:
out = None
cause = exc
if out and out.use_cache:
try:
with self._open_cached(out, remote, mode, encoding) as fd:
yield fd
return
except FileNotFoundError as exc:
raise FileMissingError(path) from exc
abs_path = os.path.join(self.root_dir, path)
if os.path.exists(abs_path):
with open(abs_path, mode=mode, encoding=encoding) as fd:
yield fd
return
raise FileMissingError(path) from cause
def _open_cached(self, out, remote=None, mode="r", encoding=None):
if out.isdir():
raise IsADirectoryError("Can't open a dir")
cache_file = self.cache.local.checksum_to_path_info(out.checksum)
cache_file = fspath_py35(cache_file)
if os.path.exists(cache_file):
return open(cache_file, mode=mode, encoding=encoding)
try:
remote_obj = self.cloud.get_remote(remote)
remote_info = remote_obj.checksum_to_path_info(out.checksum)
return remote_obj.open(remote_info, mode=mode, encoding=encoding)
except RemoteActionNotImplemented:
with self.state:
cache_info = out.get_used_cache(remote=remote)
self.cloud.pull(cache_info, remote=remote)
return open(cache_file, mode=mode, encoding=encoding)
with tree.open(
os.path.join(self.root_dir, path),
mode=mode,
encoding=encoding,
remote=remote,
) as fobj:
yield fobj
except FileNotFoundError as exc:
raise FileMissingError(path) from exc
except IsADirectoryError as exc:
raise DvcIsADirectoryError(f"'{path}' is a directory") from exc
def close(self):
self.scm.close()
+5 -4
View File
@@ -3,17 +3,18 @@ import os
import colorama
from . import locked
from dvc.dvcfile import Dvcfile, is_dvc_file
from ..exceptions import (
RecursiveAddingWhileUsingFilename,
OverlappingOutputPathsError,
OutputDuplicationError,
OverlappingOutputPathsError,
RecursiveAddingWhileUsingFilename,
)
from ..output.base import OutputDoesNotExistError
from ..progress import Tqdm
from ..repo.scm_context import scm_context
from ..utils import LARGE_DIR_SIZE, resolve_paths
from . import locked
logger = logging.getLogger(__name__)
@@ -144,6 +145,6 @@ def _create_stages(repo, targets, fname, pbar=None):
stages.append(stage)
if pbar is not None:
pbar.update_desc(out)
pbar.update_msg(out)
return stages
+4 -1
View File
@@ -25,13 +25,16 @@ def brancher( # noqa: E302
return
saved_tree = self.tree
revs = revs or []
revs = revs.copy() if revs else []
scm = self.scm
self.tree = WorkingTree(self.root_dir)
yield "working tree"
if revs and "working tree" in revs:
revs.remove("working tree")
if all_commits:
revs = scm.list_all_commits()
else:
+19 -11
View File
@@ -1,8 +1,11 @@
import logging
import os
from dvc.compat import fspath
from dvc.exceptions import CheckoutError, CheckoutErrorSuggestGit
from dvc.exceptions import (
CheckoutError,
CheckoutErrorSuggestGit,
NoOutputOrStageError,
)
from dvc.progress import Tqdm
from dvc.utils import relpath
@@ -19,11 +22,11 @@ def _get_unused_links(repo):
return repo.state.get_unused_links(used)
def _fspath_dir(path, root):
def _fspath_dir(path):
if not os.path.exists(str(path)):
return str(path)
path = relpath(fspath(path), root)
path = relpath(path)
return os.path.join(path, "") if os.path.isdir(path) else path
@@ -57,9 +60,12 @@ def _checkout(
targets = [None]
unused = _get_unused_links(self)
stats["deleted"] = [_fspath_dir(u, self.root_dir) for u in unused]
stats["deleted"] = [_fspath_dir(u) for u in unused]
self.state.remove_links(unused)
if isinstance(targets, str):
targets = [targets]
pairs = set()
for target in targets:
try:
@@ -68,7 +74,11 @@ def _checkout(
target, with_deps=with_deps, recursive=recursive
)
)
except (StageFileDoesNotExistError, StageFileBadNameError) as exc:
except (
StageFileDoesNotExistError,
StageFileBadNameError,
NoOutputOrStageError,
) as exc:
if not target:
raise
raise CheckoutErrorSuggestGit(target) from exc
@@ -80,14 +90,12 @@ def _checkout(
for stage, filter_info in pairs:
result = stage.checkout(
force=force,
progress_callback=pbar.update_desc,
progress_callback=pbar.update_msg,
relink=relink,
filter_info=filter_info,
)
for data in ["failed", "added", "modified"]:
stats[data].extend(
_fspath_dir(path, self.root_dir) for path in result[data]
)
for key, items in result.items():
stats[key].extend(_fspath_dir(path) for path in items)
if stats.get("failed"):
raise CheckoutError(stats["failed"], stats)
+39 -2
View File
@@ -1,12 +1,49 @@
from . import locked
from dvc import prompt
from dvc.dvcfile import Dvcfile
from dvc.stage.exceptions import StageCommitError
from . import locked
def _prepare_message(stage, changes):
changed_deps, changed_outs, changed_stage = changes
if changed_deps and changed_outs:
msg = "dependencies {deps} and outputs {outs} of {stage} changed."
elif changed_deps:
msg = "dependencies {deps} of {stage} changed."
elif changed_outs:
msg = "outputs {outs} of {stage} changed."
else:
msg = "{stage_changed}"
msg += " Are you sure you want to commit it?"
kw = {
"stage": stage,
"deps": changed_deps,
"outs": changed_outs,
"stage_changed": changed_stage,
}
return msg.format_map(kw)
def prompt_to_commit(stage, changes, force=False):
if not (force or prompt.confirm(_prepare_message(stage, changes))):
raise StageCommitError(
"unable to commit changed {}. Use `-f|--force` to "
"force.".format(stage)
)
@locked
def commit(self, target, with_deps=False, recursive=False, force=False):
stages = self.collect(target, with_deps=with_deps, recursive=recursive)
for stage in stages:
stage.check_can_commit(force=force)
changes = stage.changed_entries()
if any(changes):
prompt_to_commit(stage, changes, force=force)
stage.save()
stage.commit()
Dvcfile(self, stage.path).dump(stage)
return stages
+1
View File
@@ -1,4 +1,5 @@
from dvc.utils.fs import remove
from . import locked
+6 -1
View File
@@ -52,6 +52,9 @@ def diff(self, a_rev="HEAD", b_rev=None):
if _exists(output)
}
if self.scm.no_commits:
return {}
working_tree = self.tree
a_tree = self.scm.get_tree(a_rev)
b_tree = self.scm.get_tree(b_rev) if b_rev else working_tree
@@ -71,7 +74,7 @@ def diff(self, a_rev="HEAD", b_rev=None):
deleted = sorted(set(old) - set(new))
modified = sorted(set(old) & set(new))
return {
ret = {
"added": [{"path": path, "hash": new[path]} for path in added],
"deleted": [{"path": path, "hash": old[path]} for path in deleted],
"modified": [
@@ -80,3 +83,5 @@ def diff(self, a_rev="HEAD", b_rev=None):
if old[path] != new[path]
],
}
return ret if any(ret.values()) else {}
+15 -53
View File
@@ -1,10 +1,8 @@
import logging
from dvc.cache import NamedCache
from dvc.config import NoRemoteError
from dvc.exceptions import DownloadError, OutputNotFoundError
from dvc.exceptions import DownloadError
from dvc.scm.base import CloneError
from dvc.path_info import PathInfo
logger = logging.getLogger(__name__)
@@ -20,6 +18,7 @@ def _fetch(
all_tags=False,
recursive=False,
all_commits=False,
run_cache=False,
):
"""Download data items from a cloud and imported repositories
@@ -33,6 +32,12 @@ def _fetch(
config.NoRemoteError: thrown when downloading only local files and no
remote is configured
"""
used_run_cache = self.stage_cache.pull(remote) if run_cache else []
if isinstance(targets, str):
targets = [targets]
used = self.used_cache(
targets,
all_branches=all_branches,
@@ -43,6 +48,7 @@ def _fetch(
remote=remote,
jobs=jobs,
recursive=recursive,
used_run_cache=used_run_cache,
)
downloaded = 0
@@ -50,7 +56,7 @@ def _fetch(
try:
downloaded += self.cloud.pull(
used, jobs, remote=remote, show_checksums=show_checksums
used, jobs, remote=remote, show_checksums=show_checksums,
)
except NoRemoteError:
if not used.external and used["local"]:
@@ -70,35 +76,15 @@ def _fetch(
def _fetch_external(self, repo_url, repo_rev, files, jobs):
from dvc.external_repo import external_repo, ExternalRepo
from dvc.external_repo import external_repo
failed, downloaded = 0, 0
try:
with external_repo(repo_url, repo_rev) as repo:
is_dvc_repo = isinstance(repo, ExternalRepo)
# gather git-only tracked files if dvc repo
git_files = [] if is_dvc_repo else files
if is_dvc_repo:
repo.cache.local.cache_dir = self.cache.local.cache_dir
with repo.state:
cache = NamedCache()
for name in files:
try:
out = repo.find_out_by_relpath(name)
except OutputNotFoundError:
# try to add to cache if they are git-tracked files
git_files.append(name)
else:
cache.update(out.get_used_cache())
try:
downloaded += repo.cloud.pull(cache, jobs=jobs)
except DownloadError as exc:
failed += exc.amount
d, f = _git_to_cache(self.cache.local, repo.root_dir, git_files)
downloaded += d
failed += f
with repo.use_cache(self.cache.local):
d, f, _ = repo.fetch_external(files, jobs=jobs)
downloaded += d
failed += f
except CloneError:
failed += 1
logger.exception(
@@ -106,27 +92,3 @@ def _fetch_external(self, repo_url, repo_rev, files, jobs):
)
return downloaded, failed
def _git_to_cache(cache, repo_root, files):
"""Save files from a git repo directly to the cache."""
failed = set()
num_downloads = 0
repo_root = PathInfo(repo_root)
for file in files:
info = cache.save_info(repo_root / file)
if info.get(cache.PARAM_CHECKSUM) is None:
failed.add(file)
continue
if cache.changed_cache(info[cache.PARAM_CHECKSUM]):
logger.debug("fetched '%s' from '%s' repo", file, repo_root)
num_downloads += 1
cache.save(repo_root / file, info, save_link=False)
if failed:
logger.exception(
"failed to fetch data for {}".format(", ".join(failed))
)
return num_downloads, len(failed)
+2 -2
View File
@@ -1,9 +1,9 @@
import logging
from . import locked
from dvc.cache import NamedCache
from dvc.exceptions import InvalidArgumentError
from . import locked
logger = logging.getLogger(__name__)
@@ -11,7 +11,7 @@ logger = logging.getLogger(__name__)
def _do_gc(typ, func, clist, jobs=None):
removed = func(clist, jobs=jobs)
if not removed:
logger.info("No unused '{}' cache to remove.".format(typ))
logger.info(f"No unused '{typ}' cache to remove.")
def _raise_error_if_all_disabled(**kwargs):
+1 -3
View File
@@ -6,7 +6,6 @@ import shortuuid
from dvc.exceptions import DvcException
from dvc.utils import resolve_output
from dvc.utils.fs import remove
from dvc.path_info import PathInfo
logger = logging.getLogger(__name__)
@@ -51,7 +50,6 @@ def get(url, path, out=None, rev=None):
# Also, we can't use theoretical "move" link type here, because
# the same cache file might be used a few times in a directory.
repo.cache.local.cache_types = ["reflink", "hardlink", "copy"]
repo.pull_to(path, PathInfo(out))
repo.get_external(path, out)
finally:
remove(tmp_dir)
+9 -4
View File
@@ -1,10 +1,11 @@
import os
from . import locked as locked_repo
from dvc.repo.scm_context import scm_context
from dvc.utils import resolve_output, resolve_paths, relpath
from dvc.utils import relpath, resolve_output, resolve_paths
from dvc.utils.fs import path_isin
from ..exceptions import OutputDuplicationError
from . import locked as locked_repo
@locked_repo
@@ -16,8 +17,12 @@ def imp_url(self, url, out=None, fname=None, erepo=None, locked=True):
out = resolve_output(url, out)
path, wdir, out = resolve_paths(self, out)
# NOTE: when user is importing something from within his own repository
if os.path.exists(url) and path_isin(os.path.abspath(url), self.root_dir):
# NOTE: when user is importing something from within their own repository
if (
erepo is None
and os.path.exists(url)
and path_isin(os.path.abspath(url), self.root_dir)
):
url = relpath(url, wdir)
stage = create_stage(
+2 -3
View File
@@ -9,8 +9,7 @@ from dvc.exceptions import InitError, InvalidArgumentError
from dvc.repo import Repo
from dvc.scm import SCM
from dvc.scm.base import SCMError
from dvc.utils import boxify
from dvc.utils import relpath
from dvc.utils import boxify, relpath
from dvc.utils.fs import remove
logger = logging.getLogger(__name__)
@@ -102,7 +101,7 @@ def init(root_dir=os.curdir, no_scm=False, force=False, subdir=False):
proj = Repo(root_dir)
scm.add([config.files["repo"]])
scm.add([config.files["repo"], proj.plot_templates.templates_dir])
if scm.ignore_file:
scm.add([os.path.join(dvc_dir, scm.ignore_file)])
+2 -4
View File
@@ -3,13 +3,11 @@ from . import locked
@locked
def lock(self, target, unlock=False):
from .. import dvcfile
from dvc.utils import parse_target
path, name = parse_target(target)
dvcfile = dvcfile.Dvcfile(self, path)
stage = dvcfile.stages[name]
stage = self.get_stage(path, name)
stage.locked = False if unlock else True
dvcfile.dump(stage, update_pipeline=True)
stage.dvcfile.dump(stage, update_pipeline=True)
return stage
+38 -41
View File
@@ -1,10 +1,10 @@
from dvc.path_info import PathInfo
from dvc.exceptions import PathMissingError
from dvc.path_info import PathInfo
@staticmethod
def ls(
url, path=None, rev=None, recursive=None, outs_only=False,
url, path=None, rev=None, recursive=None, dvc_only=False,
):
"""Methods for getting files and outputs for the repo.
@@ -13,7 +13,7 @@ def ls(
path (str, optional): relative path into the repo
rev (str, optional): SHA commit, branch or tag name
recursive (bool, optional): recursively walk the repo
outs_only (bool, optional): show only DVC-artifacts
dvc_only (bool, optional): show only DVC-artifacts
Returns:
list of `entry`
@@ -28,25 +28,16 @@ def ls(
}
"""
from dvc.external_repo import external_repo
from dvc.repo import Repo
with external_repo(url, rev) as repo:
path_info = PathInfo(repo.root_dir)
if path:
path_info /= path
ret = {}
if isinstance(repo, Repo):
ret = _ls(repo, path_info, recursive, True)
nondvc = {}
if not outs_only:
nondvc = _ls(repo, path_info, recursive, False)
ret.update(nondvc)
ret = _ls(repo, path_info, recursive, dvc_only)
if path and not ret:
raise PathMissingError(path, repo, output_only=outs_only)
raise PathMissingError(path, repo, dvc_only=dvc_only)
ret_list = []
for path, info in ret.items():
@@ -56,46 +47,52 @@ def ls(
return ret_list
def _ls(repo, path_info, recursive=None, dvc=False):
from dvc.ignore import CleanTree
from dvc.repo.tree import DvcTree
from dvc.scm.tree import WorkingTree
def _ls(repo, path_info, recursive=None, dvc_only=False):
from dvc.repo.tree import RepoTree
if dvc:
tree = DvcTree(repo)
else:
tree = CleanTree(WorkingTree(repo.root_dir))
# use our own RepoTree instance instead of repo.repo_tree since we do not
# want fetch/stream enabled for ls
tree = RepoTree(repo)
ret = {}
try:
for root, dirs, files in tree.walk(path_info.fspath):
for root, dirs, files in tree.walk(path_info.fspath, dvcfiles=True):
for fname in files:
info = PathInfo(root) / fname
path = str(info.relative_to(path_info))
ret[path] = {
"isout": dvc,
"isdir": False,
"isexec": False if dvc else tree.isexec(info.fspath),
}
dvc = tree.isdvc(info)
if dvc or not dvc_only:
path = str(info.relative_to(path_info))
ret[path] = {
"isout": dvc,
"isdir": False,
"isexec": False if dvc else tree.isexec(info),
}
if not recursive:
for dname in dirs:
info = PathInfo(root) / dname
path = str(info.relative_to(path_info))
ret[path] = {
"isout": tree.isdvc(info.fspath) if dvc else False,
"isdir": True,
"isexec": False if dvc else tree.isexec(info.fspath),
}
if not dvc_only or (
tree.dvctree and tree.dvctree.exists(info)
):
dvc = tree.isdvc(info)
path = str(info.relative_to(path_info))
ret[path] = {
"isout": dvc,
"isdir": True,
"isexec": False if dvc else tree.isexec(info),
}
break
except NotADirectoryError:
return {
path_info.name: {
"isout": dvc,
"isdir": False,
"isexec": False if dvc else tree.isexec(path_info.fspath),
dvc = tree.isdvc(path_info)
if dvc or not dvc_only:
return {
path_info.name: {
"isout": dvc,
"isdir": False,
"isexec": False if dvc else tree.isexec(path_info),
}
}
}
return {}
except FileNotFoundError:
return {}
+1 -1
View File
@@ -1,4 +1,4 @@
class Metrics(object):
class Metrics:
def __init__(self, repo):
self.repo = repo

Some files were not shown because too many files have changed in this diff Show More