Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,338 Commits 7 Branches 579 Tags
45944ddde7e1ad147823f11b1efbd7f07b300edb
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 45944ddde7 metrics: rework metrics implementation (#3658) 
Dropping support for all types except json and yaml. No more xpath
support.

Fixes #3572
 2020-04-22 13:39:59 +03:00
Ruslan Kuprieiev bca5edf108 gc: don't forget to load remote index (#3660) 
Kudos @pmrowla
 2020-04-21 20:49:36 +03:00
Ruslan Kuprieiev 11e56dbdf9 tests: add some DvcTree unit tests (#3640) 2020-04-15 21:39:39 +03:00
Ruslan Kuprieiev c60f364774 dvc: introduce DvcTree (#3639) 
This is useful as it is in `dvc ls` implementation, but this is also a
prerequisite for `RepoTree` that will combine git(workspace) files and
dvc files into one tree, allowing things like `dvc metrics` and
`external_repo` to have a universal interface to access whatever
file/dir they want without cumbersome git/dvc path detection.
 2020-04-15 18:33:09 +03:00
Ruslan Kuprieiev 45c2c72b60 remote: don't use PathInfo.from_posix (#3637) 
Using it to load dirs takes around 50 sec compared to 0.4 with simple
`replace()`. Path objects are great, but not when you need lots of them.

Related to #3635
 2020-04-15 17:47:52 +03:00
Ruslan Kuprieiev 8455b7bece params: diff: introduce --all (#3631) 
* params: diff: introduce --all

Makes `dvc params diff` also show unchanged params.

* params: diff: show all present parameters
 2020-04-15 13:21:24 +03:00
Ruslan Kuprieiev fc6a10f486 stage: resolve stage path before creating the stage (#3630) 
Current logic is extremely complex and messy, hiding a lot of bugs
inside. There are pretty much only 2 scenarios for stage creation:

1) `dvc add/import/etc` when dvcfile is created beside the output file
2) `dvc run` when dvcfile is created at the place of execution

both are transparently affected by `--fname` flag. Since we've removed
`--cwd` complexity already, there is no reason to generalize stage path
resolution for those two scenarios.
 2020-04-12 18:16:13 +03:00
Ruslan Kuprieiev 63fe2893a8 run: remove deprecated --cwd (#3623) 
It has been deprecated for a long time now and it significantly
complicates the code, hiding multiple bugs.
 2020-04-11 19:57:05 +03:00
Ruslan Kuprieiev 43dcc97512 dvc: bump to 0.93.0 2020-04-10 23:22:30 +03:00
Ruslan Kuprieiev ce2481bbe8 remove: try deleting before trying to chmod (#3620) 
Windows (or rather NTFS) doesn't allow deleting read-only files, which made us
try to chmod +w first. But that is not necessery on Linux and Mac, and
has led to unnecessary cache file mode changes, which was causing issues
in shared cache scenario when a user that doesn't own a file was trying
to automatically protect it.

Fixes #3618
 2020-04-10 23:21:16 +03:00
Ruslan Kuprieiev 0e854f9534 params: diff: sort table (#3617) 2020-04-09 21:23:18 +03:00
Ruslan Kuprieiev cdef709e8a params: fix bug when collecting multiple params (#3615) 2020-04-09 15:41:35 +03:00
Ruslan Kuprieiev c1ac6bb06b dvc: introduce dvc params diff (#3573) 
* params: use repo.tree

* remote: local: use `repo.tree` for `exists()`

* param: handle missing params

* stage: move status warning to debug

* dvc: introduce `dvc params diff`

Fixes #3528
 2020-04-09 05:18:49 +03:00
Ruslan Kuprieiev 290a9fd434 analytics: collect runtime info in the worker (#3610) 
Shaves off 0.2 seconds from any dvc command.

Related to #3475
 2020-04-08 15:57:05 +03:00
Ruslan Kuprieiev 26980ae45b metrics: diff: fix float precision formatting (#3609) 
Texttable will try to detect the column type automatically, which
results in a low precision of the float that it is showing. We need to
disable that auto-formatting and just use `str(val)` instead.
 2020-04-07 19:42:25 +03:00
Ruslan Kuprieiev 6ab06bbf9e dvc: bump to 0.92.1 2020-04-05 04:35:22 +03:00
Ruslan Kuprieiev 7bd473371c push/pull/status/metrics: add support for --all-commits (#3587) 
First steps for #1691
 2020-04-04 19:30:11 +03:00
Ruslan Kuprieiev 8e675a9d42 metrics: diff: don't print "No changes" (#3576) 
Makes it easier to use in shell scripts. E.g. `-z $(dvc metrics diff)`, which
has the same semantics as `git diff`. Also syncs up with the behaviour
of `dvc diff`.

Fixes #3469
 2020-04-02 04:54:40 +03:00
Ruslan Kuprieiev a338fad036 dvc: bump to 0.92.0 2020-04-02 01:15:02 +03:00
Ruslan Kuprieiev b02babee9e dvc: bump to 0.91.4 2020-04-01 17:03:40 +03:00
Ruslan Kuprieiev 55644f376b dvc: bump to 0.91.3 2020-03-31 12:11:05 +03:00
Ruslan Kuprieiev 38e818bb1d remote: skip non-cache looking files when estimating remote size (#3557) 
* remote: base: improve exception message

* remote: use `all()` instead of `list_cache_paths()`

`list_cache_paths()` doesn't ignore paths that don't look like hashes,
which makes dvc raise an exception when it tries to convert such paths
to hashes. We should use `all()` instead.

Discord context:
https://discordapp.com/channels/485586884165107732/485596304961962003/694454282719592458

* remote: all: print debug message when skipping paths

This will allow us to see it from verbose logs coming from users.
 2020-03-31 12:10:01 +03:00
Ruslan Kuprieiev 0c2e4d0b72 dvc: bump to 0.91.2 2020-03-30 02:10:26 +03:00
Ruslan Kuprieiev 02f92b3ac8 dvc: bump to 0.91.1 2020-03-26 03:42:47 +02:00
Ruslan Kuprieiev 974f81791f protect: ignore EROFS (#3527) 
* protect: fix bug in mode verification

* protect: ignore EROFS

Fixes #3510
 2020-03-26 03:33:48 +02:00
Ruslan Kuprieiev 912a98e05e metrics: diff: fix bug with unchanged raw metrics (#3535) 2020-03-26 03:02:34 +02:00
Ruslan Kuprieiev 9de8d1e0fa params: use dpath for xpaths (#3531) 
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:

```
angles:
    - 2
    - 3
    - 100
    - 3.2
    - 36.2
    - 3.6
```

because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
 2020-03-25 06:17:09 +02:00
Ruslan Kuprieiev 0bf6fdae14 metrics: diff: handle deleted/new metric (#3529) 
Partial fix for https://github.com/iterative/dvc/issues/3469
 2020-03-25 05:23:21 +02:00
Ruslan Kuprieiev 1ddab1077b dvc: bump to 0.91.0 2020-03-24 16:15:12 +02:00
Ruslan Kuprieiev 8f546f253b Revert "Restyle #3393 introduce hyper parameters and config (#3516)" (#3517) 
This reverts commit 65d22c7dbd.
 2020-03-20 00:54:21 +02:00
Ruslan Kuprieiev caf5b931b9 readme: fix flowchart link 2020-03-19 19:21:49 +02:00
Ruslan Kuprieiev a1fe6c6f44 dvc: bump to 0.90.2 2020-03-18 23:11:17 +02:00
Ruslan Kuprieiev 7edee656da pipeline: fix --outs output (#3507) 
Fixes #3497
 2020-03-18 23:03:41 +02:00
Ruslan Kuprieiev 498ee509da dvc: use consistent messages for InvalidArgumentError (#3506) 
Per https://discordapp.com/channels/485586884165107732/563406153334128681/689896480898810034
 2020-03-18 21:34:46 +02:00
Ruslan Kuprieiev 55f86df5b1 dvc: bump to 0.90.1 2020-03-18 19:20:22 +02:00
Ruslan Kuprieiev eae432818a git: tree: use item.path instead of item.name (#3505) 
We are currently using this [0] hack to work around the fact that
item.name
is not always a basename. Even though `IndexObject` has it defined that
way
[1], but `Submodule` - doesn't [2]. So using `item.name` might cause
issues
such as [3], because GitPython doesn't pass name parameter when simply
going through objects [4]. In other places, when working specifically
with
submodules, it sets `_name` attribute explicitly [5].

Note that we still don't walk into submodules when walking through a git
repo. Might want to reconsider that later as for now we haven't heard
anything about it from actual users.

Fixes #3481

[0] https://github.com/iterative/dvc/blob/0.90.0/dvc/scm/git/tree.py#L15
[1]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/base.py#L170
[2]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/submodule/base.py#L1123
[3] https://github.com/gitpython-developers/GitPython/issues/597
[4]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/tree.py#L237
[5]
https://github.com/gitpython-developers/GitPython/blob/3.1.0/git/objects/submodule/base.py#L357
 2020-03-18 19:18:58 +02:00
Ruslan Kuprieiev aef3dfba13 dvc: bump to 0.90.0 2020-03-18 01:24:19 +02:00
Ruslan Kuprieiev 42c8fab47f version: add supported remotes (#3498) 
Since our remotes have optional dependencies, it is really useful to be
able to see what is currently supported. Also helps debug issues with
packages such as brew/conda/etc.
 2020-03-18 01:14:59 +02:00
Ruslan Kuprieiev 6559b9d222 dvc: use protected mode by default (#3472) 
* dvc: use protected mode by default

Fixes #3261
Fixes #2041

* unprotect: adjust help message

https://github.com/iterative/dvc.org/pull/1058/files/21aab371a487acf6f6e6201b29cd832e7c55ed23#r393324668
 2020-03-17 19:16:21 +02:00
Ruslan Kuprieiev 02daaf391a dvc: bump to 0.89.0 2020-03-17 02:08:56 +02:00
Ruslan Kuprieiev 2ffebb0b5e logger: use standard lazy formatting (#3487) 
Unfortunately overriding `format` is not enough in order to use our own
custom formatting in logger. `Record` uses percent formatting that is
hardcoded inside, so we would also have to override it, which is just
way too much effort that will inevitably break as we discover other
methods that are not ready for non-% formatting. So we are just giving
up and sticking to default %-formatting for lazy log messages.

Fixes #3397
 2020-03-17 00:38:00 +02:00
Ruslan Kuprieiev 3f3384b7c4 pre-commit: use official dvc repo 2020-03-16 20:41:29 +02:00
Ruslan Kuprieiev b223696079 install: use hooks from master 
Same as with shell completion, it is better to use the newest hooks.
 2020-03-16 20:40:02 +02:00
Ruslan Kuprieiev ef96e6b8ff dvc: bump to 0.88.0 2020-03-10 22:47:01 +02:00
Ruslan Kuprieiev 40519005f5 travis: hadoop: update tar location 2020-03-10 10:20:34 +02:00
Ruslan Kuprieiev 37e4a5dcd4 travis: use us-west-2 aws s3 region 2020-03-09 19:10:47 +02:00
Ruslan Kuprieiev 6479adb1a3 test: s3: move to dvc-temp bucket 2020-03-09 18:45:59 +02:00
Ruslan Kuprieiev f05e6e0529 metrics show: update -h output to match docs (#3420) 
per https://github.com/iterative/dvc.org/pull/1014/commits/2c3452119545987f77b764e4aa6fc94d051092a7
 2020-03-05 16:11:43 +02:00
Ruslan Kuprieiev 06c634adee pipelines: fix edges gathering (#3421) 2020-03-05 14:16:57 +02:00
Ruslan Kuprieiev abc4e3c2d0 dvc: bump to 0.87.0 2020-02-28 02:21:39 +02:00
Ruslan Kuprieiev 432e399f44 deps: remove pyfastcopy 
https://github.com/iterative/dvc/issues/3395
 2020-02-25 15:53:43 +02:00
Ruslan Kuprieiev 2c46bddcfa dvc: bump to 0.86.5 2020-02-21 00:11:41 +02:00
Ruslan Kuprieiev e4bf5c1ddb dvc: bump to 0.86.4 2020-02-20 03:46:51 +02:00
Ruslan Kuprieiev 6474725d4f test: repro: prevent dvc.stages resetting (#3368) 2020-02-20 03:45:28 +02:00
Ruslan Kuprieiev c5677decd3 dvc: bump to 0.86.3 2020-02-20 02:40:14 +02:00
Ruslan Kuprieiev 6dba652b25 dvc: bump to 0.86.2 2020-02-17 20:30:51 +02:00
Ruslan Kuprieiev d2a6c1fc62 external_repo: fix bug in path resolution (#3350) 2020-02-17 20:30:04 +02:00
Ruslan Kuprieiev 130180cc5c setup.py: remove gitdb2 dependency and update gitpython 2020-02-17 18:12:30 +02:00
Ruslan Kuprieiev 72f1313cc3 setup: fix gitpython version 2020-02-17 17:54:53 +02:00
Ruslan Kuprieiev a90d7009f5 dvc: bump to 0.86.1 2020-02-17 17:08:37 +02:00
Ruslan Kuprieiev 4986395998 setup.py: limit gitpython version 
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=122147&view=logs&jobId=4f922444-fdfe-5dcf-b824-02f86439ef14&j=4f922444-fdfe-5dcf-b824-02f86439ef14&t=937c195f-508d-5135-dc9f-d4b5730df0f7
 2020-02-17 17:03:33 +02:00
Ruslan Kuprieiev 19edc1ab1d dvc: bump to 0.86.0 2020-02-17 12:44:49 +02:00
Ruslan Kuprieiev 4087b020db dvc: bump to 0.85.0 2020-02-15 01:14:37 +02:00
Ruslan Kuprieiev 8a2699dfbb schema: use a more flexible hash schema (#3333) 
Fixes #3331
 2020-02-15 01:13:45 +02:00
Ruslan Kuprieiev 9a78c0e2fd dvc: bump to 0.84.0 2020-02-11 22:09:59 +02:00
Ruslan Kuprieiev 7200bddd09 dvc: bump to 0.83.1 2020-02-11 22:08:30 +02:00
Ruslan Kuprieiev 2361bceb70 repro: fix bug in _get_active_graph (#3301) 
We are currently removing all of the deps stages from the DAG even if
those deps are referenced by some parallel branches of execution.
The new approach is to remove edges and then gradually remove nodes if
there are no more `in` edges left.

Discord context:
https://discordapp.com/channels/485586884165107732/485596304961962003/676464313942409265
 2020-02-11 01:45:43 +02:00
Ruslan Kuprieiev 0c8897949e diff: use rev instead of ref (#3299) 
Fixes #3245
 2020-02-11 00:13:23 +02:00
Ruslan Kuprieiev 65ee446642 dvc: bump to 0.83.0 2020-02-10 01:42:31 +02:00
Ruslan Kuprieiev d305803015 analytics: ignore request.post errors (#3292) 
Users have been complaining about dvc analytics errors when they are not
connected to the internet.
 2020-02-10 01:41:42 +02:00
Ruslan Kuprieiev 2786a4e59b logger: allow exc_info in all record types (#3290) 
This better reflects the original logger formatter.

This also fixes annoying redundant newline between traceback and "need
any help" footer.
 2020-02-09 22:22:20 +02:00
Ruslan Kuprieiev afcbc7e17a deps: update pydrive2 version 2020-02-09 18:48:38 +02:00
Ruslan Kuprieiev 0bb5a8210a dvc: bump to 0.82.10 2020-02-09 18:47:24 +02:00
Ruslan Kuprieiev d9ed98b951 Merge pull request #3270 from mroutis/fix-3164 
rwlock: include actionable in error message
 2020-02-05 13:05:49 +02:00
Ruslan Kuprieiev cb06a91368 dvc: bump to 0.82.9 2020-02-05 03:20:52 +02:00
Ruslan Kuprieiev 93d1b6dcb0 Merge pull request #3217 from fabiosantoscode/feature/1744-pipeline-eliminate-extra-connection 
pipeline: show: outs: eliminate extra edges in DAG
 2020-02-04 02:02:31 +02:00
Ruslan Kuprieiev ed04a9f5ff Merge pull request #3276 from fabiosantoscode/patch-1 
commit: add missing space
 2020-02-04 01:49:27 +02:00
Ruslan Kuprieiev de1f67afbd Merge pull request #3244 from jorgeorpinel/master 
diff/metics diff: update argument descriptions, et al.
 2020-02-03 02:15:54 +02:00
Ruslan Kuprieiev 633e126ba9 dvc: bump to 0.82.8 2020-02-02 17:59:01 +02:00
Ruslan Kuprieiev 82eaa52fab Merge pull request #3274 from efiop/3248 
remote: bring back manual close_pools()
 2020-02-02 17:56:14 +02:00
Ruslan Kuprieiev 945a517f6c Merge pull request #3260 from mroutis/change-diff-format 
diff: format summary to make it more human readable
 2020-02-01 01:11:09 +02:00
Ruslan Kuprieiev 2674689442 dvc: bump to 0.82.7 2020-01-31 23:30:32 +02:00
Ruslan Kuprieiev fd6bd29303 setup: move pydot to install_requires 
https://github.com/iterative/dvc/issues/3267
 2020-01-31 22:51:41 +02:00
Ruslan Kuprieiev 92bbd437f3 Merge pull request #3268 from Suor/isdir 
dvc: do not check for isdir on recursive out collect
 2020-01-31 22:45:35 +02:00
Ruslan Kuprieiev 2613e122ff Merge pull request #3263 from casperdcl/shfmt 
lint:beautysh pre-commit hook
 2020-01-31 22:44:30 +02:00
Ruslan Kuprieiev d864caf79b Merge pull request #3251 from Suor/summon-publish 
api: prepare for dvcx summon/publish
 2020-01-30 19:56:43 +02:00
Ruslan Kuprieiev fd1e6f457e Merge pull request #3229 from mroutis/enhance-diff 
diff: reimplement interface and tests from scratch
 2020-01-30 19:24:17 +02:00
Ruslan Kuprieiev b78ef78507 dvc: bump to 0.82.6 2020-01-30 12:03:23 +02:00
Ruslan Kuprieiev 14075c4658 Merge pull request #3253 from efiop/3241 
install: fix post-checkout hook
 2020-01-30 12:02:55 +02:00
Ruslan Kuprieiev 90773ed4f3 dvc: bump to 0.82.5 2020-01-29 22:08:41 +02:00
Ruslan Kuprieiev 3d62db5a02 Merge pull request #3252 from iterative/corpora-issue 
gdrive: Init drive if corpora property value requested.
 2020-01-29 22:07:59 +02:00
Ruslan Kuprieiev 57dae7a0c1 Merge pull request #2846 from casperdcl/add 
UI progress cleanup
 2020-01-29 19:38:05 +02:00
Ruslan Kuprieiev 12cae708a8 dvc: bump to 0.82.4 2020-01-29 00:46:11 +02:00
Ruslan Kuprieiev d16e71d767 Merge pull request #3239 from iterative/fix-shared-drives-support 
[WIP] Set corpora and driveId params values based on type of remote disk - shared or user's.
 2020-01-29 00:44:41 +02:00
Ruslan Kuprieiev 242b5b18f6 Merge pull request #3193 from mroutis/fix-3192 
config: verify before saving a modification
 2020-01-27 18:08:53 +02:00
Ruslan Kuprieiev c38589bb0d Merge pull request #3234 from Suor/git-pull-p2 
erepo: make it better
 2020-01-27 14:19:59 +02:00
Ruslan Kuprieiev a42d4f2c90 dvc: bump to 0.82.3 2020-01-24 23:36:32 +02:00
Ruslan Kuprieiev 4d5faab19c Merge pull request #3213 from fabiosantoscode/feature/3121-push-no-remote-specified-hint 
push: improve 'no remote specified' hint
 2020-01-24 22:46:55 +02:00
Ruslan Kuprieiev 0bd0c7a61e Merge pull request #3232 from iterative/2865 
Remove usage of google-api-python-client deps
 2020-01-24 18:02:24 +02:00
Ruslan Kuprieiev c2def08551 Merge pull request #3228 from efiop/3107 
install: post-checkout: run only on branches/tags
 2020-01-24 17:02:53 +02:00