Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,985 Commits 7 Branches 579 Tags
edd83446591fcdbb5f1e58754755b68195dd2b1c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 4b17685351 cache: refactor dir_info conversion (#4844) 2020-11-04 21:42:14 +02:00
Ruslan Kuprieiev e1b82c5222 dvc: add size/nfiles for deps/outs (#4836) 
* dvc: add size for deps/outs

Related to #3256

* dvc: add nfiles for deps/outs

* dvc: put size/nfiles into the hash_info
 2020-11-04 19:22:57 +02:00
Ruslan Kuprieiev 7da3de451f build: test built binary (#4839) 2020-11-03 23:30:41 +02:00
Ruslan Kuprieiev 4a3f06df3a oss: require pycryptodome<3.9.9 
3.9.9 doesn't have windows wheels https://github.com/Legrandin/pycryptodome/issues/465
 2020-11-03 20:28:44 +02:00
Ruslan Kuprieiev 2225764438 tests: set subprocess._active to None (#4818) 
To better reflect https://github.com/python/cpython/pull/14360
 2020-11-03 19:31:04 +02:00
Ruslan Kuprieiev 92752e0392 dvc: upgrade dulwich to 0.20.11 
This one has wheels for python 3.9
 2020-10-30 19:16:43 +02:00
Ruslan Kuprieiev 32fa8e33c4 git: use dulwich for .gitignore (#4809) 
In my tests `Git._ignore` is taking around ~65seconds for the whole test
suit(~50K invocations). The reason is that `gitpython` is launching a real
`git check-ignore` process, that takes time and resources.

Switching to `pygit2` or `dulwich` makes it drop to ~5seconds, saving us
almost a minute of the test run time and a lot of resources by not
launching any processes. I've chosen `dulwich` over `pygit2` here
because it is much more pythonic and as long as the performance is
on-par with `pygit2` it is better for us to use `dulwich`.
 2020-10-30 01:01:58 +02:00
Ruslan Kuprieiev 9b136f0c3f pyarrow: upgrade to 2.0.0 (#4795) 2020-10-27 19:14:30 +02:00
Ruslan Kuprieiev 411b388960 dvc: bump to 1.9.1 2020-10-27 17:33:27 +02:00
Ruslan Kuprieiev 0736c64188 gha: don't create gs creds file if no creds 2020-10-27 00:23:34 +02:00
Ruslan Kuprieiev bfecd7d8e8 gha: add gs creds (#4791) 
* gha: add gs creds

Related to #4418

* tests: gs: don't use gcloud
 2020-10-26 19:20:58 +02:00
Ruslan Kuprieiev a5bdfa90f0 gha: add aws creds (#4790) 
Related to #4418
 2020-10-26 17:02:07 +02:00
Ruslan Kuprieiev 0669d8d885 dvc: bump to 1.9.0 2020-10-22 20:22:18 +03:00
Ruslan Kuprieiev 2c7d89939c gdrive: get rid of empty file hack (#4774) 2020-10-22 20:07:55 +03:00
Ruslan Kuprieiev e48055077d plots: don't allow in non-dvc repo (#4745) 2020-10-20 11:31:57 +03:00
Ruslan Kuprieiev 21a6be7c58 Delete .restyled.yaml 
Since we've decided to stop using it. Also disabled in the repo settings.
 2020-10-15 15:07:58 +03:00
Ruslan Kuprieiev 32cef509b7 run-cache: don't save always_changed stages (#4719) 
Fixes #4335
 2020-10-14 17:43:16 +03:00
Ruslan Kuprieiev b11fe83828 dvc: bump to 1.8.4 2020-10-13 23:57:29 +03:00
Ruslan Kuprieiev 0649242578 dvc: bump to 1.8.3 2020-10-13 13:17:22 +03:00
Ruslan Kuprieiev ca490f699e ls: pass repo_tree instead of repo (#4700) 2020-10-12 12:37:19 +03:00
Ruslan Kuprieiev 3ceef7dbf1 dvc: get rid of ExternalGitRepo (#4685) 2020-10-08 22:04:48 +03:00
Ruslan Kuprieiev 060fc4fbb7 state: use granularly (#4676) 
* state: add simple reentrance

* state: pass Repo instead of LocalCache

* state: use granularly

Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
 2020-10-08 08:36:14 +03:00
Ruslan Kuprieiev 4496dbf5b5 release-drafter: add ui to enhancements 2020-10-08 03:01:38 +03:00
Ruslan Kuprieiev 334afebd31 dvc: bump to 1.8.2 2020-10-08 02:57:22 +03:00
Ruslan Kuprieiev 775e1e8374 repo: don't create .dvc in read-only repos (#4674) 
* lock: use ABC

* lock: add LockNoop

* plots: remove unused code

* plots: create templates only on init

* dvc: don't create .dvc in read-only repos

* repo: explicitly init LockNoop
 2020-10-07 23:51:14 +03:00
Ruslan Kuprieiev 15b1e7072b gha: don't build packages for PRs 
Related to #4468
 2020-10-06 20:16:37 +03:00
Ruslan Kuprieiev 7424f22e2a repro: support --pull (#4538) 
* repro: support --pull

This option allows run-cache to pull the cache for the outputs
when restoring a stage instead of pulling everything at once during
`dvc pull --run-cache`.

Fixes #4223

* repro: adjust --pull help
 2020-10-06 04:22:19 +03:00
Ruslan Kuprieiev 9ee5018c63 tests: gdrive: skip if env var is empty 2020-10-06 01:43:39 +03:00
Ruslan Kuprieiev 026047aa80 gha: set GDRIVE_CREDENTIALS_DATA 2020-10-05 23:23:02 +03:00
Ruslan Kuprieiev 3878c50ea8 yaml: wrap ruamel's DuplicateKeyError (#4668) 
* yaml: wrap ruamel's DuplicateKeyError

Unfortunately ruamel's exception doesn't contain much info that we could
reuse, so we can only reasonably extract the exc.problem and have
something like:

```
ERROR: failed to obtain data status - unable to read: 'dvc.yaml', found
duplicate key "metrics" with value "[]" (original value: "[]")
```

Fixes #4664

* status: remove unneeded error prefix
 2020-10-05 23:02:59 +03:00
Ruslan Kuprieiev 3ffb6320ac dvc: print version to debug log (#4666) 
Related to #4664
 2020-10-05 21:04:31 +03:00
Ruslan Kuprieiev 52fe76cefc metrics: diff: make table more intuitive (#4655) 
* metrics: diff: make table more intuitive

Currently we have a "diff not supported" marker in the table, that is
used whenever dvc is not able to diff the values (e.g. no old one, no
new one or format error). It is quite ugly and it is much more intuitive
to just show both old and new values and now show that marker at all, as
if there is no old/new value it is intuitive that we can't calculate the
change. Next step is to add old/new "bad format"/etc markers for
unsupported types or malformed/missing metric files.

E.g., before:

```
Path          Metric    Value    Change
metrics.json  foo       1.2      diff not supported
```

after this PR:

```
Path          Metric    Old    New    Change
metrics.json  foo       None   1.2
```

* metrics/params: use em dash in diff table
 2020-10-05 19:37:34 +03:00
Ruslan Kuprieiev 3d2493a391 dvc: bump to 1.8.1 2020-10-01 11:15:30 +03:00
Ruslan Kuprieiev d7a35b9852 gha: enable hdfs tests (#4644) 2020-09-30 05:43:34 +03:00
Ruslan Kuprieiev 00267706b7 dvc: bump to 1.8.0 2020-09-30 03:39:33 +03:00
Ruslan Kuprieiev 1fcb94dee8 hdfs: add a progress bar (#4643) 2020-09-30 03:36:50 +03:00
Ruslan Kuprieiev dc6abba679 gdrive: fix comment formatting 2020-09-30 03:10:39 +03:00
Ruslan Kuprieiev fdc455a08f gdrive: use PyDrive2 issue link instead of PyDrive 2020-09-29 23:29:35 +03:00
Ruslan Kuprieiev 7b590e7c5d gdrive: revert back to the hack for uploading empty files (#4632) 
* gdrive: bring back the progress workaround

* WIP cloud: test empty files
 2020-09-29 13:49:49 +03:00
Ruslan Kuprieiev e3a7857b13 repo: move index_dir to RemoteIndex (#4631) 2020-09-29 03:32:33 +03:00
Ruslan Kuprieiev 644c2df5e8 plots: fix typo 2020-09-29 00:55:38 +03:00
Ruslan Kuprieiev de24794cd5 repo: move templates to plots (#4629) 2020-09-29 00:38:11 +03:00
Ruslan Kuprieiev 1ea8cc9e54 metrics/plots: support non-dvc repos (#4627) 2020-09-28 23:06:22 +03:00
Ruslan Kuprieiev 9643ff7229 dvc: bump to 1.7.9 2020-09-25 02:30:14 +03:00
Ruslan Kuprieiev 8da40af3e4 ssh: download: don't use self.path_info (#4619) 2020-09-25 02:27:55 +03:00
Ruslan Kuprieiev 2f030e870b Update packages.yaml 2020-09-24 18:53:50 +03:00
Ruslan Kuprieiev a2adc956b9 dvc: bump to 1.7.8 2020-09-24 18:46:36 +03:00
Ruslan Kuprieiev deaa27dde2 gha: don't build packages when not needed 2020-09-24 18:44:10 +03:00
Ruslan Kuprieiev b784216f68 dvc: bump to 1.7.7 2020-09-24 18:38:07 +03:00
Ruslan Kuprieiev 2cbd46ba75 gha: publish packages only on released release 2020-09-24 18:31:05 +03:00
Ruslan Kuprieiev 954e4784ab dvc: bump to 1.7.6 2020-09-24 17:45:53 +03:00
Ruslan Kuprieiev 62edbaf07c ssh: install paramiko with invoke (#4617) 
Paramiko has recently added support for `Match`
https://github.com/paramiko/paramiko/issues/717
which resulted in some users getting an import error about `invoke`
module, as it is not a default dependecy for paramiko.
@Pseudomanifold reported that installing `invoke` #4589 worked for
him, which means that this newly added paramiko feature works in
his case. That feature seems to not affect configs without `Match`,
so we could give this a try and install it by default.

Our conda package will be updated during the next release.

Fixes #4589
Fixes #4616
 2020-09-24 17:45:06 +03:00
Ruslan Kuprieiev 749fec381f dvc: bump to 1.7.5 2020-09-24 16:26:04 +03:00
Ruslan Kuprieiev 21de21fe37 gha: adjust the test workflow (#4615) 
* gha: move lint check to tests workflow

* gha: tests: don't fail fast

* scripts: remove redundant check
 2020-09-24 16:13:03 +03:00
Ruslan Kuprieiev 0756af5012 setup: remove wheel from install_requires (#4614) 
Kudos to @skshetry for catching this one in https://github.com/iterative/dvc/pull/4606
 2020-09-24 15:39:54 +03:00
Ruslan Kuprieiev 2457633990 gha: build packages (#4607) 2020-09-24 14:48:04 +03:00
Ruslan Kuprieiev 4a82974da4 run-cache: respect cache.shared config option (#4596) 
Fixes #4595
 2020-09-24 01:00:53 +03:00
Ruslan Kuprieiev 688da04d4a tests: cache test directories (#4601) 
* tests: ignore more hdfs tests

* tests: remove unused mark

* gha: don't limit the number of test workers

* cache created test dirs
 2020-09-23 23:22:27 +03:00
Ruslan Kuprieiev 38dd1c9fe7 tests: allow failure for hdfs tests (#4600) 2020-09-23 00:53:32 +03:00
Ruslan Kuprieiev ece7d6ba67 dvc: bump to 1.7.4 2020-09-21 23:50:53 +03:00
Ruslan Kuprieiev cf5d461f4d api: enter state when trying to get a hash (#4584) 
`get_hash` is an internal api, so conventions should be respected.

Fixes #4575
 2020-09-21 23:37:51 +03:00
Ruslan Kuprieiev ff3b686058 dvc: bump to 1.7.3 2020-09-21 20:24:02 +03:00
Ruslan Kuprieiev 0b9cd4c93c gdrive: don't use hacks for empty files (#4588) 
This bug is fixed in PyDrive2 1.6.2.
 2020-09-21 20:23:19 +03:00
Ruslan Kuprieiev de768375ed dvc: bump to 1.7.2 2020-09-15 12:14:49 +03:00
Ruslan Kuprieiev 2f88fa88c4 pydrive2: require six >= 1.13.0 
https://github.com/iterative/PyDrive2/pull/56
 2020-09-15 11:31:52 +03:00
Ruslan Kuprieiev 4ebacf1e0a pydrive2: require >=1.6.1 
https://github.com/iterative/PyDrive2/pull/56
 2020-09-15 11:06:36 +03:00
Ruslan Kuprieiev bc5611968f dvc: bump to 1.7.1 2020-09-14 12:16:14 +03:00
Ruslan Kuprieiev 8c19492ba6 tree: validate resolved remote:// sections (#4569) 
* tree: validate resolved remote:// sections

* dvc: use checksum_jobs instead of hash_jobs

This is a regression introduced during one of the refactorings.
 2020-09-14 12:15:38 +03:00
Ruslan Kuprieiev 61be19b9b5 Update version.py 2020-09-14 07:31:49 +03:00
Ruslan Kuprieiev 424396b068 tree: resolve deep remote:// urls (#4568) 
Fixes #4556
 2020-09-14 05:49:54 +03:00
Ruslan Kuprieiev cf7e534e2d tree: repo: handle dirty state (#4554) 2020-09-11 17:25:33 +03:00
Ruslan Kuprieiev 1ca2dc19c2 tree: open: handle dirty state in RepoTree (#4548) 
We used to use that logic in DvcTree back before we had proper trees.
Now we have a RepoTree, that combines git/workspace tree with the
DvcTree and dirty state clearly belongs to the git/workspace and not
the dvc itself, as dvc state only depends on dvc-files and lock-files.

Prerequisite for #4523
 2020-09-09 18:44:11 +03:00
Ruslan Kuprieiev 61bf1194c3 tests: add gs etag 
Kudos to @skshetry
 2020-09-08 09:29:56 +03:00
Ruslan Kuprieiev ccdcb1f4c1 codecov: remove old token 
Kudos to @skshetry
 2020-09-08 09:24:38 +03:00
Ruslan Kuprieiev 226322d312 gdrive: add workaround for #4507 
Temporary workaround. Proper solution will be introduced in https://github.com/iterative/PyDrive2/pull/48
 2020-09-08 05:27:20 +03:00
Ruslan Kuprieiev 18e80b25ac codecov: add token 2020-09-07 23:16:30 +03:00
Ruslan Kuprieiev 93a298343f run: consolidate run-cache logic (#4535) 
Existing lock files are just an edge case of run-cache.

Prerequisite for #4223
 2020-09-06 17:46:25 +03:00
Ruslan Kuprieiev 636a019289 dvc: compute dir hash without external cache (#4528) 
Fixes #4144
 2020-09-05 16:31:34 +03:00
Ruslan Kuprieiev 4901765cd4 get/import: fix bug when working with mixed dirs (#4516) 
Fixes #4510
 2020-09-02 05:53:09 +03:00
Ruslan Kuprieiev 981dd401fa cache: use get_file_hash instead of get_hash (#4514) 
This is what we've been doing before. No need to try to pull it up from
the state db, as this is a clearly newly generated file that will get
deleted.
 2020-09-02 03:37:17 +03:00
Ruslan Kuprieiev 9887de6a65 dvc: separate cache and tree hashes (#4511) 
Related to #4144 , #3069 , #1676
 2020-09-02 00:30:02 +03:00
Ruslan Kuprieiev fc321683d8 dvc: use HashInfo to interact with cache (#4506) 2020-09-01 03:32:51 +03:00
Ruslan Kuprieiev 56500601bd deps/outs: get rid of self.info (#4502) 
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.

Related to #4144, #3069, #1676
 2020-08-31 22:46:18 +03:00
Ruslan Kuprieiev 54402f3de5 gitignore: ignore /cache instead of cache (#4501) 2020-08-31 05:53:44 +03:00
Ruslan Kuprieiev 1633245769 dvc: bump to 1.6.6 2020-08-31 03:16:40 +03:00
Ruslan Kuprieiev f52763a0b3 dvc: use flatten-dict instead of flatten-json (#4500) 
* dvc: use flatten-dict instead of flatten-json

Fixes #4499

* utils: add [un]flatten

* pyinstaller: add flatten-dict hook
 2020-08-31 03:15:52 +03:00
Ruslan Kuprieiev 4ee3d0d602 dvc: bump to 1.6.5 2020-08-31 00:20:15 +03:00
Ruslan Kuprieiev fcbbfac7f8 dvc: show "Having any troubles?" only on unexpected errors (#4496) 
* dvc: dont catch generic exceptions

* dvc: show "Having any troubles?" only on unexpected errors

Fixes #4126
 2020-08-30 21:08:12 +03:00
Ruslan KuprieievandSaugat Pachhai 1a35128457 dvc: use HashInfo (#4495) 
* dvc: use HashInfo

Related to #4144 , #3069 , #1676

* Update dvc/tree/s3.py

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
 2020-08-30 21:07:57 +03:00
Ruslan Kuprieiev 69c63a8374 dvc: bump to 1.6.4 2020-08-29 21:55:22 +03:00
Ruslan Kuprieiev 4013256c91 external: avoid unnecessary hash computation (#4486) 
* dvc: use cache state when saving downloaded cache

Related to #4485

* tree: dont compute hash from scratch for output dirs

Fixes #4485

* tree: fix get_hash
 2020-08-29 21:15:44 +03:00
Ruslan Kuprieiev be2e0778b4 dvc: bump to 1.6.3 2020-08-28 05:18:18 +03:00
Ruslan Kuprieiev a61291039e tests: use s3 client instead of resource 2020-08-28 05:16:06 +03:00
Ruslan Kuprieiev d5fcf7377d dvc: bump to 1.6.2 2020-08-27 23:11:12 +03:00
Ruslan KuprieievandJorge Orpinel b9d685a650 s3: provide more helpful messages on common errors (#4480) 
* s3: provide more helpful messages on common errors

Fix #4478

* utils: introduce error_link

* Update dvc/tree/s3.py

Co-authored-by: Jorge Orpinel <[email protected]>

* fix tests

* dont forget the link

* remove redundant message

Co-authored-by: Jorge Orpinel <[email protected]>
 2020-08-27 23:09:12 +03:00
Ruslan Kuprieiev 718c9c98ce serialization: dont rely on the default hash type (#4466) 
Prerequisite for #4144 and #3069
 2020-08-25 18:34:09 +03:00
Ruslan Kuprieiev 026c4fa47d dvc: bump to 1.6.1 2020-08-25 02:03:35 +03:00
Ruslan Kuprieiev 6c352eeaf0 hdfs: implement isdir and isfile (#4460) 
Fixes #4332
 2020-08-25 02:02:33 +03:00
Ruslan Kuprieiev d89c16dfe8 deps: freeze pylint at 2.5.3 2020-08-21 17:31:36 +03:00
Ruslan Kuprieiev 41e6c22e7d dvc: move Dvc/RepoTree to dvc/tree (#4427) 
This is where it really belongs.
 2020-08-19 22:30:55 +03:00