Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,391 Commits 7 Branches 579 Tags
6a406abde8bc7a2e8b9d200483463a9268fd3263
Commit Graph
231 Commits
Author SHA1 Message Date
Ivan Shcheklein 7ca055e868 fix: bump PyDrive2 to fix Gdrive remote download 2020-05-06 20:11:33 -07:00
Casper da Costa-Luis 26e203242f gdrive: download: stream & add progress (#3722) 
* wip

* gdrive: add progress

Part of #2865
See https://github.com/iterative/dvc/issues/2865#issuecomment-622984785

* gdrive: move towards next pydrive2 release

- depends on https://github.com/iterative/PyDrive2/pull/30

* update to latest pydrive>=1.4.11

* avoid unneeded API call

* progress: gdrive: ensure proper bar_format
 2020-05-06 17:02:16 +03:00
Paweł RedzyńskiandRuslan Kuprieiev e553511085 Metrics - plotting for multiple revisions initial (#3577) 
* init

* rename to plot data insertion basig on dicts update

* revision support

* roll back revision

* plot makedirs for backward compatibility

* log path

* pretty plot link to visualization page

* make target default title

* efiop review

* efiop review

* plot multiple initial

* add some missing metric file tests

* proper id generation

* proper id generation

* add confusion matrix template

* refactor tests

* plot from dvct file

* plot from dvct

* brush up commands

* fix confusion matrix multiple plot

* plot: change confusion matrix data schema

* should be working as intended

* support for src file in dvct files

* minor fixes

* plot: support json templates

* plot: rename confusion template

* plot: polish command behaviour

* fix test for json

* plot: test command

* some minor fixes for tests

* plot: unit test loading

* plot: unit test loading

* plot: handle TODOS

* cleanup

* use mocker

* plot: support tsv

* plot: command refactoring

* plot: fix windows issues with tests

* plot: test: some more windows fixes

* plot: _load_from_revisions complexity fix

* plot: reduce complexity

* plot: complexity reduction

* plot: deepsource suggestions

* plot: move template path evaluation

* fixup

* fixup

* exception on no datafile and no template

* json metric load with OrderedDict

* plot: improve handling non-existing files on revisions

* plot: improve handling non-existing files on revisions

* change default plot path

* some exceptions and fixes

* add yaml metrics support

* fixup

* some more suggestions

* default filename fix

* efiop review requests

* log exception on failur

* move revisions deduction to commands

* json templates

* extract template filling to separate method

* some parsing improvements

* add columns functionality

* extract default data transformation to separate method

* plot: initial support for jsonpath

* plot: rename columns to filters, tests are dict based

* plot: fixups

* plot: refactoring

* repo: plot: convert to package

* plot: data loading refactor, support searching for data

* plot: raise if wrong fields provided

* plot: command description

* plot: default: pass y axis info for default plot

* plot: get rid of fieldnames, expect ordered data

* plot: handle default plot in separate method

* plot: fix default

* plot: command option names fixes

* refactoring

* fixes

* plot: provide option for stdout redirection

* plot: rename show-json to no-html

* plot: add no-csv-header option

* plot: improve error message for wrongly structured metric

* plot: match template name exactly, whit suffix appended only

* plot: dmpetrov and ivan review

* plot: refactor --stdout help message

* plot: move template to repo/plot

* plot: add -x and -y options

* plot: add -x and -y options

* plot: command: order change

* plot: scatter

* plot: rename confusion matrix template, new name generation format

* plot: add title anchor

* plot: review from jorgeorpinel

* plot: rename filter and result options to select and file

* plot: add --title, --x-title, --y-title

* plot: xlab ylab

* Update dvc/repo/plot/template.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plot/template.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* efiop review

* plot: bash completion

* plot: static code analysis fixes

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-05-01 22:45:03 +03:00
Dmitry ChichkovandRuslan Kuprieiev 40c6b567e3 setup: relax python-dateutil pip version constraint to include v2.8.2. #3701 (#3702) 
* Relax python-dateutil pip version constraint to include v2.8.2.  #3701

[WIP]. Attempt to relax upper constraints on the version of python-dateutil, to include 2.8.1 and 2.8.2. 

Original constraint was:
    "python-dateutil<2.8.1,>=2.1",  # Consolidates azure-blob-storage and boto3

* Update setup.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-04-29 23:15:13 +03:00
Ruslan Kuprieiev 6167702345 flake8: freeze to 3.7.9 2020-04-24 10:50:24 +03:00
Marcel Ribeiro Dantas 211f56c043 Updates latest version of PyDrive2 (#3624) 
This latest version of PyDrive2 fixes an issue for Google Drive remote management in machines with an old version of httplib2.
 2020-04-12 18:39:40 +03:00
Ivan Shcheklein 1b8e0725d8 gdrive: bump pydrive2 version to fix large file uploads (#3592) 2020-04-05 04:34:38 +03:00
Casper da Costa-Luis b9d560c584 progress: better threading (#3583) 
Fixes #3565
Continues #3500 <- #2905
 2020-04-02 22:10:15 +01:00
Casper da Costa-Luis eeb6b60fe6 progress: limit maximum running bars (#3500) 
* progress: limit maximum running bars

Fixes #2905
Closes #3453

* progress: add hidden message for overflow bars

* progress: fix nested bars properly

Just use latest `tqdm` :)
 2020-03-29 20:54:32 +03:00
SdgJlbl d8ac973362 Update PyYAML required version (#3545) 2020-03-27 21:21:21 +02:00
Max Hora 42e3b8aa25 gdrive: config option to move remote file to trash only (#3478) 2020-03-25 20:39:39 +02:00
Ruslan Kuprieiev 9de8d1e0fa params: use dpath for xpaths (#3531) 
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:

```
angles:
    - 2
    - 3
    - 100
    - 3.2
    - 36.2
    - 3.6
```

because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
 2020-03-25 06:17:09 +02:00
Saugat PachhaiandCasper da Costa-Luis 109ded4068 dvc: support Python3.8 (#3463) 
* py38: make wheels

- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows

* use realpath to resolve symbolic links

* do not build pkgs for Windows and Linux on Python3.8

* reorder

* pyarrow: document regression fix

Co-authored-by: Casper da Costa-Luis <[email protected]>
 2020-03-16 19:18:33 +00:00
Andrew HareAndrew HareRuslan Kuprieiev
 		c4164b3497 Use pre-commit for Git hooks (#3406) 
* Initial design of using pre-commit for Git hooks

* Refactor to reuse existing code and provide merge capabilities

* Linter fixes

* More cleanup

* scm: make use_precommit_tool optional

* dvc: install our own hooks

* install: fix typo

* install: fix bugs in pre-commit integration

* install: don't forget 'always_run' for post-checkout

* install: no need to run pre-commit install if conf file doesn't exist

* install: fix bugs

* install: use python hooks

* add .pre-commit-hooks.yaml

Co-authored-by: Andrew Hare <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-03-16 20:34:01 +02:00
Max Risuhin 3ff804b3b2 gdrive: Google Service account support (#3269) 2020-03-11 15:49:33 +02:00
Saugat Pachhai 7f94518008 deps: upgrade jsonpath_ng to latest version (#3464) 
Fix #3456
 2020-03-10 11:18:52 +02:00
Alexander Schepanovski 357a6dd467 dvc: optimize building graph (#3443) 
Use trie structure to efficiently check out, dep and stage paths being
prefixes of each other.
 2020-03-09 12:13:07 +02:00
rxxg d7d5d8fcca Remove Linux pyfastcopy (#3412) 
Fixes #3395
 2020-02-26 22:35:48 +02:00
Ruslan Kuprieiev 432e399f44 deps: remove pyfastcopy 
https://github.com/iterative/dvc/issues/3395
 2020-02-25 15:53:43 +02:00
Casper da Costa-Luis ae99e1b02b metadata: add Zenodo, tidy badges and keywords (#3373) 
* docs: add Zenodo badge

* docs: fix rst image syntax

* docs: collect and re-use badges

* metadata: add zenodo

Fixes #2978

* metadata: sync keywords

* docs: add citation in readme

* docs: one-line badges
 2020-02-21 04:01:59 +02:00
Ruslan Kuprieiev 130180cc5c setup.py: remove gitdb2 dependency and update gitpython 2020-02-17 18:12:30 +02:00
Ruslan Kuprieiev 72f1313cc3 setup: fix gitpython version 2020-02-17 17:54:53 +02:00
Ruslan Kuprieiev 4986395998 setup.py: limit gitpython version 
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=122147&view=logs&jobId=4f922444-fdfe-5dcf-b824-02f86439ef14&j=4f922444-fdfe-5dcf-b824-02f86439ef14&t=937c195f-508d-5135-dc9f-d4b5730df0f7
 2020-02-17 17:03:33 +02:00
Mr. Outis 753aef4910 black: bump version 19.3b0 -> 19.10b0 (#3307) 2020-02-12 02:41:24 +02:00
Ruslan Kuprieiev afcbc7e17a deps: update pydrive2 version 2020-02-09 18:48:38 +02:00
Ruslan Kuprieiev de1f67afbd Merge pull request #3244 from jorgeorpinel/master 
diff/metics diff: update argument descriptions, et al.
 2020-02-03 02:15:54 +02:00
Ruslan Kuprieiev fd6bd29303 setup: move pydot to install_requires 
https://github.com/iterative/dvc/issues/3267
 2020-01-31 22:51:41 +02:00
Jorge Orpinel 039f196dd8 Update Dmitry's email address in several places 
per https://github.com/iterative/dvc/pull/3244#pullrequestreview-349888033
 2020-01-30 12:25:40 -06:00
Jorge Orpinel aa849ab2d5 Merge branch 'master' of github.com:iterative/dvc 2020-01-30 12:24:22 -06:00
Jorge Orpinel 1672cdfb06 Update Dmitry's email address ) 2020-01-29 00:39:46 -06:00
Max Risuhin 7b0e44e293 gdrive: support corpora and driveId 2020-01-28 11:50:21 -08:00
Ruslan Kuprieiev 2f1224b27b setup: update pydrive2 version 2020-01-24 09:47:52 +02:00
Ruslan Kuprieiev 1eb82ac707 dvc: replace flatten_dict with flatten_json 
flatten_json has a conda package, while flatten_dict doesn't and I'm not
willing to maintain yet another conda package :)
 2020-01-21 00:14:22 +02:00
Ruslan Kuprieiev 6ebd053c0f Merge pull request #3165 from mroutis/remove-py2-compat 
setup: remove py2 backports
 2020-01-17 04:28:22 +02:00
Ruslan Kuprieiev 74bbc3bc04 Merge pull request #3135 from rxxg/master 
Perf improvement for file copies
 2020-01-16 18:46:58 +02:00
Mr. Outis 48326620ec setup: remove py2 backports 2020-01-15 19:13:01 -06:00
Ruslan Kuprieiev 97a29a1f7e dvc: use pydrive2 instead of pydrive 
PyDrive2 is our maintained fork.
 2020-01-15 19:13:29 +02:00
rxxg e170852e97 Merge branch 'master' into master 2020-01-15 12:53:40 +01:00
rxxg 8fca139246 Use released version of speedcopy library 2020-01-15 12:50:43 +01:00
Ruslan Kuprieiev 9cafef1fdd Merge pull request #3051 from efiop/2995 
metrics: introduce `diff`
 2020-01-14 14:18:40 +02:00
Ruslan Kuprieiev f18bfdc674 setup: fix typo 2020-01-14 11:22:29 +02:00
Ruslan Kuprieiev ebb0243e87 setup: rollback pydrive dep 
pypi doesn't allow direct github deps https://travis-ci.com/iterative/dvc/jobs/275282583
 2020-01-14 11:15:11 +02:00
rxxg 674541f922 Use pyfastcopy for non-Windows systems using Python < 3.8 2020-01-14 09:32:09 +01:00
rxxg 811fc5317c Use speedcopy module with bugfix 2020-01-14 09:32:09 +01:00
rxxg 09223442f5 Use speedcopy module for faster copies across network boundaries 2020-01-14 09:32:09 +01:00
Ruslan Kuprieiev e393e29ef5 metrics: introduce diff 
This first implementation is based on existing `dvc metrics show`
functionality, but is also able to auto-detect json to properly
show diff for it.
 2020-01-13 06:45:48 +02:00
Max Risuhin 57111a7788 gdrive: add support for teamDriveId 2020-01-11 07:41:12 -08:00
Ruslan Kuprieiev c064865c80 build: use build-requirements.txt 2020-01-11 04:55:34 +02:00
Ruslan Kuprieiev ba45c9cd5b setup: get rid of py2 reqs 2020-01-08 02:16:06 +02:00
Ruslan Kuprieiev 381b279e53 Merge pull request #3022 from mroutis/remove-future-imports 
py3: remove future imports
 2019-12-31 04:08:22 +02:00