Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-04 15:28:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,669 Commits 7 Branches 579 Tags
1.2.0
Commit Graph
255 Commits
Author SHA1 Message Date
Peter Rowlands (변기호) a56c8bbab6 Experiments proof of concept (#4199) 
* experiments: add initial experiments.show

* only includes metrics + params (no code/data information)

* experiments: basic `dvc experiments show` functionality

* experiments: clone and run repro inside clone workspace

* repro: add -e/--experiment option

* experiments: hash experiments to identify duplicates

* experiments: update show

* experiments: checkout experiment after running repro

* experiments: add `dvc experiments checkout`

* experiments: add simple `dvc experiments diff` command

* just shows combined output from `metrics diff` and `plots diff` for
  now

* add very simple tests

* add some useful output messages

* experiments: suppress help messages while feature is in development

* experiment related commands will be accessible but hidden from the
  default command help messages

* add simple experiments command tests

* experiments: disable feature by default

* experiments only enabled in test environment (DVC_TEST) or when
  core.experiments config option is true

* use fetch on checkout missing revs
 2020-07-17 05:41:11 +03:00
faucct 332d2198f4 Bump networkx (#4219) 
To avoid conflict with https://github.com/aws-cloudformation/cfn-python-lint
 2020-07-16 13:57:26 +03:00
Ruslan Kuprieiev 375998c664 tests: hdfs: use hadoop/docker fixtures (#4183) 
Fixed #4054
 2020-07-10 21:38:48 +03:00
Ruslan Kuprieiev 0899b277c0 tests: launch oss/azure containers from fixtures (#4178) 
* tests: launch oss/azure containers from fixtures

Related to #4054

* ci: hdfs: remove unneeded fix

We are using pyarrow 0.17.0 already.
 2020-07-08 19:54:17 +03:00
Casper da Costa-Luis 5d2ac9fca2 completion: update dependency (#4148) 
* completion: use new simpler `shtab` API

* completion: use FILE|DIR
 2020-07-02 00:15:39 +03:00
Ruslan Kuprieiev 27af048d3e azure: use knack directly instead of azure-cli-core (#4147) 2020-07-01 22:54:25 +03:00
Ruslan Kuprieiev 00c135f7d5 setup: limit flufl version 
flufl.lock released 4.0 in which there is no finalize method anymore, hence
```
dvc/lock.py:143:16: E1101: Instance of 'HardlinkLock' has no 'finalize' member (no-member)
```
 2020-07-01 01:55:30 +03:00
Ruslan Kuprieiev 0275a0d8d8 azure: use CLI config (#4137) 
Fixes #4129
 2020-06-30 14:13:25 +03:00
Saugat Pachhai 1c0f5cdf4b pylint: enable additional checks (#4122) 
Created a separate pylint plugin to disable some checks for tests.
 2020-06-29 19:15:05 +03:00
Ruslan Kuprieiev 83a2afc7b0 setup: update shtab 2020-06-27 02:36:56 +03:00
Saugat Pachhai ad306d7bf3 Enable most of the pylint checks (#4111) 2020-06-25 18:10:14 +03:00
Saugat Pachhai b81ace54df Enable pylint checks (#4092) 
* pylint: enable checks for imports

* ci: add github action for pylint

* enable pylint on pre-commit

* Install requirements

* use local pylint installation rather than of pre-commit's

* fix import errors

* more checks

* Silence unused imports

* Make a copy of dict
 2020-06-24 18:02:39 +03:00
Casper da Costa-Luis 061513a2ae completion: misc fixes (#4076) 
* command: add metavars

- makes help output look the same as before #3921

* deps: update shtab version

Part of #4070
 2020-06-22 02:09:43 +03:00
Casper da Costa-Luis fa0f182028 completion: autogenerate (#3921) 
* cli: split main_parser

* cli: basic parser tree printing

* cli: major completion tidy

* completion: more missing root commands

* completion: much automagic such beauty

- fixes #2985 automatically generate bash completion

* tests: bash completion

* tests: completion: tidy script a little

* minor tidy

* command: positional name consistency

* completion: bash: add logging

* double-check renames and consistency

* cli: add completion subcommand

* ci: completion: update test

* minor logging

* more completion logging verbosity

* fix completion of files

* add shtab

* purge unneeded completion => shtab

* completion: neaten UI

* completion: use `shtab>=0.0.2`

* completion: purge hardcoded completions!

* revert unneeded renames for now

* fix isort

* completion: tidy CLI API

* fix snap build

* fix snap build again

* snap: internally build completion

* allow repo-less completion

* command: tidy headless logic

* use CmdBaseNoRepo

* better DVC-file completion

* merge command.choices -> command.completion

* drop choices class

* remove completion optionals

* tidy docs
 2020-06-19 04:54:20 +03:00
Saugat Pachhai 4af3919062 Add flake8 extensions (#4067) 
* remove unnecessary generators/comprehensions

* Use dict literals

* remove unused variables

* misc fixes

ignore .iteritems, is a false +ve
remove function call in default argument

* Flake8 config: complexity, show source on err, show err counts

* flake8: add extensions

* remove flake8-string-format

* Add flake8-debugger support

* Add flake8-string-format extension

* remove flake8-debugger

It's having some problems to install on Windows in Travis
 2020-06-18 14:36:30 +03:00
Ruslan Kuprieiev 51a8c782ae tests: create remote fixtures (#4063) 
* tests: use remote fixtures

* tests: use remote fixtures for data cloud tests

* tests: treat remotes as tmp dirs
 2020-06-18 01:36:13 +03:00
Casper da Costa-Luis 4d140a9944 gdrive: add open (#3916) 
* gdrive: add open

Fixes #3408
Related #2865
Fixes #3897

* dependency: add gdrive

* test: api: open: gdrive

* rollback pydrive dependency

* Revert "dependency: add gdrive"

This reverts commit fd333261d2e9fe1364cc278138c79890a1f5fe7a.

* tests: remotes: GDrive: fix get_url

* tests: api: fully test GDrive

* minor typo fix

* attempt gdrive tests fix

* fix gdrive credentials config

* replace GDrivePathNotFound => FileMissingError

* fix test_open_external[GDrive]

* add ensure_dir_scm

* minor exception fix
 2020-06-13 23:53:25 +03:00
Saugat Pachhai c1e5b10cbd remove unnecessary dependencies (#3987) 
Fix #3971
 2020-06-09 12:43:34 +03:00
Saugat Pachhai b47b2c5673 tests: update mock-ssh-server (#3986) 2020-06-09 12:39:10 +03:00
Ruslan Kuprieiev 036ce9398e tests: move mock-ssh-server git dependency (#3953) 
This one doesn't allow us to release new pre-releases, because pypi
forbids that:

https://travis-ci.com/github/iterative/dvc/jobs/343343967#L2447

Let's install it separately for now, until there is a new official
mock-ssh-server release.
 2020-06-04 05:54:29 +03:00
Saugat Pachhai ec312d8601 use fork of mockssh-server (#3910) 2020-05-30 08:31:59 +05:45
Ruslan Kuprieiev b7ad2dc841 test: fix hdfs/azurite/oss tests (#3864) 
* travis: fix typo

* tests: azure: fix expected etag format

* test: ssh: increase the number of tries

* travis: don't install docker

* dvc: update pyarrow

* tests: ssh: don't leak connections

* remote: don't fail when cache/remote doesn't have prefix subdir
 2020-05-23 23:46:40 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792) 
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
 2020-05-14 01:46:42 +03:00
Saugat PachhaiandRuslan Kuprieiev 666e203bbb params/metrics: diff: implement tabulated markdown output (--show-md) (#3757) 
* params/metrics: diff: implement tabulated markdown output

Via `--show-md` on `diff`s.

* table: show None on missing val

* tests: use `dedent`, add tests for markdown

* diff: add --show-md

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-05-10 18:39:51 +03:00
Ivan Shcheklein 7ca055e868 fix: bump PyDrive2 to fix Gdrive remote download 2020-05-06 20:11:33 -07:00
Casper da Costa-Luis 26e203242f gdrive: download: stream & add progress (#3722) 
* wip

* gdrive: add progress

Part of #2865
See https://github.com/iterative/dvc/issues/2865#issuecomment-622984785

* gdrive: move towards next pydrive2 release

- depends on https://github.com/iterative/PyDrive2/pull/30

* update to latest pydrive>=1.4.11

* avoid unneeded API call

* progress: gdrive: ensure proper bar_format
 2020-05-06 17:02:16 +03:00
Paweł RedzyńskiandRuslan Kuprieiev e553511085 Metrics - plotting for multiple revisions initial (#3577) 
* init

* rename to plot data insertion basig on dicts update

* revision support

* roll back revision

* plot makedirs for backward compatibility

* log path

* pretty plot link to visualization page

* make target default title

* efiop review

* efiop review

* plot multiple initial

* add some missing metric file tests

* proper id generation

* proper id generation

* add confusion matrix template

* refactor tests

* plot from dvct file

* plot from dvct

* brush up commands

* fix confusion matrix multiple plot

* plot: change confusion matrix data schema

* should be working as intended

* support for src file in dvct files

* minor fixes

* plot: support json templates

* plot: rename confusion template

* plot: polish command behaviour

* fix test for json

* plot: test command

* some minor fixes for tests

* plot: unit test loading

* plot: unit test loading

* plot: handle TODOS

* cleanup

* use mocker

* plot: support tsv

* plot: command refactoring

* plot: fix windows issues with tests

* plot: test: some more windows fixes

* plot: _load_from_revisions complexity fix

* plot: reduce complexity

* plot: complexity reduction

* plot: deepsource suggestions

* plot: move template path evaluation

* fixup

* fixup

* exception on no datafile and no template

* json metric load with OrderedDict

* plot: improve handling non-existing files on revisions

* plot: improve handling non-existing files on revisions

* change default plot path

* some exceptions and fixes

* add yaml metrics support

* fixup

* some more suggestions

* default filename fix

* efiop review requests

* log exception on failur

* move revisions deduction to commands

* json templates

* extract template filling to separate method

* some parsing improvements

* add columns functionality

* extract default data transformation to separate method

* plot: initial support for jsonpath

* plot: rename columns to filters, tests are dict based

* plot: fixups

* plot: refactoring

* repo: plot: convert to package

* plot: data loading refactor, support searching for data

* plot: raise if wrong fields provided

* plot: command description

* plot: default: pass y axis info for default plot

* plot: get rid of fieldnames, expect ordered data

* plot: handle default plot in separate method

* plot: fix default

* plot: command option names fixes

* refactoring

* fixes

* plot: provide option for stdout redirection

* plot: rename show-json to no-html

* plot: add no-csv-header option

* plot: improve error message for wrongly structured metric

* plot: match template name exactly, whit suffix appended only

* plot: dmpetrov and ivan review

* plot: refactor --stdout help message

* plot: move template to repo/plot

* plot: add -x and -y options

* plot: add -x and -y options

* plot: command: order change

* plot: scatter

* plot: rename confusion matrix template, new name generation format

* plot: add title anchor

* plot: review from jorgeorpinel

* plot: rename filter and result options to select and file

* plot: add --title, --x-title, --y-title

* plot: xlab ylab

* Update dvc/repo/plot/template.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plot/template.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* efiop review

* plot: bash completion

* plot: static code analysis fixes

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-05-01 22:45:03 +03:00
Dmitry ChichkovandRuslan Kuprieiev 40c6b567e3 setup: relax python-dateutil pip version constraint to include v2.8.2. #3701 (#3702) 
* Relax python-dateutil pip version constraint to include v2.8.2.  #3701

[WIP]. Attempt to relax upper constraints on the version of python-dateutil, to include 2.8.1 and 2.8.2. 

Original constraint was:
    "python-dateutil<2.8.1,>=2.1",  # Consolidates azure-blob-storage and boto3

* Update setup.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-04-29 23:15:13 +03:00
Ruslan Kuprieiev 6167702345 flake8: freeze to 3.7.9 2020-04-24 10:50:24 +03:00
Marcel Ribeiro Dantas 211f56c043 Updates latest version of PyDrive2 (#3624) 
This latest version of PyDrive2 fixes an issue for Google Drive remote management in machines with an old version of httplib2.
 2020-04-12 18:39:40 +03:00
Ivan Shcheklein 1b8e0725d8 gdrive: bump pydrive2 version to fix large file uploads (#3592) 2020-04-05 04:34:38 +03:00
Casper da Costa-Luis b9d560c584 progress: better threading (#3583) 
Fixes #3565
Continues #3500 <- #2905
 2020-04-02 22:10:15 +01:00
Casper da Costa-Luis eeb6b60fe6 progress: limit maximum running bars (#3500) 
* progress: limit maximum running bars

Fixes #2905
Closes #3453

* progress: add hidden message for overflow bars

* progress: fix nested bars properly

Just use latest `tqdm` :)
 2020-03-29 20:54:32 +03:00
SdgJlbl d8ac973362 Update PyYAML required version (#3545) 2020-03-27 21:21:21 +02:00
Max Hora 42e3b8aa25 gdrive: config option to move remote file to trash only (#3478) 2020-03-25 20:39:39 +02:00
Ruslan Kuprieiev 9de8d1e0fa params: use dpath for xpaths (#3531) 
Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:

```
angles:
    - 2
    - 3
    - 100
    - 3.2
    - 36.2
    - 3.6
```

because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
 2020-03-25 06:17:09 +02:00
Saugat PachhaiandCasper da Costa-Luis 109ded4068 dvc: support Python3.8 (#3463) 
* py38: make wheels

- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows

* use realpath to resolve symbolic links

* do not build pkgs for Windows and Linux on Python3.8

* reorder

* pyarrow: document regression fix

Co-authored-by: Casper da Costa-Luis <[email protected]>
 2020-03-16 19:18:33 +00:00
Andrew HareAndrew HareRuslan Kuprieiev
 		c4164b3497 Use pre-commit for Git hooks (#3406) 
* Initial design of using pre-commit for Git hooks

* Refactor to reuse existing code and provide merge capabilities

* Linter fixes

* More cleanup

* scm: make use_precommit_tool optional

* dvc: install our own hooks

* install: fix typo

* install: fix bugs in pre-commit integration

* install: don't forget 'always_run' for post-checkout

* install: no need to run pre-commit install if conf file doesn't exist

* install: fix bugs

* install: use python hooks

* add .pre-commit-hooks.yaml

Co-authored-by: Andrew Hare <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-03-16 20:34:01 +02:00
Max Risuhin 3ff804b3b2 gdrive: Google Service account support (#3269) 2020-03-11 15:49:33 +02:00
Saugat Pachhai 7f94518008 deps: upgrade jsonpath_ng to latest version (#3464) 
Fix #3456
 2020-03-10 11:18:52 +02:00
Alexander Schepanovski 357a6dd467 dvc: optimize building graph (#3443) 
Use trie structure to efficiently check out, dep and stage paths being
prefixes of each other.
 2020-03-09 12:13:07 +02:00
rxxg d7d5d8fcca Remove Linux pyfastcopy (#3412) 
Fixes #3395
 2020-02-26 22:35:48 +02:00
Ruslan Kuprieiev 432e399f44 deps: remove pyfastcopy 
https://github.com/iterative/dvc/issues/3395
 2020-02-25 15:53:43 +02:00
Casper da Costa-Luis ae99e1b02b metadata: add Zenodo, tidy badges and keywords (#3373) 
* docs: add Zenodo badge

* docs: fix rst image syntax

* docs: collect and re-use badges

* metadata: add zenodo

Fixes #2978

* metadata: sync keywords

* docs: add citation in readme

* docs: one-line badges
 2020-02-21 04:01:59 +02:00
Ruslan Kuprieiev 130180cc5c setup.py: remove gitdb2 dependency and update gitpython 2020-02-17 18:12:30 +02:00
Ruslan Kuprieiev 72f1313cc3 setup: fix gitpython version 2020-02-17 17:54:53 +02:00
Ruslan Kuprieiev 4986395998 setup.py: limit gitpython version 
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=122147&view=logs&jobId=4f922444-fdfe-5dcf-b824-02f86439ef14&j=4f922444-fdfe-5dcf-b824-02f86439ef14&t=937c195f-508d-5135-dc9f-d4b5730df0f7
 2020-02-17 17:03:33 +02:00
Mr. Outis 753aef4910 black: bump version 19.3b0 -> 19.10b0 (#3307) 2020-02-12 02:41:24 +02:00
Ruslan Kuprieiev afcbc7e17a deps: update pydrive2 version 2020-02-09 18:48:38 +02:00
Ruslan Kuprieiev de1f67afbd Merge pull request #3244 from jorgeorpinel/master 
diff/metics diff: update argument descriptions, et al.
 2020-02-03 02:15:54 +02:00