841 Commits
Author SHA1 Message Date
b652670cbf Dependabot/pip/pylint 2.11.1 (#6843)
* build(deps): Bump pylint from 2.9.6 to 2.11.1

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.11.1.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.11.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix pylint problems

* Updation after rebase.

* system: use binary mode

* Set remote execution script encoding type locale

* Update dvc/machine/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-10-20 14:34:10 +03:00
Gao 6de3e734ed Revert "build(deps): Bump pylint from 2.9.6 to 2.11.1 (#6636)" (#6842)
This reverts commit 464620806c.
2021-10-20 12:41:00 +03:00
Gao 4b2bec89ec dvc machine: add a new command rename to it (#6633)
* `dvc machine`: add a new command `rename` to it

fix #6481
1. add new command `dvc machine rename`
2. add a unit test for the call.
3. add several functional tests for it.t push myself fix6481

* Move rename logic to tpi

* use tpi.state_mv instead of rename

* Rebase to a newer version

* Update tpi version

* Remove unused assert and try catch
2021-10-20 15:57:44 +09:00
f964a44e7c Exp show md (#6799)
* Added `--md`/`--show-md` option to `exp show`.

* Added test_show_experiments_md

* Fixed re-defining all_experiments

* Fixed all_experiments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added dest markdown

* Added textwrap.dedent

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update experiments.py

* Apply suggestions from code review

Co-authored-by: Saugat Pachhai <[email protected]>

* Update tests/unit/command/test_compat_flag.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
2021-10-20 11:04:57 +05:45
464620806c build(deps): Bump pylint from 2.9.6 to 2.11.1 (#6636)
* build(deps): Bump pylint from 2.9.6 to 2.11.1

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.11.1.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.11.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix pylint problems
* system: use binary mode

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-10-20 10:32:44 +08:00
966dc8a1ca machine: Add a new command dvc machine status (#6649)
* machine: Add a new command `dvc machine status`

fix#6482
1. add a new command `dvc machien status`
2. add a unit test for the cli call.
3. add a unit test for the shown result.

* Update dvc/command/machine.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update dvc/command/machine.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update machine.py

* Remove sensitive fields

* Update dvc/machine/__init__.py

Co-authored-by: Casper da Costa-Luis <[email protected]>

* Update tests/func/machine/__init__.py

Co-authored-by: Casper da Costa-Luis <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use white list to limit shown fields

Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Casper da Costa-Luis <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-10-19 17:38:11 +09:00
David de la Iglesia Castro 4f16852005 exp/metrics/params diff: Replaced Old/New with a_rev/b_rev (#6814)
* Replaced Old/New with a_rev/b_rev

* Fixed tests
2021-10-18 21:09:02 +03:00
Ruslan Kuprieiev 7f5c53c10e ui: write_json: don't forget json kwargs (#6810)
Regression from #6743

Fixes #6803
2021-10-15 01:30:22 +03:00
David de la Iglesia Castro 7e4f443002 Added test_show_experiments_sort_by (#6798)
* Added test_show_experiments_sort_by

* Added comment

* help formatting
2021-10-14 14:49:00 +03:00
2a3a93314b pathspec 0.9.0 quickfix (#6689)
* pathspec 0.9.0 quickfix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* restrict requirement properly

Co-authored-by: Gao <[email protected]>

* revert change_rule update

* adjust tests to pathspec 0.9.0

* lower bound restrict pathspec

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Gao <[email protected]>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2021-10-14 14:25:03 +03:00
Paweł Redzyński a1b146669c plots: load templates from package (#6550)
* plots: vega: make interactive linear default plot

* plots: stop dumping default templates

* plots: move templates to package initial

* plots: update default plots schema

* plots: write defaults on init

* templates: migrate to importlib.resources

* fixup

* fixup
2021-10-14 08:40:31 +02:00
Saugat PachhaiandRuslan Kuprieiev 8abda93e3e use setuptools_scm and build_requires (#6775)
* use setuptools_scm and build_requires

* remove kwargs from setup.py

* fix tests; fetch in depth in github actions

* use build to build sdist/wheel

* try updating pip and setuptools

* update pip

* remove or unpin wheel

* bring back the version in pyinstaller

* Update scripts/build.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-10-12 17:58:14 +03:00
Ruslan Kuprieiev 5cfe61928e tree: use oids instead of objects (#6785) 2021-10-12 05:39:10 +03:00
Ruslan Kuprieiev 32467ada32 objects: extract size/nfiles to Meta (#6784)
* stage: capture metadata

* tree: capture metadata

* objects: extract size/nfiles from HashInfo/HashFile/Tree to Meta
2021-10-12 03:06:18 +03:00
Saugat Pachhai 4e17279440 exp init: add config support (#6761) 2021-10-08 20:14:01 +05:45
Batuhan Taskaya 4f3b28e268 webhdfs: migrate to fsspec (#6662)
* webhdfs: migrate to fsspec

* skip the tests that can't use hdfs
2021-10-07 17:02:31 +03:00
Batuhan Taskaya 054feae778 hdfs: migrate to fsspec (#6604)
* hdfs: migrate to fsspec

* normalize windows paths when delegating back to fsspec
2021-10-06 15:44:27 +03:00
eggqq007andZhenyu Gao 9c300a0a1c command: output prettify json in tty (#6743)
1. update diff.py,experiments.py,metrics.py,params.py,status.py to
print prettify json
2. rewrite write_json in __init__.py to accept default function
3. add new unit test for write_json in tests/unit/ui/test_console.py
4. Fix #6712

Co-authored-by: Zhenyu Gao <[email protected]>
2021-10-05 20:19:56 +05:45
Saugat Pachhai 1e0b2d7af0 benedict: monkeypatch, performance improvements, bugfixes (#6706)
Fixes #6476, fixes #6423 and closes #6484.
2021-10-01 18:16:47 +09:00
Saugat Pachhai 9a1d74048a cli: add short alias for --show-{csv,json,md} (#6711)
As `--csv`/`--json`/`--md`
2021-09-30 20:38:32 +05:45
Saugat Pachhai 05eaeef012 utils.http: remove open url (#6700)
This was used before on opening http urls as a stream,
but we no longer use those (as we use fsspec's open method).

One thing to note however is that, compared to previous
implementation, fsspec's http filesystem's open is not very
robust to failures.
2021-09-28 19:19:52 +05:45
Saugat Pachhai e7c20aa1d8 better exception messages for schema errors (#6679)
* Generate prettier schema error messages

* add test for #6285
2021-09-28 11:12:16 +05:45
Saugat Pachhai 058377f33e ui: support pager universally (#6692) 2021-09-27 18:11:20 +05:45
Amey Kulkarni fbd8f5c185 Issue 6651: Added alias ls for dvc list (#6683) 2021-09-25 21:14:11 +03:00
Batuhan Taskaya 0bf1802d4b ssh: fix path normalization on Windows (#6517) 2021-09-23 18:23:40 +03:00
Saugat Pachhai eefede92b2 merge yaml parsing and schema validation into one, aka strictyaml.load (#6661) 2021-09-23 00:03:25 +03:00
b286c311f4 reincorporate PR 6111 into current master, re: #4420 (#6419)
* rebased PR 6111 onto master

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use make_tmp_dir fixture in tests

Co-authored-by: U119511 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Alexander Meier <[email protected]>
Co-authored-by: meierale <[email protected]>
2021-09-18 22:36:50 +03:00
Nikita Kodenko 696c704632 tests: replace several unittest.mock usages with mocker (#6544)
* tests: several unittest.mock usages replaced with mocker

* tests: fixes in tests/unit/repo/test_reproduce.py

* tests: rolled back WiP tests
2021-09-17 18:22:34 +03:00
David de la Iglesia Castro 21585d18ab Force sorted order in plots rendering (#6601)
* Force sorted order in plots

* Fix type hint

* Fixed mypy

* Updates test_plots

* Removed reduntant test

* Removed unused import
2021-09-16 14:25:03 +08:00
Batuhan Taskaya 8a97b21b76 ssh: ask password before setting it (#6615) 2021-09-15 15:26:57 +03:00
Saugat Pachhai fbcc3f389c TmpDir: add serialize utilities (#6620) 2021-09-15 16:13:06 +05:45
GaoandPeter Rowlands 18d147e2f8 machine: add machine list&machine modify (#6602)
* machine: add `machine list`&`machine modify`

1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands

* Update dvc/config_schema.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update tests/func/machine/test_machine_config.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Some reviewed problems

1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-09-15 15:40:16 +09:00
Saugat Pachhai 237e9d4354 fs.download: add callback support (#6610)
* fs.download: add callback support

* fix lint issues

* compat for gdrive, use fs to upload text for tests

* fix typing issues

* skip gdrive test

* Update tests/func/test_fs.py
2021-09-14 15:53:16 +05:45
Saugat Pachhaiandpre-commit-ci[bot] 788b4d3bb5 fs.upload: add support for optional callback (#6546)
* fs.upload: add support for optional callback

Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-14 10:55:45 +05:45
Batuhan Taskaya ff6e88296d ssh: support passphrases for private keys (#6566) 2021-09-09 01:51:02 +03:00
Batuhan Taskaya b4e0c75a30 http: pass prepared filesystem arguments (#6565) 2021-09-08 20:40:01 +03:00
GaoandPeter Rowlands 6106b1341b Show csv format of experiments (#6468)
* Show csv format of experiments

fix #5989 and #5446 

1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-09-08 16:59:01 +08:00
Batuhan Taskaya 6b582bd4d9 http: migrate to fsspec & aiohttp (#6525) 2021-09-07 16:41:30 +03:00
4cc06339c8 Clean up remotes's exps (#6471)
* Clean up remotes's exps

fix #6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix  #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail

Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
2021-09-05 10:06:07 +08:00
Casper da Costa-LuisandPeter Rowlands 0578643f08 machine: move to tpi[ssh] (#6528)
* machine: move to tpi[ssh]

- depends on https://github.com/iterative/tpi

* lazy import tpi

* fix lint warning

Co-authored-by: Peter Rowlands <[email protected]>
2021-09-03 14:37:24 +09:00
Peter Rowlands (변기호) a69edcc2c8 tpi: lazily import python_terraform (#6526)
* machine: lazy load tpi/python_terraform only when needed

* remove terraform from package installers

* update tpi test imports
2021-09-02 17:09:31 +03:00
Batuhan Taskaya 64bf633b0f ssh: support multiple keys (#6519) 2021-09-02 12:48:34 +03:00
David de la Iglesia Castroandpre-commit-ci[bot] e5cf9b203d Add auto-refresh option to html (#6400)
* Added refresh tag

* Formatted

* Added refresh_seconds to live

* Added refresh test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add refresh_seconds to render

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update html.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-01 17:17:23 +08:00
Peter Rowlands (변기호) 9fc4c60749 machine: add dvc machine ssh (#6501)
* machine: add naive asyncssh shell client

* machine: add `dvc machine ssh` command

* add ssh requirements to terraform install extras

* update asyncssh comments

* fix py <3.7

* fix async mocking in py<3.8

* use mock instead of unittest.mock
2021-09-01 16:20:13 +09:00
Peter Rowlands (변기호) 4ee98299ee dvc machine: implement initial terraform support (#6462)
* tpi: add naive terraform-provider-iterative wrapper

* executor: add naive backend/terraform wrapper

* setup: add dvc[terraform] install extras

* config: add feature.executor feature flag

* fix named executor config sections

* executor: add poc add/modify/remove/init/destroy commands

* relax python-terraform requirement

* ci: install terraform extras

* tpi: support .tf.json format

* s/executor/machine

* terraform: use separate dirs per machine

* s/init/create/

* reserve local machine name

* add initial ssh skeleton

* add initial machine unit tests

* add initial machine command tests

* add tests for add/remove

* enable feature by default in test env

* close temp pemfile object so it can be used on windows

* cleanup pemfile syntax
2021-08-27 12:46:24 +03:00
Ruslan Kuprieiev 4382b80d3b dvcfile: support remote per output (#6486)
Related to #2095
2021-08-27 12:45:25 +03:00
Paweł Redzyński ab08db5065 plots: introduce support for images (#6431)
* plots: support images initial

* plots: data: unify PlotsData

* plots: data conversion refactor

* plots: renderers: abstract functionality

* plots: uncommenting tests

* plots: self implement grouping

* plots: rename group

* plots: extract rendering tests

* plots: move rendering tests to unit

* plots: add image rendering tests

* plots: dont use dpath to find vega plot

* plots: test find_vega

* plots: test render function

* plots: move rendering out of repo package

* plots: refactoring

* plots: self review refactor
2021-08-26 20:00:31 +03:00
Paweł Redzyński a7175501aa ruamel.yaml: adjust to new DuplicatedKeyError (#6464) 2021-08-19 20:02:16 +03:00
GaoandPeter Rowlands 57899ee86d Remove all of the experiments in the workspace (#6394)
* Remove all of the experiments in the workspace

fix #5676
1. remove all the experiments in workspace
2. add tests for it

* Only remove baseline ones

* Add a new arguments --all

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Rename and shortcut for these commands

* Solve the issue from removing a changing list.

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* delete workspace option

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-08-19 10:18:51 +09:00
Ruslan Kuprieiev 33c6a9b19b analytics: introduce DVC_NO_ANALYTICS (#6453)
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.

Kudos @skshetry
2021-08-17 15:16:40 +03:00