Commit Graph
4133 Commits
Author SHA1 Message Date
dae08255d0 output: use remote kwarg instead of odb (#6600)
* output: use remote kwarg instead of odb

Fixes #6596

* Update dvc/output.py

Co-authored-by: Dave Berenbaum <[email protected]>

* add test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: Dave Berenbaum <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-14 18:34:20 +03:00
Saugat Pachhai 237e9d4354 fs.download: add callback support (#6610)
* fs.download: add callback support

* fix lint issues

* compat for gdrive, use fs to upload text for tests

* fix typing issues

* skip gdrive test

* Update tests/func/test_fs.py
2021-09-14 15:53:16 +05:45
Saugat Pachhaiandpre-commit-ci[bot] 788b4d3bb5 fs.upload: add support for optional callback (#6546)
* fs.upload: add support for optional callback

Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-14 10:55:45 +05:45
Ruslan Kuprieiev 049a856cd9 dvc: bump to 2.7.3 2021-09-13 21:52:48 +03:00
5c8a642500 Added GitAuthError (#6493)
* Specify SSH protocol for `git_remote`

* Update experiments.py

* Added "a valid"

* Format

* Added GitRemoteAuthError

* Added GitRemoteAuthError

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed unused import

* Lint

* Removed GitRemoteAuthError

* Reformatted

* Updated error message

* Use shorter title

* Fix URL

* flake8

* Flake8

* Removed extra line

* Apply suggestions from code review

Co-authored-by: Jorge Orpinel <[email protected]>

* valid -> authenticated

* Added test

* Update experiments.py

* Removed authenticated

* Minor edit

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jorge Orpinel <[email protected]>
2021-09-13 17:56:34 +09:00
Saugat Pachhai d2d6c730c4 objects.transfer: minor refactoring, move lazy taskset inside custom executor (#6591) 2021-09-13 11:34:43 +05:45
Ruslan Kuprieiev 3a100382bc diff: don't forget to use dvcignore and state (#6595) 2021-09-11 00:13:07 +03:00
Saugat Pachhai 4bd6d9f172 fs: merge upload_fobj and upload (#6570)
* fs: merge upload_fobj and upload

* Update dvc/fs/base.py
2021-09-09 13:08:04 +03:00
Batuhan Taskaya ff6e88296d ssh: support passphrases for private keys (#6566) 2021-09-09 01:51:02 +03:00
Ruslan Kuprieiev 2d9f36176a state: get rid of self.fs (#6567) 2021-09-09 01:43:09 +03:00
Ruslan Kuprieiev 31479f02ab dvc: bump to 2.7.2 2021-09-08 20:40:32 +03:00
Batuhan Taskaya b4e0c75a30 http: pass prepared filesystem arguments (#6565) 2021-09-08 20:40:01 +03:00
GaoandPeter Rowlands 6106b1341b Show csv format of experiments (#6468)
* Show csv format of experiments

fix #5989 and #5446 

1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-09-08 16:59:01 +08:00
Casper da Costa-Luis 73c493500b remove tpi.base dependency (#6554)
* remove tpi.base dependency

* potential future-proof constructor
2021-09-08 06:09:52 +00:00
Ruslan Kuprieiev 2b63cedfcd dvc: bump to 2.7.1 2021-09-07 21:16:49 +03:00
Batuhan Taskaya 6b582bd4d9 http: migrate to fsspec & aiohttp (#6525) 2021-09-07 16:41:30 +03:00
Peter Rowlands (변기호) f7a82f6fc0 machine: support instance size & startup script config (#6537)
* machine: install dvc via startup_script by default

* machine: support configuring instance type and startup script

* accept startup_script as a config relpath

* use pushd/popd in default install script
2021-09-07 15:07:22 +09:00
Ruslan Kuprieiev cbd49365c6 dvc: bump to 2.7.0 2021-09-06 20:21:55 +03:00
Saugat Pachhai c944d4af5f extract transfer related logic from odb to fs.utils.transfer (#6539)
* extract transfer related logic from odb to fs.utils.transfer

* don't makedirs on dirinfo

As local odb needs to set a mode for dir, we cannot create a directory
in the transfer
2021-09-06 22:02:00 +05:45
Batuhan Taskaya b8b6ace687 setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01 (#6524)
* setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01

* local: use os.path.lexists() for the time being
2021-09-06 13:44:24 +03:00
Batuhan Taskaya 348dfc0496 s3: inline bucket region caching (#6406) 2021-09-06 12:20:52 +03:00
4cc06339c8 Clean up remotes's exps (#6471)
* Clean up remotes's exps

fix #6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix  #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail

Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
2021-09-05 10:06:07 +08:00
Casper da Costa-LuisandPeter Rowlands 0578643f08 machine: move to tpi[ssh] (#6528)
* machine: move to tpi[ssh]

- depends on https://github.com/iterative/tpi

* lazy import tpi

* fix lint warning

Co-authored-by: Peter Rowlands <[email protected]>
2021-09-03 14:37:24 +09:00
Peter Rowlands (변기호) a69edcc2c8 tpi: lazily import python_terraform (#6526)
* machine: lazy load tpi/python_terraform only when needed

* remove terraform from package installers

* update tpi test imports
2021-09-02 17:09:31 +03:00
Batuhan Taskaya 64bf633b0f ssh: support multiple keys (#6519) 2021-09-02 12:48:34 +03:00
David de la Iglesia Castroandpre-commit-ci[bot] e5cf9b203d Add auto-refresh option to html (#6400)
* Added refresh tag

* Formatted

* Added refresh_seconds to live

* Added refresh test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add refresh_seconds to render

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update html.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-01 17:17:23 +08:00
Peter Rowlands (변기호) 9fc4c60749 machine: add dvc machine ssh (#6501)
* machine: add naive asyncssh shell client

* machine: add `dvc machine ssh` command

* add ssh requirements to terraform install extras

* update asyncssh comments

* fix py <3.7

* fix async mocking in py<3.8

* use mock instead of unittest.mock
2021-09-01 16:20:13 +09:00
Ruslan Kuprieiev 37717a7ef3 hdfs: don't read the whole file into memory (#6498)
* hdfs: don't read the whole file into memory

Oversight from #4973 🤦

* hdfs: use default block size of 128MB
2021-08-27 23:19:47 +03:00
Peter Rowlands (변기호) 4ee98299ee dvc machine: implement initial terraform support (#6462)
* tpi: add naive terraform-provider-iterative wrapper

* executor: add naive backend/terraform wrapper

* setup: add dvc[terraform] install extras

* config: add feature.executor feature flag

* fix named executor config sections

* executor: add poc add/modify/remove/init/destroy commands

* relax python-terraform requirement

* ci: install terraform extras

* tpi: support .tf.json format

* s/executor/machine

* terraform: use separate dirs per machine

* s/init/create/

* reserve local machine name

* add initial ssh skeleton

* add initial machine unit tests

* add initial machine command tests

* add tests for add/remove

* enable feature by default in test env

* close temp pemfile object so it can be used on windows

* cleanup pemfile syntax
2021-08-27 12:46:24 +03:00
Ruslan Kuprieiev 4382b80d3b dvcfile: support remote per output (#6486)
Related to #2095
2021-08-27 12:45:25 +03:00
Ruslan Kuprieiev 95ba68d181 ref: use fs.checksum instead of get_mtime_and_size (#6495) 2021-08-27 12:37:08 +09:00
Paweł Redzyński ab08db5065 plots: introduce support for images (#6431)
* plots: support images initial

* plots: data: unify PlotsData

* plots: data conversion refactor

* plots: renderers: abstract functionality

* plots: uncommenting tests

* plots: self implement grouping

* plots: rename group

* plots: extract rendering tests

* plots: move rendering tests to unit

* plots: add image rendering tests

* plots: dont use dpath to find vega plot

* plots: test find_vega

* plots: test render function

* plots: move rendering out of repo package

* plots: refactoring

* plots: self review refactor
2021-08-26 20:00:31 +03:00
Saugat Pachhai 73811f67d7 fix console appearing on daemon in Windows (#6450)
* fix console appearing on daemon in Windows

* fix entrypoint
2021-08-25 22:15:22 +03:00
b13aec0f2b [6478] Fix 401 unauthorized for directories in basic auth protected http remote (#6479)
* added auth to iter_url and created custom open for http fs

* added default value to auth

* Update dvc/utils/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/fs/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* pr fixes

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-24 18:39:29 +03:00
Ruslan Kuprieiev 13d1276a12 dvc: bump to 2.6.4 2021-08-20 20:00:55 +03:00
Paweł Redzyński a7175501aa ruamel.yaml: adjust to new DuplicatedKeyError (#6464) 2021-08-19 20:02:16 +03:00
GaoandPeter Rowlands 57899ee86d Remove all of the experiments in the workspace (#6394)
* Remove all of the experiments in the workspace

fix #5676
1. remove all the experiments in workspace
2. add tests for it

* Only remove baseline ones

* Add a new arguments --all

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Rename and shortcut for these commands

* Solve the issue from removing a changing list.

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* delete workspace option

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-08-19 10:18:51 +09:00
Saugat Pachhai 4b9a4114fe exp run: pass recursive args to checkout (#6458) 2021-08-18 15:17:33 +05:45
Ruslan Kuprieiev 82e559a51a dvc: bump to 2.6.3 2021-08-17 15:17:48 +03:00
Ruslan Kuprieiev 33c6a9b19b analytics: introduce DVC_NO_ANALYTICS (#6453)
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.

Kudos @skshetry
2021-08-17 15:16:40 +03:00
David de la Iglesia Castro 40abf34c8c Update dvc exp branch help (#6439) 2021-08-17 11:14:44 +09:00
Ruslan Kuprieiev 9fef38e778 dvc: bump to 2.6.2 2021-08-17 02:52:07 +03:00
Batuhan Taskaya b44e1c3646 fetched: correctly order mixed odbs (#6436) 2021-08-17 02:51:08 +03:00
Ruslan Kuprieiev c9b3d9dbae dvc: bump to 2.6.1 2021-08-17 01:49:06 +03:00
Ruslan Kuprieiev c2bf0dc713 analytics: add DVC_ANALYTICS env var (#6448) 2021-08-17 01:48:38 +03:00
Saugat Pachhai 451bd3ebc8 Change ui messaging for the updater (#6437) 2021-08-16 18:31:44 +03:00
Arjun Vikram 41bc81910f plots: upgrade to latest Vega library versions (#6371) 2021-08-16 16:22:09 +02:00
Ruslan Kuprieiev 0aec0e3249 updater: only notify in dvc doctor (#6247)
We've used updater for years now to notify our users ASAP about new
releases. These days dvc is more stable and we've actually been
receiving numerous complains about updater being too annoying and
even introduced a config flag to turn it off, which a lot of people do
and which defeats its purpose. Instead of introducing some complex logic
that would notify more rarely and once for the user instead of once for
every dvc project they have - this PR just turns it off for everything,
except `dvc doctor/version`, which is the command that people use when
they are running into issues and that will give them a hint to just try
out the newest version first, as it might be already fixed.

This PR is a more of a compromise, to get the most out of updater while
removing a pain point of it being too annoying, without investing much
time into it and not having to maintain complex unnecessary logic. We
might get back to introducing something more intricate in the future,
if the need arises.

Fixes #5912
2021-08-16 17:28:49 +05:45
Batuhan Taskaya 8406e605ab fs: find(..., detail=True) now casts values properly (#6432)
* fs: find(..., detail=True) now casts values properly

* bump pydrive2
2021-08-16 14:26:07 +03:00
Ruslan Kuprieiev 13c1314099 dvc: bump to 2.6.0 2021-08-15 17:39:36 +03:00