Commit Graph
6815 Commits
Author SHA1 Message Date
Batuhan Taskaya b4e0c75a30 http: pass prepared filesystem arguments (#6565) 2021-09-08 20:40:01 +03:00
Saugat Pachhai 378486dbf5 Update setup.py (#6557) 2021-09-08 15:13:07 +05:45
GaoandPeter Rowlands 6106b1341b Show csv format of experiments (#6468)
* Show csv format of experiments

fix #5989 and #5446 

1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-09-08 16:59:01 +08:00
Casper da Costa-Luis 73c493500b remove tpi.base dependency (#6554)
* remove tpi.base dependency

* potential future-proof constructor
2021-09-08 06:09:52 +00:00
Casper da Costa-Luis 820d89606c add contrib avatars (#6552) 2021-09-08 11:19:49 +05:45
Ruslan Kuprieiev 2b63cedfcd dvc: bump to 2.7.1 2.7.1 2021-09-07 21:16:49 +03:00
Saugat Pachhai 1744028e09 Allow running package build manually on other PRs (#6553) 2021-09-07 22:28:50 +05:45
Saugat Pachhai 82db0867fe Update hook-dvc.py (#6551) 2021-09-07 22:17:03 +05:45
Batuhan Taskaya 6b582bd4d9 http: migrate to fsspec & aiohttp (#6525) 2021-09-07 16:41:30 +03:00
Saugat Pachhai f9bb38b96f Set timeout for jobs in Github Actions (#6545)
* Set timeout for jobs in Github Actions

Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.

The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.

Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.

Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.

This should give us early feedback on the issue.

* Update setup.cfg

* Update setup.cfg
2021-09-07 14:08:22 +05:45
Peter Rowlands (변기호) f7a82f6fc0 machine: support instance size & startup script config (#6537)
* machine: install dvc via startup_script by default

* machine: support configuring instance type and startup script

* accept startup_script as a config relpath

* use pushd/popd in default install script
2021-09-07 15:07:22 +09:00
Ruslan Kuprieiev cbd49365c6 dvc: bump to 2.7.0 2.7.0 2.6.5 2021-09-06 20:21:55 +03:00
Saugat Pachhai c944d4af5f extract transfer related logic from odb to fs.utils.transfer (#6539)
* extract transfer related logic from odb to fs.utils.transfer

* don't makedirs on dirinfo

As local odb needs to set a mode for dir, we cannot create a directory
in the transfer
2021-09-06 22:02:00 +05:45
Saugat Pachhai 0db0128d28 try to fix flaky dvclive test (#6541)
Making sure that script takes a bit more time than the await time
I am not really sure this was the cause though.
Also the tests are flaky only in MacOS, but I haven't been able to reproduce.
2021-09-06 21:37:34 +05:45
Batuhan Taskaya b8b6ace687 setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01 (#6524)
* setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01

* local: use os.path.lexists() for the time being
2021-09-06 13:44:24 +03:00
Batuhan Taskaya 348dfc0496 s3: inline bucket region caching (#6406) 2021-09-06 12:20:52 +03:00
Saugat Pachhai db93709f21 Remove travis ci badge (#6538)
Also stopping from misleading users with always green badge.
2021-09-06 11:55:15 +03:00
Hongbo Miao 23d6697610 docs: add missing stage name (#6535)
Fix

> ERROR: `-n|--name` is required
2021-09-05 02:24:35 +00:00
4cc06339c8 Clean up remotes's exps (#6471)
* Clean up remotes's exps

fix #6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix  #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail

Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
2021-09-05 10:06:07 +08:00
David de la Iglesia Castroandpre-commit-ci[bot] b99cd0be11 Added dvclive to tests (#6518)
* Added dvclive to tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Moved dvclive to rest_requirements

* Update setup.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-03 22:50:41 +03:00
Casper da Costa-LuisandPeter Rowlands 0578643f08 machine: move to tpi[ssh] (#6528)
* machine: move to tpi[ssh]

- depends on https://github.com/iterative/tpi

* lazy import tpi

* fix lint warning

Co-authored-by: Peter Rowlands <[email protected]>
2021-09-03 14:37:24 +09:00
dependabot[bot]andSaugat Pachhai 2778c05e49 build(deps): Remove google-crc32c (#6529)
* build(deps): Update google-crc32c requirement from <=1.1.2 to <1.1.5

Updates the requirements on [google-crc32c](https://github.com/googleapis/python-crc32c) to permit the latest version.
- [Release notes](https://github.com/googleapis/python-crc32c/releases)
- [Changelog](https://github.com/googleapis/python-crc32c/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-crc32c/compare/v0.0.1...v1.1.4)

---
updated-dependencies:
- dependency-name: google-crc32c
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update test_requirements.txt

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
2021-09-03 08:20:53 +05:45
Ruslan Kuprieiev 6287b13ba3 gha: allow running tests on other branches (#6520)
Simplifies development for all contributors.
2021-09-02 17:55:54 +03:00
Peter Rowlands (변기호) a69edcc2c8 tpi: lazily import python_terraform (#6526)
* machine: lazy load tpi/python_terraform only when needed

* remove terraform from package installers

* update tpi test imports
2021-09-02 17:09:31 +03:00
Batuhan Taskaya 64bf633b0f ssh: support multiple keys (#6519) 2021-09-02 12:48:34 +03:00
David de la Iglesia Castroandpre-commit-ci[bot] e5cf9b203d Add auto-refresh option to html (#6400)
* Added refresh tag

* Formatted

* Added refresh_seconds to live

* Added refresh test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add refresh_seconds to render

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update html.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-09-01 17:17:23 +08:00
Peter Rowlands (변기호) 9fc4c60749 machine: add dvc machine ssh (#6501)
* machine: add naive asyncssh shell client

* machine: add `dvc machine ssh` command

* add ssh requirements to terraform install extras

* update asyncssh comments

* fix py <3.7

* fix async mocking in py<3.8

* use mock instead of unittest.mock
2021-09-01 16:20:13 +09:00
Saugat Pachhai b5fe9b4596 restrict google-crc32c for Python3.10 (#6510) 2021-08-31 09:06:28 +05:45
dependabot[bot] 03e692b4c2 build(deps): Bump pytest from 6.2.4 to 6.2.5 (#6509)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.4 to 6.2.5.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/6.2.4...6.2.5)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-31 08:50:20 +05:45
pre-commit-ci[bot] d06a96720f [pre-commit.ci] pre-commit autoupdate (#6504)
updates:
- [github.com/ambv/black: 21.7b0 → 21.8b0](https://github.com/ambv/black/compare/21.7b0...21.8b0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-30 21:31:00 +03:00
Saugat Pachhai ea2a38a207 restrict typing_extensions version for py3.10 (#6503) 2021-08-30 17:19:39 +05:45
Saugat Pachhai 5b9bdac965 add terraform to tests dependencies (#6502) 2021-08-30 17:04:47 +05:45
Ruslan Kuprieiev 5d7476a539 gha: packages: don't forget terraform 2021-08-27 23:43:28 +03:00
Ruslan Kuprieiev 341ab4fe9b gha: packages: don't forget terraform 2021-08-27 23:38:21 +03:00
Ruslan Kuprieiev 37717a7ef3 hdfs: don't read the whole file into memory (#6498)
* hdfs: don't read the whole file into memory

Oversight from #4973 🤦

* hdfs: use default block size of 128MB
2021-08-27 23:19:47 +03:00
Peter Rowlands (변기호) 4ee98299ee dvc machine: implement initial terraform support (#6462)
* tpi: add naive terraform-provider-iterative wrapper

* executor: add naive backend/terraform wrapper

* setup: add dvc[terraform] install extras

* config: add feature.executor feature flag

* fix named executor config sections

* executor: add poc add/modify/remove/init/destroy commands

* relax python-terraform requirement

* ci: install terraform extras

* tpi: support .tf.json format

* s/executor/machine

* terraform: use separate dirs per machine

* s/init/create/

* reserve local machine name

* add initial ssh skeleton

* add initial machine unit tests

* add initial machine command tests

* add tests for add/remove

* enable feature by default in test env

* close temp pemfile object so it can be used on windows

* cleanup pemfile syntax
2021-08-27 12:46:24 +03:00
Ruslan Kuprieiev 4382b80d3b dvcfile: support remote per output (#6486)
Related to #2095
2021-08-27 12:45:25 +03:00
Ruslan Kuprieiev 95ba68d181 ref: use fs.checksum instead of get_mtime_and_size (#6495) 2021-08-27 12:37:08 +09:00
Paweł Redzyński ab08db5065 plots: introduce support for images (#6431)
* plots: support images initial

* plots: data: unify PlotsData

* plots: data conversion refactor

* plots: renderers: abstract functionality

* plots: uncommenting tests

* plots: self implement grouping

* plots: rename group

* plots: extract rendering tests

* plots: move rendering tests to unit

* plots: add image rendering tests

* plots: dont use dpath to find vega plot

* plots: test find_vega

* plots: test render function

* plots: move rendering out of repo package

* plots: refactoring

* plots: self review refactor
2021-08-26 20:00:31 +03:00
Saugat Pachhai 73811f67d7 fix console appearing on daemon in Windows (#6450)
* fix console appearing on daemon in Windows

* fix entrypoint
2021-08-25 22:15:22 +03:00
HimanshuandJorge Orpinel 60d4b4791b Update README.rst (#6461)
* Update README.rst

Fixes typos and makes the related technologies stand out from their description.

* Apply suggestions from code review

* Update README.rst

* Wrapped modified portions to 80 chars or less

Left others untouched to simplify review

Co-authored-by: Jorge Orpinel <[email protected]>
2021-08-25 22:14:27 +03:00
dependabot[bot] ceb0964fcf build(deps): Bump codecov/codecov-action from 2.0.2 to 2.0.3 (#6487)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-25 08:22:52 +05:45
b13aec0f2b [6478] Fix 401 unauthorized for directories in basic auth protected http remote (#6479)
* added auth to iter_url and created custom open for http fs

* added default value to auth

* Update dvc/utils/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/fs/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* pr fixes

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-24 18:39:29 +03:00
Ruslan Kuprieiev 13d1276a12 dvc: bump to 2.6.4 2.6.4 2021-08-20 20:00:55 +03:00
Ruslan Kuprieiev 7da20a7c74 setup: upgrade s3fs
Fixes https://github.com/iterative/dvc/issues/6466
2021-08-20 19:43:14 +03:00
Paweł Redzyński a7175501aa ruamel.yaml: adjust to new DuplicatedKeyError (#6464) 2021-08-19 20:02:16 +03:00
GaoandPeter Rowlands 57899ee86d Remove all of the experiments in the workspace (#6394)
* Remove all of the experiments in the workspace

fix #5676
1. remove all the experiments in workspace
2. add tests for it

* Only remove baseline ones

* Add a new arguments --all

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Rename and shortcut for these commands

* Solve the issue from removing a changing list.

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* delete workspace option

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-08-19 10:18:51 +09:00
Saugat Pachhai 4b9a4114fe exp run: pass recursive args to checkout (#6458) 2021-08-18 15:17:33 +05:45
Ruslan Kuprieiev 82e559a51a dvc: bump to 2.6.3 2.6.3 2021-08-17 15:17:48 +03:00
Ruslan Kuprieiev 33c6a9b19b analytics: introduce DVC_NO_ANALYTICS (#6453)
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.

Kudos @skshetry
2021-08-17 15:16:40 +03:00