Commit Graph
6788 Commits
Author SHA1 Message Date
Saugat Pachhai b5fe9b4596 restrict google-crc32c for Python3.10 (#6510) 2021-08-31 09:06:28 +05:45
dependabot[bot] 03e692b4c2 build(deps): Bump pytest from 6.2.4 to 6.2.5 (#6509)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.4 to 6.2.5.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/6.2.4...6.2.5)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-31 08:50:20 +05:45
pre-commit-ci[bot] d06a96720f [pre-commit.ci] pre-commit autoupdate (#6504)
updates:
- [github.com/ambv/black: 21.7b0 → 21.8b0](https://github.com/ambv/black/compare/21.7b0...21.8b0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-30 21:31:00 +03:00
Saugat Pachhai ea2a38a207 restrict typing_extensions version for py3.10 (#6503) 2021-08-30 17:19:39 +05:45
Saugat Pachhai 5b9bdac965 add terraform to tests dependencies (#6502) 2021-08-30 17:04:47 +05:45
Ruslan Kuprieiev 5d7476a539 gha: packages: don't forget terraform 2021-08-27 23:43:28 +03:00
Ruslan Kuprieiev 341ab4fe9b gha: packages: don't forget terraform 2021-08-27 23:38:21 +03:00
Ruslan Kuprieiev 37717a7ef3 hdfs: don't read the whole file into memory (#6498)
* hdfs: don't read the whole file into memory

Oversight from #4973 🤦

* hdfs: use default block size of 128MB
2021-08-27 23:19:47 +03:00
Peter Rowlands (변기호) 4ee98299ee dvc machine: implement initial terraform support (#6462)
* tpi: add naive terraform-provider-iterative wrapper

* executor: add naive backend/terraform wrapper

* setup: add dvc[terraform] install extras

* config: add feature.executor feature flag

* fix named executor config sections

* executor: add poc add/modify/remove/init/destroy commands

* relax python-terraform requirement

* ci: install terraform extras

* tpi: support .tf.json format

* s/executor/machine

* terraform: use separate dirs per machine

* s/init/create/

* reserve local machine name

* add initial ssh skeleton

* add initial machine unit tests

* add initial machine command tests

* add tests for add/remove

* enable feature by default in test env

* close temp pemfile object so it can be used on windows

* cleanup pemfile syntax
2021-08-27 12:46:24 +03:00
Ruslan Kuprieiev 4382b80d3b dvcfile: support remote per output (#6486)
Related to #2095
2021-08-27 12:45:25 +03:00
Ruslan Kuprieiev 95ba68d181 ref: use fs.checksum instead of get_mtime_and_size (#6495) 2021-08-27 12:37:08 +09:00
Paweł Redzyński ab08db5065 plots: introduce support for images (#6431)
* plots: support images initial

* plots: data: unify PlotsData

* plots: data conversion refactor

* plots: renderers: abstract functionality

* plots: uncommenting tests

* plots: self implement grouping

* plots: rename group

* plots: extract rendering tests

* plots: move rendering tests to unit

* plots: add image rendering tests

* plots: dont use dpath to find vega plot

* plots: test find_vega

* plots: test render function

* plots: move rendering out of repo package

* plots: refactoring

* plots: self review refactor
2021-08-26 20:00:31 +03:00
Saugat Pachhai 73811f67d7 fix console appearing on daemon in Windows (#6450)
* fix console appearing on daemon in Windows

* fix entrypoint
2021-08-25 22:15:22 +03:00
HimanshuandJorge Orpinel 60d4b4791b Update README.rst (#6461)
* Update README.rst

Fixes typos and makes the related technologies stand out from their description.

* Apply suggestions from code review

* Update README.rst

* Wrapped modified portions to 80 chars or less

Left others untouched to simplify review

Co-authored-by: Jorge Orpinel <[email protected]>
2021-08-25 22:14:27 +03:00
dependabot[bot] ceb0964fcf build(deps): Bump codecov/codecov-action from 2.0.2 to 2.0.3 (#6487)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-25 08:22:52 +05:45
b13aec0f2b [6478] Fix 401 unauthorized for directories in basic auth protected http remote (#6479)
* added auth to iter_url and created custom open for http fs

* added default value to auth

* Update dvc/utils/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/fs/http.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* pr fixes

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-24 18:39:29 +03:00
Ruslan Kuprieiev 13d1276a12 dvc: bump to 2.6.4 2.6.4 2021-08-20 20:00:55 +03:00
Ruslan Kuprieiev 7da20a7c74 setup: upgrade s3fs
Fixes https://github.com/iterative/dvc/issues/6466
2021-08-20 19:43:14 +03:00
Paweł Redzyński a7175501aa ruamel.yaml: adjust to new DuplicatedKeyError (#6464) 2021-08-19 20:02:16 +03:00
GaoandPeter Rowlands 57899ee86d Remove all of the experiments in the workspace (#6394)
* Remove all of the experiments in the workspace

fix #5676
1. remove all the experiments in workspace
2. add tests for it

* Only remove baseline ones

* Add a new arguments --all

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Rename and shortcut for these commands

* Solve the issue from removing a changing list.

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/repo/experiments/remove.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* delete workspace option

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-08-19 10:18:51 +09:00
Saugat Pachhai 4b9a4114fe exp run: pass recursive args to checkout (#6458) 2021-08-18 15:17:33 +05:45
Ruslan Kuprieiev 82e559a51a dvc: bump to 2.6.3 2.6.3 2021-08-17 15:17:48 +03:00
Ruslan Kuprieiev 33c6a9b19b analytics: introduce DVC_NO_ANALYTICS (#6453)
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.

Kudos @skshetry
2021-08-17 15:16:40 +03:00
Saugat Pachhai 14cc748139 Increase coverage target to 90%. 2021-08-17 12:23:35 +05:45
David de la Iglesia Castro 40abf34c8c Update dvc exp branch help (#6439) 2021-08-17 11:14:44 +09:00
Ruslan Kuprieiev 9fef38e778 dvc: bump to 2.6.2 2.6.2 2021-08-17 02:52:07 +03:00
Batuhan Taskaya b44e1c3646 fetched: correctly order mixed odbs (#6436) 2021-08-17 02:51:08 +03:00
Ruslan Kuprieiev c9b3d9dbae dvc: bump to 2.6.1 2.6.1 2021-08-17 01:49:06 +03:00
Ruslan Kuprieiev c2bf0dc713 analytics: add DVC_ANALYTICS env var (#6448) 2021-08-17 01:48:38 +03:00
Saugat Pachhai 451bd3ebc8 Change ui messaging for the updater (#6437) 2021-08-16 18:31:44 +03:00
Arjun Vikram 41bc81910f plots: upgrade to latest Vega library versions (#6371) 2021-08-16 16:22:09 +02:00
Ruslan Kuprieiev 0aec0e3249 updater: only notify in dvc doctor (#6247)
We've used updater for years now to notify our users ASAP about new
releases. These days dvc is more stable and we've actually been
receiving numerous complains about updater being too annoying and
even introduced a config flag to turn it off, which a lot of people do
and which defeats its purpose. Instead of introducing some complex logic
that would notify more rarely and once for the user instead of once for
every dvc project they have - this PR just turns it off for everything,
except `dvc doctor/version`, which is the command that people use when
they are running into issues and that will give them a hint to just try
out the newest version first, as it might be already fixed.

This PR is a more of a compromise, to get the most out of updater while
removing a pain point of it being too annoying, without investing much
time into it and not having to maintain complex unnecessary logic. We
might get back to introducing something more intricate in the future,
if the need arises.

Fixes #5912
2021-08-16 17:28:49 +05:45
Batuhan Taskaya 8406e605ab fs: find(..., detail=True) now casts values properly (#6432)
* fs: find(..., detail=True) now casts values properly

* bump pydrive2
2021-08-16 14:26:07 +03:00
Saugat Pachhai 1bb7f39e14 add support for Python 3.10 (#6422)
* add support for Python 3.10

pyarrow is not available, so we are skipping tests for it.
And, restricting `hdfs` extras to be not installed in Python3.10.

Pathlib removed `_init` method and `init` parameter, so just working
around that. It's not a big change.

* skip windows on 3.10; update pip/wheel/setuptools during install

pywin32 is not yet available for 3.10 Python, so skipping those for now

* install libgit2 with Python3.10 as pygit2 wheel is not available

* install libgit2

* only run on macOS, libgit2-1.1 not available on ubuntu
2021-08-16 16:24:28 +05:45
dependabot[bot] a285d2ba47 build(deps): Bump pygments from 2.9.0 to 2.10.0 (#6430)
Bumps [pygments](https://github.com/pygments/pygments) from 2.9.0 to 2.10.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.9.0...2.10.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-16 10:03:34 +03:00
Ruslan Kuprieiev 13c1314099 dvc: bump to 2.6.0 2.6.0 2021-08-15 17:39:36 +03:00
Ruslan Kuprieiev 27b54198a4 pyinstaller: build: check dvc doctor output (#6425) 2021-08-15 17:30:17 +03:00
Saugat Pachhai a7f01b65a5 Unpin networkx
Networkx v2.6.2 now does not require any additional requirements. All of the other 2.6 releases have been yanked. So, it's safe for us to unpin it.
2021-08-13 14:53:10 +05:45
Gao de97a49401 Remove a special queued experiments (#6393)
* Remove a special queued experiments

fix #6157
1. dvc exp remove not accept queued experiments name
2. add some tests for this feature

* Extract tests and add revision support

1. Extract remove experiments to a new file.
2. revision can be used to remove special queued experiment

* Accept shortened revisions

* Split removing committed and queued exp functions

* Api name change

* Return to the old API name

* shorten some functions

* Error message change

* Better test cases, more corner case

* Still raise exception in a mixed case
2021-08-13 15:15:21 +08:00
dependabot[bot]andBatuhan Taskaya 4c957bf4ab build(deps): bump google-cloud-storage from 1.41.1 to 1.42.0 (#6415)
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.41.1 to 1.42.0.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.41.1...v1.42.0)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: Batuhan Taskaya <[email protected]>
2021-08-12 13:15:34 +03:00
6dd435eefe Push running checkpoint to remote (#6332)
* Push running checkpoint to remote

Fix #6182

* Update dvc/repo/experiments/executor/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Get the branch name

* Finish this PR

1. move env name to dvc/env.py.
2. add some tests for it.

* Update after finally commit

* Some problems found in the review

1. change the behavior of self remote
2. do not use string DVC_EXP_AUTO_PUSH
3. downgrade the logger level in auto push
4. use full branch ref

* Update dvc/repo/experiments/executor/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* add a remote check before auto push

* Split current one env into two

* Rename DVC_EXP_CHECKPOINT_PUSH to DVC_EXP_AUTO_PUSH

* value error

* Remove getenv_bool switch to env2bool

* Better on handling Dulwich exceptions

* Name changing removing prints

* Update dvc/env.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Some changes in review

* Add validation to repo URL

* Add new type of SCMError

* Update dvc/repo/experiments/executor/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Use spy tests call counts and args

* Update dvc/scm/git/backend/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/scm/git/backend/dulwich.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove validate_git_repo to validate_git_remote

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-12 15:17:08 +08:00
Saugat Pachhai fb96b13f13 Fix issue while sending analytics report (#6407)
* demonstrate issue while sending analytics report

When sending analytics, we are not flushing the data to the temp file
which causes failure at times, especially on Windows

* close file before calling daemonized analytics send command

This way, the contents bufferred will get a chance to be flushed.
2021-08-10 16:07:46 +03:00
Peter Rowlands (변기호) 6c646a7070 pygit: account for locked dirs on windows during checkout (#6408) 2021-08-10 15:18:32 +03:00
dependabot[bot] 955aedee13 build(deps): bump jaraco-windows from 5.5.0 to 5.6.0 (#6396)
Bumps [jaraco-windows](https://github.com/jaraco/jaraco.windows) from 5.5.0 to 5.6.0.
- [Release notes](https://github.com/jaraco/jaraco.windows/releases)
- [Changelog](https://github.com/jaraco/jaraco.windows/blob/main/CHANGES.rst)
- [Commits](https://github.com/jaraco/jaraco.windows/compare/v5.5.0...v5.6.0)

---
updated-dependencies:
- dependency-name: jaraco-windows
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-10 15:16:15 +03:00
Ruslan Kuprieiev e99e6cd367 repofs: use underlying fs.download to download files (#6401) 2021-08-10 13:12:56 +03:00
Batuhan Taskaya 1f8278252a gdrive: migrate to pydrive2.fs (#6399) 2021-08-10 12:59:44 +03:00
dependabot[bot] 38cd7d01c2 build(deps): bump wheel from 0.36.2 to 0.37.0 (#6404)
Bumps [wheel](https://github.com/pypa/wheel) from 0.36.2 to 0.37.0.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/master/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.36.2...0.37.0)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-10 08:35:38 +05:45
Maxime RischardandRuslan Kuprieiev 63b710b7c0 More granular read-write lock requests in the run() and reproduce() methods of the Stage class (#5815)
* Remove rwlock decorator for the run() method of the Stage class.

Instead of pre-emptively grabbing read locks on the dependencies
and write locks on the outputs when calling the run() method,
delegate the locking mechanism to other methods.
This results in more selective locking that is less likely
to cause a LockError when executing steps in parallel.

* Remove rwlocked decorator from reproduce() method of Stage class.

* Reinstate read-lock (not write) for run method of Stage class

* fix bugs and typos

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-08-10 01:49:21 +03:00
Ruslan Kuprieiev d97a19e97d refodb: cache ref objects (#6392)
* refodb: cache ref objects

* odb: ignore FileNotFoundError when removing
2021-08-09 13:53:38 +03:00
Saugat Pachhai fd76e0991f cleanup index, and add test for used_objs (#6390) 2021-08-06 15:35:01 +05:45