Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,834 Commits 7 Branches 579 Tags
788b4d3bb5e8fbcb4428143a84d44a26cb063599
Commit Graph
6834 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Saugat Pachhaiandpre-commit-ci[bot] 788b4d3bb5 fs.upload: add support for optional callback (#6546) 
* fs.upload: add support for optional callback

Also removes _upload methods in other fses, and replaces them
with fsspec-compatible put_file which supports fsspec callback
even if they are not fsspec based filesystem.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-09-14 10:55:45 +05:45
Saugat Pachhai 40c61c4f62 Use GDriveURLInfo instead of CloudURLInfo (#6608) 
* Use GDriveURLInfo instead of CloudURLInfo

* Update gdrive.py
 2021-09-14 10:18:22 +05:45
dependabot[bot] 1a1294848b build(deps): Bump codecov/codecov-action from 2.0.3 to 2.1.0 (#6606) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2.0.3 to 2.1.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2.0.3...v2.1.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-14 08:46:38 +05:45
dependabot[bot] 93dab26313 build(deps): Bump mock-ssh-server from 0.9.0 to 0.9.1 (#6607) 
Bumps [mock-ssh-server](https://github.com/carletes/mock-ssh-server) from 0.9.0 to 0.9.1.
- [Release notes](https://github.com/carletes/mock-ssh-server/releases)
- [Commits](https://github.com/carletes/mock-ssh-server/compare/0.9.0...0.9.1)

---
updated-dependencies:
- dependency-name: mock-ssh-server
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-14 08:46:08 +05:45
Ruslan Kuprieiev 049a856cd9 dvc: bump to 2.7.3 2.7.3 2021-09-13 21:52:48 +03:00
dependabot[bot] eb6562207b build(deps): Bump adlfs from 2021.8.2 to 2021.9.1 (#6597) 
Bumps [adlfs](https://github.com/dask/adlfs) from 2021.8.2 to 2021.9.1.
- [Release notes](https://github.com/dask/adlfs/releases)
- [Changelog](https://github.com/dask/adlfs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/dask/adlfs/compare/2021.08.2...2021.09.1)

---
updated-dependencies:
- dependency-name: adlfs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-13 17:59:53 +09:00
David de la Iglesia CastroJorge Orpinelpre-commit-ci[bot]
 		5c8a642500 Added GitAuthError (#6493) 
* Specify SSH protocol for `git_remote`

* Update experiments.py

* Added "a valid"

* Format

* Added GitRemoteAuthError

* Added GitRemoteAuthError

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed unused import

* Lint

* Removed GitRemoteAuthError

* Reformatted

* Updated error message

* Use shorter title

* Fix URL

* flake8

* Flake8

* Removed extra line

* Apply suggestions from code review

Co-authored-by: Jorge Orpinel <[email protected]>

* valid -> authenticated

* Added test

* Update experiments.py

* Removed authenticated

* Minor edit

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jorge Orpinel <[email protected]>
 2021-09-13 17:56:34 +09:00
Saugat Pachhai d2d6c730c4 objects.transfer: minor refactoring, move lazy taskset inside custom executor (#6591) 2021-09-13 11:34:43 +05:45
Ruslan Kuprieiev 3a100382bc diff: don't forget to use dvcignore and state (#6595) 2021-09-11 00:13:07 +03:00
dependabot[bot]andSaugat Pachhai 61fe8bc59c build(deps): Bump grandalf from 0.6 to 0.7 (#6575) 
* build(deps): Bump grandalf from 0.6 to 0.7

Bumps [grandalf](https://github.com/bdcht/grandalf) from 0.6 to 0.7.
- [Release notes](https://github.com/bdcht/grandalf/releases)
- [Commits](https://github.com/bdcht/grandalf/compare/v0.6...v0.7)

---
updated-dependencies:
- dependency-name: grandalf
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* grandalf dependency require range >=0.6

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
 2021-09-09 16:23:49 +05:45
dependabot[bot] eaac8d08a6 build(deps): Bump hdfs from 2.5.8 to 2.6.0 (#6577) 
Bumps [hdfs](https://github.com/mtth/hdfs) from 2.5.8 to 2.6.0.
- [Release notes](https://github.com/mtth/hdfs/releases)
- [Changelog](https://github.com/mtth/hdfs/blob/master/CHANGES)
- [Commits](https://github.com/mtth/hdfs/commits)

---
updated-dependencies:
- dependency-name: hdfs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-09 16:06:40 +05:45
dependabot[bot] 3522318dc9 build(deps): Bump google-cloud-storage from 1.42.0 to 1.42.1 (#6568) 
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.42.0 to 1.42.1.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.42.0...v1.42.1)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-09 16:04:22 +05:45
Saugat Pachhai 4bd6d9f172 fs: merge upload_fobj and upload (#6570) 
* fs: merge upload_fobj and upload

* Update dvc/fs/base.py
 2021-09-09 13:08:04 +03:00
Saugat Pachhai ef017befbe Move install requires to requirements/default.txt (#6569) 
* Move install requires to requirements/default.txt

In the same spirit as #6560.

* Bump typing_extensions to >=3.10.0.2
 2021-09-09 15:23:01 +05:45
Paweł Redzyński fa31fd1ef4 tests: requirements: remove beautifoulsoup4 (#6571) 2021-09-09 08:59:20 +00:00
Batuhan Taskaya 5592330769 project: formalize extra dependencies (#6560) 2021-09-09 09:01:45 +05:45
Batuhan Taskaya ff6e88296d ssh: support passphrases for private keys (#6566) 2021-09-09 01:51:02 +03:00
Ruslan Kuprieiev 2d9f36176a state: get rid of self.fs (#6567) 2021-09-09 01:43:09 +03:00
Ruslan Kuprieiev 31479f02ab dvc: bump to 2.7.2 2.7.2 2021-09-08 20:40:32 +03:00
Batuhan Taskaya b4e0c75a30 http: pass prepared filesystem arguments (#6565) 2021-09-08 20:40:01 +03:00
Saugat Pachhai 378486dbf5 Update setup.py (#6557) 2021-09-08 15:13:07 +05:45
GaoandPeter Rowlands 6106b1341b Show csv format of experiments (#6468) 
* Show csv format of experiments

fix #5989 and #5446 

1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
 2021-09-08 16:59:01 +08:00
Casper da Costa-Luis 73c493500b remove tpi.base dependency (#6554) 
* remove tpi.base dependency

* potential future-proof constructor
 2021-09-08 06:09:52 +00:00
Casper da Costa-Luis 820d89606c add contrib avatars (#6552) 2021-09-08 11:19:49 +05:45
Ruslan Kuprieiev 2b63cedfcd dvc: bump to 2.7.1 2.7.1 2021-09-07 21:16:49 +03:00
Saugat Pachhai 1744028e09 Allow running package build manually on other PRs (#6553) 2021-09-07 22:28:50 +05:45
Saugat Pachhai 82db0867fe Update hook-dvc.py (#6551) 2021-09-07 22:17:03 +05:45
Batuhan Taskaya 6b582bd4d9 http: migrate to fsspec & aiohttp (#6525) 2021-09-07 16:41:30 +03:00
Saugat Pachhai f9bb38b96f Set timeout for jobs in Github Actions (#6545) 
* Set timeout for jobs in Github Actions

Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.

The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.

Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.

Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.

This should give us early feedback on the issue.

* Update setup.cfg

* Update setup.cfg
 2021-09-07 14:08:22 +05:45
Peter Rowlands (변기호) f7a82f6fc0 machine: support instance size & startup script config (#6537) 
* machine: install dvc via startup_script by default

* machine: support configuring instance type and startup script

* accept startup_script as a config relpath

* use pushd/popd in default install script
 2021-09-07 15:07:22 +09:00
Ruslan Kuprieiev cbd49365c6 dvc: bump to 2.7.0 2.7.0 2.6.5 2021-09-06 20:21:55 +03:00
Saugat Pachhai c944d4af5f extract transfer related logic from odb to fs.utils.transfer (#6539) 
* extract transfer related logic from odb to fs.utils.transfer

* don't makedirs on dirinfo

As local odb needs to set a mode for dir, we cannot create a directory
in the transfer
 2021-09-06 22:02:00 +05:45
Saugat Pachhai 0db0128d28 try to fix flaky dvclive test (#6541) 
Making sure that script takes a bit more time than the await time
I am not really sure this was the cause though.
Also the tests are flaky only in MacOS, but I haven't been able to reproduce.
 2021-09-06 21:37:34 +05:45
Batuhan Taskaya b8b6ace687 setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01 (#6524) 
* setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01

* local: use os.path.lexists() for the time being
 2021-09-06 13:44:24 +03:00
Batuhan Taskaya 348dfc0496 s3: inline bucket region caching (#6406) 2021-09-06 12:20:52 +03:00
Saugat Pachhai db93709f21 Remove travis ci badge (#6538) 
Also stopping from misleading users with always green badge.
 2021-09-06 11:55:15 +03:00
Hongbo Miao 23d6697610 docs: add missing stage name (#6535) 
Fix

> ERROR: `-n|--name` is required
 2021-09-05 02:24:35 +00:00
GaoJorge Orpinelpre-commit-ci[bot]Dave Berenbaum
 		4cc06339c8 Clean up remotes's exps (#6471) 
* Clean up remotes's exps

fix #6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix  #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail

Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
 2021-09-05 10:06:07 +08:00
David de la Iglesia Castroandpre-commit-ci[bot] b99cd0be11 Added dvclive to tests (#6518) 
* Added dvclive to tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Moved dvclive to rest_requirements

* Update setup.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-09-03 22:50:41 +03:00
Casper da Costa-LuisandPeter Rowlands 0578643f08 machine: move to tpi[ssh] (#6528) 
* machine: move to tpi[ssh]

- depends on https://github.com/iterative/tpi

* lazy import tpi

* fix lint warning

Co-authored-by: Peter Rowlands <[email protected]>
 2021-09-03 14:37:24 +09:00
dependabot[bot]andSaugat Pachhai 2778c05e49 build(deps): Remove google-crc32c (#6529) 
* build(deps): Update google-crc32c requirement from <=1.1.2 to <1.1.5

Updates the requirements on [google-crc32c](https://github.com/googleapis/python-crc32c) to permit the latest version.
- [Release notes](https://github.com/googleapis/python-crc32c/releases)
- [Changelog](https://github.com/googleapis/python-crc32c/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-crc32c/compare/v0.0.1...v1.1.4)

---
updated-dependencies:
- dependency-name: google-crc32c
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update test_requirements.txt

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
 2021-09-03 08:20:53 +05:45
Ruslan Kuprieiev 6287b13ba3 gha: allow running tests on other branches (#6520) 
Simplifies development for all contributors.
 2021-09-02 17:55:54 +03:00
Peter Rowlands (변기호) a69edcc2c8 tpi: lazily import python_terraform (#6526) 
* machine: lazy load tpi/python_terraform only when needed

* remove terraform from package installers

* update tpi test imports
 2021-09-02 17:09:31 +03:00
Batuhan Taskaya 64bf633b0f ssh: support multiple keys (#6519) 2021-09-02 12:48:34 +03:00
David de la Iglesia Castroandpre-commit-ci[bot] e5cf9b203d Add auto-refresh option to html (#6400) 
* Added refresh tag

* Formatted

* Added refresh_seconds to live

* Added refresh test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add refresh_seconds to render

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update html.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-09-01 17:17:23 +08:00
Peter Rowlands (변기호) 9fc4c60749 machine: add dvc machine ssh (#6501) 
* machine: add naive asyncssh shell client

* machine: add `dvc machine ssh` command

* add ssh requirements to terraform install extras

* update asyncssh comments

* fix py <3.7

* fix async mocking in py<3.8

* use mock instead of unittest.mock
 2021-09-01 16:20:13 +09:00
Saugat Pachhai b5fe9b4596 restrict google-crc32c for Python3.10 (#6510) 2021-08-31 09:06:28 +05:45
dependabot[bot] 03e692b4c2 build(deps): Bump pytest from 6.2.4 to 6.2.5 (#6509) 
Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.4 to 6.2.5.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/6.2.4...6.2.5)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-31 08:50:20 +05:45
pre-commit-ci[bot] d06a96720f [pre-commit.ci] pre-commit autoupdate (#6504) 
updates:
- [github.com/ambv/black: 21.7b0 → 21.8b0](https://github.com/ambv/black/compare/21.7b0...21.8b0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-08-30 21:31:00 +03:00
Saugat Pachhai ea2a38a207 restrict typing_extensions version for py3.10 (#6503) 2021-08-30 17:19:39 +05:45