Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,827 Commits 7 Branches 579 Tags
d2d6c730c4302779651704582aaa820466e31c34
Commit Graph
6827 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Saugat Pachhai d2d6c730c4 objects.transfer: minor refactoring, move lazy taskset inside custom executor (#6591) 2021-09-13 11:34:43 +05:45
Ruslan Kuprieiev 3a100382bc diff: don't forget to use dvcignore and state (#6595) 2021-09-11 00:13:07 +03:00
dependabot[bot]andSaugat Pachhai 61fe8bc59c build(deps): Bump grandalf from 0.6 to 0.7 (#6575) 
* build(deps): Bump grandalf from 0.6 to 0.7

Bumps [grandalf](https://github.com/bdcht/grandalf) from 0.6 to 0.7.
- [Release notes](https://github.com/bdcht/grandalf/releases)
- [Commits](https://github.com/bdcht/grandalf/compare/v0.6...v0.7)

---
updated-dependencies:
- dependency-name: grandalf
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* grandalf dependency require range >=0.6

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
 2021-09-09 16:23:49 +05:45
dependabot[bot] eaac8d08a6 build(deps): Bump hdfs from 2.5.8 to 2.6.0 (#6577) 
Bumps [hdfs](https://github.com/mtth/hdfs) from 2.5.8 to 2.6.0.
- [Release notes](https://github.com/mtth/hdfs/releases)
- [Changelog](https://github.com/mtth/hdfs/blob/master/CHANGES)
- [Commits](https://github.com/mtth/hdfs/commits)

---
updated-dependencies:
- dependency-name: hdfs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-09 16:06:40 +05:45
dependabot[bot] 3522318dc9 build(deps): Bump google-cloud-storage from 1.42.0 to 1.42.1 (#6568) 
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.42.0 to 1.42.1.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.42.0...v1.42.1)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-09 16:04:22 +05:45
Saugat Pachhai 4bd6d9f172 fs: merge upload_fobj and upload (#6570) 
* fs: merge upload_fobj and upload

* Update dvc/fs/base.py
 2021-09-09 13:08:04 +03:00
Saugat Pachhai ef017befbe Move install requires to requirements/default.txt (#6569) 
* Move install requires to requirements/default.txt

In the same spirit as #6560.

* Bump typing_extensions to >=3.10.0.2
 2021-09-09 15:23:01 +05:45
Paweł Redzyński fa31fd1ef4 tests: requirements: remove beautifoulsoup4 (#6571) 2021-09-09 08:59:20 +00:00
Batuhan Taskaya 5592330769 project: formalize extra dependencies (#6560) 2021-09-09 09:01:45 +05:45
Batuhan Taskaya ff6e88296d ssh: support passphrases for private keys (#6566) 2021-09-09 01:51:02 +03:00
Ruslan Kuprieiev 2d9f36176a state: get rid of self.fs (#6567) 2021-09-09 01:43:09 +03:00
Ruslan Kuprieiev 31479f02ab dvc: bump to 2.7.2 2.7.2 2021-09-08 20:40:32 +03:00
Batuhan Taskaya b4e0c75a30 http: pass prepared filesystem arguments (#6565) 2021-09-08 20:40:01 +03:00
Saugat Pachhai 378486dbf5 Update setup.py (#6557) 2021-09-08 15:13:07 +05:45
GaoandPeter Rowlands 6106b1341b Show csv format of experiments (#6468) 
* Show csv format of experiments

fix #5989 and #5446 

1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
 2021-09-08 16:59:01 +08:00
Casper da Costa-Luis 73c493500b remove tpi.base dependency (#6554) 
* remove tpi.base dependency

* potential future-proof constructor
 2021-09-08 06:09:52 +00:00
Casper da Costa-Luis 820d89606c add contrib avatars (#6552) 2021-09-08 11:19:49 +05:45
Ruslan Kuprieiev 2b63cedfcd dvc: bump to 2.7.1 2.7.1 2021-09-07 21:16:49 +03:00
Saugat Pachhai 1744028e09 Allow running package build manually on other PRs (#6553) 2021-09-07 22:28:50 +05:45
Saugat Pachhai 82db0867fe Update hook-dvc.py (#6551) 2021-09-07 22:17:03 +05:45
Batuhan Taskaya 6b582bd4d9 http: migrate to fsspec & aiohttp (#6525) 2021-09-07 16:41:30 +03:00
Saugat Pachhai f9bb38b96f Set timeout for jobs in Github Actions (#6545) 
* Set timeout for jobs in Github Actions

Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.

The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.

Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.

Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.

This should give us early feedback on the issue.

* Update setup.cfg

* Update setup.cfg
 2021-09-07 14:08:22 +05:45
Peter Rowlands (변기호) f7a82f6fc0 machine: support instance size & startup script config (#6537) 
* machine: install dvc via startup_script by default

* machine: support configuring instance type and startup script

* accept startup_script as a config relpath

* use pushd/popd in default install script
 2021-09-07 15:07:22 +09:00
Ruslan Kuprieiev cbd49365c6 dvc: bump to 2.7.0 2.7.0 2.6.5 2021-09-06 20:21:55 +03:00
Saugat Pachhai c944d4af5f extract transfer related logic from odb to fs.utils.transfer (#6539) 
* extract transfer related logic from odb to fs.utils.transfer

* don't makedirs on dirinfo

As local odb needs to set a mode for dir, we cannot create a directory
in the transfer
 2021-09-06 22:02:00 +05:45
Saugat Pachhai 0db0128d28 try to fix flaky dvclive test (#6541) 
Making sure that script takes a bit more time than the await time
I am not really sure this was the cause though.
Also the tests are flaky only in MacOS, but I haven't been able to reproduce.
 2021-09-06 21:37:34 +05:45
Batuhan Taskaya b8b6ace687 setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01 (#6524) 
* setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01

* local: use os.path.lexists() for the time being
 2021-09-06 13:44:24 +03:00
Batuhan Taskaya 348dfc0496 s3: inline bucket region caching (#6406) 2021-09-06 12:20:52 +03:00
Saugat Pachhai db93709f21 Remove travis ci badge (#6538) 
Also stopping from misleading users with always green badge.
 2021-09-06 11:55:15 +03:00
Hongbo Miao 23d6697610 docs: add missing stage name (#6535) 
Fix

> ERROR: `-n|--name` is required
 2021-09-05 02:24:35 +00:00
GaoJorge Orpinelpre-commit-ci[bot]Dave Berenbaum
 		4cc06339c8 Clean up remotes's exps (#6471) 
* Clean up remotes's exps

fix #6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix  #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail

Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
 2021-09-05 10:06:07 +08:00
David de la Iglesia Castroandpre-commit-ci[bot] b99cd0be11 Added dvclive to tests (#6518) 
* Added dvclive to tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Moved dvclive to rest_requirements

* Update setup.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-09-03 22:50:41 +03:00
Casper da Costa-LuisandPeter Rowlands 0578643f08 machine: move to tpi[ssh] (#6528) 
* machine: move to tpi[ssh]

- depends on https://github.com/iterative/tpi

* lazy import tpi

* fix lint warning

Co-authored-by: Peter Rowlands <[email protected]>
 2021-09-03 14:37:24 +09:00
dependabot[bot]andSaugat Pachhai 2778c05e49 build(deps): Remove google-crc32c (#6529) 
* build(deps): Update google-crc32c requirement from <=1.1.2 to <1.1.5

Updates the requirements on [google-crc32c](https://github.com/googleapis/python-crc32c) to permit the latest version.
- [Release notes](https://github.com/googleapis/python-crc32c/releases)
- [Changelog](https://github.com/googleapis/python-crc32c/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-crc32c/compare/v0.0.1...v1.1.4)

---
updated-dependencies:
- dependency-name: google-crc32c
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update test_requirements.txt

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
 2021-09-03 08:20:53 +05:45
Ruslan Kuprieiev 6287b13ba3 gha: allow running tests on other branches (#6520) 
Simplifies development for all contributors.
 2021-09-02 17:55:54 +03:00
Peter Rowlands (변기호) a69edcc2c8 tpi: lazily import python_terraform (#6526) 
* machine: lazy load tpi/python_terraform only when needed

* remove terraform from package installers

* update tpi test imports
 2021-09-02 17:09:31 +03:00
Batuhan Taskaya 64bf633b0f ssh: support multiple keys (#6519) 2021-09-02 12:48:34 +03:00
David de la Iglesia Castroandpre-commit-ci[bot] e5cf9b203d Add auto-refresh option to html (#6400) 
* Added refresh tag

* Formatted

* Added refresh_seconds to live

* Added refresh test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add refresh_seconds to render

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update html.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-09-01 17:17:23 +08:00
Peter Rowlands (변기호) 9fc4c60749 machine: add dvc machine ssh (#6501) 
* machine: add naive asyncssh shell client

* machine: add `dvc machine ssh` command

* add ssh requirements to terraform install extras

* update asyncssh comments

* fix py <3.7

* fix async mocking in py<3.8

* use mock instead of unittest.mock
 2021-09-01 16:20:13 +09:00
Saugat Pachhai b5fe9b4596 restrict google-crc32c for Python3.10 (#6510) 2021-08-31 09:06:28 +05:45
dependabot[bot] 03e692b4c2 build(deps): Bump pytest from 6.2.4 to 6.2.5 (#6509) 
Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.4 to 6.2.5.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/6.2.4...6.2.5)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-31 08:50:20 +05:45
pre-commit-ci[bot] d06a96720f [pre-commit.ci] pre-commit autoupdate (#6504) 
updates:
- [github.com/ambv/black: 21.7b0 → 21.8b0](https://github.com/ambv/black/compare/21.7b0...21.8b0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-08-30 21:31:00 +03:00
Saugat Pachhai ea2a38a207 restrict typing_extensions version for py3.10 (#6503) 2021-08-30 17:19:39 +05:45
Saugat Pachhai 5b9bdac965 add terraform to tests dependencies (#6502) 2021-08-30 17:04:47 +05:45
Ruslan Kuprieiev 5d7476a539 gha: packages: don't forget terraform 2021-08-27 23:43:28 +03:00
Ruslan Kuprieiev 341ab4fe9b gha: packages: don't forget terraform 2021-08-27 23:38:21 +03:00
Ruslan Kuprieiev 37717a7ef3 hdfs: don't read the whole file into memory (#6498) 
* hdfs: don't read the whole file into memory

Oversight from #4973 🤦

* hdfs: use default block size of 128MB
 2021-08-27 23:19:47 +03:00
Peter Rowlands (변기호) 4ee98299ee dvc machine: implement initial terraform support (#6462) 
* tpi: add naive terraform-provider-iterative wrapper

* executor: add naive backend/terraform wrapper

* setup: add dvc[terraform] install extras

* config: add feature.executor feature flag

* fix named executor config sections

* executor: add poc add/modify/remove/init/destroy commands

* relax python-terraform requirement

* ci: install terraform extras

* tpi: support .tf.json format

* s/executor/machine

* terraform: use separate dirs per machine

* s/init/create/

* reserve local machine name

* add initial ssh skeleton

* add initial machine unit tests

* add initial machine command tests

* add tests for add/remove

* enable feature by default in test env

* close temp pemfile object so it can be used on windows

* cleanup pemfile syntax
 2021-08-27 12:46:24 +03:00
Ruslan Kuprieiev 4382b80d3b dvcfile: support remote per output (#6486) 
Related to #2095
 2021-08-27 12:45:25 +03:00
Ruslan Kuprieiev 95ba68d181 ref: use fs.checksum instead of get_mtime_and_size (#6495) 2021-08-27 12:37:08 +09:00