Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,437 Commits 7 Branches 579 Tags
112ea1d448393877a8778b4182d67ba7b8439cfe
Commit Graph
7437 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Saugat Pachhai (सौगात) 112ea1d448 dvc.objects: add callbacks for listing and estimating hashes 
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.

The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.

This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
 2022-04-26 07:38:24 +05:45
Saugat Pachhai (सौगात) f39e35445a fs.upload/download: simplify, remove unused kwargs, callbacks 
1. This PR simplifies `fs.upload`/`fs.download`/`fs.download_file`.
2. Removes unused kwargs.
3. `download` uses `dvc.utils.threadpool.ThreadPoolExecutor` which
   supports canceling futures.
4. `download` for directories supports nested/branched callbacks
   and progress bars.
 2022-04-26 06:57:10 +05:45
Paweł Redzyński 7211bd02ed fsspec wrapper: remove can handle dirs 2022-04-25 19:48:20 +03:00
Saugat Pachhai (सौगात) 83795ff3df fs.put_file: add support for uploading fileobjects 
Also changed the callbacks to inherit from our own so that we can
add few more utilities to it.
 2022-04-25 21:23:05 +05:45
Saugat Pachhai (सौगात) 8ead29805f fs: rename getsize to size 2022-04-25 18:52:46 +05:45
Saugat Pachhai fecc81e951 Revert "dvc.objects: add callbacks for listing and estimating hashes" 
This reverts commit 7599c6e20d.
 2022-04-25 18:37:05 +05:45
Saugat Pachhai (सौगात) 7599c6e20d dvc.objects: add callbacks for listing and estimating hashes 
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.

The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.

This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
 2022-04-25 18:36:47 +05:45
Saugat Pachhai (सौगात) 68a3cd9a13 fs: unify fs.base.FileSystem with FSSpecWrapper 
Since all of our filesystems are fsspec-compliant, there's no need for compat wrapper.
 2022-04-25 16:01:06 +09:00
David de la Iglesia Castro a394498067 render.match: Skip errored revisions in get_files 2022-04-22 16:44:52 +02:00
Saugat Pachhai (सौगात) 7708a8e292 gitfs: add regression test for broken stage.collect on gitfs when cwd != repo_root 2022-04-21 21:03:08 +05:45
Saugat Pachhai (सौगात) 23e62a9ff7 fs: make localfs fsspec-compliant 
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.

Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
 2022-04-21 12:44:55 +03:00
Saugat Pachhai (सौगात) fed0bfb2b3 tests: add test for dvc.api.open with remote 
Covers #7341
 2022-04-21 15:12:01 +05:45
Saugat Pachhai (सौगात) f22da81999 tests: make xfail-marked-tests to be strict by default 2022-04-21 15:06:47 +05:45
pre-commit-ci[bot] 22a988b113 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2022-04-21 15:06:31 +05:45
joncrall 77a8e870ff Add log statement to collect repo 2022-04-21 15:06:31 +05:45
David de la Iglesia Castro f20d6a7048 dag: Add --mermaid option. 2022-04-21 14:03:45 +05:45
Saugat Pachhai (सौगात) 6ffba62b75 executors: add support for cancel_futures=True in Python<3.9 
Also adds a kwarg `cancel_on_error` on the constructor to support
cancelling futures on `shutdown`. This is not done by default to
be backward compatible with `concurrent.futures.ThreadPool` executor.
 2022-04-21 14:03:10 +05:45
David de la Iglesia Castro 5c561572b0 exp init: Replace live section with metrics and plots. 
Closes #7534
 2022-04-21 08:55:49 +02:00
Peter Rowlands 444bf27e6f exp show: add internal --no-fetch option to disable temp/queue ref fetch 2022-04-21 15:45:59 +09:00
Peter Rowlands ba82faee53 exp show: don't acquire repo lock 2022-04-21 15:45:59 +09:00
dependabot[bot] 72d57f13bc build(deps-dev): Bump pylint from 2.13.5 to 2.13.7 
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.5 to 2.13.7.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.5...v2.13.7)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
 2022-04-21 08:22:47 +05:45
Saugat Pachhai (सौगात) 02b2ab6804 stage/exp list: alias --names-only to --name-only 
Fixes #7570.
 2022-04-21 06:53:16 +05:45
Ruslan Kuprieiev fd440df5b3 dvcignore: use fs.path instead of os.path 2022-04-20 12:27:26 +03:00
Saugat Pachhai (सौगात) cc8a0ce134 completion: decouple completion related stuff from command 
It also now uses the main parser itself that invokes the command rather than
creating a new parser object.
 2022-04-20 13:29:33 +05:45
Ivan Shcheklein 5c1e7be909 bump pydrive: improves and fixes thread safety 2022-04-20 06:06:28 +05:45
David de la Iglesia Castro 179e080c2f hdfs: Fix _strip_protocol and _checksum 
Match behavior previous to #6946
 2022-04-19 16:22:49 +03:00
Saugat Pachhai (सौगात) f3a7ec7f48 completion: fix dvc exp completion 2022-04-19 17:28:22 +05:45
Saugat Pachhai (सौगात) c0580c740b fs: move NoDirectoriesMixin to http filesystem 2022-04-19 11:54:34 +05:45
dependabot[bot] b4dcadbc9c build(deps-dev): Bump pylint from 2.12.2 to 2.13.5 (#7518) 
build(deps-dev): Bump pylint from 2.12.2 to 2.13.5
 2022-04-19 11:35:10 +05:45
Ruslan Kuprieiev 55219e9089 fs: get rid of CallbackMixin in azure/gs/hdfs/webhdfs 
No longer needed, as all of those filesystems already support callbacks.
 2022-04-18 17:34:06 +03:00
David de la Iglesia Castro 70b57d4703 live: Remove html generation 
Remove commands/live.
Remove LiveTask.
Bump dvclive.
 2022-04-18 15:47:31 +02:00
Saugat Pachhai (सौगात) 5eb16ab05e parsing: use relative paths; force absolute paths to relative 2022-04-18 14:19:56 +03:00
Ruslan Kuprieiev 517f45d45f repofs: migrate to fsspec 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 53c7f44480 repofs: rename _main_repo -> repo 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 9d75e8261b repofs: use repo paths 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev b3b0ec1fba repofs: unify ls/walk/info/exists/etc behaviour 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 0a3f5cbdd5 repofs: base walk on ls 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev bdd21fd481 repofs: use fs.ls 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 394118ef3d repofs: don't assume repofs paths are the same as underlying fs paths 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev f500c66aef repofs: use fs.walk directly 2022-04-18 14:17:20 +03:00
dependabot[bot] 318235f072 build(deps): Bump pyinstaller from 4.10 to 5.0 
Bumps [pyinstaller](https://github.com/pyinstaller/pyinstaller) from 4.10 to 5.0.
- [Release notes](https://github.com/pyinstaller/pyinstaller/releases)
- [Changelog](https://github.com/pyinstaller/pyinstaller/blob/develop/doc/CHANGES.rst)
- [Commits](https://github.com/pyinstaller/pyinstaller/compare/v4.10...v5.0)

---
updated-dependencies:
- dependency-name: pyinstaller
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
 2022-04-18 14:13:52 +08:00
Peter Rowlands ab395dfb7a exp show: prefer exp names for experiments 2022-04-15 14:02:35 +09:00
Nick SorrosandPaweł Redzyński dd9bc116a3 Update test name to be more informative 
Co-authored-by: Paweł Redzyński <[email protected]>
 2022-04-13 18:23:14 +02:00
Nick Sorros 0a3dd7a97f Move test to output and use mocker 2022-04-13 18:23:14 +02:00
Nick Sorros 5da72fc8ff Patch get_remote_odb instead 2022-04-13 18:23:14 +02:00
Nick SorrosandPaweł Redzyński d8dac6f50d Reraise RemoteMissingDepsError as is 
Co-authored-by: Paweł Redzyński <[email protected]>
 2022-04-13 18:23:14 +02:00
Nick Sorros 36c27ed17f Add test for remote missing when adding data dir 2022-04-13 18:23:14 +02:00
pre-commit-ci[bot] 43f5faa652 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2022-04-13 18:23:14 +02:00
Nick Sorros 968f9b38bf Raise informative missing s3 extra 2022-04-13 18:23:14 +02:00
pre-commit-ci[bot] aab7f8af4d [pre-commit.ci] pre-commit autoupdate 
updates:
- [github.com/pre-commit/pre-commit-hooks: v4.1.0 → v4.2.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.1.0...v4.2.0)
 2022-04-12 10:46:46 +09:00