Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-07 07:18:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,447 Commits 7 Branches 579 Tags
0d48598e4bc16811eeddf585a5262f36ec5261cc
Commit Graph
7447 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Saugat Pachhai (सौगात) 0d48598e4b fs.ssh: remove unused vars 2022-04-27 17:56:08 +05:45
Saugat Pachhai (सौगात) 5d64f6504f fs.s3: remove custom put_file/get_file 2022-04-27 17:53:17 +05:45
Saugat Pachhai (सौगात) 3d54039614 scm: fix progress bar, show progress and repo being cloned 
Before, as we were not passing `total`, the Tqdm progress bar would not
show the bar or not show any progress on the bar. As we get that value lazily,
we should force it to always show the bar.
Also updated the progress bar to say which repo is being cloned.
 2022-04-27 16:42:28 +05:45
Saugat Pachhai (सौगात) cda855b97a fs: add cat/cat_file/cat_ranges/pipe/pipe_file/touch methods 2022-04-27 15:57:17 +05:45
David de la Iglesia Castro bea7d56bd5 plots: Fix repo with removed plots 
Changes in dvc-render 0.0.5 caused error.
 2022-04-27 11:53:30 +02:00
Paweł Redzyński 10bcc9971d plots: tests: integration tests for studio 
Related: #7549
 2022-04-26 21:32:56 +02:00
Paweł Redzyński 700b648da9 plots: tests: integration test for vscode 
Closes: #7549
 2022-04-26 21:32:56 +02:00
Saugat Pachhai (सौगात) f84f9cf184 fs: do not pass size to underlying fs.put_file 2022-04-26 18:37:17 +05:45
Saugat Pachhai (सौगात) b53ebc1a5e build: try fixing building package 2022-04-26 17:28:36 +05:45
Saugat Pachhai (सौगात) 31e8eed402 fsspec.callbacks: add rich based callbacks 
We have some issues, so we cannot enable them just yet, but will
be good to merge this so that we can play with it.
 2022-04-26 12:28:32 +05:45
Saugat Pachhai (सौगात) 112ea1d448 dvc.objects: add callbacks for listing and estimating hashes 
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.

The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.

This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
 2022-04-26 07:38:24 +05:45
Saugat Pachhai (सौगात) f39e35445a fs.upload/download: simplify, remove unused kwargs, callbacks 
1. This PR simplifies `fs.upload`/`fs.download`/`fs.download_file`.
2. Removes unused kwargs.
3. `download` uses `dvc.utils.threadpool.ThreadPoolExecutor` which
   supports canceling futures.
4. `download` for directories supports nested/branched callbacks
   and progress bars.
 2022-04-26 06:57:10 +05:45
Paweł Redzyński 7211bd02ed fsspec wrapper: remove can handle dirs 2022-04-25 19:48:20 +03:00
Saugat Pachhai (सौगात) 83795ff3df fs.put_file: add support for uploading fileobjects 
Also changed the callbacks to inherit from our own so that we can
add few more utilities to it.
 2022-04-25 21:23:05 +05:45
Saugat Pachhai (सौगात) 8ead29805f fs: rename getsize to size 2022-04-25 18:52:46 +05:45
Saugat Pachhai fecc81e951 Revert "dvc.objects: add callbacks for listing and estimating hashes" 
This reverts commit 7599c6e20d.
 2022-04-25 18:37:05 +05:45
Saugat Pachhai (सौगात) 7599c6e20d dvc.objects: add callbacks for listing and estimating hashes 
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.

The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.

This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
 2022-04-25 18:36:47 +05:45
Saugat Pachhai (सौगात) 68a3cd9a13 fs: unify fs.base.FileSystem with FSSpecWrapper 
Since all of our filesystems are fsspec-compliant, there's no need for compat wrapper.
 2022-04-25 16:01:06 +09:00
David de la Iglesia Castro a394498067 render.match: Skip errored revisions in get_files 2022-04-22 16:44:52 +02:00
Saugat Pachhai (सौगात) 7708a8e292 gitfs: add regression test for broken stage.collect on gitfs when cwd != repo_root 2022-04-21 21:03:08 +05:45
Saugat Pachhai (सौगात) 23e62a9ff7 fs: make localfs fsspec-compliant 
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.

Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
 2022-04-21 12:44:55 +03:00
Saugat Pachhai (सौगात) fed0bfb2b3 tests: add test for dvc.api.open with remote 
Covers #7341
 2022-04-21 15:12:01 +05:45
Saugat Pachhai (सौगात) f22da81999 tests: make xfail-marked-tests to be strict by default 2022-04-21 15:06:47 +05:45
pre-commit-ci[bot] 22a988b113 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2022-04-21 15:06:31 +05:45
joncrall 77a8e870ff Add log statement to collect repo 2022-04-21 15:06:31 +05:45
David de la Iglesia Castro f20d6a7048 dag: Add --mermaid option. 2022-04-21 14:03:45 +05:45
Saugat Pachhai (सौगात) 6ffba62b75 executors: add support for cancel_futures=True in Python<3.9 
Also adds a kwarg `cancel_on_error` on the constructor to support
cancelling futures on `shutdown`. This is not done by default to
be backward compatible with `concurrent.futures.ThreadPool` executor.
 2022-04-21 14:03:10 +05:45
David de la Iglesia Castro 5c561572b0 exp init: Replace live section with metrics and plots. 
Closes #7534
 2022-04-21 08:55:49 +02:00
Peter Rowlands 444bf27e6f exp show: add internal --no-fetch option to disable temp/queue ref fetch 2022-04-21 15:45:59 +09:00
Peter Rowlands ba82faee53 exp show: don't acquire repo lock 2022-04-21 15:45:59 +09:00
dependabot[bot] 72d57f13bc build(deps-dev): Bump pylint from 2.13.5 to 2.13.7 
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.5 to 2.13.7.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.5...v2.13.7)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
 2022-04-21 08:22:47 +05:45
Saugat Pachhai (सौगात) 02b2ab6804 stage/exp list: alias --names-only to --name-only 
Fixes #7570.
 2022-04-21 06:53:16 +05:45
Ruslan Kuprieiev fd440df5b3 dvcignore: use fs.path instead of os.path 2022-04-20 12:27:26 +03:00
Saugat Pachhai (सौगात) cc8a0ce134 completion: decouple completion related stuff from command 
It also now uses the main parser itself that invokes the command rather than
creating a new parser object.
 2022-04-20 13:29:33 +05:45
Ivan Shcheklein 5c1e7be909 bump pydrive: improves and fixes thread safety 2022-04-20 06:06:28 +05:45
David de la Iglesia Castro 179e080c2f hdfs: Fix _strip_protocol and _checksum 
Match behavior previous to #6946
 2022-04-19 16:22:49 +03:00
Saugat Pachhai (सौगात) f3a7ec7f48 completion: fix dvc exp completion 2022-04-19 17:28:22 +05:45
Saugat Pachhai (सौगात) c0580c740b fs: move NoDirectoriesMixin to http filesystem 2022-04-19 11:54:34 +05:45
dependabot[bot] b4dcadbc9c build(deps-dev): Bump pylint from 2.12.2 to 2.13.5 (#7518) 
build(deps-dev): Bump pylint from 2.12.2 to 2.13.5
 2022-04-19 11:35:10 +05:45
Ruslan Kuprieiev 55219e9089 fs: get rid of CallbackMixin in azure/gs/hdfs/webhdfs 
No longer needed, as all of those filesystems already support callbacks.
 2022-04-18 17:34:06 +03:00
David de la Iglesia Castro 70b57d4703 live: Remove html generation 
Remove commands/live.
Remove LiveTask.
Bump dvclive.
 2022-04-18 15:47:31 +02:00
Saugat Pachhai (सौगात) 5eb16ab05e parsing: use relative paths; force absolute paths to relative 2022-04-18 14:19:56 +03:00
Ruslan Kuprieiev 517f45d45f repofs: migrate to fsspec 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 53c7f44480 repofs: rename _main_repo -> repo 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 9d75e8261b repofs: use repo paths 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev b3b0ec1fba repofs: unify ls/walk/info/exists/etc behaviour 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 0a3f5cbdd5 repofs: base walk on ls 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev bdd21fd481 repofs: use fs.ls 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 394118ef3d repofs: don't assume repofs paths are the same as underlying fs paths 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev f500c66aef repofs: use fs.walk directly 2022-04-18 14:17:20 +03:00