Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,483 Commits 7 Branches 579 Tags
7a1f133f292946050646b3dcd118d61ec850fb73
Commit Graph
7483 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
karajan1001 7a1f133f29 dvc ls: not raise PathMissingError on empty dir. 
fix #5841
 2022-05-11 16:38:58 +03:00
Gao 45a01c6a30 Revert "dvc ls: not raise PathMissingError on empty dir." 
This reverts commit 09afd2a0cd.
 2022-05-11 14:54:13 +03:00
karajan1001 09afd2a0cd dvc ls: not raise PathMissingError on empty dir. 
fix #5841
 2022-05-11 11:24:32 +03:00
Saugat Pachhai (सौगात) d92c6fed67 fs.callbacks: simplify, ensure None does not break them, lazy richcallbacks 2022-05-10 20:37:21 +05:45
Saugat Pachhai (सौगात) f10dbef0cf TqdmCallback: minor fixup, check for None explicitly 
Tqdm supports __bool__, so we should really use `is not None` test
rather than boolean check. This is not used anywhere right now, but
I'd prefer being correct.
 2022-05-10 14:22:13 +05:45
Saugat Pachhai (सौगात) 8fb2f65d4b TqdmCallback: make it lazy 2022-05-10 13:41:54 +05:45
dberenbaum 477f266dc5 exp init: fixes #7534; simplifies/updates exp init --live 2022-05-09 20:58:27 +05:45
Ruslan Kuprieiev b14a972cd0 repofs: use / as root_marker 
This makes dvcfs/repofs much easier to work with and allows it to support
isabs/abspath/etc correctly, as previously any full path (with root_marker "")
would be impossible to distinguish from a relative one.

This also detaches external workspaces from the main repo workspace in
DvcFileSystem, which was necessary to be able to distinguish dvcfs abspaths
from local external output paths. Note that currently we still have limited
support for those (e.g. get/api supports them, but import doesn't) and this
change keeps it that way.
 2022-05-08 17:19:53 +03:00
Saugat Pachhai (सौगात) 562e61235b objects: do not inherit from DvcException 2022-05-06 13:24:41 +03:00
David de la Iglesia Castro 9bd9ac4a37 FileSystem: Handle encoding via kwargs. 
Drop if None.

Closes #7691
 2022-05-06 02:34:38 +03:00
Saugat Pachhai (सौगात) f89f4ac152 fs.put: introduce non-recursive put 2022-05-05 21:22:29 +03:00
Saugat Pachhai (सौगात) 54dd3ae90c fs.get: use async implementation when supported by the filesystem 2022-05-05 14:15:21 +03:00
dberenbaum 35d2963069 repro/run/stage: reorder options 2022-05-05 16:42:29 +08:00
Samuel Knight 1f5f2f5c97 dvcfile: preserve 'remote' on add and commit 
Preserves 'remote' output on dvc add and dvc commit.

Fixes #7356
 2022-05-05 08:36:37 +05:45
Saugat Pachhai (सौगात) c64db917f5 fs: add more fsspec methods 2022-05-05 08:16:34 +05:45
Paweł Redzyński 9d476aab8b plots: image converter return absolute paths 2022-05-04 10:57:13 +02:00
Saugat Pachhai (सौगात) 5067e1f5bc fs.utils.transfer: remove tqdm progress bars 2022-05-03 15:15:48 +05:45
Saugat Pachhai (सौगात) 8a0c5680e0 fs: remove download 2022-05-03 14:42:19 +05:45
Saugat Pachhai (सौगात) 61699d1d66 fs: remove upload function 2022-05-03 11:22:21 +05:45
dependabot[bot] 94c776182e build(deps-dev): Bump pytest from 7.1.1 to 7.1.2 
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.1.1 to 7.1.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/7.1.1...7.1.2)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
 2022-05-03 10:18:39 +05:45
dependabot[bot] e3d3500db8 build(deps-dev): Bump pylint from 2.13.7 to 2.13.8 
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.13.7 to 2.13.8.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.13.7...v2.13.8)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
 2022-05-03 09:48:49 +05:45
Saugat Pachhai (सौगात) d0e05c4b66 deps: bump s3fs to 2022.02.0 
The callbacks on _get_file/_put_file was only supported on >2022.02.0.
Fixes #7669.
 2022-05-02 12:57:00 +03:00
Ruslan Kuprieiev 58d960a47c Revert "fs.utils: add upload that can be conditionally atomic" 
This reverts commit dd187df667.
 2022-04-30 17:34:39 +03:00
Saugat Pachhai (सौगात) dd187df667 fs.utils: add upload that can be conditionally atomic 
The PR also makes odb.add operations atomic, but the checkout is still
not atomic.
 2022-04-29 23:02:15 +03:00
Saugat Pachhai (सौगात) b71ea349d2 repofs: only use dvcfs when --dvc-only is specified 2022-04-29 12:03:46 +03:00
karajan1001 c81d841fd7 exp pull/push: better result handling in exp sharing(#7448) 
fix: #7448
Current we skip on those experiments which already existed on both sides, but
do not exclude them from the result list. And if we have diverged experiment,
pulling/pushing them will stop the current progress and raise an exception, but
didn't give any information on the condition of other experiments.

1. Change the exp pull and push to match the new API in `scmrepo`.
2. Handle sync status after `exp pull/push` finished.
3. Bump scmrepo to 0.0.19

Co-authored-by:
2.10.2 		2022-04-28 19:56:15 +08:00
Saugat Pachhai (सौगात) 127a9c74b2 fs.put_file: fix function signature for size 2022-04-28 15:54:40 +05:45
Saugat Pachhai (सौगात) e30d2fdda5 fs.put_file: don't require size to be passed 2022-04-28 15:31:28 +05:45
Saugat Pachhai (सौगात) ae4f56c5ad fs.put_file: don't require size to be passed 2022-04-28 15:31:28 +05:45
Ruslan Kuprieiev 8cdcf9b8bb utils: remove PathStringTrie 2022-04-27 22:40:38 +03:00
Ruslan Kuprieiev 55a4532f1d ignore: use Trie instead of PathStringTrie 2022-04-27 22:40:38 +03:00
Ruslan Kuprieiev 43b9cab01b repofs: use Trie instead of PathStringTrie 2022-04-27 22:40:38 +03:00
Saugat Pachhai (सौगात) 36d53bd403 dvc.stage: log time duration on stage collection walk in verbose mode 2022-04-27 18:04:14 +03:00
Saugat Pachhai (सौगात) 46fac33ba5 fs: remove unused CHUNK_SIZE var 
There is no support for chunk_size kwarg in fs.open.
Also, `upload_fobj` is now able to figure out proper
chunk size from fileobj, so I don't think we need this on _upload_file.
 2022-04-27 19:06:46 +05:45
David de la Iglesia Castro 8b68e4fa75 bump dvclive 0.7.3 2022-04-27 18:06:53 +05:45
Saugat Pachhai (सौगात) 4f1fdf0b0c fs.ossfs: remove unused var COPY_POLL_SECONDS 2022-04-27 18:06:24 +05:45
Saugat Pachhai (सौगात) 0d48598e4b fs.ssh: remove unused vars 2022-04-27 17:56:08 +05:45
Saugat Pachhai (सौगात) 5d64f6504f fs.s3: remove custom put_file/get_file 2022-04-27 17:53:17 +05:45
Saugat Pachhai (सौगात) 3d54039614 scm: fix progress bar, show progress and repo being cloned 
Before, as we were not passing `total`, the Tqdm progress bar would not
show the bar or not show any progress on the bar. As we get that value lazily,
we should force it to always show the bar.
Also updated the progress bar to say which repo is being cloned.
 2022-04-27 16:42:28 +05:45
Saugat Pachhai (सौगात) cda855b97a fs: add cat/cat_file/cat_ranges/pipe/pipe_file/touch methods 2022-04-27 15:57:17 +05:45
David de la Iglesia Castro bea7d56bd5 plots: Fix repo with removed plots 
Changes in dvc-render 0.0.5 caused error.
 2022-04-27 11:53:30 +02:00
Paweł Redzyński 10bcc9971d plots: tests: integration tests for studio 
Related: #7549
 2022-04-26 21:32:56 +02:00
Paweł Redzyński 700b648da9 plots: tests: integration test for vscode 
Closes: #7549
 2022-04-26 21:32:56 +02:00
Saugat Pachhai (सौगात) f84f9cf184 fs: do not pass size to underlying fs.put_file 2022-04-26 18:37:17 +05:45
Saugat Pachhai (सौगात) b53ebc1a5e build: try fixing building package 2022-04-26 17:28:36 +05:45
Saugat Pachhai (सौगात) 31e8eed402 fsspec.callbacks: add rich based callbacks 
We have some issues, so we cannot enable them just yet, but will
be good to merge this so that we can play with it.
 2022-04-26 12:28:32 +05:45
Saugat Pachhai (सौगात) 112ea1d448 dvc.objects: add callbacks for listing and estimating hashes 
This PR gets rid of Tqdm progress bars inside `dvc.objects`.
Those are replaced with multi-phase callback in `hashes_exist`, and
simple callable that takes `completed` value in
`odb._estimate_remote_size()`.

The `odb.list_hashes_exists()` and `odb.all()` now return iterables,
hence the caller has the responsibility of handling progress.

This is now handled in `dvc.data` by using a custom `Tqdm` progressbar.
This is ugly and is temporary, but my motivation is to remove tqdm from dvc.objects
first. We can gradually get rid of these from `dvc.data` too, but
that requires more knowledge about `dvc.data` and involves more
design work. All in time, I guess. :)
 2022-04-26 07:38:24 +05:45
Saugat Pachhai (सौगात) f39e35445a fs.upload/download: simplify, remove unused kwargs, callbacks 
1. This PR simplifies `fs.upload`/`fs.download`/`fs.download_file`.
2. Removes unused kwargs.
3. `download` uses `dvc.utils.threadpool.ThreadPoolExecutor` which
   supports canceling futures.
4. `download` for directories supports nested/branched callbacks
   and progress bars.
 2022-04-26 06:57:10 +05:45
Paweł Redzyński 7211bd02ed fsspec wrapper: remove can handle dirs 2022-04-25 19:48:20 +03:00
Saugat Pachhai (सौगात) 83795ff3df fs.put_file: add support for uploading fileobjects 
Also changed the callbacks to inherit from our own so that we can
add few more utilities to it.
 2022-04-25 21:23:05 +05:45