Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-10 07:18:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,525 Commits 7 Branches 579 Tags
e8a439df8d1a3c5ca4b21f062a99df387ce4fe2d
Commit Graph
1362 Commits
Author SHA1 Message Date
Saugat Pachhai (सौगात) 570e3a5359 dvc.fs: move fs to dvc.objects.fs 
Some of the APIs/FileSystems are reimported inside dvc.fs module.
 2022-05-16 13:15:19 +03:00
Saugat Pachhai (सौगात) 3f5757a0a5 dvc.objects: move hashing functions to dvc.objects 
This commit moves `HashedStreamReader`/`get_hash_file`/`file_md5`
into `dvc.objects.hash`.

The `get_hash_file` has been renamed to `hash_file`.
This also copies `relpath` and `is_exec` to the `dvc.fs.utils`.
 2022-05-12 20:33:18 +03:00
Daniele Trifirò 7a795f79db dvc.data: save and try loading raw dir objects 
When staging a directory, always save a "raw dir object" to the odb.
If the corresponding ".dir" object has not been added to the odb,
`stage()` calls can load the tree from the raw dir object instead of
rebuilding it by walking the directory.

This can lead to significant speed improvements when calling
`dvc status` for modified directories.

Fixes #7390
 2022-05-12 19:27:36 +03:00
Saugat Pachhai (सौगात) ee29888e18 dvc.objects: move get_mtime_and_size to dvc.objects.utils 2022-05-12 18:02:42 +03:00
Saugat Pachhai (सौगात) 11eaa87849 add typings to istextfile and system 
Also moves umask to dvc.fs.system
 2022-05-12 18:02:42 +03:00
Saugat Pachhai (सौगात) b4faa5616f system: change System class to a dvc.fs.system module 2022-05-12 18:02:42 +03:00
Saugat Pachhai (सौगात) 4add88f627 objects: move System to dvc.fs 2022-05-12 18:02:42 +03:00
Saugat Pachhai (सौगात) aac2fc0ca0 dvc.fs: rename fs.utils to fs.generic 
In the spirit of fsspec.generic, it is for cross-filesystem operations.
Trying to avoid `fs.utils` as much as possible, but we have some
functions that are really utilities coming, so I am making a place
for them. :)
 2022-05-12 18:02:15 +03:00
Ruslan Kuprieiev 005d56eedf fs: rename RepoFileSystem to DvcFileSystem 2022-05-12 00:39:41 +03:00
Karthikeyan Singaravelan b8d738220d Revert other tests and test only glob exception. 2022-05-11 18:37:09 +03:00
pre-commit-ci[bot] fb36db450f [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2022-05-11 18:37:09 +03:00
Karthikeyan SingaravelanandDavid de la Iglesia Castro 946ceb288d Update tests/func/test_data_cloud.py 
Co-authored-by: David de la Iglesia Castro <[email protected]>
 2022-05-11 18:37:09 +03:00
Karthikeyan Singaravelan 8d99ce06aa Move test to test_data_cloud. 2022-05-11 18:37:09 +03:00
Karthikeyan Singaravelan 9b3873c4f0 Move exception raising to glob_targets to cover all commands. 2022-05-11 18:37:09 +03:00
Karthikeyan Singaravelan e177f041c8 Fail on sync when there is no match for glob. 2022-05-11 18:37:09 +03:00
Ruslan Kuprieiev a80a85e6bd dvc: detach gitfs from local paths 
Our gitfs paths are currently tied to the local path in which the cloned
repository resides in, which results in a lot of hacky assumptions throughout
the code.

This PR detaches gitfs from local paths completely, making dvc work with a
proper fsspec gitfs implementation.

Part of #7543
 2022-05-11 18:17:39 +03:00
Saugat Pachhai (सौगात) 14c9348f93 dvc.objects: move hash_info, state and meta into dvc.objects 2022-05-11 17:35:48 +03:00
karajan1001 7a1f133f29 dvc ls: not raise PathMissingError on empty dir. 
fix #5841
 2022-05-11 16:38:58 +03:00
Gao 45a01c6a30 Revert "dvc ls: not raise PathMissingError on empty dir." 
This reverts commit 09afd2a0cd.
 2022-05-11 14:54:13 +03:00
karajan1001 09afd2a0cd dvc ls: not raise PathMissingError on empty dir. 
fix #5841
 2022-05-11 11:24:32 +03:00
Saugat Pachhai (सौगात) d92c6fed67 fs.callbacks: simplify, ensure None does not break them, lazy richcallbacks 2022-05-10 20:37:21 +05:45
dberenbaum 477f266dc5 exp init: fixes #7534; simplifies/updates exp init --live 2022-05-09 20:58:27 +05:45
Samuel Knight 1f5f2f5c97 dvcfile: preserve 'remote' on add and commit 
Preserves 'remote' output on dvc add and dvc commit.

Fixes #7356
 2022-05-05 08:36:37 +05:45
Saugat Pachhai (सौगात) 8a0c5680e0 fs: remove download 2022-05-03 14:42:19 +05:45
Saugat Pachhai (सौगात) 61699d1d66 fs: remove upload function 2022-05-03 11:22:21 +05:45
Ruslan Kuprieiev 58d960a47c Revert "fs.utils: add upload that can be conditionally atomic" 
This reverts commit dd187df667.
 2022-04-30 17:34:39 +03:00
Saugat Pachhai (सौगात) dd187df667 fs.utils: add upload that can be conditionally atomic 
The PR also makes odb.add operations atomic, but the checkout is still
not atomic.
 2022-04-29 23:02:15 +03:00
karajan1001 c81d841fd7 exp pull/push: better result handling in exp sharing(#7448) 
fix: #7448
Current we skip on those experiments which already existed on both sides, but
do not exclude them from the result list. And if we have diverged experiment,
pulling/pushing them will stop the current progress and raise an exception, but
didn't give any information on the condition of other experiments.

1. Change the exp pull and push to match the new API in `scmrepo`.
2. Handle sync status after `exp pull/push` finished.
3. Bump scmrepo to 0.0.19

Co-authored-by:
 2022-04-28 19:56:15 +08:00
Paweł Redzyński 700b648da9 plots: tests: integration test for vscode 
Closes: #7549
 2022-04-26 21:32:56 +02:00
Saugat Pachhai (सौगात) 83795ff3df fs.put_file: add support for uploading fileobjects 
Also changed the callbacks to inherit from our own so that we can
add few more utilities to it.
 2022-04-25 21:23:05 +05:45
Saugat Pachhai (सौगात) 8ead29805f fs: rename getsize to size 2022-04-25 18:52:46 +05:45
Saugat Pachhai (सौगात) 7708a8e292 gitfs: add regression test for broken stage.collect on gitfs when cwd != repo_root 2022-04-21 21:03:08 +05:45
Saugat Pachhai (सौगात) 23e62a9ff7 fs: make localfs fsspec-compliant 
It is not completely fsspec compliant, but it's upto the level
required by our FSSpecWrapper.

Next thing to tackle would be to merging `fs.base.FileSystem`
and `fs.fsspec_wrapper.FSSpecWrapper` into one filesystem
that is lazily-loaded and is itself fsspec-compliant.
 2022-04-21 12:44:55 +03:00
Saugat Pachhai (सौगात) fed0bfb2b3 tests: add test for dvc.api.open with remote 
Covers #7341
 2022-04-21 15:12:01 +05:45
Saugat Pachhai (सौगात) f22da81999 tests: make xfail-marked-tests to be strict by default 2022-04-21 15:06:47 +05:45
David de la Iglesia Castro 5c561572b0 exp init: Replace live section with metrics and plots. 
Closes #7534
 2022-04-21 08:55:49 +02:00
Ruslan Kuprieiev fd440df5b3 dvcignore: use fs.path instead of os.path 2022-04-20 12:27:26 +03:00
David de la Iglesia Castro 70b57d4703 live: Remove html generation 
Remove commands/live.
Remove LiveTask.
Bump dvclive.
 2022-04-18 15:47:31 +02:00
Saugat Pachhai (सौगात) 5eb16ab05e parsing: use relative paths; force absolute paths to relative 2022-04-18 14:19:56 +03:00
Ruslan Kuprieiev 517f45d45f repofs: migrate to fsspec 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev 9d75e8261b repofs: use repo paths 2022-04-18 14:17:20 +03:00
Ruslan Kuprieiev b3b0ec1fba repofs: unify ls/walk/info/exists/etc behaviour 2022-04-18 14:17:20 +03:00
Nick Sorros 0a3dd7a97f Move test to output and use mocker 2022-04-13 18:23:14 +02:00
Nick Sorros 5da72fc8ff Patch get_remote_odb instead 2022-04-13 18:23:14 +02:00
Nick Sorros 36c27ed17f Add test for remote missing when adding data dir 2022-04-13 18:23:14 +02:00
Peter Rowlands 69acbe33c9 erepo: include exp refs in clones 
- add test for dvc get --rev=<exp>
 2022-04-09 10:27:06 +09:00
Fabian 44af5b3b7e index: add getitem 
* allow access to a stage via `getitem` in addition to iterating over the Index.
 2022-04-07 21:05:47 +05:45
David de la Iglesia Castro 117b22d58a exp show: pcp: Allow to --drop Experiment 2022-04-04 11:48:55 +02:00
Saugat Pachhai (सौगात) 565a813e2a setup.cfg: tests: try to remove more dependencies 2022-03-31 13:36:43 +05:45
Saugat Pachhai (सौगात) 5ff1d76826 ui: add revision info on dvc.yaml/dvc.lock/.dvc syntax and schema errors 2022-03-29 17:49:03 +05:45