Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,805 Commits 7 Branches 579 Tags
e16c0c8b2f2fb8dd54ce250583296b92cde3835c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev e16c0c8b2f checkout/fetch: finer control over steps 
See https://github.com/iterative/dvc-data/pull/390 and https://github.com/iterative/dvc-data/pull/391
 2023-06-18 18:58:30 +03:00
Ruslan Kuprieiev 33e7645603 tests: odb: index: simplify test 
For some unknown reason the mock in this test leaks into the outside tests.

I wasn't able to determine why in the reasonable amount of time, so instead just
simplifying it to not use mocker at all.

That whole file should really be migrated to dvc-data, but that needs some time.
 2023-06-18 18:58:30 +03:00
Ruslan Kuprieiev c827654fe7 tests: use mocker instead of unittest.mock where possible 
Had the "pleasure" of finding a mocker.patch leak into other tests and one
of the suspicions is that mixing unittest.mock and mocker might be somehow
related.

In any case, we should use mocker where possible anyway.
 2023-06-18 17:10:45 +03:00
Ruslan Kuprieiev fd73b389c6 api: show/exp: add config option 
Fixes #9613
 2023-06-16 17:09:38 +03:00
Ruslan Kuprieiev 0433565f1f api: add config support for open/read 
Fixes #9610
 2023-06-15 18:30:14 +03:00
Ruslan Kuprieiev a6e49a72de diff: handle revisions pointing to the same sha 
Fixes #9604
 2023-06-14 18:48:18 +03:00
Ruslan Kuprieiev 97348cfba7 Revert "checkout/fetch: finer control over steps" 
This reverts commit c5e3d42899.
 2023-06-14 03:39:43 +03:00
Ruslan Kuprieiev c5e3d42899 checkout/fetch: finer control over steps 
See https://github.com/iterative/dvc-data/pull/390 and https://github.com/iterative/dvc-data/pull/391
 2023-06-13 23:11:14 +03:00
Ruslan Kuprieiev 5c1487e2ec deps: bump dvc-data to >=1.10.0 2023-06-13 14:45:36 +03:00
Ruslan Kuprieiev bbb77360e9 deps: bump dvc-data to >=1.9.0 
Fixes #9541
 2023-06-12 22:03:29 +03:00
Ruslan Kuprieiev 1af150a5e9 index: use out.hash_info for complete imports 2023-06-12 16:55:36 +03:00
Ruslan Kuprieiev 2ed677057d fetch: cache collected 
This avoids recollecting the cache files over and over again.

For example, `dvc fetch` noop in `dvc-bench` went from ~30sec down to ~10sec
for me.

Next step is to also cache remote index, so we don't have to waste those
~10sec checking files in the remote.

Related https://github.com/iterative/dvc-data/pull/376
 2023-06-12 04:31:37 +03:00
Ruslan Kuprieiev 0597311439 deps: bump dvc-data to >=1.5.0 2023-06-12 01:04:24 +03:00
Ruslan Kuprieiev fc8ec6aa95 fetch: show progress when collecting 2023-06-10 22:48:16 +03:00
Ruslan Kuprieiev 3dca69b678 deps: bump dvc-data to >= 1.2.0 
This brings https://github.com/iterative/dvc-data/pull/373 , which improves UI for `checkout` by not visually hanging during `diff` stage inside.
 2023-06-10 21:33:25 +03:00
Ruslan Kuprieiev 4c0bb8d316 fetch: use index fetch 2023-06-10 21:18:57 +03:00
Ruslan Kuprieiev 1cb371d4d7 repo: commit: don't require target 
Same as CLI.
 2023-06-10 19:19:40 +03:00
Ruslan Kuprieiev c96795dcfb tests: checkout: fix test_checkout_relink_protected on windows 2023-06-09 22:36:33 +03:00
Ruslan Kuprieiev 8f768addf9 checkout: don't validate cache 
Part of #9561
 2023-06-09 18:41:19 +03:00
Ruslan Kuprieiev af1fb722c8 dvc: drop cached external outputs 
Fixes #9531

Docs https://github.com/iterative/dvc.org/pull/4574 Kudos @dberenbaum
 2023-06-09 18:34:36 +03:00
Ruslan Kuprieiev 8b1c1a31f3 deps: bump dvc-data to >=1.0.3 2023-06-09 18:16:51 +03:00
Ruslan Kuprieiev 262419ef35 deps: bump dvc-data to >=1.0.2 
Related https://github.com/iterative/dvc/issues/9566
 2023-06-09 16:26:31 +03:00
Ruslan Kuprieiev d0291e8039 checkout: let index checkout error-out on missing hashes 
This ensures we don't assume that everything needs to have hash_info to be
checked out (e.g. we can totally checkout directly from FileStorage).

Related https://github.com/iterative/dvc/pull/9424
 2023-06-07 14:56:19 +03:00
Ruslan Kuprieiev 78637713cf checkout: use index checkout 
Needed for https://github.com/iterative/dvc/pull/9424
 2023-06-04 23:27:01 +03:00
Ruslan Kuprieiev f9104275f3 deps: install tzdata for testing with celery (#9537) 
Temporary fix for CI on windows with py >=3.9.
 2023-06-04 22:18:23 +03:00
Ruslan Kuprieiev bce3b5d2a1 update: make sure we update hashes/checksums on --to-remote (#9536) 
Discovered in #9444
 2023-06-04 19:36:54 +03:00
Ruslan Kuprieiev 6543580100 dvcfs: don't use datafs isout/isdvc 2023-05-31 23:14:28 +03:00
Ruslan Kuprieiev e69deaa132 import: don't forget to set other cache config options (#9423) 2023-05-16 15:53:41 +03:00
Ruslan Kuprieiev 6b6084a848 tests: import: check that dvcfs is using main repo's cache (#9417) 
Followup for https://github.com/iterative/dvc/pull/9415
 2023-05-08 00:34:00 +03:00
Ruslan Kuprieiev 967dd5ea2b import: don't forget to set cache.dir (#9415) 
* import: don't forget to set cache.dir

* repo: open: try setting remote for local repos
 2023-05-07 05:28:58 +03:00
Ruslan Kuprieiev ad39bf14ed repo: external_repo -> _external_repo 2023-05-07 04:14:25 +03:00
Ruslan Kuprieiev 01416df5a4 get: use Repo.open instead of external_repo 
This should use `Repo.open` just like `dvc import`, to benefit from
handling local repos without cloning and redundant downloaads.
 2023-05-07 04:14:25 +03:00
Ruslan Kuprieiev eb592e7e33 repo: consolidate Repo.open logic 2023-05-07 04:14:25 +03:00
Ruslan Kuprieiev ac3eef439f gha: try setting PYTHONUTF8 
Trying to see if that will help with encoding error on windows with python 3.11
 2023-05-05 21:55:31 +03:00
Ruslan Kuprieiev b044bcce5e tests: lockfile: use different way of changing a cmd 2023-05-05 04:16:58 +03:00
Ruslan Kuprieiev a251cffdb5 deps: unpin ruamel.yaml 2023-05-05 04:16:58 +03:00
Ruslan Kuprieiev 9b7240e9e0 deps: temporarily pin ruamel.yaml 
Temporary workaround for https://github.com/iterative/dvc/issues/9397
 2023-05-04 03:22:48 +03:00
Ruslan Kuprieiev 7e1036b30d doctor: add system/global config location 2023-04-28 01:23:44 +03:00
Ruslan Kuprieiev a15dbc8e03 info: get_fs_type -> _get_fs_type 
This is not used anywhere outside so should be private like the rest. Likely a leftover from the past.
 2023-04-28 00:53:03 +03:00
Ruslan Kuprieiev 4e02197245 info: replace self -> repo 
Minor cleanup of an old leftover.
 2023-04-28 00:40:11 +03:00
Ruslan Kuprieiev ce0f6c187e deps: put lower bound on dvc-http 
People are still running into `ImportError: cannot import name 'fsspec_loop' from 'fsspec.asyn' ` because of it.
 2023-04-12 22:26:31 +03:00
Ruslan Kuprieiev 40f7f34889 api: data: set default remote instead of using it directly 
Fixes #9139
Fixes #7341 (see https://github.com/iterative/dvc/pull/7363#issuecomment-1032959843)
 2023-04-05 02:47:35 +03:00
Ruslan Kuprieiev 649ee67e4b config: introduce core.site_cache_dir 2023-04-03 00:29:44 +03:00
Ruslan Kuprieiev 7cdb3b0ae2 repo: site_cache_dir: isolate by user 
Related #9249
 2023-04-03 00:29:44 +03:00
Ruslan Kuprieiev ecdc44e010 dep: repo: use dvcfs 
Makes dvc get/import/etc use dvcfs, which already supports cloud versioning,
circular imports and other stuff.

Also makes dvc import behave more like dvc import-url, so that we can use
the same existing logic for fetching those using index instead of objects.

Fixes #8789
Related iterative/studio#4782
 2023-04-02 03:55:02 +03:00
Ruslan Kuprieiev 35bb261b89 config: don't look for .dvc dir implicitly 
Currently if dvc_dir doesn't exists (e.g. we are creating an uninitialized
repo), config will try looking for `.dvc` dir in the current directory,
which might be inside of another dvc repository resulting in mixed configs.

Use explicit `from_cwd` method to make `Config` try to find repo configs
if available to avoid the usage mixup.
 2023-03-29 16:27:54 +03:00
Ruslan Kuprieiev d41512c85c deps: bump dvc-data to 0.46.0 (#9241) 
* utils: determine_linecol: ignore mypy attr-defined error

* deps: bump dvc-data to 0.46.0
 2023-03-26 22:06:23 +03:00
Ruslan Kuprieiev 40484e8d57 dvcfs: handle broken symlink/TreeError (#9231) 
* plots: handle broken symlink/TreeError

* dvcfs: ls: raise FileNotFoundError on broken symlink/TreeError

* deps: bump dvc-data to 0.45.1
 2023-03-23 15:38:21 +02:00
Ruslan Kuprieiev 2d5c15bee1 erepo: move legacy exceptions to api (#9228) 
* erepo: get rid of NoRemoteInExternalRepoError

This exception hasn't been making any sense at all since we've introduced
get and import, as they can be chained in repos.

* erepo: get rid of NoOutputInExternalRepoError

This is dead code these days.

* erepo: get rid of FileMissingError and PathMissingError

* api: preserve legacy exc wrappers
 2023-03-23 01:40:42 +02:00
Ruslan Kuprieiev dfe76c54d7 erepo: set subrepos/uninitialized explicitly 2023-03-22 02:26:44 +02:00
Ruslan Kuprieiev 392c15e1d4 erepo: pass rev instead of fs/scm 
Towards unifying `Repo.open` and `external_repo` behaviour to get rid
of the latter.
 2023-03-22 00:21:25 +02:00
Ruslan Kuprieiev 990e834f1d repo: open: handle bare git repos 
If we are asked to open a potentially unitialized repo, we need to make sure
it is not a bare git repo, so that we don't accidentally start putting dvc
files inside git control dir.

Discovered while working on #8789
 2023-03-16 02:03:31 +02:00
Ruslan Kuprieiev 33a2449c91 deps: repo: use _get_rev when creating dvcfs 2023-03-16 01:05:04 +02:00
Ruslan Kuprieiev e2ce95e17e deps: repo: pass url explicitly to DVCFileSystem 
Otherwise our wrapper ignores it.
 2023-03-16 00:35:15 +02:00
Ruslan Kuprieiev 53f9953cd1 index: data: support imports 
Stepping stone to simplifying `dvc fetch/pull` by using index.

Fetch handles regular imports through index already, but not repo
imports because their processing is much more involved (e.g. chained
imports) in the current arch.

With `FileStorage` support introduced into `DataIndex` and `datafs`
supporting imports overall, `dvcfs` can now handle repo imports (even
chained ones). This will soon allow us to handle repo imports the same
way we handle regular ones, improve performance and get rid of a lot of
messy code (e.g. DependencyRepo).

Related https://github.com/iterative/scmrepo/issues/207
Related https://github.com/iterative/dvc-data/pull/315
Related https://github.com/iterative/studio/issues/5261
 2023-03-15 20:32:13 +02:00
Ruslan Kuprieiev 37ccc5e8d8 repo: create site_cache_dir with proper perms 
Fixes #9184
 2023-03-15 14:26:17 +02:00
Ruslan Kuprieiev c8745dd580 deps: bump dvc-data to 0.44.0 2023-03-15 03:41:19 +02:00
Ruslan Kuprieiev 1cf8b5e582 install: pre-commit: install all deps by default 
Fixes #9178
 2023-03-15 01:19:42 +02:00
Ruslan Kuprieiev b56f6b69dd exp: push/pull: don't include baseline commit 
Fixes #6592
 2023-03-14 14:00:14 +02:00
Ruslan Kuprieiev 54a6709d99 Update pyproject.toml 2023-03-13 21:03:33 +02:00
Ruslan Kuprieiev e4a5a2aec5 deps: bump dvc-data to >=0.43.0 2023-03-13 21:03:33 +02:00
Ruslan Kuprieiev 612f3039e0 repo: move state/index/data_index to site_cache_dir 
We use all of these for optimization and should not depend on a potentially slow or broken
filesystem on which repository resides (could be HDD or CIFS/NFS/overlayfs/etc).

This PR means a few things:
    * all odd filesystems that dvc repo might be on will work quickly out of the box
    * when we start caching cloned repositories for `dvc get/import`, we will be
      able to cache indexes for them as well (note that bare clones don't have a
      worktree so there is no place for .dvc/tmp with the old approach).
    * `index.dir` and `state.dir` are now obsolete.
    * `dvc init` will cleanup site_cache_dir.

Related #5562
 2023-03-13 02:58:23 +02:00
Ruslan Kuprieiev 17f523bf86 deps: bump dvc-data to >=0.42.3 2023-03-11 02:01:31 +02:00
Ruslan Kuprieiev a0bf2d997a repo: data_index: fallback to in-memory 
Makes for a much simpler resource management elsewhere.
 2023-03-11 01:23:12 +02:00
Ruslan Kuprieiev 8ed9b911db deps: bump dvc-data to >= 0.42.2 2023-03-10 22:40:59 +02:00
Ruslan Kuprieiev 3aa9f51e36 output: suppress cloud.pull errors on cache collection 
Fixes https://github.com/iterative/studio/issues/5261
 2023-03-09 22:44:32 +02:00
Ruslan Kuprieiev 4779c248f8 deps: bump dvc-data to >=0.42.1 2023-03-02 17:53:07 +02:00
Ruslan Kuprieiev 59a4c4f67c remote: set hash_name as md5 
Fixes #7977
 2023-03-01 13:26:24 +02:00
Ruslan Kuprieiev 546becddaf deps: bump dvc-gs to 2.22.0 2023-03-01 08:36:33 +02:00
Ruslan Kuprieiev c0754ee5b3 deps: bump dvc-data to 0.42.0 2023-02-28 14:48:34 +02:00
Ruslan Kuprieiev a91528c3c9 Revert "testing: fixtures: fix docker regression" 
This reverts commit b80d84e689.
 2023-02-27 15:02:22 +02:00
Ruslan Kuprieiev b80d84e689 testing: fixtures: fix docker regression 2023-02-27 14:17:07 +02:00
Ruslan Kuprieiev 81da91a973 exp: run: don't relink during commit 
Part of #8809
 2023-02-24 08:09:35 +02:00
Ruslan Kuprieiev 37761a2627 data: status: get rid of --not-in-remote 2023-02-21 12:32:40 +02:00
Ruslan Kuprieiev 968a8fde99 data: status: introduce --remote-refresh 2023-02-21 12:32:40 +02:00
Ruslan Kuprieiev 09ccd9c7e4 repo: data_index: store in .dvc/tmp/index/data 2023-02-20 21:57:27 +02:00
Ruslan Kuprieiev cd9dd0a312 index: use has_node instead of ls 
Current code doesn't work for prefixes longer than one, which makes index cache
effectively not work and it should've used has_node from the very start anyway.
 2023-02-20 17:45:15 +02:00
Ruslan Kuprieiev 6fbbbeccc5 index: store data index under tree prefix 
Tidying it up, so root of the index will look like this:

```
tree/
remote/
```

This hasn't been released yet, so we are not breaking anything.
 2023-02-20 13:33:22 +02:00
Ruslan Kuprieiev c34cbf63af data: status: introduce --not-in-remote 
Also supports cloud versioning.

Fixes #8761
 2023-02-20 12:25:17 +02:00
Ruslan Kuprieiev b76aad8d56 data: status: rename with_missing to not_in_cache 2023-02-20 12:25:17 +02:00
Ruslan Kuprieiev 68bf095eae remote: rename index prefix from remotes to remote 2023-02-20 12:24:46 +02:00
Ruslan Kuprieiev ea6e01d0e0 dvcfs: implement get_file 
Currently we use `open`, which is much slower. For example,

```
$ dvc get dvc-bench data/small/dataset
```

goes down from 110sec to 50sec.

Discovered as a part of https://github.com/iterative/dvc-bench/issues/419
 2023-02-20 11:10:13 +02:00
Ruslan Kuprieiev 098685b8f3 deps: bump dvc-data to 0.40.4 
Improves index view related performance (e.g. `dvc diff`).
 2023-02-18 11:35:01 +02:00
Ruslan Kuprieiev 8f549c8a8d config: enable data index caching by default 2023-02-18 08:00:58 +02:00
Ruslan Kuprieiev e56d97bb28 gha: user ubuntu 20.04 for deb packages 
18.04 will reach EOL in april and will be dropped by gha.

18.04 also comes with an old sqlite version which forces us to use obsolete
syntax.
 2023-02-18 08:00:58 +02:00
Ruslan Kuprieiev 07b72c20e9 repo: move data_index to .dvc/tmp/index 2023-02-17 21:43:57 +02:00
Ruslan Kuprieiev 592f251481 repo: use tmp_dir if available 2023-02-17 21:43:57 +02:00
Ruslan Kuprieiev 890c0020ed Revert "repo: use tmp_dir if available" 
This reverts commit 7ccf35b47e.
 2023-02-17 21:25:46 +02:00
Ruslan Kuprieiev 7ce0b1cba2 Revert "repo: move data_index to .dvc/tmp/index" 
This reverts commit d68599a5f7.
 2023-02-17 21:25:46 +02:00
Ruslan Kuprieiev b8a9c37af5 Revert "config: enable data index caching by default" 
This reverts commit 1af63acae5.
 2023-02-17 21:25:46 +02:00
Ruslan Kuprieiev 1af63acae5 config: enable data index caching by default 2023-02-17 19:55:25 +02:00
Ruslan Kuprieiev d68599a5f7 repo: move data_index to .dvc/tmp/index 2023-02-17 19:55:25 +02:00
Ruslan Kuprieiev 7ccf35b47e repo: use tmp_dir if available 2023-02-17 19:55:25 +02:00
Ruslan Kuprieiev 4f49023a08 config: don't set default cache dir 
Setting default values is very untypical for all other applications and the
info config has is not enough to make a decision on the default cache dir,
especially when dealing with gitfs.

Fixes #8705
 2023-02-17 07:57:39 +02:00
Ruslan Kuprieiev c7be921f44 cachemgr: don't use tmp_dir 
We used to use it in gdrive before https://github.com/iterative/PyDrive2/pull/211
 2023-02-16 06:43:20 +02:00
Ruslan Kuprieiev c95c3f0f7a deps: limit types-tqdm 
See https://github.com/iterative/dvc/pull/9036
 2023-02-16 06:43:20 +02:00
Ruslan Kuprieiev cbe46cccd8 data: status: fix typo 
Leftover from #9031
 2023-02-15 07:18:19 +02:00
Ruslan Kuprieiev 49b5157abe diff: use index 2023-02-15 06:40:02 +02:00
Ruslan Kuprieiev 6f87c14bf9 data: status: bring back ui.status 2023-02-15 05:57:50 +02:00
Ruslan Kuprieiev 0f80d16ba7 data: status: fix typing 2023-02-15 05:57:50 +02:00