Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,439 Commits 7 Branches 579 Tags
890c0020ed5712d28b51e798bc9826661111032b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 890c0020ed Revert "repo: use tmp_dir if available" 
This reverts commit 7ccf35b47e.
 2023-02-17 21:25:46 +02:00
Ruslan Kuprieiev 7ce0b1cba2 Revert "repo: move data_index to .dvc/tmp/index" 
This reverts commit d68599a5f7.
 2023-02-17 21:25:46 +02:00
Ruslan Kuprieiev b8a9c37af5 Revert "config: enable data index caching by default" 
This reverts commit 1af63acae5.
 2023-02-17 21:25:46 +02:00
Ruslan Kuprieiev 1af63acae5 config: enable data index caching by default 2023-02-17 19:55:25 +02:00
Ruslan Kuprieiev d68599a5f7 repo: move data_index to .dvc/tmp/index 2023-02-17 19:55:25 +02:00
Ruslan Kuprieiev 7ccf35b47e repo: use tmp_dir if available 2023-02-17 19:55:25 +02:00
Ruslan Kuprieiev 4f49023a08 config: don't set default cache dir 
Setting default values is very untypical for all other applications and the
info config has is not enough to make a decision on the default cache dir,
especially when dealing with gitfs.

Fixes #8705
 2023-02-17 07:57:39 +02:00
Ruslan Kuprieiev c7be921f44 cachemgr: don't use tmp_dir 
We used to use it in gdrive before https://github.com/iterative/PyDrive2/pull/211
 2023-02-16 06:43:20 +02:00
Ruslan Kuprieiev c95c3f0f7a deps: limit types-tqdm 
See https://github.com/iterative/dvc/pull/9036
 2023-02-16 06:43:20 +02:00
Ruslan Kuprieiev cbe46cccd8 data: status: fix typo 
Leftover from #9031
 2023-02-15 07:18:19 +02:00
Ruslan Kuprieiev 49b5157abe diff: use index 2023-02-15 06:40:02 +02:00
Ruslan Kuprieiev 6f87c14bf9 data: status: bring back ui.status 2023-02-15 05:57:50 +02:00
Ruslan Kuprieiev 0f80d16ba7 data: status: fix typing 2023-02-15 05:57:50 +02:00
Ruslan Kuprieiev 4b23b4f8ae repo: cache index for cloud versioning 
This allows us to reuse cloud versioning index that we've built in other
operations like `data status`.

Note, this only works with `dvc config feature.data_index_cache true`
right now.

Also note that index is not populated for regular remotes, it will come
in a separate PR as there are some http-related issues to figure out.

Related #8962
 2023-02-15 04:53:48 +02:00
Ruslan Kuprieiev 446871d84f index: build_data_index: use dvcignore 
Fixes #9008
 2023-02-14 21:33:43 +02:00
Ruslan Kuprieiev e73d662686 repo: introduce data_index 
Extracting from `index.data` as a "root" data index that contains indexes
for different revisions/workspaces.

This is a setup for putting cloud versioning remote indexes and also replacing
old state.
 2023-02-13 06:31:44 +02:00
Ruslan Kuprieiev 86a6fb104a index: experimental persistent data index 
Related https://github.com/iterative/dvc-data/issues/208
 2023-02-13 02:57:30 +02:00
Ruslan Kuprieiev 3b097dcccd deps: bump dvc-data to 0.40.1 2023-02-12 09:12:08 +02:00
Ruslan Kuprieiev f7ebcbb974 deps: bump dvc-data to 0.39.3 2023-02-09 05:44:50 +02:00
Ruslan Kuprieiev 9156bb80db output: rename odb -> cache 2023-02-09 00:44:18 +02:00
Ruslan Kuprieiev 58295ca0ab repo: rename odb -> cache 
We used to have it named `cache` before, but these days it is clear that
you can have three odbs: just odb (e.g. in-memory), cache (typically local)
or remote (typically cloud). Fixing naming here before proceeding with
introducing `out.odb` and `out.cache`.

In preparation for migration to https://github.com/iterative/dvc-data/pull/304
 2023-02-08 23:05:45 +02:00
Ruslan Kuprieiev a8a5b29fb7 deps: bump dvc-data to 0.38.1 (#8976) 2023-02-07 06:58:15 +02:00
Ruslan Kuprieiev 8e1b75feb5 index: build_data_index: handle empty dirs (#8959) 
Fixes https://github.com/iterative/dvc/issues/8958
 2023-02-06 01:44:15 +02:00
Ruslan Kuprieiev 24023c6977 index: introduce data_keys (#8938) 
`data_keys` are similar to `outs`, but they provide more granularity, so we
can use them in `build_data_index` to rebuild index views with filtered
outputs. E.g. when running `dvc diff dir/file`, workspace index should only
have `dir/file` in it.

This also could be used for standalone (non-output) data in the future.

Pre-requisite for #8930
 2023-02-02 05:50:16 +02:00
Ruslan Kuprieiev 5310bcdcec deps: bump dvc-data to 0.37.3 (#8923) 2023-02-02 04:18:18 +02:00
Ruslan Kuprieiev f661cee8f6 stage: use repo.fs directly without copying (#8929) 
Otherwise when using brancher we end up in a strange situation where even
though we are collecting targets from `gitfs`, we are still using `localfs`
as `self.fs`, which make operations like `self.fs.abspath` in
`collect_granular` return invalid paths.
 2023-02-01 05:21:53 +02:00
Ruslan Kuprieiev e07c85a993 index: collect_targets: introduce onerror (#8925) 2023-02-01 03:49:27 +02:00
Ruslan Kuprieiev 58d9e379fe tests: gc: use timedelta again (#8926) 
Leftover from #8912
 2023-02-01 03:14:22 +02:00
Ruslan Kuprieiev 10184e9370 tests: gc: use timedelta (#8912) 
Bug from https://github.com/iterative/dvc/pull/8880
 2023-01-31 05:18:09 +02:00
Ruslan Kuprieiev 2690f05cee pre-commit: bump isort to 5.12.0 
https://github.com/PyCQA/isort/issues/2077
 2023-01-31 04:02:24 +02:00
Ruslan Kuprieiev 34c2c95115 index: expose rev property (#8909) 
index: add rev property

This is very useful to identify indexes that are built from git revisions
and so are describing an immutable state and can be used for caching.

We were already using this for `__repr__`.
 2023-01-30 20:18:54 +02:00
Ruslan Kuprieiev b46cfbda5d deps: bump dvc-data to 0.36.0 (#8885) 2023-01-26 00:22:12 +02:00
Ruslan Kuprieiev 8bda3650c5 deps: bump dvc-data to 0.35.3 2023-01-24 23:06:34 +02:00
Ruslan Kuprieiev 6a6bc7035a data: status: use data index (#8846) 
Pre-requisite for #8761
 2023-01-23 21:29:32 +02:00
Ruslan Kuprieiev ccc97622fa deps: bump dvc-ssh to 2.21.0 (#8865) 
* deps: bump dvc-ssh to 2.21.0

* Update pyproject.toml
 2023-01-23 01:36:52 +02:00
Ruslan Kuprieiev 916b5d169f deps: bump dvc-azure to 2.21.0 2023-01-23 00:46:58 +02:00
Ruslan Kuprieiev b50d1ee674 deps: bump dvc-data to 0.35.0 2023-01-20 04:16:12 +02:00
Ruslan Kuprieiev 4660443d0e deps: bump dvc-data to 0.31.4 2023-01-16 04:09:00 +02:00
Ruslan Kuprieiev a92ace0054 deps: bump dvc-data to 0.31.3 2023-01-16 01:16:35 +02:00
Ruslan Kuprieiev aabdddc2b7 index: introduce build_data_index 
This is a generally useful function that builds a data index from repo index.

This is currently only used in cloud versioning, but it is going to be used in
`status`, `diff` and other data-centric operations
 2023-01-15 01:45:09 +02:00
Ruslan Kuprieiev c02285ab49 deps: bump dvc-data to 0.29.0 2023-01-11 03:03:38 +02:00
Ruslan Kuprieiev b25ef5cdfb metrics/params: show: add hide_workspace flag 2023-01-09 16:57:18 +02:00
Ruslan Kuprieiev 0e5674ffea ls: compute relparts once for every dir 
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~25 to ~12
seconds.

`relparts` is an expensive operation as it calls `relpath` that calls `abspath`,
so we want to avoid calling it as much as possible.
 2023-01-08 03:44:50 +02:00
Ruslan Kuprieiev 684fc32e36 dvcfs: ls: don't call info() 
Cuts down `dvc ls . data/mnist/dataset --recursive` for me from ~37 to ~25
seconds.

Note that this also changes behaviour for ls with a broken symlink, as we
were not able to see it before because of a failing `info()` call, but now
we can follow a normal UNIX ls behaviour more closely.
 2023-01-08 00:37:40 +02:00
Ruslan Kuprieiev ab3f8fb1a4 Create hook-asyncssh.py 
Fixes #7577
 2022-11-24 23:25:00 +02:00
Ruslan Kuprieiev 2a888970e8 build: fpm: don't double quote cli option 
Leftover from #8611
 2022-11-24 01:34:18 +02:00
Ruslan Kuprieiev 73fdfc8059 build: fpm: don't create .build-id/* files 
These are leaking into our rpm packages.

Related https://github.com/iterative/dvc-s3-repo/issues/86
 2022-11-24 01:02:44 +02:00
Ruslan Kuprieiev 8dc11417d2 testing: api: test opening a file in subdir 
Related to #8420
 2022-11-23 17:47:13 +02:00
Ruslan KuprieievandPeter Rowlands b36861ada9 tests: import: use odb.clear() 
Co-authored-by: Peter Rowlands <[email protected]>
 2022-11-17 10:56:01 +02:00
Ruslan Kuprieiev d482e26c99 deps: bump dvc-data to 0.26.0 2022-11-17 10:56:01 +02:00
Ruslan Kuprieiev f87e78d754 deps: bump dvc-data to 0.25.2 2022-10-31 14:41:23 +02:00
Ruslan Kuprieiev 5899440b46 deps: bump dvc-data to 0.25.1 2022-10-30 00:59:39 +03:00
Ruslan Kuprieiev 61ce52f259 yappi: add --yappi-separate-threads flag 2022-10-29 21:23:37 +03:00
Ruslan Kuprieiev 475ef3ce65 Delete release-drafter.yml 2022-10-29 19:39:55 +03:00
Ruslan Kuprieiev 8f82a421a0 index: support multiple targets within output in IndexView 
stage_infos might contain the same stage multiple times, but with different
filter_info, so we shouldn't convert it to dict, as that makes us lose all
but one filter_info's.
 2022-10-25 13:37:44 +03:00
Ruslan Kuprieiev 23534063ca push: use build_tree instead of tree_from_index 2022-10-21 17:02:33 +03:00
Ruslan Kuprieiev d56957416e index: data: don't forget to set entry.key 2022-10-21 17:02:33 +03:00
Ruslan Kuprieiev 6711b653a5 tests: add tests for dvcignore.walk 2022-10-10 21:43:44 +03:00
Ruslan Kuprieiev faef8eec2c tests: ignore: use dvcignore.find directly 2022-10-10 21:43:44 +03:00
Ruslan Kuprieiev 11766c8839 ignore: walk: support detail=True 2022-10-10 21:43:44 +03:00
Ruslan Kuprieiev 890fbf3a6e deps: bump dvc-data to 0.14.0 2022-10-01 21:56:49 +03:00
Ruslan Kuprieiev 8c484645fa cloud-versioning: better handling for directories 2022-09-26 02:54:17 +03:00
Ruslan Kuprieiev 10cfa0d5d7 fetch: use md5() before save() 2022-09-23 15:12:19 +03:00
Ruslan Kuprieiev d2a31eb660 push: use update_pipeline=False for cloud versioning 2022-09-19 17:21:43 +03:00
Ruslan Kuprieiev 625b735502 deps: bump dvc-s3 2022-09-19 14:57:35 +03:00
Ruslan Kuprieiev 35e6f082a4 analytics: use iterative-telemetry for user_id lookup 2022-09-17 00:45:45 +03:00
Ruslan Kuprieiev 4531fec2b7 deps: bump dvc-data to 0.10.0 2022-09-16 22:56:56 +03:00
Ruslan Kuprieiev 3cbf319e0d loader: don't forget to set meta.isdir 
Leftover from #8273

Fixes #8300
 2022-09-15 22:53:48 +03:00
Ruslan Kuprieiev 477a6e177f index: use tree obj if using inline files 
Leftover from #8273
 2022-09-15 21:25:28 +03:00
Ruslan Kuprieiev 7e2248bbe5 deps: bump dvc-data to 0.7.1 2022-09-07 23:07:35 +03:00
Ruslan Kuprieiev 6cada6dc12 dvc: cloud versioning POC 2022-09-07 21:43:34 +03:00
Ruslan Kuprieiev 89d4461908 deps: temporarily limit dulwich to <0.20.46 
Looks like the new release broke some data status logic.
 2022-09-07 02:56:14 +03:00
Ruslan Kuprieiev a124d7e866 deps: bump dvc-data to 0.6.3 2022-09-07 02:23:29 +03:00
Ruslan Kuprieiev 628be2ae49 deps: bump dvc-data to 0.6.1 2022-09-06 20:30:20 +03:00
Ruslan Kuprieiev aff65d42bb deps: bump dvc-azure and dvc-s3 to 2.20.0 2022-09-01 13:00:47 +03:00
Ruslan Kuprieiev e307a62dc0 deps: bump dvc-data to 0.4.0 2022-09-01 02:59:24 +03:00
Ruslan Kuprieiev e7daeb1c3e deps: bump dvc-data to 0.3.0 2022-08-30 22:46:27 +03:00
Ruslan Kuprieiev 9cb242c8a5 deps: bump dvc-http and dvc-gs 2022-08-28 23:13:18 +03:00
Ruslan Kuprieiev 11802426de pyinstaller: don't forget dvc_* plugins 
Fixes #8194
 2022-08-28 21:05:25 +03:00
Ruslan Kuprieiev 3e183d62ed deps: bump dvc-* plugins to 2.19.0 
These are the same packages but with correct versions to sync up with existing packages that we generated in conda over the years.
 2022-08-25 14:44:13 +03:00
Ruslan Kuprieiev 460d08057c Create hook-fsspec.py 2022-08-24 23:39:46 +03:00
Ruslan Kuprieiev fc509d4e4a Delete hook-googleapiclient.model.py 
This is already provided by pyinstaller's standard hooks https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-googleapiclient.model.py
 2022-08-18 21:07:57 +03:00
Ruslan Kuprieiev a67fea2235 doctor: ignore missing plugins 2022-08-18 21:02:41 +03:00
Ruslan Kuprieiev a79ea15827 tests: webhdfs: fix linter 2022-08-18 12:43:12 +03:00
Ruslan Kuprieiev 8db9a2f70f deps: bump dvc-data to 0.1.19 2022-08-18 12:43:12 +03:00
Ruslan Kuprieiev 3fc9f323a7 fs: don't import SSHFileSystem 2022-08-18 00:23:40 +03:00
Ruslan Kuprieiev bf0aa5e8aa repo: data: ignore pylint false-positives 2022-08-17 23:02:22 +03:00
Ruslan Kuprieiev 3b3866323e deps: bump dvc-data to 0.1.18 2022-08-17 23:02:22 +03:00
Ruslan Kuprieiev 3c3cbee7fd deps: bump dvc-http to 0.0.2 2022-08-16 22:31:05 +03:00
Ruslan Kuprieiev 4a95020155 gs: add endpointurl 2022-08-14 13:39:03 +03:00
Ruslan Kuprieiev 3a429a869e tests: lockfile: use fs_path 2022-08-13 23:09:21 +03:00
Ruslan Kuprieiev 99465b25b5 deps: bump dvc-data to 0.1.17 2022-08-13 22:43:46 +03:00
Ruslan Kuprieiev fcdad4d229 deps: bump dvc-data to 0.1.16 2022-08-13 22:25:24 +03:00
Ruslan Kuprieiev 2e8b463795 deps: bump dvc-data 2022-08-12 11:58:44 +03:00
Ruslan Kuprieiev 704191b15a output: resolve relative to stage.wdir only on the same fs 
Leftover from https://github.com/iterative/dvc/pull/8091
 2022-08-04 19:25:31 +03:00
Ruslan Kuprieiev 0ce4e769f1 repo: make lock/rwlock kinda work on non local fs 2022-08-03 21:10:28 +03:00
Ruslan Kuprieiev 2c3655f7ff repo: use fs.makedirs instead of os.makedirs 2022-08-03 21:10:28 +03:00
Ruslan Kuprieiev 52bdd90840 repo: support basic workflow on s3fs/adlfs/etc 2022-08-03 21:10:28 +03:00
Ruslan Kuprieiev fcc7240617 odb: use odb.path instead of odb.cache_dir 2022-08-03 21:10:28 +03:00
Ruslan Kuprieiev ecd98bdf7d odbmgr: separate repo and local odbs 2022-08-03 21:10:28 +03:00