Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
9,182 Commits 7 Branches 579 Tags
ac0397e536bf4aaf017919b303b7992bd1eb5e92
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 84b2e685e2 repo: index: don't set cache storage if there is no out.cache (#10275) 2024-02-01 15:15:35 +02:00
Ruslan Kuprieievandpre-commit-ci[bot] 0bb2f3d10c repo: site_cache_dir: separate subdir from root_dir (#10273) 
* repo: site_cache_dir: separate subdir from root_dir

Currently we join scm root_dir with subrepo root dir to have something that would represent that subrepo, but that directory might not exist in reality as we might be using a bare git repo. So we should just treat those as separate things.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update __init__.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-02-01 09:32:22 +02:00
Ruslan Kuprieievanddberenbaum df5b582c3f fetch: handle missing/modified imports (#10220) 
tests: pull after imp-url no-download

Co-authored-by: dberenbaum <[email protected]>
 2024-01-25 18:34:06 +02:00
Ruslan Kuprieiev ba06c0504a deps: bump dvc-data to >=3.8 (#10254) 2024-01-24 02:42:07 +02:00
Ruslan KuprieievandDave Berenbaum 78ab929ad1 config: azure: don't forget Coerce() for timeouts (#10247) 
Co-authored-by: Dave Berenbaum <[email protected]>
 2024-01-19 03:19:37 +02:00
Ruslan Kuprieiev f1de712710 index: data_tree: handle partial imports 2024-01-04 18:17:45 +02:00
Ruslan Kuprieiev fe008d9748 index: load files from import 2024-01-03 13:16:05 +02:00
Ruslan Kuprieiev dad464285e deps: bump dvc-data to >=3.6 2024-01-02 18:20:34 +02:00
Ruslan Kuprieiev 041ea17233 index: fix storage for imports 2024-01-02 13:38:39 +02:00
Ruslan Kuprieiev 987b1c4213 output: get rid of out.odb 
We use out.files to build index these days, making out.odb redundant.
 2024-01-02 07:24:43 +02:00
Ruslan Kuprieiev f2e479422f index: mark remote as read_only for imports 
Fixes #10206
 2023-12-28 08:08:45 +02:00
Ruslan Kuprieiev 73820cc596 push: don't try to update meta for non versioned remotes 
Per https://github.com/iterative/dvc/pull/10191#issuecomment-1870716262
 2023-12-28 06:11:16 +02:00
Ruslan Kuprieiev 402c107823 ui: push: use blank pbar for the header 
Same as in `dvc fetch`.
 2023-12-23 04:40:23 +02:00
Ruslan Kuprieiev b73754edd1 deps: bump dvc-data to >=3.3 2023-12-23 01:19:21 +02:00
Ruslan Kuprieiev 333f2f04f0 push: set desc explicitly 
Leftover from https://github.com/iterative/dvc/pull/10190
 2023-12-22 08:03:29 +02:00
Ruslan Kuprieiev 470c50a22b push: use set_size instead of set_total 
From https://github.com/iterative/dvc/pull/10191
 2023-12-22 07:55:52 +02:00
Ruslan Kuprieiev 01f49b7f24 push: only collect needed files when updating version_ids 2023-12-22 07:38:26 +02:00
Ruslan Kuprieiev 19f2246bee push: leave ui messages 
Same as we do in pull/fetch/checkout.
 2023-12-22 05:51:54 +02:00
Ruslan Kuprieiev 6a44783d72 index: load meta from outs.files 2023-12-22 05:26:18 +02:00
Ruslan Kuprieiev 98a286d88d repo: don't forget to close data_index when dropping 2023-12-22 05:26:18 +02:00
Ruslan Kuprieiev 499c2dda01 testing: don't forget to check dvc.lock for cloud versioned dirs 2023-12-21 01:05:08 +02:00
Ruslan Kuprieiev 5f2615ff53 deps: bump dvc-data to >=0.3.1 2023-12-21 01:05:08 +02:00
Ruslan Kuprieiev 3e4e143bee deps: update dvc-data to >=3 2023-12-15 08:02:40 +02:00
Ruslan Kuprieiev b1211366cd dvcfs: make repo creation lazy 2023-12-14 21:43:24 +02:00
Ruslan Kuprieiev 92823f1d84 tests: repro: don't depend on particular order 2023-12-14 21:43:24 +02:00
Ruslan Kuprieiev 3275f1ebf5 deps: bump dvc-data to >= 2.25.0 2023-12-14 05:04:36 +02:00
Ruslan Kuprieiev 4392af6d95 cli: debug: finalize profiler reports on errors 
Even if a command failed, it might still be useful to look at the profiler report.
 2023-12-13 20:34:30 +02:00
Ruslan Kuprieiev 8ccae0a086 dvcfs: implement basic du 2023-12-12 23:56:19 +02:00
Ruslan Kuprieiev 1d47b03cd5 bench: mark test_get/import as flaky (#10148) 
Sometimes seeing those error out in 2.x and early 3.x with:

```
ERROR: unexpected error - [Errno 95] no more link types left to try out: [Errno 95] Operation not supported
```

It seems like we've fixed it at some point, but I was not able to quickly find a link.
 2023-12-08 17:15:57 +02:00
Ruslan Kuprieiev 1e6b0242aa deps: bump dvc-data to 2.22.6 (#10144) 2023-12-06 18:41:52 +02:00
Ruslan Kuprieiev 9622157565 deps: bump dvc-data to >=2.22.5 (#10141) 2023-12-06 13:32:37 +02:00
Ruslan Kuprieiev 6ff4342f02 deps: bump dvc-data to >=2.22.4 (#10139) 
* deps: bump dvc-data to >=2.22.4

* Update test_odb.py
 2023-12-06 11:51:05 +02:00
Ruslan Kuprieiev 1aed230cd0 deps: bump dvc-oss to >=2.22.0 
First release with unpinned fsspec version.
 2023-12-05 18:48:36 +02:00
Ruslan Kuprieiev e18fd34b2c testing: bench: use --dataset instead of --size (#10134) 
This is no longer just about size (e.g. we have mnist/openorca/used-cars),
so the --dataset makes more sense.
 2023-12-04 14:43:37 +02:00
Ruslan Kuprieiev c0b64b0a10 testing: bench: add openorca (#10133) 
Need to move the data into dvc at some point.
 2023-12-04 12:43:50 +02:00
Ruslan Kuprieievandpre-commit-ci[bot] 3927e75ef1 testing: bench: add pytest_report_header (#10132) 
* testing: bench: add pytest_report_header

Useful for sanity checks in logs.

* Update dvc/testing/benchmarks/plugin.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-12-04 11:23:12 +02:00
Ruslan Kuprieiev b6d231f9cf testing: import virtualenv lazily 
Fixes https://github.com/iterative/dvc/issues/10118
 2023-11-28 22:20:34 +02:00
Ruslan Kuprieiev 322fcec5ca bench: test_sharing: use fetch instead of pull 2023-11-28 19:31:32 +02:00
Ruslan Kuprieiev 1e5b3603cc bench: test fetch instead of pull 
`pull` is literally `fetch` + `checkout` and our main interest there is downloading, so let's just test fetch instead.
 2023-11-28 19:31:32 +02:00
Ruslan Kuprieiev ffa630e8d5 push: handle missing/failed version_aware file 
Previously we used recorded index which had stub entries
for missing/failed outputs, but now we actually collect
everything ourselves and those are missing.
 2023-11-28 18:52:46 +02:00
Ruslan Kuprieiev 1e9b60dc48 deps: bump dvc-data to >=2.22.1 2023-11-27 16:17:42 +02:00
Ruslan Kuprieiev 2b3b15f767 deps: bump dvc-data to 2.22.0 
Related https://github.com/iterative/dvc/issues/10059
 2023-11-17 20:01:57 +02:00
Ruslan Kuprieiev 8c9358957c push: fix cloud versioning 2023-11-15 16:28:17 +02:00
Ruslan Kuprieiev e9f2da4451 repo: drop_data_index: delete all first level nodes 2023-11-13 11:23:08 +02:00
Ruslan Kuprieiev d60557dd08 deps: bump dvc-data to >=2.20.0 2023-10-31 04:51:30 +02:00
Ruslan Kuprieiev 3fee81f1be push: use index push 2023-10-26 20:58:45 +03:00
Ruslan Kuprieiev 4a0d56a793 cli: DictAction: set default metavar (#10047) 2023-10-20 16:46:14 +03:00
Ruslan Kuprieiev b35b9420a1 config: azure: expose timeouts (#10027) 2023-10-17 19:10:08 +03:00
Ruslan Kuprieiev 46c65fdba2 worktree: don't modify index entries in place (#10024) 2023-10-17 16:30:19 +03:00
Ruslan Kuprieiev 72fd265a8f get-url/ls-url: support remote:// with in-repo config 2023-10-13 18:58:23 +03:00
Ruslan Kuprieiev 50d6788411 exp: preserve hidden alias -M for --message (#10006) 
Followup for #10005
 2023-10-10 04:41:43 +03:00
Ruslan Kuprieiev 6394a503be deps: bump dvc-data to >=2.18.1 
To ensure people are getting a fix related to iterative/dvc#9826
 2023-10-04 16:03:22 +03:00
Ruslan Kuprieiev 00c10eb6bd repo: bump site_cache_dir salt 
Caused by a cache corruption fix in https://github.com/iterative/dvc-data/pull/447
 2023-10-04 04:01:26 +03:00
Ruslan Kuprieiev b0a25f11b7 deps: bump dvc-data to >= 2.18.0 2023-10-04 04:01:26 +03:00
Ruslan Kuprieiev 9f4b67cec4 site_cache_dir: use .dvc/tmp/btime and fallback to st_birthtime 
This will ensure that we won't be using index cache if you delete your repo
and start/clone fresh.
 2023-09-29 20:30:28 +03:00
Ruslan Kuprieiev 4d5a0163ea diff: handle granular targets 2023-09-28 21:54:06 +03:00
Ruslan Kuprieiev 0053b1ccd2 index: check parents when trying to detect hash_name 2023-09-28 21:54:06 +03:00
Ruslan Kuprieiev f981500862 cli: data_sync: show warning about missing default remote (#9979) 2023-09-26 23:29:33 +03:00
Ruslan Kuprieiev 2e1166d520 fetch: don't clear pbars 
There is more we can do during the `Fetching` stage, but it will
require additional tweaks, so just keeping it simple here.

Similar to https://github.com/iterative/dvc/pull/9974
 2023-09-26 01:27:59 +03:00
Ruslan Kuprieiev 18b4d610d6 checkout: tweak messages and don't clear pbars 2023-09-25 23:17:26 +03:00
Ruslan Kuprieiev b77218aa5b fetch: use ui.progress instead of raw Callback 2023-09-25 22:21:42 +03:00
Ruslan Kuprieiev 1fa9f1f6ea index: fix fs cache location 
Existing logic might result in weird fs cache location like
`.dvc/cache/files/md5/fs`, while we really only want
`.dvc/cache/fs`, since it is not an odb cache.
 2023-09-23 05:44:05 +03:00
Ruslan Kuprieiev 4bf19debb4 index: don't load external outs 2023-09-19 01:21:04 +03:00
Ruslan Kuprieiev e9c7cd63a2 dvcfs: add tests for fsid 2023-09-11 21:20:12 +03:00
Ruslan Kuprieiev 784f5a5d63 deps: bump dvc-data to 2.16.0 2023-09-07 04:13:56 +03:00
Ruslan Kuprieiev 1710e379f5 dvcfs: add basic fsid 
Implements fsspec's fsid https://github.com/fsspec/filesystem_spec/pull/1122

Required for #9904
 2023-09-07 04:13:56 +03:00
Ruslan Kuprieiev f47d21b0c2 [WIP] ls-url/get-url/import-url: introduce --fs-config (#9888) 
ls-url/get-url/import-url: introduce --fs-config

Fixes #9886
 2023-09-01 02:17:17 +03:00
Ruslan Kuprieiev 7a9da0f4a1 cli: utils: DictAction: limit split to 2 elements (#9887) 2023-08-30 03:15:21 +03:00
Ruslan Kuprieiev 8bfe8e3328 site_cache_dir: add salt to token (#9884) 
See comment in the code.
 2023-08-29 17:48:38 +03:00
Ruslan Kuprieiev c290d5c008 repo: don't try to gitignore custom cache locations (#9883) 
If user uses some custom cache location, there is really no reason for us
to try to gitignore it, as we have no idea what he is planning to do with
that and overall it often might be sign of this being misused. Maybe we could
also have some warnings/errors, but that's out of scope for this PR.

Fixes https://github.com/iterative/studio/issues/7405
 2023-08-29 16:31:37 +03:00
Ruslan Kuprieiev 736ca9a5e5 cache: check-missing: add type, oid name and oid 2023-08-23 02:07:32 +03:00
Ruslan Kuprieiev cae1b1b647 ls/ls-url: use tqmd instead of humanize.naturalsize 
Per https://github.com/iterative/dvc/pull/9857#issuecomment-1683185006
 2023-08-22 20:21:08 +03:00
Ruslan Kuprieiev 31ffce4fba ls/ls-url: use humanize.naturalsize instead of tqdm 
Currently we get odd formatting like

```
456    .devcontainer.json
139    .dvcignore
61.0   .gitattributes
96.0   .github
7.00   .gitignore
5.76k  README.md
160    data
1.84k  dvc.lock
519    dvc.yaml
4.00   foo
96.0   models
96.0   notebooks
225    params.yaml
46.0   requirements.txt
128    results
160    src
```

(notice stuff like 46.0)

and there doesn't seem to be a simple way to fix that in tqdm, in addition to
the general feeling that we are abusing tqdm's util for this.

So let's just use `humanize`, which gets us a more pleasing result:

```
456B  .devcontainer.json
139B  .dvcignore
61B   .gitattributes
96B   .github
7B    .gitignore
5.8K  README.md
160B  data
1.8K  dvc.lock
519B  dvc.yaml
4B    foo
96B   models
96B   notebooks
225B  params.yaml
46B   requirements.txt
128B  results
160B  src
```
 2023-08-18 02:54:31 +03:00
Ruslan Kuprieiev 72a42fce20 Update local.py 2023-08-18 00:31:52 +03:00
Ruslan Kuprieiev 1805edbd32 import: set core.remote when both --remote and --remote-config are specified 
From https://github.com/iterative/dvc/pull/9853#issuecomment-1682797083
 2023-08-18 00:16:33 +03:00
Ruslan Kuprieiev 1723abcd79 ls/ls-url: introduce --size 
I really need it only for studio right now, but this is really nice to have in a daily life
as well.

E.g.

```
$ dvc ls-url . --size
108    Metadata-and-Results.dvc
97     AudioMP3.dvc
97     AudioWAV.dvc
12181  README.md
54     .gitignore
168    .gitattributes
99     VideoFlash.dvc
312    LICENSE.txt
139    .dvcignore
96     AudioMP3
96     Asset
192    .dvc
64     Metadata-and-Results
64     AudioWAV
192    Scripts
64     VideoFlash
384    .git
96     .vscode
```

```
$ dvc list . --size
139    .dvcignore
168    .gitattributes
54     .gitignore
96     .vscode
96     Asset
96     AudioMP3
97     AudioMP3.dvc
64     AudioWAV
97     AudioWAV.dvc
312    LICENSE.txt
64     Metadata-and-Results
108    Metadata-and-Results.dvc
12181  README.md
192    Scripts
64     VideoFlash
99     VideoFlash.dvc
```
 2023-08-18 00:08:48 +03:00
Ruslan Kuprieiev 3081c60414 dep: repo: handle dict as config (#9853) 2023-08-17 19:37:04 +03:00
Ruslan Kuprieiev f0c1ff0219 fs: make _prepare_credentails/fs_args lazy (#9848) 
* tests: unit: move s3 tests to dvc-s3

* deps: bump dvc-data to 2.14.0

* tests: unit: move azure tests to dvc-azure
 2023-08-16 05:51:20 +03:00
Ruslan Kuprieiev 27c548575c pyinstaller: declare dvc_ssh as hidden import (#9840) 
Likely with removal of dvc-machine pyinstaller stopped picking up dvc_ssh automatically, so we need to declare it explicitly just like the rest of plugins.
 2023-08-13 00:26:43 +03:00
Ruslan Kuprieiev bee04c6d5f deps: bump dvc-data to 2.13.1 
Fixes #9733
 2023-08-11 05:53:19 +03:00
Ruslan Kuprieiev e9545c8b55 fetch: merge config instead of updating 
Update will replace sections instead of merging them together, which
is not desirable here.

Related https://github.com/iterative/studio/pull/6541
 2023-08-10 04:52:08 +03:00
Ruslan Kuprieiev 09cfd911bb config: use local config from local_dvc_dir 
This used to work like so but by accident, when we didn't reload config
at all when collecting other revisions.

Regression from https://github.com/iterative/dvc/pull/9758
 2023-08-10 04:36:42 +03:00
Ruslan Kuprieiev 4f61a63afd config: resolve: handle ~ home dir 
Fixes #9822
 2023-08-10 03:28:32 +03:00
Ruslan Kuprieiev c71b07b996 index: plot_keys: relparts -> parts 
`_plot_sources` is messed up and needs a rewrite hard, but this will
do for now for studio purposes.

Related https://github.com/iterative/studio/pull/6541
 2023-08-08 17:14:28 +03:00
Ruslan Kuprieiev f4c0946f65 repo: cache: support some view options 2023-08-08 01:09:50 +03:00
Ruslan Kuprieiev e35fb76dac repo: add basic cache_check_missing 2023-08-07 22:23:49 +03:00
Ruslan Kuprieiev 722839dda6 fetch: add basic onerror 
Related https://github.com/iterative/studio/pull/6541
 2023-08-07 03:24:39 +03:00
Ruslan Kuprieiev d5533348f7 deps: bump dvc-data to 2.12.0 2023-08-07 02:42:41 +03:00
Ruslan Kuprieiev 43b371c43a deps: bump dvc-data to 2.11.0 2023-08-04 15:42:25 +03:00
Ruslan Kuprieiev 9d1d51caa7 deps: bump dvc-data to 2.10.0 2023-08-04 03:40:36 +03:00
Ruslan Kuprieiev 8a382f51fd deps: bump dvc-data to 2.9.0 2023-08-03 02:05:14 +03:00
Ruslan Kuprieiev b235487ecc ls/import/get: introduce --remote and --remote-config 
Fixes #9656
 2023-08-02 16:28:16 +03:00
Ruslan Kuprieiev 7c60dfd3c5 deps: bump dvc-data to >=2.8.1 2023-07-30 04:18:09 +03:00
Ruslan Kuprieiev 225814fdde dvc: don't use realpath where not needed 2023-07-30 02:24:00 +03:00
Ruslan Kuprieiev ca0ad239f7 deps: bump dvc-data to 0.28.0 2023-07-28 01:37:46 +03:00
Ruslan Kuprieiev f57ac609e9 deps: bump dvc-data to 0.27.1 2023-07-27 20:44:34 +03:00
Ruslan Kuprieiev cc4f505f99 api: open/read: introduce remote_config 2023-07-26 16:29:14 +03:00
Ruslan Kuprieiev 4d011718a6 out: str: don't try to relpath paths outside of the repo 
This is not very useful overall, but also results in a ValueError from
relpath on windows if your repo and output are on different drives.
 2023-07-25 23:49:27 +03:00
Ruslan Kuprieiev 9c89841637 fetch: support passing config 
For now not exposing it in CLI since there is no need/use yet.

Requirement for https://github.com/iterative/studio/pull/6541
 2023-07-25 20:03:38 +03:00
Ruslan Kuprieiev 73dad4780f repo: brancher: reset parsed fields 
Related #9754 and fixes `dvc fetch`.
 2023-07-25 14:00:33 +03:00