Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,972 Commits 7 Branches 579 Tags
d81a1bbf561afd2529434c6dbcb5ba527dd73126
Commit Graph
8972 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
dberenbaum d81a1bbf56 make exp run --dry imply --temp 2023-09-13 15:06:46 +09:00
Bilel Omrani ba0834caa0 Fix Repo instantiation error when using fsspec chaining API 2023-09-12 21:41:48 +03:00
pre-commit-ci[bot] 9a4578cc72 [pre-commit.ci] pre-commit autoupdate (#9937) 
updates:
- [github.com/psf/black: 23.7.0 → 23.9.1](https://github.com/psf/black/compare/23.7.0...23.9.1)
- [github.com/astral-sh/ruff-pre-commit: v0.0.287 → v0.0.288](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.287...v0.0.288)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-09-12 20:44:17 +05:45
Helio Machadoandskshetry ccf461bc84 Migrate from PyPI tokens to Trusted Publishers (#9933) 
* Migrate Python repositories to Trusted Publishers

* Update build.yaml

* Update .github/workflows/build.yaml

Co-authored-by: Helio Machado <[email protected]>

* Update build.yaml

---------

Co-authored-by: skshetry <[email protected]>
 2023-09-12 20:27:37 +05:45
Ruslan Kuprieiev e9c7cd63a2 dvcfs: add tests for fsid 2023-09-11 21:20:12 +03:00
skshetry 33b6c1d876 params/metrics: cache remote files (#9932) 2023-09-11 19:13:47 +05:45
skshetry f2673acb53 metrics/params: support reading from cached files and directories, use repo-relative path consistently (#9909) 
* metrics

* fix path handling

* fix dvc metrics diff

* params show/diff

* share common utils

* remove onerror_collect from exp_show

* extract read_params

* move around code

* fix path handling on Windows

* rename _show to _gather_metrics

* remove _collect_vars

* add tests

* Revert "remove _collect_vars"

This reverts commit 6d2cb500c32a882497406d66fb13e613790c3436.

* bring back _collect_vars

* fix tests

* fix paths
 2023-09-11 17:41:37 +05:45
David de la Iglesia Castro 7e52c233ae import: Expose --force flag (#9927) 2023-09-08 18:43:13 +05:45
Peter Rowlands (변기호) 408d23f285 repo: use is_repo_import to check for imports in outs_graph (#9925) 
repo: use is_repo_import to check for imports
 2023-09-08 16:13:11 +05:45
David de la Iglesia Castro 7fb8a9a427 checkout: Expose --allow-missing arg. (#9919) 3.19.0 2023-09-07 09:55:50 +00:00
Ruslan Kuprieiev 784f5a5d63 deps: bump dvc-data to 2.16.0 2023-09-07 04:13:56 +03:00
Ruslan Kuprieiev 1710e379f5 dvcfs: add basic fsid 
Implements fsspec's fsid https://github.com/fsspec/filesystem_spec/pull/1122

Required for #9904
 2023-09-07 04:13:56 +03:00
Joe Scott aa9636e2ae Typo rwlock.py#107 Remove -> Removed 
closes #9920
 2023-09-06 09:21:29 -04:00
Peter Rowlands 65d412bfa3 stage.run: always set DVC_ROOT env var 2023-09-06 18:43:52 +09:00
Peter Rowlands 20606d480b exp show: drop internal legacy checkpoint support 2023-09-06 14:58:53 +09:00
dependabot[bot] e2aedb7e97 build(deps): bump actions/checkout from 3 to 4 (#9911) 
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
3.18.0 		2023-09-05 08:43:43 +05:45
pre-commit-ci[bot] bacae48f96 [pre-commit.ci] pre-commit autoupdate (#9910) 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.286 → v0.0.287](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.286...v0.0.287)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-09-05 08:34:38 +05:45
Peter Rowlands bf119e70a9 exp run: track files for skipped/run-cached stages 2023-09-04 16:23:54 +09:00
BradyJ27 3528c37b66 move: Alias mv to move (#9905) 
* Alias mv to move

* Add alias for mv

* Add aliases for rm, exp rm, exp ls
 2023-09-03 22:11:41 +03:00
Ruslan Kuprieiev f47d21b0c2 [WIP] ls-url/get-url/import-url: introduce --fs-config (#9888) 
ls-url/get-url/import-url: introduce --fs-config

Fixes #9886
 2023-09-01 02:17:17 +03:00
Ruslan Kuprieiev 7a9da0f4a1 cli: utils: DictAction: limit split to 2 elements (#9887) 2023-08-30 03:15:21 +03:00
Ruslan Kuprieiev 8bfe8e3328 site_cache_dir: add salt to token (#9884) 
See comment in the code.
3.17.0 		2023-08-29 17:48:38 +03:00
Ruslan Kuprieiev c290d5c008 repo: don't try to gitignore custom cache locations (#9883) 
If user uses some custom cache location, there is really no reason for us
to try to gitignore it, as we have no idea what he is planning to do with
that and overall it often might be sign of this being misused. Maybe we could
also have some warnings/errors, but that's out of scope for this PR.

Fixes https://github.com/iterative/studio/issues/7405
 2023-08-29 16:31:37 +03:00
Matt Seddon 3c2c1fe72a Remove dead Stage method (#9881) 2023-08-29 13:10:40 +00:00
pre-commit-ci[bot] 0602eca00e [pre-commit.ci] pre-commit autoupdate (#9882) 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.285 → v0.0.286](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.285...v0.0.286)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-08-29 07:17:32 +05:45
Matt Seddon 5b69a16ea7 Add DVC_ROOT env var into queued experiment runs (#9879) 2023-08-28 22:37:03 +00:00
Dave Berenbaumandpre-commit-ci[bot] ef260eaf2e exp show: show metrics that include separator (#9819) 
* exp show: show metrics that include separator

* sort using partition instead of lpartition

* handle ':' in file and metric names

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix sorting with leading :

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-08-24 10:31:56 -04:00
Peter Rowlands 88431b1c0a update: preserve def_path for version-aware import update 2023-08-24 16:16:43 +09:00
dependabot[bot] bd2dc9dfed build(deps): bump rtCamp/action-slack-notify from 2.2.0 to 2.2.1 (#9874) 
Bumps [rtCamp/action-slack-notify](https://github.com/rtcamp/action-slack-notify) from 2.2.0 to 2.2.1.
- [Release notes](https://github.com/rtcamp/action-slack-notify/releases)
- [Commits](https://github.com/rtcamp/action-slack-notify/compare/v2.2.0...v2.2.1)

---
updated-dependencies:
- dependency-name: rtCamp/action-slack-notify
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-24 08:30:32 +05:45
Dave Berenbaumandpre-commit-ci[bot] 428c094b8d update msg for multiple overlapping outputs (#9862) 
* update msg for multiple overlapping outputs

* update msg for multiple overlapping outputs

* fix type checking

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use sets of stages to fix mypy errors

* fix tests/func/test_add.py::test_try_adding_multiple_overlaps

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-08-24 00:06:32 +03:00
Dave Berenbaum fc955c1917 Rename partial add/remove benchmarks (#9868) 
* Rename partial add/remove benchmarks

* Update dvc/testing/benchmarks/cli/stories/test_modify_data.py
 2023-08-23 19:04:51 +05:45
skshetry 8cb7e4948d tests: dvcfs: use fspath while testing (#9872) 
get_file does not support PathLike objects. fsspec does not support them in get_file.

See https://github.com/fsspec/filesystem_spec/issues/896.
Fixes test failures in dvc-webhdfs and dvc-hdfs.
 2023-08-23 17:33:22 +05:45
skshetry 8e353f9fce deps: unpin dvc-oss (#9871) 2023-08-23 17:20:41 +05:45
dependabot[bot] a4e2bce5b2 build(deps-dev): bump dvc-oss from 2.19 to 2.20.0 (#9870) 
Bumps [dvc-oss](https://github.com/iterative/dvc-oss) from 2.19 to 2.20.0.
- [Release notes](https://github.com/iterative/dvc-oss/releases)
- [Commits](https://github.com/iterative/dvc-oss/compare/2.19.0...2.20.0)

---
updated-dependencies:
- dependency-name: dvc-oss
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-23 08:45:43 +05:45
Ruslan Kuprieiev 736ca9a5e5 cache: check-missing: add type, oid name and oid 3.16.0 2023-08-23 02:07:32 +03:00
Ruslan Kuprieiev cae1b1b647 ls/ls-url: use tqmd instead of humanize.naturalsize 
Per https://github.com/iterative/dvc/pull/9857#issuecomment-1683185006
 2023-08-22 20:21:08 +03:00
pre-commit-ci[bot]andSaugat Pachhai 3783d65dde [pre-commit.ci] pre-commit autoupdate (#9841) 
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.282 → v0.0.285](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.282...v0.0.285)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix ruff issues

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
 2023-08-22 15:22:57 +05:45
skshetry 0342e4f1c3 fix type hints (#9863) 2023-08-22 14:56:11 +05:45
skshetry fc9af7a976 assert that the test is getting parametrized correctly (#9859) 
Otherwise this may end up generating 0 or not enough commands, and passing tests
 2023-08-19 13:57:48 +05:45
skshetry fea59cfc52 tests: simplify test parametrization (#9858) 
i.e. remove metafunc/pytest_generate_tests
 2023-08-19 10:02:31 +05:45
Ruslan Kuprieiev 31ffce4fba ls/ls-url: use humanize.naturalsize instead of tqdm 
Currently we get odd formatting like

```
456    .devcontainer.json
139    .dvcignore
61.0   .gitattributes
96.0   .github
7.00   .gitignore
5.76k  README.md
160    data
1.84k  dvc.lock
519    dvc.yaml
4.00   foo
96.0   models
96.0   notebooks
225    params.yaml
46.0   requirements.txt
128    results
160    src
```

(notice stuff like 46.0)

and there doesn't seem to be a simple way to fix that in tqdm, in addition to
the general feeling that we are abusing tqdm's util for this.

So let's just use `humanize`, which gets us a more pleasing result:

```
456B  .devcontainer.json
139B  .dvcignore
61B   .gitattributes
96B   .github
7B    .gitignore
5.8K  README.md
160B  data
1.8K  dvc.lock
519B  dvc.yaml
4B    foo
96B   models
96B   notebooks
225B  params.yaml
46B   requirements.txt
128B  results
160B  src
```
 2023-08-18 02:54:31 +03:00
dependabot[bot] 2da959acd1 build(deps-dev): update pytest-docker requirement from <2,>=1 to >=1,<3 
Updates the requirements on [pytest-docker](https://github.com/avast/pytest-docker) to permit the latest version.
- [Release notes](https://github.com/avast/pytest-docker/releases)
- [Changelog](https://github.com/avast/pytest-docker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/avast/pytest-docker/compare/v1.0.0...v2.0.0)

---
updated-dependencies:
- dependency-name: pytest-docker
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
 2023-08-18 00:41:47 +03:00
Ruslan Kuprieiev 72a42fce20 Update local.py 2023-08-18 00:31:52 +03:00
Dave BerenbaumandRuslan Kuprieiev 1cbe64face Update tests/unit/command/test_experiments.py 
Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2023-08-18 00:31:52 +03:00
dberenbaum 057b0112ea exp: make run/save less noisy 2023-08-18 00:31:52 +03:00
Ruslan Kuprieiev 1805edbd32 import: set core.remote when both --remote and --remote-config are specified 
From https://github.com/iterative/dvc/pull/9853#issuecomment-1682797083
 2023-08-18 00:16:33 +03:00
Ruslan Kuprieiev 1723abcd79 ls/ls-url: introduce --size 
I really need it only for studio right now, but this is really nice to have in a daily life
as well.

E.g.

```
$ dvc ls-url . --size
108    Metadata-and-Results.dvc
97     AudioMP3.dvc
97     AudioWAV.dvc
12181  README.md
54     .gitignore
168    .gitattributes
99     VideoFlash.dvc
312    LICENSE.txt
139    .dvcignore
96     AudioMP3
96     Asset
192    .dvc
64     Metadata-and-Results
64     AudioWAV
192    Scripts
64     VideoFlash
384    .git
96     .vscode
```

```
$ dvc list . --size
139    .dvcignore
168    .gitattributes
54     .gitignore
96     .vscode
96     Asset
96     AudioMP3
97     AudioMP3.dvc
64     AudioWAV
97     AudioWAV.dvc
312    LICENSE.txt
64     Metadata-and-Results
108    Metadata-and-Results.dvc
12181  README.md
192    Scripts
64     VideoFlash
99     VideoFlash.dvc
```
 2023-08-18 00:08:48 +03:00
Ruslan Kuprieiev 3081c60414 dep: repo: handle dict as config (#9853) 2023-08-17 19:37:04 +03:00
dependabot[bot] 152991d878 build(deps-dev): bump mypy from 1.4.1 to 1.5.1 (#9852) 
Bumps [mypy](https://github.com/python/mypy) from 1.4.1 to 1.5.1.
- [Commits](https://github.com/python/mypy/compare/v1.4.1...v1.5.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-17 18:43:31 +05:45
skshetry 8098fd109d plots: fix caching of legacy format (#9849) 
* plots: fix caching of legacy format

* deps: bump dvc-data minversion to 2.15.0
 2023-08-17 07:40:07 +05:45