Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-01 23:08:43 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,828 Commits 7 Branches 579 Tags
c90947f33c649d0b1930fd01a6695aa9e3839ca8
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
skshetry c90947f33c make it a dvc repo (#9670) 2023-06-29 08:50:03 +05:45
skshetry 1ca1d78b38 deps: bump gdrive to 2.20 (#9657) 2023-06-23 09:13:35 +05:45
skshetry fd4696cc50 Revert "drop stage-level vars" (#9647) 
Revert "drop stage-level vars (#9556)"

This reverts commit 358c5e0c39.
 2023-06-22 18:44:51 +05:45
skshetry c6c2ce130a remove dvc._build (#9597) 2023-06-13 20:31:22 +05:45
skshetry be975ad9eb pyproject.toml: reformat project table (#9596) 2023-06-13 20:02:34 +05:45
skshetry 30517f6676 deps: update dvc-task/dvc-s3/pytest-cov (#9592) 
* remove deprecation warnings

* Update pyproject.toml
 2023-06-13 15:39:34 +05:45
skshetry ed664143b4 dvc.build: move logic out of __init__ to dvc.build (#9563) 2023-06-08 08:19:59 +05:45
skshetry 358c5e0c39 drop stage-level vars (#9556) 
drop stage-level vars support
 2023-06-07 11:10:08 +05:45
skshetry 9d7fba5c61 fix tests (#9555) 2023-06-07 10:28:54 +05:45
skshetry a1e339b4fa .dvc file: remove cmd support (#9554) 2023-06-07 09:30:13 +05:45
skshetry 6c58d438b3 tests: remove single_stage tests from repro/run tests (#9553) 2023-06-07 09:25:08 +05:45
skshetry 0c709b6ec1 .dvc file: remove metrics/plots support (#9552) 2023-06-06 20:42:09 +05:45
skshetry 0aeb7a8662 .dvc file: remove params support (#9551) 2023-06-06 20:05:36 +05:45
skshetry 46f3157f43 add: rename jobs to remote_jobs to mirror CLI; move validation to CLI (#9550) 
- Follow up on https://github.com/iterative/dvc/pull/9522, renames jobs kwarg to remote_jobs
  to mirror CLI.
- Also moves validation logic to CLI.
 2023-06-06 19:30:39 +05:45
skshetry 36fd69be21 add: remove file (#9548) 
* add: remove `file`

Follow up to #9547.

* Update add.py

* fix tests
 2023-06-06 18:44:19 +05:45
skshetry 3759535d43 tests: remove single_stage stage runs (#9549) 2023-06-06 18:38:07 +05:45
skshetry 26152292aa remove support for reading and generating Dvcfile (#9546) 2023-06-06 15:29:10 +05:45
skshetry 3e0f2bafbb add: small misc refactoring (#9540) 2023-06-05 18:41:17 +05:45
skshetry 355ebc72eb add: remove --recursive, rename fname to file (#9513) 
add: remove --recursive
 2023-05-26 12:07:13 +05:45
skshetry 192c9cc577 remove original PR from backport PR title (#9510) 2023-05-25 16:37:25 +05:45
skshetry 0e38525d8d cli: remove dvc run (#9508) 
Note that `Repo.run()` API still exists, which will be removed in
successive PRs. A lot of tests depend on it that it'll take time
before removing it.
 2023-05-25 16:27:14 +05:45
skshetry 4163c8f57d set destination branch to PR title on backport (#9492) 2023-05-22 16:03:36 +05:45
skshetry 6532258e4a drop top-level plots definition as mapping support (#9486) 2023-05-22 15:08:22 +05:45
skshetry abd4f4a2a8 drop Repo.stages (#9489) 
Use repo.index.stages instead.
 2023-05-22 12:51:40 +05:45
skshetry f41e535857 remove dvc.main.main alias (#9488) 
You can import it using:
```python
from dvc.cli import main

main(...)
```
 2023-05-22 12:46:46 +05:45
skshetry 1989c0fe98 stage add: introduce --run option (#9480) 
* stage add: introduce --run option

Part of #5846.

* fix ui message
 2023-05-22 12:27:20 +05:45
skshetry e089ee0365 remove data ls and support for --meta/--desc/--label/--type cli flags (#9481) 2023-05-19 20:10:34 +05:45
skshetry 3959c51373 lockfile: drop 1.0 lockfile support (#9476) 2023-05-18 17:36:05 +05:45
skshetry 90aaf5a296 commands: remove --show-json/--show-md/--show-csv flags (#9475) 2023-05-18 16:36:14 +05:45
skshetry 99daf5d113 add: fail loudly if there's no dir cache (#9466) 2023-05-17 12:03:06 +05:45
skshetry 0dbdfd2b9b run tests and codeql on 2.x branch (#9465) 2023-05-17 09:17:36 +05:45
skshetryandpre-commit-ci[bot] 6d4251798a ci(Mergify): configuration update (#9463) 
* ci(Mergify): configuration update

Signed-off-by: null <null>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: null <null>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-05-17 09:09:33 +05:45
skshetry 11d031935d packaging: use PyPI trusted publishers (#9459) 
https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers
 2023-05-17 06:26:19 +05:45
skshetry 004f49744e tests: isolate dvc config and directories (#9450) 
* tests: isolate dvc config and directories

by introducing envvars to change default values. This lets pytest
handle the deletion of tmpdirs, instead of filling /var/tmp with
hundreds of thousands of directories. Also isolates test run from
faulty global/system config files.

* move to dvc/env and sort
 2023-05-15 18:16:02 +05:45
skshetry 915dd5062e disable nested progress bar during checkout (#9446) 2023-05-12 18:43:33 +05:45
skshetry d08c85f7d1 fix checkout callback during add/commit (#9445) 
* fix checkout callback

Fix https://github.com/iterative/dvc/issues/6226.

* bump minversion of dvc-data to 0.49.1
 2023-05-12 18:14:13 +05:45
skshetry 2342099bd8 add: support virtual operation (#9389) 2023-05-11 08:31:25 +05:45
skshetry e93e8f05a3 exp push: notify studio even if cache push fails (#9380) 
* exp push: notify Studio even when cache push fails

* exp push: refactor and extract exp_ref resolving logic as the function is getting complex
 2023-05-02 08:46:10 +05:45
skshetry a4cc198ac7 exp push: soft wrap link so that the URL does not break (#9368) 
soft wrap link so that URL does not break
 2023-04-27 06:50:33 +05:45
skshetry e1bf418e72 tests: lazy tpi imports, ignore deprecation warning from setuptools (#9362) 2023-04-24 21:34:07 +05:45
skshetry b2101a85ad reproduce: remove callback (#9354) 
Callback was added for working around scm_context, since we don't use it, we don't need callback now.
 2023-04-21 13:01:00 +05:45
skshetry 10f523430b exp run: do not autostage on dvc repro (#9344) 2023-04-19 16:11:13 +05:45
skshetry 5563a595d5 utils.studio: allow any base url to be set, not just the hostname (#9341) 
This keeps it in symmetry with
https://github.com/iterative/dvc-studio-client/pull/28, and allows
setting the base url with path, not just limited to netloc.

This was always intended to work, urljoin with the added slash was
causing this bug.
 2023-04-18 22:22:26 +05:45
skshetry dd3deb6cbd add DVC_STUDIO_URL envvar support (#9340) 
add DVC_STUDIO_URL envvar
 2023-04-18 09:56:17 +05:45
skshetry 7841020b2b [DO NOT MERGE] release: remove feature flags for experiment sharing (#9257) 
release: remove feature flags for experiment sharing
 2023-04-17 12:53:27 +05:45
skshetry a91993616f logger: add DVC_SHOW_TRACEBACK envvar to show traceback on exception (#9310) 
This will come useful in debugging, where you don't have to add --verbose just to
see where it failed or go through verbose logs (which might be useful when working
with UI issues as you want to see what the user sees).

Ideally this should be handled in main when we exit with error instead of using
logger.{warning,error,exception} to log exception as the intention is to only log
last exception that makes it exit. But since we don't have that capability and
we don't abuse logging.{warning,error,exception} to log and suppress exceptions, this is a good
enough solution for now without complicating `dvc.cli.main()`.
 2023-04-06 09:56:19 +05:45
skshetry 3c570a2177 exp push: improve ui (#9308) 
* exp push: improve ui

* fix misc suggestions
- show url even when the refs are up-to-date.
- show url at the end
- add Git remote information in up-to-date UI message.

* fix diverged exps message
 2023-04-06 09:54:39 +05:45
skshetry cf52efb8c3 notify_refs; rename studio_url to base_url (#9303) 2023-04-04 17:43:05 +05:45
skshetry 8be71be540 utils.studio: disallow redirects, misc API refactoring (#9301) 2023-04-04 13:15:08 +05:45
skshetry 7c3c47b4a2 bump ruff to 0.0.260 and enable bandit (#9300) 
* bump ruff to 0.0.260 and enable bandit

* install toml extra for bandit

* suppress insecure use of hash md5
 2023-04-04 10:01:43 +05:45
skshetry 086d51d9de plots json: return definition errors unrelated to renderers (#9293) 2023-03-31 16:34:26 +05:45
skshetry 1b9d40960f utils.studio: reduce verbosity on exception logging (#9284) 2023-03-30 12:55:19 +05:45
skshetry 696fabc745 exp-push: show url to the Studio project (#9283) 2023-03-30 12:52:09 +05:45
skshetry 780dc72301 Revert "exp-push: show url to the Studio project" (#9282) 
Revert "exp-push: show url to the Studio project (#9278)"

This reverts commit 1ca3ef14a3.
 2023-03-30 08:28:44 +05:45
skshetry 1ca3ef14a3 exp-push: show url to the Studio project (#9278) 2023-03-30 08:27:47 +05:45
skshetry a6c89cac8d replace funcy.cat with itertools.chain.from_iterable (#9279) 
It's just an alias to the function from itertools.
 2023-03-29 20:56:26 +05:45
skshetry 593f1064cf utils.studio: refactor to return json response (#9277) 2023-03-29 20:02:40 +05:45
skshetry efc2d7ad3d remove unused type-ignore comment (#9258) 2023-03-28 14:30:33 +05:45
skshetry 7c65739e7e refactor notify_refs_to_studio to take in repo instance as dependency (#9256) 2023-03-28 13:53:31 +05:45
skshetry cc4816f652 utils.studio: require token (#9255) 
This is a simple refactoring which makes the caller to choose appropriate token instead of
the studio.notify_refs making a decision for them.
 2023-03-28 13:32:05 +05:45
skshetry faa1bdfbce use git remote to find out remote repo url when pinging Studio (#9254) 2023-03-28 12:17:48 +05:45
skshetry 006b6f3513 exp push/remove: show message on failure when notifying to Studio (#9253) 2023-03-28 09:40:41 +05:45
skshetry 1bc4c1dceb exp share refactor - ping on exp:remove, do not fail on error (#9248) 
studio experiments share refactor

1) Pings Studio on exp remove
2) Does not fail on any error during request.post
3) UI improvement on exp remove
 2023-03-27 22:51:50 +05:45
skshetry 531b2110cf Revert "exp push: add trailing slash to the endpoint url" (#9243) 
Revert "exp push: add trailing slash to the endpoint url (#9240)"

This reverts commit 5485275b92.
 2023-03-27 14:19:12 +05:45
skshetry c1917ed751 output: ignore missing size from files when calculating total size (#9242) 
output: ignore missing size from files when calculating hash info
 2023-03-27 12:56:43 +05:45
skshetry 43195a806d plots: return errors in json format (#9146) 
* plots: return errors in json format

* plots --json: move plots data into data key

* rename  revision key to rev

* move imports to top-level

* add tests
 2023-03-27 12:53:43 +05:45
skshetry 5485275b92 exp push: add trailing slash to the endpoint url (#9240) 2023-03-24 22:35:20 +05:45
skshetry 66406e6c99 scm: fetch_all_exps pass kwargs to scm.fetch_refspecs (#9214) 2023-03-20 21:26:24 +05:45
skshetry 8399bf612a sconfig: add support for studio token (#9213) 
config: add support for studio token
 2023-03-20 20:59:52 +05:45
skshetry 4ee3271b0c dvc.repo.exp: import some modules lazily (#9207) 2023-03-18 17:24:01 +05:45
skshetry dae8515933 plots: small refactorings (#9206) 2023-03-17 19:42:27 +05:45
skshetry a2066d75f9 add a note why we are flushing (#9205) 2023-03-17 17:52:04 +05:45
skshetry 283d8245e1 flush streams before fork (#9203) 2023-03-17 17:14:34 +05:45
skshetry c52b12ac2a dvcfs: support caching remote streams; use that in plots (#9183) 
* dvcfs: support caching remote streams; use that in plots

* plots: show progress bar, cap max_workers to 16

* bump dvc-data to 0.44.1
 2023-03-17 16:04:16 +05:45
skshetry 30c3e3e4d4 exp push: set default endpoint (#9198) 2023-03-16 17:50:38 +05:45
skshetry 58029e9719 exp push: push experiments revs to studio (#9160) 
* exp push: push experiments revs to studio

* log json data that is sent
 2023-03-16 11:24:50 +05:45
skshetry e6419b939d do not disable colorama in tests (#9187) 
skip disabling colorama in tests

It might affect pytest as it wraps sys.stderr/sys.stdout for capsys fixture.
This is a temporary hack, as we might get rid of colorama soon.
 2023-03-15 18:23:26 +05:45
skshetry 8312eb9b42 ui: disable colorama on write_json when highlighting is disabled (#9186) 
On a large json output (say ~300MB), this saves about ~6 seconds of time.
 2023-03-15 17:42:45 +05:45
skshetry 16ecd53b71 tests: remove gc.collect() from windows (#9159) 2023-03-13 13:37:57 +05:45
skshetry ef22b1d77e plots: disable syntax highlighting on plots json output (#9155) 
I don't think it's worth it to highlight given the size of json output.
 2023-03-11 00:50:06 +05:45
skshetry aaa825a9a8 config: ignore unspecified keys from feature section (#9153) 
A warning is shown to the user for the unsupported/ignored keys, and then ignored.
The warning is raised only once per interpreter session.
 2023-03-10 21:52:05 +05:45
skshetry 2b889baa5e plots: avoid printing anything on --json output (#9144) 2023-03-09 16:10:46 +05:45
skshetry 48a63ca588 try to improve test speed (#9136) 
* tests: fix logic to gc.collect() on Windows + 3.11

* dvc.cli: cache get_main_parser()

* tests: isolate environment on a session-level

* bring back gc.collect()
 2023-03-08 22:11:42 +05:45
skshetry 08a50ee36e tests: error on ResourceWarning (#9123) 
* tests: error on ResourceWarning

* run-cache: avoid leaking temp files on Windows

* set returncode to popen to avoid ResourceWarning on gc.collect()
 2023-03-08 15:36:10 +05:45
skshetry 51db1765ee migrate from appdirs to platformdirs (#9131) 
appdirs has been deprecated/unmaintained, platformdirs is now recommended.
Closes #9130.
 2023-03-07 23:13:28 +05:45
skshetry 93eea0cd42 deps: bump scmrepo to 0.1.15; close subrepo.dvc on tests teardown (#9122) 2023-03-06 20:21:42 +05:45
skshetry 25fd274090 increase timeout for tests to 40 minutes 2023-03-06 20:04:26 +05:45
skshetry ddfa607437 tests: don't leak files/descriptors when tests finish (#9119) 
close open files/descriptors when tests finish
 2023-03-06 18:06:37 +05:45
skshetry c5a3566e49 deps (tests): require pytest-xdist>=3.2, use -n=logical workers and workstealing scheduler (#9116) 2023-03-06 15:54:45 +05:45
skshetry e7d4f5b24e tests: filter warnings (#9115) 2023-03-06 15:41:45 +05:45
skshetry 4c5d052d8c update ruff to v0.0.254 (#9110) 2023-03-04 17:26:59 +05:45
skshetry 4db2e89938 build: use macos-11 to build package (#9091) 
macos-10.15 image has been deprecated for a while, see https://github.blog/changelog/2022-07-20-github-actions-the-macos-10-15-actions-runner-image-is-being-deprecated-and-will-be-removed-by-8-30-22/ and https://github.com/actions/runner-images/issues/5583.

setup-ruby removed the support for these images yesterday: https://github.com/ruby/setup-ruby/pull/467.

which fails our package builds. Now using macos-11 to build packages.
 2023-02-28 17:33:05 +05:45
skshetry 4a6f3ed64a dvc.testing: fix lockfile in docker_services fixture (#9088) 
dvc.testing.docker_services: rollback to previous state

where we were not tearing down the containers
 2023-02-28 16:54:33 +05:45
skshetry 2a2c02c184 remove typing.cast (#9082) 2023-02-26 13:15:54 +05:45
skshetry 8edaef0103 deps: drop importlib-resources and typing-extensions (#9055) 
deps: remove importlib-resources and typing-extensions
 2023-02-19 11:36:48 +05:45
skshetry fd8a8c2729 pyproject: format using pyproject-fmt (#9053) 
I haven't added pyproject-fmt to pre-commit hook because of two reasons:
1. There is a bug that keeps adding empty lines after `[[tool.mypy.overrides]]`.
2. The list gets exploded to multiple lines. However, this is a minor issue.
 2023-02-18 23:07:57 +05:45
skshetry 5426ca6f79 deps (tests): unpin test deps (#9052) 2023-02-18 22:41:50 +05:45
skshetry a89c25700f pyproject.toml: cleanup metadata (#9051) 2023-02-18 17:40:36 +05:45
skshetry 1de37556d5 bump ruff to v0.0.247 (#9050) 
* bump ruff to v0.0.247

* ruff: enable pep8-naming check

rename some camelCased APIs, especially in logger
rename G variables to graph
ignore some variable namings and help messages in dvc.commands
 2023-02-18 16:50:35 +05:45
skshetry d7ea7c358d Revert "scm: use dulwich backend when fetching exps during clone/pull" (#9041) 
Revert "scm: use dulwich backend when fetching exps during clone/pull (#9023)"

This reverts commit ed1ab7fa75.
 2023-02-16 18:07:45 +05:45