Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,701 Commits 7 Branches 579 Tags
004f49744e94c7651e4eea7b57e226f25de569e4
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
skshetry 004f49744e tests: isolate dvc config and directories (#9450) 
* tests: isolate dvc config and directories

by introducing envvars to change default values. This lets pytest
handle the deletion of tmpdirs, instead of filling /var/tmp with
hundreds of thousands of directories. Also isolates test run from
faulty global/system config files.

* move to dvc/env and sort
 2023-05-15 18:16:02 +05:45
skshetry 915dd5062e disable nested progress bar during checkout (#9446) 2023-05-12 18:43:33 +05:45
skshetry d08c85f7d1 fix checkout callback during add/commit (#9445) 
* fix checkout callback

Fix https://github.com/iterative/dvc/issues/6226.

* bump minversion of dvc-data to 0.49.1
 2023-05-12 18:14:13 +05:45
skshetry 2342099bd8 add: support virtual operation (#9389) 2023-05-11 08:31:25 +05:45
skshetry e93e8f05a3 exp push: notify studio even if cache push fails (#9380) 
* exp push: notify Studio even when cache push fails

* exp push: refactor and extract exp_ref resolving logic as the function is getting complex
 2023-05-02 08:46:10 +05:45
skshetry a4cc198ac7 exp push: soft wrap link so that the URL does not break (#9368) 
soft wrap link so that URL does not break
 2023-04-27 06:50:33 +05:45
skshetry e1bf418e72 tests: lazy tpi imports, ignore deprecation warning from setuptools (#9362) 2023-04-24 21:34:07 +05:45
skshetry b2101a85ad reproduce: remove callback (#9354) 
Callback was added for working around scm_context, since we don't use it, we don't need callback now.
 2023-04-21 13:01:00 +05:45
skshetry 10f523430b exp run: do not autostage on dvc repro (#9344) 2023-04-19 16:11:13 +05:45
skshetry 5563a595d5 utils.studio: allow any base url to be set, not just the hostname (#9341) 
This keeps it in symmetry with
https://github.com/iterative/dvc-studio-client/pull/28, and allows
setting the base url with path, not just limited to netloc.

This was always intended to work, urljoin with the added slash was
causing this bug.
 2023-04-18 22:22:26 +05:45
skshetry dd3deb6cbd add DVC_STUDIO_URL envvar support (#9340) 
add DVC_STUDIO_URL envvar
 2023-04-18 09:56:17 +05:45
skshetry 7841020b2b [DO NOT MERGE] release: remove feature flags for experiment sharing (#9257) 
release: remove feature flags for experiment sharing
 2023-04-17 12:53:27 +05:45
skshetry a91993616f logger: add DVC_SHOW_TRACEBACK envvar to show traceback on exception (#9310) 
This will come useful in debugging, where you don't have to add --verbose just to
see where it failed or go through verbose logs (which might be useful when working
with UI issues as you want to see what the user sees).

Ideally this should be handled in main when we exit with error instead of using
logger.{warning,error,exception} to log exception as the intention is to only log
last exception that makes it exit. But since we don't have that capability and
we don't abuse logging.{warning,error,exception} to log and suppress exceptions, this is a good
enough solution for now without complicating `dvc.cli.main()`.
 2023-04-06 09:56:19 +05:45
skshetry 3c570a2177 exp push: improve ui (#9308) 
* exp push: improve ui

* fix misc suggestions
- show url even when the refs are up-to-date.
- show url at the end
- add Git remote information in up-to-date UI message.

* fix diverged exps message
 2023-04-06 09:54:39 +05:45
skshetry cf52efb8c3 notify_refs; rename studio_url to base_url (#9303) 2023-04-04 17:43:05 +05:45
skshetry 8be71be540 utils.studio: disallow redirects, misc API refactoring (#9301) 2023-04-04 13:15:08 +05:45
skshetry 7c3c47b4a2 bump ruff to 0.0.260 and enable bandit (#9300) 
* bump ruff to 0.0.260 and enable bandit

* install toml extra for bandit

* suppress insecure use of hash md5
 2023-04-04 10:01:43 +05:45
skshetry 086d51d9de plots json: return definition errors unrelated to renderers (#9293) 2023-03-31 16:34:26 +05:45
skshetry 1b9d40960f utils.studio: reduce verbosity on exception logging (#9284) 2023-03-30 12:55:19 +05:45
skshetry 696fabc745 exp-push: show url to the Studio project (#9283) 2023-03-30 12:52:09 +05:45
skshetry 780dc72301 Revert "exp-push: show url to the Studio project" (#9282) 
Revert "exp-push: show url to the Studio project (#9278)"

This reverts commit 1ca3ef14a3.
 2023-03-30 08:28:44 +05:45
skshetry 1ca3ef14a3 exp-push: show url to the Studio project (#9278) 2023-03-30 08:27:47 +05:45
skshetry a6c89cac8d replace funcy.cat with itertools.chain.from_iterable (#9279) 
It's just an alias to the function from itertools.
 2023-03-29 20:56:26 +05:45
skshetry 593f1064cf utils.studio: refactor to return json response (#9277) 2023-03-29 20:02:40 +05:45
skshetry efc2d7ad3d remove unused type-ignore comment (#9258) 2023-03-28 14:30:33 +05:45
skshetry 7c65739e7e refactor notify_refs_to_studio to take in repo instance as dependency (#9256) 2023-03-28 13:53:31 +05:45
skshetry cc4816f652 utils.studio: require token (#9255) 
This is a simple refactoring which makes the caller to choose appropriate token instead of
the studio.notify_refs making a decision for them.
 2023-03-28 13:32:05 +05:45
skshetry faa1bdfbce use git remote to find out remote repo url when pinging Studio (#9254) 2023-03-28 12:17:48 +05:45
skshetry 006b6f3513 exp push/remove: show message on failure when notifying to Studio (#9253) 2023-03-28 09:40:41 +05:45
skshetry 1bc4c1dceb exp share refactor - ping on exp:remove, do not fail on error (#9248) 
studio experiments share refactor

1) Pings Studio on exp remove
2) Does not fail on any error during request.post
3) UI improvement on exp remove
 2023-03-27 22:51:50 +05:45
skshetry 531b2110cf Revert "exp push: add trailing slash to the endpoint url" (#9243) 
Revert "exp push: add trailing slash to the endpoint url (#9240)"

This reverts commit 5485275b92.
 2023-03-27 14:19:12 +05:45
skshetry c1917ed751 output: ignore missing size from files when calculating total size (#9242) 
output: ignore missing size from files when calculating hash info
 2023-03-27 12:56:43 +05:45
skshetry 43195a806d plots: return errors in json format (#9146) 
* plots: return errors in json format

* plots --json: move plots data into data key

* rename  revision key to rev

* move imports to top-level

* add tests
 2023-03-27 12:53:43 +05:45
skshetry 5485275b92 exp push: add trailing slash to the endpoint url (#9240) 2023-03-24 22:35:20 +05:45
skshetry 66406e6c99 scm: fetch_all_exps pass kwargs to scm.fetch_refspecs (#9214) 2023-03-20 21:26:24 +05:45
skshetry 8399bf612a sconfig: add support for studio token (#9213) 
config: add support for studio token
 2023-03-20 20:59:52 +05:45
skshetry 4ee3271b0c dvc.repo.exp: import some modules lazily (#9207) 2023-03-18 17:24:01 +05:45
skshetry dae8515933 plots: small refactorings (#9206) 2023-03-17 19:42:27 +05:45
skshetry a2066d75f9 add a note why we are flushing (#9205) 2023-03-17 17:52:04 +05:45
skshetry 283d8245e1 flush streams before fork (#9203) 2023-03-17 17:14:34 +05:45
skshetry c52b12ac2a dvcfs: support caching remote streams; use that in plots (#9183) 
* dvcfs: support caching remote streams; use that in plots

* plots: show progress bar, cap max_workers to 16

* bump dvc-data to 0.44.1
 2023-03-17 16:04:16 +05:45
skshetry 30c3e3e4d4 exp push: set default endpoint (#9198) 2023-03-16 17:50:38 +05:45
skshetry 58029e9719 exp push: push experiments revs to studio (#9160) 
* exp push: push experiments revs to studio

* log json data that is sent
 2023-03-16 11:24:50 +05:45
skshetry e6419b939d do not disable colorama in tests (#9187) 
skip disabling colorama in tests

It might affect pytest as it wraps sys.stderr/sys.stdout for capsys fixture.
This is a temporary hack, as we might get rid of colorama soon.
 2023-03-15 18:23:26 +05:45
skshetry 8312eb9b42 ui: disable colorama on write_json when highlighting is disabled (#9186) 
On a large json output (say ~300MB), this saves about ~6 seconds of time.
 2023-03-15 17:42:45 +05:45
skshetry 16ecd53b71 tests: remove gc.collect() from windows (#9159) 2023-03-13 13:37:57 +05:45
skshetry ef22b1d77e plots: disable syntax highlighting on plots json output (#9155) 
I don't think it's worth it to highlight given the size of json output.
 2023-03-11 00:50:06 +05:45
skshetry aaa825a9a8 config: ignore unspecified keys from feature section (#9153) 
A warning is shown to the user for the unsupported/ignored keys, and then ignored.
The warning is raised only once per interpreter session.
 2023-03-10 21:52:05 +05:45
skshetry 2b889baa5e plots: avoid printing anything on --json output (#9144) 2023-03-09 16:10:46 +05:45
skshetry 48a63ca588 try to improve test speed (#9136) 
* tests: fix logic to gc.collect() on Windows + 3.11

* dvc.cli: cache get_main_parser()

* tests: isolate environment on a session-level

* bring back gc.collect()
 2023-03-08 22:11:42 +05:45
skshetry 08a50ee36e tests: error on ResourceWarning (#9123) 
* tests: error on ResourceWarning

* run-cache: avoid leaking temp files on Windows

* set returncode to popen to avoid ResourceWarning on gc.collect()
 2023-03-08 15:36:10 +05:45
skshetry 51db1765ee migrate from appdirs to platformdirs (#9131) 
appdirs has been deprecated/unmaintained, platformdirs is now recommended.
Closes #9130.
 2023-03-07 23:13:28 +05:45
skshetry 93eea0cd42 deps: bump scmrepo to 0.1.15; close subrepo.dvc on tests teardown (#9122) 2023-03-06 20:21:42 +05:45
skshetry 25fd274090 increase timeout for tests to 40 minutes 2023-03-06 20:04:26 +05:45
skshetry ddfa607437 tests: don't leak files/descriptors when tests finish (#9119) 
close open files/descriptors when tests finish
 2023-03-06 18:06:37 +05:45
skshetry c5a3566e49 deps (tests): require pytest-xdist>=3.2, use -n=logical workers and workstealing scheduler (#9116) 2023-03-06 15:54:45 +05:45
skshetry e7d4f5b24e tests: filter warnings (#9115) 2023-03-06 15:41:45 +05:45
skshetry 4c5d052d8c update ruff to v0.0.254 (#9110) 2023-03-04 17:26:59 +05:45
skshetry 4db2e89938 build: use macos-11 to build package (#9091) 
macos-10.15 image has been deprecated for a while, see https://github.blog/changelog/2022-07-20-github-actions-the-macos-10-15-actions-runner-image-is-being-deprecated-and-will-be-removed-by-8-30-22/ and https://github.com/actions/runner-images/issues/5583.

setup-ruby removed the support for these images yesterday: https://github.com/ruby/setup-ruby/pull/467.

which fails our package builds. Now using macos-11 to build packages.
 2023-02-28 17:33:05 +05:45
skshetry 4a6f3ed64a dvc.testing: fix lockfile in docker_services fixture (#9088) 
dvc.testing.docker_services: rollback to previous state

where we were not tearing down the containers
 2023-02-28 16:54:33 +05:45
skshetry 2a2c02c184 remove typing.cast (#9082) 2023-02-26 13:15:54 +05:45
skshetry 8edaef0103 deps: drop importlib-resources and typing-extensions (#9055) 
deps: remove importlib-resources and typing-extensions
 2023-02-19 11:36:48 +05:45
skshetry fd8a8c2729 pyproject: format using pyproject-fmt (#9053) 
I haven't added pyproject-fmt to pre-commit hook because of two reasons:
1. There is a bug that keeps adding empty lines after `[[tool.mypy.overrides]]`.
2. The list gets exploded to multiple lines. However, this is a minor issue.
 2023-02-18 23:07:57 +05:45
skshetry 5426ca6f79 deps (tests): unpin test deps (#9052) 2023-02-18 22:41:50 +05:45
skshetry a89c25700f pyproject.toml: cleanup metadata (#9051) 2023-02-18 17:40:36 +05:45
skshetry 1de37556d5 bump ruff to v0.0.247 (#9050) 
* bump ruff to v0.0.247

* ruff: enable pep8-naming check

rename some camelCased APIs, especially in logger
rename G variables to graph
ignore some variable namings and help messages in dvc.commands
 2023-02-18 16:50:35 +05:45
skshetry d7ea7c358d Revert "scm: use dulwich backend when fetching exps during clone/pull" (#9041) 
Revert "scm: use dulwich backend when fetching exps during clone/pull (#9023)"

This reverts commit ed1ab7fa75.
 2023-02-16 18:07:45 +05:45
skshetry ddeb32ed33 ui: log footer support message to standard error (#9034) 2023-02-16 11:31:31 +05:45
skshetry be824ede0d render: avoid deepcopying (#9033) 
* render: avoid deepcopying

* remove unnecessary line
 2023-02-15 16:25:45 +05:45
skshetry ed1ab7fa75 scm: use dulwich backend when fetching exps during clone/pull (#9023) 
refspecs uses pygit2 backend by default, which does not support git credential
helpers. This PR forces to use dulwich backend during clone/pull operations.
 2023-02-14 19:18:40 +05:45
skshetry eff775c760 brancher: add trace when switching filesystem (#9022) 
brancher: add trace info when switching filesystem
 2023-02-14 15:27:18 +05:45
skshetry e51345e276 get/import: lazy import dvc.scm; reduces dvc --help by ~20% (#8996) 2023-02-09 15:04:55 +05:45
skshetry 00e61e18ae logging: do not disable other loggers (#8995) 
Closes https://github.com/iterative/dvc/issues/8631
 2023-02-09 15:04:29 +05:45
skshetry 5490fc00d1 logger: check isatty stream to print colored logs or not instead of depending on colorama (#8993 
logger: check isatty stream to print colored logs or not
 2023-02-09 13:50:38 +05:45
skshetry f1043676da [WIP] logging: change from dictConfig to equivalent code for creating logge… (#8985 
logging: change from dictConfig to equivalent code for creating loggers/handlers/formatters

dictConfig clears all handlers, and potentially disables existing loggers (we disabled this functionality
though). There's much more code than necessary in dictConfig than what we need.

I have tried to keep the setup equivalent to what we had in the config, so I have not tried to cleanup
the code.
 2023-02-09 09:34:14 +05:45
skshetry 4176007493 push: check if versioning is supported or not during checkout on cloud versioning (#8949)Fixes https://github.com/iterative/dvc/issues/8824 
* push: check if versioning is supported or not during checkout on cloud versioning

* bump dvc-data to 0.38.4
 2023-02-08 17:05:27 +05:45
skshetry 037e2e54f0 serialization: reorder files section metadata (#8950 
serialization: reorder files section
 2023-02-07 10:05:36 +05:45
skshetry ae80889477 logging: defer calling colorama.init() in jupyter, otherwise initialize immediately (#8969)Closes https://github.com/iterative/dvc/issues/8957 
logging: init colorama before logging if outside jupyter env

Otherwise, tty detection does not work as `colorama.init()` mutates
sys.stdout/sys.stderr, and after `dictConfig()`, the handlers will hold
on to the stream.
 2023-02-07 10:05:05 +05:45
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
skshetry 524162dc3f version/info: match dashes' length below DVC version (#8921) 2023-01-31 18:32:04 +05:45
skshetry aa15b21dce logging: log pkg version/type, platform info and cli arguments (#8918) 
logging: log pkg version and type, platform info and cli arguments
 2023-01-31 15:46:14 +05:45
skshetry 8208846003 bump ruff to 0.0.238 (#8915) 2023-01-31 11:24:03 +05:45
skshetry 5c6c1dc719 mypy: unignore module override (#8902) 2023-01-28 15:08:52 +05:45
skshetry 7ab4c59d70 test: run pytest-style checks (#8901) 2023-01-28 14:10:53 +05:45
skshetry a137b2a6bc update ruff and enable flake8-logging-format and flake8-type-checking (#8894) 
* update ruff and enable flake8-logging-format and flake8-type-checking check

* Update dvc/external_repo.py

* Update dvc/fs/dvc.py

* Update dvc/repo/experiments/__init__.py

* Update dvc/repo/experiments/queue/base.py

* Update pyproject.toml
 2023-01-27 19:14:19 +05:45
skshetry c1cd13c34f serialize: fix loading of cloud metadata from .dvc/dvc.lock files (#8889) 2023-01-26 16:18:03 +05:45
skshetry b3aa698cd4 dumpd: serialize files at the end in .dvc files (#8879) 2023-01-25 18:52:15 +05:45
skshetry bb9a3a0403 import/import-url: try not to delete dvcfile before import/import-url completes (#8876) 
* import/import-url: try not to delete dvcfile before import/import-url completes

* use repo.stage.create instead of internal create_stage utility
 2023-01-25 18:16:45 +05:45
skshetry 3427e115eb data status: check if change.old.hash_info is None or not (#8878) 
data:status: check if change.old.hash_info is None or not
 2023-01-25 14:24:38 +02:00
skshetry 3d5f4dc66c add support for output-specific remotes (#8862) 
Also splits metadata in `files` to cloud vs local. But this happens
during dump/load, so actual behaviour is changed. During runtime,
the metadata is merged and loaded. During dump, the metadata
are split if there is a `remote` set.
 2023-01-25 18:06:12 +05:45
skshetry f1c9e9365b remove setup.py file (#8851) 2023-01-20 20:26:13 +05:45
skshetry 6d80b72434 lint: run lint checks on all platforms (#8832) 2023-01-17 10:51:57 +05:45
skshetry c47c2123fb cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc (#8822) 
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.

* remove pylint direcitves

* Apply suggestions from code review

* Update pyproject.toml
 2023-01-17 08:35:19 +05:45
skshetry 309ae4d995 mypy: enable check-untyped-defs (#8815) 2023-01-15 21:40:40 +05:45
skshetry 7a65ac06b2 mypy: don't ignore all imports, also disallow implicit optional (#8814) 
* mypy: don't ignore all imports

* mypy: disallow implicit_optional
 2023-01-15 16:25:03 +05:45
skshetry b0052aa666 pre-commit update: flake8/ruff/pyupgrade/black/isort (#8813) 2023-01-15 14:56:54 +05:45
skshetry da27174e02 dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups, etc. (#8812) 
dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups etc
 2023-01-15 12:45:10 +05:45
skshetry dba7d3c320 mypy: use typesafe cached_property (#8799) 2023-01-13 09:08:59 +05:45
skshetry 5169205229 deps: bump minimum version req for grandalf to 0.7. 2023-01-12 18:38:45 +05:45
skshetry 3bcd683677 repo.check_graph: rename method (#8800) 2023-01-12 16:03:51 +05:45