Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,601 Commits 7 Branches 579 Tags
1ca3ef14a300d9d19b4fbd5fb88d20a3e1521ac4
Commit Graph
86 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
skshetry 1ca3ef14a3 exp-push: show url to the Studio project (#9278) 2023-03-30 08:27:47 +05:45
skshetry a6c89cac8d replace funcy.cat with itertools.chain.from_iterable (#9279) 
It's just an alias to the function from itertools.
 2023-03-29 20:56:26 +05:45
skshetry 593f1064cf utils.studio: refactor to return json response (#9277) 2023-03-29 20:02:40 +05:45
skshetry efc2d7ad3d remove unused type-ignore comment (#9258) 2023-03-28 14:30:33 +05:45
skshetry 7c65739e7e refactor notify_refs_to_studio to take in repo instance as dependency (#9256) 2023-03-28 13:53:31 +05:45
skshetry cc4816f652 utils.studio: require token (#9255) 
This is a simple refactoring which makes the caller to choose appropriate token instead of
the studio.notify_refs making a decision for them.
 2023-03-28 13:32:05 +05:45
skshetry faa1bdfbce use git remote to find out remote repo url when pinging Studio (#9254) 2023-03-28 12:17:48 +05:45
skshetry 006b6f3513 exp push/remove: show message on failure when notifying to Studio (#9253) 2023-03-28 09:40:41 +05:45
skshetry 1bc4c1dceb exp share refactor - ping on exp:remove, do not fail on error (#9248) 
studio experiments share refactor

1) Pings Studio on exp remove
2) Does not fail on any error during request.post
3) UI improvement on exp remove
 2023-03-27 22:51:50 +05:45
skshetry 531b2110cf Revert "exp push: add trailing slash to the endpoint url" (#9243) 
Revert "exp push: add trailing slash to the endpoint url (#9240)"

This reverts commit 5485275b92.
 2023-03-27 14:19:12 +05:45
skshetry c1917ed751 output: ignore missing size from files when calculating total size (#9242) 
output: ignore missing size from files when calculating hash info
 2023-03-27 12:56:43 +05:45
skshetry 43195a806d plots: return errors in json format (#9146) 
* plots: return errors in json format

* plots --json: move plots data into data key

* rename  revision key to rev

* move imports to top-level

* add tests
 2023-03-27 12:53:43 +05:45
skshetry 5485275b92 exp push: add trailing slash to the endpoint url (#9240) 2023-03-24 22:35:20 +05:45
skshetry 66406e6c99 scm: fetch_all_exps pass kwargs to scm.fetch_refspecs (#9214) 2023-03-20 21:26:24 +05:45
skshetry 8399bf612a sconfig: add support for studio token (#9213) 
config: add support for studio token
 2023-03-20 20:59:52 +05:45
skshetry 4ee3271b0c dvc.repo.exp: import some modules lazily (#9207) 2023-03-18 17:24:01 +05:45
skshetry dae8515933 plots: small refactorings (#9206) 2023-03-17 19:42:27 +05:45
skshetry a2066d75f9 add a note why we are flushing (#9205) 2023-03-17 17:52:04 +05:45
skshetry 283d8245e1 flush streams before fork (#9203) 2023-03-17 17:14:34 +05:45
skshetry c52b12ac2a dvcfs: support caching remote streams; use that in plots (#9183) 
* dvcfs: support caching remote streams; use that in plots

* plots: show progress bar, cap max_workers to 16

* bump dvc-data to 0.44.1
 2023-03-17 16:04:16 +05:45
skshetry 30c3e3e4d4 exp push: set default endpoint (#9198) 2023-03-16 17:50:38 +05:45
skshetry 58029e9719 exp push: push experiments revs to studio (#9160) 
* exp push: push experiments revs to studio

* log json data that is sent
 2023-03-16 11:24:50 +05:45
skshetry e6419b939d do not disable colorama in tests (#9187) 
skip disabling colorama in tests

It might affect pytest as it wraps sys.stderr/sys.stdout for capsys fixture.
This is a temporary hack, as we might get rid of colorama soon.
 2023-03-15 18:23:26 +05:45
skshetry 8312eb9b42 ui: disable colorama on write_json when highlighting is disabled (#9186) 
On a large json output (say ~300MB), this saves about ~6 seconds of time.
 2023-03-15 17:42:45 +05:45
skshetry 16ecd53b71 tests: remove gc.collect() from windows (#9159) 2023-03-13 13:37:57 +05:45
skshetry ef22b1d77e plots: disable syntax highlighting on plots json output (#9155) 
I don't think it's worth it to highlight given the size of json output.
 2023-03-11 00:50:06 +05:45
skshetry aaa825a9a8 config: ignore unspecified keys from feature section (#9153) 
A warning is shown to the user for the unsupported/ignored keys, and then ignored.
The warning is raised only once per interpreter session.
 2023-03-10 21:52:05 +05:45
skshetry 2b889baa5e plots: avoid printing anything on --json output (#9144) 2023-03-09 16:10:46 +05:45
skshetry 48a63ca588 try to improve test speed (#9136) 
* tests: fix logic to gc.collect() on Windows + 3.11

* dvc.cli: cache get_main_parser()

* tests: isolate environment on a session-level

* bring back gc.collect()
 2023-03-08 22:11:42 +05:45
skshetry 08a50ee36e tests: error on ResourceWarning (#9123) 
* tests: error on ResourceWarning

* run-cache: avoid leaking temp files on Windows

* set returncode to popen to avoid ResourceWarning on gc.collect()
 2023-03-08 15:36:10 +05:45
skshetry 51db1765ee migrate from appdirs to platformdirs (#9131) 
appdirs has been deprecated/unmaintained, platformdirs is now recommended.
Closes #9130.
 2023-03-07 23:13:28 +05:45
skshetry 93eea0cd42 deps: bump scmrepo to 0.1.15; close subrepo.dvc on tests teardown (#9122) 2023-03-06 20:21:42 +05:45
skshetry 25fd274090 increase timeout for tests to 40 minutes 2023-03-06 20:04:26 +05:45
skshetry ddfa607437 tests: don't leak files/descriptors when tests finish (#9119) 
close open files/descriptors when tests finish
 2023-03-06 18:06:37 +05:45
skshetry c5a3566e49 deps (tests): require pytest-xdist>=3.2, use -n=logical workers and workstealing scheduler (#9116) 2023-03-06 15:54:45 +05:45
skshetry e7d4f5b24e tests: filter warnings (#9115) 2023-03-06 15:41:45 +05:45
skshetry 4c5d052d8c update ruff to v0.0.254 (#9110) 2023-03-04 17:26:59 +05:45
skshetry 4db2e89938 build: use macos-11 to build package (#9091) 
macos-10.15 image has been deprecated for a while, see https://github.blog/changelog/2022-07-20-github-actions-the-macos-10-15-actions-runner-image-is-being-deprecated-and-will-be-removed-by-8-30-22/ and https://github.com/actions/runner-images/issues/5583.

setup-ruby removed the support for these images yesterday: https://github.com/ruby/setup-ruby/pull/467.

which fails our package builds. Now using macos-11 to build packages.
 2023-02-28 17:33:05 +05:45
skshetry 4a6f3ed64a dvc.testing: fix lockfile in docker_services fixture (#9088) 
dvc.testing.docker_services: rollback to previous state

where we were not tearing down the containers
 2023-02-28 16:54:33 +05:45
skshetry 2a2c02c184 remove typing.cast (#9082) 2023-02-26 13:15:54 +05:45
skshetry 8edaef0103 deps: drop importlib-resources and typing-extensions (#9055) 
deps: remove importlib-resources and typing-extensions
 2023-02-19 11:36:48 +05:45
skshetry fd8a8c2729 pyproject: format using pyproject-fmt (#9053) 
I haven't added pyproject-fmt to pre-commit hook because of two reasons:
1. There is a bug that keeps adding empty lines after `[[tool.mypy.overrides]]`.
2. The list gets exploded to multiple lines. However, this is a minor issue.
 2023-02-18 23:07:57 +05:45
skshetry 5426ca6f79 deps (tests): unpin test deps (#9052) 2023-02-18 22:41:50 +05:45
skshetry a89c25700f pyproject.toml: cleanup metadata (#9051) 2023-02-18 17:40:36 +05:45
skshetry 1de37556d5 bump ruff to v0.0.247 (#9050) 
* bump ruff to v0.0.247

* ruff: enable pep8-naming check

rename some camelCased APIs, especially in logger
rename G variables to graph
ignore some variable namings and help messages in dvc.commands
 2023-02-18 16:50:35 +05:45
skshetry d7ea7c358d Revert "scm: use dulwich backend when fetching exps during clone/pull" (#9041) 
Revert "scm: use dulwich backend when fetching exps during clone/pull (#9023)"

This reverts commit ed1ab7fa75.
 2023-02-16 18:07:45 +05:45
skshetry ddeb32ed33 ui: log footer support message to standard error (#9034) 2023-02-16 11:31:31 +05:45
skshetry be824ede0d render: avoid deepcopying (#9033) 
* render: avoid deepcopying

* remove unnecessary line
 2023-02-15 16:25:45 +05:45
skshetry ed1ab7fa75 scm: use dulwich backend when fetching exps during clone/pull (#9023) 
refspecs uses pygit2 backend by default, which does not support git credential
helpers. This PR forces to use dulwich backend during clone/pull operations.
 2023-02-14 19:18:40 +05:45
skshetry eff775c760 brancher: add trace when switching filesystem (#9022) 
brancher: add trace info when switching filesystem
 2023-02-14 15:27:18 +05:45
skshetry e51345e276 get/import: lazy import dvc.scm; reduces dvc --help by ~20% (#8996) 2023-02-09 15:04:55 +05:45
skshetry 00e61e18ae logging: do not disable other loggers (#8995) 
Closes https://github.com/iterative/dvc/issues/8631
 2023-02-09 15:04:29 +05:45
skshetry 5490fc00d1 logger: check isatty stream to print colored logs or not instead of depending on colorama (#8993 
logger: check isatty stream to print colored logs or not
 2023-02-09 13:50:38 +05:45
skshetry f1043676da [WIP] logging: change from dictConfig to equivalent code for creating logge… (#8985 
logging: change from dictConfig to equivalent code for creating loggers/handlers/formatters

dictConfig clears all handlers, and potentially disables existing loggers (we disabled this functionality
though). There's much more code than necessary in dictConfig than what we need.

I have tried to keep the setup equivalent to what we had in the config, so I have not tried to cleanup
the code.
 2023-02-09 09:34:14 +05:45
skshetry 4176007493 push: check if versioning is supported or not during checkout on cloud versioning (#8949)Fixes https://github.com/iterative/dvc/issues/8824 
* push: check if versioning is supported or not during checkout on cloud versioning

* bump dvc-data to 0.38.4
 2023-02-08 17:05:27 +05:45
skshetry 037e2e54f0 serialization: reorder files section metadata (#8950 
serialization: reorder files section
 2023-02-07 10:05:36 +05:45
skshetry ae80889477 logging: defer calling colorama.init() in jupyter, otherwise initialize immediately (#8969)Closes https://github.com/iterative/dvc/issues/8957 
logging: init colorama before logging if outside jupyter env

Otherwise, tty detection does not work as `colorama.init()` mutates
sys.stdout/sys.stderr, and after `dictConfig()`, the handlers will hold
on to the stream.
 2023-02-07 10:05:05 +05:45
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
skshetry 524162dc3f version/info: match dashes' length below DVC version (#8921) 2023-01-31 18:32:04 +05:45
skshetry aa15b21dce logging: log pkg version/type, platform info and cli arguments (#8918) 
logging: log pkg version and type, platform info and cli arguments
 2023-01-31 15:46:14 +05:45
skshetry 8208846003 bump ruff to 0.0.238 (#8915) 2023-01-31 11:24:03 +05:45
skshetry 5c6c1dc719 mypy: unignore module override (#8902) 2023-01-28 15:08:52 +05:45
skshetry 7ab4c59d70 test: run pytest-style checks (#8901) 2023-01-28 14:10:53 +05:45
skshetry a137b2a6bc update ruff and enable flake8-logging-format and flake8-type-checking (#8894) 
* update ruff and enable flake8-logging-format and flake8-type-checking check

* Update dvc/external_repo.py

* Update dvc/fs/dvc.py

* Update dvc/repo/experiments/__init__.py

* Update dvc/repo/experiments/queue/base.py

* Update pyproject.toml
 2023-01-27 19:14:19 +05:45
skshetry c1cd13c34f serialize: fix loading of cloud metadata from .dvc/dvc.lock files (#8889) 2023-01-26 16:18:03 +05:45
skshetry b3aa698cd4 dumpd: serialize files at the end in .dvc files (#8879) 2023-01-25 18:52:15 +05:45
skshetry bb9a3a0403 import/import-url: try not to delete dvcfile before import/import-url completes (#8876) 
* import/import-url: try not to delete dvcfile before import/import-url completes

* use repo.stage.create instead of internal create_stage utility
 2023-01-25 18:16:45 +05:45
skshetry 3427e115eb data status: check if change.old.hash_info is None or not (#8878) 
data:status: check if change.old.hash_info is None or not
 2023-01-25 14:24:38 +02:00
skshetry 3d5f4dc66c add support for output-specific remotes (#8862) 
Also splits metadata in `files` to cloud vs local. But this happens
during dump/load, so actual behaviour is changed. During runtime,
the metadata is merged and loaded. During dump, the metadata
are split if there is a `remote` set.
 2023-01-25 18:06:12 +05:45
skshetry f1c9e9365b remove setup.py file (#8851) 2023-01-20 20:26:13 +05:45
skshetry 6d80b72434 lint: run lint checks on all platforms (#8832) 2023-01-17 10:51:57 +05:45
skshetry c47c2123fb cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc (#8822) 
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.

* remove pylint direcitves

* Apply suggestions from code review

* Update pyproject.toml
 2023-01-17 08:35:19 +05:45
skshetry 309ae4d995 mypy: enable check-untyped-defs (#8815) 2023-01-15 21:40:40 +05:45
skshetry 7a65ac06b2 mypy: don't ignore all imports, also disallow implicit optional (#8814) 
* mypy: don't ignore all imports

* mypy: disallow implicit_optional
 2023-01-15 16:25:03 +05:45
skshetry b0052aa666 pre-commit update: flake8/ruff/pyupgrade/black/isort (#8813) 2023-01-15 14:56:54 +05:45
skshetry da27174e02 dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups, etc. (#8812) 
dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups etc
 2023-01-15 12:45:10 +05:45
skshetry dba7d3c320 mypy: use typesafe cached_property (#8799) 2023-01-13 09:08:59 +05:45
skshetry 5169205229 deps: bump minimum version req for grandalf to 0.7. 2023-01-12 18:38:45 +05:45
skshetry 3bcd683677 repo.check_graph: rename method (#8800) 2023-01-12 16:03:51 +05:45
skshetry a814f04ee0 tests: xfail test_apply_failed on JSONDecodeError/PermissionError (#8796) 2023-01-11 18:04:23 +05:45
skshetry 90a27179eb ruff: ignore complex functions and remove dvc.utils.fs.makedirs (#8795) 
We may want to prevent any new functions from getting complex.
 2023-01-11 17:58:51 +05:45
skshetry 175c578579 ruff: ban relative parent/grandparent imports (#8794) 2023-01-11 16:48:46 +05:45
skshetry 5441964fdb add ruff (#8793) 
* add ruff linter

* enable few more checks
 2023-01-11 15:20:38 +05:45
skshetry d87cdba6d8 gha: remove env vars SHELL 2023-01-10 10:19:29 +05:45
skshetry ea8a4a3db4 GHA: remove unused env from tests workflow 2022-12-27 10:26:29 +05:45
skshetry bb80eab77a deps: add support for hdfs in Python 3.11 2022-11-25 11:16:42 +05:45