Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,093 Commits 7 Branches 579 Tags
29b3dc1f9ec96fbd4116f8adabaeb17c9633afb2
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
skshetry 29b3dc1f9e fix 3.10 in setup-python 2022-11-01 16:35:24 +05:45
Saugat Pachhai (सौगात) 6e73bd0b5b use setuptools_scm provided version 2022-11-01 12:44:31 +02:00
skshetry fa5a9abdeb build: bump pyinstaller packages python version to 3.10 (#8511) 
bump pyinstaller packages python version to 3.10

As 3.11 is now released, let's bump pyinstaller packages to 3.10 (we always keep them 1 version behind).
 2022-11-01 16:24:00 +05:45
skshetry d2265f36ba ci: run only single macOS job (#8509) 
* ci: run only single macOS job

* ci: use alls-green@release gate check
 2022-11-01 14:30:46 +05:45
Saugat Pachhai (सौगात) d548618291 codecov: disable patch status 2022-11-01 14:02:59 +05:45
skshetry 48ff332f4a tests: remove old cleanup handler for Windows subprocess (#8508) 2022-11-01 13:57:50 +05:45
Saugat Pachhai (सौगात) 97f5ffc3c6 codecov: re-add codecov.yml 2022-11-01 13:56:54 +05:45
skshetry f6849d2f06 chore: remove bin dir and simplify tests.__main__ (#8507) 2022-11-01 13:05:53 +05:45
skshetry a172240536 Update release.yml 2022-11-01 11:43:25 +05:45
skshetry 129a4fbb92 fix release note config, it has to be inside changelog key (#8504) 2022-11-01 11:42:05 +05:45
skshetry c101514677 ci: add release note config (#8503) 2022-11-01 11:37:11 +05:45
skshetry 5e883bc00c tests: remove TestDir (#8502) 
remove TestDir
 2022-11-01 11:28:09 +05:45
skshetry 4b720e5820 remove .codecov.yml and MANIFEST.in, move .coveragerc to pyproject.toml (#8500) 
* remove .codecov.yml and MANIFEST.in, move .coveragerc to pyproject.toml

* remove scripts/pypi_upload.sh
 2022-11-01 10:11:55 +05:45
skshetry c0054fa57e remove zip-safe (#8498) 
See https://setuptools.pypa.io/en/latest/deprecated/zip_safe.html. `zip_safe` is not required these days.
 2022-11-01 07:50:18 +05:45
skshetry 6049494cef set name so that dependency graph works (#8497) 
See https://github.com/orgs/community/discussions/6456
 2022-11-01 07:09:21 +05:45
skshetry 120a5879bb ci: push package to test.pypi.org (#8495) 
* ci: push package to test.pypi.org

* include dvc.testing and license file
 2022-10-31 23:34:00 +05:45
skshetry 7648e715f8 move project config to pyproject.toml (#8493) 
* move config to pyproject.toml

* ci: use pyproject.toml for cache key

* pyproject.toml: use recursive dependencies for dev and all

* Update pyproject.toml
 2022-10-31 22:43:48 +05:45
skshetry ca5ad6457f ci: cache lint deps, decrease test timeout (#8494) 2022-10-31 22:15:54 +05:45
skshetry 6df70d388e update pre-commit; add some more hooks (#8492) 
* update pre-commit; add some more hooks

* Apply suggestions from code review
 2022-10-31 21:04:11 +05:45
skshetry 4ca2158b9d tests: retry celery queue on timeout (#8488) 
retry celery queue on timeout
 2022-10-31 19:38:05 +05:45
Saugat Pachhai (सौगात) 1d5d15644d debug: add --show-stack/--ss flag 2022-10-19 17:13:22 +03:00
skshetry bbaf57c5b2 _debug: minor refactor on debugtools (#8454) 
* _debug: minor refactor on debugtools

* delete test
 2022-10-19 18:01:20 +05:45
skshetry d9ce9b4fef testing.api_tests: remove comment from fixture (#8444) 2022-10-18 15:44:59 +05:45
skshetry 81abf2e412 dvcfs tests: copy pytest param instead of in-place update (#8388) 
The param was being modified in-place, due to which successive runs would fail.
 2022-10-01 14:07:38 +05:45
skshetry f6b1f01a5c aiohttp: remove AIOHTTP_NO_EXTENSIONS envvar (#8353) 
* aiohttp: remove AIOHTTP_NO_EXTENSIONS envvar

aiohttp now provides wheel for 3.11

* advertise 3.11 support
 2022-09-23 16:23:37 +05:45
Saugat Pachhai (सौगात) 6d17bcce93 cli: add help for foreach 2022-09-23 13:36:34 +03:00
skshetry ba1ab86078 document dvcfs (#8332) 2022-09-21 11:30:15 +05:45
skshetry aaa467a96b fs.callback.RichCallback: add transient support (#8335) 2022-09-20 17:03:17 +05:45
skshetry e63cb31a66 dvcfs: remove get-file implementation, make dvcfs non-cachable (#8331) 
dvcfs: remove get-file implementation
 2022-09-20 15:24:44 +05:45
skshetry 7e93fc8c4a lint: pin astroid version (#8323) 
See https://github.com/PyCQA/pylint/issues/7488. Pylint is failing on our codebase.
 2022-09-19 12:22:21 +05:45
skshetry 853a918d5c cli: rename cli util, hide data ls from the help (#8321) 2022-09-19 10:29:52 +05:45
skshetry b5f20f8467 use native concurrency (#8320) 2022-09-19 10:23:55 +05:45
skshetry 2f6bc70b45 remove mergify (#8319) 2022-09-19 09:50:56 +05:45
skshetry 391bb29be2 dvcfs: rename DvcFileSystem to DVCFileSystem (#8307) 2022-09-16 18:18:55 +05:45
Saugat Pachhai (सौगात) c1f659bdd7 dvcfs: prevent opening file object in write mode 2022-09-16 11:31:24 +03:00
skshetry f854fc6365 try to split testsuites into two (#8297) 
try to split tests
 2022-09-15 22:11:02 +05:45
skshetry 1b90683e02 dvcfs: add fsspec-compliance tests (#8296) 
dvcfs: add tests
 2022-09-15 18:42:50 +05:45
skshetry d762372d55 schema: add strict schema validation for top-level plots (#8289) 
schema: add strict schema validation for plots
 2022-09-15 16:30:10 +05:45
Saugat Pachhai (सौगात) fa45a9bbba dvcfs: default open to binary mode 2022-09-15 13:11:52 +03:00
skshetry fdc18d5483 expose dvcfs in dvc.api and add to fsspec's registry (#8287) 
* expose dvcfs in dvc.api and add to fsspec's registry

* readd __all__ to avoid exposing modules
 2022-09-14 20:02:56 +05:45
skshetry e78c4455de dvcfs: remove unused config code for opening a repo, change signature (#8277) 
dvcfs: remove unused code for opening a repo
 2022-09-13 09:26:33 +05:45
skshetry 8e5f4a8f35 data ls: new command to show metadata with outputs (#8252) 
data ls: add new command to show metadata for the outputs
 2022-09-09 17:36:24 +05:45
Saugat Pachhai (सौगात) cb3159e586 dvcfs: remove config 2022-09-09 13:16:27 +03:00
skshetry 7b301b09a1 data status: fix type hints (#8275) 2022-09-09 13:38:41 +05:45
skshetry 55b16166c8 add/import/import-url: update help message for labels/type/desc (#8271) 2022-09-08 23:09:10 +05:45
Saugat Pachhai (सौगात) 8432de5980 metadata: add support for dvc.yaml file 2022-09-08 15:10:33 +03:00
Saugat Pachhai (सौगात) c73764156e metadata: support custom metadata in .dvc files 
This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
 2022-09-08 14:10:35 +03:00
skshetry df60df5242 add metadata fields: label, type to data (#8232) 
gto like metadata fields: labels and type
 2022-09-08 16:33:47 +05:45
skshetry 071b2b8f81 add support for foreach target (#8210) 
add general support for foreach target
 2022-09-06 06:39:37 +05:45
skshetry 0b2b59a601 dvc.yaml: preserve outputs' desc on rewrites/updates to the stage (#8247) 
preserve outputs' desc on rewrites/updates to the stage

We were not loading `desc` of the output, due to which we were not preserving the desc field over rewrites. Note that the rewrite only happens in select commands for `dvc.yaml`: `freeze`/`unfreeze`/`run`/`stage-add`, so we did not notice this before.
 2022-09-05 18:49:39 +05:45
skshetry 49da2eef02 tests: set celery ping_task_timeout 3x the default (#8221) 
tests: set celery ping_timeout 3x the default
 2022-09-01 16:48:19 +05:45
skshetry 8968405770 data status: fix path for committed changes in Windows (#8220) 
The path we get from GitFileSystem is posixpath, even on Windows. We need to
convert that posixpath to os.path here.
 2022-09-01 10:02:08 +05:45
Saugat Pachhai (सौगात) 639eeed43b skip tests that depend on hydra.compose in 3.11 2022-08-31 17:24:09 +02:00
Saugat Pachhai (सौगात) d3569035c8 output.get_obj: catch ObjectCorruptedError 2022-08-30 21:31:33 +03:00
skshetry a8415b01d0 data status: fix quoting on command hints for untracked files (#8211) 
data status: fix quoting on command hints
 2022-08-30 22:51:39 +05:45
skshetry a0cbcaa3c8 Delete CODEOWNERS 2022-08-30 13:12:57 +05:45
skshetry bfad82a2e3 data status: remove --withdirs, show unknowns in CLI (#8189) 
* data status: remove --withdirs, show unknowns in CLI

* data status: only consider unknown from newer object during diff
 2022-08-30 12:11:02 +05:45
skshetry e5eec59cb8 test on 3.11 (#8196) 
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.

Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.

See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.

dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.

dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
 2022-08-29 19:58:17 +05:45
skshetry 37c8ef3406 Update dvc/commands/plots.py 2022-08-29 01:46:47 +03:00
skshetry 54ee8c0d77 data status: make untracked files to be relative paths (#8184) 2022-08-25 19:29:43 +05:45
skshetry 063eb6904d data status: test cleanup and improvements (#8185) 2022-08-25 18:50:46 +05:45
Saugat Pachhai (सौगात) 2374c72105 status: add unknown state 2022-08-25 13:23:18 +03:00
Saugat Pachhai (सौगात) 5cf1421c61 fix tests 2022-08-25 13:23:18 +03:00
skshetry a6a2feea62 Update setup.cfg 2022-08-25 13:23:18 +03:00
Saugat Pachhai (सौगात) 6c1d00c2d9 fix diff when no object exists by falling back to shallow_diff 2022-08-25 13:23:18 +03:00
Saugat Pachhai (सौगात) f72456f67c add shallow_diff 2022-08-25 13:23:18 +03:00
skshetry 2c556f58d3 data status: fix not-in-cache hint (#8181) 
See https://github.com/iterative/dvc.org/pull/3812\#discussion_r936625617.
 2022-08-25 15:39:59 +05:45
skshetry 6477ade557 build: skip install fpm and ruby on Windows (#8169) 2022-08-23 18:02:47 +05:45
skshetry 78dd045d29 ci: drop __token__, it's the default (#8157) 2022-08-22 08:40:31 +05:45
skshetry 2ddbbdf5a4 ci: refactor binary executable building workflow (#8156) 
This now uses the matrix to build executables from same jobs definition.
 2022-08-22 08:38:58 +05:45
skshetry 17a565023b build: bump ruby to 2.6 (#8154) 2022-08-20 23:32:23 +05:45
skshetry b5d5728bf8 deps: bump dvc-data to 0.1.22 and dvc-s3 to 0.0.2 (#8153) 
* deps: bump dvc-data to 0.1.22

* deps: bump dvc-s3 to 0.0.2
 2022-08-20 22:59:40 +05:45
skshetry 055641a51e deps: bump dvc-data to 0.1.21 (#8147) 2022-08-19 13:22:56 +05:45
skshetry f062b52e05 Add python version label to the readme (#8146) 2022-08-18 18:34:58 +05:45
skshetry 7a0b3e9377 Update tests/func/experiments/test_queue.py 2022-08-09 17:50:48 +08:00
skshetry 7c1e339650 data status: handle path joining properly for tree leaves/items (#8107) 2022-08-09 09:34:15 +05:45
skshetry 6f0736142b deps: bump dvc-data to 0.1.6 2022-07-28 21:45:23 +05:45
Saugat Pachhai (सौगात) d0eda1d538 data status: add spinners during diff 2022-07-25 19:32:19 +05:45
Saugat Pachhai (सौगात) 896acedb07 data status: optimize printing status list 
`--untracked-files` and `--unchanged` coupled with `--granular`
puts a lot of stress on `show_status`. Currently, with those options,
dvc spends ~26s on `_show_status` function.

Unhappy with the above result, I looked into improving the situation with
`console.out`. It did reduce that to 9s but it still felt like a lot.

Then I dropped to good old colorama, which dropped to 3s(term)/1.24(redirected).
I tested with buffering output which dropped this further to
~600ms(term)/~100ms(redirected).

This may seem like premature optimization, but this was preventing me
from figuring out perf issues in granular outputs, and I believe the
user should not be penalize for using feature that we already offer.
 2022-07-25 17:47:23 +05:45
Saugat Pachhai (सौगात) fd7dcae8aa cli/log: handle SIGPIPE errors 
`dvc <command> | head` for example could fail with SIGPIPE and
would print a traceback with BrokenPipeError exception. Now we
redirect do /dev/null in case of error as suggested in Python's
docs: https://docs.python.org/3/library/signal.html\#note-on-sigpipe.
 2022-07-25 16:38:09 +05:45
Saugat Pachhai (सौगात) c3ac4f84bf data status: fix untracked files output for subdir 2022-07-22 16:29:04 +03:00
Saugat Pachhai (सौगात) 61729ff66a use M fixture for concise tests 2022-07-22 16:29:04 +03:00
Saugat Pachhai (सौगात) 3dfcb82e5c add tests for data:status command 2022-07-22 16:29:04 +03:00
Saugat Pachhai (सौगात) 340a158595 introduce new dvc data status command 2022-07-22 16:29:04 +03:00
skshetry dbfe88510f deps: bump dvc-data to 0.0.24 2022-07-15 22:40:26 +05:45
Saugat Pachhai (सौगात) a4c0ae9c73 deps: bump dvc-data, install dvc-data cli deps in dev mode 2022-07-10 18:37:38 +05:45
skshetry 3c76885577 deps: remove setuptools_scm_git_archive 
setuptools_scm can now work with git archives, so we don't need this plugin.
 2022-06-30 17:00:24 +05:45
skshetry be3ec9dc59 deps: bump dvc-data to 0.0.16 2022-06-29 20:16:19 +05:45
skshetry 3dcd0107d2 deps: bump dvc-data to 0.0.13 2022-06-23 14:04:16 +05:45
Saugat Pachhai (सौगात) 4bc6b5699a quote node names 2022-06-15 12:43:45 +03:00
Saugat Pachhai (सौगात) af649af462 deps: bump dvc-data to 0.0.6; fix imports 2022-06-14 17:50:57 +05:45
Saugat Pachhai (सौगात) 8e6ea87b76 typing: executor: use queue.Queue instead of multiprocessing.Queue 2022-06-13 21:41:54 +05:45
Saugat Pachhai (सौगात) 7b16a377e5 deps: bump dvc-data to 0.0.3 2022-06-13 18:03:53 +05:45
Saugat Pachhai (सौगात) d30dbd50bc run-cache: http: stage cache uploads in http remotes 
Currently, there is no way to find run-cache in HTTP remotes.
In the future, we could use directory-listings from supported
webservers to find files.
Temporarily, we are restricting the run-cache uploads. This is a bit
strict, we could have only restricted the downloads, but it might be
more confusing to the users, so I went to categorically restrict
stage cache uploads and downloads for http filesystems.
 2022-06-12 13:49:56 +03:00
Saugat Pachhai (सौगात) e6db6ffc89 runcache: fix push from Windows to remote filesystems 
We were passing paths with '\\' in the filename instead of posix paths to
remote filesystems which was causing failures to upload.
Although in some filesystems, this was failing silently, so upload would seem
to have passed, but the download later would fail.
 2022-06-11 11:34:12 +03:00
Saugat Pachhai (सौगात) 6dd3f1e053 deps: bump dvc-data to 0.0.3 2022-06-10 14:07:33 +05:45
Saugat Pachhai (सौगात) 1e77be3ae3 testing: use shutil.copytree with dirs_exist_ok 
See https://docs.python.org/3/library/shutil.html#shutil.copytree.
There is dirs_exist_ok kwarg in Python >3.8.
 2022-05-30 20:40:31 +05:45
Saugat Pachhai (सौगात) 68b0d61c9e tests: move run cache test to test_remote 2022-05-24 10:16:37 +05:45
Saugat Pachhai (सौगात) 942a080567 dvc.stage.cache: fix typo, was using src filesystem to transfer 
This was working for LocalFileSystem as they work for any directories/path.
It's hard to test this these days, as we don't have remote tests inside
dvc, so it was not caught.

Potentially fixes #7718.
 2022-05-23 16:37:40 +03:00
Saugat Pachhai (सौगात) 60cbcec399 formatting: consider dvc-render as a known first party for ordering imports 2022-05-23 15:49:47 +05:45