Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-01 23:08:43 +01:00
Code Issues Packages Projects Releases Wiki Activity
9,569 Commits 7 Branches 579 Tags
optimize-ignore
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Saugat Pachhai (सौगात) 7594d2f2ed optimize dvcignore 2026-09-07 21:58:33 +05:45
Saugat Pachhai (सौगात) 046d2fd72e update copyright 2025-12-01 16:54:40 +05:45
Saugat Pachhai (सौगात) dc123102cc fix tests 2025-11-27 09:00:19 +05:45
Saugat Pachhai (सौगात) 438259cc70 fix repository owner name to treeverse 2025-11-27 08:54:23 +05:45
Saugat Pachhai (सौगात) 50487e931f Set COVERAGE_CORE during pytest run 2025-07-21 14:23:19 +05:45
Saugat Pachhai (सौगात) 3f2584d155 bench: add --benchmark-cprofile-dump 2024-08-22 14:35:23 +05:45
Saugat Pachhai (सौगात) 26ad01bec0 bench: drop add/checkout from test_sharing 2024-08-21 08:00:19 +05:45
Saugat Pachhai (सौगात) e76aaeac57 benchmarks: fix for local versions 2024-08-20 21:32:37 +05:45
Saugat Pachhai (सौगात) b9bcb45aaa ci: add python version for cache suffix 2024-08-14 15:17:31 +05:45
Saugat Pachhai (सौगात) 43e6229308 stabilize import db: remove experimental warning 2024-08-07 08:40:56 -04:00
Saugat Pachhai (सौगात) 4067a86926 fix flaky test for repro --keep-going and --ignore 2023-08-04 05:46:58 +03:00
Saugat Pachhai (सौगात) f1764bdc77 stage add: remove --external 2023-07-14 15:38:57 +03:00
Saugat Pachhai (सौगात) 21a2504bcb dvc.utils.pkg: move PKG to top-level dvc 
PKG has to be specified in dvc/_build.py now, and can be
imported using:

    from dvc import PKG

This is done to avoid importing from dvc/utils pkg which can
be expensive.
Also it's not an utility, so it is better that it remains at
the top-level namespace.
Note that we may have to fix the path for a few packages that we build,
like conda/snap/chocolatey.
 2023-06-08 00:31:00 +03:00
Saugat Pachhai (सौगात) 1be82fa56d tests: remove unnecessary variable interpolation 
dvc.run does not expand variables. It is only done when it's actually read from dvc.yaml file.
So during dvc run, the shell will try to expand the variable, which does not exist.

I don't think that the intention was to use shell variables. But I don't think we should ever
use them, as they will fail on some shells (eg: fish) and keep the scripts as simple as possible.
 2023-04-14 13:01:53 +02:00
Saugat Pachhai (सौगात) f02dd83e53 close all streams in child 2023-03-17 14:43:44 +02:00
Saugat Pachhai (सौगात) 4ec6d64629 add .git-blame-ignore-revs 2023-02-04 10:02:31 +05:45
Saugat Pachhai (सौगात) 30ec1cf5e9 index/stage: move repo collection logic to index 2023-01-11 19:38:43 +02:00
Saugat Pachhai (सौगात) 81f37be9d6 ci: timeout tests if it takes >5min 
It most likely means it's frozen and unlikely to resume again. We'd want
to see where it froze, so this will provide us with a traceback.
 2023-01-06 20:16:20 +05:45
Saugat Pachhai (सौगात) 3dfdd577e7 tests: xfail experiments-related flakey tests 2023-01-06 18:59:53 +05:45
Saugat Pachhai (सौगात) a24c389674 internal: hoist resolver/top-level metrics/plots/params data into file level 2023-01-03 17:46:06 +05:45
Saugat Pachhai (सौगात) 23e4d376c9 index: remove unused APIs 
These are being used by some users, but removing it for easier refactoring.
We are not using it ourselves.
 2022-12-20 13:17:28 +05:45
Saugat Pachhai (सौगात) 0467f6a036 add index._plot_sources 2022-12-15 13:00:27 +05:45
Saugat Pachhai (सौगात) 334f297ad3 deps: remove 3.11 checks for hydra; has 3.11 support now 2022-12-09 10:32:12 +05:45
Saugat Pachhai (सौगात) d57bb23c93 fix windows path handling 2022-12-01 15:56:26 +01:00
Saugat Pachhai (सौगात) 28402dedf1 add tests 2022-12-01 15:56:26 +01:00
Saugat Pachhai (सौगात) df60f5d9bf collect top-level params/plots/metrics together with stages 2022-12-01 15:56:26 +01:00
Saugat Pachhai (सौगात) 7ea73b2596 wip: top level params and metrics 2022-12-01 15:56:26 +01:00
Saugat Pachhai (सौगात) 8e5226ece0 cli: remove version action 2022-11-21 20:37:05 +05:45
Saugat Pachhai (सौगात) 65f61acffa tests: garbage collect sqlite connections on Windows + Python 3.11 
There seems to be a regression in Python 3.11, where the sqlite
connections are not deallocated, due to some internal changes in
Python 3.11, where they are now using LRU cache. They are not deallocated
until `gc.collect()` is not called.

See https://github.com/python/cpython/issues/97641.
This affects only Windows, because when we try to remove the tempdir for
the exp run, the sqlite connection is open which prevents us from
deleting that folder.

Although this may happen in real scenario in `exp run`, I am only fixing
the tests by mocking `dvc.close()` and extending it by calling
`gc.collect()` after it. We could also mock `State.close()` but didnot
want to mock something that is not in dvc itself.

The `diskcache` uses threadlocal for connections, so they are expected
to be garbage collected, and therefore does not provide a good way to
close the connections. The only API it offers is `self.close()` and that
only closes main thread's connection. If we had access to connection,
an easier way would have been to explicitly call `conn.close()`.
But we don''t have such option at the moment.

Related: https://github.com/iterative/dvc/pull/8404#issuecomment-1310024661
GHA Failure: https://github.com/iterative/dvc/actions/runs/3437324559/jobs/5731929385#step:5:57
 2022-11-16 08:02:43 +05:45
Saugat Pachhai (सौगात) 87d8269365 typecheck: fix type error in _git_info 2022-11-14 10:21:31 +05:45
skshetry a6bfc7db4e ci: test with 3.11 on all OSes (#8543) 
* ci: test with 3.11 on all OSes

* Update tests.yaml

* Update tests.yaml
 2022-11-09 12:05:27 +00:00
skshetry b2a275ee9e Create codeql.yml 2022-11-08 13:21:20 +05:45
skshetry 7e423b668f pkging: also post released packages to test.pypi.org 2022-11-07 18:57:53 +05:45
Saugat Pachhai (सौगात) 2c3956bdb0 exp init: fix live metrics path 2022-11-07 12:45:32 +01:00
Saugat Pachhai (सौगात) a48a19ce31 tests: ignore versions for subprojects 2022-11-07 12:45:32 +01:00
skshetry 69f78d3bb6 ci: bench: use 3.11 in benchmarks 2022-11-04 18:43:02 +05:45
skshetry 740fe504c9 Update README.rst 2022-11-04 18:27:50 +05:45
skshetry 7bb84286f5 readme: show downloads badge 2022-11-04 18:27:50 +05:45
Saugat Pachhai (सौगात) 15f35e827e fix windows 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) 4d17c171f1 tests: use mocker 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) c27005a8b7 migrate test_{repro,run}*.py to pytest 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) ecd371f8ae migrate test_move.py to pytest 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) bda1388bff migrate test_gc.py to pytest 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) c02bd7f318 migrate some files to pytest 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) fe1d805c80 migrate test_cli.py to pytest 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) f9b0de883f migrate test_checkout.py to pytest 2022-11-03 23:07:28 +05:45
Saugat Pachhai (सौगात) 1cad8ea7fd migrate test_{add,analytics,init,unprotect,local,process}.py to pytest 2022-11-03 23:07:28 +05:45
skshetry 29b3dc1f9e fix 3.10 in setup-python 2022-11-01 16:35:24 +05:45
Saugat Pachhai (सौगात) 6e73bd0b5b use setuptools_scm provided version 2022-11-01 12:44:31 +02:00
skshetry fa5a9abdeb build: bump pyinstaller packages python version to 3.10 (#8511) 
bump pyinstaller packages python version to 3.10

As 3.11 is now released, let's bump pyinstaller packages to 3.10 (we always keep them 1 version behind).
 2022-11-01 16:24:00 +05:45
skshetry d2265f36ba ci: run only single macOS job (#8509) 
* ci: run only single macOS job

* ci: use alls-green@release gate check
 2022-11-01 14:30:46 +05:45
Saugat Pachhai (सौगात) d548618291 codecov: disable patch status 2022-11-01 14:02:59 +05:45
skshetry 48ff332f4a tests: remove old cleanup handler for Windows subprocess (#8508) 2022-11-01 13:57:50 +05:45
Saugat Pachhai (सौगात) 97f5ffc3c6 codecov: re-add codecov.yml 2022-11-01 13:56:54 +05:45
skshetry f6849d2f06 chore: remove bin dir and simplify tests.__main__ (#8507) 2022-11-01 13:05:53 +05:45
skshetry a172240536 Update release.yml 2022-11-01 11:43:25 +05:45
skshetry 129a4fbb92 fix release note config, it has to be inside changelog key (#8504) 2022-11-01 11:42:05 +05:45
skshetry c101514677 ci: add release note config (#8503) 2022-11-01 11:37:11 +05:45
skshetry 5e883bc00c tests: remove TestDir (#8502) 
remove TestDir
 2022-11-01 11:28:09 +05:45
skshetry 4b720e5820 remove .codecov.yml and MANIFEST.in, move .coveragerc to pyproject.toml (#8500) 
* remove .codecov.yml and MANIFEST.in, move .coveragerc to pyproject.toml

* remove scripts/pypi_upload.sh
 2022-11-01 10:11:55 +05:45
skshetry c0054fa57e remove zip-safe (#8498) 
See https://setuptools.pypa.io/en/latest/deprecated/zip_safe.html. `zip_safe` is not required these days.
 2022-11-01 07:50:18 +05:45
skshetry 6049494cef set name so that dependency graph works (#8497) 
See https://github.com/orgs/community/discussions/6456
 2022-11-01 07:09:21 +05:45
skshetry 120a5879bb ci: push package to test.pypi.org (#8495) 
* ci: push package to test.pypi.org

* include dvc.testing and license file
 2022-10-31 23:34:00 +05:45
skshetry 7648e715f8 move project config to pyproject.toml (#8493) 
* move config to pyproject.toml

* ci: use pyproject.toml for cache key

* pyproject.toml: use recursive dependencies for dev and all

* Update pyproject.toml
 2022-10-31 22:43:48 +05:45
skshetry ca5ad6457f ci: cache lint deps, decrease test timeout (#8494) 2022-10-31 22:15:54 +05:45
skshetry 6df70d388e update pre-commit; add some more hooks (#8492) 
* update pre-commit; add some more hooks

* Apply suggestions from code review
 2022-10-31 21:04:11 +05:45
skshetry 4ca2158b9d tests: retry celery queue on timeout (#8488) 
retry celery queue on timeout
 2022-10-31 19:38:05 +05:45
Saugat Pachhai (सौगात) 1d5d15644d debug: add --show-stack/--ss flag 2022-10-19 17:13:22 +03:00
skshetry bbaf57c5b2 _debug: minor refactor on debugtools (#8454) 
* _debug: minor refactor on debugtools

* delete test
 2022-10-19 18:01:20 +05:45
skshetry d9ce9b4fef testing.api_tests: remove comment from fixture (#8444) 2022-10-18 15:44:59 +05:45
skshetry 81abf2e412 dvcfs tests: copy pytest param instead of in-place update (#8388) 
The param was being modified in-place, due to which successive runs would fail.
 2022-10-01 14:07:38 +05:45
skshetry f6b1f01a5c aiohttp: remove AIOHTTP_NO_EXTENSIONS envvar (#8353) 
* aiohttp: remove AIOHTTP_NO_EXTENSIONS envvar

aiohttp now provides wheel for 3.11

* advertise 3.11 support
 2022-09-23 16:23:37 +05:45
Saugat Pachhai (सौगात) 6d17bcce93 cli: add help for foreach 2022-09-23 13:36:34 +03:00
skshetry ba1ab86078 document dvcfs (#8332) 2022-09-21 11:30:15 +05:45
skshetry aaa467a96b fs.callback.RichCallback: add transient support (#8335) 2022-09-20 17:03:17 +05:45
skshetry e63cb31a66 dvcfs: remove get-file implementation, make dvcfs non-cachable (#8331) 
dvcfs: remove get-file implementation
 2022-09-20 15:24:44 +05:45
skshetry 7e93fc8c4a lint: pin astroid version (#8323) 
See https://github.com/PyCQA/pylint/issues/7488. Pylint is failing on our codebase.
 2022-09-19 12:22:21 +05:45
skshetry 853a918d5c cli: rename cli util, hide data ls from the help (#8321) 2022-09-19 10:29:52 +05:45
skshetry b5f20f8467 use native concurrency (#8320) 2022-09-19 10:23:55 +05:45
skshetry 2f6bc70b45 remove mergify (#8319) 2022-09-19 09:50:56 +05:45
skshetry 391bb29be2 dvcfs: rename DvcFileSystem to DVCFileSystem (#8307) 2022-09-16 18:18:55 +05:45
Saugat Pachhai (सौगात) c1f659bdd7 dvcfs: prevent opening file object in write mode 2022-09-16 11:31:24 +03:00
skshetry f854fc6365 try to split testsuites into two (#8297) 
try to split tests
 2022-09-15 22:11:02 +05:45
skshetry 1b90683e02 dvcfs: add fsspec-compliance tests (#8296) 
dvcfs: add tests
 2022-09-15 18:42:50 +05:45
skshetry d762372d55 schema: add strict schema validation for top-level plots (#8289) 
schema: add strict schema validation for plots
 2022-09-15 16:30:10 +05:45
Saugat Pachhai (सौगात) fa45a9bbba dvcfs: default open to binary mode 2022-09-15 13:11:52 +03:00
skshetry fdc18d5483 expose dvcfs in dvc.api and add to fsspec's registry (#8287) 
* expose dvcfs in dvc.api and add to fsspec's registry

* readd __all__ to avoid exposing modules
 2022-09-14 20:02:56 +05:45
skshetry e78c4455de dvcfs: remove unused config code for opening a repo, change signature (#8277) 
dvcfs: remove unused code for opening a repo
 2022-09-13 09:26:33 +05:45
skshetry 8e5f4a8f35 data ls: new command to show metadata with outputs (#8252) 
data ls: add new command to show metadata for the outputs
 2022-09-09 17:36:24 +05:45
Saugat Pachhai (सौगात) cb3159e586 dvcfs: remove config 2022-09-09 13:16:27 +03:00
skshetry 7b301b09a1 data status: fix type hints (#8275) 2022-09-09 13:38:41 +05:45
skshetry 55b16166c8 add/import/import-url: update help message for labels/type/desc (#8271) 2022-09-08 23:09:10 +05:45
Saugat Pachhai (सौगात) 8432de5980 metadata: add support for dvc.yaml file 2022-09-08 15:10:33 +03:00
Saugat Pachhai (सौगात) c73764156e metadata: support custom metadata in .dvc files 
This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
 2022-09-08 14:10:35 +03:00
skshetry df60df5242 add metadata fields: label, type to data (#8232) 
gto like metadata fields: labels and type
 2022-09-08 16:33:47 +05:45
skshetry 071b2b8f81 add support for foreach target (#8210) 
add general support for foreach target
 2022-09-06 06:39:37 +05:45
skshetry 0b2b59a601 dvc.yaml: preserve outputs' desc on rewrites/updates to the stage (#8247) 
preserve outputs' desc on rewrites/updates to the stage

We were not loading `desc` of the output, due to which we were not preserving the desc field over rewrites. Note that the rewrite only happens in select commands for `dvc.yaml`: `freeze`/`unfreeze`/`run`/`stage-add`, so we did not notice this before.
 2022-09-05 18:49:39 +05:45
skshetry 49da2eef02 tests: set celery ping_task_timeout 3x the default (#8221) 
tests: set celery ping_timeout 3x the default
 2022-09-01 16:48:19 +05:45
skshetry 8968405770 data status: fix path for committed changes in Windows (#8220) 
The path we get from GitFileSystem is posixpath, even on Windows. We need to
convert that posixpath to os.path here.
 2022-09-01 10:02:08 +05:45
Saugat Pachhai (सौगात) 639eeed43b skip tests that depend on hydra.compose in 3.11 2022-08-31 17:24:09 +02:00