Commit Graph
71 Commits
Author SHA1 Message Date
pre-commit-ci[bot]andSaugat Pachhai ca3704997b [pre-commit.ci] pre-commit autoupdate (#9632)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/asottile/pyupgrade: v3.3.1 → v3.9.0](https://github.com/asottile/pyupgrade/compare/v3.3.1...v3.9.0)
- https://github.com/charliermarsh/ruff-pre-commit → https://github.com/astral-sh/ruff-pre-commit
- [github.com/astral-sh/ruff-pre-commit: v0.0.261 → v0.0.277](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.261...v0.0.277)
- [github.com/codespell-project/codespell: v2.2.4 → v2.2.5](https://github.com/codespell-project/codespell/compare/v2.2.4...v2.2.5)

* fix ruff complaints

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2023-07-14 20:11:15 +05:45
skshetry 26152292aa remove support for reading and generating Dvcfile (#9546) 2023-06-06 15:29:10 +05:45
skshetry 3959c51373 lockfile: drop 1.0 lockfile support (#9476) 2023-05-18 17:36:05 +05:45
Alexander Guschin 32bebc0246 Add top-level artifacts: section (#9220)
* add artifacts section

* adding and removing artifacts

* delete code for add/remove; add tests

* fix pre-commit

* remove extra piece of writing to dvc.yaml

* remove extra piece

* simplify kwargs

* allow same names for artifacts in different dvc.yaml files

* set ID as path; don't return dvc.yaml(s) with no artifacts

* fix PR feedback

* issue warning at incorrect artifact name

* issue warning at incorrect artifact name

* reference right schema

* add r to regexp

* add tests for regexp

* set artifacts for SingleStageFile

* fix windows test

* make path required; simplify regexp and remove slash from it

* allow to use gto's artifacts.yaml

* remove extra piece of code for migration from gto
2023-04-05 13:31:35 +00:00
pre-commit-ci[bot]andDavid de la Iglesia Castro 48b56c83dc [pre-commit.ci] pre-commit autoupdate (#9168)
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/charliermarsh/ruff-pre-commit: v0.0.254 → v0.0.257](https://github.com/charliermarsh/ruff-pre-commit/compare/v0.0.254...v0.0.257)
- [github.com/codespell-project/codespell: v2.2.2 → v2.2.4](https://github.com/codespell-project/codespell/compare/v2.2.2...v2.2.4)

* pre-commit: fix ruff errors

* pre-commit: fix flake8

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David de la Iglesia Castro <[email protected]>
2023-03-23 13:05:31 +00:00
skshetry 4c5d052d8c update ruff to v0.0.254 (#9110) 2023-03-04 17:26:59 +05:45
skshetry 1de37556d5 bump ruff to v0.0.247 (#9050)
* bump ruff to v0.0.247

* ruff: enable pep8-naming check

rename some camelCased APIs, especially in logger
rename G variables to graph
ignore some variable namings and help messages in dvc.commands
2023-02-18 16:50:35 +05:45
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
skshetry a137b2a6bc update ruff and enable flake8-logging-format and flake8-type-checking (#8894)
* update ruff and enable flake8-logging-format and flake8-type-checking check

* Update dvc/external_repo.py

* Update dvc/fs/dvc.py

* Update dvc/repo/experiments/__init__.py

* Update dvc/repo/experiments/queue/base.py

* Update pyproject.toml
2023-01-27 19:14:19 +05:45
daniele e1254c9ece Add misc type hints (#8430) 2023-01-15 17:58:34 +05:45
skshetry 7a65ac06b2 mypy: don't ignore all imports, also disallow implicit optional (#8814)
* mypy: don't ignore all imports

* mypy: disallow implicit_optional
2023-01-15 16:25:03 +05:45
skshetry da27174e02 dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups, etc. (#8812)
dvcfile: rename PipelineFile to ProjectFile, get rid of Dvcfile class, cleanups etc
2023-01-15 12:45:10 +05:45
skshetry dba7d3c320 mypy: use typesafe cached_property (#8799) 2023-01-13 09:08:59 +05:45
Saugat Pachhai (सौगात) 30ec1cf5e9 index/stage: move repo collection logic to index 2023-01-11 19:38:43 +02:00
Saugat Pachhai (सौगात) a24c389674 internal: hoist resolver/top-level metrics/plots/params data into file level 2023-01-03 17:46:06 +05:45
Ruslan Kuprieiev 8c484645fa cloud-versioning: better handling for directories 2022-09-26 02:54:17 +03:00
dberenbaum c039e9a6e7 merge-driver: support removes and changes 2022-09-23 23:22:08 +03:00
Peter Rowlands c29e721823 output: support loading/dumping dir trees from metadata 2022-09-15 13:13:31 +03:00
skshetry 071b2b8f81 add support for foreach target (#8210)
add general support for foreach target
2022-09-06 06:39:37 +05:45
Alex Hall c59f935cc5 Switch uses of %r to %s with quotes to avoid escaped backslashes in logs, especially in windows paths 2022-07-05 02:09:25 +03:00
Alex Hall 03b789e206 logger: use lazy formatting
Fixes #1843
2022-07-05 02:09:25 +03:00
Saugat Pachhai (सौगात) 570e3a5359 dvc.fs: move fs to dvc.objects.fs
Some of the APIs/FileSystems are reimported inside dvc.fs module.
2022-05-16 13:15:19 +03:00
Saugat Pachhai (सौगात) a282e0e48c scm: restructure scm
Makes it easier to decouple, we'd only need to rename
dvc/scm/__init__.py to dvc/scm.py.
2021-11-24 11:46:40 +05:45
Saugat Pachhai (सौगात) 93e6526907 scm: split tracking of to-be git-ignored/git-staged into SCMContext 2021-11-18 15:34:45 +02:00
Saugat Pachhai e7c20aa1d8 better exception messages for schema errors (#6679)
* Generate prettier schema error messages

* add test for #6285
2021-09-28 11:12:16 +05:45
Saugat Pachhai eefede92b2 merge yaml parsing and schema validation into one, aka strictyaml.load (#6661) 2021-09-23 00:03:25 +03:00
Saugat Pachhai 6a417885b8 exceptional exceptions (#6654)
* exceptional exceptions

* backward compatibility with previous exceptions

* make linters happy :)
2021-09-21 11:27:57 +05:45
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
b349b39d62 Decoupling dvcignore and fs (#5812)
* Decoupling dvcignore and fs

[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check

* Solve all of the arugments error.

* add dvcignore to dvc list

* Solve some test_ignore cases

* add dvcignore to checkout.

1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs

* Solve two tests

* Get rid of ignore_filter

* Add ignore examine to download

* Solve a test issue

* Solve dvcignore and subrepo problems

* Solve sub repo error

* Revert some changes

* pass all tests

* Remove debug codes

* Some final exams

* dvcignore trie tree only update when needed

* reuse of the dir result in path walk.

* Some change after code review.

* Recover something from false `--force` push

* add abspath trans to dvcginore

* pass pylint

* pylint related

* Update dvc/fs/repo.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Changes related to reviews.

* dvcignore call api change

from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)

* Get rid of PathInfo in test_ignore.py

* fs.walk follow the arguments of os.walk

* Dvcignore as parameters

* pass dvcignore as arguments to checkouts

* Decoupling state and repo

* Update dvc/dvcfile.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Update dvc/objects/stage.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Solve some small problems

* Add tests to verify that dependency now not ignore

fix#5256

* Update dvc/ignore.py

* Update dvc/fs/git.py

* Update dvc/repo/__init__.py

* Some small improvements

* pass black

* Refactor is_ignored

* Update dvc/output/base.py

* repofs: use dvcignore in exists/isdir/isfile

Per https://github.com/iterative/dvc/pull/5812/files#r630604524

Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-05-12 05:29:04 +03:00
Peter Rowlands (변기호) 53414a0618 dvcfile/stage: check if dvcfile would be gitignored on stage creation (#5907)
* dvcfile: add check for gitignored when validating filenames

* update tests

* update exception using in tests

* review fixes

- clarify method name
- use FileIsGitIgnored
2021-05-04 15:14:55 +09:00
Saugat Pachhai 45477a2a81 dvc.lock: raise error if the file is gitignored (#5484) 2021-02-18 03:08:00 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Pavel Ditenbir e7686f90a6 term: replace 'DVC-file' and 'stage file' with correct terms (#3960) (#5280) 2021-02-05 16:01:42 +09:00
Batuhan Taskaya c84833bab1 params: support unicode characters (#5363)
* params: support unicode characters

* open remove() calls with utf-8
2021-02-01 15:56:53 +02:00
Saugat Pachhai b1d6918c5a stage collection: do not collect .gitignored dvcfiles (#5265)
* Do not collect .gitignored dvcfiles

* fix exc

* consistent error message

* fix imports

* add tests
2021-01-26 14:51:02 +02:00
Saugat Pachhai 257f58450a refactor run; move to stage.utils (#5262)
* refactor run; move to stage.utils

* Update dvc/repo/run.py

* fix pylint

* skip on run-cache
2021-01-13 21:19:36 +02:00
Saugat Pachhai af2a12a188 Use 'schema' top-level element in the lockfile for version (#5222) 2021-01-06 20:08:14 +02:00
Saugat Pachhai 4c53e636bc Move misc. imports from command, make them lazy (#5214) 2021-01-06 17:44:02 +05:45
Saugat Pachhai 0cc5227f47 Add versioning scheme in the dvc.lock (#5128)
* Add versioning scheme in the dvc.lock

* remove whitespace

* use actual size in tests assumptions
2021-01-06 15:44:19 +05:45
Ruslan Kuprieiev 7c45711e34 dvc: preserve comments/meta on add/imp/run (#5001) 2020-12-01 10:39:51 +02:00
Saugat Pachhai bc4c9fe9ea parametrization: prevent dumping parametrized stages (#4950)
* parametrization: prevent dumping parametrized stages

This is a bit hackish, but does the job done.
It feels like we need some unified way to be able to load
and dump, but not sure how. Till then, this dataclass
will be helpful in communicating between those.

* s/Meta/RawData

* Update dvc/stage/loader.py
2020-11-24 17:40:47 +05:45
Saugat Pachhai d5312e56a1 parametrization: Move resolver to StageLoader from Dvcfile (#4916) 2020-11-20 13:38:57 +05:45
Peter Rowlands (변기호) 3daeeaffdb experiments: fix checkpoint/persist related checkout issues (#4885)
* dvcfile: add is_lock_file to identify pipeline lock files

* experiments: clean dirty lock files before stashing experiments
2020-11-12 19:28:28 +09:00
Saugat Pachhai b0ae5e2251 Support parametrization from params or vars section in dvc.yaml (#4751)
* Implement importing from params

* Change varname

* Add tests

* fix pylint issues
2020-10-20 04:16:13 -07:00
Saugat Pachhai 6a79111047 Parametrization support on dvc.yaml (#4679)
* Custom parametrization support

* log generated data

* Add tests
2020-10-14 05:32:05 -07:00
Saugat Pachhai 2931d30ca5 dvc: change dvcfile.dump to do synchronized dump by default (#4544)
The API of `Dvcfile.dump` has been changed to
`dump(stage, update_pipeline=True, update_lock=True)`,
which will make it to always dump by default to both files,
and provide flags to suppress one or the other.

Fixes #4235
2020-09-10 13:12:00 +05:45
Saugat Pachhai 1d196b4df6 Modify yaml/toml using contextmanager, removing boilerplate (#4426) 2020-08-21 04:47:58 +03:00
Saugat Pachhai de0fe94cd0 Make serializing utils use tree, misc changes (#4425)
* Also fixes plots not getting dumped on flow style
* Make serialization utils aware of trees
* Directly load yamls/tomls using utils, rather than using parser
2020-08-20 20:38:17 +03:00
Ruslan Kuprieiev ac24b5c6f0 dvc: introduce merge-driver (#4298)
Related to #4162
2020-08-19 17:55:12 +03:00