Pre-requisite for removing fs.path_info and switching to regular strings
as paths in filesystems.
With new filesystems we no longer need the zoo of different dependency
and output types, as most of them are now just handled by the underlying
filesystem. Though deps/outs are still due for a refactor in other
aspects.
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* addressing: allow stage name addressing without colon
* repo: reorganize imports
* refactor/reorganize collect_granular
* add tests for stage addressing
* fix ds and cc issues
* fix unpack error in test
* tests: stage string representations
* exception: add error message
Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.
* rename NoOutputOrStage NoOutputOrStageError
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
* test: remove never used copy of load_stage_file()
* dvc: save comments and custom formatting in stage files
* Add a couple of TODOs
* test: test repro doesn't clear comments and formatting in .dvc
As part of this I started writing pytest-like tests and added commonly
used fixtures.
* test: simplify and prettify dvc formatting retained test
Also test that line comments after md5 fields are retained.
* test: fix tests
* dvc: fix stage and metrics opens not closed
* dvc: always use utf-8 in stage files
It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.
* test: unit test apply_diff()
* test: remove TODOs and ASK
* dvc: polish dvc-formatting PR
* dvc: move stage load/dump utils to a new stage utils file
* dvc: clean up apply_diff()
* test: test stage md5 ignores comments
* test: remove pytests starting marker
Ruslan Kuprieiev
mbiesekandRuslan Kuprieiev
Saugat Pachhai
Paweł Redzyński
Casper da Costa-Luis
Mr. Outis
Alexander Schepanovski