* Renamed --outs-only to --dvc-only
* Changed outs_only to dvc_only
* replace output_only with dvc_only
* Restyled by black
Co-authored-by: Restyled.io <[email protected]>
* stage: extract unnecessary methods out of Stage class
* stage: change naming of functions
* stage: extract dump equal to utils
* stage: extract computation of md5 to utils
* stage: extract command run related logic in run.py
* stage: reorganize run()
* stage: boolean cosmetic changes
* stage: reorder save() and extract some into func
* stage: refactor status() into smaller functions
* extract resolve_wdir into utils
* merge _changed() and changed()
* cosmetic changes
* extract dump to utils
* extract import updates/sync to imports.py
* remove meta from PipelineStage
* fix imports, params load
* make run_stage a function, fix mocks in tests
* use original cmd
* rollback imports on decorators
* fix ds and cc
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
Note: This does not change default for `pipeline` as
it cannot show multiple DAGs at the same time. If we
are able to show multiple DAGs, we can just default to
`dvc.yaml`.
To be consistent with how git does it (e.g. objects) as well as with
future cache structure where we plan on adding `dirs`/`files`/`objects`
subdirs in the future.
This patch introduces `.dvc/cache/stages` that is used to store previous
runs and their results, which could then be reused later when we stumble
upon the same command with the same deps and outs.
Format of build cache entries is single-line json, which is readable by
humans and might also be used for lock files discussed in #1871.
Related to #1871
Local part of #1234
* tests: ensure repro --downstream preserves dependency order
* repo: use reverse post-order DFS in repro --downstream
- Fixes#3602
* make test_repro and test_repro_multistage consistent
* rebase master, update multistage test
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Now, pipeline stage files can house multiple stages, and separate lockfiles
are created which has the checksums, whereas Dvcfile will be clean and human
readable and editable. The *.dvc files will be generated for output files.
It is available via a hidden flag: -n.
Fixes#1871
PR: #3584
* refactor stage into dvcfile
* Split create_stage out of dvcfile to Stage::create() and overwrite
* Remove logging from Repo::run
* tests: stop using --remove-outs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* gc: make it safer by only activating by use of certain flags
Previous plain `gc` is now not supported. The same behavior can be trigger by
`gc -w` or `gc --workspace`. When `--all-{tags, commits, branches}` are
specified, it will work as if `-w` is also specified (same as before).
* tests: adjust gc to use workspace on test_repro
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
Make `Repo.checkout` accept a list of targets, instead of having to feed
it targets one-by-one. This allows us to properly handle checkout
errors.
Fixes#2580
Imagine the scenario of shared cache. To set it up, user would do:
```
dvc config cache.dir /path
dvc config cache.shared group
```
Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.
The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.
Fixes#2563
Shankho Boron GhoshandRestyled.io
Saugat Pachhai
Ruslan Kuprieiev
Peter Rowlands (변기호)
nik123
Casper da Costa-Luis
Alexander Schepanovski
Paweł Redzyński
Mr. Outis