* Fix partial file checkouts in the same directory
* unittest => pytest
* Try to rationalize changed() to take filter_info
* Use path.read_text() to verify the directory structure
* Try to use specialized hash, fall back to general one if doesn't exist
* Cover subdirectories in hash specialization
* coverage: add a test for empty subdirectories
* Move out the dir_info filtering block
* hash_origin => path_info
* cleanup
* Move hashing functionality after the existence checks
* Simplify it a bit
* Update dvc/cache/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions
* simplify tests
* change filenames
* Some visual changes for consistency
Co-authored-by: Ruslan Kuprieiev <[email protected]>
I had to keep providing default value before when profiling
simple functions as an API. I added a default now, and
also made `debugtools` customizable so that you could
edit the code to, for example, have `pdb` enabled by default.
* experiments: use relpaths when tracking changed files
* experiments: suppress SCM hints in tempdir executors
* experiments: move multiprocessing error logging into child
* exp run: clarify help text for --temp
* Add pyinstrument: a statistical profiler
It is easier than cprofile, in that it can show stacktrace
of where things are slow.
Also refactored/moved debugging related utils to a separate
module
* use ipdb first, then fallback to pdb
* use ipdb first, then fallback to pdb
* plots: allow dir plots
* first scenario
* separate dvclive tests
* rename index to step
* another test
* refactor
* its ok to have step in data
* sort plots
* introduce --logs
* add logs to run commands
* variable name refactoring
* fix logs option
* fixup
* cmd: run: fix unit tests
* plots: modify: fix tests
* improve dir handling
* fixup
* dvclive: logs summary
* api: introduce logs summarization method
* handle no repo case
* fix granular dir plots
* api: dvclive: simplify
* dvclive: roll back logs flag
* cmd: introduce logs
* api: adjust to logs
* test: dvclive fix
* rename logs to dvclive
* live: command: add uninitialized
* outs: dvclive type
* dvclive: support providing config via evn variables
* dvclive: treat as metrics
* dvclive integration: add support for summary on/off
* dvc integration: add temporary test for configuration export
* remove requirements
* cmd: introduce logs
* rename logs to dvclive
* cmd: introduce logs
* remove logs leftowvers
* dvclive: env: adjust to chekpoints env variables
* command: live: remove unnecessary help
* fix linter suggestions
* type hinting
* review refactor
* replace dvclive with live
* stage: loading: move live loading to separate method
* stage: serialize: allow only single live output
* live: add show/diff
* stage: serialize: roll back single live enforcing
* fixup
* rebase fixups
* output: live: force single instance
* live: command: add unit tests
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/serialize.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* review v2
* live: fix summary tracking by experiments
* naming fixups
* api: live: add tests
* tests: api: separate live tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* cli: config: show merged config with --show-origin
Builds on #5126 and #5184 by showing the full merged config when
listing or getting config options.
References #5126
References #5184
References iterative/dvc.org#2028Fixes#5119
* Add additional tests.
* Add test for merged config.
* config: rename methods
Co-authored-by: OLOLO ALALA <[email protected]>
* cli: config: add support for --show-origin
Support the --show-origin option for config, which, similar to git,
prefixes each config option with the source file it originated from.
Fixes#5119
* Reformat with black.
* Simplify logic for local config files.
* Add test for --show-config output.
* Update tests/func/test_config.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* stage: support explicit allow_missing in checkout
* exp: explicitly clear EXEC_BRANCH and checkout with allow_missing on
starting experiment run
* git: wrap stash apply errors with SCMError
* git: add `git status` to backends
* exp: reset staged git changes before workspace exp runs to avoid merge
conflicts
* test untracked (and staged) pipeline file and deps for both workspace
and tmpdir runs
* experiments: explicitly track pipeline files and uncached deps for exp
commmits
* fix exp typing errors
* dvc/tree/local: fix it for no repo/cache case
fixes#5069
* Update dvc/output/base.py
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* exp apply: show passed exp name in CLI output instead of resolved git sha
* exp res: warn that --name will be ignored when resuming without
modifications
* exp gc: don't re-use gc parameter check message
* exp: update help strings
* exp: warn if exp dir contains untracked files after tmpdir execution
* #4578 added is_user_executable to dvc file
* Addressed PR feedback and introduced test_add_executable
Also fixed all tests in test_add.py
* Added a test to checkout executable file
* Pop isexec in dvc/output/__init__.py:loadd_from
* Implement test that file is executable after checkout
* Address PR feedback
* Address PR feedback
* Changed BaseTree.isexec to use path_info as an argument
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* fix formatting
* add: preserve exec
* checkout: fix setting exec
* tests: refactor checkout exec test
* tests: fix windows cases
* tests: add: fix exec test on windows
* Update dvc/tree/local.py
* Update dvc/output/base.py
* Update dvc/output/base.py
* Update dvc/cache/base.py
* Update tests/func/test_add.py
* tests: add: skip whole exec test on windows
* save isexec to lockfiles
* fix formatting
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: move executor into a module
* experiments: separate executor into base/local
* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default
* git: restore HEAD using reset not checkout in detached_head()
* experiments: support calling BaseExecutor.reproduce directly for
workspace runs
* experiments: run single experiments in local workspace by default
* update experiments tests for workspace changes
* experiments: update checkpoints for workspace runs
* update checkpoints tests
* update test_show
* add force param for apply
* exp apply: add --force option
* exp run/res: add --temp option to force tempdir execution for single
experiments
* force apply in non-apply specific tests
* Cleanup log messages when loading stages
1. Simplified callback stage debug message
2. Reduced no lock entry message to trace
If lockfile does not have some entry, it will simply complain
that lockfile is not uptodate once.
If there is no lockfile or empty lockfile, it will complain that
lockfile not found.
3. Annoying parametrization related wdir-interpolated debug message
removed.
* Update dvc/stage/__init__.py
* black
* Skip paramdependency for vars
Previously, `configs` which was a list of PathInfo was being used
to check if the `vars` should be skipped or not.
But, the file is stored as a string that made that check redundant
which might have caused #5117 as they were being stored twice,
once as a expanded paramdependency and next as a flattened dict
from vars, which later during `flatten`, it might have panicked
from that duplication.
* Use defaultdict
* change add_vars to collect_vars
Batuhan TaskayaandRuslan Kuprieiev
Saugat Pachhai
Peter Rowlands (변기호)
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి)
Paweł RedzyńskiandRuslan Kuprieiev
Jorge Orpinel
Michael StegemanandOLOLO ALALA
Pavel Ditenbir
Dmitry Kropachev
Ioana Grigoropol
devramx
