* dvcignore: Fix incorrect ignored output computation
* dvcignore: Fixed unignored patterns
Unignored patterns were shown in the dvc status output and caused Error.
Also made a bit of a refactoring in `matches` and `ignore`.
* dvcignore: Fix functional tests
Not sure if I follow, but tests assume that unignored files should trigger `check-ignore`. What was the intention?
* dvcignore: Fix stage.wdir being MagicMock
* Use `is_ignored` instead of `check_ignore`
1. use is_ignored to judge DVC ignore status.
2. use check_ignore to show ignore patterns.
* Solve tests fails
* Add a new tests
* Return to the old check_ignore logic.
* Pass the pylint
* Simplify ignore tests
Co-authored-by: karajan1001 <[email protected]>
* Implement export-to-remote functionality
* Redistrubition of functionality from tree to remote
* Pass chunk size into the open_url
* Move functionality away from remote to cache
* Basic progress bars for both transfer types
* Proper job calculation
* Redesign .open() usage in order to work with different trees
* Align the signatures of other open() functions to accept **kwargs
* Implement chunked uploading to the ssh
* Use local cache to compute directory hash (md5)
* S3: multipart upload test
* Wrap local cache's hash_to_path with PathInfo
* Use .isdir() instead of a try/except statement for calculating the from_infos
* New CLI layout (add/import-url)
* Adjust CLI tests to accept new arguments
* tests: import_url/add func/unit tests
* Introduce cache.transfer()
* Invalid CLI combination tests
* Store dir_info key in the transfer_directory
* LocalTree.upload_fobj
* Tests for utils
* Lazy imports on some places
* --to-remote migration (and repo._transfer)
* Use proper temporary location, more efficient pbar calc
* Directory transfer logic rewrite
* Use shutil.copyfileobj()
* Rebase...
* Better tests
* Inline transfer operation to .add() / .transfer()
* Open in the 'rb' mode for the tests
* More tests, better invalid option handling at repo level
* .read() certain amount of bytes to validate the content
* Better invalid argument handling
* Better progress bars by using .getsize() API
* For fallback upload, use the from_info.name instead of temporary naem
* introduce --jobs for add/import-url
* pygit2: implement get/set/remove refs
* pygit: implement iter_refs
* pygit: implement resolve_rev
* pygit2: implement working tree checkout
* pygit2: implement reset, explicitly read/write index when used
* pygit2: checkout does not need to reset index
* pygit2: handle "-" branch shorthand in checkout
* tests: install post-checkout hook should be run via gitpython
* exp apply: use stricter exception handling for merge conflict
* pygit: fix index read/write usage
* gitpython: handle untracked file conflicts in stash apply
* fix flaky/race index issue
* tests: upgrade azurite and enable loose mode
Azure tests started failing recently. They still work with real azure, so this is on azurite side.
* tests: use cloud instead of workspace
also gdrive doesn't support it yet :(
* scm: implement gitignore; use `pygit2` backend
* Add is_ignored implementation on `pygit2` and `gitpython` backends.
* Also implement `._reset()` on `repo.scm` which is most of the changes.
* fix tests
* use _backend_func
* minor fixups
* S3Tree._list_paths now recognizes the max_items limit
* Use modern approach + add a test
* refactor tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* lint
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Fix partial file checkouts in the same directory
* unittest => pytest
* Try to rationalize changed() to take filter_info
* Use path.read_text() to verify the directory structure
* Try to use specialized hash, fall back to general one if doesn't exist
* Cover subdirectories in hash specialization
* coverage: add a test for empty subdirectories
* Move out the dir_info filtering block
* hash_origin => path_info
* cleanup
* Move hashing functionality after the existence checks
* Simplify it a bit
* Update dvc/cache/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions
* simplify tests
* change filenames
* Some visual changes for consistency
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* plots: allow dir plots
* first scenario
* separate dvclive tests
* rename index to step
* another test
* refactor
* its ok to have step in data
* sort plots
* introduce --logs
* add logs to run commands
* variable name refactoring
* fix logs option
* fixup
* cmd: run: fix unit tests
* plots: modify: fix tests
* improve dir handling
* fixup
* dvclive: logs summary
* api: introduce logs summarization method
* handle no repo case
* fix granular dir plots
* api: dvclive: simplify
* dvclive: roll back logs flag
* cmd: introduce logs
* api: adjust to logs
* test: dvclive fix
* rename logs to dvclive
* live: command: add uninitialized
* outs: dvclive type
* dvclive: support providing config via evn variables
* dvclive: treat as metrics
* dvclive integration: add support for summary on/off
* dvc integration: add temporary test for configuration export
* remove requirements
* cmd: introduce logs
* rename logs to dvclive
* cmd: introduce logs
* remove logs leftowvers
* dvclive: env: adjust to chekpoints env variables
* command: live: remove unnecessary help
* fix linter suggestions
* type hinting
* review refactor
* replace dvclive with live
* stage: loading: move live loading to separate method
* stage: serialize: allow only single live output
* live: add show/diff
* stage: serialize: roll back single live enforcing
* fixup
* rebase fixups
* output: live: force single instance
* live: command: add unit tests
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/serialize.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* review v2
* live: fix summary tracking by experiments
* naming fixups
* api: live: add tests
* tests: api: separate live tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* cli: config: show merged config with --show-origin
Builds on #5126 and #5184 by showing the full merged config when
listing or getting config options.
References #5126
References #5184
References iterative/dvc.org#2028Fixes#5119
* Add additional tests.
* Add test for merged config.
* config: rename methods
Co-authored-by: OLOLO ALALA <[email protected]>
* cli: config: add support for --show-origin
Support the --show-origin option for config, which, similar to git,
prefixes each config option with the source file it originated from.
Fixes#5119
* Reformat with black.
* Simplify logic for local config files.
* Add test for --show-config output.
* Update tests/func/test_config.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* stage: support explicit allow_missing in checkout
* exp: explicitly clear EXEC_BRANCH and checkout with allow_missing on
starting experiment run
* git: wrap stash apply errors with SCMError
* git: add `git status` to backends
* exp: reset staged git changes before workspace exp runs to avoid merge
conflicts
* test untracked (and staged) pipeline file and deps for both workspace
and tmpdir runs
* experiments: explicitly track pipeline files and uncached deps for exp
commmits
* fix exp typing errors
* dvc/tree/local: fix it for no repo/cache case
fixes#5069
* Update dvc/output/base.py
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* #4578 added is_user_executable to dvc file
* Addressed PR feedback and introduced test_add_executable
Also fixed all tests in test_add.py
* Added a test to checkout executable file
* Pop isexec in dvc/output/__init__.py:loadd_from
* Implement test that file is executable after checkout
* Address PR feedback
* Address PR feedback
* Changed BaseTree.isexec to use path_info as an argument
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* fix formatting
* add: preserve exec
* checkout: fix setting exec
* tests: refactor checkout exec test
* tests: fix windows cases
* tests: add: fix exec test on windows
* Update dvc/tree/local.py
* Update dvc/output/base.py
* Update dvc/output/base.py
* Update dvc/cache/base.py
* Update tests/func/test_add.py
* tests: add: skip whole exec test on windows
* save isexec to lockfiles
* fix formatting
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: move executor into a module
* experiments: separate executor into base/local
* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default
* git: restore HEAD using reset not checkout in detached_head()
* experiments: support calling BaseExecutor.reproduce directly for
workspace runs
* experiments: run single experiments in local workspace by default
* update experiments tests for workspace changes
* experiments: update checkpoints for workspace runs
* update checkpoints tests
* update test_show
* add force param for apply
* exp apply: add --force option
* exp run/res: add --temp option to force tempdir execution for single
experiments
* force apply in non-apply specific tests
Currently, we iterate over targets manually in CLI, which results in a
complicated code and a DAG collection/check overhead. With this PR we
collect the DAG once, figure out all the needed targets within it and
just reproduce them in one pass.
minor refactors, code comments
Rename interim to definitions
provide location of entry during init for definition
small fixups to set_temporarily
Refactor loop to use dict comprehension
Provide same interface for definitions
Make it work for composite lists
Check for sequence rather than a list
refactor tests
Fixes#4955Fixes#4957Fixes#4963
Currently we don't handle bad name well:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: unexpected error - not enough values to unpack (expected 2, got 1)
```
With this PR, we check name to conform to either `remote.name.option` or
`section.option`, so we'll give a more friendly:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: argument name: name argument should look like remote.name.option or section.option
usage: dvc config [-h] [--global | --system | --local] [-q | -v] [-u] [-l] [name] [value]
Get or set config options.
Documentation: <https://man.dvc.org/config>
positional arguments:
name Option name (remote.name.option or section.option).
value Option value.
optional arguments:
-h, --help show this help message and exit
--global Use global config.
--system Use system config.
--local Use local config.
-q, --quiet Be quiet.
-v, --verbose Be verbose.
-u, --unset Unset option.
-l, --list List all defined config values.
```
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.
So, that's why, the `collect` also does a graph check for this
reason.
Fixes#5027Fixes#4010
* experiments: add utils.exp_commits for iterating over experiment commits
* repo: add support for all_experiments in brancher and used_cache
* git: make all_commits return commits in standard refs instead of --all
* gc: add --all-experiments to preserve cache for experiments
* update gc tests
* repo: support pushing and fetching specific revs internally
* exp push: push cache for experiment by default
* exp pull: pull cache for experiment by default
* fix empty git repo bug
Timofey Yefimovandkarajan1001
Batuhan Taskaya
Ruslan Kuprieiev
Peter Rowlands (변기호)
Saugat Pachhai
Dmitry Kropachev
Paweł Redzyński
mbiesekandRuslan Kuprieiev
Michael StegemanandOLOLO ALALA
Ioana Grigoropol
devramx
Sandeep MistryandRuslan Kuprieiev
Batuhan Taskaya