We didn't mock `os.kill` in `test_dead_process` and the pid it
kills might coincidentlly be the same with some real process.
1. Add a mocker to mock out `os.kill` in `test_dead_process`.
2. Use reset_mock to replace setting it again.
fix: #7062
1. Add `send_signal`, `kill`, `terminate` to processManager.
2. Add tests for each of them.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Because our machine name now is the key value of the machine config
dict. And the machine config value will been overwritten by it when the
config is loaded. We can remove the field `name` from the config.
Borrowed this idea from pandas api. Current dropna methods will drop a
column or row if any data is missing. But it is also quite useful to
drop those columns or rows only if all of its elements are missing. This
is mostly used for pruning a table.
fix: #6940
`--global/system/project/local` help message are in some irrelevant
command `status`, `ssh`, `create`, `destroy`. These commands are more
related to running instances rather than `machine config`, level
options didn't work for them.
1. Remove `--global/system/project/local` options in `status`, `ssh`,
`create`, `destroy`.
2. Add a test for it.
3. Correct a typo error.
fix: #6838
1. make machine status name optional.
2. show all machine status if no name provide.
3. UI changes for this cmd.
4. add a new test for it.
Co-authored-by: Dave Berenbaum <[email protected]>
* Add experiment name check.
1. Add experiment name check (https://git-scm.com/docs/git-check-ref-format)
2. Add duplicate exp name check.
3. Add some unit test for it.
* Ban slash / in dvc exp names
* Use dulwich backend for ref name checking
* Some bug fix
* Update dvc/repo/experiments/__init__.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/__init__.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some review changes
* Make some funtion more reusable.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* `dvc machine`: add a new command `rename` to it
fix#6481
1. add new command `dvc machine rename`
2. add a unit test for the call.
3. add several functional tests for it.t push myself fix6481
* Move rename logic to tpi
* use tpi.state_mv instead of rename
* Rebase to a newer version
* Update tpi version
* Remove unused assert and try catch
* machine: Add a new command `dvc machine status`
fix#6482
1. add a new command `dvc machien status`
2. add a unit test for the cli call.
3. add a unit test for the shown result.
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update dvc/command/machine.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Update machine.py
* Remove sensitive fields
* Update dvc/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* Update tests/func/machine/__init__.py
Co-authored-by: Casper da Costa-Luis <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use white list to limit shown fields
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Casper da Costa-Luis <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* machine: add `machine list`&`machine modify`
1. add new command `dvc machine list`
2. complete `dvc machine modify`
3. add new unit tests for this two commands to ensure the call.
4. add functionality tests for two commands
* Update dvc/config_schema.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update tests/func/machine/test_machine_config.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some reviewed problems
1. capitalize global variables.
2. remove precision restriction.
3. solve windows path seperator problem.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
The current ci failure was caused by wrong order of the experiments.
Because our test script runs so fast that they might share the same
commit timestamp, that makes them in an uncertain order. And then gives
a wrong branch_base.
* Show csv format of experiments
fix#5989 and #5446
1. add --show-csv to dvc exp show
2. add a new functional test for csv format
3. add a new unit test for show_experiements
4. solve name duplication in table common headers (Created,Parent, etc... ), metrics files and param files.
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Clean up remotes's exps
fix#6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
* Remove all of the experiments in the workspace
fix#5676
1. remove all the experiments in workspace
2. add tests for it
* Only remove baseline ones
* Add a new arguments --all
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Rename and shortcut for these commands
* Solve the issue from removing a changing list.
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/repo/experiments/remove.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* delete workspace option
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Remove a special queued experiments
fix#6157
1. dvc exp remove not accept queued experiments name
2. add some tests for this feature
* Extract tests and add revision support
1. Extract remove experiments to a new file.
2. revision can be used to remove special queued experiment
* Accept shortened revisions
* Split removing committed and queued exp functions
* Api name change
* Return to the old API name
* shorten some functions
* Error message change
* Better test cases, more corner case
* Still raise exception in a mixed case
* Push running checkpoint to remote
Fix#6182
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Get the branch name
* Finish this PR
1. move env name to dvc/env.py.
2. add some tests for it.
* Update after finally commit
* Some problems found in the review
1. change the behavior of self remote
2. do not use string DVC_EXP_AUTO_PUSH
3. downgrade the logger level in auto push
4. use full branch ref
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* add a remote check before auto push
* Split current one env into two
* Rename DVC_EXP_CHECKPOINT_PUSH to DVC_EXP_AUTO_PUSH
* value error
* Remove getenv_bool switch to env2bool
* Better on handling Dulwich exceptions
* Name changing removing prints
* Update dvc/env.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Some changes in review
* Add validation to repo URL
* Add new type of SCMError
* Update dvc/repo/experiments/executor/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Use spy tests call counts and args
* Update dvc/scm/git/backend/base.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Update dvc/scm/git/backend/dulwich.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove validate_git_repo to validate_git_remote
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/objects/db/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* bump into ossfs 2021.7.2
* Bump into ossfs 2021.7.3OC
* Switch to real_oss_server
* Switch to a real oss server
* Use the oss bucket located in US.
* Remove some emulator related code
* Add envs to ci
* remove some oss pytest XFAIL
Because we didn't use emulator, we can remove some pytest XFAILs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Decoupling dvcignore and fs
[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check
* Solve all of the arugments error.
* add dvcignore to dvc list
* Solve some test_ignore cases
* add dvcignore to checkout.
1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs
* Solve two tests
* Get rid of ignore_filter
* Add ignore examine to download
* Solve a test issue
* Solve dvcignore and subrepo problems
* Solve sub repo error
* Revert some changes
* pass all tests
* Remove debug codes
* Some final exams
* dvcignore trie tree only update when needed
* reuse of the dir result in path walk.
* Some change after code review.
* Recover something from false `--force` push
* add abspath trans to dvcginore
* pass pylint
* pylint related
* Update dvc/fs/repo.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Changes related to reviews.
* dvcignore call api change
from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)
* Get rid of PathInfo in test_ignore.py
* fs.walk follow the arguments of os.walk
* Dvcignore as parameters
* pass dvcignore as arguments to checkouts
* Decoupling state and repo
* Update dvc/dvcfile.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Update dvc/objects/stage.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Solve some small problems
* Add tests to verify that dependency now not ignore
fix#5256
* Update dvc/ignore.py
* Update dvc/fs/git.py
* Update dvc/repo/__init__.py
* Some small improvements
* pass black
* Refactor is_ignored
* Update dvc/output/base.py
* repofs: use dvcignore in exists/isdir/isfile
Per https://github.com/iterative/dvc/pull/5812/files#r630604524
Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* `check-ignore` show sub-repo patterns
fix#4555
1. Sub-repo pattern now would be considered in `check-ignore` command.
2. New outputs for sub-repo patterns in a `detail`
3. Related tests.
* message change
* Print dvc version info for debugging
fix#4095
1. seperate info from dvc version.
2. print dvc version info when some unexpected error occured.
* Psutil NoneType error
* Update tests/func/test_version.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Move `package` outside dvc/utils/pkg.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
fix#4482
According to the discussion. Make `DVC status [outputs]` output format the same with `DVC status [stages]` instead of `DVC status -c [outputs]`
* Outputs as target supporting for `dvc status`
fix#4191
1. Add a related test which would fail on current version.
* Pass the tests
1. add deps to the tests.
2. make change to pass the tests.
* Update tests/func/test_status.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Solve some change request.
Co-authored-by: Saugat Pachhai <[email protected]>
Fix#4333, as `is_ignored_file` and `is_ignored_dir` and `__call__`
share the same call `DvcIgnorePatterns.matches`, we only need to focus
on the out of repo situation.
1. Add tests for the `is_ignored_file` and `is_ignored_dir` for the out
of repo condition.
2. Now `is_ignored_file` and `is_ignored_dir` returns false for those
files outside the repo.
* Remove dvcignore reset in `repo._reset`
Repo `add`,`checkout`,etc will not change `.dvcignore`, and can remove dvcignore reset after these operations.
* Reset dvcignore manually after a subrepo created.
1. subrepo created during session open time.
* Error message when no match found.
1. remove error message when no match found.
* Test update.
1.Test update cover return 0 and 1
* Remove unused arguments
* Error message show in dvc check-ignore
fix#4361
1. Remove error message when input a empty string
2. Remove error message when read input from stream
2. Remove error message when interrupt by `CTRL-C`
* Two new mode `-all` and `--stdin` for check-ignore
Fix#4321
Introduce new arguments --all for dvc check-ignore
Add tests for dvc check-ignore --all
* `--stdin` and targets only one
1. `--stdin` and targets can and only can have one
2. more tests
* New tests for stdin mode and solve a problem
1. Add a new tests with mock stdin for interactive mode
2. Solve the problem of `assert "" in caplog.text` in tests always `True`.
* Remove path_input interface
1. remove path_input and make _ask to a public method instead
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* fix styling
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update some tests first
fix#3736
* first edition
* solve failure in Windows
* For Windows ci
* Some help ducuments issue.
* Update dvc/ignore.py
abspath
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Refactor with OutOfWorkSpaceError
* Solve a bug
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
* Revert "Refactor with OutOfWorkSpaceError"
This reverts commit 27eec49ed67346c89157edea998b6b5fa81d88f5.
* Two change request
1. Argument `targets`'s description.
2. Error handling of `_get_normalize_path`
* Update dvc/main.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* `check_ignore` now only accept one path a time
1. Add a new test for the out side repo cases
2. check ignore now only check one file not file lists
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* Update
1. unit test update
2. two might cause bugs
* dvc ignore
* Some refactoring
* clean
* Config tree use
* Solve sub dir repo problem. Add more tests
* typo error
* Change request
Co-authored-by: karajan1001 <[email protected]>
* Remove Duplicate ignore Match
* Continue Optimize Dvcignore
fix#3869
* Add a new test with multi ignore files
* Solve merging two dvc files
* Solve Code Climate
* For Windows
* Complete addition of patterns. Add one test
* Systematic test
* Change request
* Change request
* Seperate path sepcification math
* Rename and add comment
* rename change_dirname to private
* Update dvc/pathspec_math.py
list comprehension
Co-authored-by: Alexander Schepanovski <[email protected]>
* Change request
* Update dvc/ignore.py
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Alexander Schepanovski <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Add more tests according to .gitignore
* Test update
* Update test
* Solve question of path
* update one test
* Update 2 tests
* Add one tests
* comment those failed tests caused by other package
* parametrize some tests
Co-authored-by: karajan1001 <[email protected]>
* remote config validation
Fixes#3552
* solving config test fail
* Validation problem in config merging
sloving validation problem in config merging from different levels.
Only those config in lower levels can affect the validation process.
* reduce Cognitive Complexity
* Two request changes
1. Duplicate load current level of config
2. rewrite test to pytest style
* black it
* config: don't use all_levels
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
karajan1001
