884 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 81eda7e2b7 fs: get rid of CHECKSUM_DIR_SUFFIX and is_dir_hash
These are leftovers from old remotes, that clearly have nothing
to do with filesystems.
2021-12-10 10:53:10 +02:00
pre-commit-ci[bot] 0bc101964e [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2021-12-07 09:55:03 +05:45
Saugat Pachhai (सौगात) 3319808ed2 exp init: use shortlinks for doclinks 2021-12-07 09:55:03 +05:45
Saugat Pachhai (सौगात) cd37ebca37 exp init: minor refactoring 2021-12-07 08:40:46 +05:45
dberenbaum 4385720fab exp init: update tests 2021-12-07 07:34:31 +05:45
Saugat Pachhai (सौगात) 389fd48b5a exp init: change type to --type=live to --type=dl 2021-12-07 05:56:23 +05:45
Ruslan Kuprieiev ec73300782 odb: make some listing methods private
These are not used anywhere outside of odb, so we can make them private
to reduce the scope for the upcoming refactoring.
2021-12-06 14:10:50 +02:00
Paweł Redzyński 954ac6ab03 plots: roll back templates in packages change
Fixes: #6911
Reverts: a1b146669c
2021-12-06 09:55:13 +02:00
karajan1001 9cf1f45e71 proc: implement cleanup/removal (#7063)
fix: #7063
Add `cleanup` and `removal` function to ProcessManager
2021-12-06 15:25:15 +09:00
David de la Iglesia Castro 728731d4d7 ui: Add open_browser.
Use `webbrowser.open` and relpath for WSL plattform.
2021-12-05 15:44:13 +02:00
karajan1001 eb26b9a78d ci: fix ci failure
We didn't mock `os.kill` in `test_dead_process` and the pid it
kills might coincidentlly be the same with some real process.

1. Add a mocker to mock out `os.kill` in `test_dead_process`.
2. Use reset_mock to replace setting it again.
2021-12-04 19:40:57 +09:00
Ruslan Kuprieiev b6f2a8e653 tests: remove redundant remote tests
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.

This PR also removes some redundant/broken remote tests.
2021-12-03 22:45:41 +02:00
karajan1001andPeter Rowlands 7d68863ba2 proc: implement process signaling(#7062)
fix: #7062
1. Add `send_signal`, `kill`, `terminate` to processManager.
2. Add tests for each of them.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-12-02 16:39:41 +09:00
Peter Rowlands 15eb326188 proc: initial ManagedProcess implementation
- supports running (backgrounded) processes with redirected output
2021-11-29 07:00:38 +02:00
Saugat Pachhai (सौगात) ea52ab2260 remove dvc.scm and dvc.fs.git module; rename dvc.fs.scm to dvc.fs.git 2021-11-25 19:09:58 +02:00
Saugat Pachhai (सौगात) f76ef84c88 exceptions: migrate other scm exceptions to internal 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) f2f4227656 exceptions: migrate RevError 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) c19a11152a scm: move FileNotInRepoError and MergeConflictError to scm.exceptions 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) 00e53c5c32 tests: fix black 2021-11-22 17:33:12 +05:45
Saugat Pachhai (सौगात) 0aaf96bce4 exp init: show 'n to omit' on --explicit mode 2021-11-22 17:20:16 +05:45
Saugat Pachhai (सौगात) cb941a836d tests: cover exp init 2021-11-19 18:58:23 +05:45
Ruslan Kuprieiev c9dd137426 fs: BaseFileSystem -> FileSystem
Following fsspec conventions and new object conventions.
2021-11-18 19:58:23 +02:00
Saugat Pachhai (सौगात) 93e6526907 scm: split tracking of to-be git-ignored/git-staged into SCMContext 2021-11-18 15:34:45 +02:00
Saugat Pachhai (सौगात) eac030598c ui.prompt: move prompts from exp init to ui.prompt
Some fixes along the way to make it more generic, but it still has
lots of features tied with exp init.
2021-11-18 13:22:45 +05:45
karajan1001 b24a31b834 Machine: use table for list outputs(#6892)
fix: #6892

Current output of machine list is quite urgly, it is originaly used to
show configurations. In this PR, we use a new table format for it.
2021-11-18 11:22:13 +08:00
karajan1001 45553c5b30 TabularData: add a new how arg to dropna(#6892)
Borrowed this idea from pandas api. Current dropna methods will drop a
column or row if any data is missing. But it is also quite useful to
drop those columns or rows only if all of its elements are missing. This
is mostly used for pruning a table.
2021-11-18 11:22:13 +08:00
Peter Rowlands 25dca0b665 dulwich: workaround github SSH server in asyncssh vendor
- allow mismatched RSA signature algo names in initial
  SSH_MSG_USERAUTH_PK_OK response
- unit test overridden asyncssh publickey auth compatibility with
  github ssh server
2021-11-17 12:48:34 +02:00
karajan1001 eea67a6396 machine:remove help text for irrelevant commands (#6940)
fix: #6940

`--global/system/project/local` help message are in some irrelevant
command `status`, `ssh`, `create`, `destroy`. These commands are more
related to running instances rather than `machine config`, level
options didn't work for them.

1. Remove `--global/system/project/local` options in `status`, `ssh`,
   `create`, `destroy`.
2. Add a test for it.
3. Correct a typo error.
2021-11-17 16:25:48 +08:00
karajan1001andDave Berenbaum f43c04f935 Make machine status name optional
fix: #6838
1. make machine status name optional.
2. show all machine status if no name provide.
3. UI changes for this cmd.
4. add a new test for it.

Co-authored-by: Dave Berenbaum <[email protected]>
2021-11-17 15:44:36 +08:00
Paweł Redzyński e0fd0790fc plots: special chars handling for vega plots paths
Fixes: #6894
2021-11-17 01:57:35 +02:00
Ruslan Kuprieiev 88660f6a66 dvc: use transfer in checkout/doctor/odb
All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.

Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.

This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).

Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.

Continuing from #6963

Fixes #6980
2021-11-15 22:50:00 +02:00
David de la Iglesia Castro 7df6090a9c params: Use ast.literal_eval for python params.
Add unit tests for `parse_py`

Closes  #6953
2021-11-15 17:52:49 +02:00
David de la Iglesia Castro 4a1985d7cd dvc render: Add RENDERERS registry.
Related: #6840, #4455, #6943
2021-11-15 16:37:24 +01:00
David de la Iglesia Castro 28dd39a1a0 exp diff: Remove --old option.
After #6814 the header of `diff_table` is `a_rev/`b_rev` instead of `Old`/`New` so the name of the option was counterintuitive.

In addition, make `a_rev` show by default, for consistency with output format of other `diff_table` uses (`dvc metrics diff` / `dvc params diff`).
2021-11-12 12:58:12 +02:00
Paweł Redzyński 36ed182f26 argparse: dont override cmd
Related: #6956, https://bugs.python.org/issue45235
2021-11-11 01:14:03 +02:00
David de la Iglesia Castro cb6f625dd9 TabularData: Add drop_duplicates method.
This method mimics pandas `drop_duplicates` when axis=rows.

For axis=cols this method will drop columns with 0-variance (only
1 unique value). NaNs (`_fill_value`) won't be counted as unique values.

So the following column (`-` would be the `_fill_value`) will be dropped:

foo
-
foo
foo
2021-11-10 16:52:18 +02:00
Batuhan Taskaya 9aec8f1926 dvc: get rid of path_info
Intermediate PR in removing path_info from our code base.

It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.

For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).

Fixes #6774
2021-11-10 00:52:48 +02:00
Guðmundur HeimissonandRuslan Kuprieiev c5b1a73dc5 webhdfs: expose kerberos and https options (#6936)
* Add fsspec options for webhdfs

* Add requests-kerberos dependency

* Update dvc/fs/webhdfs.py

Simplify config.pop.

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/fs/webhdfs.py

* Simplify kerb_kwargs config setting.

Co-authored-by: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-11-07 20:42:10 +02:00
Paweł Redzyński 22cc2a5cbc plots: fix resolve template path bug
Fix: #6854
2021-11-02 16:49:56 +02:00
David de la Iglesia Castro 24abc6f328 TabularData: Add dropna method.
This method mimics pandas `dropna`.
2021-10-29 15:28:36 +03:00
David de la Iglesia Castro 4dd544121f plots: Use URI for --open.
We were previously using relpath in *all* cases, mainly to support WSL but relpath doesn't work
on macOS.

Added a check for using relpath *only* for WSL.

Fixes #6092
2021-10-29 13:27:12 +02:00
Ruslan Kuprieiev 922eb20e44 staging: don't use move 2021-10-27 20:19:17 +03:00
GaoandPeter Rowlands c07592131a exp run: add experiment name check. (#6848)
* Add experiment name check.

1. Add experiment name check (https://git-scm.com/docs/git-check-ref-format)
2. Add duplicate exp name check.
3. Add some unit test for it.

* Ban slash / in dvc exp names

* Use dulwich backend for ref name checking

* Some bug fix

* Update dvc/repo/experiments/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/repo/experiments/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Some review changes

* Make some funtion more reusable.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-10-25 20:16:37 +09:00
b652670cbf Dependabot/pip/pylint 2.11.1 (#6843)
* build(deps): Bump pylint from 2.9.6 to 2.11.1

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.11.1.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.11.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix pylint problems

* Updation after rebase.

* system: use binary mode

* Set remote execution script encoding type locale

* Update dvc/machine/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-10-20 14:34:10 +03:00
Gao 6de3e734ed Revert "build(deps): Bump pylint from 2.9.6 to 2.11.1 (#6636)" (#6842)
This reverts commit 464620806c.
2021-10-20 12:41:00 +03:00
Gao 4b2bec89ec dvc machine: add a new command rename to it (#6633)
* `dvc machine`: add a new command `rename` to it

fix #6481
1. add new command `dvc machine rename`
2. add a unit test for the call.
3. add several functional tests for it.t push myself fix6481

* Move rename logic to tpi

* use tpi.state_mv instead of rename

* Rebase to a newer version

* Update tpi version

* Remove unused assert and try catch
2021-10-20 15:57:44 +09:00
f964a44e7c Exp show md (#6799)
* Added `--md`/`--show-md` option to `exp show`.

* Added test_show_experiments_md

* Fixed re-defining all_experiments

* Fixed all_experiments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added dest markdown

* Added textwrap.dedent

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update experiments.py

* Apply suggestions from code review

Co-authored-by: Saugat Pachhai <[email protected]>

* Update tests/unit/command/test_compat_flag.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Saugat Pachhai <[email protected]>
2021-10-20 11:04:57 +05:45
464620806c build(deps): Bump pylint from 2.9.6 to 2.11.1 (#6636)
* build(deps): Bump pylint from 2.9.6 to 2.11.1

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.11.1.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.11.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix pylint problems
* system: use binary mode

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-10-20 10:32:44 +08:00
966dc8a1ca machine: Add a new command dvc machine status (#6649)
* machine: Add a new command `dvc machine status`

fix#6482
1. add a new command `dvc machien status`
2. add a unit test for the cli call.
3. add a unit test for the shown result.

* Update dvc/command/machine.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update dvc/command/machine.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update machine.py

* Remove sensitive fields

* Update dvc/machine/__init__.py

Co-authored-by: Casper da Costa-Luis <[email protected]>

* Update tests/func/machine/__init__.py

Co-authored-by: Casper da Costa-Luis <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use white list to limit shown fields

Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Casper da Costa-Luis <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-10-19 17:38:11 +09:00
David de la Iglesia Castro 4f16852005 exp/metrics/params diff: Replaced Old/New with a_rev/b_rev (#6814)
* Replaced Old/New with a_rev/b_rev

* Fixed tests
2021-10-18 21:09:02 +03:00