Commit Graph
2456 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 04e93da0bd fs: remove unused RemoteCmdError 2021-12-23 19:02:44 +02:00
Ruslan Kuprieiev 72aaca4343 odb: move verify option to config
These options are part of the default config for each particular remote.
Plugins will define their default config on theirown, but for now we keep
those in one config schema.
2021-12-23 12:49:16 +02:00
Saugat Pachhai (सौगात) ba40d91de6 fs.hdfs: skip port if not available 2021-12-22 16:43:56 +02:00
dependabot[bot] 96acb64fec build(deps): Bump scmrepo from 0.0.6 to 0.0.7
Bumps [scmrepo](https://github.com/iterative/scmrepo) from 0.0.6 to 0.0.7.
- [Release notes](https://github.com/iterative/scmrepo/releases)
- [Commits](https://github.com/iterative/scmrepo/compare/0.0.6...0.0.7)

---
updated-dependencies:
- dependency-name: scmrepo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

tests: fix invalid set_ref in exp func test
2021-12-22 17:52:30 +09:00
David de la Iglesia Castro 4d5be46ab4 exp show --html: Handle fill_value.
Adds new argument `fill_value` to ParallelCoordinatesRenderer.

For scalar columns, prevent converting to categorical by explictly inserting None when value is missing (value == fill_value).

For categorical columns, replace `fill_value` with `"Missing"` and ensure it is the last item so it gets rendered on top of the column.
2021-12-17 14:45:12 +09:00
David de la Iglesia Castro 5ee82d4000 exp show: Add parallel coordinates plot.
Uses `TabularData.to_parallel_coordinates`.
Adds new arguments: `html`, `out`, `open`

Reuses `--sort-by` to define colorscale.

Closes #4455
2021-12-17 14:45:12 +09:00
David de la Iglesia Castro a5b7c5ea19 TabularData: Add to_parallel_coordinates.
Uses `ParallelCoordinatesRenderer` and `dvc.render.html.write`.

pre-requisite #4455
2021-12-17 14:45:12 +09:00
David de la Iglesia Castro 19b3c82698 dvc render: Add plotly parallel coords renderer
New renderer based on plotly. Not exposed to `dvc plots`.
Generate plotly datapoints from `TabularData`.

pre-requisite #4455
2021-12-17 14:45:12 +09:00
David de la Iglesia Castro 9bd60f62d7 exp show: Add alternating row colors and update border style.
Use `box.HORIZONTALS`.

rich_table: Support `box.HORIZONTALS`

Closes #7128
2021-12-17 14:31:53 +09:00
Ruslan Kuprieiev 30d6839a5f diff: use relpaths
Regression from path_info removal.

Also fixes #7147
2021-12-15 17:27:38 +02:00
Paweł Redzyński e24de3a0f3 plots: introduce --json option
Fixes: #7073
2021-12-15 11:05:57 +01:00
Peter Rowlands 25ef812ecf exp: use process manager dirs for serialized executors
- partially refactor existing pidfile behavior into using proc manager
- old pidfiles are now json-serialized executor info files placed in
  proc manager subdirs (alongside proc manager pidfile/logs/etc)
- move multiprocessing based (attached) execution into separate
  execution path from future "detached" execution support
2021-12-15 15:55:32 +09:00
Peter Rowlands 37f8010b42 exp: initial executor manager refactor
- introduce ExecutorManager classes
- move executor init + execution from experiments.__init__ into the
  manager classes
- make executors serializable via ExecutorInfo
2021-12-15 15:55:32 +09:00
Saugat Pachhai (सौगात) 3ca8e26169 tests: get rid of GitFS related tests
Also refactored unittest-based tests for LocalFileSystem to pytest
2021-12-14 17:55:26 +05:45
karajan1001andPeter Rowlands 0deab0118a exp: split exp sub command (#5489)
related to #5489
Current dvc/command/experiments.py is too long for switch between cmd class
and the argument parser.

Split the sub command in dvc/command/experiments.py to individual files

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-12-14 19:16:04 +08:00
Saugat Pachhai (सौगात) ba4955738f tests: remove func.test_scm 2021-12-13 21:03:24 +05:45
Ruslan Kuprieiev 81eda7e2b7 fs: get rid of CHECKSUM_DIR_SUFFIX and is_dir_hash
These are leftovers from old remotes, that clearly have nothing
to do with filesystems.
2021-12-10 10:53:10 +02:00
pre-commit-ci[bot] 0bc101964e [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2021-12-07 09:55:03 +05:45
Saugat Pachhai (सौगात) 3319808ed2 exp init: use shortlinks for doclinks 2021-12-07 09:55:03 +05:45
Saugat Pachhai (सौगात) cd37ebca37 exp init: minor refactoring 2021-12-07 08:40:46 +05:45
dberenbaum 4385720fab exp init: update tests 2021-12-07 07:34:31 +05:45
Saugat Pachhai (सौगात) 389fd48b5a exp init: change type to --type=live to --type=dl 2021-12-07 05:56:23 +05:45
Ruslan Kuprieiev ec73300782 odb: make some listing methods private
These are not used anywhere outside of odb, so we can make them private
to reduce the scope for the upcoming refactoring.
2021-12-06 14:10:50 +02:00
Paweł Redzyński 954ac6ab03 plots: roll back templates in packages change
Fixes: #6911
Reverts: a1b146669c
2021-12-06 09:55:13 +02:00
karajan1001 9cf1f45e71 proc: implement cleanup/removal (#7063)
fix: #7063
Add `cleanup` and `removal` function to ProcessManager
2021-12-06 15:25:15 +09:00
David de la Iglesia Castro 728731d4d7 ui: Add open_browser.
Use `webbrowser.open` and relpath for WSL plattform.
2021-12-05 15:44:13 +02:00
karajan1001 eb26b9a78d ci: fix ci failure
We didn't mock `os.kill` in `test_dead_process` and the pid it
kills might coincidentlly be the same with some real process.

1. Add a mocker to mock out `os.kill` in `test_dead_process`.
2. Use reset_mock to replace setting it again.
2021-12-04 19:40:57 +09:00
Ruslan Kuprieiev b6f2a8e653 tests: remove redundant remote tests
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.

This PR also removes some redundant/broken remote tests.
2021-12-03 22:45:41 +02:00
karajan1001andPeter Rowlands 7d68863ba2 proc: implement process signaling(#7062)
fix: #7062
1. Add `send_signal`, `kill`, `terminate` to processManager.
2. Add tests for each of them.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-12-02 16:39:41 +09:00
Ruslan Kuprieiev 09f48a5aaa tests: introduce dvc.testing
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
2021-12-02 09:08:40 +02:00
Peter Rowlands 15eb326188 proc: initial ManagedProcess implementation
- supports running (backgrounded) processes with redirected output
2021-11-29 07:00:38 +02:00
Saugat Pachhai (सौगात) ea52ab2260 remove dvc.scm and dvc.fs.git module; rename dvc.fs.scm to dvc.fs.git 2021-11-25 19:09:58 +02:00
Saugat Pachhai (सौगात) 95c726b297 fs.git: refactor GitFileSystem to be fsspec compliant
Also adds a wrapper for DVC to use as GitFileSystem in dvc.fs.scm.
This is temporary, and will be renamed back to dvc.fs.git after the
split.

Note that this filesystem is not fully compliant, it's only
meant to be compatible with our previous implementation.
2021-11-25 08:32:20 +05:45
Saugat Pachhai (सौगात) a282e0e48c scm: restructure scm
Makes it easier to decouple, we'd only need to rename
dvc/scm/__init__.py to dvc/scm.py.
2021-11-24 11:46:40 +05:45
Saugat Pachhai (सौगात) 48581715b6 scm: remove exp related behaviours from scm.resolve_rev 2021-11-24 11:39:39 +05:45
Saugat Pachhai (सौगात) a0f4e12fb4 scm: unified progress reporting in a structured way
This adds a support for reporting progress from Git in a structured
format. The gitpython already provides somewhat structured progress
reporting, dulwich on the other hand only provides git-like line
messages.

As gitpython uses git binary, it already parse this line format
in a structured way. This PR makes dulwich use the same machinery
to unify progress reporting for both dulwich and gitpython python.

This does add a dependency on gitpython even on dulwich backend,
so I am iffy if we should do this.
The alternative is to translate gitpython's progress reports to
string, and just use string messages as progress reports on both
dulwich and gitpython.
2021-11-23 19:50:12 +02:00
Saugat Pachhai (सौगात) f76ef84c88 exceptions: migrate other scm exceptions to internal 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) f2f4227656 exceptions: migrate RevError 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) c19a11152a scm: move FileNotInRepoError and MergeConflictError to scm.exceptions 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) cab60b54f8 live: import webbrowser lazily; only patch webbrowser.open 2021-11-23 22:08:21 +05:45
David de la Iglesia Castro fc7caa487f tests: Patch webbrowser.open on session level
Closes #7014
2021-11-23 22:08:21 +05:45
Saugat Pachhai (सौगात) 568abb3e28 scm: remove dependency from DvcException 2021-11-23 13:19:41 +02:00
Saugat Pachhai (सौगात) 00e53c5c32 tests: fix black 2021-11-22 17:33:12 +05:45
Saugat Pachhai (सौगात) 0aaf96bce4 exp init: show 'n to omit' on --explicit mode 2021-11-22 17:20:16 +05:45
Saugat Pachhai (सौगात) e07b99ed33 tests: increase timeout to 2 second for prompts test 2021-11-22 15:10:55 +05:45
Saugat Pachhai (सौगात) 7bc906011f tests: timeout only on functions, excluding tests fixtures
Sometimes, `tmp_dir` fixtures take longer than a second. Using `func_only`
makes it only consider duration of test function run, excluding fixtures
duration.
2021-11-22 15:10:55 +05:45
Saugat Pachhai (सौगात) b84227db89 scm: remove ignore list 2021-11-22 07:16:54 +02:00
David de la Iglesia Castro 318b269ad1 live: Call webbrowser.open.
Use `funcy.once_per_args` decorator to prevent opening the html multiple times if user closes first time.

Closes iterative/dvclive#180
2021-11-22 03:35:23 +02:00
Saugat Pachhai (सौगात) bf313d92aa exp init: timeout prompts' test
The `exp init` tests may just keep waiting on the prompts blocking tests.
The blocking of prompts is likely a failure in tests setup or a bug
in the implementation. It's hard to find out these failures or avoid it.
If they take longer than 1 second, they most likely have failed as
most of test takes 0.1 second at max.
2021-11-22 03:27:16 +02:00
Saugat Pachhai (सौगात) cb941a836d tests: cover exp init 2021-11-19 18:58:23 +05:45