Commit Graph
2430 Commits
Author SHA1 Message Date
karajan1001 eb26b9a78d ci: fix ci failure
We didn't mock `os.kill` in `test_dead_process` and the pid it
kills might coincidentlly be the same with some real process.

1. Add a mocker to mock out `os.kill` in `test_dead_process`.
2. Use reset_mock to replace setting it again.
2021-12-04 19:40:57 +09:00
Ruslan Kuprieiev b6f2a8e653 tests: remove redundant remote tests
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.

This PR also removes some redundant/broken remote tests.
2021-12-03 22:45:41 +02:00
karajan1001andPeter Rowlands 7d68863ba2 proc: implement process signaling(#7062)
fix: #7062
1. Add `send_signal`, `kill`, `terminate` to processManager.
2. Add tests for each of them.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-12-02 16:39:41 +09:00
Ruslan Kuprieiev 09f48a5aaa tests: introduce dvc.testing
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
2021-12-02 09:08:40 +02:00
Peter Rowlands 15eb326188 proc: initial ManagedProcess implementation
- supports running (backgrounded) processes with redirected output
2021-11-29 07:00:38 +02:00
Saugat Pachhai (सौगात) ea52ab2260 remove dvc.scm and dvc.fs.git module; rename dvc.fs.scm to dvc.fs.git 2021-11-25 19:09:58 +02:00
Saugat Pachhai (सौगात) 95c726b297 fs.git: refactor GitFileSystem to be fsspec compliant
Also adds a wrapper for DVC to use as GitFileSystem in dvc.fs.scm.
This is temporary, and will be renamed back to dvc.fs.git after the
split.

Note that this filesystem is not fully compliant, it's only
meant to be compatible with our previous implementation.
2021-11-25 08:32:20 +05:45
Saugat Pachhai (सौगात) a282e0e48c scm: restructure scm
Makes it easier to decouple, we'd only need to rename
dvc/scm/__init__.py to dvc/scm.py.
2021-11-24 11:46:40 +05:45
Saugat Pachhai (सौगात) 48581715b6 scm: remove exp related behaviours from scm.resolve_rev 2021-11-24 11:39:39 +05:45
Saugat Pachhai (सौगात) a0f4e12fb4 scm: unified progress reporting in a structured way
This adds a support for reporting progress from Git in a structured
format. The gitpython already provides somewhat structured progress
reporting, dulwich on the other hand only provides git-like line
messages.

As gitpython uses git binary, it already parse this line format
in a structured way. This PR makes dulwich use the same machinery
to unify progress reporting for both dulwich and gitpython python.

This does add a dependency on gitpython even on dulwich backend,
so I am iffy if we should do this.
The alternative is to translate gitpython's progress reports to
string, and just use string messages as progress reports on both
dulwich and gitpython.
2021-11-23 19:50:12 +02:00
Saugat Pachhai (सौगात) f76ef84c88 exceptions: migrate other scm exceptions to internal 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) f2f4227656 exceptions: migrate RevError 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) c19a11152a scm: move FileNotInRepoError and MergeConflictError to scm.exceptions 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) cab60b54f8 live: import webbrowser lazily; only patch webbrowser.open 2021-11-23 22:08:21 +05:45
David de la Iglesia Castro fc7caa487f tests: Patch webbrowser.open on session level
Closes #7014
2021-11-23 22:08:21 +05:45
Saugat Pachhai (सौगात) 568abb3e28 scm: remove dependency from DvcException 2021-11-23 13:19:41 +02:00
Saugat Pachhai (सौगात) 00e53c5c32 tests: fix black 2021-11-22 17:33:12 +05:45
Saugat Pachhai (सौगात) 0aaf96bce4 exp init: show 'n to omit' on --explicit mode 2021-11-22 17:20:16 +05:45
Saugat Pachhai (सौगात) e07b99ed33 tests: increase timeout to 2 second for prompts test 2021-11-22 15:10:55 +05:45
Saugat Pachhai (सौगात) 7bc906011f tests: timeout only on functions, excluding tests fixtures
Sometimes, `tmp_dir` fixtures take longer than a second. Using `func_only`
makes it only consider duration of test function run, excluding fixtures
duration.
2021-11-22 15:10:55 +05:45
Saugat Pachhai (सौगात) b84227db89 scm: remove ignore list 2021-11-22 07:16:54 +02:00
David de la Iglesia Castro 318b269ad1 live: Call webbrowser.open.
Use `funcy.once_per_args` decorator to prevent opening the html multiple times if user closes first time.

Closes iterative/dvclive#180
2021-11-22 03:35:23 +02:00
Saugat Pachhai (सौगात) bf313d92aa exp init: timeout prompts' test
The `exp init` tests may just keep waiting on the prompts blocking tests.
The blocking of prompts is likely a failure in tests setup or a bug
in the implementation. It's hard to find out these failures or avoid it.
If they take longer than 1 second, they most likely have failed as
most of test takes 0.1 second at max.
2021-11-22 03:27:16 +02:00
Saugat Pachhai (सौगात) cb941a836d tests: cover exp init 2021-11-19 18:58:23 +05:45
Ruslan Kuprieiev c9dd137426 fs: BaseFileSystem -> FileSystem
Following fsspec conventions and new object conventions.
2021-11-18 19:58:23 +02:00
Saugat Pachhai (सौगात) 93e6526907 scm: split tracking of to-be git-ignored/git-staged into SCMContext 2021-11-18 15:34:45 +02:00
Saugat Pachhai (सौगात) eac030598c ui.prompt: move prompts from exp init to ui.prompt
Some fixes along the way to make it more generic, but it still has
lots of features tied with exp init.
2021-11-18 13:22:45 +05:45
karajan1001 068b8371aa Machine: table output for machine status(#6892)
Change the output format of `dvc machine status` command to tables for a
better readability.
2021-11-18 11:22:13 +08:00
karajan1001 888b6392bb Machine:remove field name in config(#6892)
Because our machine name now is the key value of the machine config
dict. And the machine config value will been overwritten by it when the
config is loaded. We can remove the field `name` from the config.
2021-11-18 11:22:13 +08:00
karajan1001 b24a31b834 Machine: use table for list outputs(#6892)
fix: #6892

Current output of machine list is quite urgly, it is originaly used to
show configurations. In this PR, we use a new table format for it.
2021-11-18 11:22:13 +08:00
karajan1001 45553c5b30 TabularData: add a new how arg to dropna(#6892)
Borrowed this idea from pandas api. Current dropna methods will drop a
column or row if any data is missing. But it is also quite useful to
drop those columns or rows only if all of its elements are missing. This
is mostly used for pruning a table.
2021-11-18 11:22:13 +08:00
Peter Rowlands 25dca0b665 dulwich: workaround github SSH server in asyncssh vendor
- allow mismatched RSA signature algo names in initial
  SSH_MSG_USERAUTH_PK_OK response
- unit test overridden asyncssh publickey auth compatibility with
  github ssh server
2021-11-17 12:48:34 +02:00
karajan1001 eea67a6396 machine:remove help text for irrelevant commands (#6940)
fix: #6940

`--global/system/project/local` help message are in some irrelevant
command `status`, `ssh`, `create`, `destroy`. These commands are more
related to running instances rather than `machine config`, level
options didn't work for them.

1. Remove `--global/system/project/local` options in `status`, `ssh`,
   `create`, `destroy`.
2. Add a test for it.
3. Correct a typo error.
2021-11-17 16:25:48 +08:00
karajan1001andDave Berenbaum f43c04f935 Make machine status name optional
fix: #6838
1. make machine status name optional.
2. show all machine status if no name provide.
3. UI changes for this cmd.
4. add a new test for it.

Co-authored-by: Dave Berenbaum <[email protected]>
2021-11-17 15:44:36 +08:00
Paweł Redzyński e0fd0790fc plots: special chars handling for vega plots paths
Fixes: #6894
2021-11-17 01:57:35 +02:00
Ruslan Kuprieiev 88660f6a66 dvc: use transfer in checkout/doctor/odb
All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.

Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.

This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).

Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.

Continuing from #6963

Fixes #6980
2021-11-15 22:50:00 +02:00
David de la Iglesia Castro 7df6090a9c params: Use ast.literal_eval for python params.
Add unit tests for `parse_py`

Closes  #6953
2021-11-15 17:52:49 +02:00
David de la Iglesia Castro 4a1985d7cd dvc render: Add RENDERERS registry.
Related: #6840, #4455, #6943
2021-11-15 16:37:24 +01:00
David de la Iglesia Castro c8903dc3a6 live: Add test for logging images with dvclive
Closes iterative/dvclive#190
2021-11-15 12:47:02 +01:00
David de la Iglesia Castro 28dd39a1a0 exp diff: Remove --old option.
After #6814 the header of `diff_table` is `a_rev/`b_rev` instead of `Old`/`New` so the name of the option was counterintuitive.

In addition, make `a_rev` show by default, for consistency with output format of other `diff_table` uses (`dvc metrics diff` / `dvc params diff`).
2021-11-12 12:58:12 +02:00
Ruslan Kuprieiev 7e3e257c7f utils: fs: remove unused fs_copy 2021-11-11 01:26:37 +02:00
Paweł Redzyński 36ed182f26 argparse: dont override cmd
Related: #6956, https://bugs.python.org/issue45235
2021-11-11 01:14:03 +02:00
David de la Iglesia Castro 72066f233b exp show: Add --only-changed option.
Use TabularData `drop_duplicates` to only show metrics/params with values varying across the selected experiments.

Fixes #5966
2021-11-10 16:52:18 +02:00
David de la Iglesia Castro cb6f625dd9 TabularData: Add drop_duplicates method.
This method mimics pandas `drop_duplicates` when axis=rows.

For axis=cols this method will drop columns with 0-variance (only
1 unique value). NaNs (`_fill_value`) won't be counted as unique values.

So the following column (`-` would be the `_fill_value`) will be dropped:

foo
-
foo
foo
2021-11-10 16:52:18 +02:00
Batuhan Taskaya 9aec8f1926 dvc: get rid of path_info
Intermediate PR in removing path_info from our code base.

It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.

For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).

Fixes #6774
2021-11-10 00:52:48 +02:00
Guðmundur HeimissonandRuslan Kuprieiev c5b1a73dc5 webhdfs: expose kerberos and https options (#6936)
* Add fsspec options for webhdfs

* Add requests-kerberos dependency

* Update dvc/fs/webhdfs.py

Simplify config.pop.

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/fs/webhdfs.py

* Simplify kerb_kwargs config setting.

Co-authored-by: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-11-07 20:42:10 +02:00
Paweł Redzyński 22cc2a5cbc plots: fix resolve template path bug
Fix: #6854
2021-11-02 16:49:56 +02:00
Saugat Pachhai (सौगात) 8fc959060a tests: skip based on conditional import on pyarrow
From now on, pyarrow dependency is optional for the tests.
2021-11-01 19:15:28 +05:45
David de la Iglesia Castro 24abc6f328 TabularData: Add dropna method.
This method mimics pandas `dropna`.
2021-10-29 15:28:36 +03:00
David de la Iglesia Castro 4dd544121f plots: Use URI for --open.
We were previously using relpath in *all* cases, mainly to support WSL but relpath doesn't work
on macOS.

Added a check for using relpath *only* for WSL.

Fixes #6092
2021-10-29 13:27:12 +02:00