Commit Graph
7060 Commits
Author SHA1 Message Date
karajan1001 45553c5b30 TabularData: add a new how arg to dropna(#6892)
Borrowed this idea from pandas api. Current dropna methods will drop a
column or row if any data is missing. But it is also quite useful to
drop those columns or rows only if all of its elements are missing. This
is mostly used for pruning a table.
2021-11-18 11:22:13 +08:00
Peter Rowlands 28f57933db dulwich: flush stdin channel on asyncssh vendor write 2021-11-17 12:48:34 +02:00
Peter Rowlands 25dca0b665 dulwich: workaround github SSH server in asyncssh vendor
- allow mismatched RSA signature algo names in initial
  SSH_MSG_USERAUTH_PK_OK response
- unit test overridden asyncssh publickey auth compatibility with
  github ssh server
2021-11-17 12:48:34 +02:00
Peter Rowlands 45c4e1998d dulwich: fallback to default client config in asyncssh vendor
- vendor should use ssh-agent or default IdentityFile(s) when no SSH key
  has been provided by the user
- encrypted (fallback) keys will always be passed to ssh-agent
2021-11-17 12:48:34 +02:00
karajan1001 eea67a6396 machine:remove help text for irrelevant commands (#6940)
fix: #6940

`--global/system/project/local` help message are in some irrelevant
command `status`, `ssh`, `create`, `destroy`. These commands are more
related to running instances rather than `machine config`, level
options didn't work for them.

1. Remove `--global/system/project/local` options in `status`, `ssh`,
   `create`, `destroy`.
2. Add a test for it.
3. Correct a typo error.
2021-11-17 16:25:48 +08:00
karajan1001andDave Berenbaum f43c04f935 Make machine status name optional
fix: #6838
1. make machine status name optional.
2. show all machine status if no name provide.
3. UI changes for this cmd.
4. add a new test for it.

Co-authored-by: Dave Berenbaum <[email protected]>
2021-11-17 15:44:36 +08:00
dependabot[bot] 3486435a7f build: bump filelock from 3.3.2 to 3.4.0
Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.3.2 to 3.4.0.
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/py-filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.3.2...3.4.0)

---
updated-dependencies:
- dependency-name: filelock
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-17 14:04:17 +08:00
Paweł Redzyński e0fd0790fc plots: special chars handling for vega plots paths
Fixes: #6894
2021-11-17 01:57:35 +02:00
dberenbaum f99d3cda57 scm: hint to autostage 2021-11-16 18:01:18 +02:00
pre-commit-ci[bot] 003461f9a2 [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/timothycrosley/isort: 5.9.3 → 5.10.1](https://github.com/timothycrosley/isort/compare/5.9.3...5.10.1)
2021-11-16 07:55:56 +05:45
Ruslan Kuprieiev 88660f6a66 dvc: use transfer in checkout/doctor/odb
All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.

Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.

This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).

Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.

Continuing from #6963

Fixes #6980
2021-11-15 22:50:00 +02:00
David de la Iglesia Castro 7df6090a9c params: Use ast.literal_eval for python params.
Add unit tests for `parse_py`

Closes  #6953
2021-11-15 17:52:49 +02:00
Dimitar Todorov fdba79c519 change method override to call super instead of repeating the calls 2021-11-15 17:50:14 +02:00
Димитър Тодоров (Dimitar Todorov)andSaugat Pachhai 2f6b6cc586 Update dvc/fs/http.py
Co-authored-by: Saugat Pachhai <[email protected]>
2021-11-15 17:50:14 +02:00
Dimitar Todorov 0fea622266 push: fix http storage ignoring method setting
Add method overriding the default put_file in http.py
where the upload method is passed to the http implementation.

Fixes: #6694
2021-11-15 17:50:14 +02:00
David de la Iglesia Castro 4a1985d7cd dvc render: Add RENDERERS registry.
Related: #6840, #4455, #6943
2021-11-15 16:37:24 +01:00
David de la Iglesia Castro c8903dc3a6 live: Add test for logging images with dvclive
Closes iterative/dvclive#190
2021-11-15 12:47:02 +01:00
David de la Iglesia Castro 28dd39a1a0 exp diff: Remove --old option.
After #6814 the header of `diff_table` is `a_rev/`b_rev` instead of `Old`/`New` so the name of the option was counterintuitive.

In addition, make `a_rev` show by default, for consistency with output format of other `diff_table` uses (`dvc metrics diff` / `dvc params diff`).
2021-11-12 12:58:12 +02:00
dependabot[bot] ef9d8580cc build: bump types-requests from 2.25.12 to 2.26.0
Bumps [types-requests](https://github.com/python/typeshed) from 2.25.12 to 2.26.0.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-12 12:40:38 +02:00
dependabot[bot] 1af23f32b7 build: bump pyinstaller from 4.6 to 4.7
Bumps [pyinstaller](https://github.com/pyinstaller/pyinstaller) from 4.6 to 4.7.
- [Release notes](https://github.com/pyinstaller/pyinstaller/releases)
- [Changelog](https://github.com/pyinstaller/pyinstaller/blob/v4.7/doc/CHANGES.rst)
- [Commits](https://github.com/pyinstaller/pyinstaller/compare/v4.6...v4.7)

---
updated-dependencies:
- dependency-name: pyinstaller
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-12 12:40:27 +02:00
Saugat Pachhai (सौगात) 7b488887b4 exp init: suppress plots if live is selected as a stage type 2021-11-12 11:25:39 +05:45
Saugat Pachhai (सौगात) 8262dbd37c exp init: hide metrics/plots from worktree on --type=live 2021-11-12 11:21:58 +05:45
Ruslan Kuprieiev 264d78b637 ostage: don't forget to convert https etags
Fixes #6966
2021-11-12 00:54:44 +02:00
Ruslan Kuprieiev 1b2400842d checkout: catch ENOSYS 2021-11-11 22:38:29 +02:00
Saugat Pachhai (सौगात) 81db879c51 hardlink: fallback to other link methods if 38 error code is received
Fixes #6962, where GDdrive mounted on Colab was failing when trying to hardlink
with the message that the 'Function not implemented'.
2021-11-11 22:38:29 +02:00
Saugat Pachhai (सौगात) d668cbeb45 exp init: warn for code/data if they don't exist 2021-11-11 08:09:37 +05:45
Saugat Pachhai (सौगात) a466241552 exp init: validate params as soon as it's entered in prompt
It'll give a error message to the user and reprompt them.
This mechanism can be extended to other prompts, and we
can show a warning message and/or reprompt them based on
validations.

Closes #6865. Part of #6446.
2021-11-11 08:09:37 +05:45
Ruslan Kuprieiev 7e3e257c7f utils: fs: remove unused fs_copy 2021-11-11 01:26:37 +02:00
Paweł Redzyński ab56e4bb9b gha: limit python version to 3.9.7
Related: https://bugs.python.org/issue45235
Closes: #6956
2021-11-11 01:14:03 +02:00
Paweł Redzyński 36ed182f26 argparse: dont override cmd
Related: #6956, https://bugs.python.org/issue45235
2021-11-11 01:14:03 +02:00
Ruslan Kuprieiev 045d1f5e3e localfs: remove LocalPath
fs.path is meant to be lightweight, and we should try to avoid using os.path.

Leftover from #6946
2021-11-10 20:10:48 +02:00
Ruslan Kuprieiev 5214aec1bb fspath: introduce relparts 2021-11-10 20:10:48 +02:00
David de la Iglesia Castro 72066f233b exp show: Add --only-changed option.
Use TabularData `drop_duplicates` to only show metrics/params with values varying across the selected experiments.

Fixes #5966
2021-11-10 16:52:18 +02:00
David de la Iglesia Castro cb6f625dd9 TabularData: Add drop_duplicates method.
This method mimics pandas `drop_duplicates` when axis=rows.

For axis=cols this method will drop columns with 0-variance (only
1 unique value). NaNs (`_fill_value`) won't be counted as unique values.

So the following column (`-` would be the `_fill_value`) will be dropped:

foo
-
foo
foo
2021-11-10 16:52:18 +02:00
dependabot[bot] 87fadd3225 build: bump dvclive from 0.4.2 to 0.4.3
Bumps [dvclive](https://github.com/iterative/dvclive) from 0.4.2 to 0.4.3.
- [Release notes](https://github.com/iterative/dvclive/releases)
- [Commits](https://github.com/iterative/dvclive/compare/0.4.2...0.4.3)

---
updated-dependencies:
- dependency-name: dvclive
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-10 11:46:41 +01:00
dependabot[bot] 8f028028b9 build: bump types-requests from 2.25.11 to 2.25.12
Bumps [types-requests](https://github.com/python/typeshed) from 2.25.11 to 2.25.12.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-10 11:45:42 +01:00
Batuhan Taskaya 9aec8f1926 dvc: get rid of path_info
Intermediate PR in removing path_info from our code base.

It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.

For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).

Fixes #6774
2021-11-10 00:52:48 +02:00
dependabot[bot] 4175f9ffa4 build: bump s3/gcsfs from 2021.10.1 to 2021.11.0
Bumps [gcsfs](https://github.com/fsspec/gcsfs) from 2021.10.1 to 2021.11.0.
- [Release notes](https://github.com/fsspec/gcsfs/releases)
Bumps [s3fs](https://github.com/fsspec/s3fs) from 2021.10.1 to 2021.11.0.
- [Release notes](https://github.com/fsspec/s3fs/releases)
- [Changelog](https://github.com/fsspec/s3fs/blob/main/release-procedure.md)

---
updated-dependencies:
- dependency-name: s3fs
  dependency-type: direct:development
  update-type: version-update:semver-minor
- dependency-name: gcsfs
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2.8.3
2021-11-08 11:13:52 +08:00
Guðmundur HeimissonandRuslan Kuprieiev c5b1a73dc5 webhdfs: expose kerberos and https options (#6936)
* Add fsspec options for webhdfs

* Add requests-kerberos dependency

* Update dvc/fs/webhdfs.py

Simplify config.pop.

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/fs/webhdfs.py

* Simplify kerb_kwargs config setting.

Co-authored-by: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-11-07 20:42:10 +02:00
Saugat Pachhai (सौगात) 044e79000d deps: bump rich to >=10.13.0
It comes with the support for no. of parameters that matches with json.dumps.
See: https://github.com/willmcgugan/rich/pull/1547 and
https://github.com/willmcgugan/rich/pull/1644.

Also added support for printing json to stderr. Before we used to use json.dumps
and rich.json.JSON based on isatty. Now we use rich.print_json on both
tty and non-tty conditions, which may avoid different behaviours in the
future.
2021-11-07 19:38:06 +02:00
karajan1001 be6128470f machine status: include two new field in the output
1. include instance_type and instance_gpu in the output.
2. rerange the field by importance.
2021-11-04 15:10:45 +02:00
Saugat Pachhai (सौगात) dd0d4f5a2c exp init: don't try tracking params if they are skipped 2021-11-03 19:21:10 +05:45
Saugat Pachhai (सौगात) 6e0dfb94e4 exp init: hide message about workspace tree on --explicit 2021-11-03 18:59:25 +05:45
Saugat Pachhai (सौगात) ce6c8a5678 exp init: fix --explicit failing with KeyError
Fixes #6917
2021-11-03 16:42:45 +05:45
dependabot[bot] b927337a30 build(deps): bump actions/checkout from 2.3.5 to 2.4.0
Bumps [actions/checkout](https://github.com/actions/checkout) from 2.3.5 to 2.4.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2.3.5...v2.4.0)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-03 12:47:22 +02:00
Saugat Pachhai (सौगात) 03b372a073 exp init: ask for command before paths 2021-11-03 05:23:40 +05:45
Saugat Pachhai (सौगात) 915a755e2f build: remove numpy restriction 2021-11-03 05:23:17 +05:45
Paweł Redzyński 22cc2a5cbc plots: fix resolve template path bug
Fix: #6854
2021-11-02 16:49:56 +02:00
Saugat Pachhai (सौगात) a67d8a44b3 exp init: make confirmation prompt at the end similar to others 2021-11-02 17:24:09 +05:45
Saugat Pachhai (सौगात) 80a52e65ad exp init: minor refactor on way it returns prompted values 2021-11-02 14:57:40 +05:45