Commit Graph
7098 Commits
Author SHA1 Message Date
Saugat Pachhai (सौगात) 281b97c8ef scm: remove dependency from dvc's utils in scm
This removes dependency from dvc.utils and dvc.types from DVC.
Some utils like fix_env, is_binary have been reimplemented.
Regarding fix_env and is_binary, I think eventually this
should be a responsibility of DVC itself to set.

relpath implementation in DVC also works with paths in
different drives in Windows. I have changed this to use
os.path.relpath for now.
2021-11-24 17:00:12 +02:00
Saugat Pachhai (सौगात) a282e0e48c scm: restructure scm
Makes it easier to decouple, we'd only need to rename
dvc/scm/__init__.py to dvc/scm.py.
2021-11-24 11:46:40 +05:45
Saugat Pachhai (सौगात) 48581715b6 scm: remove exp related behaviours from scm.resolve_rev 2021-11-24 11:39:39 +05:45
dependabot[bot] 13ca959e98 build(deps): Bump actions/cache from 2.1.6 to 2.1.7
Bumps [actions/cache](https://github.com/actions/cache) from 2.1.6 to 2.1.7.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.6...v2.1.7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-24 08:47:13 +05:45
Saugat Pachhai (सौगात) a0f4e12fb4 scm: unified progress reporting in a structured way
This adds a support for reporting progress from Git in a structured
format. The gitpython already provides somewhat structured progress
reporting, dulwich on the other hand only provides git-like line
messages.

As gitpython uses git binary, it already parse this line format
in a structured way. This PR makes dulwich use the same machinery
to unify progress reporting for both dulwich and gitpython python.

This does add a dependency on gitpython even on dulwich backend,
so I am iffy if we should do this.
The alternative is to translate gitpython's progress reports to
string, and just use string messages as progress reports on both
dulwich and gitpython.
2021-11-23 19:50:12 +02:00
Saugat Pachhai (सौगात) d95dd27773 scm: map internal SCMError to DVC's exception type 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) f76ef84c88 exceptions: migrate other scm exceptions to internal 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) f2f4227656 exceptions: migrate RevError 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) 7ca8fbbc82 exceptions: migrate CloneError 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) 8ec1ff44de exceptions: move AuthError and InvalidRemote errors 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) c19a11152a scm: move FileNotInRepoError and MergeConflictError to scm.exceptions 2021-11-23 19:21:51 +02:00
Saugat Pachhai (सौगात) cab60b54f8 live: import webbrowser lazily; only patch webbrowser.open 2021-11-23 22:08:21 +05:45
David de la Iglesia Castro fc7caa487f tests: Patch webbrowser.open on session level
Closes #7014
2021-11-23 22:08:21 +05:45
Saugat Pachhai (सौगात) 568abb3e28 scm: remove dependency from DvcException 2021-11-23 13:19:41 +02:00
Kwon-Young ChoiandPeter Rowlands 3e7cff3e35 exp: respect repo hardlink_lock config for the exp scm lock
- Will fix #7029

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-11-23 19:51:42 +09:00
Saugat Pachhai (सौगात) 7dc5420548 metrics/params diff: catch NoSCMError instead of a broad exception
instead of catching broad exception, NoSCMError should work here.
2021-11-23 13:21:15 +05:45
pre-commit-ci[bot] 644e9ede92 [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/ambv/black: 21.10b0 → 21.11b1](https://github.com/ambv/black/compare/21.10b0...21.11b1)
2021-11-22 21:36:19 +02:00
Paweł Redzyński a9512e8585 plots: collect: return callable instead of data
Related to: #6923, iterative/viewer#2657
2021-11-22 21:31:20 +02:00
Saugat Pachhai (सौगात) 00e53c5c32 tests: fix black 2021-11-22 17:33:12 +05:45
Saugat Pachhai (सौगात) 0aaf96bce4 exp init: show 'n to omit' on --explicit mode 2021-11-22 17:20:16 +05:45
Saugat Pachhai (सौगात) e07b99ed33 tests: increase timeout to 2 second for prompts test 2021-11-22 15:10:55 +05:45
Saugat Pachhai (सौगात) 7bc906011f tests: timeout only on functions, excluding tests fixtures
Sometimes, `tmp_dir` fixtures take longer than a second. Using `func_only`
makes it only consider duration of test function run, excluding fixtures
duration.
2021-11-22 15:10:55 +05:45
Saugat Pachhai (सौगात) b84227db89 scm: remove ignore list 2021-11-22 07:16:54 +02:00
David de la Iglesia Castro 318b269ad1 live: Call webbrowser.open.
Use `funcy.once_per_args` decorator to prevent opening the html multiple times if user closes first time.

Closes iterative/dvclive#180
2021-11-22 03:35:23 +02:00
Ruslan Kuprieiev 1176b6d875 Update setup.cfg 2021-11-22 03:31:23 +02:00
dependabot[bot] ee6539292b build(deps): Bump adlfs from 2021.9.1 to 2021.10.0
Bumps [adlfs](https://github.com/dask/adlfs) from 2021.9.1 to 2021.10.0.
- [Release notes](https://github.com/dask/adlfs/releases)
- [Changelog](https://github.com/dask/adlfs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/dask/adlfs/compare/2021.09.1...2021.10.0)

---
updated-dependencies:
- dependency-name: adlfs
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-22 03:31:23 +02:00
Saugat Pachhai (सौगात) bf313d92aa exp init: timeout prompts' test
The `exp init` tests may just keep waiting on the prompts blocking tests.
The blocking of prompts is likely a failure in tests setup or a bug
in the implementation. It's hard to find out these failures or avoid it.
If they take longer than 1 second, they most likely have failed as
most of test takes 0.1 second at max.
2021-11-22 03:27:16 +02:00
Saugat Pachhai (सौगात) 46289182d2 tests: run on Windows with Python 3.10
pygit2 wheels should be now available.
Had to skip pytest-docker for Windows + Python 3.10 due to
a dependency issue with it's dependency, see:
https://github.com/docker/docker-py/issues/2902.
2021-11-20 18:52:31 +02:00
Saugat Pachhai (सौगात) cb941a836d tests: cover exp init 2021-11-19 18:58:23 +05:45
Gustavo José de Sousa a70b3087b4 build: bump sshfs from 2021.8.1 to 2021.11.2
This version uses bumps asyncssh to a version that supports ProxyJump.
2021-11-18 23:58:57 +02:00
Ruslan Kuprieiev c9dd137426 fs: BaseFileSystem -> FileSystem
Following fsspec conventions and new object conventions.
2021-11-18 19:58:23 +02:00
Saugat Pachhai (सौगात) 93e6526907 scm: split tracking of to-be git-ignored/git-staged into SCMContext 2021-11-18 15:34:45 +02:00
dependabot[bot] 230eb7087d build: bump google-cloud-storage from 1.42.3 to 1.43.0
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.42.3 to 1.43.0.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.42.3...v1.43.0)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-18 16:36:00 +05:45
Saugat Pachhai (सौगात) c3b91e5f85 exp init: interactive mode refactor and simplification 2021-11-18 16:06:52 +05:45
Saugat Pachhai (सौगात) eac030598c ui.prompt: move prompts from exp init to ui.prompt
Some fixes along the way to make it more generic, but it still has
lots of features tied with exp init.
2021-11-18 13:22:45 +05:45
karajan1001 068b8371aa Machine: table output for machine status(#6892)
Change the output format of `dvc machine status` command to tables for a
better readability.
2021-11-18 11:22:13 +08:00
karajan1001 888b6392bb Machine:remove field name in config(#6892)
Because our machine name now is the key value of the machine config
dict. And the machine config value will been overwritten by it when the
config is loaded. We can remove the field `name` from the config.
2021-11-18 11:22:13 +08:00
karajan1001 b24a31b834 Machine: use table for list outputs(#6892)
fix: #6892

Current output of machine list is quite urgly, it is originaly used to
show configurations. In this PR, we use a new table format for it.
2021-11-18 11:22:13 +08:00
karajan1001 45553c5b30 TabularData: add a new how arg to dropna(#6892)
Borrowed this idea from pandas api. Current dropna methods will drop a
column or row if any data is missing. But it is also quite useful to
drop those columns or rows only if all of its elements are missing. This
is mostly used for pruning a table.
2021-11-18 11:22:13 +08:00
Peter Rowlands 28f57933db dulwich: flush stdin channel on asyncssh vendor write 2021-11-17 12:48:34 +02:00
Peter Rowlands 25dca0b665 dulwich: workaround github SSH server in asyncssh vendor
- allow mismatched RSA signature algo names in initial
  SSH_MSG_USERAUTH_PK_OK response
- unit test overridden asyncssh publickey auth compatibility with
  github ssh server
2021-11-17 12:48:34 +02:00
Peter Rowlands 45c4e1998d dulwich: fallback to default client config in asyncssh vendor
- vendor should use ssh-agent or default IdentityFile(s) when no SSH key
  has been provided by the user
- encrypted (fallback) keys will always be passed to ssh-agent
2021-11-17 12:48:34 +02:00
karajan1001 eea67a6396 machine:remove help text for irrelevant commands (#6940)
fix: #6940

`--global/system/project/local` help message are in some irrelevant
command `status`, `ssh`, `create`, `destroy`. These commands are more
related to running instances rather than `machine config`, level
options didn't work for them.

1. Remove `--global/system/project/local` options in `status`, `ssh`,
   `create`, `destroy`.
2. Add a test for it.
3. Correct a typo error.
2021-11-17 16:25:48 +08:00
karajan1001andDave Berenbaum f43c04f935 Make machine status name optional
fix: #6838
1. make machine status name optional.
2. show all machine status if no name provide.
3. UI changes for this cmd.
4. add a new test for it.

Co-authored-by: Dave Berenbaum <[email protected]>
2021-11-17 15:44:36 +08:00
dependabot[bot] 3486435a7f build: bump filelock from 3.3.2 to 3.4.0
Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.3.2 to 3.4.0.
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/py-filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.3.2...3.4.0)

---
updated-dependencies:
- dependency-name: filelock
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
2021-11-17 14:04:17 +08:00
Paweł Redzyński e0fd0790fc plots: special chars handling for vega plots paths
Fixes: #6894
2021-11-17 01:57:35 +02:00
dberenbaum f99d3cda57 scm: hint to autostage 2021-11-16 18:01:18 +02:00
pre-commit-ci[bot] 003461f9a2 [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/timothycrosley/isort: 5.9.3 → 5.10.1](https://github.com/timothycrosley/isort/compare/5.9.3...5.10.1)
2021-11-16 07:55:56 +05:45
Ruslan Kuprieiev 88660f6a66 dvc: use transfer in checkout/doctor/odb
All three of those use more or less the same logic for testing different
link types and maybe then using them. So now instead of having 3
implementations, we'll have just one.

Filesystem `transfer` now will not try to catch particular errno's, but
instead will just ignore any OSError and move down to the next link type
to try out, and if the last one fails - it will raise the last exception,
so if it is a common problem - we'll be able to see it.

This patch also removes `odb.confirmed_cache_type`, as it was caching
link type that is not really a cache-level thing, but rather a checkout thing,
that should only be cached for particular objects we are linking, as multiple
objects could be linked to different filesystems from the same odb (e.g.
external local output on hdd and local data on your ssd).

Next step here will be to introduce metadata diff, so that we can get rid of
temporary Link wrapper and introduce Transfer and corresponding transfer
config, that will know what links to use and could have some other things
like callbacks, so we don't continue passing dozens of kwargs along the stack.

Continuing from #6963

Fixes #6980
2021-11-15 22:50:00 +02:00
David de la Iglesia Castro 7df6090a9c params: Use ast.literal_eval for python params.
Add unit tests for `parse_py`

Closes  #6953
2021-11-15 17:52:49 +02:00