Commit Graph
24 Commits
Author SHA1 Message Date
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
skshetry a137b2a6bc update ruff and enable flake8-logging-format and flake8-type-checking (#8894)
* update ruff and enable flake8-logging-format and flake8-type-checking check

* Update dvc/external_repo.py

* Update dvc/fs/dvc.py

* Update dvc/repo/experiments/__init__.py

* Update dvc/repo/experiments/queue/base.py

* Update pyproject.toml
2023-01-27 19:14:19 +05:45
daniele e1254c9ece Add misc type hints (#8430) 2023-01-15 17:58:34 +05:45
skshetry 90a27179eb ruff: ignore complex functions and remove dvc.utils.fs.makedirs (#8795)
We may want to prevent any new functions from getting complex.
2023-01-11 17:58:51 +05:45
skshetry 5441964fdb add ruff (#8793)
* add ruff linter

* enable few more checks
2023-01-11 15:20:38 +05:45
skshetry 6df70d388e update pre-commit; add some more hooks (#8492)
* update pre-commit; add some more hooks

* Apply suggestions from code review
2022-10-31 21:04:11 +05:45
Ruslan Kuprieiev 3fc9f323a7 fs: don't import SSHFileSystem 2022-08-18 00:23:40 +03:00
Saugat Pachhai (सौगात) 570e3a5359 dvc.fs: move fs to dvc.objects.fs
Some of the APIs/FileSystems are reimported inside dvc.fs module.
2022-05-16 13:15:19 +03:00
Saugat Pachhai (सौगात) 0d48598e4b fs.ssh: remove unused vars 2022-04-27 17:56:08 +05:45
Peter Rowlands fb16cd06e3 machine: update behavior for exp ssh executors
- log startup script finished message
- add setup_script option for runtime env setup
2022-01-13 16:34:20 +09:00
Peter Rowlands (변기호) 0d895e2661 exp: add initial ssh executor (#6833)
* executor: move local specific git initialization out of base

* machine: return kwargs for executor init instead of executor object

* executor: add initial ssh executor

* ssh-executor: disable strict host checking for dvc-machine instances

* initial tests for ssh executor

* drop GIT_SSH_COMMAND usage

* push: accept destination odb as a parameter

* separate cache initialization into executor.init_cache

* add test for ssh executor cache init

* remove dead code

* machine: fixture-ize default test scenarios

* test SSHExecutor.from_machine

* tpi: return default SSH port for constructing executors
2021-10-26 16:08:17 +09:00
b652670cbf Dependabot/pip/pylint 2.11.1 (#6843)
* build(deps): Bump pylint from 2.9.6 to 2.11.1

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.11.1.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.11.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix pylint problems

* Updation after rebase.

* system: use binary mode

* Set remote execution script encoding type locale

* Update dvc/machine/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-10-20 14:34:10 +03:00
Gao 6de3e734ed Revert "build(deps): Bump pylint from 2.9.6 to 2.11.1 (#6636)" (#6842)
This reverts commit 464620806c.
2021-10-20 12:41:00 +03:00
Gao 4b2bec89ec dvc machine: add a new command rename to it (#6633)
* `dvc machine`: add a new command `rename` to it

fix #6481
1. add new command `dvc machine rename`
2. add a unit test for the call.
3. add several functional tests for it.t push myself fix6481

* Move rename logic to tpi

* use tpi.state_mv instead of rename

* Rebase to a newer version

* Update tpi version

* Remove unused assert and try catch
2021-10-20 15:57:44 +09:00
464620806c build(deps): Bump pylint from 2.9.6 to 2.11.1 (#6636)
* build(deps): Bump pylint from 2.9.6 to 2.11.1

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.11.1.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.11.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix pylint problems
* system: use binary mode

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-10-20 10:32:44 +08:00
966dc8a1ca machine: Add a new command dvc machine status (#6649)
* machine: Add a new command `dvc machine status`

fix#6482
1. add a new command `dvc machien status`
2. add a unit test for the cli call.
3. add a unit test for the shown result.

* Update dvc/command/machine.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update dvc/command/machine.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update machine.py

* Remove sensitive fields

* Update dvc/machine/__init__.py

Co-authored-by: Casper da Costa-Luis <[email protected]>

* Update tests/func/machine/__init__.py

Co-authored-by: Casper da Costa-Luis <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use white list to limit shown fields

Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Casper da Costa-Luis <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-10-19 17:38:11 +09:00
Peter Rowlands (변기호) b8478a8b04 machine: re-add instance dir management from TPI (#6748)
* machine: re-add instance dir management from TPI

* update default constructor

* machine: lazy load Repo.machine
2021-10-14 12:40:30 +09:00
Ruslan Kuprieiev 53255c22b3 gha: tests: temporarily disable some remote tests (#6789)
* gha: tests: temporarily disable some remote tests

See #6788

* machine: upgrade tpi version

```
dvc/machine/backend/terraform.py:28:19: E1101: Instance of 'TerraformBackend' has no '_default_resource' member; maybe 'default_resource'? (no-member)
```

* tests: remotes: require --s3 for s3 tests
2021-10-12 17:26:47 +03:00
Casper da Costa-Luis 73c493500b remove tpi.base dependency (#6554)
* remove tpi.base dependency

* potential future-proof constructor
2021-09-08 06:09:52 +00:00
Peter Rowlands (변기호) f7a82f6fc0 machine: support instance size & startup script config (#6537)
* machine: install dvc via startup_script by default

* machine: support configuring instance type and startup script

* accept startup_script as a config relpath

* use pushd/popd in default install script
2021-09-07 15:07:22 +09:00
Casper da Costa-LuisandPeter Rowlands 0578643f08 machine: move to tpi[ssh] (#6528)
* machine: move to tpi[ssh]

- depends on https://github.com/iterative/tpi

* lazy import tpi

* fix lint warning

Co-authored-by: Peter Rowlands <[email protected]>
2021-09-03 14:37:24 +09:00
Peter Rowlands (변기호) a69edcc2c8 tpi: lazily import python_terraform (#6526)
* machine: lazy load tpi/python_terraform only when needed

* remove terraform from package installers

* update tpi test imports
2021-09-02 17:09:31 +03:00
Peter Rowlands (변기호) 9fc4c60749 machine: add dvc machine ssh (#6501)
* machine: add naive asyncssh shell client

* machine: add `dvc machine ssh` command

* add ssh requirements to terraform install extras

* update asyncssh comments

* fix py <3.7

* fix async mocking in py<3.8

* use mock instead of unittest.mock
2021-09-01 16:20:13 +09:00
Peter Rowlands (변기호) 4ee98299ee dvc machine: implement initial terraform support (#6462)
* tpi: add naive terraform-provider-iterative wrapper

* executor: add naive backend/terraform wrapper

* setup: add dvc[terraform] install extras

* config: add feature.executor feature flag

* fix named executor config sections

* executor: add poc add/modify/remove/init/destroy commands

* relax python-terraform requirement

* ci: install terraform extras

* tpi: support .tf.json format

* s/executor/machine

* terraform: use separate dirs per machine

* s/init/create/

* reserve local machine name

* add initial ssh skeleton

* add initial machine unit tests

* add initial machine command tests

* add tests for add/remove

* enable feature by default in test env

* close temp pemfile object so it can be used on windows

* cleanup pemfile syntax
2021-08-27 12:46:24 +03:00