Commit Graph
10 Commits
Author SHA1 Message Date
skshetry 48f1f52c1b refactor: dvc.daemon (#10045)
* minor refactor to dvc.daemon

* fix tests

* run daemon only inside a subprocess
2023-10-30 11:09:55 +05:45
skshetry 17f5508f49 daemon: add logging support, tests and generalize popen based runs for future (#10033)
* use popen to run updater/analytics in detached mode

subprocess does not provide a way to avoid a zombie process. Not to confuse with orphaned processes, which will be `wait`-ed by `pid 1`.

It might not work for long-running processes, but I don't think zombie processes are going to be a problem for us, as the parent process immediately exits after spawning a child process.

Even in case of `queue` based experiment runner, parent will exit before than the child process. So I don't think it's a requirement here. But it does feel wrong to leak the processes, so I have to sleep on that.

---

This PR:

1. makes `daemonized` process inherit the logging verbosity, so if the parent process used `-vv`, the daemonized process will run the same way.
2. Add `DVC_DAEMON_LOGFILE` envvar, that the daemon will log to the file. By default, the verbosity is the same as running parent processes. Without `DVC_DAEMON_LOGFILE`, the output will be redirected to `/dev/null`.
3. Adds debug/trace for updater/analytics.

I hope by doing this, we can make our life easier on debugging and make daemon more transparent/observable.

* add tests

* fixes

* use kwargs

* pass all os.environ with some modified
2023-10-19 15:14:41 +05:45
skshetry 5441964fdb add ruff (#8793)
* add ruff linter

* enable few more checks
2023-01-11 15:20:38 +05:45
Nikita KodenkoandSaugat Pachhai e731e35e6c tests: moved few unittest tests to pytest (#6339)
* tests: moved few unittest tests to pytest

* tests: unittest.mock -> pytest-mock in few tests

* tests: test_prompt: unittest.mock -> pytest-mock

* Update tests/unit/test_daemon.py

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
2021-07-22 18:06:54 +03:00
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
Saugat Pachhai a172067c7d fix errors raised by pylint (#6081)
astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.

Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).

Example errors that we were getting because of this:

```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
2021-05-31 05:46:18 +00:00
Saugat Pachhai ad306d7bf3 Enable most of the pylint checks (#4111) 2020-06-25 18:10:14 +03:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev 7c540da5c4 daemon: don't spawn recursively
Fixes #1803

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-07 20:12:23 +03:00
Ivan Shcheklein 75421f0f21 migrate to pytest 2019-03-17 14:49:49 -07:00