Commit Graph
22 Commits
Author SHA1 Message Date
Rishabh SharmaandRishabh Sharma 8d8939fb67 DVC_STAGE: Added environment variable to track DVC stage name (#10357)
DVC_STAGE_NAME: Added environment variable to track DVC stage name

Fixes #10355

Co-authored-by: Rishabh Sharma <[email protected]>
2024-03-20 13:28:02 -07:00
skshetry 0100c14d6a import-db: support importing a table (#10219)
* import-db: support importing a table

* unhide the import-db command
2024-01-05 09:26:07 +05:45
skshetry cbc111b519 remove flake8/bandit (#10082)
ruff already runs most of the flake8/bandit checks
2023-11-09 21:26:37 +05:45
skshetry 17f5508f49 daemon: add logging support, tests and generalize popen based runs for future (#10033)
* use popen to run updater/analytics in detached mode

subprocess does not provide a way to avoid a zombie process. Not to confuse with orphaned processes, which will be `wait`-ed by `pid 1`.

It might not work for long-running processes, but I don't think zombie processes are going to be a problem for us, as the parent process immediately exits after spawning a child process.

Even in case of `queue` based experiment runner, parent will exit before than the child process. So I don't think it's a requirement here. But it does feel wrong to leak the processes, so I have to sleep on that.

---

This PR:

1. makes `daemonized` process inherit the logging verbosity, so if the parent process used `-vv`, the daemonized process will run the same way.
2. Add `DVC_DAEMON_LOGFILE` envvar, that the daemon will log to the file. By default, the verbosity is the same as running parent processes. Without `DVC_DAEMON_LOGFILE`, the output will be redirected to `/dev/null`.
3. Adds debug/trace for updater/analytics.

I hope by doing this, we can make our life easier on debugging and make daemon more transparent/observable.

* add tests

* fixes

* use kwargs

* pass all os.environ with some modified
2023-10-19 15:14:41 +05:45
David de la Iglesia Castro 25313f5d82 exp: Drop checkpoints (#9271)
exp: Drop `checkpoints`.

Closes #9221
2023-05-23 11:32:18 +05:45
Dave Berenbaum 7f6890ff25 Live metrics config (#9433)
* exp: live metrics config token and offline

* refactor live metrics config

* use get_studio_config

* revert to using env vars for studio config

* update dvc-studio-client to support live metrics config changes

* upgrade dvc-studio-client

* don't change environ when posting live metrics events
2023-05-17 09:21:01 +00:00
skshetry 004f49744e tests: isolate dvc config and directories (#9450)
* tests: isolate dvc config and directories

by introducing envvars to change default values. This lets pytest
handle the deletion of tmpdirs, instead of filling /var/tmp with
hundreds of thousands of directories. Also isolates test run from
faulty global/system config files.

* move to dvc/env and sort
2023-05-15 18:16:02 +05:45
skshetry dd3deb6cbd add DVC_STUDIO_URL envvar support (#9340)
add DVC_STUDIO_URL envvar
2023-04-18 09:56:17 +05:45
skshetry a91993616f logger: add DVC_SHOW_TRACEBACK envvar to show traceback on exception (#9310)
This will come useful in debugging, where you don't have to add --verbose just to
see where it failed or go through verbose logs (which might be useful when working
with UI issues as you want to see what the user sees).

Ideally this should be handled in main when we exit with error instead of using
logger.{warning,error,exception} to log exception as the intention is to only log
last exception that makes it exit. But since we don't have that capability and
we don't abuse logging.{warning,error,exception} to log and suppress exceptions, this is a good
enough solution for now without complicating `dvc.cli.main()`.
2023-04-06 09:56:19 +05:45
David de la Iglesia Castro 8124396218 exp: Expose baseline and name via run_env.
Required by DVCLive for the experiments onboarding and live metrics features.

Related (not sure if closes): https://github.com/iterative/dvc/issues/6668
2022-11-28 13:13:04 +01:00
David de la Iglesia Castro 1de983f0de Remove live section support. 2022-11-07 12:45:32 +01:00
Ruslan Kuprieiev 33c6a9b19b analytics: introduce DVC_NO_ANALYTICS (#6453)
Replacing DVC_ANALYTICS. This is following our other env conventions and
will not have effect if set to false/no/etc, and will only work if set
to true/yes/y/etc. Will disable analytics even if it is explicitly
enabled in our configs.

Kudos @skshetry
2021-08-17 15:16:40 +03:00
Ruslan Kuprieiev c2bf0dc713 analytics: add DVC_ANALYTICS env var (#6448) 2021-08-17 01:48:38 +03:00
6dd435eefe Push running checkpoint to remote (#6332)
* Push running checkpoint to remote

Fix #6182

* Update dvc/repo/experiments/executor/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Get the branch name

* Finish this PR

1. move env name to dvc/env.py.
2. add some tests for it.

* Update after finally commit

* Some problems found in the review

1. change the behavior of self remote
2. do not use string DVC_EXP_AUTO_PUSH
3. downgrade the logger level in auto push
4. use full branch ref

* Update dvc/repo/experiments/executor/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* add a remote check before auto push

* Split current one env into two

* Rename DVC_EXP_CHECKPOINT_PUSH to DVC_EXP_AUTO_PUSH

* value error

* Remove getenv_bool switch to env2bool

* Better on handling Dulwich exceptions

* Name changing removing prints

* Update dvc/env.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Some changes in review

* Add validation to repo URL

* Add new type of SCMError

* Update dvc/repo/experiments/executor/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Use spy tests call counts and args

* Update dvc/scm/git/backend/base.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/scm/git/backend/dulwich.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove validate_git_repo to validate_git_remote

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-08-12 15:17:08 +08:00
Saugat Pachhai f2998630a3 env: track DVC_IGNORE_ISATTY as a env that we support (#6202) 2021-06-21 05:35:39 +00:00
Paweł Redzyński 31e4a3325d live integration: rename report to html (#5366) 2021-02-02 13:44:38 +01:00
Paweł Redzyński 6a9c0c5ce5 live/checkpoints: test continue (#5300)
* live: test resuming

* initial test, not working

* experiments: introduce DVCLIVE_RESUME env variable on resume
2021-01-24 22:06:37 +02:00
Paweł Redzyński 386d0c463a live: add report switch (#5223) 2021-01-07 13:52:21 +02:00
Paweł RedzyńskiandRuslan Kuprieiev 862ab8de62 live logs initial (#4902)
* plots: allow dir plots

* first scenario

* separate dvclive tests

* rename index to step

* another test

* refactor

* its ok to have step in data

* sort plots

* introduce --logs

* add logs to run commands

* variable name refactoring

* fix logs option

* fixup

* cmd: run: fix unit tests

* plots: modify: fix tests

* improve dir handling

* fixup

* dvclive: logs summary

* api: introduce logs summarization method

* handle no repo case

* fix granular dir plots

* api: dvclive: simplify

* dvclive: roll back logs flag

* cmd: introduce logs

* api: adjust to logs

* test: dvclive fix

* rename logs to dvclive

* live: command: add uninitialized

* outs: dvclive type

* dvclive: support providing config via evn variables

* dvclive: treat as metrics

* dvclive integration: add support for summary on/off

* dvc integration: add temporary test for configuration export

* remove requirements

* cmd: introduce logs

* rename logs to dvclive

* cmd: introduce logs

* remove logs leftowvers

* dvclive: env: adjust to chekpoints env variables

* command: live: remove unnecessary help

* fix linter suggestions

* type hinting

* review refactor

* replace dvclive with live

* stage: loading: move live loading to separate method

* stage: serialize: allow only single live output

* live: add show/diff

* stage: serialize: roll back single live enforcing

* fixup

* rebase fixups

* output: live: force single instance

* live: command: add unit tests

* Update dvc/output/base.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/stage/__init__.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/stage/serialize.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* review v2

* live: fix summary tracking by experiments

* naming fixups

* api: live: add tests

* tests: api: separate live tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-04 11:29:41 +02:00
Peter Rowlands (변기호) 07676b2531 checkpoints: set DVC_ROOT environment variable (#4877)
* checkpoints: set DVC_ROOT environment variable

* env: add DVC_ROOT, DVC_CHECKPOINT
2020-11-12 16:26:07 +09:00
Tymoteusz Jankowski 3e163f1934 Rename and move DVC_PAGER_ENV_NAME to env.py file 2019-12-01 19:13:56 +01:00
Ruslan Kuprieiev 7c540da5c4 daemon: don't spawn recursively
Fixes #1803

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-07 20:12:23 +03:00