Commit Graph
1037 Commits
Author SHA1 Message Date
Peter Rowlands 67cbdf6de8 exp queue: remove dvc queue attach command 2022-07-12 21:39:49 +09:00
Peter Rowlands f12091002f exp queue: add dvc queue logs command 2022-07-12 21:39:49 +09:00
Peter Rowlands 8ab46c1549 exp: simplify shutdown behavior 2022-07-12 21:39:49 +09:00
Peter Rowlands c0db513cf0 dvc-task: bump to 0.0.11 2022-07-12 21:39:49 +09:00
Peter Rowlands 10aa833f80 dvc-task: bump to 0.0.10 2022-07-12 21:39:49 +09:00
Peter Rowlands 9f5af04963 exp: update tests for match_queue_entry refactor 2022-07-12 21:39:49 +09:00
karajan1001 3d0ee0d202 exp queue: monitor queue status (#7590)
fix: #7590
1. Add two new command `queue status` and `queue attach`
2. Add unit test for the CLI
3. Make `Running` exps can be found by `get_queue_entry_by_names`
2022-07-12 21:39:49 +09:00
karajan1001 6b8c87b58a Add more unit test and implement a new method
1. Add some more unit test for `celery_queue.kill`
2. Implement `kill` argument for `celery_queue.shutdown`
3. Use dataclass's internal methods instead of `__eq__`
2022-07-12 21:39:49 +09:00
karajan1001 12fc8c4e7d exp queue: Add two command queue start and stop (#7589)
fix: #7589

1. Add two sub-command `dvc queue start` and `dvc queue stop`
2. Add a unit test to test them
2022-07-12 21:39:49 +09:00
karajan1001andPeter Rowlands a6cb4eb72e exp queue: remove queued experiments(#7591)
fix: #7591
1. Add a new command `dvc queue`
2. Add two sub-command `dvc queue remove` and `dvc queue kill`
3. Add a unit test to test them

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2022-07-12 21:39:49 +09:00
karajan1001 4df5a1321d exp: implement internal queue.kill support(#7587)
fix: #7587
1. Add implement kill method for local queue class
2. Add a test to make sure the following job will be success after the
   original job was killed.
3. Some refactoring work on `exp remove`
2022-07-12 21:39:49 +09:00
Peter Rowlands dc4292c602 exp: implement queue shutdown 2022-07-12 21:39:49 +09:00
Peter Rowlands 56f90eb3c9 exp: add test_queue unit test helper for celery queue 2022-07-12 21:39:49 +09:00
Peter Rowlands 7113568c60 exp: refactor base.py into refs.py and exceptions.py 2022-07-12 21:39:49 +09:00
15ada9a1c3 Implement --date to exp gc
1. Implement `--time` to `exp gc`
2. add a unit test for it.
3. Rename `--time` to `--date` and change the warning msg.
4. To get a better msg for `gc` and `exp gc`

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: Dave Berenbaum <[email protected]>
2022-07-12 13:53:46 +02:00
karajan1001andPeter Rowlands 217eb13495 Add argument commit_time to gc/brancher and iter_revs
1. Add argument `commit_time` to `gc`/`brancher` and `iter_revs`
2. Add unit test for iter_revs.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2022-07-12 13:53:46 +02:00
karajan1001andDave Berenbaum 20d1f7a87d gc: add a new flag --time (#678)
fix #678
1. Add a new flag `--time` to `dvc gc`.
2. Modify the unit test of `dvc gc`
3. Fix an argument checking error and update the error msg.

Co-authored-by: Dave Berenbaum <[email protected]>
2022-07-12 13:53:46 +02:00
karajan1001 9c59689aa7 Gc: Move old unittest gc tests to pytest 2022-07-12 13:53:46 +02:00
Ruslan Kuprieiev 8b3020aaf3 deps: bump dvc-data to 0.0.20 2022-07-08 21:37:38 +03:00
Paweł Redzyński fbb98a87d1 plots: introduce flexible plots configuration to dvcfiles
Closes: #7086
2022-07-01 13:17:24 +02:00
David de la Iglesia Castro c336507e51 render: image_converter: Support slash in revision.
Fixes #7934
2022-06-27 10:53:03 +02:00
dberenbaum 2773e99afc config: remove core.jobs dead code 2022-06-22 11:36:01 +08:00
David de la Iglesia Castro 20f69a121e api: Move to sub-package.
Split into sub-modules.

Split tests.
2022-06-21 20:52:19 +03:00
David de la Iglesia Castro 66c2e8cbd8 repo.collect: Add duplicates option.
Defaults to `False`.
If `True`, multiple `outs` sharing a provided `target_path` will not be filtered.
2022-06-21 20:52:19 +03:00
Ruslan Kuprieiev 2e21c92c4f dvc: use dvc-data 0.0.8 2022-06-17 22:08:33 +03:00
David de la Iglesia CastroandPeter Rowlands 1663b97eaa api: open: Raise ValueError if rev is used in wrong mode.
Closes #7405

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2022-06-15 18:19:27 +02:00
Saugat Pachhai (सौगात) 4bc6b5699a quote node names 2022-06-15 12:43:45 +03:00
Saugat Pachhai (सौगात) af649af462 deps: bump dvc-data to 0.0.6; fix imports 2022-06-14 17:50:57 +05:45
Ruslan Kuprieiev ef689c5593 dvc: upgrade to dvc-objects 0.0.4 2022-06-13 18:03:53 +05:45
Daniele Trifirò 793f97bdd5 fix dvc-objects/data imports 2022-06-13 18:03:53 +05:45
Ruslan Kuprieiev 9ac2f49b57 Revert "fix dvc-objects/data imports"
This reverts commit 677da2b22c.
2022-06-10 12:01:07 +03:00
Daniele Trifirò 677da2b22c fix dvc-objects/data imports 2022-06-10 14:07:33 +05:45
dberenbaum 33b3afa2f5 exp init: create output dirs 2022-06-07 17:55:24 +05:45
Paweł Redzyński 20c7b0e3e6 plots: grouping: stop using dpath.util.search 2022-06-03 16:48:33 +03:00
Peter Rowlands 1cc803d12b exp: output troubleshooting link on shallow merge failure 2022-06-02 01:15:16 -07:00
David de la Iglesia Castro 3d791778de plots: Pass templates_dir to match_renderers.
We were forgetting to pass default template dir.

Fixes #7817.
2022-05-31 16:17:58 +02:00
Ruslan Kuprieiev 844f77b4f9 dvc: drop python 3.7
Also upgraded with `pyupgrade --py38-plus`

Fixes #7708
2022-05-30 16:51:35 +03:00
David de la Iglesia Castro 109d0e4e28 dag: mermaid: Use quotation marks to escape node name.
Fixes #7731
2022-05-24 18:27:58 +02:00
Saugat Pachhai (सौगात) 75fa487406 dvc: remove unused utils and redundant tests
The tests should be in dvc-objects repo npw, and will be migrated soon.
The nanotime and diskcache are specified in dvc-objects, so we don't need
those here.
2022-05-20 19:17:00 +05:45
Saugat Pachhai (सौगात) 9a2feaa4ed dvc: remove dvc.data, fix imports 2022-05-20 15:13:50 +05:45
Saugat Pachhai (सौगात) 7bc98e3fb3 tests: remove objects related tests 2022-05-20 14:02:08 +05:45
Saugat Pachhai (सौगात) 37921d618d dvc: remove dvc.objects, fix imports 2022-05-20 14:02:08 +05:45
Saugat Pachhai (सौगात) 95c6cea36d tests: restructure some tests for objects/data 2022-05-19 18:38:21 +05:45
Saugat Pachhai (सौगात) 2057c471bb data.checkout: remove dependency on DVC; prompts and DvcException 2022-05-19 15:11:27 +03:00
Saugat Pachhai (सौगात) f23d31af64 fs/data: move repo related utils to dvc
This moves get_fs_config/get_fs_cloud and ODBManager to DVC
as they require DVC related utilities, and we don't need them
strictly in dvc-objects/dvc-data.
This also adds support to ReferenceHashFile to load from
any filesystem, using it's module name and filename.
Now we can get rid of DVCFileSystem dependency in
ReferenceHashFile.
This is a temporary solution, we really need a registry.
(I am not quite sure if this works in pyinstaller mode,
but I think it should, it's how fsspec does it if I understand
correctly.)
2022-05-18 19:39:09 +03:00
Saugat Pachhai (सौगात) 32cd6c2c03 dvc.objects: use stub ignore in dvc.objects/dvc.data
This commit:
* defines a `Ignore` protocol in `dvc.objects`, that expects a `find` method.
  This will be used in dvc-data and dvc-objects instead of depending on dvc.ignore.
  dvcignore is expected to respect this protocol, which it does.
* renames `dvcignore` kwarg to `ignore` in `dvc-data` and `dvc-objects`.
* teaches dvc.cli that `IgnoreInCollectedError` is a known exception
   and is logged correctly and adds a test for the behaviour.

I am not happy that we need to hardcode `.dvcignore` in dvc-data
but I don't see any easy way, since we always try to avoid
staging if the tree contains .dvcignore file, even if .dvcignore
is not passed.
2022-05-18 12:22:28 +03:00
Saugat Pachhai (सौगात) 3872f79dad objects.fs: do not use format_link, DvcException and fix_env from DVC
* Introduce ConfigError and AuthError in dvc.objects.fs and raise it
  instead of dvc.config.ConfigError.
* Provide hints and links by catching those errors in dvc.cli.main,
  i.e decouple UI messages from filesystems for AuthError and
  ConfigError.
* Copy flatten/unflatten utils to dvc.objects.fs.utils.
2022-05-17 18:23:29 +03:00
Saugat Pachhai (सौगात) 48176b80f6 fs: do not use DvcException in filesystems
* Refactor check_requires, move UI messages to dvc.cli
* Refactor link related error
2022-05-17 16:48:38 +03:00
Saugat Pachhai (सौगात) 8c9a9e9a1f objects.fs: move schemes to objects.fs 2022-05-17 17:02:14 +05:45
Saugat Pachhai (सौगात) 36605eea4c objects.hash: rename HashedStreamReader to HashStreamFile 2022-05-17 13:25:38 +03:00