Commit Graph
63 Commits
Author SHA1 Message Date
Saugat Pachhai (सौगात) af649af462 deps: bump dvc-data to 0.0.6; fix imports 2022-06-14 17:50:57 +05:45
Daniele Trifirò 793f97bdd5 fix dvc-objects/data imports 2022-06-13 18:03:53 +05:45
Ruslan Kuprieiev 9ac2f49b57 Revert "fix dvc-objects/data imports"
This reverts commit 677da2b22c.
2022-06-10 12:01:07 +03:00
Daniele Trifirò 677da2b22c fix dvc-objects/data imports 2022-06-10 14:07:33 +05:45
Saugat Pachhai (सौगात) 37921d618d dvc: remove dvc.objects, fix imports 2022-05-20 14:02:08 +05:45
Saugat Pachhai (सौगात) 14c9348f93 dvc.objects: move hash_info, state and meta into dvc.objects 2022-05-11 17:35:48 +03:00
karajan1001 c81d841fd7 exp pull/push: better result handling in exp sharing(#7448)
fix: #7448
Current we skip on those experiments which already existed on both sides, but
do not exclude them from the result list. And if we have diverged experiment,
pulling/pushing them will stop the current progress and raise an exception, but
didn't give any information on the condition of other experiments.

1. Change the exp pull and push to match the new API in `scmrepo`.
2. Handle sync status after `exp pull/push` finished.
3. Bump scmrepo to 0.0.19

Co-authored-by:
2022-04-28 19:56:15 +08:00
David de la Iglesia Castro ddd6c97f71 render: Extract to dvc_render
Create and use separate package for rendering logic.
Closes #6944
2022-03-28 19:34:05 +02:00
Paweł Redzyński 7bc14a5ddb plots: make templates generation explicit
Fixes: #6389
2022-02-01 15:09:25 +01:00
Paweł Redzyński a3144482ba render: vega: data processing outside of renderers
Fixes: #6943
2022-01-07 10:39:12 +01:00
karajan1001andPeter Rowlands b6a2402d27 exp remove: refactor exp utils (#7196)
fix: #7196

1. Refactor exp_ref_by_name in `dvc/repo/experiments/utils.py`.

In previous the exp_ref_by_name walk throught all of the references to
match only one exp name. This makes multi exp name seeking to be a job
of `O(mn)` complexity, Walking though the exp ref and return all of the
exp ref with greatly reduce this complexity to `O(n)`

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2022-01-06 12:20:29 +08:00
Alexander Schepanovski 9ee428ef17 Fix memory leak in Index.build_graph()
`@memoize()` memory lives as long as decorated function lives, which is
permanent in this case. Since `self` is referenced as a memory key no
`Index` will ever be freed.

Changed to use `@cached_property` whose cache lives as long as the
parent instance.
2022-01-03 13:41:10 +02:00
Ruslan Kuprieiev 35d364a527 repo: use fs instead of scm
SCM is used for both reading and writing operations, but in the former case -
as filesystem and in the latter - as actual scm/git. So we should distinguish
those and use fs directly, which follows the overall philosophy of being able
to use Repo on any fs.

Keeping `rev=` option for `Repo` because it is a very convenient way to get
repo instance from local path but for particular branch/tag/etc.

Pre-requisite for git/dvc/repofs refactorings.
2021-12-31 02:44:21 +02:00
Saugat Pachhai (सौगात) 93e6526907 scm: split tracking of to-be git-ignored/git-staged into SCMContext 2021-11-18 15:34:45 +02:00
Batuhan Taskaya 9aec8f1926 dvc: get rid of path_info
Intermediate PR in removing path_info from our code base.

It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.

For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).

Fixes #6774
2021-11-10 00:52:48 +02:00
GaoandPeter Rowlands c07592131a exp run: add experiment name check. (#6848)
* Add experiment name check.

1. Add experiment name check (https://git-scm.com/docs/git-check-ref-format)
2. Add duplicate exp name check.
3. Add some unit test for it.

* Ban slash / in dvc exp names

* Use dulwich backend for ref name checking

* Some bug fix

* Update dvc/repo/experiments/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Update dvc/repo/experiments/__init__.py

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Some review changes

* Make some funtion more reusable.

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-10-25 20:16:37 +09:00
Nikita Kodenko 696c704632 tests: replace several unittest.mock usages with mocker (#6544)
* tests: several unittest.mock usages replaced with mocker

* tests: fixes in tests/unit/repo/test_reproduce.py

* tests: rolled back WiP tests
2021-09-17 18:22:34 +03:00
4cc06339c8 Clean up remotes's exps (#6471)
* Clean up remotes's exps

fix #6006
1. add a new argument `--git-remote` to `dvc exp remove`
2. support remote a special remote exp
3. add tests for it
4. fix  #6421
5. add a test for #6421.
6. extract some functions from the `dvc pull`, `dvc push`, `dvc remove` to utils.
7. add a tests for this new util function `resolve_exp_ref`
8. add `__init__.py` to some test package for the pytest fail

Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Dave Berenbaum <[email protected]>
2021-09-05 10:06:07 +08:00
Paweł Redzyński ab08db5065 plots: introduce support for images (#6431)
* plots: support images initial

* plots: data: unify PlotsData

* plots: data conversion refactor

* plots: renderers: abstract functionality

* plots: uncommenting tests

* plots: self implement grouping

* plots: rename group

* plots: extract rendering tests

* plots: move rendering tests to unit

* plots: add image rendering tests

* plots: dont use dpath to find vega plot

* plots: test find_vega

* plots: test render function

* plots: move rendering out of repo package

* plots: refactoring

* plots: self review refactor
2021-08-26 20:00:31 +03:00
Saugat Pachhai 6c33fbb617 introduce index (#6300)
* introduce index

* migrate tests to use index

* migrate checkout/diff to use index.outs

* test for index

* fix typings

* add few more tests
2021-08-05 14:23:40 +03:00
Peter Rowlands (변기호) 114a07e7a4 objects: use object IDs and references instead of naive objs in status/transfer (#6360)
* objects.status: use object IDs instead of naive objects

* objects.transfer: use object IDs instead of naive objects

* objects: add ReferenceHashFile

* odb: add ReferenceObjectDB

* repo: collect used object IDs instead of naive objs

* support basic/naive fs serialization

* objects.stage: stage file objects in addition to trees

* objects: remove save(), update transfer()

* output/dep: update for osave removal

* path_info: support both url and path attributes

* status/gc: use object IDs instead of naive objs

* diff: update staging usage

* commit: update staging usage

* tests: clean staging when needed

* tests: update for save/transfer/stage changes

* dvcfs: don't allow config serialization (repofs should be used instead)

* utils: return mtime as str, size as int

* tests: update unit tests for save/stage/transfer changes

* objects: use Repo.open when needed in ReferenceHashFile

* fix ls test staging

* objects.stage: don't skip hash check when loading from state

- individual hashes will still be loaded from state if they are still
  valid

* ReferenceHashFile: serialize repo fs paths relative to repo root

* preserve state when staging

* objects.tree: skip hash computation in digest() when we have already
loaded hash from state

* always check file existence in referencehashfile

* update repofs test since stage no longer allows nonexistent files

* odb: only protect on check() if hash was verified

* staging: use unique memfs namespace for staging per dest ODB

* tests: clean full staging namespace

* refodb: preserve repofs cache dir

* repodependency: stage imports to local ODB and strict check circular
imports

* fetch: fetch staging last

* staging: make dest odb mandatory and preserve state

* refodb: cache nonlocal fs's

* fix rebase errors

* repofs: support full instantiation from fs config

* repofs: preserve subrepo factory for erepos

* tests: clarify repofs dirty file behavior
2021-08-04 13:55:52 +03:00
Paweł Redzyński 67b42ebe4f plots: cleanup data extraction (#6355)
* plots: cleanup data extraction

* fixup
2021-07-23 10:28:36 +09:00
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
Peter Rowlands (변기호) 1d3524f144 odb: treat external repo dependencies as objects (#6109)
* objects: add ExternalRepoFile object type

* RepoDependency: implement get_obj (use ExternalRepoFile)

* drop distinction between used external and used objects

* update tests

* remove unneeded fetch from ExternalRepoFile

* wrap erepo errors as an object error

* update cloud/remote functions for new used_objs behavior

* fix update() rev/locked behavior

* update import tests

* remove ExternalRepoFile

* move get_obj and get_used_objs implementations back into repo dependency

* return objects plus associated ODB (remote) in get_used_objs

* odb: add tmp_dir field and support creating Remote from an odb

* return dict mapping odb -> objects in get_used_objs

* use dummy git odb for git imports

* fetch: support pulling from specific odb

* update cloud funcs for get_used_objs usage

* update tests for used_objs behavior

* use fetch in dep.download

* catch circular imports from local fs repos

* add tests for chained and circular imports

* move tmp_dir into odb config

* update return docstring for repo.used_objs
2021-06-17 14:41:33 +03:00
eb46c03c6c output/remote: get rid of NamedCache (#6008)
* objects: make HashInfo & HashFile hashable

* stage/output: collect objects (HashFiles) instead of NamedCache

* remote/data_cloud: use objs instead of NamedCache

* repo: use objects instead of NamedCache

* rerun black after rebase

* remove NamedCache usage from run-cache

* drop NamedCache

* update HashInfo.__hash__

* update unit tests

* revert remote_name change

* start cleaning up test failures

* fix tree obj expansion

* update missing cache logging

* fix external dict handling

* update naive odb.gc

* update state save behavior for pull

* update test_data_cloud

* update test_remote

* fix gc with multiple repos

* objects.tree: support filtering w/preserved hash_info

* fix stage cache restore

* drop fs.path_info from HashFile.__hash__

* objects: add optional name attribute

* revert missing cache logging name/suffix changes

* revert name changes in remote tests

* review fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update dvc/remote/base.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-06-02 22:19:58 +03:00
Peter Rowlands (변기호) 60ea24410e exp: use exp baseline for DVC commands that compare to HEAD during exp run (#5818)
* exp: add util method for getting baseline HEAD during run

* exp: use baseline for dvc commands which compare to HEAD during exp run

* add test for new util method

* update plots diff test mock
2021-04-19 14:42:58 +09:00
Ruslan Kuprieiev 8f568ab197 state: use diskcache (#5505)
* state: only load when needed

* state: don't use separate load/dump

* checkout: group link removal logic together

* state: use diskcache.Cache

* staging: use pbar and save to state

* state: don't pass the whole repo to __init__

* staging: explicitly pass state object

* fs: get rid of odb property

* state: don't close the connection to cache

* repo: close: close state
2021-02-23 20:51:34 +02:00
Ruslan Kuprieiev e0842ce01e dvc: rename cache -> odb (#5481) 2021-02-16 22:49:25 +02:00
Batuhan Taskaya e5f91f5c7f cache: don't initalize cloud cache until its needed (#5264) 2021-01-13 17:01:06 +02:00
Ruslan Kuprieiev 64dfaffd7c [WIP] repro: simplify the logic for frozen stages (#5114)
Fixes #5082
2020-12-16 14:28:00 +02:00
Saugat Pachhai 189e88f1db Run graph checks on collect/find_outs_by_path (#5035)
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.

So, that's why, the `collect` also does a graph check for this
reason.

Fixes #5027
Fixes #4010
2020-12-10 12:20:45 +02:00
Saugat Pachhai bc1d10532a Move Repo.collect and collect_granular to Repo.stage (#4993) 2020-11-30 14:28:16 +05:45
Ruslan Kuprieiev 2a6c4291ce dag: support output as target (#4908)
Fixes #4904
2020-11-20 06:57:12 +02:00
Ruslan Kuprieiev 060fc4fbb7 state: use granularly (#4676)
* state: add simple reentrance

* state: pass Repo instead of LocalCache

* state: use granularly

Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
2020-10-08 08:36:14 +03:00
Alexander Schepanovski 67ddb7b41f Make repo lock reentrable and provide a context manager (#4416) 2020-09-08 17:40:34 +03:00
Peter Rowlands (변기호) d6bb4341dd plots: add dvc plots -e/--experiment option (#4488)
* plots: add -e/--experiment option for plots diff

* experiments: add method to get baseline for a given experiment

* plots: default to diff against baseline instead of workspace/head for experiments

* use exact match for experiments RE since name-rev may return relative
commits
2020-08-29 00:57:28 +03:00
Ruslan Kuprieiev 41e6c22e7d dvc: move Dvc/RepoTree to dvc/tree (#4427)
This is where it really belongs.
2020-08-19 22:30:55 +03:00
Saugat Pachhai b773ba73cf tests: add .dvcignore on the tests expectations (#4396)
Since #4229, we have started creating .dvcignore file after `dvc init`
This commit only adds that in assertions
2020-08-15 21:24:29 +05:45
Saugat Pachhai 0b1d4f10d6 RepoTree: add support for subrepo traversal (#4381)
Right now, no actual "functionality" is surfaced as
`subrepos` is always set to False by default for now.
2020-08-15 17:27:00 +03:00
Ruslan Kuprieiev b8a42939ad config: use repo.tree (#4190)
Fixes #4188
2020-07-12 06:45:04 +03:00
Ruslan Kuprieiev 4181c0ec4a run: hide and forbid --file for multistage files (#4086)
Fixes #4084
2020-06-23 03:46:59 +03:00
Peter Rowlands (변기호) f516dd252a trees: handle tree.walk errors the same way as os.walk (#4059)
* trees: BaseTree.walk should handle errors like os.walk

* use onerror callback if exceptions should be logged or raised

* repo.ls: use onerror for tree.walk
2020-06-17 13:25:14 +03:00
Peter Rowlands (변기호) a0c229cf9b remote: continue remote/cache/tree refactoring (#4041)
* remote: move constants into trees

* remote: replace Remote() Cache() helper functions

* output: update for remote changes

* cache/data_cloud: update for remote changes

* dependency: update for remote changes

* stage: update for remote changes

* remote: move remaining non-index methods into tree

* tests: update remote unit tests

* remote: LocalCache needs to lock remote index, not cache index

* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH

* tests: repo tree open needs state context

* remote: make Remote.gc a class method

* operates on obj.tree, so that GC can be done with both remote and
  cache instances

* bug fixes

* tests: update test_data_cloud

* bug fixes

* tests: update func tests
2020-06-15 16:22:55 +03:00
Ruslan Kuprieiev 87276faef2 RepoTree: work with dirty repos (#4005)
Currently `RepoTree.open` will throw FileNotFoundError if you have a
file that is tracked by dvc, but which has no md5 or a cache for which
is missing, even though you have the file itself in your workspace.

This corresponds very nicely to the `--no-commit` or `--no-exec`
scenario, where you might not want to commit your artifacts to cache
just yet.

Fixes #3974
2020-06-11 09:50:27 +03:00
Ruslan Kuprieiev 9ee62ef8f0 dvc: rename "working tree" to "workspace" (#3914)
Fixes #3892
2020-05-31 17:39:04 +03:00
Ruslan Kuprieiev 0bb49d2549 dvc: rename dvc lock -> dvc freeze (#3915)
Fixes #3901
2020-05-31 03:38:21 +03:00
Saugat PachhaiandRuslan Kuprieiev 2fecc66cb9 addressing: allow stage name addressing without ":" (#3842)
* addressing: allow stage name addressing without colon

* repo: reorganize imports

* refactor/reorganize collect_granular

* add tests for stage addressing

* fix ds and cc issues

* fix unpack error in test

* tests: stage string representations

* exception: add error message

Also remove `cause` when throwing NoOutputOrStage, as
CheckoutErrorSuggestGit will start chaining all the messages
in cause of the exception.

* rename NoOutputOrStage NoOutputOrStageError

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-26 03:51:51 +03:00
Peter Rowlands (변기호) 75e795c491 remote: support saving RepoTree objects directly to cache (#3825)
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache

* RepoTree: support filtering out dvcfiles during walk()

* RepoTree: support recursive isdvc()

* remote: support computing checksums for tree objects

* RepoTree: pull dir contents on walk() if in `fetch` mode

* tests: add func test for DVC dir pull on RepoTree.walk()

* DvcTree: support granular open()

* RepoTree: add get_file_checksum()

* remote: revert checksum changes

* remote: add tree support for cache.save()

* tests: test saving directly to cache from RepoTree

* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree

* RepoTree: state context should be entered by open()/walk() caller

* RepoTree: only pull dir contents when cache has changed

* tree: update tree tests

* fix ds warnings, update comments

* fix windows tests
2020-05-21 06:31:28 +03:00
Ruslan Kuprieiev 09136f2378 plots: add plot markers to DVC files (#3807)
* run: add --plot/--plot-no-cache

Part of #3409

* plot: move show/diff to API

* plots: show marked plots by default

Involves some refactoring to make plots comply with the rest of the code
base.

* dvc.yaml: use top-level metrics and plots

* dvc.yaml: add support for plot template

* plots: add support for templates in dvc.yaml

* dvc.yaml: allow metrics/plots to have persist/cache

* plots: render single page by-default and introduce --show-json

* plot: use "working tree" instead of "workspace"

Just to sync this with the rest of the codebase. We should consider
renaming it to "workspace" everywhere, but for now I opt to not touch
it to not break something that depends on it.

* plots: rename -f|--file to -o|--out

Better corresponds to other commands like `import`.

* plots: use RepoTree and brancher instead of api

* stage: get rid of OutputParams

* tests: plots: add simple diff test

* plots: use `--targets` instead of `--datafile`

Makes it comply with `dvc metrics/params` CLI options.

* tests: unit: plots: add --show-json test

* stage: cleanup exception for format error

Based on feedback from the users.

* lockfile: use relpath in corruption error
2020-05-20 04:48:53 +03:00
Peter Rowlands (변기호) afbb3a0add repo: Support streaming and pulling files on RepoTree/DvcTree.open() (#3810)
* DvcTree: support stream or fetch on open()

* tests: test DvcTree dir cache support for walk()

* RepoTree: finish implementation of BaseTree functions

* repo: use RepoTree/DvcTree.open() for repo.open_by_relpath

* repo: only use RepoTree when needed for now

* fix docstring

* RepoTree: only use DvcTree.open for DVC outs
2020-05-18 15:16:11 +03:00