Compare commits

..
20 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 7504b68c12 dvc: bump to 1.0.0a7 2020-06-02 03:34:04 +03:00
Ruslan Kuprieiev 86f3457dec dvc: remove dvc metrics add/remove (#3928)
We will be re-introducing `metrics modify` in the future and
will likely revisit `add/remove` too, so it is better to get
rid of these for now and introduce later with a clear scneario
in mind.

Fixes #3849
Fixes #3907
2020-06-02 00:17:03 +03:00
Saugat Pachhai 219662597b refactor: rename utils.stage to utils.yaml (#3923) 2020-06-01 16:49:46 +03:00
Alexander Schepanovski 536966f9d1 Fix modifying plot (#3926)
- no error when modifying plot with no props yet
- do not turn into non-plot if all props are unset
2020-06-01 16:19:11 +03:00
Saugat Pachhai 076eadceba bugfix: rerun stage when command is changed (#3922)
Previously, the stage would not rerun if the deps/outs are cached
even if the cmd has changed.

Versions affected: >0.93.0

Fixes #3918
2020-06-01 14:04:31 +05:45
Ruslan Kuprieiev d1e08b6157 dvc: replace pipeline with dag (#3905)
`pipeline list` is no more, it had some very questionable applications
(none to be precise) and was duplicating old `pipeline show` functionality.
We might introduce a different command to assist in listing (e.g. for
shell completion) in the future.

New `dvc dag` replaces `dvc pipeline show` and by-default will show
all pipelines in the project in ASCII (so there is no more --ascii)
and also `--dot` support, just like before. When given a target,
`dvc dag` (in any mode) will show DAG that leads to the target,
skipping the descendants, unless `--full` is specified.
2020-06-01 02:35:36 +03:00
Saugat Pachhai edfa1741db implement removing stage (#3881)
This should allow us to be able to remove a stage from a dvcfile,
(if it's a single stage file, it removes it), and make us able to
delete the overwrite the stage on run as well.

This commit:
* changes `run` `--overwrite-dvcfile` flag to `--overwrite`
* removes prompt in `run` for overwrite and fails hard
* changes `remove`: now it by default removes stage entry and unprotect outs
* `remove` has `--outs` to remove outputs instead of unprotecting them
2020-05-31 17:41:19 +03:00
Ruslan Kuprieiev 9ee62ef8f0 dvc: rename "working tree" to "workspace" (#3914)
Fixes #3892
2020-05-31 17:39:04 +03:00
Ruslan Kuprieiev 0bb49d2549 dvc: rename dvc lock -> dvc freeze (#3915)
Fixes #3901
2020-05-31 03:38:21 +03:00
Alexander SchepanovskiandRuslan Kuprieiev 516d82e411 Add plots props and plots modify params (#3911)
* Add plots props and `plots modify` params

* Update dvc/command/plots.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Update dvc/repo/plots/show.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-05-30 14:37:11 +03:00
Saugat Pachhai ec312d8601 use fork of mockssh-server (#3910) 2020-05-30 08:31:59 +05:45
Ruslan Kuprieiev f504245f35 pre-commit: don't run dvc in parallel (#3912)
* pre-commit: use local hooks

* pre-commit: don't run dvc in parallel

By default pre-commit will try to run hooks in parallel, which results
in some dvc instances failing to acquire repo locks.

Fixes #3751
2020-05-29 21:20:46 +03:00
Saugat Pachhai bb50fb8664 use higher level API selectors instead of select (#3899) 2020-05-29 09:25:07 +05:45
Jorge Orpinel 777a9f80ae Update bug report template (#3903)
done directly from GH
2020-05-29 08:09:35 +05:45
Ruslan Kuprieiev 02a52d038a tests: make ssh_server session scoped (#3898)
Fixes #3888
2020-05-28 16:09:58 +03:00
Peter Rowlands (변기호) 7d0a06248d dvcignore: use correct root .dvcignore for subrepos (#3895)
* ignore: dvcignore root is distinct from tree_root

* for subrepos, dvcignore root should be the subrepo's .dvcignore, not
  host repo's .dvcignore

* tests: add test for subrepo dvcignore behavior
2020-05-28 20:52:48 +09:00
Ruslan Kuprieiev 2d8c6043ca plots: replace --show-json with --show-vega (#3891)
Fixes #3890
2020-05-28 14:42:41 +03:00
Nicholas A. Del Grosso 3f996048e8 Bugfix: _credentials_location (#3896)
_credentials_location  to credentials_location to removed attributeerror
2020-05-28 14:41:34 +03:00
Peter Rowlands (변기호) 8d9435e8ee brancher: only yield git revisions that contain repo_root (#3883)
* tests: add test for metrics show -T with pre-existing tags

* brancher: ignore git revisions that do not contain repo

* tests: update test to include cmd

* CleanTree: tree_root for subrepos should be the repo root, not scm root

* only force subrepo root for git trees

* fix windows shell quotes
2020-05-28 12:26:29 +09:00
Saugat Pachhai 889a17e2fc modify issue template (#3887)
* modify issue template

* change bug issue template
* auto add labels
* use emojis
* add link to Discord chat
2020-05-28 08:21:02 +05:45
88 changed files with 1365 additions and 1523 deletions
+14 -5
View File
@@ -1,10 +1,19 @@
---
name: Bug report
about: Create a report to help us improve
name: "\U0001F41B Bug Report"
about: Create a bug report to help us improve DVC
labels: bug, triage
---
## Bug Report
**Please provide information about your setup**
DVC version(i.e. `dvc --version`), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))
### Please provide information about your setup
**Output of `dvc version`:**
```console
$ dvc version
```
**Additional Information (if any):**
If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
+5
View File
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: "🤗 Need help?"
url: https://dvc.org/chat
about: If you have a question, ask us on Discord. Please join with this invite 👉
+2 -2
View File
@@ -1,6 +1,6 @@
---
name: Feature request
name: "\U0001F680 Feature Request"
about: Suggest an idea for this project
labels: feature request, triage
---
+26 -7
View File
@@ -27,18 +27,37 @@ repos:
repo: https://github.com/lovesegfault/beautysh
rev: master
- hooks:
- id: dvc-pre-commit
language_version: python3
- args:
- git-hook
- pre-commit
entry: dvc
id: dvc-pre-commit
language: system
name: DVC pre-commit
stages:
- commit
- id: dvc-pre-push
language_version: python3
verbose: true
require_serial: true
- args:
- git-hook
- pre-push
entry: dvc
id: dvc-pre-push
language: system
name: DVC pre-push
stages:
- push
require_serial: true
- always_run: true
args:
- git-hook
- post-checkout
entry: dvc
id: dvc-post-checkout
language_version: python3
language: system
minimum_pre_commit_version: 2.2.0
name: DVC post-checkout
stages:
- post-checkout
repo: https://github.com/iterative/dvc
rev: master
require_serial: true
repo: local
+3
View File
@@ -6,6 +6,7 @@
language: python
language_version: python3
name: DVC pre-commit
require_serial: true
stages:
- commit
verbose: true
@@ -17,6 +18,7 @@
language: python
language_version: python3
name: DVC pre-push
require_serial: true
stages:
- push
- always_run: true
@@ -29,5 +31,6 @@
language_version: python3
minimum_pre_commit_version: 2.2.0
name: DVC post-checkout
require_serial: true
stages:
- post-checkout
+4 -4
View File
@@ -10,9 +10,11 @@ from .command import (
commit,
config,
daemon,
dag,
data_sync,
destroy,
diff,
freeze,
gc,
get,
get_url,
@@ -21,12 +23,10 @@ from .command import (
imp_url,
init,
install,
lock,
ls,
metrics,
move,
params,
pipeline,
plots,
remote,
remove,
@@ -66,8 +66,8 @@ COMMANDS = [
install,
root,
ls,
lock,
pipeline,
freeze,
dag,
daemon,
commit,
diff,
+115
View File
@@ -0,0 +1,115 @@
import argparse
import logging
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
def _show_ascii(G):
from dvc.repo.graph import get_pipelines
from dvc.dagascii import draw
pipelines = get_pipelines(G)
ret = []
for pipeline in pipelines:
ret.append(draw(pipeline.nodes, pipeline.edges))
return "\n".join(ret)
def _show_dot(G):
import io
from networkx.drawing.nx_pydot import write_dot
dot_file = io.StringIO()
write_dot(G, dot_file)
return dot_file.getvalue()
def _build(G, target=None, full=False):
import networkx as nx
from dvc.repo.graph import get_pipeline, get_pipelines
if target:
H = get_pipeline(get_pipelines(G), target)
if not full:
descendants = nx.descendants(G, target)
descendants.add(target)
H.remove_nodes_from(set(G.nodes()) - descendants)
else:
H = G
def _relabel(stage):
return stage.addressing
return nx.relabel_nodes(H, _relabel, copy=False)
class CmdDAG(CmdBase):
def run(self):
try:
target = None
if self.args.target:
stages = self.repo.collect(self.args.target)
if len(stages) > 1:
logger.error(
f"'{self.args.target}' contains more than one stage "
"{stages}, please specify one stage"
)
return 1
target = stages[0]
G = _build(self.repo.graph, target=target, full=self.args.full,)
if self.args.dot:
logger.info(_show_dot(G))
else:
from dvc.utils.pager import pager
pager(_show_ascii(G))
return 0
except DvcException:
msg = "failed to show"
if self.args.target:
msg += f"a pipeline for '{target}'"
else:
msg += "pipelines"
logger.exception(msg)
return 1
def add_parser(subparsers, parent_parser):
DAG_HELP = "Visualize DVC project DAG."
dag_parser = subparsers.add_parser(
"dag",
parents=[parent_parser],
description=append_doc_link(DAG_HELP, "dag"),
help=DAG_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
dag_parser.add_argument(
"--dot",
action="store_true",
default=False,
help="Print DAG with .dot format.",
)
dag_parser.add_argument(
"--full",
action="store_true",
default=False,
help=(
"Show full DAG that the target belongs too, instead of "
"showing DAG consisting only of ancestors."
),
)
dag_parser.add_argument(
"target",
nargs="?",
help="Stage or output to show pipeline for. Optional. "
"(Finds all stages in the workspace by default.)",
)
dag_parser.set_defaults(func=CmdDAG)
+57
View File
@@ -0,0 +1,57 @@
import argparse
import logging
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdFreezeBase(CmdBase):
def _run(self, func, name):
ret = 0
for target in self.args.targets:
try:
func(target)
except DvcException:
logger.exception(f"failed to {name} '{target}'")
ret = 1
return ret
class CmdFreeze(CmdFreezeBase):
def run(self):
return self._run(self.repo.freeze, "freeze")
class CmdUnfreeze(CmdFreezeBase):
def run(self):
return self._run(self.repo.unfreeze, "unfreeze")
def add_parser(subparsers, parent_parser):
FREEZE_HELP = "Freeze DVC-files."
freeze_parser = subparsers.add_parser(
"freeze",
parents=[parent_parser],
description=append_doc_link(FREEZE_HELP, "freeze"),
help=FREEZE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
freeze_parser.add_argument(
"targets", nargs="+", help="DVC-files to freeze."
)
freeze_parser.set_defaults(func=CmdFreeze)
UNFREEZE_HELP = "Unfreeze DVC-files."
unfreeze_parser = subparsers.add_parser(
"unfreeze",
parents=[parent_parser],
description=append_doc_link(UNFREEZE_HELP, "unfreeze"),
help=UNFREEZE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
unfreeze_parser.add_argument(
"targets", nargs="+", help="DVC-files to unfreeze."
)
unfreeze_parser.set_defaults(func=CmdUnfreeze)
+2 -2
View File
@@ -21,7 +21,7 @@ class CmdGC(CmdBase):
msg = "This will remove all cache except items used in "
msg += "the working tree"
msg += "the workspace"
if self.args.all_commits:
msg += " and all git commits"
elif self.args.all_branches and self.args.all_tags:
@@ -77,7 +77,7 @@ def add_parser(subparsers, parent_parser):
"--workspace",
action="store_true",
default=False,
help="Keep data files used in the current workspace (working tree).",
help="Keep data files used in the current workspace.",
)
gc_parser.add_argument(
"-a",
-59
View File
@@ -1,59 +0,0 @@
import argparse
import logging
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdLockBase(CmdBase):
def _run(self, unlock):
for target in self.args.targets:
try:
self.repo.lock_stage(target, unlock=unlock)
except DvcException:
logger.exception(
"failed to {}lock '{}'".format(
"un" if unlock else "", target
)
)
return 1
return 0
class CmdLock(CmdLockBase):
def run(self):
return self._run(False)
class CmdUnlock(CmdLockBase):
def run(self):
return self._run(True)
def add_parser(subparsers, parent_parser):
LOCK_HELP = "Lock DVC-files."
lock_parser = subparsers.add_parser(
"lock",
parents=[parent_parser],
description=append_doc_link(LOCK_HELP, "lock"),
help=LOCK_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
lock_parser.add_argument("targets", nargs="+", help="DVC-files to lock.")
lock_parser.set_defaults(func=CmdLock)
UNLOCK_HELP = "Unlock DVC-files."
unlock_parser = subparsers.add_parser(
"unlock",
parents=[parent_parser],
description=append_doc_link(UNLOCK_HELP, "unlock"),
help=UNLOCK_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
unlock_parser.add_argument(
"targets", nargs="+", help="DVC-files to unlock."
)
unlock_parser.set_defaults(func=CmdUnlock)
+1 -47
View File
@@ -63,30 +63,6 @@ class CmdMetricsShow(CmdBase):
return 0
class CmdMetricsAdd(CmdBase):
def run(self):
try:
self.repo.metrics.add(self.args.path)
except DvcException:
msg = f"failed to add metric file '{self.args.path}'"
logger.exception(msg)
return 1
return 0
class CmdMetricsRemove(CmdBase):
def run(self):
try:
self.repo.metrics.remove(self.args.path)
except DvcException:
msg = f"failed to remove metric file '{self.args.path}'"
logger.exception(msg)
return 1
return 0
def _show_diff(diff, markdown=False, no_path=False, old=False):
from collections import OrderedDict
@@ -145,7 +121,7 @@ class CmdMetricsDiff(CmdBase):
def add_parser(subparsers, parent_parser):
METRICS_HELP = "Commands to add, manage, collect, and display metrics."
METRICS_HELP = "Commands to display and compare metrics."
metrics_parser = subparsers.add_parser(
"metrics",
@@ -162,17 +138,6 @@ def add_parser(subparsers, parent_parser):
fix_subparsers(metrics_subparsers)
METRICS_ADD_HELP = "Mark a DVC-tracked file as a metric."
metrics_add_parser = metrics_subparsers.add_parser(
"add",
parents=[parent_parser],
description=append_doc_link(METRICS_ADD_HELP, "metrics/add"),
help=METRICS_ADD_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
metrics_add_parser.add_argument("path", help="Path to a metric file.")
metrics_add_parser.set_defaults(func=CmdMetricsAdd)
METRICS_SHOW_HELP = "Print metrics, with optional formatting."
metrics_show_parser = metrics_subparsers.add_parser(
"show",
@@ -291,14 +256,3 @@ def add_parser(subparsers, parent_parser):
help="Show old metric value.",
)
metrics_diff_parser.set_defaults(func=CmdMetricsDiff)
METRICS_REMOVE_HELP = "Remove metric mark on a DVC-tracked file."
metrics_remove_parser = metrics_subparsers.add_parser(
"remove",
parents=[parent_parser],
description=append_doc_link(METRICS_REMOVE_HELP, "metrics/remove"),
help=METRICS_REMOVE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
metrics_remove_parser.add_argument("path", help="Path to a metric file.")
metrics_remove_parser.set_defaults(func=CmdMetricsRemove)
-265
View File
@@ -1,265 +0,0 @@
import argparse
import logging
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
class CmdPipelineShow(CmdBase):
def _show(self, target, commands, outs, locked):
import networkx
from dvc.utils import parse_target
path, name = parse_target(target)
stage = self.repo.get_stage(path, name)
G = self.repo.graph
stages = networkx.dfs_postorder_nodes(G, stage)
if locked:
stages = [s for s in stages if s.locked]
for stage in stages:
if commands:
if stage.cmd is None:
continue
logger.info(stage.cmd)
elif outs:
for out in stage.outs:
logger.info(str(out))
else:
logger.info(stage.addressing)
@staticmethod
def _build_output_graph(G, target_stage):
import networkx
from itertools import product
nodes = {str(out) for out in target_stage.outs}
edges = []
for from_stage, to_stage in networkx.edge_dfs(G, target_stage):
from_stage_deps = {dep.path_info.parts for dep in from_stage.deps}
to_outs = {
to_out
for to_out in to_stage.outs
if to_out.path_info.parts in from_stage_deps
}
from_outs = {
from_out
for from_out in from_stage.outs
if str(from_out) in nodes
}
nodes |= {str(to_out) for to_out in to_outs}
for from_out, to_out in product(from_outs, to_outs):
edges.append((str(from_out), str(to_out)))
return nodes, edges
def _build_graph(self, target, commands=False, outs=False):
import networkx
from dvc.repo.graph import get_pipeline
from dvc.utils import parse_target
path, name = parse_target(target)
target_stage = self.repo.get_stage(path, name)
G = get_pipeline(self.repo.pipelines, target_stage)
nodes = set()
for stage in networkx.dfs_preorder_nodes(G, target_stage):
if commands:
if stage.cmd is None:
continue
nodes.add(stage.cmd)
elif not outs:
nodes.add(stage.addressing)
edges = []
for from_stage, to_stage in networkx.edge_dfs(G, target_stage):
if commands:
if to_stage.cmd is None:
continue
edges.append((from_stage.cmd, to_stage.cmd))
elif not outs:
edges.append((from_stage.addressing, to_stage.addressing))
if outs:
nodes, edges = self._build_output_graph(G, target_stage)
return list(nodes), edges, networkx.is_tree(G)
def _show_ascii(self, target, commands, outs):
from dvc.dagascii import draw
nodes, edges, _ = self._build_graph(target, commands, outs)
if not nodes:
return
draw(nodes, edges)
def _show_dependencies_tree(self, target, commands, outs):
from treelib import Tree
nodes, edges, is_tree = self._build_graph(target, commands, outs)
if not nodes:
return
if not is_tree:
raise DvcException(
"DAG is not a tree, can not print it in tree-structure way, "
"please use --ascii instead"
)
tree = Tree()
tree.create_node(target, target) # Root node
observe_list = [target]
while len(observe_list) > 0:
current_root = observe_list[0]
for edge in edges:
if edge[0] == current_root:
tree.create_node(edge[1], edge[1], parent=current_root)
observe_list.append(edge[1])
observe_list.pop(0)
tree.show()
def _write_dot(self, target, commands, outs):
import io
import networkx
from networkx.drawing.nx_pydot import write_dot
_, edges, _ = self._build_graph(target, commands, outs)
edges = [edge[::-1] for edge in edges]
simple_g = networkx.DiGraph()
simple_g.add_edges_from(edges)
dot_file = io.StringIO()
write_dot(simple_g, dot_file)
logger.info(dot_file.getvalue())
def run(self):
from dvc.dvcfile import DVC_FILE
if not self.args.targets:
self.args.targets = [DVC_FILE]
for target in self.args.targets:
try:
if self.args.ascii:
self._show_ascii(
target, self.args.commands, self.args.outs
)
elif self.args.dot:
self._write_dot(target, self.args.commands, self.args.outs)
elif self.args.tree:
self._show_dependencies_tree(
target, self.args.commands, self.args.outs
)
else:
self._show(
target,
self.args.commands,
self.args.outs,
self.args.locked,
)
except DvcException:
msg = f"failed to show pipeline for '{target}'"
logger.exception(msg)
return 1
return 0
class CmdPipelineList(CmdBase):
def run(self):
pipelines = self.repo.pipelines
for pipeline in pipelines:
for stage in pipeline:
logger.info(stage.addressing)
if len(pipeline) != 0:
logger.info("=" * 80)
logger.info("{} pipelines total".format(len(pipelines)))
return 0
def add_parser(subparsers, parent_parser):
PIPELINE_HELP = "Manage pipelines."
pipeline_parser = subparsers.add_parser(
"pipeline",
parents=[parent_parser],
description=append_doc_link(PIPELINE_HELP, "pipeline"),
help=PIPELINE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
pipeline_subparsers = pipeline_parser.add_subparsers(
dest="cmd",
help="Use `dvc pipeline CMD --help` for command-specific help.",
)
fix_subparsers(pipeline_subparsers)
PIPELINE_LIST_HELP = "List connected groups of stages (pipelines)."
pipeline_list_parser = pipeline_subparsers.add_parser(
"list",
parents=[parent_parser],
description=append_doc_link(PIPELINE_LIST_HELP, "pipeline/list"),
help=PIPELINE_LIST_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
pipeline_list_parser.set_defaults(func=CmdPipelineList)
PIPELINE_SHOW_HELP = "Show stages in a pipeline."
pipeline_show_parser = pipeline_subparsers.add_parser(
"show",
parents=[parent_parser],
description=append_doc_link(PIPELINE_SHOW_HELP, "pipeline/show"),
help=PIPELINE_SHOW_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
pipeline_show_group = pipeline_show_parser.add_mutually_exclusive_group()
pipeline_show_group.add_argument(
"-c",
"--commands",
action="store_true",
default=False,
help="Print commands instead of paths to DVC-files.",
)
pipeline_show_group.add_argument(
"-o",
"--outs",
action="store_true",
default=False,
help="Print output files instead of paths to DVC-files.",
)
pipeline_show_parser.add_argument(
"-l",
"--locked",
action="store_true",
default=False,
help="Print locked DVC stages",
)
pipeline_show_parser.add_argument(
"--ascii",
action="store_true",
default=False,
help="Output DAG as ASCII.",
)
pipeline_show_parser.add_argument(
"--dot",
action="store_true",
default=False,
help="Print DAG with .dot format.",
)
pipeline_show_parser.add_argument(
"--tree",
action="store_true",
default=False,
help="Output DAG as Dependencies Tree.",
)
pipeline_show_parser.add_argument(
"targets",
nargs="*",
help="DVC-files to show pipeline for. Optional. "
"(Finds all DVC-files in the workspace by default.)",
)
pipeline_show_parser.set_defaults(func=CmdPipelineShow)
+80 -86
View File
@@ -4,6 +4,7 @@ import os
from dvc.command.base import CmdBase, append_doc_link, fix_subparsers
from dvc.exceptions import DvcException
from dvc.schema import PLOT_PROPS
from dvc.utils import format_link
logger = logging.getLogger(__name__)
@@ -32,23 +33,28 @@ class CmdPlots(CmdBase):
def _func(self, *args, **kwargs):
raise NotImplementedError
def _props(self):
# Pass only props specified by user, to not shadow ones from plot def
props = {p: getattr(self.args, p) for p in PLOT_PROPS}
return {k: v for k, v in props.items() if v is not None}
def run(self):
if self.args.show_vega:
if not self.args.targets:
logger.error("please specify a target for `--show-vega`")
return 1
if len(self.args.targets) > 1:
logger.error(
"you can only specify one target for `--show-vega`"
)
return 1
try:
plots = self._func(
targets=self.args.targets,
template=self.args.template,
x_field=self.args.x,
y_field=self.args.y,
csv_header=not self.args.no_csv_header,
title=self.args.title,
x_title=self.args.xlab,
y_title=self.args.ylab,
)
plots = self._func(targets=self.args.targets, props=self._props())
if self.args.show_json:
import json
logger.info(json.dumps(plots))
if self.args.show_vega:
target = self.args.targets[0]
logger.info(plots[target])
return 0
divs = [
@@ -82,6 +88,14 @@ class CmdPlotsDiff(CmdPlots):
return self.repo.plots.diff(*args, revs=self.args.revisions, **kwargs)
class CmdPlotsModify(CmdPlots):
def run(self):
self.repo.plots.modify(
self.args.target, props=self._props(), unset=self.args.unset,
)
return 0
def add_parser(subparsers, parent_parser):
PLOTS_HELP = (
"Generating plots for metrics stored in structured files "
@@ -110,51 +124,13 @@ def add_parser(subparsers, parent_parser):
help=SHOW_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_show_parser.add_argument(
"-t",
"--template",
nargs="?",
default=None,
help=(
"Special JSON or HTML schema file to inject with the data. "
"See {}".format(
format_link("https://man.dvc.org/plots#plot-templates")
)
),
)
plots_show_parser.add_argument(
"-o", "--out", default=None, help="Destination path to save plots to.",
)
plots_show_parser.add_argument(
"-x", default=None, help="Field name for x axis."
)
plots_show_parser.add_argument(
"-y", default=None, help="Field name for y axis."
)
plots_show_parser.add_argument(
"--no-csv-header",
action="store_true",
default=False,
help="Required when CSV or TSV datafile does not have a header.",
)
plots_show_parser.add_argument(
"--show-json",
action="store_true",
default=False,
help="Show output in JSON format.",
)
plots_show_parser.add_argument("--title", default=None, help="Plot title.")
plots_show_parser.add_argument(
"--xlab", default=None, help="X axis title."
)
plots_show_parser.add_argument(
"--ylab", default=None, help="Y axis title."
)
plots_show_parser.add_argument(
"targets",
nargs="*",
help="Metrics files to visualize. Shows all plots by default.",
help="Plots files to visualize. Shows all plots by default.",
)
_add_props_arguments(plots_show_parser)
_add_output_arguments(plots_show_parser)
plots_show_parser.set_defaults(func=CmdPlotsShow)
PLOTS_DIFF_HELP = (
@@ -169,6 +145,37 @@ def add_parser(subparsers, parent_parser):
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_diff_parser.add_argument(
"--targets",
nargs="*",
help="Plots file to visualize. Shows all plots by default.",
)
plots_diff_parser.add_argument(
"revisions", nargs="*", default=None, help="Git commits to plot from",
)
_add_props_arguments(plots_diff_parser)
_add_output_arguments(plots_diff_parser)
plots_diff_parser.set_defaults(func=CmdPlotsDiff)
PLOTS_MODIFY_HELP = "Modify plot props associated with a target file."
plots_modify_parser = plots_subparsers.add_parser(
"modify",
parents=[parent_parser],
description=append_doc_link(PLOTS_MODIFY_HELP, "plots/modify"),
help=PLOTS_MODIFY_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
plots_modify_parser.add_argument(
"target", help="Plot file to set props to.",
)
_add_props_arguments(plots_modify_parser)
plots_modify_parser.add_argument(
"--unset", nargs="*", help="Props to unset.",
)
plots_modify_parser.set_defaults(func=CmdPlotsModify)
def _add_props_arguments(parser):
parser.add_argument(
"-t",
"--template",
nargs="?",
@@ -180,40 +187,27 @@ def add_parser(subparsers, parent_parser):
)
),
)
plots_diff_parser.add_argument(
"--targets",
nargs="*",
help="Metrics file to visualize. Shows all plots by default.",
)
plots_diff_parser.add_argument(
"-o", "--out", default=None, help="Destination path to save plots to.",
)
plots_diff_parser.add_argument(
"-x", default=None, help="Field name for x axis."
)
plots_diff_parser.add_argument(
"-y", default=None, help="Field name for y axis."
)
plots_diff_parser.add_argument(
parser.add_argument("-x", default=None, help="Field name for x axis.")
parser.add_argument("-y", default=None, help="Field name for y axis.")
parser.add_argument(
"--no-csv-header",
action="store_true",
default=False,
action="store_false",
dest="csv_header",
default=None, # Use default None to distinguish when it's not used
help="Provided CSV ot TSV datafile does not have a header.",
)
plots_diff_parser.add_argument(
"--show-json",
parser.add_argument("--title", default=None, help="Plot title.")
parser.add_argument("--xlab", default=None, help="X axis title.")
parser.add_argument("--ylab", default=None, help="Y axis title.")
def _add_output_arguments(parser):
parser.add_argument(
"-o", "--out", default=None, help="Destination path to save plots to.",
)
parser.add_argument(
"--show-vega",
action="store_true",
default=False,
help="Show output in JSON format.",
help="Show output in VEGA format.",
)
plots_diff_parser.add_argument("--title", default=None, help="Plot title.")
plots_diff_parser.add_argument(
"--xlab", default=None, help="X axis title."
)
plots_diff_parser.add_argument(
"--ylab", default=None, help="Y axis title."
)
plots_diff_parser.add_argument(
"revisions", nargs="*", default=None, help="Git commits to plot from",
)
plots_diff_parser.set_defaults(func=CmdPlotsDiff)
+4 -41
View File
@@ -1,7 +1,6 @@
import argparse
import logging
import dvc.prompt as prompt
from dvc.command.base import CmdBase, append_doc_link
from dvc.exceptions import DvcException
@@ -9,30 +8,10 @@ logger = logging.getLogger(__name__)
class CmdRemove(CmdBase):
def _is_dvc_only(self, target):
if not self.args.purge:
return True
if self.args.force:
return False
msg = "Are you sure you want to remove '{}' with its outputs?".format(
target
)
if prompt.confirm(msg):
return False
raise DvcException(
"Cannot purge without a confirmation from the user."
" Use `-f` to force."
)
def run(self):
for target in self.args.targets:
try:
dvc_only = self._is_dvc_only(target)
self.repo.remove(target, dvc_only=dvc_only)
self.repo.remove(target, outs=self.args.outs)
except DvcException:
logger.exception(f"failed to remove '{target}'")
return 1
@@ -40,7 +19,7 @@ class CmdRemove(CmdBase):
def add_parser(subparsers, parent_parser):
REMOVE_HELP = "Remove DVC-tracked files or directories."
REMOVE_HELP = "Remove stage entry and unprotect outputs"
remove_parser = subparsers.add_parser(
"remove",
parents=[parent_parser],
@@ -48,27 +27,11 @@ def add_parser(subparsers, parent_parser):
help=REMOVE_HELP,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
remove_parser_group = remove_parser.add_mutually_exclusive_group()
remove_parser_group.add_argument(
"-o",
remove_parser.add_argument(
"--outs",
action="store_true",
default=True,
help="Only remove DVC-file outputs. (Default)",
)
remove_parser_group.add_argument(
"-p",
"--purge",
action="store_true",
default=False,
help="Remove DVC-file and all its outputs.",
)
remove_parser.add_argument(
"-f",
"--force",
action="store_true",
default=False,
help="Force purge.",
help="Remove outputs as well.",
)
remove_parser.add_argument(
"targets", nargs="+", help="DVC-files to remove."
+3 -3
View File
@@ -45,7 +45,7 @@ class CmdRun(CmdBase):
fname=self.args.file,
wdir=self.args.wdir,
no_exec=self.args.no_exec,
overwrite=self.args.overwrite_dvcfile,
overwrite=self.args.overwrite,
run_cache=not self.args.no_run_cache,
no_commit=self.args.no_commit,
outs_persist=self.args.outs_persist,
@@ -177,10 +177,10 @@ def add_parser(subparsers, parent_parser):
help="Only create stage file without actually running it.",
)
run_parser.add_argument(
"--overwrite-dvcfile",
"--overwrite",
action="store_true",
default=False,
help="Overwrite existing DVC-file without asking for confirmation.",
help="Overwrite existing stage",
)
run_parser.add_argument(
"--no-run-cache",
+2 -3
View File
@@ -96,10 +96,9 @@ class AsciiCanvas:
def draw(self):
"""Draws ASCII canvas on the screen."""
pager = find_pager()
lines = map("".join, self.canvas)
joined_lines = os.linesep.join(lines)
pager(joined_lines)
return joined_lines
def point(self, x, y, char):
"""Create a point on ASCII canvas.
@@ -316,4 +315,4 @@ def draw(vertexes, edges):
int(round(x - minx)) + 1, int(round(y - miny)) + 1, vertex.data
)
canvas.draw()
return canvas.draw()
+45 -14
View File
@@ -17,11 +17,7 @@ from dvc.stage.exceptions import (
from dvc.stage.loader import SingleStageLoader, StageLoader
from dvc.utils import relpath
from dvc.utils.collections import apply_diff
from dvc.utils.stage import (
dump_stage_file,
parse_stage,
parse_stage_for_update,
)
from dvc.utils.yaml import dump_yaml, parse_yaml, parse_yaml_for_update
logger = logging.getLogger(__name__)
@@ -101,7 +97,7 @@ class FileMixin:
with self.repo.tree.open(self.path) as fd:
stage_text = fd.read()
d = parse_stage(stage_text, self.path)
d = parse_yaml(stage_text, self.path)
self.validate(d, self.relpath)
return d, stage_text
@@ -149,8 +145,8 @@ class SingleStageFile(FileMixin):
logger.debug(
"Saving information to '{file}'.".format(file=relpath(self.path))
)
dump_stage_file(self.path, serialize.to_single_stage_file(stage))
self.repo.scm.track_file(relpath(self.path))
dump_yaml(self.path, serialize.to_single_stage_file(stage))
self.repo.scm.track_file(self.relpath)
def remove_with_prompt(self, force=False):
if not self.exists():
@@ -165,6 +161,9 @@ class SingleStageFile(FileMixin):
self.remove()
def remove_stage(self, stage):
self.remove()
class PipelineFile(FileMixin):
"""Abstraction for pipelines file, .yaml + .lock combined."""
@@ -195,7 +194,7 @@ class PipelineFile(FileMixin):
data = {}
if self.exists():
with open(self.path) as fd:
data = parse_stage_for_update(fd.read(), self.path)
data = parse_yaml_for_update(fd.read(), self.path)
else:
logger.info("Creating '%s'", self.relpath)
open(self.path, "w+").close()
@@ -211,8 +210,8 @@ class PipelineFile(FileMixin):
logger.info(
"Adding stage '%s' to '%s'", stage.name, self.relpath,
)
dump_stage_file(self.path, data)
self.repo.scm.track_file(relpath(self.path))
dump_yaml(self.path, data)
self.repo.scm.track_file(self.relpath)
@property
def stage(self):
@@ -234,6 +233,22 @@ class PipelineFile(FileMixin):
super().remove()
self._lockfile.remove()
def remove_stage(self, stage):
self._lockfile.remove_stage(stage)
if not self.exists():
return
with open(self.path, "r") as f:
d = parse_yaml_for_update(f.read(), self.path)
self.validate(d, self.path)
if stage.name not in d.get("stages", {}):
return
logger.debug("Removing '%s' from '%s'", stage.name, self.path)
del d["stages"][stage.name]
dump_yaml(self.path, d)
class Lockfile(FileMixin):
from dvc.schema import COMPILED_LOCKFILE_SCHEMA as SCHEMA
@@ -242,7 +257,7 @@ class Lockfile(FileMixin):
if not self.exists():
return {}
with self.repo.tree.open(self.path) as fd:
data = parse_stage(fd.read(), self.path)
data = parse_yaml(fd.read(), self.path)
try:
self.validate(data, fname=self.relpath)
except StageFileFormatError:
@@ -260,17 +275,33 @@ class Lockfile(FileMixin):
open(self.path, "w+").close()
else:
with self.repo.tree.open(self.path, "r") as fd:
data = parse_stage_for_update(fd.read(), self.path)
data = parse_yaml_for_update(fd.read(), self.path)
modified = data.get(stage.name, {}) != stage_data.get(
stage.name, {}
)
if modified:
logger.info("Updating lock file '%s'", self.relpath)
data.update(stage_data)
dump_stage_file(self.path, data)
dump_yaml(self.path, data)
if modified:
self.repo.scm.track_file(self.relpath)
def remove_stage(self, stage):
if not self.exists():
return
with open(self.path) as f:
d = parse_yaml_for_update(f.read(), self.path)
self.validate(d, self.path)
if stage.name not in d:
return
logger.debug("Removing '%s' from '%s'", stage.name, self.path)
del d[stage.name]
dump_yaml(self.path, d)
class Dvcfile:
def __new__(cls, repo, path, **kwargs):
+4 -5
View File
@@ -173,17 +173,16 @@ class BadMetricError(DvcException):
class NoMetricsError(DvcException):
def __init__(self):
super().__init__(
"no metric files in this repository. "
"Use `dvc metrics add` to add a metric file to track."
"no metric files in this repository. Use `-m/-M` options for "
"`dvc run` to mark stage outputs as metrics."
)
class StageFileCorruptedError(DvcException):
class YAMLFileCorruptedError(DvcException):
def __init__(self, path):
path = relpath(path)
super().__init__(
"unable to read DVC-file: {} "
"YAML file structure is corrupted".format(path)
f"unable to read: '{path}', YAML file structure is corrupted"
)
+15 -7
View File
@@ -92,13 +92,14 @@ class DvcIgnoreRepo(DvcIgnore):
class DvcIgnoreFilter:
def __init__(self, tree):
def __init__(self, tree, root_dir):
self.tree = tree
self.root_dir = root_dir
self.ignores = {
DvcIgnoreDirs([".git", ".hg", ".dvc"]),
DvcIgnoreRepo(),
}
for root, dirs, files in self.tree.walk(self.tree.tree_root):
for root, dirs, files in self.tree.walk(self.root_dir):
self._update(root)
dirs[:], files[:] = self(root, dirs, files)
@@ -115,16 +116,20 @@ class DvcIgnoreFilter:
class CleanTree(BaseTree):
def __init__(self, tree):
def __init__(self, tree, tree_root=None):
self.tree = tree
if tree_root:
self._tree_root = tree_root
else:
self._tree_root = self.tree.tree_root
@cached_property
def dvcignore(self):
return DvcIgnoreFilter(self.tree)
return DvcIgnoreFilter(self.tree, self.tree_root)
@property
def tree_root(self):
return self.tree.tree_root
return self._tree_root
def open(self, path, mode="r", encoding="utf-8"):
if self.isfile(path):
@@ -146,8 +151,11 @@ class CleanTree(BaseTree):
)
def _valid_dirname(self, path):
dirname, basename = os.path.split(os.path.normpath(path))
dirs, _ = self.dvcignore(os.path.abspath(dirname), [basename], [])
path = os.path.abspath(path)
if path == self.tree_root:
return True
dirname, basename = os.path.split(path)
dirs, _ = self.dvcignore(dirname, [basename], [])
if dirs:
return True
return False
+6
View File
@@ -56,6 +56,12 @@ class BaseOutput:
PARAM_METRIC_XPATH = "xpath"
PARAM_PLOT = "plot"
PARAM_PLOT_TEMPLATE = "template"
PARAM_PLOT_X = "x"
PARAM_PLOT_Y = "y"
PARAM_PLOT_XLAB = "xlab"
PARAM_PLOT_YLAB = "ylab"
PARAM_PLOT_TITLE = "title"
PARAM_PLOT_CSV_HEADER = "csv_header"
PARAM_PERSIST = "persist"
METRIC_SCHEMA = Any(
+1 -1
View File
@@ -340,7 +340,7 @@ class GDriveRemote(BaseRemote):
"use the `gdrive_user_credentials_file` config\n"
" option if you use multiple GDrive remotes with "
"different email accounts.\n\nDetails".format(
item_id, self.path_info, self._credentials_location
item_id, self.path_info, self.credentials_location
)
) from exc
raise
+7 -3
View File
@@ -237,11 +237,15 @@ class SSHConnection:
or channel.recv_ready()
or channel.recv_stderr_ready()
):
import select
import selectors
selector = selectors.DefaultSelector()
selector.register(stdout.channel, selectors.EVENT_READ)
got_chunk = False
readq, _, _ = select.select([stdout.channel], [], [], self.timeout)
for c in readq:
events = selector.select(self.timeout)
for key, _ in events:
c = key.fileobj
if c.recv_ready():
stdout_chunks.append(stdout.channel.recv(len(c.in_buffer)))
got_chunk = True
+10 -3
View File
@@ -17,6 +17,7 @@ from dvc.exceptions import (
from dvc.ignore import CleanTree
from dvc.path_info import PathInfo
from dvc.repo.tree import RepoTree
from dvc.scm.tree import is_working_tree
from dvc.utils.fs import path_isin
from ..stage.exceptions import StageFileDoesNotExistError, StageNotFound
@@ -47,7 +48,7 @@ class Repo:
from dvc.repo.add import add
from dvc.repo.remove import remove
from dvc.repo.ls import ls
from dvc.repo.lock import lock as lock_stage
from dvc.repo.freeze import freeze, unfreeze
from dvc.repo.move import move
from dvc.repo.run import run
from dvc.repo.imp import imp
@@ -133,7 +134,13 @@ class Repo:
@tree.setter
def tree(self, tree):
self._tree = tree if isinstance(tree, CleanTree) else CleanTree(tree)
if is_working_tree(tree) or tree.tree_root == self.root_dir:
root = None
else:
root = self.root_dir
self._tree = (
tree if isinstance(tree, CleanTree) else CleanTree(tree, root)
)
# Our graph cache is no longer valid, as it was based on the previous
# tree.
self._reset()
@@ -434,7 +441,7 @@ class Repo:
stages = stages or self.stages
outs = Trie() # Use trie to efficiently find overlapping outs and deps
for stage in filter(bool, stages):
for stage in filter(bool, stages): # bug? not using it later
for out in stage.outs:
out_key = out.path_info.parts
+8 -5
View File
@@ -18,7 +18,7 @@ def brancher( # noqa: E302
str: the display name for the currently selected tree, it could be:
- a git revision identifier
- empty string it there is no branches to iterate over
- "Working Tree" if there are uncommitted changes in the SCM repo
- "workspace" if there are uncommitted changes in the SCM repo
"""
if not any([revs, all_branches, all_tags, all_commits]):
yield ""
@@ -30,10 +30,10 @@ def brancher( # noqa: E302
scm = self.scm
self.tree = WorkingTree(self.root_dir)
yield "working tree"
yield "workspace"
if revs and "working tree" in revs:
revs.remove("working tree")
if revs and "workspace" in revs:
revs.remove("workspace")
if all_commits:
revs = scm.list_all_commits()
@@ -48,6 +48,9 @@ def brancher( # noqa: E302
if revs:
for sha, names in group_by(scm.resolve_rev, revs).items():
self.tree = scm.get_tree(sha)
yield ", ".join(names)
# ignore revs that don't contain repo root
# (i.e. revs from before a subdir=True repo was init'ed)
if self.tree.exists(self.root_dir):
yield ", ".join(names)
finally:
self.tree = saved_tree
+1 -1
View File
@@ -7,7 +7,7 @@ from dvc.scm.tree import is_working_tree
@locked
def diff(self, a_rev="HEAD", b_rev=None):
"""
By default, it compares the working tree with the last commit's tree.
By default, it compares the workspace with the last commit's tree.
This implementation differs from `git diff` since DVC doesn't have
the concept of `index`, but it keeps the same interface, thus,
+21
View File
@@ -0,0 +1,21 @@
from . import locked
@locked
def _set(repo, target, frozen):
from dvc.utils import parse_target
path, name = parse_target(target)
stage = repo.get_stage(path, name)
stage.frozen = frozen
stage.dvcfile.dump(stage, update_pipeline=True)
return stage
def freeze(repo, target):
return _set(repo, target, True)
def unfreeze(repo, target):
return _set(repo, target, False)
+1 -1
View File
@@ -3,4 +3,4 @@ def imp(self, url, path, out=None, rev=None):
if rev is not None:
erepo["rev"] = rev
return self.imp_url(path, out=out, erepo=erepo, locked=True)
return self.imp_url(path, out=out, erepo=erepo, frozen=True)
+4 -4
View File
@@ -5,12 +5,12 @@ from dvc.utils import relpath, resolve_output, resolve_paths
from dvc.utils.fs import path_isin
from ..exceptions import OutputDuplicationError
from . import locked as locked_repo
from . import locked
@locked_repo
@locked
@scm_context
def imp_url(self, url, out=None, fname=None, erepo=None, locked=True):
def imp_url(self, url, out=None, fname=None, erepo=None, frozen=True):
from dvc.dvcfile import Dvcfile
from dvc.stage import Stage, create_stage
@@ -48,7 +48,7 @@ def imp_url(self, url, out=None, fname=None, erepo=None, locked=True):
stage.run()
stage.locked = locked
stage.frozen = frozen
dvcfile.dump(stage)
-13
View File
@@ -1,13 +0,0 @@
from . import locked
@locked
def lock(self, target, unlock=False):
from dvc.utils import parse_target
path, name = parse_target(target)
stage = self.get_stage(path, name)
stage.locked = False if unlock else True
stage.dvcfile.dump(stage, update_pipeline=True)
return stage
-10
View File
@@ -2,21 +2,11 @@ class Metrics:
def __init__(self, repo):
self.repo = repo
def add(self, *args, **kwargs):
from dvc.repo.metrics.add import add
return add(self.repo, *args, **kwargs)
def show(self, *args, **kwargs):
from dvc.repo.metrics.show import show
return show(self.repo, *args, **kwargs)
def remove(self, *args, **kwargs):
from dvc.repo.metrics.remove import remove
return remove(self.repo, *args, **kwargs)
def diff(self, *args, **kwargs):
from .diff import diff
-5
View File
@@ -1,5 +0,0 @@
from dvc.repo.metrics.modify import modify
def add(repo, path):
modify(repo, path)
-22
View File
@@ -1,22 +0,0 @@
from dvc.dvcfile import Dvcfile
from dvc.exceptions import DvcException
from dvc.repo import locked
@locked
def modify(repo, path, delete=False):
outs = repo.find_outs_by_path(path)
assert len(outs) == 1
out = outs[0]
if out.scheme != "local":
msg = "output '{}' scheme '{}' is not supported for metrics"
raise DvcException(msg.format(out.path, out.path_info.scheme))
if delete:
out.metric = None
out.verify_metric()
dvcfile = Dvcfile(repo, out.stage.path)
dvcfile.dump(out.stage)
-5
View File
@@ -1,5 +0,0 @@
from dvc.repo.metrics.modify import modify
def remove(repo, path):
modify(repo, path, delete=True)
+3 -3
View File
@@ -104,13 +104,13 @@ def show(
if not res:
raise NoMetricsError()
# Hide working tree metrics if they are the same as in the active branch
# Hide workspace metrics if they are the same as in the active branch
try:
active_branch = repo.scm.active_branch()
except TypeError:
pass # Detached head
else:
if res.get("working tree") == res.get(active_branch):
res.pop("working tree", None)
if res.get("workspace") == res.get(active_branch):
res.pop("workspace", None)
return res
+3 -3
View File
@@ -58,13 +58,13 @@ def show(repo, revs=None):
if not res:
raise NoParamsError("no parameter configs files in this repository")
# Hide working tree params if they are the same as in the active branch
# Hide workspace params if they are the same as in the active branch
try:
active_branch = repo.scm.active_branch()
except TypeError:
pass # Detached head
else:
if res.get("working tree") == res.get(active_branch):
res.pop("working tree", None)
if res.get("workspace") == res.get(active_branch):
res.pop("workspace", None)
return res
+22
View File
@@ -11,3 +11,25 @@ class Plots:
from .diff import diff
return diff(self.repo, *args, **kwargs)
def modify(self, path, props=None, unset=None):
from dvc.dvcfile import Dvcfile
(out,) = self.repo.find_outs_by_path(path)
# This out will become a plot unless it is one already
if not isinstance(out.plot, dict):
out.plot = {}
for field in unset or ():
out.plot.pop(field, None)
out.plot.update(props or {})
# Empty dict will move it to non-plots
if not out.plot:
out.plot = True
out.verify_metric()
dvcfile = Dvcfile(self.repo, out.stage.path)
dvcfile.dump(out.stage, update_pipeline=True)
+1 -1
View File
@@ -258,7 +258,7 @@ def _load_from_revisions(repo, datafile, revisions):
exceptions = []
for rev in repo.brancher(revs=revisions):
if rev == "working tree" and rev not in revisions:
if rev == "workspace" and rev not in revisions:
continue
try:
+1 -1
View File
@@ -3,7 +3,7 @@ def _revisions(revs, is_dirty):
if len(revisions) <= 1:
if len(revisions) == 0 and is_dirty:
revisions.append("HEAD")
revisions.append("working tree")
revisions.append("workspace")
return revisions
+86 -90
View File
@@ -1,13 +1,14 @@
import copy
import logging
import os
from collections import defaultdict
from funcy import first, last
from funcy import first, last, project
from dvc.exceptions import DvcException
from dvc.repo import locked
from dvc.schema import PLOT_PROPS
from .data import PlotData
from .data import NoMetricInHistoryError, PlotData
from .template import NoDataForTemplateError, Template
logger = logging.getLogger(__name__)
@@ -43,29 +44,24 @@ def _evaluate_templatepath(repo, template=None):
@locked
def fill_template(
repo,
datafile,
template_path,
revisions,
fields=None,
path=None,
csv_header=True,
x_field=None,
y_field=None,
**kwargs,
repo, datafile, template_path, revisions, props,
):
if x_field and fields:
fields.add(x_field)
# Copy things to not modify passed values
props = props.copy()
fields = copy.copy(props.get("fields"))
if y_field and fields:
fields.add(y_field)
if props.get("x") and fields:
fields.add(props.get("x"))
if props.get("y") and fields:
fields.add(props.get("y"))
template_datafiles, x_anchor, y_anchor = _parse_template(
template_path, datafile
)
append_index = x_anchor and not x_field
append_index = x_anchor and not props.get("x")
if append_index:
x_field = PlotData.INDEX_FIELD
props["x"] = PlotData.INDEX_FIELD
template_data = {}
for template_datafile in template_datafiles:
@@ -76,13 +72,13 @@ def fill_template(
for pd in plot_datas:
rev_data_points = pd.to_datapoints(
fields=fields,
path=path,
csv_header=csv_header,
path=props.get("path"),
csv_header=props.get("csv_header", True),
append_index=append_index,
)
if y_anchor and not y_field:
y_field = _infer_y_field(rev_data_points, x_field)
if y_anchor and not props.get("y"):
props["y"] = _infer_y_field(rev_data_points, props.get("x"))
tmp_data.extend(rev_data_points)
template_data[template_datafile] = tmp_data
@@ -91,12 +87,7 @@ def fill_template(
raise NoDataForTemplateError(template_path)
content = Template.fill(
template_path,
template_data,
priority_datafile=datafile,
x_field=x_field,
y_field=y_field,
**kwargs,
template_path, template_data, priority_datafile=datafile, props=props,
)
path = datafile or ",".join(template_datafiles)
@@ -113,17 +104,17 @@ def _infer_y_field(rev_data_points, x_field):
return y_field
def _show(repo, datafile=None, template=None, revs=None, **kwargs):
def _show(repo, datafile=None, revs=None, props=None):
if revs is None:
revs = ["working tree"]
revs = ["workspace"]
if not datafile and not template:
if not datafile and not props.get("template"):
raise NoDataOrTemplateProvided()
template_path = _evaluate_templatepath(repo, template)
template_path = _evaluate_templatepath(repo, props.get("template"))
plot_datafile, plot_content = fill_template(
repo, datafile, template_path, revs, **kwargs
repo, datafile, template_path, revs, props
)
return plot_datafile, plot_content
@@ -148,67 +139,72 @@ def _parse_template(template_path, priority_datafile):
)
def _collect_plots(repo):
plots = defaultdict(set)
def _collect_plots(repo, targets=None):
from dvc.exceptions import OutputNotFoundError
from contextlib import suppress
for stage in repo.stages:
for out in stage.outs:
if not out.plot:
continue
if isinstance(out.plot, dict):
template = out.plot[out.PARAM_PLOT_TEMPLATE]
else:
template = None
plots[str(out)].add(template)
return plots
def show(repo, targets=None, template=None, revs=None, **kwargs) -> dict:
if isinstance(targets, str):
targets = [targets]
def _targets_to_outs(targets):
for t in targets:
with suppress(OutputNotFoundError):
(out,) = repo.find_outs_by_path(t)
yield out
if targets:
for target in targets:
return {
target: _show(
repo,
datafile=target,
template=template,
revs=revs,
**kwargs,
)[1]
}
if not revs:
plots = _collect_plots(repo)
outs = _targets_to_outs(targets)
else:
plots = defaultdict(set)
for rev in repo.brancher(revs=revs):
for plot, templates in _collect_plots(repo).items():
plots[plot].update(templates)
outs = (out for stage in repo.stages for out in stage.outs if out.plot)
return {str(out): _plot_props(out) for out in outs}
def _plot_props(out):
if not out.plot:
raise DvcException(
f"'{out}' is not a plot. Use `dvc plots modify` to change that."
)
if isinstance(out.plot, list):
raise DvcException("Multiple plots per data file not supported yet.")
if isinstance(out.plot, bool):
return {}
return project(out.plot, PLOT_PROPS)
def show(repo, targets=None, revs=None, props=None) -> dict:
if isinstance(targets, str):
targets = [targets]
if props is None:
props = {}
# Collect plot data files with associated props
plots = {}
for rev in repo.brancher(revs=revs):
if revs is not None and rev not in revs:
continue
for datafile, file_props in _collect_plots(repo, targets).items():
# props from command line overwrite plot props from out definition
full_props = {**file_props, **props}
if datafile in plots:
saved_rev, saved_props = plots[datafile]
if saved_props != props:
logger.warning(
f"Inconsistent plot props for '{datafile}' in "
f"'{saved_rev}' and '{rev}'. "
f"Going to use ones from '{saved_rev}'"
)
else:
plots[datafile] = rev, full_props
if not plots:
datafile, plot = _show(
repo, datafile=None, template=template, revs=revs, **kwargs
)
if targets:
raise NoMetricInHistoryError(", ".join(targets))
datafile, plot = _show(repo, datafile=None, revs=revs, props=props)
return {datafile: plot}
ret = {}
for plot, templates in plots.items():
tmplt = template
if len(templates) == 1:
tmplt = list(templates)[0]
elif not template:
raise TooManyTemplatesError(
f"'{plot}' uses multiple templates '{templates}'. "
"Use `-t|--template` to specify the template to use. "
)
ret[plot] = _show(
repo, datafile=plot, template=tmplt, revs=revs, **kwargs
)[1]
return ret
return {
datafile: _show(repo, datafile=datafile, revs=revs, props=props)[1]
for datafile, (_, props) in plots.items()
}
+25 -41
View File
@@ -86,29 +86,24 @@ class Template:
@staticmethod
def fill(
template_path,
data,
priority_datafile=None,
x_field=None,
y_field=None,
title=None,
x_title=None,
y_title=None,
template_path, data, priority_datafile=None, props=None,
):
props = props or {}
with open(template_path) as fobj:
result_content = fobj.read()
if x_field:
Template._check_field_exists(data, x_field)
if y_field:
Template._check_field_exists(data, y_field)
if props.get("x"):
Template._check_field_exists(data, props.get("x"))
if props.get("y"):
Template._check_field_exists(data, props.get("y"))
result_content = Template._replace_data_anchors(
result_content, data, priority_datafile
)
result_content = Template._replace_metadata_anchors(
result_content, title, x_field, x_title, y_field, y_title
result_content, props
)
return result_content
@@ -122,34 +117,23 @@ class Template:
raise NoFieldInDataError(field)
@staticmethod
def _replace_metadata_anchors(
result_content, title, x_field, x_title, y_field, y_title
):
if Template.TITLE_ANCHOR in result_content:
if title:
result_content = result_content.replace(
Template.TITLE_ANCHOR, title
)
else:
result_content = result_content.replace(
Template.TITLE_ANCHOR, ""
)
if Template.X_ANCHOR in result_content and x_field:
result_content = result_content.replace(Template.X_ANCHOR, x_field)
if Template.Y_ANCHOR in result_content and y_field:
result_content = result_content.replace(Template.Y_ANCHOR, y_field)
if Template.X_TITLE_ANCHOR in result_content:
if not x_title and x_field:
x_title = x_field
result_content = result_content.replace(
Template.X_TITLE_ANCHOR, x_title
)
if Template.Y_TITLE_ANCHOR in result_content:
if not y_title and y_field:
y_title = y_field
result_content = result_content.replace(
Template.Y_TITLE_ANCHOR, y_title
)
def _replace_metadata_anchors(result_content, props):
props.setdefault("title", "")
props.setdefault("xlab", props.get("x"))
props.setdefault("ylab", props.get("y"))
replace_pairs = [
(Template.TITLE_ANCHOR, "title"),
(Template.X_ANCHOR, "x"),
(Template.Y_ANCHOR, "y"),
(Template.X_TITLE_ANCHOR, "xlab"),
(Template.Y_TITLE_ANCHOR, "ylab"),
]
for anchor, key in replace_pairs:
value = props.get(key)
if anchor in result_content and value is not None:
result_content = result_content.replace(anchor, value)
return result_content
@staticmethod
+3 -7
View File
@@ -7,15 +7,11 @@ logger = logging.getLogger(__name__)
@locked
def remove(self, target, dvc_only=False):
from ..dvcfile import Dvcfile, is_valid_filename
def remove(self, target, outs=False):
path, name = parse_target(target)
stages = self.get_stages(path, name)
for stage in stages:
stage.remove_outs(force=True)
if path and is_valid_filename(path) and not dvc_only:
Dvcfile(self, path).remove()
for stage in stages:
stage.remove(remove_outs=outs, force=outs)
return stages
+3 -3
View File
@@ -10,9 +10,9 @@ logger = logging.getLogger(__name__)
def _reproduce_stage(stage, **kwargs):
if stage.locked and not stage.is_import:
if stage.frozen and not stage.is_import:
logger.warning(
"{} is locked. Its dependencies are"
"{} is frozen. Its dependencies are"
" not going to be reproduced.".format(stage)
)
@@ -34,7 +34,7 @@ def _get_active_graph(G):
active = G.copy()
for stage in G:
if not stage.locked:
if not stage.frozen:
continue
active.remove_edges_from(G.out_edges(stage))
for edge in G.out_edges(stage):
+10 -4
View File
@@ -3,7 +3,11 @@ import os
from funcy import concat, first
from dvc.exceptions import InvalidArgumentError
from dvc.stage.exceptions import DuplicateStageName, InvalidStageName
from dvc.stage.exceptions import (
DuplicateStageName,
InvalidStageName,
StageFileAlreadyExistsError,
)
from ..exceptions import OutputDuplicationError
from . import locked
@@ -74,10 +78,12 @@ def run(self, fname=None, no_exec=False, single_stage=False, **kwargs):
dvcfile = Dvcfile(self, stage.path)
if dvcfile.exists():
if stage_name and stage_name in dvcfile.stages:
if kwargs.get("overwrite", True):
dvcfile.remove_stage(stage)
elif stage_cls != PipelineStage:
raise StageFileAlreadyExistsError(dvcfile.relpath)
elif stage_name and stage_name in dvcfile.stages:
raise DuplicateStageName(stage_name, dvcfile)
if stage_cls != PipelineStage:
dvcfile.remove_with_prompt(force=kwargs.get("overwrite", True))
try:
self.check_modified_graph([stage])
+2 -2
View File
@@ -14,9 +14,9 @@ def _joint_status(stages):
status = {}
for stage in stages:
if stage.locked and not stage.is_repo_import:
if stage.frozen and not stage.is_repo_import:
logger.warning(
"{} is locked. Its dependencies are"
"{} is frozen. Its dependencies are"
" not going to be shown in the status output.".format(stage)
)
+17 -4
View File
@@ -11,7 +11,8 @@ SINGLE_STAGE_SCHEMA = {
StageParams.PARAM_WDIR: Any(str, None),
StageParams.PARAM_DEPS: Any([dependency.SCHEMA], None),
StageParams.PARAM_OUTS: Any([output.SCHEMA], None),
StageParams.PARAM_LOCKED: bool,
StageParams.PARAM_LOCKED: bool, # backard compatibility
StageParams.PARAM_FROZEN: bool,
StageParams.PARAM_META: object,
StageParams.PARAM_ALWAYS_CHANGED: bool,
}
@@ -29,8 +30,20 @@ OUT_PSTAGE_DETAILED_SCHEMA = {
str: {BaseOutput.PARAM_CACHE: bool, BaseOutput.PARAM_PERSIST: bool}
}
PLOT_PSTAGE_SCHEMA = OUT_PSTAGE_DETAILED_SCHEMA.copy()
PLOT_PSTAGE_SCHEMA[str][BaseOutput.PARAM_PLOT_TEMPLATE] = str
PLOT_PROPS = {
BaseOutput.PARAM_PLOT_TEMPLATE: str,
BaseOutput.PARAM_PLOT_X: str,
BaseOutput.PARAM_PLOT_Y: str,
BaseOutput.PARAM_PLOT_XLAB: str,
BaseOutput.PARAM_PLOT_YLAB: str,
BaseOutput.PARAM_PLOT_TITLE: str,
BaseOutput.PARAM_PLOT_CSV_HEADER: bool,
}
PLOT_PROPS_SCHEMA = {
**OUT_PSTAGE_DETAILED_SCHEMA[str],
**PLOT_PROPS,
}
PLOT_PSTAGE_SCHEMA = {str: Any(PLOT_PROPS_SCHEMA, [PLOT_PROPS_SCHEMA])}
PARAM_PSTAGE_NON_DEFAULT_SCHEMA = {str: [str]}
@@ -42,7 +55,7 @@ SINGLE_PIPELINE_STAGE_SCHEMA = {
Optional(StageParams.PARAM_PARAMS): [
Any(str, PARAM_PSTAGE_NON_DEFAULT_SCHEMA)
],
Optional(StageParams.PARAM_LOCKED): bool,
Optional(StageParams.PARAM_FROZEN): bool,
Optional(StageParams.PARAM_META): object,
Optional(StageParams.PARAM_ALWAYS_CHANGED): bool,
Optional(StageParams.PARAM_OUTS): [
+3 -3
View File
@@ -10,7 +10,7 @@ from dvc.output import BaseOutput
from dvc.stage.params import StageParams
from dvc.stage.utils import resolve_wdir
from dvc.utils.collections import apply_diff
from dvc.utils.stage import parse_stage_for_update
from dvc.utils.yaml import parse_yaml_for_update
if TYPE_CHECKING:
from dvc.stage import PipelineStage, Stage
@@ -115,7 +115,7 @@ def to_pipeline_file(stage: "PipelineStage"):
stage.PARAM_PLOTS,
_get_outs([out for out in stage.outs if out.plot]),
),
(stage.PARAM_LOCKED, stage.locked),
(stage.PARAM_FROZEN, stage.frozen),
(stage.PARAM_ALWAYS_CHANGED, stage.always_changed),
]
return {
@@ -165,7 +165,7 @@ def to_single_stage_file(stage: "Stage"):
# - apply changes to a returned structure
# - serialize it
if stage._stage_text is not None:
saved_state = parse_stage_for_update(stage._stage_text, stage.path)
saved_state = parse_yaml_for_update(stage._stage_text, stage.path)
# Stage doesn't work with meta in any way, so .dumpd() doesn't
# have it. We simply copy it over.
if "meta" in saved_state:
+13 -10
View File
@@ -42,6 +42,7 @@ def loads_from(cls, repo, path, wdir, data):
[
Stage.PARAM_CMD,
Stage.PARAM_LOCKED,
Stage.PARAM_FROZEN,
Stage.PARAM_ALWAYS_CHANGED,
Stage.PARAM_MD5,
"name",
@@ -93,7 +94,8 @@ class Stage(params.StageParams):
deps=None,
outs=None,
md5=None,
locked=False,
locked=False, # backward compatibility
frozen=False,
always_changed=False,
stage_text=None,
dvcfile=None,
@@ -110,7 +112,7 @@ class Stage(params.StageParams):
self.outs = outs
self.deps = deps
self.md5 = md5
self.locked = locked
self.frozen = locked or frozen
self.always_changed = always_changed
self._stage_text = stage_text
self._dvcfile = dvcfile
@@ -201,7 +203,7 @@ class Stage(params.StageParams):
return isinstance(self.deps[0], dependency.RepoDependency)
def changed_deps(self):
if self.locked:
if self.frozen:
return False
if self.is_callback:
@@ -280,12 +282,13 @@ class Stage(params.StageParams):
out.unprotect()
@rwlocked(write=["outs"])
def remove(self, force=False, remove_outs=True):
def remove(self, force=False, remove_outs=True, purge=True):
if remove_outs:
self.remove_outs(ignore_remove=True, force=force)
else:
self.unprotect_outs()
self.dvcfile.remove()
if purge:
self.dvcfile.remove_stage(self)
@rwlocked(read=["deps"], write=["outs"])
def reproduce(self, interactive=False, **kwargs):
@@ -401,16 +404,16 @@ class Stage(params.StageParams):
@rwlocked(read=["deps"], write=["outs"])
def run(self, dry=False, no_commit=False, force=False, run_cache=True):
if (self.cmd or self.is_import) and not self.locked and not dry:
if (self.cmd or self.is_import) and not self.frozen and not dry:
self.remove_outs(ignore_remove=False, force=False)
if not self.locked and self.is_import:
if not self.frozen and self.is_import:
sync_import(self, dry, force)
elif not self.locked and self.cmd:
elif not self.frozen and self.cmd:
run_stage(self, dry, force, run_cache)
else:
args = (
("outputs", "locked ") if self.locked else ("data sources", "")
("outputs", "frozen ") if self.frozen else ("data sources", "")
)
logger.info("Verifying %s in %s%s", *args, self)
if not dry:
@@ -452,7 +455,7 @@ class Stage(params.StageParams):
ret = []
show_import = self.is_repo_import and check_updates
if not self.locked or show_import:
if not self.frozen or show_import:
self._status_deps(ret)
self._status_outs(ret)
self._status_always_changed(ret)
+2 -2
View File
@@ -11,7 +11,7 @@ from dvc.serialize import to_single_stage_lockfile
from dvc.stage.loader import StageLoader
from dvc.utils import dict_sha256, relpath
from dvc.utils.fs import makedirs
from dvc.utils.stage import dump_stage_file
from dvc.utils.yaml import dump_yaml
logger = logging.getLogger(__name__)
@@ -154,7 +154,7 @@ class StageCache:
path = self._get_cache_path(cache_key, cache_value)
dpath = os.path.dirname(path)
makedirs(dpath, exist_ok=True)
dump_stage_file(path, cache)
dump_yaml(path, cache)
def is_cached(self, stage):
return bool(self._load(stage))
+3 -3
View File
@@ -5,12 +5,12 @@ logger = logging.getLogger(__name__)
def update_import(stage, rev=None):
stage.deps[0].update(rev=rev)
locked = stage.locked
stage.locked = False
frozen = stage.frozen
stage.frozen = False
try:
stage.reproduce()
finally:
stage.locked = locked
stage.frozen = frozen
def sync_import(stage, dry=False, force=False):
+2 -1
View File
@@ -4,7 +4,8 @@ class StageParams:
PARAM_WDIR = "wdir"
PARAM_DEPS = "deps"
PARAM_OUTS = "outs"
PARAM_LOCKED = "locked"
PARAM_LOCKED = "locked" # backward compatibility
PARAM_FROZEN = "frozen"
PARAM_META = "meta"
PARAM_ALWAYS_CHANGED = "always_changed"
PARAM_PARAMS = "params"
+1
View File
@@ -82,6 +82,7 @@ def _is_cached(stage):
cached = (
not stage.is_callback
and not stage.always_changed
and not stage.changed_stage()
and stage.already_cached()
)
if cached:
+3 -2
View File
@@ -136,7 +136,8 @@ def compute_md5(stage):
return dict_md5(
d,
exclude=[
stage.PARAM_LOCKED,
stage.PARAM_LOCKED, # backward compatibility
stage.PARAM_FROZEN,
BaseOutput.PARAM_METRIC,
BaseOutput.PARAM_PERSIST,
],
@@ -155,7 +156,7 @@ def get_dump(stage):
stage.PARAM_MD5: stage.md5,
stage.PARAM_CMD: stage.cmd,
stage.PARAM_WDIR: resolve_wdir(stage.wdir, stage.path),
stage.PARAM_LOCKED: stage.locked,
stage.PARAM_FROZEN: stage.frozen,
stage.PARAM_DEPS: [d.dumpd() for d in stage.deps],
stage.PARAM_OUTS: [o.dumpd() for o in stage.outs],
stage.PARAM_ALWAYS_CHANGED: stage.always_changed,
+48
View File
@@ -0,0 +1,48 @@
"""Draws DAG in ASCII."""
import logging
import os
import pydoc
import sys
from dvc.env import DVC_PAGER
from dvc.utils import format_link
logger = logging.getLogger(__name__)
DEFAULT_PAGER = "less"
DEFAULT_PAGER_FORMATTED = "{} --chop-long-lines --clear-screen".format(
DEFAULT_PAGER
)
def make_pager(cmd):
def _pager(text):
return pydoc.tempfilepager(pydoc.plain(text), cmd)
return _pager
def find_pager():
if not sys.stdout.isatty():
return pydoc.plainpager
env_pager = os.getenv(DVC_PAGER)
if env_pager:
return make_pager(env_pager)
if os.system(f"({DEFAULT_PAGER}) 2>{os.devnull}") == 0:
return make_pager(DEFAULT_PAGER_FORMATTED)
logger.warning(
"Unable to find `less` in the PATH. Check out "
"{} for more info.".format(
format_link("https://man.dvc.org/pipeline/show")
)
)
return pydoc.plainpager
def pager(text):
find_pager()(text)
+10 -10
View File
@@ -4,7 +4,7 @@ import yaml
from ruamel.yaml import YAML
from ruamel.yaml.error import YAMLError
from dvc.exceptions import StageFileCorruptedError
from dvc.exceptions import YAMLFileCorruptedError
try:
from yaml import CSafeLoader as SafeLoader
@@ -12,35 +12,35 @@ except ImportError:
from yaml import SafeLoader
def load_stage_file(path):
def load_yaml(path):
with open(path, encoding="utf-8") as fd:
return parse_stage(fd.read(), path)
return parse_yaml(fd.read(), path)
def parse_stage(text, path):
def parse_yaml(text, path):
try:
return yaml.load(text, Loader=SafeLoader) or {}
except yaml.error.YAMLError as exc:
raise StageFileCorruptedError(path) from exc
raise YAMLFileCorruptedError(path) from exc
def parse_stage_for_update(text, path):
def parse_yaml_for_update(text, path):
"""Parses text into Python structure.
Unlike `parse_stage()` this returns ordered dicts, values have special
Unlike `parse_yaml()` this returns ordered dicts, values have special
attributes to store comments and line breaks. This allows us to preserve
all of those upon dump.
This one is, however, several times slower than simple `parse_stage()`.
This one is, however, several times slower than simple `parse_yaml()`.
"""
try:
yaml = YAML()
return yaml.load(text) or {}
except YAMLError as exc:
raise StageFileCorruptedError(path) from exc
raise YAMLFileCorruptedError(path) from exc
def dump_stage_file(path, data):
def dump_yaml(path, data):
with open(path, "w", encoding="utf-8") as fd:
yaml = YAML()
yaml.default_flow_style = False
+1 -1
View File
@@ -6,7 +6,7 @@
import os
import subprocess
_BASE_VERSION = "1.0.0a6"
_BASE_VERSION = "1.0.0a7"
def _generate_version(base_version):
+1 -1
View File
@@ -3,7 +3,7 @@
set -x
set -e
pip install Pygments collective.checkdocs pre-commit
pip install Pygments collective.checkdocs pre-commit .
# stop the build if there are any readme formatting errors
python setup.py checkdocs
+10 -17
View File
@@ -4,9 +4,10 @@
# - https://opensource.com/article/18/3/creating-bash-completion-script
# - https://stackoverflow.com/questions/12933362
_dvc_commands='add cache checkout commit config destroy diff fetch get-url get gc \
import-url import init install lock list metrics move params pipeline plots pull push \
remote remove repro root run status unlock unprotect update version'
_dvc_commands='add cache checkout commit config dag destroy diff fetch freeze \
get-url get gc import-url import init install list metrics move params \
plots pull push remote remove repro root run status unfreeze unprotect \
update version'
_dvc_options='-h --help -V --version'
_dvc_global_options='-h --help -q --quiet -v --verbose'
@@ -20,6 +21,8 @@ _dvc_checkout_COMPGEN=_dvc_compgen_DVCFiles
_dvc_commit='-f --force -d --with-deps -R --recursive'
_dvc_commit_COMPGEN=_dvc_compgen_DVCFiles
_dvc_config='-u --unset --local --system --global'
_dvc_dag='--dot --full'
_dvc_dag_COMPGEN=_dvc_compgen_DVCFiles
_dvc_destroy='-f --force'
_dvc_diff='-t --show-json --show-hash --show-md'
_dvc_fetch='-j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -R --recursive'
@@ -33,26 +36,16 @@ _dvc_init='--no-scm -f --force'
_dvc_install=''
_dvc_list='-R --recursive --dvc-only --rev'
_dvc_list_COMPGEN=_dvc_compgen_files
_dvc_lock=''
_dvc_lock_COMPGEN=_dvc_compgen_DVCFiles
_dvc_metrics='add diff modify remove show'
_dvc_metrics_add='-t --type -x --xpath'
_dvc_metrics_add_COMPGEN=_dvc_compgen_files
_dvc_freeze=''
_dvc_freeze_COMPGEN=_dvc_compgen_DVCFiles
_dvc_metrics='diff show'
_dvc_metrics_diff='--targets -t --type -x --xpath -R --show-json --show-md --no-path --old'
_dvc_metrics_modify='-t --type -x --xpath'
_dvc_metrics_modify_COMPGEN=_dvc_compgen_files
_dvc_metrics_remove=''
_dvc_metrics_remove_COMPGEN=_dvc_compgen_files
_dvc_metrics_show='-t --type -x --xpath -a --all-branches -T --all-tags -R --recursive'
_dvc_metrics_show_COMPGEN=_dvc_compgen_files
_dvc_move=''
_dvc_move_COMPGEN=_dvc_compgen_files
_dvc_params='diff'
_dvc_params_diff='--all --show-json --show-md --no-path'
_dvc_pipeline='list show'
_dvc_pipeline_list=''
_dvc_pipeline_show='-c --commands -o --outs --ascii --dot --tree -l --locked'
_dvc_pipeline_show_COMPGEN=_dvc_compgen_DVCFiles
_dvc_plots='show diff'
_dvc_plots_show='-t --template -o --out -x -y --show-json --no-csv-header --title --xlab --ylab'
_dvc_plots_diff='-t --template --targets -o --out -x -y --show-json --no-csv-header --title --xlab --ylab'
@@ -76,7 +69,7 @@ _dvc_run='--no-exec -f --file -d --deps -o --outs -O --outs-no-cache --outs-pers
_dvc_run_COMPGEN=_dvc_compgen_DVCFiles
_dvc_status='-j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -c --cloud'
_dvc_status_COMPGEN=_dvc_compgen_DVCFiles
_dvc_unlock_COMPGEN=_dvc_compgen_DVCFiles
_dvc_unfreeze_COMPGEN=_dvc_compgen_DVCFiles
_dvc_unprotect_COMPGEN=_dvc_compgen_files
_dvc_update='--rev'
_dvc_update_COMPGEN=_dvc_compgen_DVCFiles
+15 -13
View File
@@ -17,6 +17,7 @@ _dvc_commands() {
"checkout:Checkout data files from cache."
"commit:Save changed data to cache and update DVC-files."
"config:Get or set config settings."
"dag:Visualize DVC project DAG."
"destroy:Remove DVC-files, local DVC config and data cache."
"diff:Show added, modified, or deleted data between commits in the DVC repository, or between a commit and the workspace."
"fetch:Get files or directories tracked by DVC from remote storage into the cache."
@@ -29,10 +30,9 @@ _dvc_commands() {
"install:Install DVC git hooks into the repository."
"list:List repository contents, including files and directories tracked by DVC and by Git."
"lock:Lock DVC-file."
"metrics:Commands to add, manage, collect and display metrics."
"metrics:Commands to display and compare metrics."
"move:Rename or move a DVC controlled data file or a directory."
"params:Commands to display params."
"pipeline:Manage pipelines."
"pull:Pull data files from a DVC remote storage."
"push:Push data files to a DVC remote storage."
"plots:Generate plot for metrics structured as JSON, CSV or TSV."
@@ -96,6 +96,12 @@ _dvc_config=(
{-u,--unset}"[Unset option.]"
)
_dvc_dag=(
"--dot[Print DAG in DOT format.]"
"--full[Show full DAG that the target belongs too, instead of showing DAG consisting only of ancestors.]"
"1:Stage:"
)
_dvc_destroy=(
{-f,--force}"[Force destruction.]"
)
@@ -132,7 +138,7 @@ _dvc_get=(
)
_dvc_gc=(
{-w,--workspace}"[Keep data files used in the current workspace (working tree).]"
{-w,--workspace}"[Keep data files used in the current workspace.]"
{-a,--all-branches}"[Keep data files for the tips of all Git branches.]"
"--all-commits[Keep data files for all Git commits.]"
{-T,--all-tags}"[Keep data files for all Git tags.]"
@@ -163,7 +169,7 @@ _dvc_init=(
_dvc_install=()
_dvc_lock=(
_dvc_freeze=(
"*:Stages:_files -g '(*.dvc|Dvcfile)'"
)
@@ -176,7 +182,7 @@ _dvc_list=(
)
_dvc_metrics=(
"1:Sub command:(add show diff modify remove)"
"1:Sub command:(show diff)"
)
_dvc_move=(
@@ -188,10 +194,6 @@ _dvc_params=(
"1:Sub command:(diff)"
)
_dvc_pipeline=(
"1:Sub command:(show list)"
)
_dvc_pull=(
{-j,--jobs}"[Number of jobs to run simultaneously.]:Number of jobs:"
{-r,--remote}"[Remote repository to pull from.]:Remote repository:"
@@ -278,7 +280,7 @@ _dvc_status=(
"*:Stages:_files -g '(*.dvc|Dvcfile)'"
)
_dvc_unlock=(
_dvc_unfreeze=(
"*:Stages:_files -g '(*.dvc|Dvcfile)'"
)
@@ -305,9 +307,11 @@ case $words[1] in
checkout) _arguments $_dvc_global_options $_dvc_checkout ;;
commit) _arguments $_dvc_global_options $_dvc_commit ;;
config) _arguments $_dvc_global_options $_dvc_config ;;
dag) _arguments $_dvc_global_options $_dvc_dag ;;
destroy) _arguments $_dvc_global_options $_dvc_destroy ;;
diff) _arguments $_dvc_global_options $_dvc_diff ;;
fetch) _arguments $_dvc_global_options $_dvc_fetch ;;
freeze) _arguments $_dvc_global_options $_dvc_freeze ;;
get-url) _arguments $_dvc_global_options $_dvc_geturl ;;
get) _arguments $_dvc_global_options $_dvc_get ;;
gc) _arguments $_dvc_global_options $_dvc_gc ;;
@@ -315,12 +319,10 @@ case $words[1] in
import) _arguments $_dvc_global_options $_dvc_import ;;
init) _arguments $_dvc_global_options $_dvc_init ;;
install) _arguments $_dvc_global_options $_dvc_install ;;
lock) _arguments $_dvc_global_options $_dvc_lock ;;
list) _arguments $_dvc_global_options $_dvc_list ;;
metrics) _arguments $_dvc_global_options $_dvc_metrics ;;
move) _arguments $_dvc_global_options $_dvc_move ;;
params) _arguments $_dvc_global_options $_dvc_params ;;
pipeline) _arguments $_dvc_global_options $_dvc_pipeline ;;
pull) _arguments $_dvc_global_options $_dvc_pull ;;
push) _arguments $_dvc_global_options $_dvc_push ;;
plots) _arguments $_dvc_global_options $_dvc_plots ;;
@@ -330,7 +332,7 @@ case $words[1] in
root) _arguments $_dvc_global_options $_dvc_root ;;
run) _arguments $_dvc_global_options $_dvc_run ;;
status) _arguments $_dvc_global_options $_dvc_status ;;
unlock) _arguments $_dvc_global_options $_dvc_unlock ;;
unfreeze) _arguments $_dvc_global_options $_dvc_unfreeze ;;
unprotect) _arguments $_dvc_global_options $_dvc_unprotect ;;
update) _arguments $_dvc_global_options $_dvc_update ;;
esac
+1 -1
View File
@@ -12,6 +12,6 @@ select=B,C,E,F,W,T4,B9
[isort]
include_trailing_comma=true
known_first_party=dvc,tests
known_third_party=PyInstaller,RangeHTTPServer,boto3,colorama,configobj,distro,dpath,flaky,flufl,funcy,git,google,grandalf,mock,mockssh,moto,nanotime,packaging,paramiko,pathspec,pytest,requests,ruamel,setuptools,shortuuid,tqdm,voluptuous,yaml,zc
known_third_party=PyInstaller,RangeHTTPServer,boto3,colorama,configobj,distro,dpath,flaky,flufl,funcy,git,google,grandalf,mock,mockssh,moto,nanotime,networkx,packaging,paramiko,pathspec,pytest,requests,ruamel,setuptools,shortuuid,tqdm,voluptuous,yaml,zc
line_length=79
multi_line_output=3
+1 -1
View File
@@ -121,7 +121,7 @@ tests_requirements = [
"flake8-docstrings",
"pydocstyle<4.0",
"jaraco.windows==3.9.2",
"mock-ssh-server>=0.6.0",
"mock-ssh-server@git+https://github.com/iterative/mock-ssh-server.git",
"moto==1.3.14.dev464",
"rangehttpserver==1.2.0",
"beautifulsoup4==4.4.0",
+1 -1
View File
@@ -34,7 +34,7 @@ user = "user"
key_path = os.path.join(here, f"{user}.key")
@pytest.fixture
@pytest.fixture(scope="session")
def ssh_server():
users = {user: key_path}
with mockssh.Server(users) as s:
+38 -1
View File
@@ -1,5 +1,8 @@
import os
import pytest
from dvc.repo import Repo
from dvc.repo.metrics.show import NoMetricsError
@@ -53,6 +56,40 @@ def test_show_branch(tmp_dir, scm, dvc, run_copy_metrics):
tmp_dir.scm_gen("metrics.yaml", "foo: 2", commit="branch")
assert dvc.metrics.show(revs=["branch"]) == {
"working tree": {"metrics.yaml": {"foo": 1}},
"workspace": {"metrics.yaml": {"foo": 1}},
"branch": {"metrics.yaml": {"foo": 2}},
}
def test_show_subrepo_with_preexisting_tags(tmp_dir, scm):
tmp_dir.gen("foo", "foo")
scm.add("foo")
scm.commit("init")
scm.tag("no-metrics")
tmp_dir.gen({"subdir": {}})
subrepo_dir = tmp_dir / "subdir"
with subrepo_dir.chdir():
dvc = Repo.init(subdir=True)
scm.commit("init dvc")
dvc.run(
cmd="echo foo: 1 > metrics.yaml",
metrics=["metrics.yaml"],
single_stage=True,
)
scm.add(
[
str(subrepo_dir / "metrics.yaml"),
str(subrepo_dir / "metrics.yaml.dvc"),
]
)
scm.commit("init metrics")
scm.tag("v1")
expected_path = os.path.join("subdir", "metrics.yaml")
assert dvc.metrics.show(all_tags=True) == {
"workspace": {expected_path: {"foo": 1}},
"v1": {expected_path: {"foo": 1}},
}
+1 -1
View File
@@ -49,7 +49,7 @@ def test_show_branch(tmp_dir, scm, dvc):
tmp_dir.scm_gen("params.yaml", "foo: baz", commit="branch")
assert dvc.params.show(revs=["branch"]) == {
"working tree": {"params.yaml": {"foo": "bar"}},
"workspace": {"params.yaml": {"foo": "bar"}},
"branch": {"params.yaml": {"foo": "baz"}},
}
+3 -3
View File
@@ -28,12 +28,12 @@ def test_diff_dirty(tmp_dir, scm, dvc, run_copy_metrics):
"metric_t.json", "metric.json", plots_no_cache=["metric.json"]
)
plot_string = dvc.plots.diff(fields={"y"})["metric.json"]
plot_string = dvc.plots.diff(props={"fields": {"y"}})["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"y": 5, PlotData.INDEX_FIELD: 0, "rev": "working tree"},
{"y": 6, PlotData.INDEX_FIELD: 1, "rev": "working tree"},
{"y": 5, PlotData.INDEX_FIELD: 0, "rev": "workspace"},
{"y": 6, PlotData.INDEX_FIELD: 1, "rev": "workspace"},
{"y": 3, PlotData.INDEX_FIELD: 0, "rev": "HEAD"},
{"y": 5, PlotData.INDEX_FIELD: 1, "rev": "HEAD"},
]
+83 -70
View File
@@ -22,10 +22,6 @@ from dvc.repo.plots.template import (
)
def _remove_whitespace(value):
return value.replace(" ", "").replace("\n", "")
def _write_csv(metric, filename, header=True):
with open(filename, "w", newline="") as csvobj:
if header:
@@ -53,18 +49,19 @@ def test_plot_csv_one_column(tmp_dir, scm, dvc, run_copy_metrics):
"metric_t.csv", "metric.csv", plots_no_cache=["metric.csv"]
)
plot_string = dvc.plots.show(
csv_header=False,
x_title="x_title",
y_title="y_title",
title="mytitle",
)["metric.csv"]
props = {
"csv_header": False,
"xlab": "x_title",
"ylab": "y_title",
"title": "mytitle",
}
plot_string = dvc.plots.show(props=props)["metric.csv"]
plot_content = json.loads(plot_string)
assert plot_content["title"] == "mytitle"
assert plot_content["data"]["values"] == [
{"0": "2", PlotData.INDEX_FIELD: 0, "rev": "working tree"},
{"0": "3", PlotData.INDEX_FIELD: 1, "rev": "working tree"},
{"0": "2", PlotData.INDEX_FIELD: 0, "rev": "workspace"},
{"0": "3", PlotData.INDEX_FIELD: 1, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == PlotData.INDEX_FIELD
assert plot_content["encoding"]["y"]["field"] == "0"
@@ -89,14 +86,14 @@ def test_plot_csv_multiple_columns(tmp_dir, scm, dvc, run_copy_metrics):
{
"val": "2",
PlotData.INDEX_FIELD: 0,
"rev": "working tree",
"rev": "workspace",
"first_val": "100",
"second_val": "100",
},
{
"val": "3",
PlotData.INDEX_FIELD: 1,
"rev": "working tree",
"rev": "workspace",
"first_val": "200",
"second_val": "300",
},
@@ -115,21 +112,20 @@ def test_plot_csv_choose_axes(tmp_dir, scm, dvc, run_copy_metrics):
"metric_t.csv", "metric.csv", plots_no_cache=["metric.csv"]
)
plot_string = dvc.plots.show(x_field="first_val", y_field="second_val")[
"metric.csv"
]
props = {"x": "first_val", "y": "second_val"}
plot_string = dvc.plots.show(props=props)["metric.csv"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{
"val": "2",
"rev": "working tree",
"rev": "workspace",
"first_val": "100",
"second_val": "100",
},
{
"val": "3",
"rev": "working tree",
"rev": "workspace",
"first_val": "200",
"second_val": "300",
},
@@ -152,8 +148,8 @@ def test_plot_json_single_val(tmp_dir, scm, dvc, run_copy_metrics):
plot_json = json.loads(plot_string)
assert plot_json["data"]["values"] == [
{"val": 2, PlotData.INDEX_FIELD: 0, "rev": "working tree"},
{"val": 3, PlotData.INDEX_FIELD: 1, "rev": "working tree"},
{"val": 2, PlotData.INDEX_FIELD: 0, "rev": "workspace"},
{"val": 3, PlotData.INDEX_FIELD: 1, "rev": "workspace"},
]
assert plot_json["encoding"]["x"]["field"] == PlotData.INDEX_FIELD
assert plot_json["encoding"]["y"]["field"] == "val"
@@ -180,13 +176,13 @@ def test_plot_json_multiple_val(tmp_dir, scm, dvc, run_copy_metrics):
"val": 2,
PlotData.INDEX_FIELD: 0,
"first_val": 100,
"rev": "working tree",
"rev": "workspace",
},
{
"val": 3,
PlotData.INDEX_FIELD: 1,
"first_val": 200,
"rev": "working tree",
"rev": "workspace",
},
]
assert plot_content["encoding"]["x"]["field"] == PlotData.INDEX_FIELD
@@ -206,14 +202,13 @@ def test_plot_confusion(tmp_dir, dvc, run_copy_metrics):
commit="first run",
)
plot_string = dvc.plots.show(
template="confusion", x_field="predicted", y_field="actual",
)["metric.json"]
props = {"template": "confusion", "x": "predicted", "y": "actual"}
plot_string = dvc.plots.show(props=props)["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"predicted": "B", "actual": "A", "rev": "working tree"},
{"predicted": "A", "actual": "A", "rev": "working tree"},
{"predicted": "B", "actual": "A", "rev": "workspace"},
{"predicted": "A", "actual": "A", "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == "predicted"
assert plot_content["encoding"]["y"]["field"] == "actual"
@@ -248,9 +243,9 @@ def test_plot_multiple_revs_default(tmp_dir, scm, dvc, run_copy_metrics):
plots_no_cache=["metric.json"],
commit="third",
)
plot_string = dvc.plots.show(fields={"y"}, revs=["HEAD", "v2", "v1"],)[
"metric.json"
]
plot_string = dvc.plots.show(
revs=["HEAD", "v2", "v1"], props={"fields": {"y"}}
)["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
@@ -291,9 +286,10 @@ def test_plot_multiple_revs(tmp_dir, scm, dvc, run_copy_metrics):
scm.add(["metric.json", stage.path])
scm.commit("third")
plot_string = dvc.plots.show(
template="template.json", revs=["HEAD", "v2", "v1"],
)["metric.json"]
props = {"template": "template.json"}
plot_string = dvc.plots.show(revs=["HEAD", "v2", "v1"], props=props)[
"metric.json"
]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
@@ -379,14 +375,13 @@ def test_custom_template(tmp_dir, scm, dvc, custom_template, run_copy_metrics):
tag="v1",
)
plot_string = dvc.plots.show(
template=os.fspath(custom_template), x_field="a", y_field="b"
)["metric.json"]
props = {"template": os.fspath(custom_template), "x": "a", "y": "b"}
plot_string = dvc.plots.show(props=props)["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"a": 1, "b": 2, "rev": "working tree"},
{"a": 2, "b": 3, "rev": "working tree"},
{"a": 1, "b": 2, "rev": "workspace"},
{"a": 2, "b": 3, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == "a"
assert plot_content["encoding"]["y"]["field"] == "b"
@@ -413,14 +408,13 @@ def test_custom_template_with_specified_data(
tag="v1",
)
plot_string = dvc.plots.show(
template=os.fspath(custom_template), x_field="a", y_field="b",
)["metric.json"]
props = {"template": os.fspath(custom_template), "x": "a", "y": "b"}
plot_string = dvc.plots.show(props=props)["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"a": 1, "b": 2, "rev": "working tree"},
{"a": 2, "b": 3, "rev": "working tree"},
{"a": 1, "b": 2, "rev": "workspace"},
{"a": 2, "b": 3, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == "a"
assert plot_content["encoding"]["y"]["field"] == "b"
@@ -444,19 +438,20 @@ def test_plot_override_specified_data_source(
run_copy_metrics(
"metric1.json",
"metric2.json",
outs_no_cache=["metric2.json"],
plots_no_cache=["metric2.json"],
commit="init",
tag="v1",
)
plot_string = dvc.plots.show(
targets=["metric2.json"], template="newtemplate.json", x_field="a"
)["metric2.json"]
props = {"template": "newtemplate.json", "x": "a"}
plot_string = dvc.plots.show(targets=["metric2.json"], props=props)[
"metric2.json"
]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"a": 1, "b": 2, "rev": "working tree"},
{"a": 2, "b": 3, "rev": "working tree"},
{"a": 1, "b": 2, "rev": "workspace"},
{"a": 2, "b": 3, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == "a"
assert plot_content["encoding"]["y"]["field"] == "b"
@@ -467,18 +462,35 @@ def test_should_raise_on_no_template_and_datafile(tmp_dir, dvc):
dvc.plots.show()
def test_should_raise_on_no_template(tmp_dir, dvc):
def test_should_raise_on_no_template(tmp_dir, dvc, run_copy_metrics):
metric = [{"val": 2}, {"val": 3}]
_write_json(tmp_dir, metric, "metric_t.json")
run_copy_metrics(
"metric_t.json",
"metric.json",
plots_no_cache=["metric.json"],
commit="first run",
)
with pytest.raises(TemplateNotFoundError):
dvc.plots.show("metric.json", template="non_existing_template.json")
props = {"template": "non_existing_template.json"}
dvc.plots.show("metric.json", props=props)
def test_plot_no_data(tmp_dir, dvc):
with pytest.raises(NoDataForTemplateError):
dvc.plots.show(template="default")
dvc.plots.show(props={"template": "default"})
def test_plot_wrong_metric_type(tmp_dir, scm, dvc):
tmp_dir.scm_gen("metric.txt", "content", commit="initial")
def test_plot_wrong_metric_type(tmp_dir, scm, dvc, run_copy_metrics):
tmp_dir.gen("metric_t.txt", "some text")
run_copy_metrics(
"metric_t.txt",
"metric.txt",
plots_no_cache=["metric.txt"],
commit="add text metric",
)
with pytest.raises(PlotMetricTypeError):
dvc.plots.show(targets=["metric.txt"])
@@ -496,17 +508,18 @@ def test_plot_choose_columns(
tag="v1",
)
plot_string = dvc.plots.show(
template=os.fspath(custom_template),
fields={"b", "c"},
x_field="b",
y_field="c",
)["metric.json"]
props = {
"template": os.fspath(custom_template),
"fields": {"b", "c"},
"x": "b",
"y": "c",
}
plot_string = dvc.plots.show(props=props)["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"b": 2, "c": 3, "rev": "working tree"},
{"b": 3, "c": 4, "rev": "working tree"},
{"b": 2, "c": 3, "rev": "workspace"},
{"b": 3, "c": 4, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == "b"
assert plot_content["encoding"]["y"]["field"] == "c"
@@ -523,12 +536,12 @@ def test_plot_default_choose_column(tmp_dir, scm, dvc, run_copy_metrics):
tag="v1",
)
plot_string = dvc.plots.show(fields={"b"})["metric.json"]
plot_string = dvc.plots.show(props={"fields": {"b"}})["metric.json"]
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{PlotData.INDEX_FIELD: 0, "b": 2, "rev": "working tree"},
{PlotData.INDEX_FIELD: 1, "b": 3, "rev": "working tree"},
{PlotData.INDEX_FIELD: 0, "b": 2, "rev": "workspace"},
{PlotData.INDEX_FIELD: 1, "b": 3, "rev": "workspace"},
]
assert plot_content["encoding"]["x"]["field"] == PlotData.INDEX_FIELD
assert plot_content["encoding"]["y"]["field"] == "b"
@@ -547,8 +560,8 @@ def test_plot_yaml(tmp_dir, scm, dvc, run_copy_metrics):
plot_content = json.loads(plot_string)
assert plot_content["data"]["values"] == [
{"val": 2, PlotData.INDEX_FIELD: 0, "rev": "working tree"},
{"val": 3, PlotData.INDEX_FIELD: 1, "rev": "working tree"},
{"val": 2, PlotData.INDEX_FIELD: 0, "rev": "workspace"},
{"val": 3, PlotData.INDEX_FIELD: 1, "rev": "workspace"},
]
@@ -563,7 +576,7 @@ def test_raise_on_wrong_field(tmp_dir, scm, dvc, run_copy_metrics):
)
with pytest.raises(NoFieldInDataError):
dvc.plots.show("metric.json", x_field="no_val")
dvc.plots.show("metric.json", props={"x": "no_val"})
with pytest.raises(NoFieldInDataError):
dvc.plots.show("metric.json", y_field="no_val")
dvc.plots.show("metric.json", props={"y": "no_val"})
+8 -9
View File
@@ -17,7 +17,7 @@ from dvc.exceptions import (
OutputDuplicationError,
OverlappingOutputPathsError,
RecursiveAddingWhileUsingFilename,
StageFileCorruptedError,
YAMLFileCorruptedError,
)
from dvc.main import main
from dvc.output.base import OutputAlreadyTrackedError, OutputIsStageFileError
@@ -27,7 +27,7 @@ from dvc.stage import Stage
from dvc.system import System
from dvc.utils import LARGE_DIR_SIZE, file_md5, relpath
from dvc.utils.fs import path_isin
from dvc.utils.stage import load_stage_file
from dvc.utils.yaml import load_yaml
from tests.basic_env import TestDvc
from tests.utils import get_gitignore_content
@@ -46,7 +46,7 @@ def test_add(tmp_dir, dvc):
assert stage.outs[0].info["md5"] == md5
assert stage.md5 is None
assert load_stage_file("foo.dvc") == {
assert load_yaml("foo.dvc") == {
"outs": [{"md5": "acbd18db4cc2f85cedef654fccc4a4d8", "path": "foo"}],
}
@@ -220,14 +220,14 @@ class TestAddLocalRemoteFile(TestDvc):
ret = main(["add", foo])
self.assertEqual(ret, 0)
d = load_stage_file("foo.dvc")
d = load_yaml("foo.dvc")
self.assertEqual(d["outs"][0]["path"], foo)
bar = os.path.join(cwd, self.BAR)
ret = main(["add", bar])
self.assertEqual(ret, 0)
d = load_stage_file("bar.dvc")
d = load_yaml("bar.dvc")
self.assertEqual(d["outs"][0]["path"], self.BAR)
@@ -371,7 +371,7 @@ class SymlinkAddTestBase(TestDvc):
stage_file = self.data_file_name + DVC_FILE_SUFFIX
self.assertTrue(os.path.exists(stage_file))
d = load_stage_file(stage_file)
d = load_yaml(stage_file)
relative_data_path = posixpath.join(
self.link_name, self.data_file_name
)
@@ -436,8 +436,7 @@ class TestShouldThrowProperExceptionOnCorruptedStageFile(TestDvc):
assert 1 == ret
expected_error = (
"unable to read DVC-file: {} "
"YAML file structure is corrupted".format(foo_stage)
f"unable to read: '{foo_stage}', YAML file structure is corrupted"
)
assert expected_error in self._caplog.text
@@ -478,7 +477,7 @@ def test_failed_add_cleanup(tmp_dir, scm, dvc):
dvc.add("foo")
tmp_dir.gen("foo.dvc", "- broken\nyaml")
with pytest.raises(StageFileCorruptedError):
with pytest.raises(YAMLFileCorruptedError):
dvc.add("bar")
assert not os.path.exists("bar.dvc")
+4 -4
View File
@@ -24,7 +24,7 @@ from dvc.stage.exceptions import StageFileDoesNotExistError
from dvc.system import System
from dvc.utils import relpath
from dvc.utils.fs import walk_files
from dvc.utils.stage import dump_stage_file, load_stage_file
from dvc.utils.yaml import dump_yaml, load_yaml
from tests.basic_env import TestDvc, TestDvcGit
from tests.func.test_repro import TestRepro
from tests.remotes import S3
@@ -221,7 +221,7 @@ class TestCheckoutSelectiveRemove(CheckoutBase):
self.assertEqual(0, ret)
stage_path = self.DATA_DIR + DVC_FILE_SUFFIX
stage = load_stage_file(stage_path)
stage = load_yaml(stage_path)
staged_files = self.outs_info(stage)
# move instead of remove, to lock inode assigned to stage_files[0].path
@@ -304,10 +304,10 @@ class TestGitIgnoreWhenCheckout(CheckoutBase):
class TestCheckoutMissingMd5InStageFile(TestRepro):
def test(self):
d = load_stage_file(self.file1_stage)
d = load_yaml(self.file1_stage)
del d[Stage.PARAM_OUTS][0][LocalRemote.PARAM_CHECKSUM]
del d[Stage.PARAM_DEPS][0][LocalRemote.PARAM_CHECKSUM]
dump_stage_file(self.file1_stage, d)
dump_yaml(self.file1_stage, d)
with pytest.raises(CheckoutError):
self.dvc.checkout(force=True)
+4 -4
View File
@@ -2,7 +2,7 @@ import pytest
from dvc.dvcfile import PIPELINE_FILE
from dvc.stage.exceptions import StageCommitError
from dvc.utils.stage import dump_stage_file, load_stage_file
from dvc.utils.yaml import dump_yaml, load_yaml
def test_commit_recursive(tmp_dir, dvc):
@@ -63,15 +63,15 @@ def test_commit_changed_md5(tmp_dir, dvc):
tmp_dir.gen({"file": "file content"})
(stage,) = dvc.add("file", no_commit=True)
stage_file_content = load_stage_file(stage.path)
stage_file_content = load_yaml(stage.path)
stage_file_content["md5"] = "1111111111"
dump_stage_file(stage.path, stage_file_content)
dump_yaml(stage.path, stage_file_content)
with pytest.raises(StageCommitError):
dvc.commit(stage.path)
dvc.commit(stage.path, force=True)
assert "md5" not in load_stage_file(stage.path)
assert "md5" not in load_yaml(stage.path)
def test_commit_no_exec(tmp_dir, dvc):
+3 -3
View File
@@ -27,7 +27,7 @@ from dvc.remote.base import STATUS_DELETED, STATUS_NEW, STATUS_OK
from dvc.stage.exceptions import StageNotFound
from dvc.utils import file_md5
from dvc.utils.fs import remove
from dvc.utils.stage import dump_stage_file, load_stage_file
from dvc.utils.yaml import dump_yaml, load_yaml
from tests.basic_env import TestDvc
from tests.remotes import (
GCP,
@@ -523,9 +523,9 @@ class TestWarnOnOutdatedStage(TestDvc):
self.main(["push"])
stage_file_path = stage.relpath
content = load_stage_file(stage_file_path)
content = load_yaml(stage_file_path)
del content["outs"][0]["md5"]
dump_stage_file(stage_file_path, content)
dump_yaml(stage_file_path, content)
with self._caplog.at_level(logging.WARNING, logger="dvc"):
self._caplog.clear()
+132 -2
View File
@@ -1,8 +1,14 @@
import textwrap
import pytest
from dvc.dvcfile import PIPELINE_FILE, Dvcfile
from dvc.stage.exceptions import StageFileDoesNotExistError
from dvc.dvcfile import PIPELINE_FILE, PIPELINE_LOCK, Dvcfile
from dvc.stage.exceptions import (
StageFileDoesNotExistError,
StageFileFormatError,
)
from dvc.stage.loader import StageNotFound
from dvc.utils.yaml import dump_yaml
def test_run_load_one_for_multistage(tmp_dir, dvc):
@@ -160,3 +166,127 @@ def test_stage_collection(tmp_dir, dvc):
single_stage=True,
)
assert {s for s in dvc.stages} == {stage1, stage3, stage2}
def test_remove_stage(tmp_dir, dvc, run_copy):
tmp_dir.gen("foo", "foo")
stage = run_copy("foo", "bar", name="copy-foo-bar")
stage2 = run_copy("bar", "foobar", name="copy-bar-foobar")
dvc_file = Dvcfile(dvc, PIPELINE_FILE)
assert dvc_file.exists()
assert {"copy-bar-foobar", "copy-foo-bar"} == set(
dvc_file._load()[0]["stages"].keys()
)
dvc_file.remove_stage(stage)
assert ["copy-bar-foobar"] == list(dvc_file._load()[0]["stages"].keys())
# sanity check
stage2.reload()
# re-check to see if it fails if there's no stage entry
dvc_file.remove_stage(stage)
dvc_file.remove(force=True)
# should not fail when there's no file at all.
dvc_file.remove_stage(stage)
def test_remove_stage_lockfile(tmp_dir, dvc, run_copy):
tmp_dir.gen("foo", "foo")
stage = run_copy("foo", "bar", name="copy-foo-bar")
stage2 = run_copy("bar", "foobar", name="copy-bar-foobar")
dvc_file = Dvcfile(dvc, PIPELINE_FILE)
lock_file = dvc_file._lockfile
assert dvc_file.exists()
assert lock_file.exists()
assert {"copy-bar-foobar", "copy-foo-bar"} == set(lock_file.load().keys())
lock_file.remove_stage(stage)
assert ["copy-bar-foobar"] == list(lock_file.load().keys())
# sanity check
stage2.reload()
# re-check to see if it fails if there's no stage entry
lock_file.remove_stage(stage)
lock_file.remove()
# should not fail when there's no file at all.
lock_file.remove_stage(stage)
def test_remove_stage_dvcfiles(tmp_dir, dvc, run_copy):
tmp_dir.gen("foo", "foo")
stage = run_copy("foo", "bar", single_stage=True)
dvc_file = Dvcfile(dvc, stage.path)
assert dvc_file.exists()
dvc_file.remove_stage(stage)
assert not dvc_file.exists()
# re-check to see if it fails if there's no stage entry
dvc_file.remove_stage(stage)
dvc_file.remove(force=True)
# should not fail when there's no file at all.
dvc_file.remove_stage(stage)
def test_remove_stage_on_lockfile_format_error(tmp_dir, dvc, run_copy):
tmp_dir.gen("foo", "foo")
stage = run_copy("foo", "bar", name="copy-foo-bar")
dvc_file = Dvcfile(dvc, stage.path)
lock_file = dvc_file._lockfile
data = dvc_file._load()[0]
lock_data = lock_file.load()
lock_data["gibberish"] = True
data["gibberish"] = True
dump_yaml(lock_file.relpath, lock_data)
with pytest.raises(StageFileFormatError):
dvc_file.remove_stage(stage)
lock_file.remove()
dvc_file.dump(stage)
dump_yaml(dvc_file.relpath, data)
with pytest.raises(StageFileFormatError):
dvc_file.remove_stage(stage)
def test_remove_stage_preserves_comment(tmp_dir, dvc, run_copy):
tmp_dir.gen(
"dvc.yaml",
textwrap.dedent(
"""\
stages:
generate-foo:
cmd: "echo foo > foo"
# This copies 'foo' text to 'foo' file.
outs:
- foo
copy-foo-bar:
cmd: "python copy.py foo bar"
deps:
- foo
outs:
- bar"""
),
)
dvc.reproduce(PIPELINE_FILE)
dvc_file = Dvcfile(dvc, PIPELINE_FILE)
assert dvc_file.exists()
assert (tmp_dir / PIPELINE_LOCK).exists()
assert (tmp_dir / "foo").exists()
assert (tmp_dir / "bar").exists()
dvc_file.remove_stage(dvc_file.stages["copy-foo-bar"])
assert (
"# This copies 'foo' text to 'foo' file."
in (tmp_dir / PIPELINE_FILE).read_text()
)
+3 -3
View File
@@ -68,7 +68,7 @@ class TestGCBranchesTags(TestDvcGit):
self.dvc.scm.tag("v1.0")
self.dvc.scm.checkout("test", create_new=True)
self.dvc.remove(stages[0].relpath, dvc_only=True)
self.dvc.remove(stages[0].relpath)
with open(fname, "w+") as fobj:
fobj.write("test")
stages = self.dvc.add(fname)
@@ -77,7 +77,7 @@ class TestGCBranchesTags(TestDvcGit):
self.dvc.scm.commit("test")
self.dvc.scm.checkout("master")
self.dvc.remove(stages[0].relpath, dvc_only=True)
self.dvc.remove(stages[0].relpath)
with open(fname, "w+") as fobj:
fobj.write("trash")
stages = self.dvc.add(fname)
@@ -85,7 +85,7 @@ class TestGCBranchesTags(TestDvcGit):
self.dvc.scm.add([".gitignore", stages[0].relpath])
self.dvc.scm.commit("trash")
self.dvc.remove(stages[0].relpath, dvc_only=True)
self.dvc.remove(stages[0].relpath)
with open(fname, "w+") as fobj:
fobj.write("master")
stages = self.dvc.add(fname)
+18
View File
@@ -11,6 +11,7 @@ from dvc.ignore import (
DvcIgnoreRepo,
)
from dvc.remote import LocalRemote
from dvc.repo import Repo
from dvc.scm.tree import WorkingTree
from dvc.utils import relpath
from dvc.utils.fs import get_mtime_and_size
@@ -151,3 +152,20 @@ def test_ignore_external(tmp_dir, scm, dvc, tmp_path_factory):
remote = LocalRemote(dvc, {})
result = {relpath(f, ext_dir) for f in remote.walk_files(ext_dir)}
assert result == {"y.backup", "tmp"}
def test_ignore_subrepo(tmp_dir, scm, dvc):
tmp_dir.gen({".dvcignore": "foo", "subdir": {"foo": "foo"}})
scm.add([".dvcignore"])
scm.commit("init parent dvcignore")
subrepo_dir = tmp_dir / "subdir"
assert not dvc.tree.exists(subrepo_dir / "foo")
with subrepo_dir.chdir():
subrepo = Repo.init(subdir=True)
scm.add(str(subrepo_dir / "foo"))
scm.commit("subrepo init")
for _ in subrepo.brancher(all_commits=True):
assert subrepo.tree.exists(subrepo_dir / "foo")
+2 -2
View File
@@ -6,7 +6,7 @@ import yaml
from dvc.dvcfile import PIPELINE_LOCK
from dvc.serialize import get_params_deps
from dvc.utils.fs import remove
from dvc.utils.stage import parse_stage_for_update
from dvc.utils.yaml import parse_yaml_for_update
from tests.func.test_run_multistage import supported_params
FS_STRUCTURE = {
@@ -20,7 +20,7 @@ FS_STRUCTURE = {
def read_lock_file(file=PIPELINE_LOCK):
with open(file) as f:
data = parse_stage_for_update(f.read(), file)
data = parse_yaml_for_update(f.read(), file)
assert isinstance(data, OrderedDict)
return data
+3 -3
View File
@@ -3,7 +3,7 @@ import os
from dvc.dvcfile import DVC_FILE_SUFFIX
from dvc.exceptions import DvcException, MoveNotDataSourceError
from dvc.main import main
from dvc.utils.stage import load_stage_file
from dvc.utils.yaml import load_yaml
from tests.basic_env import TestDvc, TestDvcGit
from tests.func.test_repro import TestRepro
from tests.utils import cd
@@ -117,7 +117,7 @@ class TestMoveFileToDirectoryWithoutSpecifiedTargetName(TestDvc):
self.assertTrue(os.path.exists(target_foo_path))
self.assertTrue(os.path.exists(target_foo_stage_file_path))
new_stage = load_stage_file(target_foo_stage_file_path)
new_stage = load_yaml(target_foo_stage_file_path)
self.assertEqual(self.FOO, new_stage["outs"][0]["path"])
@@ -170,7 +170,7 @@ class TestMoveFileBetweenDirectories(TestDvc):
self.assertTrue(os.path.exists(new_data_path))
self.assertTrue(os.path.exists(new_data_stage_file))
new_stage_file = load_stage_file(new_data_stage_file)
new_stage_file = load_yaml(new_data_stage_file)
self.assertEqual(
os.path.basename(self.DATA), new_stage_file["outs"][0]["path"]
)
-354
View File
@@ -1,354 +0,0 @@
import logging
from dvc.command.pipeline import CmdPipelineList, CmdPipelineShow
from dvc.main import main
from tests.basic_env import TestDvc
from tests.func.test_repro import TestRepro, TestReproChangedDeepData
class TestPipelineShowSingle(TestDvc):
def setUp(self):
super().setUp()
self.stage = "foo.dvc"
ret = main(["add", self.FOO])
self.assertEqual(ret, 0)
def test(self):
ret = main(["pipeline", "show", self.stage])
self.assertEqual(ret, 0)
def test_commands(self):
ret = main(["pipeline", "show", self.stage, "--commands"])
self.assertEqual(ret, 0)
def test_outs(self):
ret = main(["pipeline", "show", self.stage, "--outs"])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(["pipeline", "show", "--dot", self.stage])
self.assertEqual(ret, 0)
def test_tree(self):
ret = main(["pipeline", "show", "--tree", self.stage])
self.assertEqual(ret, 0)
def test_ascii_outs(self):
ret = main(["pipeline", "show", "--ascii", self.stage, "--outs"])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(["pipeline", "show", "--dot", self.stage, "--commands"])
self.assertEqual(ret, 0)
def test_dot_outs(self):
ret = main(["pipeline", "show", "--dot", self.stage, "--outs"])
self.assertEqual(ret, 0)
def test_not_dvc_file(self):
ret = main(["pipeline", "show", self.FOO])
self.assertNotEqual(ret, 0)
def test_non_existing(self):
ret = main(["pipeline", "show", "non-existing"])
self.assertNotEqual(ret, 0)
def test_single_ascii(tmp_dir, dvc):
tmp_dir.dvc_gen("foo", "foo content")
assert main(["pipeline", "show", "--ascii", "foo.dvc"]) == 0
def test_single_ascii_commands(tmp_dir, dvc):
tmp_dir.dvc_gen("foo", "foo content")
assert main(["pipeline", "show", "--ascii", "foo.dvc", "--commands"]) == 0
class TestPipelineShow(TestRepro):
def test(self):
ret = main(["pipeline", "show", self.file1_stage])
self.assertEqual(ret, 0)
def test_commands(self):
ret = main(["pipeline", "show", self.file1_stage, "--commands"])
self.assertEqual(ret, 0)
def test_ascii(self):
ret = main(["pipeline", "show", "--ascii", self.file1_stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(["pipeline", "show", "--dot", self.file1_stage])
self.assertEqual(ret, 0)
def test_ascii_commands(self):
ret = main(
["pipeline", "show", "--ascii", self.file1_stage, "--commands"]
)
self.assertEqual(ret, 0)
def test_ascii_outs(self):
ret = main(["pipeline", "show", "--ascii", self.file1_stage, "--outs"])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(
["pipeline", "show", "--dot", self.file1_stage, "--commands"]
)
self.assertEqual(ret, 0)
def test_disconnected_stage(tmp_dir, dvc):
tmp_dir.dvc_gen({"base": "base"})
dvc.add("base")
dvc.run(
deps=["base"],
outs=["derived1"],
cmd="echo derived1 > derived1",
single_stage=True,
)
dvc.run(
deps=["base"],
outs=["derived2"],
cmd="echo derived2 > derived2",
single_stage=True,
)
final_stage = dvc.run(
deps=["derived1"],
outs=["final"],
cmd="echo final > final",
single_stage=True,
)
command = CmdPipelineShow([])
# Need to test __build_graph directly
nodes, edges, is_tree = command._build_graph(
final_stage.path, commands=False, outs=True
)
assert set(nodes) == {"final", "derived1", "base"}
assert edges == [("final", "derived1"), ("derived1", "base")]
assert is_tree is True
def test_print_locked_stages(tmp_dir, dvc, caplog):
tmp_dir.dvc_gen({"foo": "foo content", "bar": "bar content"})
dvc.lock_stage("foo.dvc")
caplog.clear()
with caplog.at_level(logging.INFO, logger="dvc"):
assert main(["pipeline", "show", "foo.dvc", "--locked"]) == 0
assert "foo.dvc" in caplog.text
assert "bar.dvc" not in caplog.text
def test_dot_outs(tmp_dir, dvc, run_copy):
tmp_dir.gen("foo", "foo content")
run_copy("foo", "file", single_stage=True)
assert main(["pipeline", "show", "--dot", "file.dvc", "--outs"]) == 0
class TestPipelineShowOuts(TestRepro):
def setUp(self):
super().setUp()
def test_outs(self):
ret = main(["pipeline", "show", self.file1_stage, "--outs"])
self.assertEqual(ret, 0)
class TestPipelineShowDeep(TestReproChangedDeepData):
def test(self):
ret = main(["pipeline", "show", self.file1_stage])
self.assertEqual(ret, 0)
def test_commands(self):
ret = main(["pipeline", "show", self.file1_stage, "--commands"])
self.assertEqual(ret, 0)
def test_outs(self):
ret = main(["pipeline", "show", self.file1_stage, "--outs"])
self.assertEqual(ret, 0)
def test_ascii(self):
ret = main(["pipeline", "show", "--ascii", self.file1_stage])
self.assertEqual(ret, 0)
def test_dot(self):
ret = main(["pipeline", "show", "--dot", self.file1_stage])
self.assertEqual(ret, 0)
def test_ascii_commands(self):
ret = main(
["pipeline", "show", "--ascii", self.file1_stage, "--commands"]
)
self.assertEqual(ret, 0)
def test_ascii_outs(self):
ret = main(["pipeline", "show", "--ascii", self.file1_stage, "--outs"])
self.assertEqual(ret, 0)
def test_dot_commands(self):
ret = main(
["pipeline", "show", "--dot", self.file1_stage, "--commands"]
)
self.assertEqual(ret, 0)
def test_dot_outs(self):
ret = main(["pipeline", "show", "--dot", self.file1_stage, "--outs"])
self.assertEqual(ret, 0)
class TestPipelineListEmpty(TestDvc):
def test(self):
ret = main(["pipeline", "list"])
self.assertEqual(ret, 0)
class TestPipelineListSingle(TestPipelineShowDeep):
def test(self):
ret = main(["pipeline", "list"])
self.assertEqual(ret, 0)
class TestDvcRepoPipeline(TestDvc):
def test_no_stages(self):
pipelines = self.dvc.pipelines
self.assertEqual(len(pipelines), 0)
def one_pipeline(self):
self.dvc.add("foo")
self.dvc.run(deps=["foo"], outs=["bar"], cmd="", single_stage=True)
self.dvc.run(
deps=["bar"], outs=["baz"], cmd="echo baz > baz", single_stage=True
)
pipelines = self.dvc.pipelines
self.assertEqual(len(pipelines), 1)
self.assertEqual(pipelines[0].nodes, 3)
self.assertEqual(pipelines[0].edges, 2)
def two_pipelines(self):
self.dvc.add("foo")
self.dvc.run(deps=["foo"], outs=["bar"], cmd="", single_stage=True)
self.dvc.run(
deps=["bar"], outs=["baz"], cmd="echo baz > baz", single_stage=True
)
self.dvc.add("code.py")
pipelines = self.dvc.pipelines
self.assertEqual(len(pipelines), 2)
self.assertEqual(pipelines[0].nodes, 3)
self.assertEqual(pipelines[0].edges, 2)
self.assertEqual(pipelines[1].nodes, 1)
self.assertEqual(pipelines[1].edges, 0)
def locked_stage(self):
self.dvc.add("foo")
self.dvc.lock_stage("foo.dvc")
pipelines = self.dvc.pipelines
self.assertEqual(len(pipelines), 0)
def test_split_pipeline(tmp_dir, scm, dvc):
tmp_dir.scm_gen("git_dep1", "git_dep1")
tmp_dir.scm_gen("git_dep2", "git_dep2")
tmp_dir.dvc_gen("data", "source file content")
dvc.run(
deps=["git_dep1", "data"],
outs=["data_train", "data_valid"],
cmd="echo train >> data_train && echo valid >> data_valid",
single_stage=True,
)
stage = dvc.run(
deps=["git_dep2", "data_train", "data_valid"],
outs=["result"],
cmd="echo result >> result",
single_stage=True,
)
command = CmdPipelineShow([])
nodes, edges, _ = command._build_graph(
stage.path, commands=False, outs=True
)
assert set(nodes) == {"data", "data_train", "data_valid", "result"}
assert set(edges) == {
("result", "data_train"),
("result", "data_valid"),
("data_train", "data"),
("data_valid", "data"),
}
def test_pipeline_list_show_multistage(tmp_dir, dvc, run_copy, caplog):
tmp_dir.gen("foo", "foo")
run_copy("foo", "bar", name="copy-foo-bar")
run_copy("bar", "foobar", single_stage=True)
command = CmdPipelineShow([])
caplog.clear()
with caplog.at_level(logging.INFO, "dvc"):
command._show("foobar.dvc", False, False, False)
output = caplog.text.splitlines()
assert "copy-foo-bar" in output[0]
assert "foobar.dvc" in output[1]
caplog.clear()
with caplog.at_level(logging.INFO, "dvc"):
command._show("copy-foo-bar", False, False, False)
assert "copy-foo-bar" in caplog.text
assert "foobar.dvc" not in caplog.text
command = CmdPipelineList([])
caplog.clear()
with caplog.at_level(logging.INFO, "dvc"):
command.run()
assert "copy-foo-bar" in caplog.text
assert "foobar.dvc" in caplog.text
assert "1 pipelines in total"
def test_pipeline_ascii_multistage(tmp_dir, dvc, run_copy):
tmp_dir.gen("foo", "foo")
run_copy("foo", "bar", name="copy-foo-bar")
run_copy("bar", "foobar", single_stage=True)
command = CmdPipelineShow([])
nodes, edges, _ = command._build_graph("foobar.dvc")
assert set(nodes) == {"copy-foo-bar", "foobar.dvc"}
assert set(edges) == {
("foobar.dvc", "copy-foo-bar"),
}
nodes, *_ = command._build_graph("copy-foo-bar")
assert set(nodes) == {"copy-foo-bar"}
def test_pipeline_multi_outputs_stages(dvc):
dvc.run(
outs=["alice", "bob"],
cmd="echo alice>alice && echo bob>bob",
single_stage=True,
)
dvc.run(
deps=["alice"],
outs=["mary", "mike"],
cmd="echo mary>mary && echo mike>mike",
single_stage=True,
)
stage = dvc.run(
deps=["mary"],
outs=["carol"],
cmd="echo carol>carol",
single_stage=True,
)
command = CmdPipelineShow([])
nodes, edges, _ = command._build_graph(stage.path, outs=True)
assert set(nodes) == {"alice", "mary", "carol"}
assert set(edges) == {("carol", "mary"), ("mary", "alice")}
+42 -107
View File
@@ -1,129 +1,64 @@
import os
from mock import patch
import pytest
from dvc.exceptions import DvcException
from dvc.main import main
from dvc.stage import Stage
from dvc.stage.exceptions import StageFileDoesNotExistError
from dvc.system import System
from dvc.utils.fs import remove
from tests.basic_env import TestDvc
class TestRemove(TestDvc):
def test(self):
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
(stage_removed,) = self.dvc.remove(stage.path, dvc_only=True)
@pytest.mark.parametrize("remove_outs", [True, False])
def test_remove(tmp_dir, dvc, run_copy, remove_outs):
(stage1,) = tmp_dir.dvc_gen("foo", "foo")
stage2 = run_copy("foo", "bar", single_stage=True)
stage3 = run_copy("bar", "foobar", name="copy-bar-foobar")
self.assertIsInstance(stage_removed, Stage)
self.assertEqual(stage.path, stage_removed.path)
self.assertFalse(os.path.isfile(self.FOO))
(stage_removed,) = self.dvc.remove(stage.path)
self.assertIsInstance(stage_removed, Stage)
self.assertEqual(stage.path, stage_removed.path)
self.assertFalse(os.path.isfile(self.FOO))
self.assertFalse(os.path.exists(stage.path))
for stage in [stage1, stage2, stage3]:
dvc.remove(stage.addressing, outs=remove_outs)
out_exists = (out.exists for out in stage.outs)
assert stage not in dvc._collect_stages()
if remove_outs:
assert not any(out_exists)
else:
assert all(out_exists)
class TestRemoveNonExistentFile(TestDvc):
def test(self):
with self.assertRaises(StageFileDoesNotExistError):
self.dvc.remove("non_existent_dvc_file")
def test_remove_non_existent_file(tmp_dir, dvc):
with pytest.raises(StageFileDoesNotExistError):
dvc.remove("non_existent_dvc_file.dvc")
with pytest.raises(StageFileDoesNotExistError):
dvc.remove("non_existent_stage_name")
class TestRemoveBrokenSymlink(TestDvc):
def test(self):
ret = main(["config", "cache.type", "symlink"])
self.assertEqual(ret, 0)
ret = main(["add", self.FOO])
self.assertEqual(ret, 0)
remove(self.dvc.cache.local.cache_dir)
self.assertTrue(System.is_symlink(self.FOO))
ret = main(["remove", self.FOO + ".dvc"])
self.assertEqual(ret, 0)
self.assertFalse(os.path.lexists(self.FOO))
class TestRemoveDirectory(TestDvc):
def test(self):
stages = self.dvc.add(self.DATA_DIR)
self.assertEqual(len(stages), 1)
stage_add = stages[0]
self.assertTrue(stage_add is not None)
(stage_removed,) = self.dvc.remove(stage_add.path)
self.assertEqual(stage_add.path, stage_removed.path)
self.assertFalse(os.path.exists(self.DATA_DIR))
self.assertFalse(os.path.exists(stage_removed.path))
class TestCmdRemove(TestDvc):
def test(self):
stages = self.dvc.add(self.FOO)
self.assertEqual(len(stages), 1)
stage = stages[0]
self.assertTrue(stage is not None)
ret = main(["remove", stage.path])
self.assertEqual(ret, 0)
ret = main(["remove", "non-existing-dvc-file"])
self.assertNotEqual(ret, 0)
class TestRemovePurge(TestDvc):
def test(self):
dvcfile = self.dvc.add(self.FOO)[0].path
ret = main(["remove", "--purge", "--force", dvcfile])
self.assertEqual(ret, 0)
self.assertFalse(os.path.exists(self.FOO))
self.assertFalse(os.path.exists(dvcfile))
@patch("dvc.prompt.confirm", return_value=False)
def test_force(self, mock_prompt):
dvcfile = self.dvc.add(self.FOO)[0].path
ret = main(["remove", "--purge", dvcfile])
mock_prompt.assert_called()
self.assertEqual(ret, 1)
self.assertRaises(DvcException)
def test_remove_pipeline_outs(tmp_dir, dvc, run_copy):
from dvc.dvcfile import PIPELINE_FILE
def test_remove_broken_symlink(tmp_dir, dvc):
tmp_dir.gen("foo", "foo")
stage = run_copy("foo", "bar", name="copy-foo-bar")
dvc.cache.local.cache_types = ["symlink"]
assert main(["remove", stage.path]) == 0
assert not (tmp_dir / "bar").exists()
assert (tmp_dir / PIPELINE_FILE).exists()
(stage,) = dvc.add("foo")
remove(dvc.cache.local.cache_dir)
assert System.is_symlink("foo")
stage = run_copy("foo", "foobar", name="copy-foo-foobar")
with pytest.raises(DvcException):
dvc.remove(stage.addressing)
assert os.path.lexists("foo")
assert (tmp_dir / stage.relpath).exists()
dvc.remove(stage.addressing, outs=True)
assert not os.path.lexists("foo")
assert not (tmp_dir / stage.relpath).exists()
def test_cmd_remove(tmp_dir, dvc):
assert main(["remove", "non-existing-dvc-file"]) == 1
(stage,) = tmp_dir.dvc_gen("foo", "foo")
assert main(["remove", stage.addressing]) == 0
assert not (tmp_dir / "foobar").exists()
assert not (tmp_dir / stage.relpath).exists()
assert (tmp_dir / "foo").exists()
stage = run_copy("foo", "baz", name="copy-foo-baz")
assert main(["remove", "--purge", "-f", stage.addressing]) == 0
assert not (tmp_dir / "baz").exists()
assert (tmp_dir / "foo").exists()
assert (tmp_dir / PIPELINE_FILE).exists()
dvc.reproduce(PIPELINE_FILE)
assert main(["remove", PIPELINE_FILE]) == 0
for file in ["bar", "foobar", "baz"]:
assert not (tmp_dir / file).exists()
assert (tmp_dir / "foo").exists()
assert (tmp_dir / PIPELINE_FILE).exists()
(stage,) = tmp_dir.dvc_gen("foo", "foo")
assert main(["remove", stage.addressing, "--outs"]) == 0
assert not (tmp_dir / stage.relpath).exists()
assert not (tmp_dir / "foo").exists()
+67 -34
View File
@@ -33,7 +33,7 @@ from dvc.stage.exceptions import StageFileDoesNotExistError
from dvc.system import System
from dvc.utils import file_md5, relpath
from dvc.utils.fs import remove
from dvc.utils.stage import dump_stage_file, load_stage_file
from dvc.utils.yaml import dump_yaml, load_yaml
from tests.basic_env import TestDvc
from tests.remotes import (
GCP,
@@ -108,7 +108,7 @@ class TestReproCyclicGraph(SingleStageRun, TestDvc):
"deps": [{"path": "baz.txt"}],
"outs": [{"path": self.FOO}],
}
dump_stage_file("cycle.dvc", stage_dump)
dump_yaml("cycle.dvc", stage_dump)
with self.assertRaises(CyclicGraphError):
self.dvc.reproduce("cycle.dvc")
@@ -149,7 +149,7 @@ class TestReproWorkingDirectoryAsOutput(TestDvc):
"cmd": f"echo something > {output}",
"outs": [{"path": output}],
}
dump_stage_file(faulty_stage_path, stage_dump)
dump_yaml(faulty_stage_path, stage_dump)
with self.assertRaises(StagePathAsOutputError):
self.dvc.reproduce(faulty_stage_path)
@@ -188,7 +188,7 @@ class TestReproWorkingDirectoryAsOutput(TestDvc):
"cmd": f"echo something > {output}",
"outs": [{"path": output}],
}
dump_stage_file(error_stage_path, stage_dump)
dump_yaml(error_stage_path, stage_dump)
# NOTE: os.walk() walks in a sorted order and we need dir2 subdirs to
# be processed before dir1 to load error.dvc first.
@@ -220,7 +220,7 @@ class TestReproWorkingDirectoryAsOutput(TestDvc):
stage = os.path.join("something-1", "a.dvc")
stage_dump = {"cmd": "echo a > a", "outs": [{"path": "a"}]}
dump_stage_file(stage, stage_dump)
dump_yaml(stage, stage_dump)
try:
self.dvc.reproduce(stage)
@@ -554,7 +554,7 @@ class TestReproPipelines(SingleStageRun, TestDvc):
self.assertEqual(ret, 0)
class TestReproLocked(TestReproChangedData):
class TestReproFrozen(TestReproChangedData):
def test(self):
file2 = "file2"
file2_stage = self._run(
@@ -567,12 +567,12 @@ class TestReproLocked(TestReproChangedData):
self.swap_foo_with_bar()
ret = main(["lock", self._get_stage_target(file2_stage)])
ret = main(["freeze", self._get_stage_target(file2_stage)])
self.assertEqual(ret, 0)
stages = self.dvc.reproduce(self._get_stage_target(file2_stage))
self.assertEqual(len(stages), 0)
ret = main(["unlock", self._get_stage_target(file2_stage)])
ret = main(["unfreeze", self._get_stage_target(file2_stage)])
self.assertEqual(ret, 0)
stages = self.dvc.reproduce(self._get_stage_target(file2_stage))
self.assertTrue(filecmp.cmp(self.file1, self.BAR, shallow=False))
@@ -581,19 +581,19 @@ class TestReproLocked(TestReproChangedData):
def test_non_existing(self):
with self.assertRaises(StageFileDoesNotExistError):
self.dvc.lock_stage("Dvcfile")
self.dvc.lock_stage("pipelines.yaml")
self.dvc.lock_stage("pipelines.yaml:name")
self.dvc.lock_stage("Dvcfile:name")
self.dvc.lock_stage("stage.dvc")
self.dvc.lock_stage("stage.dvc:name")
self.dvc.lock_stage("not-existing-stage.json")
self.dvc.freeze("Dvcfile")
self.dvc.freeze("pipelines.yaml")
self.dvc.freeze("pipelines.yaml:name")
self.dvc.freeze("Dvcfile:name")
self.dvc.freeze("stage.dvc")
self.dvc.freeze("stage.dvc:name")
self.dvc.freeze("not-existing-stage.json")
ret = main(["lock", "non-existing-stage"])
ret = main(["freeze", "non-existing-stage"])
self.assertNotEqual(ret, 0)
class TestReproLockedCallback(SingleStageRun, TestDvc):
class TestReproFrozenCallback(SingleStageRun, TestDvc):
def test(self):
file1 = "file1"
file1_stage = file1 + ".dvc"
@@ -610,26 +610,26 @@ class TestReproLockedCallback(SingleStageRun, TestDvc):
stages = self.dvc.reproduce(self._get_stage_target(stage))
self.assertEqual(len(stages), 1)
self.dvc.lock_stage(self._get_stage_target(stage))
self.dvc.freeze(self._get_stage_target(stage))
stages = self.dvc.reproduce(self._get_stage_target(stage))
self.assertEqual(len(stages), 0)
self.dvc.lock_stage(self._get_stage_target(stage), unlock=True)
self.dvc.unfreeze(self._get_stage_target(stage))
stages = self.dvc.reproduce(self._get_stage_target(stage))
self.assertEqual(len(stages), 1)
class TestReproLockedUnchanged(TestRepro):
class TestReproFrozenUnchanged(TestRepro):
def test(self):
"""
Check that locking/unlocking doesn't affect stage state
Check that freezing/unfreezing doesn't affect stage state
"""
target = self._get_stage_target(self.stage)
self.dvc.lock_stage(target)
self.dvc.freeze(target)
stages = self.dvc.reproduce(target)
self.assertEqual(len(stages), 0)
self.dvc.lock_stage(target, unlock=True)
self.dvc.unfreeze(target)
stages = self.dvc.reproduce(target)
self.assertEqual(len(stages), 0)
@@ -656,11 +656,17 @@ class TestReproMetricsAddUnchanged(TestDvc):
stages = self.dvc.reproduce(file1_stage)
self.assertEqual(len(stages), 0)
self.dvc.metrics.add(file1)
d = load_yaml(file1_stage)
d["outs"][0]["metric"] = True
dump_yaml(file1_stage, d)
stages = self.dvc.reproduce(file1_stage)
self.assertEqual(len(stages), 0)
self.dvc.metrics.remove(file1)
d = load_yaml(file1_stage)
d["outs"][0]["metric"] = False
dump_yaml(file1_stage, d)
stages = self.dvc.reproduce(file1_stage)
self.assertEqual(len(stages), 0)
@@ -775,10 +781,10 @@ class TestReproChangedDirData(SingleStageRun, TestDvc):
class TestReproMissingMd5InStageFile(TestRepro):
def test(self):
d = load_stage_file(self.file1_stage)
d = load_yaml(self.file1_stage)
del d[Stage.PARAM_OUTS][0][LocalRemote.PARAM_CHECKSUM]
del d[Stage.PARAM_DEPS][0][LocalRemote.PARAM_CHECKSUM]
dump_stage_file(self.file1_stage, d)
dump_yaml(self.file1_stage, d)
stages = self.dvc.reproduce(self.file1_stage)
self.assertEqual(len(stages), 1)
@@ -879,9 +885,9 @@ class TestReproExternalBase(SingleStageRun, TestDvc):
with patch_download as mock_download:
with patch_checkout as mock_checkout:
with patch_run as mock_run:
stage.locked = False
stage.frozen = False
stage.run()
stage.locked = True
stage.frozen = True
mock_run.assert_not_called()
mock_download.assert_not_called()
@@ -983,7 +989,8 @@ class TestReproExternalBase(SingleStageRun, TestDvc):
self.dvc.gc(workspace=True)
self.assertEqual(self.dvc.status(), {})
self.dvc.remove(cmd_stage.path, dvc_only=True)
with self.dvc.lock:
cmd_stage.remove_outs(force=True)
self.assertNotEqual(self.dvc.status([cmd_stage.addressing]), {})
self.dvc.checkout([cmd_stage.path], force=True)
@@ -1203,12 +1210,13 @@ class TestReproExternalHTTP(TestReproExternalBase):
self.assertTrue(os.path.exists(import_output))
self.assertTrue(filecmp.cmp(import_output, self.FOO, shallow=False))
self.dvc.remove("imported_file.dvc")
self.dvc.remove("imported_file.dvc", outs=True)
with StaticFileServer(handler_class=ContentMD5Handler) as httpd:
import_url = urljoin(self.get_remote(httpd.server_port), self.FOO)
import_output = "imported_file"
import_stage = self.dvc.imp_url(import_url, import_output)
assert import_stage.repo == self.dvc
self.assertTrue(os.path.exists(import_output))
self.assertTrue(filecmp.cmp(import_output, self.FOO, shallow=False))
@@ -1225,7 +1233,7 @@ class TestReproExternalHTTP(TestReproExternalBase):
self.assertEqual(ret1, 0)
self.assertEqual(ret2, 0)
self.dvc = DvcRepo(".")
self.dvc = import_stage.repo = DvcRepo(".")
run_dependency = urljoin(remote, self.BAR)
run_output = "remote_file"
@@ -1245,7 +1253,10 @@ class TestReproExternalHTTP(TestReproExternalBase):
self.assertTrue(os.path.exists(run_output))
# Pull
self.dvc.remove(import_stage.path, dvc_only=True)
with self.dvc.lock:
self.assertEqual(import_stage.repo.lock.is_locked, True)
self.assertEqual(self.dvc.lock.is_locked, True)
import_stage.remove_outs(force=True)
self.assertFalse(os.path.exists(import_output))
shutil.move(self.local_cache, cache_id)
@@ -1387,7 +1398,7 @@ class TestReproAlreadyCached(TestRepro):
with patch_download as mock_download:
with patch_checkout as mock_checkout:
assert main(["unlock", "bar.dvc"]) == 0
assert main(["unfreeze", "bar.dvc"]) == 0
ret = main(["repro", "--force", "bar.dvc"])
self.assertEqual(ret, 0)
self.assertEqual(mock_download.call_count, 1)
@@ -1765,3 +1776,25 @@ def test_ssh_dir_out(tmp_dir, dvc, ssh_server):
repo.reproduce("dir-out.dvc")
repo.reproduce("dir-out.dvc", force=True)
def test_repro_when_cmd_changes(tmp_dir, dvc, run_copy, mocker):
from dvc.dvcfile import SingleStageFile
tmp_dir.gen("foo", "foo")
stage = run_copy("foo", "bar", single_stage=True)
assert not dvc.reproduce(stage.addressing)
from dvc.stage.run import cmd_run
m = mocker.patch("dvc.stage.run.cmd_run", wraps=cmd_run)
data = SingleStageFile(dvc, stage.path)._load()[0]
data["cmd"] = " ".join(stage.cmd.split()) # change cmd spacing by two
dump_yaml(stage.path, data)
assert dvc.status([stage.addressing]) == {
stage.addressing: ["changed checksum"]
}
assert dvc.reproduce(stage.addressing)[0] == stage
m.assert_called_once_with(stage)
+22 -18
View File
@@ -10,7 +10,7 @@ from dvc.dvcfile import PIPELINE_FILE, PIPELINE_LOCK
from dvc.exceptions import CyclicGraphError
from dvc.main import main
from dvc.stage import PipelineStage
from dvc.utils.stage import dump_stage_file, parse_stage
from dvc.utils.yaml import dump_yaml, parse_yaml
from tests.func import test_repro
COPY_SCRIPT_FORMAT = dedent(
@@ -101,18 +101,18 @@ class TestReproPipelinesMultiStage(
pass
class TestReproLockedMultiStage(MultiStageRun, test_repro.TestReproLocked):
class TestReproFrozenMultiStage(MultiStageRun, test_repro.TestReproFrozen):
pass
class TestReproLockedCallbackMultiStage(
MultiStageRun, test_repro.TestReproLockedCallback
class TestReproFrozenCallbackMultiStage(
MultiStageRun, test_repro.TestReproFrozenCallback
):
pass
class TestReproLockedUnchangedMultiStage(
MultiStageRun, test_repro.TestReproLockedUnchanged
class TestReproFrozenUnchangedMultiStage(
MultiStageRun, test_repro.TestReproFrozenUnchanged
):
pass
@@ -202,9 +202,9 @@ def test_non_existing_stage_name(tmp_dir, dvc, run_copy):
run_copy("file1", "file2", name="copy-file1-file2")
with pytest.raises(DvcException):
dvc.lock_stage(":copy-file1-file3")
dvc.freeze(":copy-file1-file3")
assert main(["lock", ":copy-file1-file3"]) != 0
assert main(["freeze", ":copy-file1-file3"]) != 0
def test_downstream(tmp_dir, dvc):
@@ -307,7 +307,7 @@ def test_downstream(tmp_dir, dvc):
)
def test_repro_when_cmd_changes(tmp_dir, dvc, run_copy):
def test_repro_when_cmd_changes(tmp_dir, dvc, run_copy, mocker):
from dvc.dvcfile import PipelineFile
tmp_dir.gen("foo", "foo")
@@ -315,11 +315,15 @@ def test_repro_when_cmd_changes(tmp_dir, dvc, run_copy):
target = "copy-file"
assert not dvc.reproduce(target)
from dvc.stage.run import cmd_run
m = mocker.patch("dvc.stage.run.cmd_run", wraps=cmd_run)
stage.cmd = " ".join(stage.cmd.split()) # change cmd spacing by two
PipelineFile(dvc, PIPELINE_FILE)._dump_pipeline_file(stage)
assert dvc.status([target]) == {target: ["changed command"]}
assert dvc.reproduce(target)[0] == stage
m.assert_called_once_with(stage)
def test_repro_when_new_deps_is_added_in_dvcfile(tmp_dir, dvc, run_copy):
@@ -339,7 +343,7 @@ def test_repro_when_new_deps_is_added_in_dvcfile(tmp_dir, dvc, run_copy):
dvcfile = Dvcfile(dvc, stage.path)
data, _ = dvcfile._load()
data["stages"]["copy-file"]["deps"] += ["copy.py"]
dump_stage_file(stage.path, data)
dump_yaml(stage.path, data)
assert dvc.reproduce(target)[0] == stage
@@ -361,7 +365,7 @@ def test_repro_when_new_outs_is_added_in_dvcfile(tmp_dir, dvc):
dvcfile = Dvcfile(dvc, stage.path)
data, _ = dvcfile._load()
data["stages"]["copy-file"]["outs"] = ["foobar"]
dump_stage_file(stage.path, data)
dump_yaml(stage.path, data)
assert dvc.reproduce(target)[0] == stage
@@ -388,7 +392,7 @@ def test_repro_when_new_deps_is_moved(tmp_dir, dvc):
dvcfile = Dvcfile(dvc, stage.path)
data, _ = dvcfile._load()
data["stages"]["copy-file"]["deps"] = ["bar"]
dump_stage_file(stage.path, data)
dump_yaml(stage.path, data)
assert dvc.reproduce(target)[0] == stage
@@ -398,7 +402,7 @@ def test_repro_when_new_out_overlaps_others_stage_outs(tmp_dir, dvc):
tmp_dir.gen({"dir": {"file1": "file1"}, "foo": "foo"})
dvc.add("dir")
dump_stage_file(
dump_yaml(
PIPELINE_FILE,
{
"stages": {
@@ -419,7 +423,7 @@ def test_repro_when_new_deps_added_does_not_exist(tmp_dir, dvc):
tmp_dir.gen("copy.py", COPY_SCRIPT)
tmp_dir.gen("foo", "foo")
dump_stage_file(
dump_yaml(
PIPELINE_FILE,
{
"stages": {
@@ -440,7 +444,7 @@ def test_repro_when_new_outs_added_does_not_exist(tmp_dir, dvc):
tmp_dir.gen("copy.py", COPY_SCRIPT)
tmp_dir.gen("foo", "foo")
dump_stage_file(
dump_yaml(
PIPELINE_FILE,
{
"stages": {
@@ -459,7 +463,7 @@ def test_repro_when_new_outs_added_does_not_exist(tmp_dir, dvc):
def test_repro_when_lockfile_gets_deleted(tmp_dir, dvc):
tmp_dir.gen("copy.py", COPY_SCRIPT)
tmp_dir.gen("foo", "foo")
dump_stage_file(
dump_yaml(
PIPELINE_FILE,
{
"stages": {
@@ -491,13 +495,13 @@ def test_cyclic_graph_error(tmp_dir, dvc, run_copy):
run_copy("baz", "foobar", name="copy-baz-foobar")
with open(PIPELINE_FILE) as f:
data = parse_stage(f.read(), PIPELINE_FILE)
data = parse_yaml(f.read(), PIPELINE_FILE)
data["stages"]["copy-baz-foo"] = {
"cmd": "echo baz > foo",
"deps": ["baz"],
"outs": ["foo"],
}
dump_stage_file(PIPELINE_FILE, data)
dump_yaml(PIPELINE_FILE, data)
with pytest.raises(CyclicGraphError):
dvc.reproduce(":copy-baz-foo")
+2 -1
View File
@@ -218,7 +218,8 @@ def test_run_already_exists(tmp_dir, dvc, run_copy):
tmp_dir.dvc_gen("foo", "foo")
run_copy("foo", "bar", name="copy")
with pytest.raises(DuplicateStageName):
run_copy("bar", "foobar", name="copy")
run_copy("bar", "foobar", name="copy", overwrite=False)
run_copy("bar", "foobar", name="copy", overwrite=True)
supported_params = {
+19 -18
View File
@@ -31,7 +31,7 @@ from dvc.stage.exceptions import (
)
from dvc.system import System
from dvc.utils import file_md5
from dvc.utils.stage import load_stage_file
from dvc.utils.yaml import load_yaml
from tests.basic_env import TestDvc, TestDvcGit
@@ -350,7 +350,7 @@ class TestRunUnprotectOutsCopy(TestDvc):
ret = main(
[
"run",
"--overwrite-dvcfile",
"--overwrite",
"--no-run-cache",
"--single-stage",
"-d",
@@ -408,7 +408,7 @@ class TestRunUnprotectOutsSymlink(TestDvc):
ret = main(
[
"run",
"--overwrite-dvcfile",
"--overwrite",
"--no-run-cache",
"--single-stage",
"-d",
@@ -467,7 +467,7 @@ class TestRunUnprotectOutsHardlink(TestDvc):
ret = main(
[
"run",
"--overwrite-dvcfile",
"--overwrite",
"--no-run-cache",
"--single-stage",
"-d",
@@ -552,7 +552,7 @@ class TestCmdRunOverwrite(TestDvc):
self.FOO,
"-d",
self.CODE,
"--overwrite-dvcfile",
"--overwrite",
"--no-run-cache",
"--single-stage",
"-o",
@@ -576,7 +576,7 @@ class TestCmdRunOverwrite(TestDvc):
ret = main(
[
"run",
"--overwrite-dvcfile",
"--overwrite",
"--single-stage",
"-f",
"out.dvc",
@@ -629,13 +629,13 @@ class TestCmdRunWorkingDirectory(TestDvc):
wdir=".",
single_stage=True,
)
d = load_stage_file(stage.relpath)
d = load_yaml(stage.relpath)
self.assertNotIn(Stage.PARAM_WDIR, d.keys())
stage = self.dvc.run(
cmd=f"echo test > {self.BAR}", outs=[self.BAR], single_stage=True,
)
d = load_stage_file(stage.relpath)
d = load_yaml(stage.relpath)
self.assertNotIn(Stage.PARAM_WDIR, d.keys())
def test_fname_changes_path_and_wdir(self):
@@ -655,7 +655,7 @@ class TestCmdRunWorkingDirectory(TestDvc):
)
# Check that it is dumped properly (relative to fname)
d = load_stage_file(stage.relpath)
d = load_yaml(stage.relpath)
self.assertEqual(d[Stage.PARAM_WDIR], "..")
@@ -676,19 +676,19 @@ def test_rerun_deterministic_ignore_cache(tmp_dir, run_copy):
def test_rerun_callback(dvc):
def run_callback():
def run_callback(overwrite=False):
return dvc.run(
cmd=("echo content > out"),
cmd="echo content > out",
outs=["out"],
deps=[],
overwrite=False,
overwrite=overwrite,
single_stage=True,
)
assert run_callback() is not None
with mock.patch("dvc.prompt.confirm", return_value=True):
with pytest.raises(StageFileAlreadyExistsError):
assert run_callback() is not None
assert run_callback(overwrite=True) is not None
def test_rerun_changed_dep(tmp_dir, run_copy):
@@ -698,6 +698,7 @@ def test_rerun_changed_dep(tmp_dir, run_copy):
tmp_dir.gen("foo", "changed content")
with pytest.raises(StageFileAlreadyExistsError):
run_copy("foo", "out", overwrite=False, single_stage=True)
assert run_copy("foo", "out", overwrite=True, single_stage=True)
def test_rerun_changed_stage(tmp_dir, run_copy):
@@ -774,7 +775,7 @@ class TestRunPersist(TestDvc):
self.assertEqual(0, ret)
def stage_should_contain_persist_flag(self, stage_file):
stage_file_content = load_stage_file(stage_file)
stage_file_content = load_yaml(stage_file)
self.assertEqual(
True, stage_file_content["outs"][0][BaseOutput.PARAM_PERSIST]
)
@@ -788,7 +789,7 @@ class TestRunPersist(TestDvc):
self.assertEqual(2, len(lines))
def should_remove_persistent_outs(self, file, stage_file):
ret = main(["remove", stage_file])
ret = main(["remove", stage_file, "--outs"])
self.assertEqual(0, ret)
self.assertFalse(os.path.exists(file))
@@ -869,7 +870,7 @@ class TestRerunWithSameOutputs(TestDvc):
"run",
self._outs_command,
self.FOO,
"--overwrite-dvcfile",
"--overwrite",
"--single-stage",
f"echo {self.BAR_CONTENTS} >> {self.FOO}",
]
@@ -945,7 +946,7 @@ class TestPersistentOutput(TestDvc):
cmd = [
"run",
"--overwrite-dvcfile",
"--overwrite",
"--single-stage",
"--deps",
"immutable",
+37 -8
View File
@@ -10,7 +10,7 @@ from dvc.remote.local import LocalRemote
from dvc.repo import Repo
from dvc.stage import PipelineStage, Stage
from dvc.stage.exceptions import StageFileFormatError
from dvc.utils.stage import dump_stage_file, load_stage_file
from dvc.utils.yaml import dump_yaml, load_yaml
from tests.basic_env import TestDvc
@@ -74,12 +74,12 @@ class TestReload(TestDvc):
stage = stages[0]
self.assertTrue(stage is not None)
d = load_stage_file(stage.relpath)
d = load_yaml(stage.relpath)
# NOTE: checking that reloaded stage didn't change its checksum
md5 = "11111111111111111111111111111111"
d[stage.PARAM_MD5] = md5
dump_stage_file(stage.relpath, d)
dump_yaml(stage.relpath, d)
dvcfile = SingleStageFile(self.dvc, stage.relpath)
stage = dvcfile.stage
@@ -87,7 +87,7 @@ class TestReload(TestDvc):
self.assertTrue(stage is not None)
dvcfile.dump(stage)
d = load_stage_file(stage.relpath)
d = load_yaml(stage.relpath)
self.assertEqual(d[stage.PARAM_MD5], md5)
@@ -103,7 +103,7 @@ class TestDefaultWorkingDirectory(TestDvc):
d = stage.dumpd()
self.assertNotIn(Stage.PARAM_WDIR, d.keys())
d = load_stage_file(stage.relpath)
d = load_yaml(stage.relpath)
self.assertNotIn(Stage.PARAM_WDIR, d.keys())
with self.dvc.lock, self.dvc.state:
@@ -167,16 +167,16 @@ def test_meta_is_preserved(tmp_dir, dvc):
(stage,) = tmp_dir.dvc_gen("foo", "foo content")
# Add meta to DVC-file
data = load_stage_file(stage.path)
data = load_yaml(stage.path)
data["meta"] = {"custom_key": 42}
dump_stage_file(stage.path, data)
dump_yaml(stage.path, data)
# Loading and dumping to test that it works and meta is retained
dvcfile = SingleStageFile(dvc, stage.path)
new_stage = dvcfile.stage
dvcfile.dump(new_stage)
new_data = load_stage_file(stage.path)
new_data = load_yaml(stage.path)
assert new_data["meta"] == data["meta"]
@@ -241,3 +241,32 @@ def test_stage_on_no_path_string_repr(tmp_dir, dvc):
assert p.addressing == "No path:stage_name"
assert repr(p) == "Stage: 'No path:stage_name'"
assert str(p) == "stage: 'No path:stage_name'"
def test_stage_remove_pipeline_stage(tmp_dir, dvc, run_copy):
tmp_dir.gen("foo", "foo")
stage = run_copy("foo", "bar", name="copy-foo-bar")
run_copy("bar", "foobar", name="copy-bar-foobar")
dvc_file = stage.dvcfile
with dvc.lock:
stage.remove(purge=False)
assert stage.name in dvc_file.stages
with dvc.lock:
stage.remove()
assert stage.name not in dvc_file.stages
assert "copy-bar-foobar" in dvc_file.stages
def test_stage_remove_pointer_stage(tmp_dir, dvc, run_copy):
(stage,) = tmp_dir.dvc_gen("foo", "foo")
with dvc.lock:
stage.remove(purge=False)
assert not (tmp_dir / "foo").exists()
assert (tmp_dir / stage.relpath).exists()
with dvc.lock:
stage.remove()
assert not (tmp_dir / stage.relpath).exists()
+105
View File
@@ -0,0 +1,105 @@
import networkx as nx
import pytest
from dvc.cli import parse_args
from dvc.command.dag import CmdDAG, _build, _show_ascii, _show_dot
@pytest.mark.parametrize("fmt", [None, "--dot"])
def test_dag(tmp_dir, dvc, mocker, fmt):
tmp_dir.dvc_gen("foo", "foo")
args = ["dag", "--full", "foo.dvc"]
if fmt:
args.append(fmt)
cli_args = parse_args(args)
assert cli_args.func == CmdDAG
cmd = cli_args.func(cli_args)
mocker.patch("dvc.command.dag._build", return_value=dvc.graph)
assert cmd.run() == 0
@pytest.fixture
def graph(tmp_dir, dvc):
tmp_dir.dvc_gen("a", "a")
tmp_dir.dvc_gen("b", "b")
dvc.run(
no_exec=True, deps=["a", "c"], outs=["d", "e"], cmd="cmd1", name="1"
)
dvc.run(
no_exec=True, deps=["b", "c"], outs=["f", "g"], cmd="cmd2", name="2"
)
dvc.run(
no_exec=True,
deps=["a", "b", "c"],
outs=["h", "i"],
cmd="cmd3",
name="3",
)
dvc.run(no_exec=True, deps=["a", "h"], outs=["j"], cmd="cmd4", name="4")
return dvc.graph
def test_build(graph):
assert nx.is_isomorphic(_build(graph), graph)
def test_build_target(graph):
(stage,) = filter(
lambda s: hasattr(s, "name") and s.name == "3", graph.nodes()
)
G = _build(graph, target=stage)
assert set(G.nodes()) == {"3", "b.dvc", "a.dvc"}
assert set(G.edges()) == {("3", "a.dvc"), ("3", "b.dvc")}
def test_build_full(graph):
(stage,) = filter(
lambda s: hasattr(s, "name") and s.name == "3", graph.nodes()
)
G = _build(graph, target=stage, full=True)
assert nx.is_isomorphic(G, graph)
def test_show_ascii(graph):
assert [line.rstrip() for line in _show_ascii(graph).splitlines()] == [
" +----------------+ +----------------+", # noqa: E501
" | stage: 'a.dvc' | | stage: 'b.dvc' |", # noqa: E501
" *+----------------+**** +----------------+", # noqa: E501
" ***** * ***** *** ***", # noqa: E501
" **** * ***** ** **", # noqa: E501
" *** * *** ** **", # noqa: E501
"+------------+ ** +------------+ +------------+", # noqa: E501
"| stage: '1' | ** | stage: '3' | | stage: '2' |", # noqa: E501
"+------------+ *** +------------+ +------------+", # noqa: E501
" ** ***",
" ** **",
" ** **",
" +------------+",
" | stage: '4' |",
" +------------+",
]
def test_show_dot(graph):
assert _show_dot(graph) == (
"strict digraph {\n"
"stage;\n"
"stage;\n"
"stage;\n"
"stage;\n"
"stage;\n"
"stage;\n"
"\"stage: '1'\" -> \"stage: 'a.dvc'\";\n"
"\"stage: '2'\" -> \"stage: 'b.dvc'\";\n"
"\"stage: '3'\" -> \"stage: 'a.dvc'\";\n"
"\"stage: '3'\" -> \"stage: 'b.dvc'\";\n"
"\"stage: '4'\" -> \"stage: 'a.dvc'\";\n"
"\"stage: '4'\" -> \"stage: '3'\";\n"
"}\n"
)
+28 -21
View File
@@ -13,7 +13,7 @@ def test_metrics_diff(dvc, mocker):
"template",
"--targets",
"datafile",
"--show-json",
"--show-vega",
"-x",
"x_field",
"-y",
@@ -32,21 +32,24 @@ def test_metrics_diff(dvc, mocker):
assert cli_args.func == CmdPlotsDiff
cmd = cli_args.func(cli_args)
m = mocker.patch("dvc.repo.plots.diff.diff", return_value={})
m = mocker.patch(
"dvc.repo.plots.diff.diff", return_value={"datafile": "filledtemplate"}
)
assert cmd.run() == 0
m.assert_called_once_with(
cmd.repo,
targets=["datafile"],
template="template",
revs=["HEAD", "tag1", "tag2"],
x_field="x_field",
y_field="y_field",
csv_header=True,
title="my_title",
x_title="x_title",
y_title="y_title",
props={
"template": "template",
"x": "x_field",
"y": "y_field",
"title": "my_title",
"xlab": "x_title",
"ylab": "y_title",
},
)
@@ -59,7 +62,7 @@ def test_metrics_show(dvc, mocker):
"result.extension",
"-t",
"template",
"--show-json",
"--show-vega",
"--no-csv-header",
"datafile",
]
@@ -68,30 +71,34 @@ def test_metrics_show(dvc, mocker):
cmd = cli_args.func(cli_args)
m = mocker.patch("dvc.repo.plots.show.show", return_value={})
m = mocker.patch(
"dvc.repo.plots.show.show", return_value={"datafile": "filledtemplate"}
)
assert cmd.run() == 0
m.assert_called_once_with(
cmd.repo,
targets=["datafile"],
template="template",
x_field=None,
y_field=None,
csv_header=False,
title=None,
x_title=None,
y_title=None,
props={"template": "template", "csv_header": False},
)
def test_plots_show_json(dvc, mocker, caplog):
def test_plots_show_vega(dvc, mocker, caplog):
cli_args = parse_args(
["plots", "diff", "HEAD~10", "HEAD~1", "--show-json"]
[
"plots",
"diff",
"HEAD~10",
"HEAD~1",
"--show-vega",
"--targets",
"plots.csv",
]
)
cmd = cli_args.func(cli_args)
mocker.patch(
"dvc.repo.plots.diff.diff", return_value={"plots.csv": "plothtml"}
)
assert cmd.run() == 0
assert '{"plots.csv": "plothtml"}\n' in caplog.text
assert "plothtml" in caplog.text
+1 -1
View File
@@ -27,7 +27,7 @@ def test_run(mocker, dvc):
"--wdir",
"wdir",
"--no-exec",
"--overwrite-dvcfile",
"--overwrite",
"--no-run-cache",
"--no-commit",
"--outs-persist",
+4 -4
View File
@@ -6,10 +6,10 @@ from dvc.repo.plots.diff import _revisions
@pytest.mark.parametrize(
"arg_revisions,is_dirty,expected_revisions",
[
([], False, ["working tree"]),
([], True, ["HEAD", "working tree"]),
(["v1", "v2", "working tree"], False, ["v1", "v2", "working tree"]),
(["v1", "v2", "working tree"], True, ["v1", "v2", "working tree"]),
([], False, ["workspace"]),
([], True, ["HEAD", "workspace"]),
(["v1", "v2", "workspace"], False, ["v1", "v2", "workspace"]),
(["v1", "v2", "workspace"], True, ["v1", "v2", "workspace"]),
],
)
def test_revisions(mocker, arg_revisions, is_dirty, expected_revisions):
+1 -1
View File
@@ -38,7 +38,7 @@ def test_open_in_history(tmp_dir, scm, dvc):
dvc.scm.commit("foofoo")
for rev in dvc.brancher(revs=["HEAD~1"]):
if rev == "working tree":
if rev == "workspace":
continue
tree = RepoTree(dvc)
+2 -2
View File
@@ -15,7 +15,7 @@ def test_get_active_graph(tmp_dir, dvc):
single_stage=True, deps=["foo"], outs=["baz"], cmd="echo baz > baz"
)
dvc.lock_stage("bar.dvc")
dvc.freeze("bar.dvc")
graph = dvc.graph
active_graph = _get_active_graph(graph)
@@ -25,7 +25,7 @@ def test_get_active_graph(tmp_dir, dvc):
(baz_stage, foo_stage),
}
dvc.lock_stage("baz.dvc")
dvc.freeze("baz.dvc")
graph = dvc.graph
active_graph = _get_active_graph(graph)
+1 -1
View File
@@ -38,7 +38,7 @@ def test_open_in_history(tmp_dir, scm, dvc):
dvc.scm.commit("foofoo")
for rev in dvc.brancher(revs=["HEAD~1"]):
if rev == "working tree":
if rev == "workspace":
continue
tree = DvcTree(dvc)