Compare commits

...
26 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 4bca30e4d6 dvc: bump to 1.11.9 2020-12-23 13:17:46 +02:00
mergify[bot]andPeter Rowlands 7a84c104fd lock: fix AttributeError in dvc.lock.HardlinkLock (#5152) (#5153)
(cherry picked from commit afe96974c3)

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2020-12-23 13:17:08 +02:00
mergify[bot]andSaugat Pachhai 1aa7e5dbec params/metrics diff: collect from revisions/workspace twice (bp #5121) (#5129)
(cherry picked from commit 723477ed5a)

Co-authored-by: Saugat Pachhai <[email protected]>
2020-12-18 14:22:19 +05:45
mergify[bot]andSaugat Pachhai fe0d6ec3e0 params diff: skip params dependency for vars (bp #5123) (#5127)
* Skip paramdependency for vars

Previously, `configs` which was a list of PathInfo was being used
to check if the `vars` should be skipped or not.

But, the file is stored as a string that made that check redundant
which might have caused #5117 as they were being stored twice,
once as a expanded paramdependency and next as a flattened dict
from vars, which later during `flatten`, it might have panicked
from that duplication.

* Use defaultdict

* change add_vars to collect_vars

(cherry picked from commit dcce3a31a9)

Co-authored-by: Saugat Pachhai <[email protected]>
2020-12-18 12:09:22 +05:45
Ruslan Kuprieiev a3891995e6 dvc: bump to 1.11.8 2020-12-16 15:05:48 +02:00
05f3eb67ce repro: simplify the logic for frozen stages (bp #5114) (#5116)
* [WIP] repro: simplify the logic for frozen stages (#5114)

Fixes #5082

(cherry picked from commit 64dfaffd7c)

# Conflicts:
#	dvc/repo/reproduce.py

* resolve conflicts

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-16 15:04:57 +02:00
29db8a1a95 repro: accept multiple targets (bp #5111) (#5115)
* repro: accept multiple targets (#5111)

Currently, we iterate over targets manually in CLI, which results in a
complicated code and a DAG collection/check overhead. With this PR we
collect the DAG once, figure out all the needed targets within it and
just reproduce them in one pass.

(cherry picked from commit aec8d65791)

# Conflicts:
#	dvc/command/base.py
#	dvc/command/experiments.py
#	dvc/command/repro.py
#	tests/func/test_repro_multistage.py

* resolve conflicts

* resolve test conflicts

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-12-16 05:17:55 +02:00
Ruslan Kuprieiev 4082757aa1 dvc: bump to 1.11.7 2020-12-15 19:55:12 +02:00
mergify[bot]andJoão Rafael 8a11595a93 Remove version restriction from networkx (#5110) (#5112)
(cherry picked from commit 84a3a59bc7)

Co-authored-by: João Rafael <[email protected]>
2020-12-15 19:54:25 +02:00
Ruslan Kuprieiev 60546dcd98 dvc: bump to 1.11.6 2020-12-14 20:24:50 +02:00
mergify[bot]andpeper0 1b4326aefd fix making too many ssh connections when pushing run cache (#4732) (#4733) (#5097)
* fix making too many ssh connections when pushing run cache (#4732)

* reformat

* generator should exit on GeneratorExit exception

* Update dvc/tree/ssh/__init__.py

* pool: close on Exception and not BaseException

Co-authored-by: Tomasz Łakota <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
(cherry picked from commit 6e92352001)

Co-authored-by: peper0 <[email protected]>
2020-12-14 16:55:18 +02:00
Ruslan Kuprieiev 1138afe495 dvc: bump to 1.11.5 2020-12-14 13:11:06 +02:00
Ruslan Kuprieiev 7916ce4bfb cli: config: check name format (#5094)
Currently we don't handle bad name well:

```
(dvc-3.8.3) ➜  dvc git:(config-bug) ✗ dvc config section
ERROR: unexpected error - not enough values to unpack (expected 2, got 1)
```

With this PR, we check name to conform to either `remote.name.option` or
`section.option`, so we'll give a more friendly:

```
(dvc-3.8.3) ➜  dvc git:(config-bug) ✗ dvc config section
ERROR: argument name: name argument should look like remote.name.option or section.option
usage: dvc config [-h] [--global | --system | --local] [-q | -v] [-u] [-l] [name] [value]

Get or set config options.
Documentation: <https://man.dvc.org/config>

positional arguments:
  name           Option name (remote.name.option or section.option).
  value          Option value.

optional arguments:
  -h, --help     show this help message and exit
  --global       Use global config.
  --system       Use system config.
  --local        Use local config.
  -q, --quiet    Be quiet.
  -v, --verbose  Be verbose.
  -u, --unset    Unset option.
  -l, --list     List all defined config values.
```
2020-12-14 13:10:11 +02:00
Ruslan Kuprieiev c5db57900d gha: run tests on other branches 2020-12-14 12:37:06 +02:00
Ruslan Kuprieiev c9ef5d3ccb dvc: bump to 1.11.4 2020-12-13 02:33:04 +02:00
Ruslan Kuprieiev cd72149b5f system: _reflink_darwin: use fsencode for path (#5086)
Fixes #5083
2020-12-13 02:32:37 +02:00
Ruslan Kuprieiev 80c1f10f08 dvc: bump to 1.11.3 2020-12-08 18:09:48 +02:00
Saugat Pachhai 573546d824 Pin numpy for windows build (#5057) 2020-12-08 18:06:38 +02:00
Casper da Costa-Luis 41b20cd675 tidy snap release (#5016) 2020-12-02 20:21:00 +02:00
Ruslan Kuprieiev cb90bd0c8d dvc: bump to 1.11.2 2020-12-02 19:28:15 +02:00
Ruslan Kuprieiev 82896be988 gha: fix snap action name 2020-12-02 19:12:35 +02:00
Ruslan Kuprieiev 84ae8e02f7 snap: fix channels 2020-12-02 19:12:02 +02:00
Ruslan Kuprieiev a97d5fef72 snap: fix channel names 2020-12-02 18:56:16 +02:00
Ruslan Kuprieiev ed02ff7c30 dvc: bump to 1.11.1 2020-12-02 18:28:29 +02:00
Ruslan Kuprieiev 363ad46326 gha: build packages on other branches 2020-12-02 18:07:26 +02:00
Ruslan Kuprieiev 776ed1d2dd snap: publish to v1 channel 2020-12-02 18:06:08 +02:00
24 changed files with 216 additions and 225 deletions
+3 -5
View File
@@ -1,8 +1,6 @@
name: Build packages
on:
push:
branches:
- master
push: {}
release:
types: [released]
jobs:
@@ -121,11 +119,11 @@ jobs:
with:
store_login: ${{ secrets.STORE_LOGIN }}
snap: ${{ steps.snapcraft.outputs.snap }}
release: edge
release: edge,v1/edge
- name: Publish to the stable channel
uses: snapcore/action-publish@v1
if: github.event_name == 'release'
with:
store_login: ${{ secrets.STORE_LOGIN }}
snap: ${{ steps.snapcraft.outputs.snap }}
release: stable
release: stable,v1/stable
+2 -6
View File
@@ -1,12 +1,8 @@
name: Tests
on:
pull_request:
branches:
- master
push:
branches:
- master
- pull_request
- push
env:
DVC_TEST: "true"
-8
View File
@@ -45,14 +45,6 @@ class CmdBase(ABC):
updater = Updater(self.repo.tmp_dir, hardlink_lock=hardlink_lock)
updater.check()
@property
def default_targets(self):
"""Default targets for `dvc repro`."""
from dvc.dvcfile import PIPELINE_FILE
logger.debug(f"assuming default target '{PIPELINE_FILE}'.")
return [PIPELINE_FILE]
@abstractmethod
def run(self):
pass
+38 -10
View File
@@ -6,6 +6,25 @@ from dvc.config import Config, ConfigError
logger = logging.getLogger(__name__)
NAME_REGEX = r"^(?P<remote>remote\.)?(?P<section>[^\.]*)\.(?P<option>[^\.]*)$"
def _name_type(value):
import re
match = re.match(NAME_REGEX, value)
if not match:
raise argparse.ArgumentTypeError(
"name argument should look like "
"remote.name.option or "
"section.option"
)
return (
bool(match.group("remote")),
match.group("section").lower(),
match.group("option").lower(),
)
class CmdConfig(CmdBaseNoRepo):
def __init__(self, args):
@@ -14,20 +33,24 @@ class CmdConfig(CmdBaseNoRepo):
self.config = Config(validate=False)
def run(self):
section, opt = self.args.name.lower().strip().split(".", 1)
remote, section, opt = self.args.name
if self.args.value is None and not self.args.unset:
conf = self.config.load_one(self.args.level)
self._check(conf, section, opt)
if remote:
conf = conf["remote"]
self._check(conf, remote, section, opt)
logger.info(conf[section][opt])
return 0
with self.config.edit(self.args.level) as conf:
if remote:
conf = conf["remote"]
if self.args.unset:
self._check(conf, section, opt)
self._check(conf, remote, section, opt)
del conf[section][opt]
else:
self._check(conf, section)
self._check(conf, remote, section)
conf[section][opt] = self.args.value
if self.args.name == "cache.type":
@@ -39,14 +62,15 @@ class CmdConfig(CmdBaseNoRepo):
return 0
def _check(self, conf, section, opt=None):
def _check(self, conf, remote, section, opt=None):
name = "remote" if remote else "section"
if section not in conf:
msg = "section {} doesn't exist"
raise ConfigError(msg.format(self.args.name))
raise ConfigError(f"{name} '{section}' doesn't exist")
if opt and opt not in conf[section]:
msg = "option {} doesn't exist"
raise ConfigError(msg.format(self.args.name))
raise ConfigError(
f"option '{opt}' doesn't exist in {name} '{section}'"
)
parent_config_parser = argparse.ArgumentParser(add_help=False)
@@ -92,6 +116,10 @@ def add_parser(subparsers, parent_parser):
action="store_true",
help="Unset option.",
)
config_parser.add_argument("name", help="Option name.")
config_parser.add_argument(
"name",
type=_name_type,
help="Option name (remote.name.option or section.option).",
)
config_parser.add_argument("value", nargs="?", help="Option value.")
config_parser.set_defaults(func=CmdConfig)
+10 -24
View File
@@ -451,32 +451,18 @@ class CmdExperimentsRun(CmdRepro):
saved_dir = os.path.realpath(os.curdir)
os.chdir(self.args.cwd)
# Dirty hack so the for loop below can at least enter once
if self.args.all_pipelines:
self.args.targets = [None]
elif not self.args.targets:
self.args.targets = self.default_targets
ret = 0
for target in self.args.targets:
try:
self.repo.experiments.run(
target,
name=self.args.name,
queue=self.args.queue,
run_all=self.args.run_all,
jobs=self.args.jobs,
params=self.args.params,
checkpoint_resume=self.args.checkpoint_resume,
**self._repro_kwargs,
)
except DvcException:
logger.exception("")
ret = 1
break
self.repo.experiments.run(
name=self.args.name,
queue=self.args.queue,
run_all=self.args.run_all,
jobs=self.args.jobs,
params=self.args.params,
checkpoint_resume=self.args.checkpoint_resume,
**self._repro_kwargs,
)
os.chdir(saved_dir)
return ret
return 0
class CmdExperimentsGC(CmdRepro):
+12 -28
View File
@@ -7,7 +7,6 @@ from dvc.command.base import CmdBase, append_doc_link
from dvc.command.metrics import _show_metrics
from dvc.command.status import CmdDataStatus
from dvc.dvcfile import PIPELINE_FILE
from dvc.exceptions import DvcException
logger = logging.getLogger(__name__)
@@ -17,40 +16,25 @@ class CmdRepro(CmdBase):
saved_dir = os.path.realpath(os.curdir)
os.chdir(self.args.cwd)
# Dirty hack so the for loop below can at least enter once
if self.args.all_pipelines:
self.args.targets = [None]
elif not self.args.targets:
self.args.targets = self.default_targets
stages = self.repo.reproduce(**self._repro_kwargs)
if len(stages) == 0:
logger.info(CmdDataStatus.UP_TO_DATE_MSG)
else:
logger.info(
"Use `dvc push` to send your updates to " "remote storage."
)
ret = 0
for target in self.args.targets:
try:
stages = self.repo.reproduce(target, **self._repro_kwargs)
if len(stages) == 0:
logger.info(CmdDataStatus.UP_TO_DATE_MSG)
else:
logger.info(
"Use `dvc push` to send your updates to "
"remote storage."
)
if self.args.metrics:
metrics = self.repo.metrics.show()
logger.info(_show_metrics(metrics))
except DvcException:
logger.exception("")
ret = 1
break
if self.args.metrics:
metrics = self.repo.metrics.show()
logger.info(_show_metrics(metrics))
os.chdir(saved_dir)
return ret
return 0
@property
def _repro_kwargs(self):
return {
"targets": self.args.targets,
"single_item": self.args.single_item,
"force": self.args.force,
"dry": self.args.dry,
+1 -2
View File
@@ -148,9 +148,8 @@ class HardlinkLock(flufl.lock.Lock, LockBase):
): # pylint: disable=super-init-not-called
import socket
super().__init__(lockfile)
self._tmp_dir = tmp_dir
super().__init__(lockfile)
# NOTE: this is basically Lock.__init__ copy-paste, except that
# instead of using `socket.getfqdn()` we use `socket.gethostname()`
+4 -4
View File
@@ -37,13 +37,13 @@ def _parse_params(path_params: Iterable):
@locked
def run(
repo,
target: Optional[str] = None,
targets: Optional[Iterable] = None,
params: Optional[Iterable] = None,
run_all: Optional[bool] = False,
jobs: Optional[int] = 1,
**kwargs,
) -> dict:
"""Reproduce the specified target as an experiment.
"""Reproduce the specified targets as an experiment.
Accepts the same additional kwargs as Repo.reproduce.
@@ -59,10 +59,10 @@ def run(
params = []
try:
return repo.experiments.reproduce_one(
target=target, params=params, **kwargs
targets=targets, params=params, **kwargs
)
except UnchangedExperimentError:
# If experiment contains no changes, just run regular repro
kwargs.pop("queue", None)
kwargs.pop("checkpoint_resume", None)
return {None: repo.reproduce(target=target, **kwargs)}
return {None: repo.reproduce(targets=targets, **kwargs)}
+9 -7
View File
@@ -3,12 +3,10 @@ from dvc.utils.diff import diff as _diff
from dvc.utils.diff import format_dict
def _get_metrics(repo, *args, rev=None, **kwargs):
def _get_metrics(repo, *args, revs=None, **kwargs):
try:
metrics = repo.metrics.show(
*args, **kwargs, revs=[rev] if rev else None
)
return metrics.get(rev or "", {})
metrics = repo.metrics.show(*args, **kwargs, revs=revs)
return metrics
except NoMetricsError:
return {}
@@ -19,8 +17,12 @@ def diff(repo, *args, a_rev=None, b_rev=None, **kwargs):
with_unchanged = kwargs.pop("all", False)
old = _get_metrics(repo, *args, **kwargs, rev=(a_rev or "HEAD"))
new = _get_metrics(repo, *args, **kwargs, rev=b_rev)
a_rev = a_rev or "HEAD"
b_rev = b_rev or "workspace"
metrics = _get_metrics(repo, *args, **kwargs, revs=[a_rev, b_rev])
old = metrics.get(a_rev, {})
new = metrics.get(b_rev, {})
return _diff(
format_dict(old), format_dict(new), with_unchanged=with_unchanged
+9 -5
View File
@@ -4,10 +4,10 @@ from dvc.utils.diff import format_dict
from .show import NoParamsError
def _get_params(repo, *args, rev=None, **kwargs):
def _get_params(repo, *args, revs=None, **kwargs):
try:
params = repo.params.show(*args, **kwargs, revs=[rev] if rev else None)
return params.get(rev or "", {})
params = repo.params.show(*args, **kwargs, revs=revs)
return params
except NoParamsError:
return {}
@@ -18,8 +18,12 @@ def diff(repo, *args, a_rev=None, b_rev=None, **kwargs):
with_unchanged = kwargs.pop("all", False)
old = _get_params(repo, *args, **kwargs, rev=(a_rev or "HEAD"))
new = _get_params(repo, *args, **kwargs, rev=b_rev)
a_rev = a_rev or "HEAD"
b_rev = b_rev or "workspace"
params = _get_params(repo, *args, **kwargs, revs=[a_rev, b_rev])
old = params.get(a_rev, {})
new = params.get(b_rev, {})
return _diff(
format_dict(old), format_dict(new), with_unchanged=with_unchanged
+11 -5
View File
@@ -1,4 +1,5 @@
import logging
from collections import defaultdict
from typing import TYPE_CHECKING, List
from dvc.dependency.param import ParamsDependency
@@ -50,17 +51,18 @@ def _read_params(repo, configs, rev):
return res
def _add_vars(repo, configs, params):
def _collect_vars(repo, params):
vars_params = defaultdict(dict)
for stage in repo.stages:
if isinstance(stage, PipelineStage) and stage.tracked_vars:
for file, vars_ in stage.tracked_vars.items():
# `params` file are shown regardless of `tracked` or not
# to reduce noise and duplication, they are skipped
if file in configs:
if file in params:
continue
params[file] = params.get(file, {})
params[file].update(vars_)
vars_params[file].update(vars_)
return vars_params
@locked
@@ -70,7 +72,11 @@ def show(repo, revs=None):
for branch in repo.brancher(revs=revs):
configs = _collect_configs(repo, branch)
params = _read_params(repo, configs, branch)
_add_vars(repo, configs, params)
vars_params = _collect_vars(repo, params)
# NOTE: only those that are not added as a ParamDependency are included
# so we don't need to recursively merge them yet.
params.update(vars_params)
if params:
res[branch] = params
+65 -75
View File
@@ -2,7 +2,7 @@ import logging
import typing
from functools import partial
from dvc.exceptions import InvalidArgumentError, ReproductionError
from dvc.exceptions import ReproductionError
from dvc.repo.scm_context import scm_context
from dvc.stage.run import CheckpointKilledError
@@ -54,33 +54,11 @@ def _dump_stage(stage):
dvcfile.dump(stage, update_pipeline=False)
def _get_active_graph(G):
import networkx as nx
active = G.copy()
for stage in G:
if not stage.frozen:
continue
active.remove_edges_from(G.out_edges(stage))
for edge in G.out_edges(stage):
_, to_stage = edge
for node in nx.dfs_preorder_nodes(G, to_stage):
# NOTE: `in_degree` will return InDegreeView({}) if stage
# no longer exists in the `active` DAG.
if not active.in_degree(node):
# NOTE: if some edge no longer exists `remove_edges_from`
# will ignore it without error.
active.remove_edges_from(G.out_edges(node))
active.remove_node(node)
return active
@locked
@scm_context
def reproduce(
self: "Repo",
target=None,
targets=None,
recursive=False,
pipeline=False,
all_pipelines=False,
@@ -88,41 +66,46 @@ def reproduce(
):
glob = kwargs.pop("glob", False)
accept_group = not glob
assert target is None or isinstance(target, str)
if not target and not all_pipelines:
raise InvalidArgumentError(
"Neither `target` nor `--all-pipelines` are specified."
)
if isinstance(targets, str):
targets = [targets]
if not all_pipelines and not targets:
from dvc.dvcfile import PIPELINE_FILE
targets = [PIPELINE_FILE]
interactive = kwargs.get("interactive", False)
if not interactive:
kwargs["interactive"] = self.config["core"].get("interactive", False)
active_graph = _get_active_graph(self.graph)
active_pipelines = get_pipelines(active_graph)
stages = set()
if pipeline or all_pipelines:
pipelines = get_pipelines(self.graph)
if all_pipelines:
pipelines = active_pipelines
used_pipelines = pipelines
else:
stage = self.stage.get_target(target)
pipelines = [get_pipeline(active_pipelines, stage)]
used_pipelines = []
for target in targets:
stage = self.stage.get_target(target)
used_pipelines.append(get_pipeline(pipelines, stage))
targets = []
for pipeline in pipelines:
for stage in pipeline:
if pipeline.in_degree(stage) == 0:
targets.append(stage)
for pline in used_pipelines:
for stage in pline:
if pline.in_degree(stage) == 0:
stages.add(stage)
else:
targets = self.stage.collect(
target,
recursive=recursive,
graph=active_graph,
accept_group=accept_group,
glob=glob,
)
for target in targets:
stages.update(
self.stage.collect(
target,
recursive=recursive,
accept_group=accept_group,
glob=glob,
)
)
return _reproduce_stages(active_graph, targets, **kwargs)
return _reproduce_stages(self.graph, list(stages), **kwargs)
def _reproduce_stages(
@@ -163,7 +146,7 @@ def _reproduce_stages(
The derived evaluation of _downstream_ B would be: [B, D, E]
"""
pipeline = _get_pipeline(G, stages, downstream, single_item)
steps = _get_steps(G, stages, downstream, single_item)
force_downstream = kwargs.pop("force_downstream", False)
result = []
@@ -171,7 +154,7 @@ def _reproduce_stages(
# `ret` is used to add a cosmetic newline.
ret = []
checkpoint_func = kwargs.pop("checkpoint_func", None)
for stage in pipeline:
for stage in steps:
if ret:
logger.info("")
@@ -205,36 +188,43 @@ def _reproduce_stages(
return result
def _get_pipeline(G, stages, downstream, single_item):
def _get_steps(G, stages, downstream, single_item):
import networkx as nx
if single_item:
all_pipelines = stages
else:
all_pipelines = []
for stage in stages:
if downstream:
# NOTE (py3 only):
# Python's `deepcopy` defaults to pickle/unpickle the object.
# Stages are complex objects (with references to `repo`,
# `outs`, and `deps`) that cause struggles when you try
# to serialize them. We need to create a copy of the graph
# itself, and then reverse it, instead of using
# graph.reverse() directly because it calls `deepcopy`
# underneath -- unless copy=False is specified.
nodes = nx.dfs_postorder_nodes(
G.copy().reverse(copy=False), stage
)
all_pipelines += reversed(list(nodes))
else:
all_pipelines += nx.dfs_postorder_nodes(G, stage)
active = G.copy()
if not single_item:
# NOTE: frozen stages don't matter for single_item
for stage in G:
if stage.frozen:
# NOTE: disconnect frozen stage from its dependencies
active.remove_edges_from(G.out_edges(stage))
pipeline = []
all_pipelines = []
for stage in stages:
if downstream:
# NOTE (py3 only):
# Python's `deepcopy` defaults to pickle/unpickle the object.
# Stages are complex objects (with references to `repo`,
# `outs`, and `deps`) that cause struggles when you try
# to serialize them. We need to create a copy of the graph
# itself, and then reverse it, instead of using
# graph.reverse() directly because it calls `deepcopy`
# underneath -- unless copy=False is specified.
nodes = nx.dfs_postorder_nodes(active.reverse(copy=False), stage)
all_pipelines += reversed(list(nodes))
else:
all_pipelines += nx.dfs_postorder_nodes(active, stage)
steps = []
for stage in all_pipelines:
if stage not in pipeline:
pipeline.append(stage)
if stage not in steps:
# NOTE: order of steps still matters for single_item
if single_item and stage not in stages:
continue
return pipeline
steps.append(stage)
return steps
def _repro_callback(experiments_callback, unchanged, stages):
+2 -2
View File
@@ -71,8 +71,8 @@ class System:
clonefile.restype = ctypes.c_int
return clonefile(
ctypes.c_char_p(src.encode("utf-8")),
ctypes.c_char_p(dst.encode("utf-8")),
ctypes.c_char_p(os.fsencode(src)),
ctypes.c_char_p(os.fsencode(dst)),
ctypes.c_int(0),
)
+1 -1
View File
@@ -11,7 +11,7 @@ def get_connection(conn_func, *args, **kwargs):
conn = pool.get_connection()
try:
yield conn
except BaseException:
except Exception:
conn.close()
raise
else:
+1 -1
View File
@@ -6,7 +6,7 @@
import os
import subprocess
_BASE_VERSION = "1.11.0"
_BASE_VERSION = "1.11.9"
def _generate_version(base_version):
+1
View File
@@ -1,2 +1,3 @@
PyInstaller==3.5
numpy==1.19.3
psutil
+1 -1
View File
@@ -71,7 +71,7 @@ install_requires = [
"flufl.lock>=3.2,<4",
"win-unicode-console>=0.5; sys_platform == 'win32'",
"pywin32>=225; sys_platform == 'win32'",
"networkx>=2.1,<2.5",
"networkx>=2.1",
"pydot>=1.2.4",
"speedcopy>=2.0.1; python_version < '3.8' and sys_platform == 'win32'",
"dataclasses==0.7; python_version < '3.7'",
+2 -2
View File
@@ -141,14 +141,14 @@ class TestGCMultipleDvcRepos(TestDvc):
class TestConfig(TestDvc):
def test(self):
name = "param"
name = "section.option"
value = "1"
args = parse_args(["config", "-u", "--unset", name, value])
self.assertIsInstance(args.func(args), CmdConfig)
self.assertEqual(args.unset, True)
self.assertEqual(args.name, name)
self.assertEqual(args.name, (False, "section", "option"))
self.assertEqual(args.value, value)
+16
View File
@@ -145,3 +145,19 @@ def test_load_relative_paths(dvc, field, remote_url):
assert cfg["remote"]["test"][field] == os.path.join(
dvc_dir, "..", "file.txt"
)
def test_config_remote(tmp_dir, dvc, caplog):
(tmp_dir / ".dvc" / "config").write_text(
"['remote \"myremote\"']\n"
" url = s3://bucket/path\n"
" region = myregion\n"
)
caplog.clear()
assert main(["config", "remote.myremote.url"]) == 0
assert "s3://bucket/path" in caplog.text
caplog.clear()
assert main(["config", "remote.myremote.region"]) == 0
assert "myregion" in caplog.text
+13
View File
@@ -170,6 +170,19 @@ def test_non_existing_stage_name(tmp_dir, dvc, run_copy):
assert main(["freeze", ":copy-file1-file3"]) != 0
def test_repro_frozen(tmp_dir, dvc, run_copy):
(data_stage,) = tmp_dir.dvc_gen("data", "foo")
stage0 = run_copy("data", "stage0", name="copy-data-stage0")
run_copy("stage0", "stage1", name="copy-data-stage1")
run_copy("stage1", "stage2", name="copy-data-stage2")
dvc.freeze("copy-data-stage1")
tmp_dir.gen("data", "bar")
stages = dvc.reproduce()
assert stages == [data_stage, stage0]
def test_downstream(tmp_dir, dvc):
# The dependency graph should look like this:
#
+11
View File
@@ -0,0 +1,11 @@
import pytest
from dvc.cli import DvcParserError, parse_args
@pytest.mark.parametrize(
"name", ["way.too.long", "no_option", "remote.way.too.long"],
)
def test_config_bad_name(name):
with pytest.raises(DvcParserError):
parse_args(["config", name])
+1 -4
View File
@@ -9,7 +9,6 @@ from dvc.command.experiments import (
CmdExperimentsRun,
CmdExperimentsShow,
)
from dvc.dvcfile import PIPELINE_FILE
from dvc.exceptions import InvalidArgumentError
from .test_repro import default_arguments as repro_arguments
@@ -100,9 +99,7 @@ def test_experiments_run(dvc, mocker, args, resume):
mocker.patch.object(cmd.repo, "reproduce")
mocker.patch.object(cmd.repo.experiments, "run")
cmd.run()
cmd.repo.experiments.run.assert_called_with(
PIPELINE_FILE, **default_arguments
)
cmd.repo.experiments.run.assert_called_with(**default_arguments)
def test_experiments_gc(dvc, mocker):
+3 -3
View File
@@ -1,6 +1,5 @@
from dvc.cli import parse_args
from dvc.command.repro import CmdRepro
from dvc.dvcfile import PIPELINE_FILE
default_arguments = {
"all_pipelines": False,
@@ -16,6 +15,7 @@ default_arguments = {
"force_downstream": False,
"pull": False,
"glob": False,
"targets": [],
}
@@ -23,7 +23,7 @@ def test_default_arguments(dvc, mocker):
cmd = CmdRepro(parse_args(["repro"]))
mocker.patch.object(cmd.repo, "reproduce")
cmd.run()
cmd.repo.reproduce.assert_called_with(PIPELINE_FILE, **default_arguments)
cmd.repo.reproduce.assert_called_with(**default_arguments)
def test_downstream(dvc, mocker):
@@ -32,4 +32,4 @@ def test_downstream(dvc, mocker):
cmd.run()
arguments = default_arguments.copy()
arguments.update({"downstream": True})
cmd.repo.reproduce.assert_called_with(PIPELINE_FILE, **arguments)
cmd.repo.reproduce.assert_called_with(**arguments)
-32
View File
@@ -1,37 +1,5 @@
import mock
from dvc.repo.reproduce import _get_active_graph
def test_get_active_graph(tmp_dir, dvc):
(pre_foo_stage,) = tmp_dir.dvc_gen({"pre-foo": "pre-foo"})
foo_stage = dvc.run(
single_stage=True, deps=["pre-foo"], outs=["foo"], cmd="echo foo > foo"
)
bar_stage = dvc.run(
single_stage=True, deps=["foo"], outs=["bar"], cmd="echo bar > bar"
)
baz_stage = dvc.run(
single_stage=True, deps=["foo"], outs=["baz"], cmd="echo baz > baz"
)
dvc.freeze("bar.dvc")
graph = dvc.graph
active_graph = _get_active_graph(graph)
assert active_graph.nodes == graph.nodes
assert set(active_graph.edges) == {
(foo_stage, pre_foo_stage),
(baz_stage, foo_stage),
}
dvc.freeze("baz.dvc")
graph = dvc.graph
active_graph = _get_active_graph(graph)
assert set(active_graph.nodes) == {bar_stage, baz_stage}
assert not active_graph.edges
@mock.patch("dvc.repo.reproduce._reproduce_stage", returns=[])
def test_number_reproduces(reproduce_stage_mock, tmp_dir, dvc):