dvc: bump to 0.34.2

dvc: exclude `persist` when computing md5 for the stage

.appveyor.yml

@@ -75,6 +75,7 @@ test_script:
   - python -mtests
 
 after_test:
+  - if "%APPVEYOR_PULL_REQUEST_NUMBER%"=="" pip install codecov
   - if "%APPVEYOR_PULL_REQUEST_NUMBER%"=="" codecov
   - if "%APPVEYOR_REPO_TAG%"=="true" python -m dvc pull
   - if "%APPVEYOR_REPO_TAG%"=="true" .\scripts\build_windows.cmd

.dvc/config

@@ -1,4 +1,4 @@
 ['remote "dvc"']
-url = s3://dvc-share/dvc/
+url = https://dvc.org/s3/dvc
 [core]
 remote = dvc

.gitignore

@@ -27,3 +27,6 @@ innosetup/config.ini
 dvc/version.py
 
 *.swp
+
+pip-wheel-metadata/
+.vscode/

README.rst

@@ -6,7 +6,7 @@
 • `Docs <https://dvc.org/doc>`_
 • `Twitter <https://twitter.com/iterativeai>`_
 • `Chat (Community & Support) <https://dvc.org/chat>`_
-• `Tutorial <https://dvc.org/doc/tutorial>`_
+• `Tutorial <https://dvc.org/doc/get-started>`_
 • `Mailing List <https://sweedom.us10.list-manage.com/subscribe/post?u=a08bf93caae4063c4e6a351f6&id=24c0ecc49a>`_
 
 .. image:: https://travis-ci.com/iterative/dvc.svg?branch=master
@@ -28,17 +28,18 @@
 |
 
 **Data Science Version Control** or **DVC** is an **open-source** tool for data science and
-machine learning projects. With a simple and flexible Git-like architecture and interface it
-helps data scientists:
+machine learning projects. Key features:
 
-#. manage **machine learning models** - versioning, including data sets and transformations (scripts) that were
-   used to generate models;
+#. simple **command line** Git-like exprerience. Does not require installing and maintaining
+   any databases. Does not depend on any proprietary online services;
 
-#. make projects **reproducible**;
+#. it manages and versions **datasets** and **machine learning models**. Data is saved in
+   S3, Google cloud, Azure, SSH server, HDFS or even local HDD RAID;
 
-#. make projects **shareable**;
+#. it makes projects **reproducible** and **shareable**, it helps answering question how
+   the model was build;
 
-#. manage experiments with branching and **metrics** tracking;
+#. it helps manage experiments with Git tags or branches and **metrics** tracking;
 
 It aims to replace tools like Excel and Docs that are being commonly used as a knowledge repo and
 a ledger for the team, ad-hoc scripts to track and move deploy different model versions, ad-hoc
@@ -50,14 +51,30 @@ data file suffixes and prefixes.
 How DVC works
 =============
 
-DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache.
-To store and share data files cache DVC supports remotes - any cloud (S3, Azure, Google Cloud, etc) or any on-premise
-network storage (via SSH, for example).
+We encourage you to read our `Get Started <https://dvc.org/doc/get-started>`_ to better understand what DVC
+is and how does it fit your scenarios.
+
+The easiest (but not perfect!) *analogy* to describe it: DVC is Git (or Git-lfs to be precise) + ``makefiles``
+made right and tailored specifically for ML and Data Science scenarios.
+
+#. ``Git/Git-lfs`` part - DVC helps you storing and sharing data artifacts, models. It connects them with your
+   Git repository.
+#. ``Makefiles`` part - DVC describes how one data or model artifact was build from another data. 
+
+DVC usually runs along with Git. Git is used as usual to store and version code and DVC meta-files. DVC helps
+to store data and model files seamlessly out of Git while preserving almost the same user experience as if they
+were stored in Git itself. To store and share data files cache DVC supports remotes - any cloud (S3, Azure,
+Google Cloud, etc) or any on-premise network storage (via SSH, for example). It does not require 
 
 .. image:: https://dvc.org/static/img/flow.gif
    :target: https://dvc.org/static/img/flow.gif
    :alt: how_dvc_works
 
+DVC pipelines (aka computational graph) feature connects code and data together. In a very explicit way you can
+specify, run, and save information that a certain command with certain dependencies needs to be run to produce
+a model. See the quick start section below or check `Get Started <https://dvc.org/doc/get-started>`_ tutorial to
+learn more.
+
 Quick start
 ===========
 
@@ -66,7 +83,7 @@ Please read `Get Started <https://dvc.org/doc/get-started>`_ for the full versio
 +-----------------------------------+-------------------------------------------------------------------+
 | Step                              | Command                                                           |
 +===================================+===================================================================+
-| Track code and data together      | | ``$ git add train.py``                                          |
+| Track data                        | | ``$ git add train.py``                                          |
 |                                   | | ``$ dvc add images.zip``                                        |
 +-----------------------------------+-------------------------------------------------------------------+
 | Connect code and data by commands | | ``$ dvc run -d images.zip -o images/ unzip -q images.zip``      |
@@ -79,27 +96,30 @@ Please read `Get Started <https://dvc.org/doc/get-started>`_ for the full versio
 |                                   | | ``$ git commit -m 'The baseline model'``                        |
 |                                   | | ``$ git push``                                                  |
 +-----------------------------------+-------------------------------------------------------------------+
-| Share data and ML models          | | ``$ dvc remote add myremote s3://mybucket/image_cnn``           |
-|                                   | | ``$ dvc config core.remote myremote``                           |
+| Share data and ML models          | | ``$ dvc remote add myremote -d s3://mybucket/image_cnn``        |
 |                                   | | ``$ dvc push``                                                  |
 +-----------------------------------+-------------------------------------------------------------------+
 
 Installation
 ============
 
-There are three options to install DVC: ``pip``, Homebrew, or an OS-specific package:
+Read this `instruction <https://dvc.org/doc/get-started/install>`_ to get more details. There are three
+options to install DVC: ``pip``, Homebrew, or an OS-specific package:
 
 pip (PyPI)
 ----------
 
-Stable
-^^^^^^
 .. code-block:: bash
 
    pip install dvc
 
-Development
-^^^^^^^^^^^
+Depending on the remote storage type you plan to use to keep and share your data, you might need to specify
+one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``ssh``. Or ``all_remotes`` to include them all.
+The command should look like this: ``pip install dvc[s3]`` - it installs the ``boto3`` library along with
+DVC to support the AWS S3 storage.
+
+To install the development version, run:
+
 .. code-block:: bash
 
    pip install git+git://github.com/iterative/dvc

dvc/__init__.py

@@ -10,7 +10,7 @@ import os
 import warnings
 
 
-VERSION_BASE = "0.32.1"
+VERSION_BASE = "0.34.2"
 __version__ = VERSION_BASE
 
 PACKAGEPATH = os.path.abspath(os.path.dirname(__file__))
@@ -57,6 +57,7 @@ if os.path.exists(os.path.join(HOMEPATH, "setup.py")):
     if (
         os.getenv("APPVEYOR_REPO_TAG", "").lower() != "true"
         and os.getenv("TRAVIS_TAG", "") == ""
+        and os.getenv("DVC_TEST", "").lower() != "true"
     ):
         __version__ = _update_version_file()
     else:  # pragma: no cover

dvc/analytics.py

@@ -201,6 +201,13 @@ class Analytics(object):
             json.dump(self.info, fobj)
             return fobj.name
 
+    @staticmethod
+    def _is_enabled_config(config):
+        from dvc.config import Config
+
+        core = config.config.get(Config.SECTION_CORE, {})
+        return core.get(Config.SECTION_CORE_ANALYTICS, True)
+
     @staticmethod
     def _is_enabled(cmd=None):
         from dvc.config import Config
@@ -226,10 +233,9 @@ class Analytics(object):
             config = cmd.config
             assert config is not None
 
-        core = config.config.get(Config.SECTION_CORE, {})
-        enabled = core.get(Config.SECTION_CORE_ANALYTICS, True)
+        enabled = Analytics._is_enabled_config(config)
         logger.debug(
-            "Analytics is {}abled.".format("en" if enabled else "dis")
+            "Analytics is {}.".format("enabled" if enabled else "disabled")
         )
         return enabled
 

dvc/cli.py

@@ -9,6 +9,7 @@ import argparse
 import dvc.logger as logger
 from dvc.command.base import fix_subparsers
 import dvc.command.init as init
+import dvc.command.pkg as pkg
 import dvc.command.destroy as destroy
 import dvc.command.remove as remove
 import dvc.command.move as move
@@ -30,12 +31,14 @@ import dvc.command.lock as lock
 import dvc.command.pipeline as pipeline
 import dvc.command.daemon as daemon
 import dvc.command.commit as commit
+import dvc.command.tag as tag
 from dvc.exceptions import DvcParserError
 from dvc import VERSION
 
 
 COMMANDS = [
     init,
+    pkg,
     destroy,
     add,
     remove,
@@ -58,6 +61,7 @@ COMMANDS = [
     pipeline,
     daemon,
     commit,
+    tag,
 ]
 
 

dvc/command/add.py

@@ -1,8 +1,10 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdAdd(CmdBase):
@@ -26,9 +28,14 @@ class CmdAdd(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    ADD_HELP = "Add files/directories to dvc."
+    ADD_HELP = "Take data files or directories under DVC control."
+
     add_parser = subparsers.add_parser(
-        "add", parents=[parent_parser], description=ADD_HELP, help=ADD_HELP
+        "add",
+        parents=[parent_parser],
+        description=append_doc_link(ADD_HELP, "add"),
+        help=ADD_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     add_parser.add_argument(
         "-R",

dvc/command/base.py

@@ -1,5 +1,7 @@
 from __future__ import unicode_literals
 
+import colorama
+
 import dvc.logger as logger
 
 
@@ -18,6 +20,19 @@ def fix_subparsers(subparsers):
         subparsers.dest = "cmd"
 
 
+def append_doc_link(help_message, path):
+    if not path:
+        return help_message
+    doc_base = "https://man.dvc.org/"
+    return "{message}\ndocumentation: {blue}{base}{path}{nc}".format(
+        message=help_message,
+        base=doc_base,
+        path=path,
+        blue=colorama.Fore.CYAN,
+        nc=colorama.Fore.RESET,
+    )
+
+
 class CmdBase(object):
     def __init__(self, args):
         from dvc.repo import Repo

dvc/command/cache.py

@@ -2,7 +2,7 @@ from __future__ import unicode_literals
 
 import argparse
 
-from dvc.command.base import fix_subparsers
+from dvc.command.base import fix_subparsers, append_doc_link
 from dvc.command.remote import CmdRemoteAdd
 from dvc.command.config import CmdConfig
 
@@ -21,11 +21,13 @@ def add_parser(subparsers, parent_parser):
     from dvc.command.config import parent_config_parser
 
     CACHE_HELP = "Manage cache settings."
+
     cache_parser = subparsers.add_parser(
         "cache",
         parents=[parent_parser],
-        description=CACHE_HELP,
+        description=append_doc_link(CACHE_HELP, "cache"),
         help=CACHE_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
 
     cache_subparsers = cache_parser.add_subparsers(
@@ -38,11 +40,13 @@ def add_parser(subparsers, parent_parser):
         add_help=False, parents=[parent_config_parser]
     )
     CACHE_DIR_HELP = "Configure cache directory location."
+
     cache_dir_parser = cache_subparsers.add_parser(
         "dir",
         parents=[parent_cache_config_parser],
-        description=CACHE_DIR_HELP,
+        description=append_doc_link(CACHE_HELP, "cache-dir"),
         help=CACHE_DIR_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     cache_dir_parser.add_argument(
         "-u",
@@ -53,7 +57,6 @@ def add_parser(subparsers, parent_parser):
     )
     cache_dir_parser.add_argument(
         "value",
-        nargs="?",
         default=None,
         help="Path to cache directory. Relative paths are resolved relative "
         "to the current directory and saved to config relative to the "

dvc/command/checkout.py

@@ -1,6 +1,8 @@
 from __future__ import unicode_literals
 
-from dvc.command.base import CmdBase
+import argparse
+
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdCheckout(CmdBase):
@@ -13,17 +15,20 @@ class CmdCheckout(CmdBase):
                     target=target,
                     with_deps=self.args.with_deps,
                     force=self.args.force,
+                    recursive=self.args.recursive,
                 )
         return 0
 
 
 def add_parser(subparsers, parent_parser):
     CHECKOUT_HELP = "Checkout data files from cache."
+
     checkout_parser = subparsers.add_parser(
         "checkout",
         parents=[parent_parser],
-        description=CHECKOUT_HELP,
+        description=append_doc_link(CHECKOUT_HELP, "checkout"),
         help=CHECKOUT_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     checkout_parser.add_argument(
         "-d",
@@ -39,5 +44,12 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Do not prompt when removing working directory files.",
     )
+    checkout_parser.add_argument(
+        "-R",
+        "--recursive",
+        action="store_true",
+        default=False,
+        help="Checkout all subdirectories of the specified directory.",
+    )
     checkout_parser.add_argument("targets", nargs="*", help="DVC files.")
     checkout_parser.set_defaults(func=CmdCheckout)

dvc/command/commit.py

@@ -1,8 +1,10 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdCommit(CmdBase):
@@ -29,12 +31,14 @@ class CmdCommit(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    COMMIT_HELP = "Record changes to the repository."
+    COMMIT_HELP = "Save changed data to cache and update DVC files."
+
     commit_parser = subparsers.add_parser(
         "commit",
         parents=[parent_parser],
-        description=COMMIT_HELP,
+        description=append_doc_link(COMMIT_HELP, "commit"),
         help=COMMIT_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     commit_parser.add_argument(
         "-f",

dvc/command/config.py

@@ -4,7 +4,7 @@ import os
 import argparse
 
 import dvc.logger as logger
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 from dvc.config import Config
 from dvc.exceptions import DvcException
 
@@ -99,11 +99,13 @@ parent_config_parser.add_argument(
 
 def add_parser(subparsers, parent_parser):
     CONFIG_HELP = "Get or set config options."
+
     config_parser = subparsers.add_parser(
         "config",
         parents=[parent_config_parser, parent_parser],
-        description=CONFIG_HELP,
+        description=append_doc_link(CONFIG_HELP, "config"),
         help=CONFIG_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     config_parser.add_argument(
         "-u",

dvc/command/daemon.py

@@ -43,7 +43,7 @@ def add_parser(subparsers, parent_parser):
         "daemon",
         parents=[parent_parser],
         description=DAEMON_HELP,
-        help=DAEMON_HELP,
+        add_help=False,
     )
 
     daemon_subparsers = daemon_parser.add_subparsers(

dvc/command/data_sync.py

@@ -3,7 +3,7 @@ from __future__ import unicode_literals
 import argparse
 
 import dvc.logger as logger
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdDataBase(CmdBase):
@@ -120,12 +120,14 @@ def add_parser(subparsers, _parent_parser):
     from dvc.command.status import CmdDataStatus
 
     # Pull
-    PULL_HELP = "Pull data files from the cloud."
+    PULL_HELP = "Pull data files from a DVC remote storage."
+
     pull_parser = subparsers.add_parser(
         "pull",
         parents=[shared_parent_parser()],
-        description=PULL_HELP,
+        description=append_doc_link(PULL_HELP, "pull"),
         help=PULL_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     pull_parser.add_argument(
         "-r", "--remote", help="Remote repository to pull from."
@@ -168,12 +170,14 @@ def add_parser(subparsers, _parent_parser):
     pull_parser.set_defaults(func=CmdDataPull)
 
     # Push
-    PUSH_HELP = "Push data files to the cloud."
+    PUSH_HELP = "Push data files to a DVC remote storage."
+
     push_parser = subparsers.add_parser(
         "push",
         parents=[shared_parent_parser()],
-        description=PUSH_HELP,
+        description=append_doc_link(PUSH_HELP, "push"),
         help=PUSH_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     push_parser.add_argument(
         "-r", "--remote", help="Remote repository to push to."
@@ -209,12 +213,14 @@ def add_parser(subparsers, _parent_parser):
     push_parser.set_defaults(func=CmdDataPush)
 
     # Fetch
-    FETCH_HELP = "Fetch data files from the cloud."
+    FETCH_HELP = "Fetch data files from a DVC remote storage."
+
     fetch_parser = subparsers.add_parser(
         "fetch",
         parents=[shared_parent_parser()],
-        description=FETCH_HELP,
+        description=append_doc_link(FETCH_HELP, "fetch"),
         help=FETCH_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     fetch_parser.add_argument(
         "-r", "--remote", help="Remote repository to fetch from."
@@ -250,13 +256,17 @@ def add_parser(subparsers, _parent_parser):
     fetch_parser.set_defaults(func=CmdDataFetch)
 
     # Status
-    STATUS_HELP = "Show the repo status."
+    STATUS_HELP = (
+        "Show changed stages, compare local cache and a remote storage."
+    )
+
     status_parser = subparsers.add_parser(
         "status",
         parents=[shared_parent_parser()],
-        description=STATUS_HELP,
+        description=append_doc_link(STATUS_HELP, "status"),
         help=STATUS_HELP,
         conflict_handler="resolve",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     status_parser.add_argument(
         "-q",

dvc/command/destroy.py

@@ -1,8 +1,10 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.prompt as prompt
 import dvc.logger as logger
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 from dvc.exceptions import DvcException
 
 
@@ -30,15 +32,14 @@ class CmdDestroy(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    DESTROY_HELP = (
-        "Destroy dvc. Will remove all repo's information, "
-        "data files and cache."
-    )
+    DESTROY_HELP = "Remove DVC files, local DVC config and data cache."
+
     destroy_parser = subparsers.add_parser(
         "destroy",
         parents=[parent_parser],
-        description=DESTROY_HELP,
+        description=append_doc_link(DESTROY_HELP, "destroy"),
         help=DESTROY_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     destroy_parser.add_argument(
         "-f",

dvc/command/gc.py

@@ -1,10 +1,11 @@
 from __future__ import unicode_literals
 
+import argparse
 import os
 import dvc.prompt as prompt
 import dvc.logger as logger
 
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdGC(CmdBase):
@@ -46,13 +47,17 @@ class CmdGC(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    GC_HELP = (
-        "Collect garbage from your DVC cache. \n"
-        "Deletes all files in the cache which are not in use by the "
-        "specified git references (defaults to just HEAD)."
+    GC_HELP = "Collect unused data from DVC cache or a remote storage."
+    GC_DESCRIPTION = (
+        "Deletes all files in the cache or a remote which are not in\n"
+        "use by the specified git references (defaults to just HEAD)."
     )
     gc_parser = subparsers.add_parser(
-        "gc", parents=[parent_parser], description=GC_HELP, help=GC_HELP
+        "gc",
+        parents=[parent_parser],
+        description=append_doc_link(GC_DESCRIPTION, "gc"),
+        help=GC_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     gc_parser.add_argument(
         "-a",
@@ -76,7 +81,7 @@ def add_parser(subparsers, parent_parser):
         help="Collect garbage in remote repository.",
     )
     gc_parser.add_argument(
-        "-r", "--remote", help="Remote repository to collect garbage in."
+        "-r", "--remote", help="Remote storage to collect garbage in."
     )
     gc_parser.add_argument(
         "-f",

dvc/command/imp.py

@@ -1,11 +1,12 @@
 from __future__ import unicode_literals
 
+import argparse
 import os
 
 import dvc.logger as logger
 from dvc.utils.compat import urlparse
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdImport(CmdBase):
@@ -28,25 +29,26 @@ class CmdImport(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    IMPORT_HELP = "Import files from URL."
+    IMPORT_HELP = "Download or copy files from URL and take under DVC control."
+
     import_parser = subparsers.add_parser(
         "import",
         parents=[parent_parser],
-        description=IMPORT_HELP,
+        description=append_doc_link(IMPORT_HELP, "import"),
         help=IMPORT_HELP,
+        formatter_class=argparse.RawTextHelpFormatter,
     )
     import_parser.add_argument(
         "url",
-        help="URL. Supported urls: "
-        "'/path/to/file', "
-        "'C:\\\\path\\to\\file', "
-        "'https://example.com/path/to/file', "
-        "'s3://bucket/path/to/file', "
-        "'gs://bucket/path/to/file', "
-        "'hdfs://example.com/path/to/file', "
-        "'ssh://example.com:/path/to/file', "
-        "'remote://myremote/path/to/file'(see "
-        "`dvc remote` commands). ",
+        help="Supported urls:\n"
+        "/path/to/file\n"
+        "C:\\\\path\\to\\file\n"
+        "https://example.com/path/to/file\n"
+        "s3://bucket/path/to/file\n"
+        "gs://bucket/path/to/file\n"
+        "hdfs://example.com/path/to/file\n"
+        "ssh://example.com:/path/to/file\n"
+        "remote://myremote/path/to/file (see `dvc remote`)",
     )
     import_parser.add_argument(
         "--resume",
@@ -54,5 +56,7 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Resume previously started download.",
     )
-    import_parser.add_argument("out", nargs="?", help="Output.")
+    import_parser.add_argument(
+        "out", nargs="?", help="Destination path to put files to."
+    )
     import_parser.set_defaults(func=CmdImport)

dvc/command/init.py

@@ -1,6 +1,9 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
+from dvc.command.base import append_doc_link
 
 
 class CmdInit(object):
@@ -24,11 +27,18 @@ class CmdInit(object):
 
 def add_parser(subparsers, parent_parser):
     """Setup parser for `dvc init`."""
-    INIT_HELP = (
-        "Initialize dvc over a directory " "(should already be a git dir)."
+    INIT_HELP = "Initialize DVC in the current directory."
+    INIT_DESCRIPTION = (
+        "Initialize DVC in the current directory. Expects directory\n"
+        "to be a Git repository unless --no-scm option is specified."
     )
+
     init_parser = subparsers.add_parser(
-        "init", parents=[parent_parser], description=INIT_HELP, help=INIT_HELP
+        "init",
+        parents=[parent_parser],
+        description=append_doc_link(INIT_DESCRIPTION, "init"),
+        help=INIT_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     init_parser.add_argument(
         "--no-scm",
@@ -42,6 +52,9 @@ def add_parser(subparsers, parent_parser):
         "--force",
         action="store_true",
         default=False,
-        help="Overwrite '.dvc' if it exists. Will remove all local cache.",
+        help=(
+            "Overwrite existing '.dvc' directory. "
+            "This operation removes local cache."
+        ),
     )
     init_parser.set_defaults(func=CmdInit)

dvc/command/install.py

@@ -1,7 +1,9 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdInstall(CmdBase):
@@ -15,11 +17,12 @@ class CmdInstall(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    INSTALL_HELP = "Install dvc hooks into the repository."
+    INSTALL_HELP = "Install DVC git hooks into the repository."
     install_parser = subparsers.add_parser(
         "install",
         parents=[parent_parser],
-        description=INSTALL_HELP,
+        description=append_doc_link(INSTALL_HELP, "install"),
         help=INSTALL_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     install_parser.set_defaults(func=CmdInstall)

dvc/command/lock.py

@@ -1,12 +1,14 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdLockBase(CmdBase):
-    def run(self, unlock=False):
+    def _run(self, unlock):
         for target in self.args.targets:
             try:
                 self.repo.lock_stage(target, unlock=unlock)
@@ -23,18 +25,22 @@ class CmdLockBase(CmdBase):
 
 class CmdLock(CmdLockBase):
     def run(self):
-        return super(CmdLock, self).run(False)
+        return self._run(False)
 
 
 class CmdUnlock(CmdLockBase):
     def run(self):
-        return super(CmdUnlock, self).run(True)
+        return self._run(True)
 
 
 def add_parser(subparsers, parent_parser):
     LOCK_HELP = "Lock DVC file."
     lock_parser = subparsers.add_parser(
-        "lock", parents=[parent_parser], description=LOCK_HELP, help=LOCK_HELP
+        "lock",
+        parents=[parent_parser],
+        description=append_doc_link(LOCK_HELP, "lock"),
+        help=LOCK_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     lock_parser.add_argument("targets", nargs="+", help="DVC files.")
     lock_parser.set_defaults(func=CmdLock)
@@ -43,8 +49,9 @@ def add_parser(subparsers, parent_parser):
     unlock_parser = subparsers.add_parser(
         "unlock",
         parents=[parent_parser],
-        description=UNLOCK_HELP,
+        description=append_doc_link(UNLOCK_HELP, "unlock"),
         help=UNLOCK_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     unlock_parser.add_argument("targets", nargs="+", help="DVC files.")
     unlock_parser.set_defaults(func=CmdUnlock)

dvc/command/metrics.py

@@ -1,19 +1,36 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase, fix_subparsers
+from dvc.command.base import CmdBase, fix_subparsers, append_doc_link
+
+
+def show_metrics(metrics, all_branches=False, all_tags=False):
+    """
+    Args:
+        metrics (list): Where each element is either a `list`
+            if an xpath was specified, otherwise a `str`
+    """
+    for branch, val in metrics.items():
+        if all_branches or all_tags:
+            logger.info("{branch}:".format(branch=branch))
+
+        for fname, metric in val.items():
+            lines = metric if type(metric) is list else metric.splitlines()
+
+            if len(lines) > 1:
+                logger.info("\t{fname}:".format(fname=fname))
+
+                for line in lines:
+                    logger.info("\t\t{content}".format(content=line))
+
+            else:
+                logger.info("\t{}: {}".format(fname, metric))
 
 
 class CmdMetricsShow(CmdBase):
-    def _show(self, metrics):
-        for branch, val in metrics.items():
-            if self.args.all_branches or self.args.all_tags:
-                logger.info("{}:".format(branch))
-
-            for fname, metric in val.items():
-                logger.info("\t{}: {}".format(fname, metric))
-
     def run(self):
         typ = self.args.type
         xpath = self.args.xpath
@@ -27,7 +44,7 @@ class CmdMetricsShow(CmdBase):
                 recursive=self.args.recursive,
             )
 
-            self._show(metrics)
+            show_metrics(metrics, self.args.all_branches, self.args.all_tags)
         except DvcException:
             logger.error("failed to show metrics")
             return 1
@@ -75,15 +92,14 @@ class CmdMetricsRemove(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    METRICS_HELP = (
-        "A set of commands to add, manage, collect and display project "
-        "metrics."
-    )
+    METRICS_HELP = "Commands to add, manage, collect and display metrics."
+
     metrics_parser = subparsers.add_parser(
         "metrics",
         parents=[parent_parser],
-        description=METRICS_HELP,
+        description=append_doc_link(METRICS_HELP, "metrics"),
         help=METRICS_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
 
     metrics_subparsers = metrics_parser.add_subparsers(
@@ -97,14 +113,21 @@ def add_parser(subparsers, parent_parser):
     metrics_show_parser = metrics_subparsers.add_parser(
         "show",
         parents=[parent_parser],
-        description=METRICS_SHOW_HELP,
+        description=append_doc_link(METRICS_SHOW_HELP, "metrics-show"),
         help=METRICS_SHOW_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     metrics_show_parser.add_argument(
         "path", nargs="?", help="Path to a metric file or a directory."
     )
     metrics_show_parser.add_argument(
-        "-t", "--type", help="Type of metrics (raw/json/tsv/htsv/csv/hcsv)."
+        "-t",
+        "--type",
+        help=(
+            "Type of metrics (json/tsv/htsv/csv/hcsv). "
+            "It can be detected by the file extension automatically. "
+            "Unsupported types will be treated as raw."
+        ),
     )
     metrics_show_parser.add_argument(
         "-x", "--xpath", help="json/tsv/htsv/csv/hcsv path."
@@ -139,8 +162,9 @@ def add_parser(subparsers, parent_parser):
     metrics_add_parser = metrics_subparsers.add_parser(
         "add",
         parents=[parent_parser],
-        description=METRICS_ADD_HELP,
+        description=append_doc_link(METRICS_ADD_HELP, "metrics-add"),
         help=METRICS_ADD_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     metrics_add_parser.add_argument(
         "-t", "--type", help="Type of metrics (raw/json/tsv/htsv/csv/hcsv)."
@@ -155,8 +179,9 @@ def add_parser(subparsers, parent_parser):
     metrics_modify_parser = metrics_subparsers.add_parser(
         "modify",
         parents=[parent_parser],
-        description=METRICS_MODIFY_HELP,
+        description=append_doc_link(METRICS_MODIFY_HELP, "metrics-modify"),
         help=METRICS_MODIFY_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     metrics_modify_parser.add_argument(
         "-t", "--type", help="Type of metrics (raw/json/tsv/htsv/csv/hcsv)."
@@ -171,8 +196,9 @@ def add_parser(subparsers, parent_parser):
     metrics_remove_parser = metrics_subparsers.add_parser(
         "remove",
         parents=[parent_parser],
-        description=METRICS_REMOVE_HELP,
+        description=append_doc_link(METRICS_REMOVE_HELP, "metrics-remove"),
         help=METRICS_REMOVE_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     metrics_remove_parser.add_argument("path", help="Path to a metric file.")
     metrics_remove_parser.set_defaults(func=CmdMetricsRemove)

dvc/command/move.py

@@ -1,8 +1,10 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdMove(CmdBase):
@@ -19,14 +21,19 @@ class CmdMove(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    description = (
-        "Rename or move a data file or a directory that "
-        "is under DVC control. It renames and modifies "
-        "the corresponding DVC file to reflect the changes."
+    MOVE_HELP = "Rename or move a DVC controlled data file or a directory."
+    MOVE_DESCRIPTION = (
+        "Rename or move a DVC controlled data file or a directory.\n"
+        "It renames and modifies the corresponding DVC file to reflect the"
+        " changes."
     )
-    help = "Rename or move a DVC controlled data file or a directory."
+
     move_parser = subparsers.add_parser(
-        "move", parents=[parent_parser], description=description, help=help
+        "move",
+        parents=[parent_parser],
+        description=append_doc_link(MOVE_DESCRIPTION, "move"),
+        help=MOVE_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     move_parser.add_argument(
         "src", help="Source path to a data file or directory."

dvc/command/pipeline.py

@@ -1,12 +1,14 @@
 from __future__ import unicode_literals
 
+import argparse
+
 from dvc.utils.compat import str
 
 import os
 
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase, fix_subparsers
+from dvc.command.base import CmdBase, fix_subparsers, append_doc_link
 
 
 class CmdPipelineShow(CmdBase):
@@ -71,23 +73,47 @@ class CmdPipelineShow(CmdBase):
             else:
                 edges.append((from_stage.relpath, to_stage.relpath))
 
-        return nodes, edges
+        return nodes, edges, networkx.is_tree(G)
 
     def _show_ascii(self, target, commands, outs):
         from dvc.dagascii import draw
 
-        nodes, edges = self.__build_graph(target, commands, outs)
+        nodes, edges, _ = self.__build_graph(target, commands, outs)
 
         if not nodes:
             return
 
         draw(nodes, edges)
 
+    def _show_dependencies_tree(self, target, commands, outs):
+        from treelib import Tree
+
+        nodes, edges, is_tree = self.__build_graph(target, commands, outs)
+        if not nodes:
+            return
+        if not is_tree:
+            raise DvcException(
+                "DAG is not a tree, can not print it in tree-structure way, "
+                "please use --ascii instead"
+            )
+
+        tree = Tree()
+        tree.create_node(target, target)  # Root node
+        observe_list = [target]
+        while len(observe_list) > 0:
+            current_root = observe_list[0]
+            for edge in edges:
+                if edge[0] == current_root:
+                    tree.create_node(edge[1], edge[1], parent=current_root)
+                    observe_list.append(edge[1])
+            observe_list.pop(0)
+        tree.show()
+
     def __write_dot(self, target, commands, outs, filename):
         import networkx
         from networkx.drawing.nx_pydot import write_dot
 
-        _, edges = self.__build_graph(target, commands, outs)
+        _, edges, _ = self.__build_graph(target, commands, outs)
         edges = [edge[::-1] for edge in edges]
 
         simple_g = networkx.DiGraph()
@@ -111,6 +137,10 @@ class CmdPipelineShow(CmdBase):
                         self.args.outs,
                         self.args.dot,
                     )
+                elif self.args.tree:
+                    self._show_dependencies_tree(
+                        target, self.args.commands, self.args.outs
+                    )
                 else:
                     self._show(target, self.args.commands, self.args.outs)
             except DvcException:
@@ -137,12 +167,13 @@ class CmdPipelineList(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    PIPELINE_HELP = "Manage pipeline."
+    PIPELINE_HELP = "Manage pipelines."
     pipeline_parser = subparsers.add_parser(
         "pipeline",
         parents=[parent_parser],
-        description=PIPELINE_HELP,
+        description=append_doc_link(PIPELINE_HELP, "pipeline"),
         help=PIPELINE_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
 
     pipeline_subparsers = pipeline_parser.add_subparsers(
@@ -156,8 +187,9 @@ def add_parser(subparsers, parent_parser):
     pipeline_show_parser = pipeline_subparsers.add_parser(
         "show",
         parents=[parent_parser],
-        description=PIPELINE_SHOW_HELP,
+        description=append_doc_link(PIPELINE_SHOW_HELP, "pipeline-show"),
         help=PIPELINE_SHOW_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     pipeline_show_group = pipeline_show_parser.add_mutually_exclusive_group()
     pipeline_show_group.add_argument(
@@ -183,6 +215,12 @@ def add_parser(subparsers, parent_parser):
     pipeline_show_parser.add_argument(
         "--dot", help="Write DAG in .dot format."
     )
+    pipeline_show_parser.add_argument(
+        "--tree",
+        action="store_true",
+        default=False,
+        help="Output DAG as Dependencies Tree.",
+    )
     pipeline_show_parser.add_argument(
         "targets", nargs="*", help="DVC files. 'Dvcfile' by default."
     )
@@ -192,7 +230,8 @@ def add_parser(subparsers, parent_parser):
     pipeline_list_parser = pipeline_subparsers.add_parser(
         "list",
         parents=[parent_parser],
-        description=PIPELINE_LIST_HELP,
+        description=append_doc_link(PIPELINE_LIST_HELP, "pipeline-list"),
         help=PIPELINE_LIST_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     pipeline_list_parser.set_defaults(func=CmdPipelineList)

dvc/command/pkg.py

@@ -0,0 +1,89 @@
+from __future__ import unicode_literals
+
+import argparse
+
+import dvc.logger as logger
+from dvc.exceptions import DvcException
+from dvc.command.base import CmdBase, fix_subparsers, append_doc_link
+from dvc.repo.pkg import PackageParams
+
+
+class CmdPkg(CmdBase):
+    def run(self, unlock=False):
+        try:
+            pkg_params = PackageParams(
+                self.args.address,
+                self.args.target_dir,
+                self.args.select,
+                self.args.file,
+            )
+            return self.repo.install_pkg(pkg_params)
+        except DvcException:
+            logger.error(
+                "failed to install package '{}'".format(self.args.address)
+            )
+            return 1
+        pass
+
+
+def add_parser(subparsers, parent_parser):
+    from dvc.command.config import parent_config_parser
+
+    PKG_HELP = "Manage DVC packages."
+    pkg_parser = subparsers.add_parser(
+        "pkg",
+        parents=[parent_parser],
+        description=append_doc_link(PKG_HELP, "pkg"),
+        help=PKG_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        add_help=False,
+    )
+
+    pkg_subparsers = pkg_parser.add_subparsers(
+        dest="cmd", help="Use dvc pkg CMD --help for command-specific help."
+    )
+
+    fix_subparsers(pkg_subparsers)
+
+    PKG_INSTALL_HELP = "Install package."
+    pkg_install_parser = pkg_subparsers.add_parser(
+        "install",
+        parents=[parent_config_parser, parent_parser],
+        description=append_doc_link(PKG_INSTALL_HELP, "pkg-install"),
+        help=PKG_INSTALL_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    pkg_install_parser.add_argument(
+        "address",
+        nargs="?",
+        default="",
+        help="Package address: git://<url> or https://github.com/...",
+    )
+    pkg_install_parser.add_argument(
+        "target_dir",
+        metavar="target",
+        nargs="?",
+        default=".",
+        help="Target directory to deploy package outputs. "
+        "Default value is the current dir.",
+    )
+    pkg_install_parser.add_argument(
+        "-s",
+        "--select",
+        metavar="OUT",
+        action="append",
+        default=[],
+        help="Select and persist only specified outputs from a package. "
+        "The parameter can be used multiple times. "
+        "All outputs will be selected by default.",
+    )
+    pkg_install_parser.add_argument(
+        "-f",
+        "--file",
+        help="Specify name of the stage file. It should be "
+        "either 'Dvcfile' or have a '.dvc' suffix (e.g. "
+        "'prepare.dvc', 'clean.dvc', etc). "
+        "By default the file has 'mod_' prefix and imported package name "
+        "followed by .dvc",
+    )
+    pkg_install_parser.set_defaults(func=CmdPkg)

dvc/command/remote.py

@@ -1,12 +1,13 @@
 from __future__ import unicode_literals
 
+import argparse
 import os
 import re
 
 import dvc.logger as logger
-from dvc.command.base import fix_subparsers
-from dvc.config import Config
+from dvc.command.base import append_doc_link, fix_subparsers
 from dvc.command.config import CmdConfig
+from dvc.config import Config
 
 
 class CmdRemoteAdd(CmdConfig):
@@ -38,6 +39,14 @@ class CmdRemoteAdd(CmdConfig):
             )
 
         section = Config.SECTION_REMOTE_FMT.format(self.args.name)
+        if (section in self.configobj.keys()) and not self.args.force:
+            logger.error(
+                "Remote with name {} already exists. "
+                "Use -f (--force) to overwrite remote "
+                "with new value".format(self.args.name)
+            )
+            return 1
+
         ret = self._set(section, Config.SECTION_REMOTE_URL, self.args.url)
         if ret != 0:
             return ret
@@ -119,12 +128,13 @@ class CmdRemoteList(CmdConfig):
 def add_parser(subparsers, parent_parser):
     from dvc.command.config import parent_config_parser
 
-    REMOTE_HELP = "Manage set of tracked repositories."
+    REMOTE_HELP = "Manage remote storage configuration."
     remote_parser = subparsers.add_parser(
         "remote",
         parents=[parent_parser],
-        description=REMOTE_HELP,
+        description=append_doc_link(REMOTE_HELP, "remote"),
         help=REMOTE_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
 
     remote_subparsers = remote_parser.add_subparsers(
@@ -138,14 +148,14 @@ def add_parser(subparsers, parent_parser):
     remote_add_parser = remote_subparsers.add_parser(
         "add",
         parents=[parent_config_parser, parent_parser],
-        description=REMOTE_ADD_HELP,
+        description=append_doc_link(REMOTE_ADD_HELP, "remote-add"),
         help=REMOTE_ADD_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     remote_add_parser.add_argument("name", help="Name.")
     remote_add_parser.add_argument(
         "url",
-        help="URL. See full list of supported urls at "
-        "https://dvc.org/doc/commands-reference/remote",
+        help="URL. See full list of supported urls at " "man.dvc.org/remote",
     )
     remote_add_parser.add_argument(
         "-d",
@@ -154,14 +164,22 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Set as default remote.",
     )
+    remote_add_parser.add_argument(
+        "-f",
+        "--force",
+        action="store_true",
+        default=False,
+        help="Force overwriting existing configs",
+    )
     remote_add_parser.set_defaults(func=CmdRemoteAdd)
 
     REMOTE_DEFAULT_HELP = "Set/unset default remote."
     remote_default_parser = remote_subparsers.add_parser(
         "default",
         parents=[parent_config_parser, parent_parser],
-        description=REMOTE_DEFAULT_HELP,
+        description=append_doc_link(REMOTE_DEFAULT_HELP, "remote-default"),
         help=REMOTE_DEFAULT_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     remote_default_parser.add_argument(
         "name", nargs="?", help="Name of the remote."
@@ -179,8 +197,9 @@ def add_parser(subparsers, parent_parser):
     remote_remove_parser = remote_subparsers.add_parser(
         "remove",
         parents=[parent_config_parser, parent_parser],
-        description=REMOTE_REMOVE_HELP,
+        description=append_doc_link(REMOTE_REMOVE_HELP, "remote-remove"),
         help=REMOTE_REMOVE_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     remote_remove_parser.add_argument("name", help="Name")
     remote_remove_parser.set_defaults(func=CmdRemoteRemove)
@@ -189,8 +208,9 @@ def add_parser(subparsers, parent_parser):
     remote_modify_parser = remote_subparsers.add_parser(
         "modify",
         parents=[parent_config_parser, parent_parser],
-        description=REMOTE_MODIFY_HELP,
+        description=append_doc_link(REMOTE_MODIFY_HELP, "remote-modify"),
         help=REMOTE_MODIFY_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     remote_modify_parser.add_argument("name", help="Name.")
     remote_modify_parser.add_argument("option", help="Option.")
@@ -204,11 +224,12 @@ def add_parser(subparsers, parent_parser):
     )
     remote_modify_parser.set_defaults(func=CmdRemoteModify)
 
-    REMOTE_LIST_HELP = "List remotes."
+    REMOTE_LIST_HELP = "List available remotes."
     remote_list_parser = remote_subparsers.add_parser(
         "list",
         parents=[parent_config_parser, parent_parser],
-        description=REMOTE_LIST_HELP,
+        description=append_doc_link(REMOTE_LIST_HELP, "remote-list"),
         help=REMOTE_LIST_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     remote_list_parser.set_defaults(func=CmdRemoteList)

dvc/command/remove.py

@@ -1,9 +1,11 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.prompt as prompt
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdRemove(CmdBase):
@@ -42,8 +44,9 @@ def add_parser(subparsers, parent_parser):
     remove_parser = subparsers.add_parser(
         "remove",
         parents=[parent_parser],
-        description=REMOVE_HELP,
+        description=append_doc_link(REMOVE_HELP, "remove"),
         help=REMOVE_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     remove_parser_group = remove_parser.add_mutually_exclusive_group()
     remove_parser_group.add_argument(

dvc/command/repro.py

@@ -1,9 +1,11 @@
 from __future__ import unicode_literals
 
+import argparse
 import os
 
 import dvc.logger as logger
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
+from dvc.command.metrics import show_metrics
 from dvc.command.status import CmdDataStatus
 from dvc.exceptions import DvcException
 
@@ -40,7 +42,8 @@ class CmdRepro(CmdBase):
                     logger.info(CmdDataStatus.UP_TO_DATE_MSG)
 
                 if self.args.metrics:
-                    self.repo.metrics.show()
+                    metrics = self.repo.metrics.show()
+                    show_metrics(metrics)
             except DvcException:
                 logger.error()
                 ret = 1
@@ -51,15 +54,18 @@ class CmdRepro(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    REPRO_HELP = "Reproduce DVC file. Default file name - 'Dvcfile'."
+    REPRO_HELP = "Check for changes and reproduce DVC file and dependencies."
     repro_parser = subparsers.add_parser(
         "repro",
         parents=[parent_parser],
-        description=REPRO_HELP,
+        description=append_doc_link(REPRO_HELP, "repro"),
         help=REPRO_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     repro_parser.add_argument(
-        "targets", nargs="*", help="DVC file to reproduce."
+        "targets",
+        nargs="*",
+        help="DVC file to reproduce (default - 'Dvcfile').",
     )
     repro_parser.add_argument(
         "-f",
@@ -80,7 +86,7 @@ def add_parser(subparsers, parent_parser):
         "-c",
         "--cwd",
         default=os.path.curdir,
-        help="Directory within your repo to reroduce from.",
+        help="Directory within your repo to reproduce from.",
     )
     repro_parser.add_argument(
         "-m",

dvc/command/root.py

@@ -1,9 +1,10 @@
 from __future__ import unicode_literals
 
+import argparse
 import os
 
 import dvc.logger as logger
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdRoot(CmdBase):
@@ -16,8 +17,12 @@ class CmdRoot(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    ROOT_HELP = "Relative path to repo's directory."
+    ROOT_HELP = "Relative path to the repository's directory."
     root_parser = subparsers.add_parser(
-        "root", parents=[parent_parser], description=ROOT_HELP, help=ROOT_HELP
+        "root",
+        parents=[parent_parser],
+        description=append_doc_link(ROOT_HELP, "root"),
+        help=ROOT_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     root_parser.set_defaults(func=CmdRoot)

dvc/command/run.py

@@ -3,7 +3,7 @@ from __future__ import unicode_literals
 import argparse
 
 import dvc.logger as logger
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 from dvc.exceptions import DvcException
 
 
@@ -18,12 +18,15 @@ class CmdRun(CmdBase):
                 self.args.outs_no_cache,
                 self.args.metrics,
                 self.args.metrics_no_cache,
+                self.args.outs_persist,
+                self.args.outs_persist_no_cache,
                 self.args.command,
             ]
         ):  # pragma: no cover
             logger.error(
-                "too few arguments. Specify at least one: '-d', "
-                "'-o', '-O', '-m', '-M', 'command'."
+                "too few arguments. Specify at least one: '-d', '-o', '-O',"
+                " '-m', '-M', '--outs-persist', '--outs-persist-no-cache',"
+                " 'command'."
             )
             return 1
 
@@ -43,6 +46,8 @@ class CmdRun(CmdBase):
                 ignore_build_cache=self.args.ignore_build_cache,
                 remove_outs=self.args.remove_outs,
                 no_commit=self.args.no_commit,
+                outs_persist=self.args.outs_persist,
+                outs_persist_no_cache=self.args.outs_persist_no_cache,
             )
         except DvcException:
             logger.error("failed to run command")
@@ -73,12 +78,13 @@ class CmdRun(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    RUN_HELP = (
-        "Generate a stage file from a given "
-        "command and execute the command."
-    )
+    RUN_HELP = "Generate a stage file from a command and execute the command."
     run_parser = subparsers.add_parser(
-        "run", parents=[parent_parser], description=RUN_HELP, help=RUN_HELP
+        "run",
+        parents=[parent_parser],
+        description=append_doc_link(RUN_HELP, "run"),
+        help=RUN_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     run_parser.add_argument(
         "-d",
@@ -174,6 +180,20 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Don't put files/directories into cache.",
     )
+    run_parser.add_argument(
+        "--outs-persist",
+        action="append",
+        default=[],
+        help="Declare output file or directory that will not be "
+        "removed upon repro.",
+    )
+    run_parser.add_argument(
+        "--outs-persist-no-cache",
+        action="append",
+        default=[],
+        help="Declare output file or directory that will not be "
+        "removed upon repro (do not put into DVC cache).",
+    )
     run_parser.add_argument(
         "command", nargs=argparse.REMAINDER, help="Command to execute."
     )

dvc/command/tag.py

@@ -0,0 +1,151 @@
+import yaml
+
+import dvc.logger as logger
+from dvc.exceptions import DvcException
+from dvc.command.base import CmdBase, fix_subparsers, append_doc_link
+
+
+class CmdTagAdd(CmdBase):
+    def run(self):
+        for target in self.args.targets:
+            try:
+                self.repo.tag.add(
+                    self.args.tag,
+                    target=target,
+                    with_deps=self.args.with_deps,
+                    recursive=self.args.recursive,
+                )
+            except DvcException:
+                logger.error("failed to add tag")
+                return 1
+        return 0
+
+
+class CmdTagRemove(CmdBase):
+    def run(self):
+        for target in self.args.targets:
+            try:
+                self.repo.tag.remove(
+                    self.args.tag,
+                    target=target,
+                    with_deps=self.args.with_deps,
+                    recursive=self.args.recursive,
+                )
+            except DvcException:
+                logger.error("failed to remove tag")
+                return 1
+        return 0
+
+
+class CmdTagList(CmdBase):
+    def run(self):
+        for target in self.args.targets:
+            try:
+                tags = self.repo.tag.list(
+                    target,
+                    with_deps=self.args.with_deps,
+                    recursive=self.args.recursive,
+                )
+                if tags:
+                    logger.info(yaml.dump(tags, default_flow_style=False))
+            except DvcException:
+                logger.error("failed list tags")
+                return 1
+        return 0
+
+
+def add_parser(subparsers, parent_parser):
+    TAG_HELP = "A set of commands to manage DVC tags."
+    tag_parser = subparsers.add_parser(
+        "tag",
+        parents=[parent_parser],
+        description=append_doc_link(TAG_HELP, "tag"),
+        add_help=False,
+    )
+
+    tag_subparsers = tag_parser.add_subparsers(
+        dest="cmd",
+        help="Use DVC tag CMD --help to display command-specific help.",
+    )
+
+    fix_subparsers(tag_subparsers)
+
+    TAG_ADD_HELP = "Add DVC tag."
+    tag_add_parser = tag_subparsers.add_parser(
+        "add",
+        parents=[parent_parser],
+        description=append_doc_link(TAG_ADD_HELP, "tag-add"),
+        help=TAG_ADD_HELP,
+    )
+    tag_add_parser.add_argument("tag", help="Dvc tag.")
+    tag_add_parser.add_argument(
+        "targets", nargs="*", default=[None], help="Dvc files."
+    )
+    tag_add_parser.add_argument(
+        "-d",
+        "--with-deps",
+        action="store_true",
+        default=False,
+        help="Add tag for all dependencies of the specified DVC file.",
+    )
+    tag_add_parser.add_argument(
+        "-R",
+        "--recursive",
+        action="store_true",
+        default=False,
+        help="Add tag for subdirectories of the specified directory.",
+    )
+    tag_add_parser.set_defaults(func=CmdTagAdd)
+
+    TAG_REMOVE_HELP = "Remove DVC tag."
+    tag_remove_parser = tag_subparsers.add_parser(
+        "remove",
+        parents=[parent_parser],
+        description=append_doc_link(TAG_REMOVE_HELP, "tag-remove"),
+        help=TAG_REMOVE_HELP,
+    )
+    tag_remove_parser.add_argument("tag", help="Dvc tag.")
+    tag_remove_parser.add_argument(
+        "targets", nargs="*", default=[None], help="Dvc files."
+    )
+    tag_remove_parser.add_argument(
+        "-d",
+        "--with-deps",
+        action="store_true",
+        default=False,
+        help="Remove tag for all dependencies of the specified DVC file.",
+    )
+    tag_remove_parser.add_argument(
+        "-R",
+        "--recursive",
+        action="store_true",
+        default=False,
+        help="Remove tag for subdirectories of the specified directory.",
+    )
+    tag_remove_parser.set_defaults(func=CmdTagRemove)
+
+    TAG_LIST_HELP = "List DVC tags."
+    tag_list_parser = tag_subparsers.add_parser(
+        "list",
+        parents=[parent_parser],
+        description=append_doc_link(TAG_LIST_HELP, "tag-list"),
+        help=TAG_LIST_HELP,
+    )
+    tag_list_parser.add_argument(
+        "targets", nargs="*", default=[None], help="Dvc files."
+    )
+    tag_list_parser.add_argument(
+        "-d",
+        "--with-deps",
+        action="store_true",
+        default=False,
+        help="List tags for all dependencies of the specified DVC file.",
+    )
+    tag_list_parser.add_argument(
+        "-R",
+        "--recursive",
+        action="store_true",
+        default=False,
+        help="List tags for subdirectories of the specified directory.",
+    )
+    tag_list_parser.set_defaults(func=CmdTagList)

dvc/command/unprotect.py

@@ -1,8 +1,10 @@
 from __future__ import unicode_literals
 
+import argparse
+
 import dvc.logger as logger
 from dvc.exceptions import DvcException
-from dvc.command.base import CmdBase
+from dvc.command.base import CmdBase, append_doc_link
 
 
 class CmdUnprotect(CmdBase):
@@ -18,12 +20,13 @@ class CmdUnprotect(CmdBase):
 
 
 def add_parser(subparsers, parent_parser):
-    UNPROTECT_HELP = "Unprotect data file/directory."
+    UNPROTECT_HELP = "Unprotect a data file/directory."
     unprotect_parser = subparsers.add_parser(
         "unprotect",
         parents=[parent_parser],
-        description=UNPROTECT_HELP,
+        description=append_doc_link(UNPROTECT_HELP, "unprotect"),
         help=UNPROTECT_HELP,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
     )
     unprotect_parser.add_argument(
         "targets", nargs="+", help="Data files/directory."

dvc/daemon.py

@@ -83,7 +83,7 @@ def daemon(args):
         os.path.dirname(os.path.dirname(file_path))
     )
 
-    logger.debug("Trying to spawn '{}' with env '{}'".format(cmd, env))
+    logger.debug("Trying to spawn '{}'".format(cmd))
 
     if os.name == "nt":
         _spawn_windows(cmd, env)

dvc/exceptions.py

@@ -227,3 +227,31 @@ class RecursiveAddingWhileUsingFilename(DvcException):
         super(RecursiveAddingWhileUsingFilename, self).__init__(
             "using fname with recursive is not allowed."
         )
+
+
+class OverlappingOutputPathsError(DvcException):
+    def __init__(self, out_1, out_2):
+        super(OverlappingOutputPathsError, self).__init__(
+            "Paths for outs:\n'{}'('{}')\n'{}'('{}')\noverlap. To avoid "
+            "unpredictable behaviour, rerun command with non overlapping outs "
+            "paths.".format(
+                str(out_1),
+                out_1.stage.relpath,
+                str(out_2),
+                out_2.stage.relpath,
+            )
+        )
+
+
+class TargetNotDirectoryError(DvcException):
+    def __init__(self, path):
+        super(TargetNotDirectoryError, self).__init__(
+            "Target: {} is not a directory".format(path)
+        )
+
+
+class CheckoutErrorSuggestGit(DvcException):
+    def __init__(self, target, cause):
+        super(CheckoutErrorSuggestGit, self).__init__(
+            "Did you mean 'git checkout {}'?".format(target), cause=cause
+        )

dvc/logger.py

@@ -22,6 +22,10 @@ def info(message):
     logger.info(message)
 
 
+# Do not annotate debug with progress_aware. Why? Debug messages are called
+# in analytics.py. Trigger introduced by @progress_aware may run even
+# if log_level != DEBUG (look at Progress.print) Which might result in
+# messy terminal after any command (because analytics run in separate thread)
 def debug(message):
     """Prints a debug message."""
     prefix = colorize("Debug", color="blue")
@@ -203,16 +207,16 @@ def colorize(message, color=None):
     )
 
 
-def _init_colorama():
-    colorama.init()
-
-
 def set_default_level():
     """Sets default log level."""
     logger.setLevel(logging.INFO)
 
 
 def _add_handlers():
+    # NOTE: We need to initialize colorama before setting the stream handlers
+    #       so it can wrap stdout/stderr and convert color codes to Windows.
+    colorama.init()
+
     formatter = "%(message)s"
 
     class _LogLevelFilter(logging.Filter):
@@ -309,4 +313,3 @@ logger = logging.getLogger("dvc")  # pylint: disable=invalid-name
 
 set_default_level()
 _add_handlers()
-_init_colorama()

dvc/output/__init__.py

@@ -33,38 +33,69 @@ OUTS_MAP = {
     "local": OutputLOCAL,
 }
 
-SCHEMA = {
-    OutputBase.PARAM_PATH: str,
-    # NOTE: currently there are only 3 possible checksum names:
-    #
-    #    1) md5 (LOCAL, SSH, GS);
-    #    2) etag (S3);
-    #    3) checksum (HDFS);
-    #
-    # so when a few types of outputs share the same name, we only need
-    # specify it once.
+# NOTE: currently there are only 3 possible checksum names:
+#
+#    1) md5 (LOCAL, SSH, GS);
+#    2) etag (S3);
+#    3) checksum (HDFS);
+#
+# so when a few types of outputs share the same name, we only need
+# specify it once.
+CHECKSUM_SCHEMA = {
     schema.Optional(RemoteLOCAL.PARAM_CHECKSUM): schema.Or(str, None),
     schema.Optional(RemoteS3.PARAM_CHECKSUM): schema.Or(str, None),
     schema.Optional(RemoteHDFS.PARAM_CHECKSUM): schema.Or(str, None),
-    schema.Optional(OutputBase.PARAM_CACHE): bool,
-    schema.Optional(OutputBase.PARAM_METRIC): OutputBase.METRIC_SCHEMA,
 }
 
+TAGS_SCHEMA = {schema.Optional(str): CHECKSUM_SCHEMA}
 
-def _get(stage, p, info, cache, metric):
+SCHEMA = CHECKSUM_SCHEMA.copy()
+SCHEMA[OutputBase.PARAM_PATH] = str
+SCHEMA[schema.Optional(OutputBase.PARAM_CACHE)] = bool
+SCHEMA[schema.Optional(OutputBase.PARAM_METRIC)] = OutputBase.METRIC_SCHEMA
+SCHEMA[schema.Optional(OutputBase.PARAM_TAGS)] = TAGS_SCHEMA
+SCHEMA[schema.Optional(OutputBase.PARAM_PERSIST)] = bool
+
+
+def _get(stage, p, info, cache, metric, persist=False, tags=None):
     parsed = urlparse(p)
     if parsed.scheme == "remote":
         name = Config.SECTION_REMOTE_FMT.format(parsed.netloc)
         sect = stage.repo.config.config[name]
         remote = Remote(stage.repo, sect)
         return OUTS_MAP[remote.scheme](
-            stage, p, info, cache=cache, remote=remote, metric=metric
+            stage,
+            p,
+            info,
+            cache=cache,
+            remote=remote,
+            metric=metric,
+            persist=persist,
+            tags=tags,
         )
 
     for o in OUTS:
         if o.supported(p):
-            return o(stage, p, info, cache=cache, remote=None, metric=metric)
-    return OutputLOCAL(stage, p, info, cache=cache, remote=None, metric=metric)
+            return o(
+                stage,
+                p,
+                info,
+                cache=cache,
+                remote=None,
+                metric=metric,
+                persist=persist,
+                tags=tags,
+            )
+    return OutputLOCAL(
+        stage,
+        p,
+        info,
+        cache=cache,
+        remote=None,
+        metric=metric,
+        persist=persist,
+        tags=tags,
+    )
 
 
 def loadd_from(stage, d_list):
@@ -73,12 +104,33 @@ def loadd_from(stage, d_list):
         p = d.pop(OutputBase.PARAM_PATH)
         cache = d.pop(OutputBase.PARAM_CACHE, True)
         metric = d.pop(OutputBase.PARAM_METRIC, False)
-        ret.append(_get(stage, p, info=d, cache=cache, metric=metric))
+        persist = d.pop(OutputBase.PARAM_PERSIST, False)
+        tags = d.pop(OutputBase.PARAM_TAGS, None)
+        ret.append(
+            _get(
+                stage,
+                p,
+                info=d,
+                cache=cache,
+                metric=metric,
+                persist=persist,
+                tags=tags,
+            )
+        )
     return ret
 
 
-def loads_from(stage, s_list, use_cache=True, metric=False):
+def loads_from(stage, s_list, use_cache=True, metric=False, persist=False):
     ret = []
     for s in s_list:
-        ret.append(_get(stage, s, info={}, cache=use_cache, metric=metric))
+        ret.append(
+            _get(
+                stage,
+                s,
+                info={},
+                cache=use_cache,
+                metric=metric,
+                persist=persist,
+            )
+        )
     return ret

dvc/output/base.py

@@ -36,6 +36,7 @@ class OutputBase(object):
     PARAM_METRIC = "metric"
     PARAM_METRIC_TYPE = "type"
     PARAM_METRIC_XPATH = "xpath"
+    PARAM_PERSIST = "persist"
 
     METRIC_SCHEMA = Or(
         None,
@@ -46,11 +47,21 @@ class OutputBase(object):
         },
     )
 
+    PARAM_TAGS = "tags"
+
     DoesNotExistError = OutputDoesNotExistError
     IsNotFileOrDirError = OutputIsNotFileOrDirError
 
     def __init__(
-        self, stage, path, info=None, remote=None, cache=True, metric=False
+        self,
+        stage,
+        path,
+        info=None,
+        remote=None,
+        cache=True,
+        metric=False,
+        persist=False,
+        tags=None,
     ):
         self.stage = stage
         self.repo = stage.repo
@@ -59,6 +70,8 @@ class OutputBase(object):
         self.remote = remote or self.REMOTE(self.repo, {})
         self.use_cache = False if self.IS_DEPENDENCY else cache
         self.metric = False if self.IS_DEPENDENCY else metric
+        self.persist = persist
+        self.tags = None if self.IS_DEPENDENCY else (tags or {})
 
         if (
             self.use_cache
@@ -84,6 +97,11 @@ class OutputBase(object):
             "is local is not supported for {}".format(self.scheme)
         )
 
+    def assign_to_stage_file(self, target_repo):
+        raise DvcException(
+            "change repo is not supported for {}".format(self.scheme)
+        )
+
     @classmethod
     def match(cls, url):
         return re.match(cls.REMOTE.REGEX, url)
@@ -181,6 +199,10 @@ class OutputBase(object):
                 del self.metric[self.PARAM_METRIC_XPATH]
 
         ret[self.PARAM_METRIC] = self.metric
+        ret[self.PARAM_PERSIST] = self.persist
+
+        if self.tags:
+            ret[self.PARAM_TAGS] = self.tags
 
         return ret
 
@@ -192,12 +214,20 @@ class OutputBase(object):
     def download(self, to_info, resume=False):
         self.remote.download([self.path_info], [to_info], resume=resume)
 
-    def checkout(self, force=False):
+    def checkout(self, force=False, progress_callback=None, tag=None):
         if not self.use_cache:
             return
 
+        if tag:
+            info = self.tags[tag]
+        else:
+            info = self.info
+
         getattr(self.repo.cache, self.scheme).checkout(
-            self.path_info, self.info, force=force
+            self.path_info,
+            info,
+            force=force,
+            progress_callback=progress_callback,
         )
 
     def remove(self, ignore_remove=False):
@@ -220,3 +250,13 @@ class OutputBase(object):
 
         if self.scheme == "local" and self.use_cache and self.is_local:
             self.repo.scm.ignore(self.path)
+
+    def get_files_number(self):
+        if self.use_cache and self.checksum:
+            return 1
+
+        return 0
+
+    def unprotect(self):
+        if self.exists:
+            self.remote.unprotect(self.path_info)

dvc/output/hdfs.py

@@ -11,10 +11,25 @@ class OutputHDFS(OutputBase):
     REMOTE = RemoteHDFS
 
     def __init__(
-        self, stage, path, info=None, remote=None, cache=True, metric=False
+        self,
+        stage,
+        path,
+        info=None,
+        remote=None,
+        cache=True,
+        metric=False,
+        persist=False,
+        tags=None,
     ):
         super(OutputHDFS, self).__init__(
-            stage, path, info=info, remote=remote, cache=cache, metric=metric
+            stage,
+            path,
+            info=info,
+            remote=remote,
+            cache=cache,
+            metric=metric,
+            persist=persist,
+            tags=tags,
         )
         if remote:
             path = posixpath.join(remote.url, urlparse(path).path.lstrip("/"))

dvc/output/local.py

@@ -14,10 +14,25 @@ class OutputLOCAL(OutputBase):
     REMOTE = RemoteLOCAL
 
     def __init__(
-        self, stage, path, info=None, remote=None, cache=True, metric=False
+        self,
+        stage,
+        path,
+        info=None,
+        remote=None,
+        cache=True,
+        metric=False,
+        persist=False,
+        tags=None,
     ):
         super(OutputLOCAL, self).__init__(
-            stage, path, info, remote=remote, cache=cache, metric=metric
+            stage,
+            path,
+            info,
+            remote=remote,
+            cache=cache,
+            metric=metric,
+            persist=persist,
+            tags=tags,
         )
         if remote:
             p = os.path.join(
@@ -33,6 +48,8 @@ class OutputLOCAL(OutputBase):
 
         self.path_info = {"scheme": "local", "path": p}
 
+        self._dir_cache = {}
+
     def __str__(self):
         return self.rel_path
 
@@ -42,6 +59,17 @@ class OutputLOCAL(OutputBase):
             self.url
         )
 
+    def assign_to_stage_file(self, stage):
+        from dvc.repo import Repo
+
+        fullpath = os.path.abspath(stage.wdir)
+        self.path_info["path"] = os.path.join(fullpath, self.stage_path)
+
+        self.repo = Repo(self.path)
+
+        self.stage = stage
+        return self
+
     @property
     def sep(self):
         return os.sep
@@ -50,10 +78,18 @@ class OutputLOCAL(OutputBase):
     def rel_path(self):
         return os.path.relpath(self.path)
 
+    @property
+    def stage_path(self):
+        return os.path.relpath(self.path, self.stage.wdir)
+
     @property
     def cache(self):
         return self.repo.cache.local.get(self.checksum)
 
+    @property
+    def is_dir_cache(self):
+        return self.repo.cache.local.is_dir_cache(self.checksum)
+
     def dumpd(self):
         ret = super(OutputLOCAL, self).dumpd()
         if self.is_local:
@@ -118,3 +154,15 @@ class OutputLOCAL(OutputBase):
                 self.repo.scm.ignore(self.path)
 
         self.info = self.remote.save_info(self.path_info)
+
+    @property
+    def dir_cache(self):
+        return self.repo.cache.local.load_dir_cache(self.checksum)
+
+    def get_files_number(self):
+        if self.cache is None:
+            return 0
+
+        if self.is_dir_cache:
+            return len(self.dir_cache)
+        return 1

dvc/output/s3.py

@@ -11,10 +11,25 @@ class OutputS3(OutputBase):
     REMOTE = RemoteS3
 
     def __init__(
-        self, stage, path, info=None, remote=None, cache=True, metric=False
+        self,
+        stage,
+        path,
+        info=None,
+        remote=None,
+        cache=True,
+        metric=False,
+        persist=False,
+        tags=None,
     ):
         super(OutputS3, self).__init__(
-            stage, path, info=info, remote=remote, cache=cache, metric=metric
+            stage,
+            path,
+            info=info,
+            remote=remote,
+            cache=cache,
+            metric=metric,
+            persist=persist,
+            tags=tags,
         )
         bucket = remote.bucket if remote else urlparse(path).netloc
         path = urlparse(path).path.lstrip("/")

dvc/output/ssh.py

@@ -12,10 +12,25 @@ class OutputSSH(OutputBase):
     REMOTE = RemoteSSH
 
     def __init__(
-        self, stage, path, info=None, remote=None, cache=True, metric=False
+        self,
+        stage,
+        path,
+        info=None,
+        remote=None,
+        cache=True,
+        metric=False,
+        persist=False,
+        tags=None,
     ):
         super(OutputSSH, self).__init__(
-            stage, path, info=info, remote=remote, cache=cache, metric=metric
+            stage,
+            path,
+            info=info,
+            remote=remote,
+            cache=cache,
+            metric=metric,
+            persist=persist,
+            tags=tags,
         )
         parsed = urlparse(path)
         host = remote.host if remote else parsed.hostname

dvc/progress.py

@@ -8,6 +8,8 @@ from dvc.utils.compat import str
 import sys
 import threading
 
+CLEARLINE_PATTERN = "\r\x1b[K"
+
 
 class Progress(object):
     """
@@ -30,14 +32,20 @@ class Progress(object):
         """Returns if all targets have finished."""
         return self._n_total == self._n_finished
 
-    def _clearln(self):
-        self._print("\r\x1b[K", end="")
+    def clearln(self):
+        self.print(CLEARLINE_PATTERN, end="")
 
     def _writeln(self, line):
-        self._clearln()
-        self._print(line, end="")
+        self.clearln()
+        self.print(line, end="")
         sys.stdout.flush()
 
+    def reset(self):
+        with self._lock:
+            self._n_total = 0
+            self._n_finished = 0
+            self._line = None
+
     def refresh(self, line=None):
         """Refreshes progress bar."""
         # Just go away if it is locked. Will update next time
@@ -64,9 +72,9 @@ class Progress(object):
             pbar = self._bar(name, 100, 100)
 
             if sys.stdout.isatty():
-                self._clearln()
+                self.clearln()
 
-            self._print(pbar)
+            self.print(pbar)
 
             self._n_finished += 1
             self._line = None
@@ -98,7 +106,7 @@ class Progress(object):
         return num + pbar + percent + target_name
 
     @staticmethod
-    def _print(*args, **kwargs):
+    def print(*args, **kwargs):
         import dvc.logger as logger
 
         if logger.is_quiet():
@@ -109,7 +117,7 @@ class Progress(object):
     def __enter__(self):
         self._lock.acquire(True)
         if self._line is not None:
-            self._clearln()
+            self.clearln()
 
     def __exit__(self, typ, value, tbck):
         if self._line is not None:
@@ -124,11 +132,26 @@ def progress_aware(f):
     @wraps(f)
     def wrapper(*args, **kwargs):
         if not progress.is_finished:
-            progress._print()
+            progress.print()
+        progress.clearln()
 
         return f(*args, **kwargs)
 
     return wrapper
 
 
+class ProgressCallback(object):
+    def __init__(self, total):
+        self.total = total
+        self.current = 0
+        progress.reset()
+
+    def update(self, name, progress_to_add=1):
+        self.current += progress_to_add
+        progress.update_target(name, self.current, self.total)
+
+    def finish(self, name):
+        progress.finish_target(name)
+
+
 progress = Progress()  # pylint: disable=invalid-name

dvc/remote/azure.py

@@ -6,6 +6,7 @@ import re
 
 try:
     from azure.storage.blob import BlockBlobService
+    from azure.common import AzureMissingResourceHttpError
 except ImportError:
     BlockBlobService = None
 
@@ -77,7 +78,12 @@ class RemoteAzure(RemoteBase):
                 connection_string=self.connection_string
             )
             logger.debug("Container name {}".format(self.bucket))
-            self.__blob_service.create_container(self.bucket)
+            try:  # verify that container exists
+                self.__blob_service.list_blobs(
+                    self.bucket, delimiter="/", num_results=1
+                )
+            except AzureMissingResourceHttpError:
+                self.__blob_service.create_container(self.bucket)
         return self.__blob_service
 
     def remove(self, path_info):

dvc/remote/base.py

@@ -322,7 +322,9 @@ class RemoteBase(object):
 
         self.remove(path_info)
 
-    def do_checkout(self, path_info, checksum, force=False):
+    def do_checkout(
+        self, path_info, checksum, force=False, progress_callback=None
+    ):
         if self.exists(path_info):
             msg = "data '{}' exists. Removing before checkout."
             logger.warning(msg.format(str(path_info)))
@@ -331,7 +333,9 @@ class RemoteBase(object):
         from_info = self.checksum_to_path_info(checksum)
         self.copy(from_info, path_info)
 
-    def checkout(self, path_info, checksum_info, force=False):
+    def checkout(
+        self, path_info, checksum_info, force=False, progress_callback=None
+    ):
         scheme = path_info["scheme"]
         if scheme not in ["", "local"] and scheme != self.scheme:
             raise NotImplementedError
@@ -339,12 +343,12 @@ class RemoteBase(object):
         checksum = checksum_info.get(self.PARAM_CHECKSUM)
         if not checksum:
             msg = "No checksum info for '{}'."
-            logger.info(msg.format(str(path_info)))
+            logger.debug(msg.format(str(path_info)))
             return
 
         if not self.changed(path_info, checksum_info):
             msg = "Data '{}' didn't change."
-            logger.info(msg.format(str(path_info)))
+            logger.debug(msg.format(str(path_info)))
             return
 
         if self.changed_cache(checksum):
@@ -354,6 +358,15 @@ class RemoteBase(object):
             return
 
         msg = "Checking out '{}' with cache '{}'."
-        logger.info(msg.format(str(path_info), checksum))
+        logger.debug(msg.format(str(path_info), checksum))
 
-        self.do_checkout(path_info, checksum, force=force)
+        self.do_checkout(
+            path_info,
+            checksum,
+            force=force,
+            progress_callback=progress_callback,
+        )
+
+    @staticmethod
+    def unprotect(path_info):
+        pass

dvc/remote/hdfs.py

@@ -77,7 +77,7 @@ class RemoteHDFS(RemoteBase):
 
     def rm(self, path_info):
         self.hadoop_fs(
-            "rm {}".format(path_info["path"]), user=path_info["user"]
+            "rm -f {}".format(path_info["path"]), user=path_info["user"]
         )
 
     def save_info(self, path_info):

dvc/remote/local.py

@@ -20,7 +20,15 @@ from dvc.remote.base import (
     STATUS_DELETED,
     STATUS_MISSING,
 )
-from dvc.utils import remove, move, copyfile, dict_md5, to_chunks, tmp_fname
+from dvc.utils import (
+    remove,
+    move,
+    copyfile,
+    dict_md5,
+    to_chunks,
+    tmp_fname,
+    walk_files,
+)
 from dvc.utils import LARGE_DIR_SIZE
 from dvc.config import Config
 from dvc.exceptions import DvcException
@@ -70,6 +78,8 @@ class RemoteLOCAL(RemoteBase):
 
         self.path_info = {"scheme": "local"}
 
+        self._dir_info = {}
+
     @staticmethod
     def compat_config(config):
         ret = config.copy()
@@ -248,9 +258,12 @@ class RemoteLOCAL(RemoteBase):
 
     def load_dir_cache(self, md5):
         path = self.get(md5)
-
         assert self.is_dir_cache(path)
 
+        dir_info = self._dir_info.get(md5)
+        if dir_info:
+            return dir_info
+
         try:
             with open(path, "r") as fd:
                 d = json.load(fd)
@@ -267,6 +280,8 @@ class RemoteLOCAL(RemoteBase):
         for info in d:
             info["relpath"] = self.to_ospath(info["relpath"])
 
+        self._dir_info[md5] = d
+
         return d
 
     def dump_dir_cache(self, md5, dir_info):
@@ -291,10 +306,11 @@ class RemoteLOCAL(RemoteBase):
     def is_dir_cache(cls, cache):
         return cache.endswith(cls.MD5_DIR_SUFFIX)
 
-    def do_checkout(self, path_info, checksum, force=False):
+    def do_checkout(
+        self, path_info, checksum, force=False, progress_callback=None
+    ):
         path = path_info["path"]
-        md5 = checksum
-        cache = self.get(md5)
+        cache = self.get(checksum)
 
         if not self.is_dir_cache(cache):
             if os.path.exists(path):
@@ -302,6 +318,8 @@ class RemoteLOCAL(RemoteBase):
 
             self.link(cache, path)
             self.state.update_link(path)
+            if progress_callback:
+                progress_callback.update(os.path.relpath(path))
             return
 
         # Create dir separately so that dir is created
@@ -309,12 +327,11 @@ class RemoteLOCAL(RemoteBase):
         if not os.path.exists(path):
             os.makedirs(path)
 
-        dir_info = self.load_dir_cache(md5)
         dir_relpath = os.path.relpath(path)
-        dir_size = len(dir_info)
-        bar = dir_size > LARGE_DIR_SIZE
 
-        logger.info("Linking directory '{}'.".format(dir_relpath))
+        logger.debug("Linking directory '{}'.".format(dir_relpath))
+
+        dir_info = self.load_dir_cache(checksum)
 
         for processed, entry in enumerate(dir_info):
             relpath = entry[self.PARAM_RELPATH]
@@ -332,16 +349,13 @@ class RemoteLOCAL(RemoteBase):
 
                 self.link(c, p)
 
-            if bar:
-                progress.update_target(dir_relpath, processed, dir_size)
+            if progress_callback:
+                progress_callback.update(os.path.relpath(p))
 
         self._discard_working_directory_changes(path, dir_info, force=force)
 
         self.state.update_link(path)
 
-        if bar:
-            progress.finish_target(dir_relpath)
-
     def already_cached(self, path_info):
         assert path_info["scheme"] in ["", "local"]
 
@@ -353,11 +367,7 @@ class RemoteLOCAL(RemoteBase):
         return not self.changed_cache(current_md5)
 
     def _discard_working_directory_changes(self, path, dir_info, force=False):
-        working_dir_files = set(
-            os.path.join(root, file)
-            for root, _, files in os.walk(str(path))
-            for file in files
-        )
+        working_dir_files = set(path for path in walk_files(path))
 
         cached_files = set(
             os.path.join(path, file["relpath"]) for file in dir_info
@@ -739,3 +749,48 @@ class RemoteLOCAL(RemoteBase):
                 "nor on remote. Missing cache files: {}".format(missing_desc)
             )
             logger.warning(msg)
+
+    @staticmethod
+    def _unprotect_file(path):
+        if System.is_symlink(path) or System.is_hardlink(path):
+            logger.debug("Unprotecting '{}'".format(path))
+            tmp = os.path.join(os.path.dirname(path), "." + str(uuid.uuid4()))
+
+            # The operations order is important here - if some application
+            # would access the file during the process of copyfile then it
+            # would get only the part of file. So, at first, the file should be
+            # copied with the temporary name, and then original file should be
+            # replaced by new.
+            copyfile(
+                path,
+                tmp,
+                name="Unprotecting '{}'".format(os.path.relpath(path)),
+            )
+            remove(path)
+            os.rename(tmp, path)
+
+        else:
+            logger.debug(
+                "Skipping copying for '{}', since it is not "
+                "a symlink or a hardlink.".format(path)
+            )
+
+        os.chmod(path, os.stat(path).st_mode | stat.S_IWRITE)
+
+    @staticmethod
+    def _unprotect_dir(path):
+        for path in walk_files(path):
+            RemoteLOCAL._unprotect_file(path)
+
+    @staticmethod
+    def unprotect(path_info):
+        path = path_info["path"]
+        if not os.path.exists(path):
+            raise DvcException(
+                "can't unprotect non-existing data '{}'".format(path)
+            )
+
+        if os.path.isdir(path):
+            RemoteLOCAL._unprotect_dir(path)
+        else:
+            RemoteLOCAL._unprotect_file(path)

dvc/remote/ssh/connection.py

@@ -1,3 +1,4 @@
+import errno
 import os
 import posixpath
 
@@ -127,7 +128,17 @@ class SSHConnection:
 
         self._sftp_connect()
 
-        for entry in self._sftp.listdir_attr(directory):
+        try:
+            dir_entries = self._sftp.listdir_attr(directory)
+        except IOError as exc:
+            raise DvcException(
+                "couldn't get the '{}' remote directory files list".format(
+                    directory
+                ),
+                cause=exc,
+            )
+
+        for entry in dir_entries:
             path = posixpath.join(directory, entry.filename)
 
             if S_ISLNK(entry.st_mode):
@@ -143,7 +154,11 @@ class SSHConnection:
 
     def remove(self, path):
         self._sftp_connect()
-        self._sftp.remove(path)
+        try:
+            self._sftp.remove(path)
+        except IOError as exc:
+            if exc.errno != errno.ENOENT:
+                raise
 
     def download(self, src, dest, no_progress_bar=False, progress_title=None):
         self._sftp_connect()

dvc/repo/__init__.py

@@ -1,12 +1,16 @@
 from __future__ import unicode_literals
 
-from dvc.utils.compat import str
-
 import os
+
 import dvc.prompt as prompt
 import dvc.logger as logger
 
-from dvc.exceptions import DvcException, NotDvcRepoError, OutputNotFoundError
+from dvc.exceptions import (
+    DvcException,
+    NotDvcRepoError,
+    OutputNotFoundError,
+    TargetNotDirectoryError,
+)
 
 
 class Repo(object):
@@ -28,6 +32,8 @@ class Repo(object):
     from dvc.repo.status import status
     from dvc.repo.gc import gc
     from dvc.repo.commit import commit
+    from dvc.repo.pkg import install_pkg
+    from dvc.repo.brancher import brancher
 
     def __init__(self, root_dir=None):
         from dvc.config import Config
@@ -38,6 +44,8 @@ class Repo(object):
         from dvc.data_cloud import DataCloud
         from dvc.updater import Updater
         from dvc.repo.metrics import Metrics
+        from dvc.scm.tree import WorkingTree
+        from dvc.repo.tag import Tag
 
         root_dir = self.find_root(root_dir)
 
@@ -46,6 +54,8 @@ class Repo(object):
 
         self.config = Config(self.dvc_dir)
 
+        self.tree = WorkingTree()
+
         self.scm = SCM(self.root_dir, repo=self)
         self.lock = Lock(self.dvc_dir)
         # NOTE: storing state and link_state in the repository itself to avoid
@@ -61,6 +71,7 @@ class Repo(object):
         self.updater = Updater(self.dvc_dir)
 
         self.metrics = Metrics(self)
+        self.tag = Tag(self)
 
         self._ignore()
 
@@ -97,11 +108,9 @@ class Repo(object):
         init(root_dir=root_dir, no_scm=no_scm, force=force)
         return Repo(root_dir)
 
-    @staticmethod
-    def unprotect(target):
-        from dvc.repo.unprotect import unprotect
-
-        return unprotect(target)
+    def unprotect(self, target):
+        path_info = {"schema": "local", "path": target}
+        return self.cache.local.unprotect(path_info)
 
     def _ignore(self):
         flist = [
@@ -132,25 +141,27 @@ class Repo(object):
             raise CyclicGraphError(cycles[0])
 
     def _get_pipeline(self, node):
-        pipelines = list(filter(lambda g: node in g.nodes(), self.pipelines()))
+        pipelines = [i for i in self.pipelines() if i.has_node(node)]
         assert len(pipelines) == 1
         return pipelines[0]
 
-    def collect(self, target, with_deps=False):
+    def collect(self, target, with_deps=False, recursive=False):
         import networkx as nx
         from dvc.stage import Stage
 
+        if not target or recursive:
+            return self.active_stages(target)
+
         stage = Stage.load(self, target)
         if not with_deps:
             return [stage]
 
         node = os.path.relpath(stage.path, self.root_dir)
         G = self._get_pipeline(node)
-        stages = nx.get_node_attributes(G, "stage")
 
-        ret = [stage]
+        ret = []
         for n in nx.dfs_postorder_nodes(G, node):
-            ret.append(stages[n])
+            ret.append(G.node[n]["stage"])
 
         return ret
 
@@ -243,7 +254,7 @@ class Repo(object):
         cache["ssh"] = []
         cache["azure"] = []
 
-        for branch in self.scm.brancher(
+        for branch in self.brancher(
             all_branches=all_branches, all_tags=all_tags
         ):
             if target:
@@ -282,6 +293,7 @@ class Repo(object):
         from dvc.exceptions import (
             OutputDuplicationError,
             StagePathAsOutputError,
+            OverlappingOutputPathsError,
         )
 
         G = nx.DiGraph()
@@ -292,7 +304,15 @@ class Repo(object):
 
         for stage in stages:
             for out in stage.outs:
-                existing = [o.stage for o in outs if o.path == out.path]
+                existing = []
+                for o in outs:
+                    if o.path == out.path:
+                        existing.append(o.stage)
+
+                    in_o_dir = out.path.startswith(o.path + o.sep)
+                    in_out_dir = o.path.startswith(out.path + out.sep)
+                    if in_o_dir or in_out_dir:
+                        raise OverlappingOutputPathsError(o, out)
 
                 if existing:
                     stages = [stage.relpath, existing[0].relpath]
@@ -357,13 +377,15 @@ class Repo(object):
 
         if not from_directory:
             from_directory = self.root_dir
+        elif not os.path.isdir(from_directory):
+            raise TargetNotDirectoryError(from_directory)
 
         stages = []
         outs = []
-        for root, dirs, files in os.walk(str(from_directory)):
+        for root, dirs, files in self.tree.walk(from_directory):
             for fname in files:
                 path = os.path.join(root, fname)
-                if not Stage.is_stage_file(path):
+                if not Stage.is_valid_filename(path):
                     continue
                 stage = Stage.load(self, path)
                 for out in stage.outs:
@@ -396,10 +418,14 @@ class Repo(object):
 
     def find_outs_by_path(self, path, outs=None, recursive=False):
         if not outs:
-            outs = [out for stage in self.stages() for out in stage.outs]
+            # there is no `from_directory=path` argument because some data
+            # files might be generated to an upper level, and so it is
+            # needed to look at all the files (project root_dir)
+            stages = self.stages()
+            outs = [out for stage in stages for out in stage.outs]
 
         abs_path = os.path.abspath(path)
-        is_dir = os.path.isdir(abs_path)
+        is_dir = self.tree.isdir(abs_path)
 
         def func(out):
             if out.path == abs_path:

dvc/repo/add.py

@@ -1,6 +1,6 @@
 import os
 
-from dvc.scm import scm_context
+from dvc.repo.scm_context import scm_context
 from dvc.stage import Stage
 from dvc.utils import walk_files
 from dvc.exceptions import RecursiveAddingWhileUsingFilename

dvc/repo/brancher.py

@@ -0,0 +1,56 @@
+def brancher(  # noqa: E302
+    self, branches=None, all_branches=False, tags=None, all_tags=False
+):
+    """Generator that iterates over specified revisions.
+
+    Args:
+        branches (list): a list of branches to iterate over.
+        all_branches (bool): iterate over all available branches.
+        tags (list): a list of tags to iterate over.
+        all_tags (bool): iterate over all available tags.
+
+    Yields:
+        str: the display name for the currently selected tree, it could be:
+            - a git revision identifier
+            - empty string it there is no branches to iterate over
+            - "Working Tree" if there are uncommited changes in the SCM repo
+    """
+    if not any([branches, all_branches, tags, all_tags]):
+        yield ""
+        return
+
+    saved_tree = self.tree
+    revs = []
+
+    scm = self.scm
+
+    if self.scm.is_dirty():
+        from dvc.scm.tree import WorkingTree
+
+        self.tree = WorkingTree()
+        yield "Working Tree"
+
+    if all_branches:
+        branches = scm.list_branches()
+
+    if all_tags:
+        tags = scm.list_tags()
+
+    if branches is None:
+        revs.extend([scm.active_branch()])
+    else:
+        revs.extend(branches)
+
+    if tags is not None:
+        revs.extend(tags)
+
+    # NOTE: it might be a good idea to wrap this loop in try/finally block
+    # to don't leave the tree on some unexpected branch after the
+    # `brancher()`, but this could cause problems on exception handling
+    # code which might expect the tree on which exception was raised to
+    # stay in place. This behavior is a subject to change.
+    for rev in revs:
+        self.tree = scm.get_tree(rev)
+        yield rev
+
+    self.tree = saved_tree

dvc/repo/checkout.py

@@ -1,7 +1,8 @@
 from __future__ import unicode_literals
 
 import dvc.logger as logger
-from dvc.exceptions import DvcException
+from dvc.exceptions import CheckoutErrorSuggestGit
+from dvc.progress import ProgressCallback
 
 
 def _cleanup_unused_links(self, all_stages):
@@ -12,23 +13,32 @@ def _cleanup_unused_links(self, all_stages):
     self.state.remove_unused_links(used)
 
 
+def get_all_files_numbers(stages):
+    return sum(stage.get_all_files_number() for stage in stages)
+
+
+def get_progress_callback(stages):
+    total_files_num = get_all_files_numbers(stages)
+    if total_files_num == 0:
+        return None
+    return ProgressCallback(total_files_num)
+
+
 def checkout(self, target=None, with_deps=False, force=False, recursive=False):
     from dvc.stage import StageFileDoesNotExistError, StageFileBadNameError
 
-    if target and not recursive:
-        all_stages = self.active_stages()
-        try:
-            stages = self.collect(target, with_deps=with_deps)
-        except (StageFileDoesNotExistError, StageFileBadNameError) as exc:
-            raise DvcException(
-                str(exc) + " Did you mean 'git checkout {}'?".format(target)
-            )
-    else:
-        all_stages = self.active_stages(target)
-        stages = all_stages
+    all_stages = self.stages()
+
+    try:
+        stages = self.collect(target, with_deps=with_deps, recursive=recursive)
+    except (StageFileDoesNotExistError, StageFileBadNameError) as exc:
+        if not target:
+            raise
+        raise CheckoutErrorSuggestGit(target, exc)
 
     with self.state:
         _cleanup_unused_links(self, all_stages)
+        progress_callback = get_progress_callback(stages)
 
         for stage in stages:
             if stage.locked:
@@ -37,4 +47,6 @@ def checkout(self, target=None, with_deps=False, force=False, recursive=False):
                     " not going to be checked out.".format(path=stage.relpath)
                 )
 
-            stage.checkout(force=force)
+            stage.checkout(force=force, progress_callback=progress_callback)
+        if progress_callback:
+            progress_callback.finish("Checkout finished!")

dvc/repo/commit.py

@@ -1,9 +1,5 @@
 def commit(self, target, with_deps=False, recursive=False, force=False):
-    if target and not recursive:
-        stages = self.collect(target, with_deps=with_deps)
-    else:
-        stages = self.active_stages(target)
-
+    stages = self.collect(target, with_deps=with_deps, recursive=recursive)
     with self.state:
         for stage in stages:
             stage.check_can_commit(force=force)

dvc/repo/imp.py

@@ -1,4 +1,4 @@
-from dvc.scm import scm_context
+from dvc.repo.scm_context import scm_context
 
 
 @scm_context

dvc/repo/init.py

@@ -4,7 +4,7 @@ import colorama
 
 import dvc.logger as logger
 from dvc.repo import Repo
-from dvc.scm import SCM, Base
+from dvc.scm import SCM, NoSCM
 from dvc.config import Config
 from dvc.exceptions import InitError
 
@@ -56,7 +56,7 @@ def init(root_dir=os.curdir, no_scm=False, force=False):
     root_dir = os.path.abspath(root_dir)
     dvc_dir = os.path.join(root_dir, Repo.DVC_DIR)
     scm = SCM(root_dir)
-    if type(scm) == Base and not no_scm:
+    if isinstance(scm, NoSCM) and not no_scm:
         raise InitError(
             "{repo} is not tracked by any supported scm tool (e.g. git). "
             "Use '--no-scm' if you don't want to use any scm.".format(

dvc/repo/metrics/show.py

@@ -7,7 +7,7 @@ from jsonpath_rw import parse
 
 import dvc.logger as logger
 from dvc.exceptions import OutputNotFoundError, BadMetricError, NoMetricsError
-from dvc.utils.compat import builtin_str, open
+from dvc.utils.compat import builtin_str, open, StringIO, csv_reader
 
 
 def _read_metric_json(fd, json_path):
@@ -66,33 +66,103 @@ def _read_typed_metric(typ, xpath, fd):
     return ret
 
 
-def _read_metric(path, typ=None, xpath=None, branch=None):
-    ret = None
-    if not os.path.exists(path):
-        return ret
+def _format_csv(content, delimiter):
+    """Format delimited text to have same column width.
 
+    Args:
+        content (str): The content of a metric.
+        delimiter (str): Value separator
+
+    Returns:
+        str: Formatted content.
+
+    Example:
+
+        >>> content = (
+            "value_mse,deviation_mse,data_set\n"
+            "0.421601,0.173461,train\n"
+            "0.67528,0.289545,testing\n"
+            "0.671502,0.297848,validation\n"
+        )
+        >>> _format_csv(content, ",")
+
+        "value_mse  deviation_mse   data_set\n"
+        "0.421601   0.173461        train\n"
+        "0.67528    0.289545        testing\n"
+        "0.671502   0.297848        validation\n"
+    """
+    reader = csv_reader(StringIO(content), delimiter=builtin_str(delimiter))
+    rows = [row for row in reader]
+    max_widths = [max(map(len, column)) for column in zip(*rows)]
+
+    lines = [
+        " ".join(
+            "{entry:{width}}".format(entry=entry, width=width + 2)
+            for entry, width in zip(row, max_widths)
+        )
+        for row in rows
+    ]
+
+    return "\n".join(lines)
+
+
+def _format_output(content, typ):
+    """Tabularize the content according to its type.
+
+    Args:
+        content (str): The content of a metric.
+        typ (str): The type of metric -- (raw|json|tsv|htsv|csv|hcsv).
+
+    Returns:
+        str: Content in a raw or tabular format.
+    """
+
+    if "csv" in str(typ):
+        return _format_csv(content, delimiter=",")
+
+    if "tsv" in str(typ):
+        return _format_csv(content, delimiter="\t")
+
+    return content
+
+
+def _read_metric(fd, typ=None, xpath=None, rel_path=None, branch=None):
     typ = typ.lower().strip() if typ else typ
-    xpath = xpath.strip() if xpath else xpath
     try:
-        with open(path, "r") as fd:
-            if not xpath:
-                ret = fd.read().strip()
-            else:
-                ret = _read_typed_metric(typ, xpath, fd)
+        if xpath:
+            return _read_typed_metric(typ, xpath.strip(), fd)
+        else:
+            return _format_output(fd.read().strip(), typ)
     # Json path library has to be replaced or wrapped in
     # order to fix this too broad except clause.
     except Exception:
         logger.warning(
             "unable to read metric in '{}' in branch '{}'".format(
-                path, branch
+                rel_path, branch
             ),
             parse_exception=True,
         )
-
-    return ret
+        return None
 
 
 def _collect_metrics(self, path, recursive, typ, xpath, branch):
+    """Gather all the metric outputs.
+
+    Args:
+        path (str): Path to a metric file or a directory.
+        recursive (bool): If path is a directory, do a recursive search for
+            metrics on the given path.
+        typ (str): The type of metric to search for, could be one of the
+            following (raw|json|tsv|htsv|csv|hcsv).
+        xpath (str): Path to search for.
+        branch (str): Branch to look up for metrics.
+
+    Returns:
+        list(tuple): (output, typ, xpath)
+            - output:
+            - typ:
+            - xpath:
+    """
     outs = [out for stage in self.stages() for out in stage.outs]
 
     if path:
@@ -123,16 +193,50 @@ def _collect_metrics(self, path, recursive, typ, xpath, branch):
     return res
 
 
+def _read_metrics_filesystem(path, typ, xpath, rel_path, branch):
+    if not os.path.exists(path):
+        return None
+    with open(path, "r") as fd:
+        return _read_metric(
+            fd, typ=typ, xpath=xpath, rel_path=rel_path, branch=branch
+        )
+
+
 def _read_metrics(self, metrics, branch):
+    """Read the content of each metric file and format it.
+
+    Args:
+        metrics (list): List of metric touples
+        branch (str): Branch to look up for metrics.
+
+    Returns:
+        A dict mapping keys with metrics path name and content.
+        For example:
+
+        {'metric.csv': ("value_mse  deviation_mse   data_set\n"
+                        "0.421601   0.173461        train\n"
+                        "0.67528    0.289545        testing\n"
+                        "0.671502   0.297848        validation\n")}
+    """
     res = {}
     for out, typ, xpath in metrics:
         assert out.scheme == "local"
+        if not typ:
+            typ = os.path.splitext(out.path.lower())[1].replace(".", "")
         if out.use_cache:
-            path = self.cache.local.get(out.checksum)
+            metric = _read_metrics_filesystem(
+                self.cache.local.get(out.checksum),
+                typ=typ,
+                xpath=xpath,
+                rel_path=out.rel_path,
+                branch=branch,
+            )
         else:
-            path = out.path
+            fd = self.tree.open(out.path)
+            metric = _read_metric(
+                fd, typ=typ, xpath=xpath, rel_path=out.rel_path, branch=branch
+            )
 
-        metric = _read_metric(path, typ=typ, xpath=xpath, branch=branch)
         if not metric:
             continue
 
@@ -151,9 +255,8 @@ def show(
     recursive=False,
 ):
     res = {}
-    for branch in self.scm.brancher(
-        all_branches=all_branches, all_tags=all_tags
-    ):
+
+    for branch in self.brancher(all_branches=all_branches, all_tags=all_tags):
         entries = _collect_metrics(self, path, recursive, typ, xpath, branch)
         metrics = _read_metrics(self, entries, branch)
         if metrics:

dvc/repo/move.py

@@ -1,7 +1,7 @@
 import os
 
 from dvc.exceptions import MoveNotDataSourceError
-from dvc.scm import scm_context
+from dvc.repo.scm_context import scm_context
 
 
 def _expand_target_path(from_path, to_path):

dvc/repo/pkg.py

@@ -0,0 +1,201 @@
+from __future__ import unicode_literals
+
+import os
+import shutil
+from git.cmd import Git
+
+import dvc.logger as logger
+from dvc.exceptions import DvcException
+from dvc.stage import Stage
+from dvc.temp_git_repo import TempGitRepo
+
+
+class PackageParams(object):
+    def __init__(self, address, target_dir, select=[], file=None):
+        self.address = address
+        self.target_dir = target_dir
+        self.select = select
+        self.file = file
+
+    @property
+    def all_addresses(self):
+        if self.address:
+            return [self.address]
+        return PackageManager.read_packages()
+
+
+class PackageManager(object):
+    PACKAGE_FILE = "package.yaml"
+
+    @staticmethod
+    def read_packages():
+        return []
+
+    @staticmethod
+    def get_package(addr):
+        for pkg_class in [GitPackage]:
+            try:
+                return pkg_class()
+            except Exception:
+                pass
+        return None
+
+    def __init__(self, addr):
+        self._addr = addr
+
+
+class Package(object):
+    MODULES_DIR = "dvc_mod"
+
+    def install_or_update(self, parent_repo, pkg_param):
+        raise NotImplementedError(
+            "A method of abstract Package class was called"
+        )
+
+    def is_in_root(self):
+        return True
+
+
+class GitPackage(Package):
+    DEF_DVC_FILE_PREFIX = "mod_"
+
+    def install_or_update(self, parent_repo, pkg_params):
+        if not self.is_in_root():
+            raise DvcException(
+                "This command can be run only from a repository root"
+            )
+
+        if not os.path.exists(self.MODULES_DIR):
+            logger.debug("Creating modules dir {}".format(self.MODULES_DIR))
+            os.makedirs(self.MODULES_DIR)
+            parent_repo.scm.ignore(os.path.abspath(self.MODULES_DIR))
+
+        module_name = (
+            Git.polish_url(pkg_params.address).strip("/").split("/")[-1]
+        )
+        if not module_name:
+            raise DvcException(
+                "Package address error: unable to extract package name"
+            )
+
+        with TempGitRepo(
+            pkg_params.address, module_name, Package.MODULES_DIR
+        ) as tmp_repo:
+            outputs_to_copy = tmp_repo.outs
+            if pkg_params.select:
+                outputs_to_copy = list(
+                    filter(
+                        lambda out: out.dvc_path in pkg_params.select,
+                        outputs_to_copy,
+                    )
+                )
+
+            fetched_stage_files = set(
+                map(lambda o: o.stage.path, outputs_to_copy)
+            )
+            tmp_repo.fetch(fetched_stage_files)
+
+            module_dir = self.create_module_dir(module_name)
+            tmp_repo.persist_to(module_dir, parent_repo)
+
+            dvc_file = self.get_dvc_file_name(
+                pkg_params.file, pkg_params.target_dir, module_name
+            )
+            try:
+                self.persist_stage_and_scm_state(
+                    parent_repo,
+                    outputs_to_copy,
+                    pkg_params.target_dir,
+                    dvc_file,
+                )
+            except Exception as ex:
+                raise DvcException(
+                    "Package '{}' was installed "
+                    "but stage file '{}' "
+                    "was not created properly: {}".format(
+                        pkg_params.address, dvc_file, ex
+                    )
+                )
+
+        parent_repo.checkout(dvc_file)
+
+    @staticmethod
+    def persist_stage_and_scm_state(
+        parent_repo, outputs_to_copy, target_dir, dvc_file
+    ):
+        stage = Stage.create(
+            repo=parent_repo,
+            fname=dvc_file,
+            validate_state=False,
+            wdir=target_dir,
+        )
+        stage.outs = list(
+            map(lambda o: o.assign_to_stage_file(stage), outputs_to_copy)
+        )
+
+        for out in stage.outs:
+            parent_repo.scm.ignore(out.path, in_curr_dir=True)
+
+        stage.dump()
+
+    @staticmethod
+    def create_module_dir(module_name):
+        module_dir = os.path.join(GitPackage.MODULES_DIR, module_name)
+        if os.path.exists(module_dir):
+            logger.info("Updating package {}".format(module_name))
+            shutil.rmtree(module_dir)
+        else:
+            logger.info("Adding package {}".format(module_name))
+        return module_dir
+
+    def get_dvc_file_name(self, stage_file, target_dir, module_name):
+        if stage_file:
+            dvc_file_path = stage_file
+        else:
+            dvc_file_name = self.DEF_DVC_FILE_PREFIX + module_name + ".dvc"
+            dvc_file_path = os.path.join(target_dir, dvc_file_name)
+        return dvc_file_path
+
+
+def install_pkg(self, pkg_params):
+    """
+    Install package.
+
+    The command can be run only from DVC project root.
+
+    E.g.
+          Having: DVC package in https://github.com/dmpetrov/tag_classifier
+
+          $ dvc pkg install https://github.com/dmpetrov/tag_classifier
+
+          Result: tag_classifier package in dvc_mod/ directory
+    """
+
+    if not os.path.isdir(pkg_params.target_dir):
+        logger.error(
+            "Unable to install package: "
+            "target directory '{}' does not exist".format(
+                pkg_params.target_dir
+            )
+        )
+        return 1
+
+    curr_dir = os.path.realpath(os.curdir)
+    if not os.path.realpath(pkg_params.target_dir).startswith(curr_dir):
+        logger.error(
+            "Unable to install package: the current dir should be"
+            " a subdirectory of the target dir {}".format(
+                pkg_params.target_dir
+            )
+        )
+        return 1
+
+    for addr in pkg_params.all_addresses:
+        try:
+            mgr = PackageManager.get_package(addr)
+            mgr.install_or_update(self, pkg_params)
+        except Exception as ex:
+            logger.error("Unable to install package: ".format(ex))
+            return 1
+
+    return 0

dvc/repo/remove.py

@@ -3,8 +3,8 @@ def remove(self, target, outs_only=False):
 
     stage = Stage.load(self, target)
     if outs_only:
-        stage.remove_outs()
+        stage.remove_outs(force=True)
     else:
-        stage.remove()
+        stage.remove(force=True)
 
     return stage

dvc/repo/reproduce.py

@@ -4,7 +4,7 @@ import os
 
 import dvc.logger as logger
 from dvc.exceptions import ReproductionError
-from dvc.scm import scm_context
+from dvc.repo.scm_context import scm_context
 
 
 def _reproduce_stage(stages, node, force, dry, interactive, no_commit):

dvc/repo/run.py

@@ -1,6 +1,6 @@
 from __future__ import unicode_literals
 
-from dvc.scm import scm_context
+from dvc.repo.scm_context import scm_context
 
 
 @scm_context
@@ -20,6 +20,8 @@ def run(
     ignore_build_cache=False,
     remove_outs=False,
     no_commit=False,
+    outs_persist=None,
+    outs_persist_no_cache=None,
 ):
     from dvc.stage import Stage
 
@@ -33,6 +35,10 @@ def run(
         metrics = []
     if metrics_no_cache is None:
         metrics_no_cache = []
+    if outs_persist is None:
+        outs_persist = []
+    if outs_persist_no_cache is None:
+        outs_persist_no_cache = []
 
     with self.state:
         stage = Stage.create(
@@ -49,6 +55,8 @@ def run(
             overwrite=overwrite,
             ignore_build_cache=ignore_build_cache,
             remove_outs=remove_outs,
+            outs_persist=outs_persist,
+            outs_persist_no_cache=outs_persist_no_cache,
         )
 
     if stage is None:

dvc/repo/scm_context.py

@@ -0,0 +1,12 @@
+def scm_context(method):
+    def run(repo, *args, **kw):
+        try:
+            result = method(repo, *args, **kw)
+            repo.scm.reset_ignores()
+            repo.scm.remind_to_track()
+            return result
+        except Exception as e:
+            repo.scm.cleanup_ignores()
+            raise e
+
+    return run

dvc/repo/status.py

@@ -6,10 +6,7 @@ import dvc.logger as logger
 def _local_status(self, target=None, with_deps=False):
     status = {}
 
-    if target:
-        stages = self.collect(target, with_deps=with_deps)
-    else:
-        stages = self.active_stages()
+    stages = self.collect(target, with_deps=with_deps)
 
     for stage in stages:
         if stage.locked:
@@ -54,7 +51,7 @@ def _cloud_status(
     for md5, info in status_info.items():
         name = info["name"]
         status = info["status"]
-        if status == cloud.STATUS_OK:
+        if status in [cloud.STATUS_OK, cloud.STATUS_MISSING]:
             continue
 
         prefix_map = {cloud.STATUS_DELETED: "deleted", cloud.STATUS_NEW: "new"}

dvc/repo/tag/__init__.py

@@ -0,0 +1,18 @@
+class Tag(object):
+    def __init__(self, repo):
+        self.repo = repo
+
+    def add(self, *args, **kwargs):
+        from dvc.repo.tag.add import add
+
+        return add(self.repo, *args, **kwargs)
+
+    def list(self, *args, **kwargs):
+        from dvc.repo.tag.list import list
+
+        return list(self.repo, *args, **kwargs)
+
+    def remove(self, *args, **kwargs):
+        from dvc.repo.tag.remove import remove
+
+        return remove(self.repo, *args, **kwargs)

dvc/repo/tag/add.py

@@ -0,0 +1,15 @@
+import dvc.logger as logger
+
+
+def add(self, tag, target=None, with_deps=False, recursive=False):
+    stages = self.collect(target, with_deps=with_deps, recursive=recursive)
+    for stage in stages:
+        changed = False
+        for out in stage.outs:
+            if not out.info:
+                logger.warning("missing checksum info for '{}'".format(out))
+                continue
+            out.tags[tag] = out.info.copy()
+            changed = True
+        if changed:
+            stage.dump()

dvc/repo/tag/list.py

@@ -0,0 +1,11 @@
+def list(self, target=None, with_deps=False, recursive=False):
+    ret = {}
+    stages = self.collect(target, with_deps=with_deps, recursive=recursive)
+    for stage in stages:
+        outs_tags = {}
+        for out in stage.outs:
+            if out.tags:
+                outs_tags[str(out)] = out.tags.copy()
+        if outs_tags:
+            ret.update({stage.relpath: outs_tags})
+    return ret

dvc/repo/tag/remove.py

@@ -0,0 +1,15 @@
+import dvc.logger as logger
+
+
+def remove(self, tag, target=None, with_deps=False, recursive=False):
+    stages = self.collect(target, with_deps=with_deps, recursive=recursive)
+    for stage in stages:
+        changed = False
+        for out in stage.outs:
+            if tag not in out.tags.keys():
+                logger.warning("tag '{}' not found for '{}'".format(tag, out))
+                continue
+            del out.tags[tag]
+            changed = True
+        if changed:
+            stage.dump()

dvc/repo/unprotect.py

@@ -1,48 +0,0 @@
-from __future__ import unicode_literals
-
-import os
-import stat
-import uuid
-
-import dvc.logger as logger
-from dvc.system import System
-from dvc.utils import copyfile, move, remove
-from dvc.exceptions import DvcException
-
-
-def _unprotect_file(path):
-    if System.is_symlink(path) or System.is_hardlink(path):
-        logger.debug("Unprotecting '{}'".format(path))
-
-        tmp = os.path.join(os.path.dirname(path), "." + str(uuid.uuid4()))
-        move(path, tmp)
-
-        copyfile(tmp, path)
-
-        remove(tmp)
-    else:
-        logger.debug(
-            "Skipping copying for '{}', since it is not "
-            "a symlink or a hardlink.".format(path)
-        )
-
-    os.chmod(path, os.stat(path).st_mode | stat.S_IWRITE)
-
-
-def _unprotect_dir(path):
-    for root, dirs, files in os.walk(str(path)):
-        for f in files:
-            path = os.path.join(root, f)
-            _unprotect_file(path)
-
-
-def unprotect(path):
-    if not os.path.exists(path):
-        raise DvcException(
-            "can't unprotect non-existing data '{}'".format(path)
-        )
-
-    if os.path.isdir(path):
-        _unprotect_dir(path)
-    else:
-        _unprotect_file(path)

dvc/scm/__init__.py

@@ -5,7 +5,11 @@ from __future__ import unicode_literals
 from dvc.scm.base import Base
 from dvc.scm.git import Git
 
-import dvc
+
+# just a sugar to point that this is an actual implementation for a dvc
+# project under no SCM control
+class NoSCM(Base):
+    pass
 
 
 def SCM(root_dir, repo=None):  # pylint: disable=invalid-name
@@ -22,20 +26,4 @@ def SCM(root_dir, repo=None):  # pylint: disable=invalid-name
     if Git.is_repo(root_dir) or Git.is_submodule(root_dir):
         return Git(root_dir, repo=repo)
 
-    return Base(root_dir, repo=repo)
-
-
-def scm_context(method):
-    def run(*args, **kw):
-        repo = args[0]
-        assert isinstance(repo, dvc.repo.Repo)
-        try:
-            result = method(*args, **kw)
-            repo.scm.reset_ignores()
-            repo.scm.remind_to_track()
-            return result
-        except Exception as e:
-            repo.scm.cleanup_ignores()
-            raise e
-
-    return run
+    return NoSCM(root_dir, repo=repo)

dvc/scm/base.py

@@ -17,12 +17,17 @@ class FileNotInRepoError(DvcException):
     """
 
 
+class FileNotInTargetSubdirError(DvcException):
+    """Thrown when trying to place .gitignore for a file that not in
+    the file subdirectory."""
+
+
 class Base(object):
     """Base class for source control management driver implementations."""
 
     def __init__(self, root_dir=os.curdir, repo=None):
         self.repo = repo
-        self.root_dir = root_dir
+        self.root_dir = os.path.realpath(root_dir)
 
     def __repr__(self):
         return "{class_name}: '{directory}'".format(
@@ -76,47 +81,6 @@ class Base(object):
     def tag(self, tag):
         """Makes SCM create a tag with a specified name."""
 
-    def brancher(
-        self, branches=None, all_branches=False, tags=None, all_tags=False
-    ):
-        """Generator that iterates over specified revisions.
-
-        Args:
-            branches (list): a list of branches to iterate over.
-            all_branches (bool): iterate over all available branches.
-            tags (list): a list of tags to iterate over.
-            all_tags (bool): iterate over all available tags.
-
-        Yields:
-            str: the current revision.
-        """
-        if not branches and not all_branches and not tags and not all_tags:
-            yield ""
-            return
-
-        saved = self.active_branch()
-        revs = []
-
-        if all_branches:
-            branches = self.list_branches()
-
-        if all_tags:
-            tags = self.list_tags()
-
-        if branches is None:
-            revs.extend([saved])
-        else:
-            revs.extend(branches)
-
-        if tags is not None:
-            revs.extend(tags)
-
-        for rev in revs:
-            self.checkout(rev)
-            yield rev
-
-        self.checkout(saved)
-
     def untracked_files(self):  # pylint: disable=no-self-use
         """Returns a list of untracked files."""
         return []
@@ -125,6 +89,10 @@ class Base(object):
         """Returns whether or not a specified path is tracked."""
         return False
 
+    def is_dirty(self):
+        """Return whether the SCM contains uncommited changes."""
+        return False
+
     def active_branch(self):  # pylint: disable=no-self-use
         """Returns current branch in the repo."""
         return ""

dvc/scm/git/__init__.py

@@ -2,17 +2,18 @@
 
 from __future__ import unicode_literals
 
-# NOTE: need this one because python 2 would re-import this module
-# (called git.py) instead of importing gitpython's git module.
-from __future__ import absolute_import
-
-from dvc.utils.compat import str, open
-
 import os
 
-import dvc.logger as logger
+from dvc.utils.compat import str, open
 from dvc.utils import fix_env
-from dvc.scm.base import Base, SCMError, FileNotInRepoError
+from dvc.scm.base import (
+    Base,
+    SCMError,
+    FileNotInRepoError,
+    FileNotInTargetSubdirError,
+)
+from dvc.scm.git.tree import GitTree
+import dvc.logger as logger
 
 
 class Git(Base):
@@ -62,28 +63,47 @@ class Git(Base):
     def ignore_file(self):
         return self.GITIGNORE
 
-    def _get_gitignore(self, path):
-        assert os.path.isabs(path)
-        # NOTE: using '/' prefix to make path unambiguous
-        entry = "/" + os.path.basename(path)
-        gitignore = os.path.join(os.path.dirname(path), self.GITIGNORE)
+    def _get_gitignore(self, path, ignore_file_dir=None):
+        if not ignore_file_dir:
+            ignore_file_dir = os.path.dirname(os.path.realpath(path))
 
-        if not gitignore.startswith(self.root_dir):
+        assert os.path.isabs(path)
+        assert os.path.isabs(ignore_file_dir)
+
+        if not path.startswith(ignore_file_dir):
+            msg = (
+                "{} file has to be located in one of '{}' subdirectories"
+                ", not outside '{}'"
+            )
+            raise FileNotInTargetSubdirError(
+                msg.format(self.GITIGNORE, path, ignore_file_dir)
+            )
+
+        entry = os.path.relpath(path, ignore_file_dir).replace(os.sep, "/")
+        # NOTE: using '/' prefix to make path unambiguous
+        if len(entry) > 0 and entry[0] != "/":
+            entry = "/" + entry
+
+        gitignore = os.path.join(ignore_file_dir, self.GITIGNORE)
+
+        if not gitignore.startswith(os.path.realpath(self.root_dir)):
             raise FileNotInRepoError(path)
 
         return entry, gitignore
 
-    def ignore(self, path):
-        entry, gitignore = self._get_gitignore(path)
+    def ignore(self, path, in_curr_dir=False):
+        base_dir = (
+            os.path.realpath(os.curdir)
+            if in_curr_dir
+            else os.path.dirname(path)
+        )
+        entry, gitignore = self._get_gitignore(path, base_dir)
 
         ignore_list = []
         if os.path.exists(gitignore):
             with open(gitignore, "r") as f:
                 ignore_list = f.readlines()
-            filtered = list(
-                filter(lambda x: x.strip() == entry.strip(), ignore_list)
-            )
-            if filtered:
+            if any(filter(lambda x: x.strip() == entry.strip(), ignore_list)):
                 return
 
         msg = "Adding '{}' to '{}'.".format(
@@ -91,17 +111,20 @@ class Git(Base):
         )
         logger.info(msg)
 
-        content = entry
-        if ignore_list:
-            content = "\n" + content
-
-        with open(gitignore, "a") as fobj:
-            fobj.write(content)
+        self._add_entry_to_gitignore(entry, gitignore, ignore_list)
 
         self.track_file(os.path.relpath(gitignore))
 
         self.ignored_paths.append(path)
 
+    @staticmethod
+    def _add_entry_to_gitignore(entry, gitignore, ignore_list):
+        content = entry
+        if ignore_list:
+            content = "\n" + content
+        with open(gitignore, "a", encoding="utf-8") as fobj:
+            fobj.write(content)
+
     def ignore_remove(self, path):
         entry, gitignore = self._get_gitignore(path)
 
@@ -153,7 +176,13 @@ class Git(Base):
         return [os.path.join(self.git.working_dir, fname) for fname in files]
 
     def is_tracked(self, path):
-        return len(self.git.git.ls_files(path)) != 0
+        # it is equivalent to `bool(self.git.git.ls_files(path))` by
+        # functionality, but ls_files fails on unicode filenames
+        path = os.path.relpath(path, self.root_dir)
+        return path in [i[0] for i in self.git.index.entries]
+
+    def is_dirty(self):
+        return self.git.is_dirty()
 
     def active_branch(self):
         return self.git.active_branch.name
@@ -203,3 +232,6 @@ class Git(Base):
         basename = os.path.basename(path)
         path_parts = os.path.normpath(path).split(os.path.sep)
         return basename == self.ignore_file or Git.GIT_DIR in path_parts
+
+    def get_tree(self, rev):
+        return GitTree(self.git, rev)

dvc/scm/git/tree.py

@@ -0,0 +1,114 @@
+import errno
+import os
+
+from dvc.utils.compat import StringIO
+
+from dvc.scm.tree import BaseTree
+
+
+# see git-fast-import(1)
+GIT_MODE_DIR = 0o40000
+GIT_MODE_FILE = 0o644
+
+
+class GitTree(BaseTree):
+    """Proxies the repo file access methods to Git objects"""
+
+    def __init__(self, git, rev):
+        """Create GitTree instance
+
+        Args:
+            git (dvc.scm.Git):
+            branch:
+        """
+        self.git = git
+        self.rev = rev
+
+    def open(self, path):
+
+        relpath = os.path.relpath(path, self.git.working_dir)
+
+        obj = self.git_object_by_path(path)
+        if obj is None:
+            msg = "No such file in branch '{}'".format(self.rev)
+            raise IOError(errno.ENOENT, msg, relpath)
+        if obj.mode == GIT_MODE_DIR:
+            raise IOError(errno.EISDIR, "Is a directory", relpath)
+
+        # GitPython's obj.data_stream is a fragile thing, it is better to
+        # read it immediately, also it needs to be to decoded if we follow
+        # the `open()` behavior (since data_stream.read() returns bytes,
+        # and `open` with default "r" mode returns str)
+        return StringIO(obj.data_stream.read().decode("utf-8"))
+
+    def exists(self, path):
+        return self.git_object_by_path(path) is not None
+
+    def isdir(self, path):
+        obj = self.git_object_by_path(path)
+        if obj is None:
+            return False
+        return obj.mode == GIT_MODE_DIR
+
+    def isfile(self, path):
+        obj = self.git_object_by_path(path)
+        if obj is None:
+            return False
+        # according to git-fast-import(1) file mode could be 644 or 755
+        return obj.mode & GIT_MODE_FILE == GIT_MODE_FILE
+
+    @staticmethod
+    def _is_tree_and_contains(obj, path):
+        if obj.mode != GIT_MODE_DIR:
+            return False
+        # see https://github.com/gitpython-developers/GitPython/issues/851
+        # `return (i in tree)` doesn't work so here is a workaround:
+        for i in obj:
+            if i.name == path:
+                return True
+        return False
+
+    def git_object_by_path(self, path):
+        path = os.path.relpath(os.path.realpath(path), self.git.working_dir)
+        assert path.split(os.sep, 1)[0] != ".."
+        tree = self.git.tree(self.rev)
+        if not path or path == ".":
+            return tree
+        for i in path.split(os.sep):
+            if not self._is_tree_and_contains(tree, i):
+                # there is no tree for specified path
+                return None
+            tree = tree[i]
+        return tree
+
+    def _walk(self, tree, topdown=True):
+
+        dirs, nondirs = [], []
+        for i in tree:
+            if i.mode == GIT_MODE_DIR:
+                dirs.append(i.name)
+            else:
+                nondirs.append(i.name)
+
+        if topdown:
+            yield os.path.normpath(tree.path), dirs, nondirs
+        for i in dirs:
+            for x in self._walk(tree[i], topdown=True):
+                yield x
+        if not topdown:
+            yield os.path.normpath(tree.path), dirs, nondirs
+
+    def walk(self, top, topdown=True):
+        """Directory tree generator.
+
+        See `os.walk` for the docs. Differences:
+        - no support for symlinks
+        - it could raise exceptions, there is no onerror argument
+        """
+
+        tree = self.git_object_by_path(top)
+        if tree is None:
+            raise IOError(errno.ENOENT, "No such file")
+
+        for x in self._walk(tree, topdown):
+            yield x

dvc/scm/tree.py

@@ -0,0 +1,65 @@
+import os
+
+from dvc.utils.compat import open
+
+
+class BaseTree(object):
+    """Abstract class to represent access to files"""
+
+    def open(self, path):
+        """Open file and return a stream."""
+
+    def exists(self, path):
+        """Test whether a path exists."""
+
+    def isdir(self, path):
+        """Return true if the pathname refers to an existing directory."""
+
+    def isfile(self, path):
+        """Test whether a path is a regular file"""
+
+    def walk(self, top, topdown=True):
+        """Directory tree generator.
+
+        See `os.walk` for the docs. Differences:
+        - no support for symlinks
+        - it could raise exceptions, there is no onerror argument
+        """
+
+
+class WorkingTree(BaseTree):
+    """Proxies the repo file access methods to working tree files"""
+
+    def open(self, path):
+        """Open file and return a stream."""
+        return open(path, encoding="utf-8")
+
+    def exists(self, path):
+        """Test whether a path exists."""
+        return os.path.exists(path)
+
+    def isdir(self, path):
+        """Return true if the pathname refers to an existing directory."""
+        return os.path.isdir(path)
+
+    def isfile(self, path):
+        """Test whether a path is a regular file"""
+        return os.path.isfile(path)
+
+    def walk(self, top, topdown=True):
+        """Directory tree generator.
+
+        See `os.walk` for the docs. Differences:
+        - no support for symlinks
+        - it could raise exceptions, there is no onerror argument
+        """
+
+        def onerror(e):
+            raise e
+
+        for root, dirs, files in os.walk(
+            top, topdown=topdown, onerror=onerror
+        ):
+            if topdown:
+                dirs[:] = [i for i in dirs if i not in (".git", ".hg", ".dvc")]
+            yield os.path.normpath(root), dirs, files

dvc/stage.py

@@ -2,6 +2,7 @@ from __future__ import unicode_literals
 
 from dvc.utils.compat import str, open
 
+import re
 import os
 import yaml
 import subprocess
@@ -14,7 +15,7 @@ import dvc.logger as logger
 import dvc.dependency as dependency
 import dvc.output as output
 from dvc.exceptions import DvcException
-from dvc.utils import dict_md5, fix_env, load_stage_file
+from dvc.utils import dict_md5, fix_env, load_stage_file_fobj
 
 
 class StageCmdFailedError(DvcException):
@@ -129,6 +130,8 @@ class Stage(object):
         Optional(PARAM_LOCKED): bool,
     }
 
+    TAG_REGEX = r"^(?P<path>.*)@(?P<tag>[^\\/@:]*)$"
+
     def __init__(
         self,
         repo,
@@ -139,6 +142,7 @@ class Stage(object):
         outs=None,
         md5=None,
         locked=False,
+        tag=None,
     ):
         if deps is None:
             deps = []
@@ -153,6 +157,7 @@ class Stage(object):
         self.deps = deps
         self.md5 = md5
         self.locked = locked
+        self.tag = tag
 
     def __repr__(self):
         return "Stage: '{path}'".format(
@@ -171,7 +176,10 @@ class Stage(object):
     @staticmethod
     def is_valid_filename(path):
         return (
-            path.endswith(Stage.STAGE_FILE_SUFFIX)
+            # path.endswith doesn't work for encoded unicode filenames on
+            # Python 2 and since Stage.STAGE_FILE_SUFFIX is ascii then it is
+            # not needed to decode the path from py2's str
+            path[-len(Stage.STAGE_FILE_SUFFIX) :] == Stage.STAGE_FILE_SUFFIX
             or os.path.basename(path) == Stage.STAGE_FILE
         )
 
@@ -208,10 +216,12 @@ class Stage(object):
             return True
 
         for dep in self.deps:
-            if dep.changed():
+            status = dep.status()
+            if status:
                 logger.warning(
-                    "Dependency '{dep}' of '{stage}' changed.".format(
-                        dep=dep, stage=self.relpath
+                    "Dependency '{dep}' of '{stage}' changed because it is "
+                    "'{status}'.".format(
+                        dep=dep, stage=self.relpath, status=status[str(dep)]
                     )
                 )
                 return True
@@ -220,10 +230,12 @@ class Stage(object):
 
     def _changed_outs(self):
         for out in self.outs:
-            if out.changed():
+            status = out.status()
+            if status:
                 logger.warning(
-                    "Output '{out}' of '{stage}' changed.".format(
-                        out=out, stage=self.relpath
+                    "Output '{out}' of '{stage}' changed because it is "
+                    "'{status}'".format(
+                        out=out, stage=self.relpath, status=status[str(out)]
                     )
                 )
                 return True
@@ -252,21 +264,25 @@ class Stage(object):
 
         return ret
 
-    def remove_outs(self, ignore_remove=False):
-        """
-        Used mainly for `dvc remove --outs`
-        """
+    def remove_outs(self, ignore_remove=False, force=False):
+        """Used mainly for `dvc remove --outs` and :func:`Stage.reproduce`."""
         for out in self.outs:
-            out.remove(ignore_remove=ignore_remove)
+            if out.persist and not force:
+                out.unprotect()
+            else:
+                logger.debug(
+                    "Removing output '{out}' of '{stage}'.".format(
+                        out=out, stage=self.relpath
+                    )
+                )
+                out.remove(ignore_remove=ignore_remove)
 
     def unprotect_outs(self):
         for out in self.outs:
-            if out.scheme != "local" or not out.exists:
-                continue
-            self.repo.unprotect(out.path)
+            out.unprotect()
 
-    def remove(self):
-        self.remove_outs(ignore_remove=True)
+    def remove(self, force=False):
+        self.remove_outs(ignore_remove=True, force=force)
         os.unlink(self.path)
 
     def reproduce(
@@ -275,10 +291,6 @@ class Stage(object):
         if not self.changed() and not force:
             return None
 
-        if (self.cmd or self.is_import) and not self.locked and not dry:
-            # Removing outputs only if we actually have command to reproduce
-            self.remove_outs(ignore_remove=False)
-
         msg = (
             "Going to reproduce '{stage}'. "
             "Are you sure you want to continue?".format(stage=self.relpath)
@@ -396,6 +408,9 @@ class Stage(object):
         overwrite=True,
         ignore_build_cache=False,
         remove_outs=False,
+        validate_state=True,
+        outs_persist=None,
+        outs_persist_no_cache=None,
     ):
         if outs is None:
             outs = []
@@ -407,6 +422,10 @@ class Stage(object):
             metrics = []
         if metrics_no_cache is None:
             metrics_no_cache = []
+        if outs_persist is None:
+            outs_persist = []
+        if outs_persist_no_cache is None:
+            outs_persist_no_cache = []
 
         # Backward compatibility for `cwd` option
         if wdir is None and cwd is not None:
@@ -423,13 +442,14 @@ class Stage(object):
 
         stage = Stage(repo=repo, wdir=wdir, cmd=cmd, locked=locked)
 
-        stage.outs = output.loads_from(stage, outs, use_cache=True)
-        stage.outs += output.loads_from(
-            stage, metrics, use_cache=True, metric=True
-        )
-        stage.outs += output.loads_from(stage, outs_no_cache, use_cache=False)
-        stage.outs += output.loads_from(
-            stage, metrics_no_cache, use_cache=False, metric=True
+        Stage._fill_stage_outputs(
+            stage,
+            outs,
+            outs_no_cache,
+            metrics,
+            metrics_no_cache,
+            outs_persist,
+            outs_persist_no_cache,
         )
         stage.deps = dependency.loads_from(stage, deps)
 
@@ -458,23 +478,49 @@ class Stage(object):
         else:
             stage.unprotect_outs()
 
-        if os.path.exists(path):
-            if not ignore_build_cache and stage.is_cached:
-                logger.info("Stage is cached, skipping.")
-                return None
+        if validate_state:
+            if os.path.exists(path):
+                if not ignore_build_cache and stage.is_cached:
+                    logger.info("Stage is cached, skipping.")
+                    return None
 
-            msg = (
-                "'{}' already exists. Do you wish to run the command and "
-                "overwrite it?".format(stage.relpath)
-            )
+                msg = (
+                    "'{}' already exists. Do you wish to run the command and "
+                    "overwrite it?".format(stage.relpath)
+                )
 
-            if not overwrite and not prompt.confirm(msg):
-                raise StageFileAlreadyExistsError(stage.relpath)
+                if not overwrite and not prompt.confirm(msg):
+                    raise StageFileAlreadyExistsError(stage.relpath)
 
-            os.unlink(path)
+                os.unlink(path)
 
         return stage
 
+    @staticmethod
+    def _fill_stage_outputs(
+        stage,
+        outs,
+        outs_no_cache,
+        metrics,
+        metrics_no_cache,
+        outs_persist,
+        outs_persist_no_cache,
+    ):
+        stage.outs = output.loads_from(stage, outs, use_cache=True)
+        stage.outs += output.loads_from(
+            stage, metrics, use_cache=True, metric=True
+        )
+        stage.outs += output.loads_from(
+            stage, outs_persist, use_cache=True, persist=True
+        )
+        stage.outs += output.loads_from(stage, outs_no_cache, use_cache=False)
+        stage.outs += output.loads_from(
+            stage, metrics_no_cache, use_cache=False, metric=True
+        )
+        stage.outs += output.loads_from(
+            stage, outs_persist_no_cache, use_cache=False, persist=True
+        )
+
     @staticmethod
     def _check_dvc_filename(fname):
         if not Stage.is_valid_filename(fname):
@@ -486,19 +532,36 @@ class Stage(object):
             )
 
     @staticmethod
-    def _check_file_exists(fname):
-        if not os.path.exists(fname):
+    def _check_file_exists(repo, fname):
+        if not repo.tree.exists(fname):
             raise StageFileDoesNotExistError(fname)
 
     @staticmethod
-    def load(repo, fname):
-        Stage._check_file_exists(fname)
-        Stage._check_dvc_filename(fname)
-
-        if not Stage.is_stage_file(fname):
+    def _check_isfile(repo, fname):
+        if not repo.tree.isfile(fname):
             raise StageFileIsNotDvcFileError(fname)
 
-        d = load_stage_file(fname)
+    @classmethod
+    def _get_path_tag(cls, s):
+        regex = re.compile(cls.TAG_REGEX)
+        match = regex.match(s)
+        if not match:
+            return s, None
+        return match.group("path"), match.group("tag")
+
+    @staticmethod
+    def load(repo, fname):
+        fname, tag = Stage._get_path_tag(fname)
+
+        # it raises the proper exceptions by priority:
+        # 1. when the file doesn't exists
+        # 2. filename is not a dvc filename
+        # 3. path doesn't represent a regular file
+        Stage._check_file_exists(repo, fname)
+        Stage._check_dvc_filename(fname)
+        Stage._check_isfile(repo, fname)
+
+        d = load_stage_file_fobj(repo.tree.open(fname), fname)
 
         Stage.validate(d, fname=os.path.relpath(fname))
         path = os.path.abspath(fname)
@@ -514,6 +577,7 @@ class Stage(object):
             cmd=d.get(Stage.PARAM_CMD),
             md5=d.get(Stage.PARAM_MD5),
             locked=d.get(Stage.PARAM_LOCKED, False),
+            tag=tag,
         )
 
         stage.deps = dependency.loadd_from(stage, d.get(Stage.PARAM_DEPS, []))
@@ -557,7 +621,7 @@ class Stage(object):
         self.repo.scm.track_file(os.path.relpath(fname))
 
     def _compute_md5(self):
-        from dvc.output.local import OutputLOCAL
+        from dvc.output.base import OutputBase
 
         d = self.dumpd()
 
@@ -575,7 +639,15 @@ class Stage(object):
         # NOTE: excluding parameters that don't affect the state of the
         # pipeline. Not excluding `OutputLOCAL.PARAM_CACHE`, because if
         # it has changed, we might not have that output in our cache.
-        m = dict_md5(d, exclude=[self.PARAM_LOCKED, OutputLOCAL.PARAM_METRIC])
+        m = dict_md5(
+            d,
+            exclude=[
+                self.PARAM_LOCKED,
+                OutputBase.PARAM_METRIC,
+                OutputBase.PARAM_TAGS,
+                OutputBase.PARAM_PERSIST,
+            ],
+        )
         logger.debug("Computed stage '{}' md5: '{}'".format(self.relpath, m))
         return m
 
@@ -681,6 +753,9 @@ class Stage(object):
             raise StageCmdFailedError(self)
 
     def run(self, dry=False, resume=False, no_commit=False, force=False):
+        if (self.cmd or self.is_import) and not self.locked and not dry:
+            self.remove_outs(ignore_remove=False, force=False)
+
         if self.locked:
             logger.info(
                 "Verifying outputs in locked stage '{stage}'".format(
@@ -737,9 +812,11 @@ class Stage(object):
         if paths:
             raise MissingDataSource(paths)
 
-    def checkout(self, force=False):
+    def checkout(self, force=False, progress_callback=None):
         for out in self.outs:
-            out.checkout(force=force)
+            out.checkout(
+                force=force, tag=self.tag, progress_callback=progress_callback
+            )
 
     @staticmethod
     def _status(entries):
@@ -782,3 +859,6 @@ class Stage(object):
                 for out in self.outs
             )
         )
+
+    def get_all_files_number(self):
+        return sum(out.get_files_number() for out in self.outs)

dvc/system.py

@@ -3,6 +3,7 @@ from __future__ import unicode_literals
 from dvc.utils.compat import str, open
 
 import os
+import errno
 
 
 class System(object):
@@ -61,8 +62,22 @@ class System(object):
     @staticmethod
     def _reflink_darwin(src, dst):
         import ctypes
+        import dvc.logger as logger
+
+        LIBC = "libc.dylib"
+        LIBC_FALLBACK = "/usr/lib/libSystem.dylib"
+        try:
+            clib = ctypes.CDLL(LIBC)
+        except OSError as exc:
+            logger.debug(
+                "unable to access '{}' (errno '{}'). "
+                "Falling back to '{}'.".format(LIBC, exc.errno, LIBC_FALLBACK)
+            )
+            if exc.errno != errno.ENOENT:
+                raise
+            # NOTE: trying to bypass System Integrity Protection (SIP)
+            clib = ctypes.CDLL(LIBC_FALLBACK)
 
-        clib = ctypes.CDLL("libc.dylib")
         if not hasattr(clib, "clonefile"):
             return -1
 

dvc/temp_git_repo.py

@@ -0,0 +1,145 @@
+import os
+import shutil
+import tempfile
+
+import git
+
+from dvc import logger
+from dvc.exceptions import DvcException
+
+
+class TempRepoException(DvcException):
+    """Raise when temporary package repository has not properly created"""
+
+    def __init__(self, temp_repo, msg, cause=None):
+        m = "temp repository '{}' error: {}".format(temp_repo.addr, msg)
+        super(TempRepoException, self).__init__(m, cause)
+
+
+class TempGitRepo(object):
+    GIT_DIR_TO_REMOVE = ".git"
+    GIT_FILES_TO_REMOVE = [".gitignore", ".gitmodules"]
+    SUFFIX = "tmp_DVC_mod"
+
+    def __init__(self, addr, module_name, modules_dir):
+        self.addr = addr
+        self.modules_dir = modules_dir
+        self._tmp_mod_prefix = module_name + "_" + self.SUFFIX + "_"
+        self._reset_state()
+
+    def _reset_state(self):
+        self._set_state(None, [])
+
+    def _set_state(self, cloned_tmp_dir, outs):
+        from dvc.repo import Repo
+
+        self.repo = Repo(cloned_tmp_dir) if cloned_tmp_dir else None
+
+        self._cloned_tmp_dir = cloned_tmp_dir
+        self.outs = outs
+        self._moved_files = []
+
+    def fetch(self, targets):
+        for target in targets:
+            try:
+                self.repo.fetch(target)
+            except Exception as ex:
+                msg = "error in fetching data from {}: {}".format(
+                    os.path.basename(target), ex
+                )
+                raise TempRepoException(self, msg)
+
+    @property
+    def is_state_set(self):
+        return self._cloned_tmp_dir is not None
+
+    def __enter__(self):
+        if self.is_state_set:
+            raise TempRepoException(self, "Git repo cloning duplication")
+
+        module_temp_dir = None
+        try:
+            module_temp_dir = self._clone_to_temp_dir()
+            self._remove_git_files_and_dirs(module_temp_dir)
+
+            outputs = self._read_outputs(module_temp_dir)
+            self._set_state(module_temp_dir, outputs)
+        finally:
+            self._clean_mod_temp_dir(module_temp_dir)
+
+        return self
+
+    def _clone_to_temp_dir(self):
+        result = tempfile.mktemp(
+            prefix=self._tmp_mod_prefix, dir=self.modules_dir
+        )
+        logger.debug(
+            "Cloning git repository {} to temp dir {}".format(
+                self.addr, result
+            )
+        )
+        git.Repo.clone_from(self.addr, result, depth=1)
+        return result
+
+    def _remove_git_files_and_dirs(self, module_temp_dir):
+        logger.debug(
+            "Removing git meta files from {}: {} dif and {}".format(
+                module_temp_dir,
+                TempGitRepo.GIT_DIR_TO_REMOVE,
+                ", ".join(TempGitRepo.GIT_FILES_TO_REMOVE),
+            )
+        )
+        shutil.rmtree(
+            os.path.join(module_temp_dir, TempGitRepo.GIT_DIR_TO_REMOVE)
+        )
+        for item in TempGitRepo.GIT_FILES_TO_REMOVE:
+            fname = os.path.join(module_temp_dir, item)
+            if os.path.exists(fname):
+                os.remove(fname)
+
+    def _clean_mod_temp_dir(self, module_temp_dir):
+        if not self.is_state_set:
+            if module_temp_dir and os.path.exists(module_temp_dir):
+                shutil.rmtree(module_temp_dir, ignore_errors=True)
+
+    def _read_outputs(self, module_temp_dir):
+        from dvc.repo import Repo
+
+        pkg_repo = Repo(module_temp_dir)
+        stages = pkg_repo.stages()
+        return [out for s in stages for out in s.outs]
+
+    def persist_to(self, module_dir, parent_repo):
+        if not self.is_state_set:
+            raise TempRepoException(self, "cannot persist")
+
+        tmp_repo_cache = self.repo.cache.local.url
+
+        for prefix in os.listdir(tmp_repo_cache):
+            if len(prefix) != 2:
+                logger.warning(
+                    "wrong dir format in cache {}: dir {}".format(
+                        tmp_repo_cache, prefix
+                    )
+                )
+            self._move_all_cache_files(parent_repo, prefix, tmp_repo_cache)
+
+        shutil.move(self._cloned_tmp_dir, module_dir)
+        self._reset_state()
+
+    @staticmethod
+    def _move_all_cache_files(parent_repo, prefix, tmp_repo_cache):
+        obj_name = os.path.join(tmp_repo_cache, prefix)
+        for suffix in os.listdir(obj_name):
+            src_path = os.path.join(tmp_repo_cache, prefix, suffix)
+            pre = os.path.join(parent_repo.cache.local.url, prefix)
+            if not os.path.exists(pre):
+                os.mkdir(pre)
+            dest = os.path.join(pre, suffix)
+            shutil.move(src_path, dest)
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        if self.is_state_set:
+            shutil.rmtree(self._cloned_tmp_dir)
+            self._reset_state()
+        pass

dvc/utils/__init__.py

@@ -232,16 +232,20 @@ def current_timestamp():
 
 
 def load_stage_file(path):
+    with open(path, "r") as fobj:
+        return load_stage_file_fobj(fobj, path)
+
+
+def load_stage_file_fobj(fobj, path):
     from dvc.exceptions import StageFileCorruptedError
 
-    with open(path, "r") as fobj:
-        try:
-            return yaml.safe_load(fobj) or {}
-        except ScannerError:
-            raise StageFileCorruptedError(path)
+    try:
+        return yaml.safe_load(fobj) or {}
+    except ScannerError:
+        raise StageFileCorruptedError(path)
 
 
 def walk_files(directory):
-    for root, _, files in os.walk(directory):
+    for root, _, files in os.walk(str(directory)):
         for f in files:
             yield os.path.join(root, f)

dvc/utils/compat.py

@@ -27,18 +27,48 @@ def encode(u, encoding=None):
     return u.encode(encoding, "replace")
 
 
-# NOTE: cast_bytes_py2 is taken from
-# https://github.com/ipython/ipython_genutils
+def csv_reader(unicode_csv_data, dialect=None, **kwargs):
+    """csv.reader doesn't support Unicode input, so need to use some tricks
+    to work around this.
+
+    Source: https://docs.python.org/2/library/csv.html#csv-examples
+    """
+    import csv
+
+    dialect = dialect or csv.excel
+
+    if is_py3:
+        # Python3 supports encoding by default, so just return the object
+        for row in csv.reader(unicode_csv_data, dialect=dialect, **kwargs):
+            yield [cell for cell in row]
+
+    else:
+        # csv.py doesn't do Unicode; encode temporarily as UTF-8:
+        reader = csv.reader(
+            utf_8_encoder(unicode_csv_data), dialect=dialect, **kwargs
+        )
+        for row in reader:
+            # decode UTF-8 back to Unicode, cell by cell:
+            yield [unicode(cell, "utf-8") for cell in row]  # noqa: F821
+
+
+def utf_8_encoder(unicode_csv_data):
+    """Source: https://docs.python.org/2/library/csv.html#csv-examples"""
+    for line in unicode_csv_data:
+        yield line.encode("utf-8")
+
+
 def cast_bytes(s, encoding=None):
+    """Source: https://github.com/ipython/ipython_genutils"""
     if not isinstance(s, bytes):
         return encode(s, encoding)
     return s
 
 
-# NOTE _makedirs is taken from
-# https://github.com/python/cpython/blob/
-# 3ce3dea60646d8a5a1c952469a2eb65f937875b3/Lib/os.py#L196-L226
 def _makedirs(name, mode=0o777, exist_ok=False):
+    """Source: https://github.com/python/cpython/blob/
+        3ce3dea60646d8a5a1c952469a2eb65f937875b3/Lib/os.py#L196-L226
+    """
     head, tail = os.path.split(name)
     if not tail:
         head, tail = os.path.split(head)

requirements.txt

@@ -1,5 +1,5 @@
 azure-storage-blob==1.3.0
-boto3==1.9.86
+boto3==1.9.115
 ply>=3.9 # See https://github.com/pyinstaller/pyinstaller/issues/1945
 configparser>=3.5.0
 zc.lockfile>=1.2.1
@@ -25,3 +25,4 @@ pydot>=1.2.4
 asciimatics>=1.10.0
 distro>=1.3.0
 appdirs>=1.4.3
+treelib>=1.5.5

scripts/ci/deploy_condition.sh

@@ -9,7 +9,7 @@ if [[ $python_ver != "2" ]]; then
     exit 0
 fi
 
-if [[ $TRAVIS_OS_NAME == "osx" && $TRAVIS_OSX_IMAGE != "xcode10.1" ]]; then
+if [[ $TRAVIS_OS_NAME == "osx" && $TRAVIS_OSX_IMAGE != "xcode8.3" ]]; then
     echo false
     exit 0
 fi

scripts/ci/script.sh

@@ -8,6 +8,7 @@ python -mtests
 if [[ "$TRAVIS_PULL_REQUEST" == "false" && \
       "$TRAVIS_SECURE_ENV_VARS" == "true" && \
       "$(python -c 'import sys; print(sys.version_info[0])')" == '2' ]]; then
+	pip install codecov
 	codecov
 fi
 

scripts/completion/dvc.bash

@@ -9,28 +9,28 @@
 #----------------------------------------------------------
 
 _dvc_commands='init destroy add import checkout run pull push fetch \
-              status repro remove move gc config remote metrics install \
-              root lock unlock pipeline'
+              status repro remove move unprotect gc config remote metrics \
+              install root lock unlock pipeline commit'
 
 _dvc_global_options="-h --help -q --quiet -V --verbose"
 _dvc_options="-h --help -v --version"
 _dvc_init="--no-scm -f --force"
 _dvc_destroy="-f --force"
-_dvc_add="-R --recursive"
-_dvc_run="--no-exec -f --file -c --cwd -d --deps -o --outs -O --outs-no-cache -M --metrics-no-cache -y --yes"
-_dvc_pull="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps"
-_dvc_push="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps"
-_dvc_fetch="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps"
-_dvc_status="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -c --cloud"
-_dvc_repro="--dry -f --force -s --single-item -c --cwd -m --metrics -i --interactive -p --pipeline -P --all-pipelines"
-_dvc_remove="--dry -o --outs -p --purge"
-_dvc_gc="-a --all-branches -T --all-tags -c --cloud -r --remote"
-_dvc_config="-u --unset --local"
+_dvc_add="-R --recursive -f --file --no-commit"
+_dvc_run="--no-exec -f --file -c --cwd -d --deps -o --outs -O --outs-no-cache -M --metrics-no-cache -y --yes --overwrite-dvc-file --ignore-build-cache --remove-outs --no-commit -w --wdir"
+_dvc_pull="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -f --force -R --recursive"
+_dvc_push="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -R --recursive"
+_dvc_fetch="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -R --recursive"
+_dvc_status="--show-checksums -j --jobs -r --remote -a --all-branches -T --all-tags -d --with-deps -c --cloud -q --quiet"
+_dvc_repro="--dry -f --force -s --single-item -c --cwd -m --metrics -i --interactive -p --pipeline -P --all-pipelines --ignore-build-cache --no-commit -s --single-item"
+_dvc_remove="--dry -o --outs -p --purge --force"
+_dvc_gc="-a --all-branches -T --all-tags -c --cloud -r --remote -f --force -p --project"
+_dvc_config="-u --unset --local --system --global"
 _dvc_checkout="$(compgen -G '*.dvc')"
 _dvc_move="$(compgen -G '*')"
 _dvc_lock="$(compgen -G '*.dvc')"
 _dvc_unlock="$(compgen -G '*.dvc')"
-_dvc_import=""
+_dvc_import="--resume"
 _dvc_remote=""
 _dvc_metrics=""
 _dvc_install=""

scripts/completion/dvc.zsh

@@ -25,6 +25,7 @@ _dvc_commands() {
     "repro:Reproduce DVC file. Default file name - 'Dvcfile'."
     "remove:Remove outputs of DVC file."
     "move:Move output of DVC file."
+    "unprotect:Unprotect data file/directory."
     "gc:Collect garbage."
     "config:Get or set config options."
     "remote:Manage set of tracked repositories."
@@ -34,6 +35,7 @@ _dvc_commands() {
     "lock:Lock DVC file."
     "unlock:Unlock DVC file."
     "pipeline:Manage pipeline."
+    "commit:Record changes to the repository."
   )
 
   _describe 'dvc commands' _commands
@@ -61,61 +63,80 @@ _dvc_destroy=(
 
 _dvc_add=(
   {-R,--recursive}"[Recursively add each file under the directory.]"
+  {-f,--file}"[Specify name of the stage file.]:File:_files"
+  "--no-commit[Don't put files/directories into cache.]"
   "1:File:_files"
 )
 
 _dvc_import=(
+  "--resume[Resume previously started download.]"
   "1:URL:"
   "2:Output:"
 )
 
 _dvc_checkout=(
-  "1:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  {-f,--force}"[Do not prompt when removing working directory files.]"
+  {-d,--with-deps}"[Checkout all dependencies of the specified target.]"
+  "1:Stages:_files -g '(*.dvc|Dvcfile)'"
+)
+
+_dvc_unprotect=(
+  "*:Data files:_files"
 )
 
 _dvc_run=(
   "--no-exec[Only create stage file without actually running it.]"
+  "--overwrite-dvcfile[Overwrite existing dvc file without asking for confirmation.]"
+  "--ignore-build-cache[Run this stage even if it has been already ran with the same command/dependencies/outputs/etc before.]"
+  "--remove-outs[Remove outputs before running the command.]"
+  "--no-commit[Don't put files/directories into cache.]"
   {-f,--file}"[Specify name of the stage file.]:File:_files"
-  {-c,--cwd}"[Directory to run your command and place state file in.]:CWD:_files -/"
+  {-c,--cwd}"[Deprecated, use -w and -f instead.]:CWD:_files -/"
+  {-w,--wdir}"[Directory to run your command and place state file in.]:WDIR:_files -/"
   "*"{-d,--deps}"[Declare dependencies for reproducible cmd.]:Dependency:_files"
   "*"{-o,--outs}"[Declare output data file or data directory.]:Output data:_files"
   "*"{-O,--outs-no-cache}"[Declare output regular file or directory.]:Output regular:_files"
-  "*"{-M,--metrics-no-cache}"[Declare output metric file or directory]:Metrics (no cache):_files"
+  "*"{-M,--metrics-no-cache}"[Declare output metric file or directory (do not put into DVC cache)]:Metrics (no cache):_files"
+  "*"{-m,--metrics}"[Declare output metric file or directory]:Metrics:_files"
   {-y,--yes}"[Automatic 'yes' answer to all prompts.]"
 )
 
 _dvc_pull=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
   "--show-checksums[Show checksums instead of file names]"
   {-j,--jobs}"[Number of jobs to run simultaneously]:Number of jobs:"
   {-r,--remote}"[Remote repository to pull from]:Remote repository:"
   {-a,--all-branches}"[Fetch cache for all branches]"
   {-T,--all-tags}"[Fetch cache for all tags]"
   {-d,--with-deps}"[Fetch cache for all dependencies of the specified target]"
+  {-f,--force}"[Do not prompt when removing working directory files.]"
+  {-R,--recursive}"[Pull cache for subdirectories of the specified directory.]"
 )
 
 _dvc_push=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
   "--show-checksums[Show checksums instead of file names]"
   {-j,--jobs}"[Number of jobs to run simultaneously]:Number of jobs:"
   {-r,--remote}"[Remote repository to pull from]:Remote repository:"
   {-a,--all-branches}"[Fetch cache for all branches]"
   {-T,--all-tags}"[Fetch cache for all tags]"
   {-d,--with-deps}"[Fetch cache for all dependencies of the specified target]"
+  {-R,--recursive}"[Push cache for subdirectories of the specified directory.]"
 )
 
 _dvc_fetch=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
   "--show-checksums[Show checksums instead of file names]"
   {-j,--jobs}"[Number of jobs to run simultaneously]:Number of jobs:"
   {-r,--remote}"[Remote repository to pull from]:Remote repository:"
   {-a,--all-branches}"[Fetch cache for all branches]"
   {-T,--all-tags}"[Fetch cache for all tags]"
   {-d,--with-deps}"[Fetch cache for all dependencies of the specified target]"
+  {-R,--recursive}"[Fetch cache for subdirectories of the specified directory.]"
 )
 
 _dvc_status=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
   "--show-checksums[Show checksums instead of file names]"
   {-j,--jobs}"[Number of jobs to run simultaneously]:Number of jobs:"
   {-r,--remote}"[Remote repository to pull from]:Remote repository:"
@@ -123,14 +144,16 @@ _dvc_status=(
   {-T,--all-tags}"[Fetch cache for all tags]"
   {-d,--with-deps}"[Fetch cache for all dependencies of the specified target]"
   {-c,--cloud}"[Show status of a local cache compared to a remote repository]"
+  {-q,--quiet}"[Suppresses all output. Exit with 0 if pipeline is up]"
 )
 
 _dvc_repro=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
   "--dry[Only print the commands that would be executed without actually executing]"
+  "--ignore-build-cache[Reproduce all descendants of a changed stage even if their direct dependencies didn't change.]"
+  "--no-commit[Don't put files/directories into cache.]"
   {-f,--force}"[Reproduce even if dependencies were not changed.]"
   {-s,--single-item}"[Reproduce only single data item without recursive dependencies check.]"
-  {-c,--cwd}"[Directory to reproduce from]:CWD:_files -/"
   {-m,--metrics}"[Show metrics after reproduction.]"
   {-i,--interactive}"[Ask for confirmation before reproducing each stage.]"
   {-p,--pipeline}"[Reproduce the whole pipeline that the specified stage file belongs to.]"
@@ -138,10 +161,11 @@ _dvc_repro=(
 )
 
 _dvc_remove=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
   "--dry[Only print the commands that would be executed without actually executing]"
   {-o,--outs}"[Only remove DVC file outputs.(default)]"
   {-p,--purge}"[Remove DVC file and all its outputs]"
+  {-f,--force}"[Force purge.]"
 )
 
 _dvc_move=(
@@ -154,19 +178,28 @@ _dvc_gc=(
   {-T,--all-tags}"[Collect garbage for all tags]"
   {-c,--cloud}"[Collect garbage in remote repository]"
   {-r,--remote}"[Remote repository to collect garbage in]:Remote repository:"
+  {-f,--force}"[Force garbage collection - automatically agree to all prompts.]"
+
+  {-p,--projects}"[Keep data files required by these projects in addition to the current one. Useful if you share a single cache across repos.]:Repos:_files"
 )
 
 _dvc_config=(
   {-u,--unset}"[Unset option.]"
   "--local[Unset option.]"
+  "--system[Use system config.]"
+  "--global[Use global config.]"
 )
 
 _dvc_remote=(
-  "1:Task:(add remove modify list)"
+  "1:Sub command:(add default remove modify list)"
+)
+
+_dvc_cache=(
+  "1:Sub command:(dir)"
 )
 
 _dvc_metrics=(
-  "1:Task:(show add modify remove)"
+  "1:Sub command:(show add modify remove)"
 )
 
 _dvc_install=()
@@ -174,15 +207,22 @@ _dvc_install=()
 _dvc_root=()
 
 _dvc_lock=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
 )
 
 _dvc_unlock=(
-  "*:DVC files:_files -g '(*.dvc|Dvcfile)'"
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
 )
 
 _dvc_pipeline=(
-  "1:Task:(show)"
+  "1:Sub command:(show,list)"
+)
+
+_dvc_commit=(
+  "*:Stages:_files -g '(*.dvc|Dvcfile)'"
+  {-d,--with-deps}"[Commit all dependencies of the specified target.]"
+  {-f,--force}"[Commit even if checksums for dependencies/outputs changed.]"
+  {-R,--recursive}"[Commit cache for subdirectories of the specified directory.]"
 )
 
 typeset -A opt_args
@@ -194,26 +234,28 @@ _arguments \
   '*::args:->args'
 
 case $words[1] in
-  init)     _arguments $_dvc_global_options   $_dvc_init     ;;
-  destroy)  _arguments $_dvc_global_options   $_dvc_destroy  ;;
-  add)      _arguments $_dvc_global_options   $_dvc_add      ;;
-  import)   _arguments $_dvc_global_options   $_dvc_import   ;;
-  checkout) _arguments $_dvc_global_options   $_dvc_checkout ;;
-  run)      _arguments $_dvc_global_options   $_dvc_run      ;;
-  pull)     _arguments $_dvc_global_options   $_dvc_pull     ;;
-  push)     _arguments $_dvc_global_options   $_dvc_push     ;;
-  fetch)    _arguments $_dvc_global_options   $_dvc_fetch    ;;
-  status)   _arguments $_dvc_global_options   $_dvc_status   ;;
-  repro)    _arguments $_dvc_global_options   $_dvc_repro    ;;
-  remove)   _arguments $_dvc_global_options   $_dvc_remove   ;;
-  move)     _arguments $_dvc_global_options   $_dvc_move     ;;
-  gc)       _arguments $_dvc_global_options   $_dvc_gc       ;;
-  config)   _arguments $_dvc_global_options   $_dvc_config   ;;
-  remote)   _arguments $_dvc_global_options   $_dvc_remote   ;;
-  metrics)  _arguments $_dvc_global_options   $_dvc_metrics  ;;
-  install)  _arguments $_dvc_global_options   $_dvc_install  ;;
-  root)     _arguments $_dvc_global_options   $_dvc_root     ;;
-  lock)     _arguments $_dvc_global_options   $_dvc_lock     ;;
-  unlock)   _arguments $_dvc_global_options   $_dvc_unlock   ;;
-  pipeline) _arguments $_dvc_global_options   $_dvc_pipeline ;;
+  init)      _arguments $_dvc_global_options   $_dvc_init      ;;
+  destroy)   _arguments $_dvc_global_options   $_dvc_destroy   ;;
+  add)       _arguments $_dvc_global_options   $_dvc_add       ;;
+  import)    _arguments $_dvc_global_options   $_dvc_import    ;;
+  checkout)  _arguments $_dvc_global_options   $_dvc_checkout  ;;
+  run)       _arguments $_dvc_global_options   $_dvc_run       ;;
+  pull)      _arguments $_dvc_global_options   $_dvc_pull      ;;
+  push)      _arguments $_dvc_global_options   $_dvc_push      ;;
+  fetch)     _arguments $_dvc_global_options   $_dvc_fetch     ;;
+  status)    _arguments $_dvc_global_options   $_dvc_status    ;;
+  repro)     _arguments $_dvc_global_options   $_dvc_repro     ;;
+  remove)    _arguments $_dvc_global_options   $_dvc_remove    ;;
+  move)      _arguments $_dvc_global_options   $_dvc_move      ;;
+  gc)        _arguments $_dvc_global_options   $_dvc_gc        ;;
+  config)    _arguments $_dvc_global_options   $_dvc_config    ;;
+  remote)    _arguments $_dvc_global_options   $_dvc_remote    ;;
+  metrics)   _arguments $_dvc_global_options   $_dvc_metrics   ;;
+  install)   _arguments $_dvc_global_options   $_dvc_install   ;;
+  root)      _arguments $_dvc_global_options   $_dvc_root      ;;
+  lock)      _arguments $_dvc_global_options   $_dvc_lock      ;;
+  unlock)    _arguments $_dvc_global_options   $_dvc_unlock    ;;
+  pipeline)  _arguments $_dvc_global_options   $_dvc_pipeline  ;;
+  commit)    _arguments $_dvc_global_options   $_dvc_commit    ;;
+  unprotect) _arguments $_dvc_global_options   $_dvc_unprotect ;;
 esac

scripts/hooks/hook-checksumdir.py

@@ -1,3 +0,0 @@
-from PyInstaller.utils.hooks import copy_metadata
-
-datas = copy_metadata("checksumdir")

scripts/hooks/hook-google.cloud.storage.py

@@ -1,3 +1,9 @@
 from PyInstaller.utils.hooks import copy_metadata
 
 datas = copy_metadata("google-cloud-storage")
+
+# NOTE: including "requests" metadata because of google-resumable-media
+# bug. See [1] for more info.
+#
+# [1] https://github.com/googleapis/google-resumable-media-python/issues/59
+datas += copy_metadata("requests")

setup.py

@@ -22,11 +22,12 @@ install_requires = [
     "asciimatics>=1.10.0",
     "distro>=1.3.0",
     "appdirs>=1.4.3",
+    "treelib>=1.5.5",
 ]
 
 # Extra dependencies for remote integrations
 gs = ["google-cloud-storage==1.13.0"]
-s3 = ["boto3==1.7.4"]
+s3 = ["boto3==1.9.115"]
 azure = ["azure-storage-blob==1.3.0"]
 ssh = ["paramiko>=2.4.1"]
 all_remotes = gs + s3 + azure + ssh

tests/__main__.py

@@ -13,14 +13,12 @@ os.putenv("DVC_HOME", REPO_ROOT)
 os.putenv("DVC_TEST", "true")
 
 if len(sys.argv) == 1:
-    scope = "--all-modules"
+    params = ""
 else:
-    scope = " ".join(sys.argv[1:])
+    params = " ".join(sys.argv[1:])
 
 cmd = (
-    "nosetests -v --processes=4 --process-timeout=600 --cover-inclusive "
-    "--cover-erase --cover-package=dvc --with-coverage --with-flaky "
-    "--logging-clear-handlers "
-    "{scope} ".format(scope=scope)
+    "py.test -v -n=4 --timeout=600 --timeout_method=thread --cov=dvc "
+    "{params} ".format(params=params)
 )
 check_call(cmd, shell=True)

tests/basic_env.py

@@ -1,4 +1,9 @@
+# encoding: utf-8
+
+from __future__ import unicode_literals
+
 import os
+import shutil
 import uuid
 import tempfile
 
@@ -7,10 +12,12 @@ from git.exc import GitCommandNotFound
 from unittest import TestCase
 
 import dvc.logger as logger
+from dvc.command.remote import CmdRemoteAdd
 from dvc.repo import Repo as DvcRepo
+from dvc.utils.compat import open, str
 
 
-class TestDir(TestCase):
+class TestDirFixture(object):
     GCP_CREDS_FILE = os.path.abspath(
         os.path.join("scripts", "ci", "gcp-creds.json")
     )
@@ -38,6 +45,15 @@ class TestDir(TestCase):
         "import sys\nimport shutil\n"
         "shutil.copyfile(sys.argv[1], sys.argv[2])"
     )
+    UNICODE = "тест"
+    UNICODE_CONTENTS = "проверка"
+
+    def __init__(self, root_dir=None):
+        if root_dir:
+            os.mkdir(root_dir)
+        else:
+            root_dir = self.mkdtemp()
+        self._root_dir = os.path.realpath(root_dir)
 
     def _pushd(self, d):
         self._saved_dir = os.path.realpath(os.curdir)
@@ -52,8 +68,12 @@ class TestDir(TestCase):
         if len(dname) > 0 and not os.path.isdir(dname):
             os.makedirs(dname)
 
-        with open(name, "a") as f:
-            f.write(contents)
+        with open(name, "a", encoding="utf-8") as f:
+            f.write(
+                contents
+                if isinstance(contents, str)
+                else contents.decode("utf-8")
+            )
 
     @staticmethod
     def mkdtemp():
@@ -62,8 +82,6 @@ class TestDir(TestCase):
         return tempfile.mkdtemp(prefix=prefix, suffix=suffix)
 
     def setUp(self):
-        self._root_dir = TestDir.mkdtemp()
-
         self._pushd(self._root_dir)
         self.create(self.FOO, self.FOO_CONTENTS)
         self.create(self.BAR, self.BAR_CONTENTS)
@@ -72,16 +90,17 @@ class TestDir(TestCase):
         os.mkdir(self.DATA_SUB_DIR)
         self.create(self.DATA, self.DATA_CONTENTS)
         self.create(self.DATA_SUB, self.DATA_SUB_CONTENTS)
+        self.create(self.UNICODE, self.UNICODE_CONTENTS)
 
     def tearDown(self):
         self._popd()
 
 
-class TestGit(TestDir):
+class TestGitFixture(TestDirFixture):
     N_RETRIES = 5
 
     def setUp(self):
-        super(TestGit, self).setUp()
+        super(TestGitFixture, self).setUp()
         # NOTE: handles EAGAIN error on BSD systems (osx in our case).
         # Otherwise when running tests you might get this exception:
         #
@@ -100,17 +119,118 @@ class TestGit(TestDir):
         self.git.index.commit("add code")
 
 
-class TestGitSubmodule(TestGit):
+class TestGitSubmoduleFixture(TestGitFixture):
+    def __init__(self, root_dir=None):
+        super(TestGitSubmoduleFixture, self).__init__(root_dir)
+
     def setUp(self):
-        super(TestGitSubmodule, self).setUp()
+        super(TestGitSubmoduleFixture, self).setUp()
         subrepo = Repo.init()
         subrepo_path = "subrepo"
         self.git.create_submodule(subrepo_path, subrepo_path, subrepo.git_dir)
         self._pushd(subrepo_path)
 
 
-class TestDvc(TestGit):
+class TestDvcFixture(TestGitFixture):
+    def __init__(self, root_dir=None):
+        super(TestDvcFixture, self).__init__(root_dir)
+
     def setUp(self):
-        super(TestDvc, self).setUp()
+        super(TestDvcFixture, self).setUp()
         self.dvc = DvcRepo.init(self._root_dir)
+        self.dvc.scm.commit("init dvc")
         logger.be_verbose()
+
+
+class TestDvcGitInitializedFixture(TestDvcFixture):
+    def __init__(self, root_dir=None):
+        super(TestDvcGitInitializedFixture, self).__init__(root_dir)
+
+    def setUp(self):
+        super(TestDvcGitInitializedFixture, self).setUp()
+        self.git.init()
+
+
+class TestDvcDataFileFixture(TestDvcGitInitializedFixture):
+    DATA_DVC_FILE = "data.dvc"
+    DATA_DIR_DVC_FILE = "data_sub_dir.dvc"
+    REMOTE = "myremote2"
+
+    def __init__(self, root_dir=None, cache_dir=None):
+        super(TestDvcDataFileFixture, self).__init__(root_dir)
+        self.cache_dir = cache_dir
+
+    def setUp(self):
+        super(TestDvcDataFileFixture, self).setUp()
+
+        self.dvc.add(self.DATA)
+        self.dvc.add(self.DATA_SUB_DIR)
+
+        if self.cache_dir:
+            shutil.copytree(self.dvc.cache.local.cache_dir, self.cache_dir)
+
+            class MockConfig:
+                system = None
+                glob = None
+                local = None
+                default = True
+                name = self.REMOTE
+                url = self.cache_dir
+
+            cmd = CmdRemoteAdd(MockConfig())
+            cmd.run()
+
+        self.git.index.add(
+            [
+                os.path.join(self.DATA_DIR, self.DATA_DVC_FILE),
+                os.path.join(self.DATA_DIR, self.DATA_DIR_DVC_FILE),
+                os.path.join(self.DATA_DIR, ".gitignore"),
+                ".dvc/config",
+                self.FOO,
+                self.BAR,
+            ]
+        )
+        self.git.index.commit("Hello world commit")
+
+
+# NOTE: Inheritance order in the classes below is important.
+
+
+class TestDir(TestDirFixture, TestCase):
+    def __init__(self, methodName, root_dir=None):
+        TestDirFixture.__init__(self, root_dir)
+        TestCase.__init__(self, methodName)
+
+
+class TestGit(TestGitFixture, TestCase):
+    def __init__(self, methodName, root_dir=None):
+        TestGitFixture.__init__(self, root_dir)
+        TestCase.__init__(self, methodName)
+
+
+class TestGitSubmodule(TestGitSubmoduleFixture, TestCase):
+    def __init__(self, methodName, root_dir=None):
+        TestGitSubmoduleFixture.__init__(self, root_dir)
+        TestCase.__init__(self, methodName)
+
+
+class TestDvc(TestDvcFixture, TestCase):
+    def __init__(self, methodName, root_dir=None):
+        TestDvcFixture.__init__(self, root_dir)
+        TestCase.__init__(self, methodName)
+
+
+class TestDvcPkg(TestDvcFixture, TestCase):
+    GIT_PKG = "git_pkg"
+    CACHE_DIR = "mycache"
+
+    def __init__(self, methodName, root_dir=None):
+        TestDvcFixture.__init__(self, root_dir)
+        TestCase.__init__(self, methodName)
+
+        self.pkg_dir = os.path.join(self._root_dir, self.GIT_PKG)
+        cache_dir = os.path.join(self._root_dir, self.CACHE_DIR)
+        self.pkg_fixture = TestDvcDataFileFixture(
+            root_dir=self.pkg_dir, cache_dir=cache_dir
+        )
+        self.pkg_fixture.setUp()

tests/requirements.txt

@@ -1,11 +1,11 @@
-pytest>=3.3.1
-nose>=1.3.7
+pytest>=4.4.0
+pytest-timeout>=1.3.3
+pytest-cov>=2.6.1
+pytest-xdist>=1.26.1
 flaky>=3.5.3
 mock>=2.0.0
-coverage>=4.5.1
-codecov>=2.0.15
 xmltodict>=0.11.0
-awscli>=1.14.18
+awscli>=1.16.125
 google-compute-engine
 pywin32; sys_platform == 'win32'
 Pygments  # required by collective.checkdocs

tests/test_add.py

@@ -39,6 +39,7 @@ class TestAdd(TestDvc):
         self.assertEqual(len(stage.deps), 0)
         self.assertEqual(stage.cmd, None)
         self.assertEqual(stage.outs[0].info["md5"], md5)
+        self.assertEqual(stage.md5, "ee343f2482f53efffc109be83cc976ac")
 
     def test_unicode(self):
         fname = "\xe1"
@@ -475,7 +476,7 @@ class TestShouldCleanUpAfterFailedAdd(TestDvc):
         self.assertFalse(os.path.exists(bar_stage_file))
 
         gitignore_content = get_gitignore_content()
-        self.assertFalse(any(self.BAR in line for line in gitignore_content))
+        self.assertNotIn("/" + self.BAR, gitignore_content)
 
 
 class TestShouldNotTrackGitInternalFiles(TestDvc):

tests/test_analytics.py

@@ -27,18 +27,24 @@ class TestAnalytics(TestDir):
         self.assertNotEqual(a.info[a.PARAM_SYSTEM_INFO], {})
 
     @mock.patch.object(os, "getenv", new=_clean_getenv)
+    @mock.patch(
+        "dvc.analytics.Analytics._is_enabled_config", return_value=True
+    )
     @mock.patch("requests.post")
-    def test_send(self, mockpost):
+    def test_send(self, mockpost, _):
         ret = main(["daemon", "analytics", Analytics().dump(), "-v"])
         self.assertEqual(ret, 0)
 
         self.assertTrue(mockpost.called)
 
     @mock.patch.object(os, "getenv", new=_clean_getenv)
+    @mock.patch(
+        "dvc.analytics.Analytics._is_enabled_config", return_value=True
+    )
     @mock.patch.object(
         requests, "post", side_effect=requests.exceptions.RequestException()
     )
-    def test_send_failed(self, mockpost):
+    def test_send_failed(self, mockpost, _):
         ret = main(["daemon", "analytics", Analytics().dump(), "-v"])
         self.assertEqual(ret, 0)
 

tests/test_cache.py

@@ -128,6 +128,10 @@ class TestCacheLinkType(TestDvc):
 
 
 class TestCmdCacheDir(TestDvc):
+    def test(self):
+        ret = main(["cache", "dir"])
+        self.assertEqual(ret, 254)
+
     def test_abs_path(self):
         dname = os.path.join(os.path.dirname(self._root_dir), "dir")
         ret = main(["cache", "dir", dname])

tests/test_checkout.py

@@ -1,20 +1,31 @@
 import os
-import mock
+import sys
+import re
+
 import yaml
 import shutil
 import filecmp
 import collections
 
+from dvc.logger import logger
 from dvc.main import main
+from dvc import progress
 from dvc.repo import Repo as DvcRepo
 from dvc.system import System
+from dvc.utils import walk_files
 from tests.basic_env import TestDvc
 from tests.test_repro import TestRepro
-from dvc.stage import Stage
+from dvc.stage import Stage, StageFileBadNameError, StageFileDoesNotExistError
 from dvc.remote.local import RemoteLOCAL
-from dvc.exceptions import DvcException, ConfirmRemoveError
+from dvc.exceptions import (
+    DvcException,
+    ConfirmRemoveError,
+    TargetNotDirectoryError,
+    CheckoutErrorSuggestGit,
+)
 
 from mock import patch
+from tests.utils.logger import MockLoggerHandlers, ConsoleFontColorsRemover
 
 
 class TestCheckout(TestRepro):
@@ -121,10 +132,7 @@ class CheckoutBase(TestDvc):
         FileInfo = collections.namedtuple("FileInfo", "path inode")
 
         paths = [
-            os.path.join(root, file)
-            for output in stage.outs
-            for root, _, files in os.walk(output.path)
-            for file in files
+            path for output in stage.outs for path in walk_files(output.path)
         ]
 
         return [
@@ -355,8 +363,8 @@ class TestCheckoutDirectory(TestRepro):
 
 
 class TestCheckoutHook(TestDvc):
-    @mock.patch("sys.stdout.isatty", return_value=True)
-    @mock.patch("dvc.utils.compat.input", side_effect=EOFError)
+    @patch("sys.stdout.isatty", return_value=True)
+    @patch("dvc.prompt.input", side_effect=EOFError)
     def test(self, mock_input, mock_isatty):
         """ Test that dvc checkout handles EOFError gracefully, which is what
         it will experience when running in a git hook.
@@ -378,27 +386,139 @@ class TestCheckoutSuggestGit(TestRepro):
         try:
             self.dvc.checkout(target="gitbranch")
         except DvcException as exc:
-            msg = str(exc)
-            self.assertEqual(
-                msg,
-                (
-                    "'gitbranch' does not exist. "
-                    "Did you mean 'git checkout gitbranch'?"
-                ),
-            )
+            self.assertIsInstance(exc, CheckoutErrorSuggestGit)
+            self.assertIsInstance(exc.cause, StageFileDoesNotExistError)
+            self.assertIsNone(exc.cause.cause)
 
         try:
             self.dvc.checkout(target=self.FOO)
         except DvcException as exc:
-            msg = str(exc)
-            self.assertEqual(
-                msg,
-                (
-                    "bad stage filename '{}'."
-                    " Stage files should be named 'Dvcfile'"
-                    " or have a '.dvc' suffix (e.g. '{}.dvc')."
-                    " Did you mean 'git checkout {}'?".format(
-                        self.FOO, self.FOO, self.FOO
+            self.assertIsInstance(exc, CheckoutErrorSuggestGit)
+            self.assertIsInstance(exc.cause, StageFileBadNameError)
+            self.assertIsNone(exc.cause.cause)
+
+
+class TestCheckoutShouldHaveSelfClearingProgressBar(TestDvc):
+    def setUp(self):
+        super(TestCheckoutShouldHaveSelfClearingProgressBar, self).setUp()
+        self._prepare_repo()
+
+    def test(self):
+        self._checkout_and_intercept_std_output()
+
+        stdout_calls = self.stdout_mock.method_calls
+        write_calls = self.filter_out_non_write_calls(stdout_calls)
+        write_calls = self.filter_out_empty_write_calls(write_calls)
+        self.write_args = [w_c[1][0] for w_c in write_calls]
+
+        progress_bars = self.get_progress_bars()
+        update_bars = progress_bars[:-1]
+        finish_bar = progress_bars[-1]
+
+        self.assertEqual(4, len(update_bars))
+        self.assertRegexpMatches(progress_bars[0], ".*\\[#{7} {23}\\] 25%.*")
+        self.assertRegexpMatches(progress_bars[1], ".*\\[#{15} {15}\\] 50%.*")
+        self.assertRegexpMatches(progress_bars[2], ".*\\[#{22} {8}\\] 75%.*")
+        self.assertRegexpMatches(progress_bars[3], ".*\\[#{30}\\] 100%.*")
+
+        self.assertCaretReturnFollowsEach(update_bars)
+        self.assertNewLineFollows(finish_bar)
+
+        self.assertAnyEndsWith(update_bars, self.FOO)
+        self.assertAnyEndsWith(update_bars, self.BAR)
+        self.assertAnyEndsWith(update_bars, self.DATA)
+        self.assertAnyEndsWith(update_bars, self.DATA_SUB)
+
+        self.assertTrue(finish_bar.endswith("Checkout finished!"))
+
+    def filter_out_empty_write_calls(self, calls):
+        def is_not_empty_write(call):
+            assert call[0] == "write"
+            return call[1][0] != ""
+
+        return list(filter(is_not_empty_write, calls))
+
+    def filter_out_non_write_calls(self, calls):
+        def is_write_call(call):
+            return call[0] == "write"
+
+        return list(filter(is_write_call, calls))
+
+    def _prepare_repo(self):
+        storage = self.mkdtemp()
+
+        ret = main(["remote", "add", "-d", "myremote", storage])
+        self.assertEqual(0, ret)
+
+        ret = main(["add", self.DATA_DIR])
+        self.assertEqual(0, ret)
+
+        ret = main(["add", self.FOO])
+        self.assertEqual(0, ret)
+
+        ret = main(["add", self.BAR])
+        self.assertEqual(0, ret)
+
+        ret = main(["push"])
+        self.assertEqual(0, ret)
+
+        shutil.rmtree(self.DATA_DIR)
+        os.unlink(self.FOO)
+        os.unlink(self.BAR)
+
+    def _checkout_and_intercept_std_output(self):
+        with MockLoggerHandlers(
+            logger
+        ), ConsoleFontColorsRemover(), patch.object(
+            sys, "stdout"
+        ) as stdout_mock:
+            self.stdout_mock = logger.handlers[0].stream = stdout_mock
+
+            ret = main(["checkout"])
+            self.assertEqual(0, ret)
+
+    def get_progress_bars(self):
+        pattern = re.compile(".*\\[.{30}\\].*%.*")
+        return [arg for arg in self.write_args if pattern.match(arg)]
+
+    def assertCaretReturnFollowsEach(self, update_bars):
+        for update_bar in update_bars:
+
+            self.assertIn(update_bar, self.write_args)
+
+            for index, arg in enumerate(self.write_args):
+                if arg == update_bar:
+                    self.assertEqual(
+                        progress.CLEARLINE_PATTERN, self.write_args[index + 1]
                     )
-                ),
-            )
+
+    def assertNewLineFollows(self, finish_bar):
+        self.assertIn(finish_bar, self.write_args)
+
+        for index, arg in enumerate(self.write_args):
+            if arg == finish_bar:
+                self.assertEqual("\n", self.write_args[index + 1])
+
+    def assertAnyEndsWith(self, update_bars, name):
+        self.assertTrue(any(ub for ub in update_bars if ub.endswith(name)))
+
+
+class TestCheckoutTargetRecursiveShouldNotRemoveOtherUsedFiles(TestDvc):
+    def test(self):
+        ret = main(["add", self.DATA_DIR, self.FOO, self.BAR])
+        self.assertEqual(0, ret)
+
+        ret = main(["checkout", "-R", self.DATA_DIR])
+        self.assertEqual(0, ret)
+
+        self.assertTrue(os.path.exists(self.FOO))
+        self.assertTrue(os.path.exists(self.BAR))
+
+
+class TestCheckoutRecursiveNotDirectory(TestDvc):
+    def test(self):
+        ret = main(["add", self.FOO])
+        self.assertEqual(0, ret)
+
+        with self.assertRaises(TargetNotDirectoryError):
+            self.dvc.checkout(target=self.FOO, recursive=True)

tests/test_data_cloud.py

@@ -628,8 +628,7 @@ class TestWarnOnOutdatedStage(TestDvc):
         url = get_local_url()
         self.main(["remote", "add", "-d", TEST_REMOTE, url])
 
-        self.dvc.add(self.FOO)
-        stage = self.dvc.run(deps=["foo"], outs=["bar"], cmd="echo bar > bar")
+        stage = self.dvc.run(outs=["bar"], cmd="echo bar > bar")
         self.main(["push"])
 
         stage_file_path = stage.relpath
@@ -675,7 +674,7 @@ class TestRecursiveSyncOperations(TestDataCloudBase):
         )
 
         self.dvc.add(self.DATA)
-        self.dvc.add(self.DATA_SUB_CONTENTS)
+        self.dvc.add(self.DATA_SUB)
 
     def _remove_local_data_files(self):
         os.remove(self.DATA)
@@ -799,15 +798,18 @@ class TestShouldWarnOnNoChecksumInLocalAndRemoteCache(TestDvc):
         with ConsoleFontColorsRemover(), MockLoggerHandlers(logger):
             reset_logger_error_output()
 
-            main(["push"])
+            ret = main(["push"])
+            self.assertEqual(ret, 0)
             self.stderr_contains_message()
 
             reset_logger_error_output()
 
-            main(["pull"])
+            ret = main(["pull", "-f"])
+            self.assertEqual(ret, 0)
             self.stderr_contains_message()
 
             reset_logger_error_output()
 
-            main(["status", "-c"])
+            ret = main(["status", "-c"])
+            self.assertEqual(ret, 0)
             self.stderr_contains_message()

tests/test_import.py

@@ -1,3 +1,5 @@
+import dvc
+
 from dvc.utils.compat import str
 
 import os
@@ -9,7 +11,7 @@ from dvc.exceptions import DvcException
 from dvc.main import main
 from mock import patch, mock_open, call
 from tests.basic_env import TestDvc
-from tests.utils import reset_logger_error_output
+from tests.utils import reset_logger_error_output, spy
 from tests.utils.httpd import StaticFileServer
 from tests.utils.logger import MockLoggerHandlers, ConsoleFontColorsRemover
 
@@ -137,3 +139,22 @@ class TestShouldNotResumeDownload(TestInterruptedDownload):
         m_handle = m()
         expected_calls = [call(b"f"), call(b"o"), call(b"o")]
         m_handle.write.assert_has_calls(expected_calls, any_order=False)
+
+
+class TestShouldRemoveOutsBeforeImport(TestDvc):
+    def setUp(self):
+        super(TestShouldRemoveOutsBeforeImport, self).setUp()
+        tmp_dir = self.mkdtemp()
+        self.external_source = os.path.join(tmp_dir, "file")
+        with open(self.external_source, "w") as fobj:
+            fobj.write("content")
+
+    def test(self):
+        remove_outs_call_counter = spy(dvc.stage.Stage.remove_outs)
+        with patch.object(
+            dvc.stage.Stage, "remove_outs", remove_outs_call_counter
+        ):
+            ret = main(["import", self.external_source])
+            self.assertEqual(0, ret)
+
+        self.assertEqual(1, remove_outs_call_counter.mock.call_count)

tests/test_install.py

@@ -1,10 +1,15 @@
 import os
+import sys
+import unittest
 
 from dvc.main import main
 
 from tests.basic_env import TestDvc
 
 
+@unittest.skipIf(
+    sys.platform == "win32", "Git hooks aren't supported on Windows"
+)
 class TestInstall(TestDvc):
     def test(self):
         ret = main(["install"])
@@ -20,6 +25,8 @@ class TestInstall(TestDvc):
         self.assertTrue(os.path.isfile(hook("pre-commit")))
 
         self.dvc.add(self.FOO)
+        self.dvc.scm.add([".gitignore", self.FOO + ".dvc"])
+        self.dvc.scm.commit("add")
         os.unlink(self.FOO)
 
         self.dvc.scm.checkout("branch", create_new=True)

tests/test_metrics.py

@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 import os
 import json
 
@@ -109,7 +110,7 @@ class TestMetrics(TestDvc):
         )
         self.assertEqual(len(ret), 3)
         for b in ["foo", "bar", "baz"]:
-            self.assertSequenceEqual(ret[b]["metric_hcsv"], "branch\n" + b)
+            self.assertEqual(ret[b]["metric_hcsv"].split(), ["branch", b])
 
     def test_type_case_normalized(self):
         ret = self.dvc.metrics.show(
@@ -167,6 +168,119 @@ class TestMetrics(TestDvc):
         for b in ["foo", "bar", "baz"]:
             self.assertSequenceEqual(ret[b]["metric_hcsv"], [[b]])
 
+    def test_formatted_output(self):
+        with open("metrics.csv", "w") as fobj:
+            # Labels are in Spanish to test unicode characters
+            fobj.write(
+                "valor_mse,desviación_mse,data_set\n"
+                "0.421601,0.173461,entrenamiento\n"
+                "0.67528,0.289545,pruebas\n"
+                "0.671502,0.297848,validación\n"
+            )
+
+        with open("metrics.tsv", "w") as fobj:
+            # Contains quoted newlines to test output correctness
+            fobj.write(
+                "value_mse\tdeviation_mse\tdata_set\n"
+                "0.421601\t0.173461\ttrain\n"
+                '0.67528\t0.289545\t"test\\ning"\n'
+                "0.671502\t0.297848\tvalidation\n"
+            )
+
+        with open("metrics.json", "w") as fobj:
+            fobj.write(
+                "{\n"
+                '     "data_set": [\n'
+                '          "train",\n'
+                '          "testing",\n'
+                '          "validation"\n'
+                "     ],\n"
+                '     "deviation_mse": [\n'
+                '          "0.173461",\n'
+                '          "0.289545",\n'
+                '          "0.297848"\n'
+                "     ],\n"
+                '     "value_mse": [\n'
+                '          "0.421601",\n'
+                '          "0.67528",\n'
+                '          "0.671502"\n'
+                "     ]\n"
+                "}"
+            )
+
+        with open("metrics.txt", "w") as fobj:
+            fobj.write("ROC_AUC: 0.64\nKS: 78.9999999996\nF_SCORE: 77\n")
+
+        self.dvc.run(
+            fname="testing_metrics_output.dvc",
+            metrics_no_cache=[
+                "metrics.csv",
+                "metrics.tsv",
+                "metrics.json",
+                "metrics.txt",
+            ],
+        )
+
+        self.dvc.metrics.modify("metrics.csv", typ="csv")
+        self.dvc.metrics.modify("metrics.tsv", typ="tsv")
+        self.dvc.metrics.modify("metrics.json", typ="json")
+
+        with MockLoggerHandlers(logger.logger):
+            reset_logger_standard_output()
+
+            ret = main(["metrics", "show"])
+            self.assertEqual(ret, 0)
+
+            expected_csv = (
+                u"\tmetrics.csv:\n"
+                u"\t\tvalor_mse   desviación_mse   data_set       \n"
+                u"\t\t0.421601    0.173461         entrenamiento  \n"
+                u"\t\t0.67528     0.289545         pruebas        \n"
+                u"\t\t0.671502    0.297848         validación"
+            )
+
+            expected_tsv = (
+                "\tmetrics.tsv:\n"
+                "\t\tvalue_mse   deviation_mse   data_set    \n"
+                "\t\t0.421601    0.173461        train       \n"
+                "\t\t0.67528     0.289545        test\\ning   \n"
+                "\t\t0.671502    0.297848        validation"
+            )
+
+            expected_txt = (
+                "\tmetrics.txt:\n"
+                "\t\tROC_AUC: 0.64\n"
+                "\t\tKS: 78.9999999996\n"
+                "\t\tF_SCORE: 77\n"
+            )
+
+            expected_json = (
+                "\tmetrics.json:\n"
+                "\t\t{\n"
+                '\t\t     "data_set": [\n'
+                '\t\t          "train",\n'
+                '\t\t          "testing",\n'
+                '\t\t          "validation"\n'
+                "\t\t     ],\n"
+                '\t\t     "deviation_mse": [\n'
+                '\t\t          "0.173461",\n'
+                '\t\t          "0.289545",\n'
+                '\t\t          "0.297848"\n'
+                "\t\t     ],\n"
+                '\t\t     "value_mse": [\n'
+                '\t\t          "0.421601",\n'
+                '\t\t          "0.67528",\n'
+                '\t\t          "0.671502"\n'
+                "\t\t     ]\n"
+                "\t\t}"
+            )
+
+            stdout = logger.logger.handlers[0].stream.getvalue()
+            self.assertIn(expected_tsv, stdout)
+            self.assertIn(expected_csv, stdout)
+            self.assertIn(expected_txt, stdout)
+            self.assertIn(expected_json, stdout)
+
 
 class TestMetricsRecursive(TestDvc):
     def setUp(self):
@@ -206,7 +320,9 @@ class TestMetricsRecursive(TestDvc):
 
         self.assertEqual(ret, 0)
 
-        self.dvc.scm.add(["nested"])
+        self.dvc.scm.add(
+            ["nested", "metric_nested.dvc", "metric_subnested.dvc"]
+        )
         self.dvc.scm.commit("nested metrics")
 
         self.dvc.scm.checkout("master")
@@ -489,7 +605,7 @@ class TestCachedMetrics(TestDvc):
         self.assertIsNotNone(stage)
         self.assertEqual(stage.relpath, "metrics.json.dvc")
 
-        self.dvc.scm.add([".gitignore", "metrics.json.dvc"])
+        self.dvc.scm.add(["code.py", ".gitignore", "metrics.json.dvc"])
         self.dvc.scm.commit(branch)
 
     def _test_metrics(self, func):
@@ -502,6 +618,10 @@ class TestCachedMetrics(TestDvc):
         func("one")
         func("two")
 
+        # TestDvc currently is based on TestGit, so it is safe to use
+        # scm.git for now
+        self.dvc.scm.git.git.clean("-fd")
+
         self.dvc = DvcRepo(".")
 
         res = self.dvc.metrics.show(
@@ -535,3 +655,58 @@ class TestCachedMetrics(TestDvc):
 
     def test_run(self):
         self._test_metrics(self._do_run)
+
+
+class TestMetricsType(TestDvc):
+    branches = ["foo", "bar", "baz"]
+    files = [
+        "metric",
+        "metric.txt",
+        "metric.json",
+        "metric.tsv",
+        "metric.htsv",
+        "metric.csv",
+        "metric.hcsv",
+    ]
+    xpaths = [None, None, "branch", "0,0", "0,branch", "0,0", "0,branch"]
+
+    def setUp(self):
+        super(TestMetricsType, self).setUp()
+        self.dvc.scm.commit("init")
+
+        for branch in self.branches:
+            self.dvc.scm.checkout(branch, create_new=True)
+            with open("metric", "w+") as fd:
+                fd.write(branch)
+            with open("metric.txt", "w+") as fd:
+                fd.write(branch)
+            with open("metric.json", "w+") as fd:
+                json.dump({"branch": branch}, fd)
+            with open("metric.csv", "w+") as fd:
+                fd.write(branch)
+            with open("metric.hcsv", "w+") as fd:
+                fd.write("branch\n")
+                fd.write(branch)
+            with open("metric.tsv", "w+") as fd:
+                fd.write(branch)
+            with open("metric.htsv", "w+") as fd:
+                fd.write("branch\n")
+                fd.write(branch)
+            self.dvc.run(metrics_no_cache=self.files, overwrite=True)
+            self.dvc.scm.add(self.files + ["metric.dvc"])
+            self.dvc.scm.commit("metric")
+
+        self.dvc.scm.checkout("master")
+
+    def test_show(self):
+        for file_name, xpath in zip(self.files, self.xpaths):
+            self._do_show(file_name, xpath)
+
+    def _do_show(self, file_name, xpath):
+        ret = self.dvc.metrics.show(file_name, xpath=xpath, all_branches=True)
+        self.assertEqual(len(ret), 3)
+        for branch in self.branches:
+            if isinstance(ret[branch][file_name], list):
+                self.assertSequenceEqual(ret[branch][file_name], [branch])
+            else:
+                self.assertSequenceEqual(ret[branch][file_name], branch)

tests/test_move.py

@@ -173,13 +173,16 @@ class TestMoveFileBetweenDirectories(TestDvc):
 
 class TestMoveFileInsideDirectory(TestDvc):
     def test(self):
-        ret = main(["add", "data_dir/data"])
+        ret = main(["add", self.DATA])
         self.assertEqual(ret, 0)
 
-        with cd("data_dir"):
-            ret = main(["move", "data", "data.txt"])
+        with cd(self.DATA_DIR):
+            ret = main(["move", os.path.basename(self.DATA), "data.txt"])
             self.assertEqual(ret, 0)
 
-        self.assertFalse(os.path.exists("data_dir/data"))
-        self.assertTrue(os.path.exists("data_dir/data.txt"))
-        self.assertTrue(os.path.exists("data_dir/data.txt.dvc"))
+        self.assertFalse(os.path.exists(self.DATA))
+
+        data_fullpath = os.path.join(self.DATA_DIR, "data.txt")
+        dvc_fullpath = os.path.join(self.DATA_DIR, "data.txt.dvc")
+        self.assertTrue(os.path.exists(data_fullpath))
+        self.assertTrue(os.path.exists(dvc_fullpath))

tests/test_output.py

@@ -23,7 +23,7 @@ class TestOutScheme(TestDvc):
     }
 
     def _get(self, path):
-        return _get(Stage(self.dvc), path, None, None, None)
+        return _get(Stage(self.dvc), path, None, None, None, None)
 
     def test(self):
         for path, cls in self.TESTS.items():

tests/test_pipeline.py

@@ -34,6 +34,10 @@ class TestPipelineShowSingle(TestDvc):
         self.assertEqual(ret, 0)
         self.assertTrue(os.path.isfile(self.dotFile))
 
+    def test_tree(self):
+        ret = main(["pipeline", "show", "--tree", self.stage])
+        self.assertEqual(ret, 0)
+
     def test_ascii_commands(self):
         ret = main(["pipeline", "show", "--ascii", self.stage, "--commands"])
         self.assertEqual(ret, 0)

tests/test_pkg.py

@@ -0,0 +1,37 @@
+import os
+
+from dvc.repo.pkg import Package, PackageParams
+from tests.basic_env import TestDvcPkg
+
+
+class TestPkgBasic(TestDvcPkg):
+    NEW_PKG_DIR = "new_pkg"
+
+    def setUp(self):
+        super(TestPkgBasic, self).setUp()
+
+        self.to_dir = os.path.join(self._root_dir, self.NEW_PKG_DIR)
+        os.mkdir(self.to_dir)
+        self.dvc.install_pkg(
+            PackageParams(self.pkg_fixture._root_dir, self.to_dir)
+        )
+
+    def test_installed_dirs_and_files(self):
+        self.assertTrue(os.path.isdir(self.pkg_dir))
+
+        mod_dir_fullpath = os.path.join(self._root_dir, Package.MODULES_DIR)
+        self.assertTrue(os.path.isdir(mod_dir_fullpath))
+
+        installed_pkg_dir = os.path.join(mod_dir_fullpath, self.GIT_PKG)
+        for file in [self.FOO, self.BAR, self.CODE]:
+            installed_file = os.path.join(installed_pkg_dir, file)
+            self.assertTrue(os.path.isfile(installed_file))
+
+    def test_target_dir_checkout(self):
+        pkg_data_file = os.path.join(self.to_dir, self.DATA)
+        self.assertTrue(os.path.exists(pkg_data_file))
+        self.assertTrue(os.path.isfile(pkg_data_file))
+
+        pkg_data_dir = os.path.join(self.to_dir, self.DATA_SUB_DIR)
+        self.assertTrue(os.path.exists(pkg_data_dir))
+        self.assertTrue(os.path.isdir(pkg_data_dir))