diff --git a/docs/doc.html b/docs/doc.html deleted file mode 100644 index b6e24028a..000000000 --- a/docs/doc.html +++ /dev/null @@ -1,559 +0,0 @@ - - - - - - - - - - - - - - Data Version Control (DVC) - Product Documentation - - - - - - -
- -
-

Getting Started with DVC

-

It is hardly possible in real life to develop a good machine learning model in a single pass. ML modeling is an iterative process and it is extremely important to keep track of your steps, dependencies between the steps, dependencies between your code and data files and all code running arguments. This becomes even more important and complicated in a team environment where data scientists’ collaboration takes a serious amount of the team’s effort.

-

Data Version Control application (aka DVC) is a tool that can help you to address such challenges while increasing your every-day productivity.

-

DVC an open source tool for data science projects. DVC makes your data science projects reproducible by automatically building data dependency graph (DAG). Your code and the dependencies could be easily shared by Git, and data - through cloud storage (AWS S3, GCP) in a single DVC environment.

-

It is quite easy to integrate DVC in your existing ML pipeline/processes without any significant effort to re-implement your ML code/application.

-

The one thing to wrap your head around is that DVC automatically derives the dependencies between the steps and builds the dependency graph (DAG) transparently to the user. This graph is used for reproducing parts of your pipeline which were affected by recent changes.

-

Not only can DVC streamline your work into a single, reproducible environment, it also makes it easy to share this environment by Git including the dependencies (DAG) — an exciting collaboration feature which gives the ability to reproduce the research results in different computers. Moreover, you can share your data files through cloud storage services like AWS S3 or Google Cloud Project (GCP) Storage since DVC does not push data files to Git repositories.

-

Note: If you are interested in reading more about practical aspects of using DVC in machine learning projects, please refer to the tutorials and articles collected in ‘Further Reading’ section. There are articles there walking you through the end-to-end implementation of DVC-based pipeline for Python- and R-based ML projects.

-

Installation

-

Sections below will walk you through the installation of DVC on your computer.

-

Installing DVC with an Installation Package

-

There are well-formed DVC installation packages available for Mac OS, Linux and Windows platforms. You can download them at https://github.com/dataversioncontrol/dvc/releases/tag/0.8.7.2-travis-macos-release-test

-

Once you start the installation package, it will walk you through DVC setup. You will have to just follow the installation instructions.

-

Installing DVC with Python pip

-

Another option to deploy DVC to your machine is to use its standard Python pip package.

-

Pre-requisites and Dependencies

-

Before you start installation of DVC using pip, please make sure the following software applications are installed on your machine

- -

Below are OS-specific notes as for the command-line C++ compiler

- -

Installation with pip

-

When you install DVC on your local machine for the first time, go to your command line prompt and type the command below:

-
pip install dvc
-

Note: if you use the special data science-centric Python environment provided by Anaconda, you can use the above-mentioned command there as well. It will work in Anaconda’s command prompt tool. As of the moment, DVC does not provide a special installation package for a native Anaconda package manager (that is, conda).

-

Installing the Development Version of DVC

-

If you like to pull the latest version of DVC from the master branch in its repo at github, you execute the following command in your command prompt:

-
pip install git+git://github.com/dataversioncontrol/dvc
-

This command will automatically upgrade your DVC version in case it is behind the latest version in the master branch of the github repo.

-

Note: if you use the special data science-centric Python environment provided by Anaconda, you can use the above-mentioned command there as well. It will work in Anaconda’s command prompt tool. As of the moment, DVC does not provide a special installation package for a native Anaconda package manager (that is, conda).

-

Configuration

-

Once you install DVC, you should be able to start using it (in its local setup) immediately.

-

However, you can optionally proceed to further configure DVC (especially if you intend to use it in a Cloud-based scenario).

-

DVC Files and Directories

-

Once installed, dvc will populate its installation folder (hereinafter referred to as .dvc) with essential shared and internal files and folders will be stored

- -

Configuring DVC to Work with Cloud-based Data Storages

-

Note: Using DVC with Cloud-based Data Storages is an optional feature. By default, DVC is configured to use local data storage, and it enables basic DVC usage scenarios out of the box.

-

If your organization or team uses a cloud storage for your data, you can leverage DVC cloud storage integration capabilities.

-

As of this version, DVC supports two types of cloud-based data storage providers * AWS * Google Cloud (GC)

-

The subsections below explain how to configure DVC to use of the data cloud storages above.

-

Using AWS as a Cloud Data Storage for DVC

-

If you decide to use AWS as a data cloud storage for your DVC repositories, you should update dvc.conf options respectively

- -

In AWS section of dvc.conf, specify essential details about your AWS data storage as follows

- -

Once you save the above-mentioned changes to dvc.conf, your instance of DVC will be ready to work with AWS as a cloud data storage.

-

Note: the current version of DVC uses cloud for data storage purposes only. It does not use it for computations.

-

Using Google Cloud as a Cloud Data Storage for DVC

-

If you decide to use AWS as a data cloud storage for your DVC repositories, you should update dvc.conf options respectively

- -

Specify additional values in GC section of dvc.conf as follows

- -

Once you save the above-mentioned changes to dvc.conf, your instance of DVC will be ready to work with Google Cloud as a cloud data storage.

-

Note: the current version of DVC uses cloud for data storage purposes only. It does not use it for computations.

-

Using DVC Commands

-

Since DVC is a command-line application, the appropriate method of use of DVC commands is essential to properly utilize it to the benefit of your machine learning projects.

-

The typical method of use of DVC is as follows

- -

Note: please refer to “Further Reading” section to see in-depth articles and tutorials on the end-to-end ML pipeline setup with DVC for real-world ML projects in Python and R.

-

DVC Commands Cheat Sheet

-

Below is the quick summary of the most important commands of DVC

- -

DVC Command Reference

-

Init Command

-

This command initializes a local DVC environment (repository) in a local directory on your machine.

-

Note: such a directory should contain either a local git repo or a remote git repo clone.

-
usage:
-
-dvc init [-h] [-q] [-v] [-G] [--data-dir DATA_DIR]
-    [--cache-dir CACHE_DIR] [--state-dir STATE_DIR] 
-
-optional arguments:
-    -h, --help             show this help message and exit
-    -q, --quiet            Be quiet.
-    -v, --verbose          Be verbose.
-    -G, --no-git-actions   Skip all git actions including reproducibility check and commits.
-    --data-dir DATA_DIR    Data directory.
-    --cache-dir CACHE_DIR  A well-formed path to the Cache directory.
-    --state-dir STATE_DIR  A well-formed path to the State directory.
-

Examples

-

Initializing DVC repository in the current directory:

-
dvc init
-

Requesting help about using dvc init command:

-
dvc init -h
-

Run command

-

This command executes an OS command (command-line utility) on your local machine. It is often used to execute the steps in your ML pipeline, for instance * Running a python or R script * Running a database SQL script * Etc.

-
usage: dvc run [-h] [-q] [-v] [-G] [--stdout STDOUT] [--stderr STDERR]
-           [-i INPUT] [-o OUTPUT] [-c CODE] [--shell] [-l]
-           command [args]
-
-positional arguments:
-    command     Command to execute
-    args        Arguments of a command (optional; it can be a list of the command-line arguments of command separated by spaces)
-
-optional arguments:
-    -h, --help                   show this help message and exit
-    -q, --quiet                  Be quiet.
-    -v, --verbose                Be verbose.
-    -G, --no-git-actions         Skip all git actions including reproducibility check and commits.
-    --stdout STDOUT              Output std output to a file.
-    --stderr STDERR              Output std error to a file.
-    -i INPUT, --input INPUT      Declare input data items for reproducible cmd.
-    -o OUTPUT, --output OUTPUT   Declare output data items for reproducible cmd.
-    -c CODE, --code CODE         Code dependencies which produce the output.
-    --shell                      Shell command
-    -l, --lock                   Lock data item - disable reproduction.
-

Examples

-

Get help for Run command:

-
dvc run -h
-

Execute a Python script as a DVC ML pipeline step:

-
# Train ML model out of the training dataset. 20170426 is another seed value.
-dvc run python code/train_model.py data/matrix-train.p 20170426 data/model.p
-

Note: In this example, the external command is Python (Python runtime). code/train_model.py is the Python script to be executed by Python. data/matrix-train.p, 20170426, and data/model.p are command-line arguments that are passed to code/train_model.py script.

-

Execute an R script as a DVC ML pipeline step:

-
dvc run Rscript code/parsingxml.R data/Posts.xml data/Posts.csv
-

Note: In this example, the external command is Rscript (R runtime script execution utility). code/parsingxml.R is the R script to be executed by Rscript. data/Posts.xml and data/Posts.csv are command-line arguments that are passed to code/parsingxml.R script.

-

Extract an XML file from an archive to data subfolder:

-
dvc run tar zxf data/Posts.xml.tgz -C data/
-

Sync command

-

This command synchronizes data file with the cloud storage (the cloud settings should be specified in dvc.conf prior to running this command).

-
usage: dvc sync [-h] [-q] [-v] [-G] [-j JOBS] targets [targets ...]
-
-positional arguments:
-targets               File or directory to sync.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check and commits.
-    -j JOBS, --jobs JOBS  Number of jobs to run simultaneously.
-

Note: this command is deprecated and it is going to be removed in one of the next releases of DVC. Technically, running dvc sync is equivalent to running a sequence of dvc pull and dvc push commands.

-

Pull Command

-

This command pulls data from the cloud storage you configured for DVC.

-
usage: dvc pull [-h] [-q] [-v] [-G] [-j JOBS] targets [targets ...]
-
-positional arguments:
-    targets               File or directory to sync.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check and commits.
-    -j JOBS, --jobs JOBS  Number of jobs to run simultaneously.
-

Examples

-

Get help for Pull command:

-
dvc pull -h
-

Push Command

-

This command pushes data files to the cloud storage you configured for DVC.

-
usage: dvc push [-h] [-q] [-v] [-G] [-j JOBS] targets [targets ...]
-
-positional arguments:
-    targets               File or directory to sync.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check
-                          and commits.
-    -j JOBS, --jobs JOBS  Number of jobs to run simultaneously.
-

Examples

-

Get help for Push command:

-
dvc push -h
-

Status Command

-

This command shows status for data files in the DVC repository

-
usage: dvc status [-h] [-q] [-v] [-G] [-j JOBS] targets [targets ...]
-
-positional arguments:
-    targets               File or directory to sync.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check
-                          and commits.
-    -j JOBS, --jobs JOBS  Number of jobs to run simultaneously.
-

Examples

-

Get help for Status command:

-
dvc status -h
-

Get status of data in training.csv file:

-
dvc status data/training.csv
-

Repro Command

-

This command reproduces the that part of the ML pipeline that is dependent on the data or code file targeted by it.

-
usage: dvc repro [-h] [-q] [-v] [-G] [-f] [-s] [target [target ...]]
-
-positional arguments:
-    target                Data items to reproduce.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check and commits.
-    -f, --force           Reproduce even if dependencies were not changed.
-    -s, --single-item     Reproduce only single data item without recursive dependencies check.
-

Examples

-

Get help for Repro command:

-
dvc repro -h
-

Reproduce the part of the pipeline where training.csv data file is involved, forcing reproduce:

-
dvc repro data/training.csv -f
-

Remove Command

-

This command removes a data item from the data directory of a DVC repository.

-
usage: dvc remove [-h] [-q] [-v] [-G] [-l] [-r] [-c] [target [target ...]]
-
-positional arguments:
-    target                Target to remove - file or directory.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check
-                          and commits.
-    -l, --keep-in-cloud   Do not remove data from cloud.
-    -r, --recursive       Remove directory recursively.
-    -c, --keep-in-cache   Do not remove data from cache.
-

Examples

-

Get help for Remove command:

-
dvc remove -h
-

Remove training.csv data file from the DVC repository:

-
dvc remove data/training.csv
-

Import Command

-

This command imports a new data file to the data directory of the DVC repository.

-
usage: dvc import [-h] [-q] [-v] [-G] [-l] [-j JOBS] [-c]
-           input [input ...] output
-
-positional arguments:
-    input             Input file/files.
-    output            Output file/directory.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check and commits.
-    -l, --lock            Lock data item - disable reproduction.
-    -j JOBS, --jobs JOBS  Number of jobs to run simultaneously.
-    -c, --continue        Resume downloading file from url
-

Examples

-

Get help for Import command:

-
dvc import -h
-

Download a file and put to data/ directory:

-
dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz data/
-

Lock Command

-

This command is used to

- -
usage: dvc lock [-h] [-q] [-v] [-G] [-l] [-u] [files [files ...]]
-
-positional arguments:
-    files                 Data items to lock or unlock.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check
-                          and commits.
-    -l, --lock            Lock data item - disable reproduction.
-    -u, --unlock          Unlock data item - enable reproduction.
-

Examples

-

Get help for Lock command:

-
dvc lock -h
-

Lock data/Posts.xml file:

-
dvc lock data/Posts.xml
-

Unlock a previously locked data/Posts.xml file:

-
dvc lock -u data/Posts.xml
-

Notes

- -

gc Command

-

This command collects the garbage in DVC environment. It is especially important when you work with large data files. Under such a condition, keeping previous versions of the large files may slow down performance/drain the disk quota thus swift removing of unnecessary files will be beneficial.

-
usage: dvc gc [-h] [-q] [-v] [-G] [-l] [-r] [-c] [target [target ...]]
-
-positional arguments:
-    target                Target to remove - file or directory.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check and commits.
-    -l, --keep-in-cloud   Do not remove data from cloud.
-    -r, --recursive       Remove directory recursively.
-    -c, --keep-in-cache   Do not remove data from cache.
-

Examples

-

Get help for gc command:

-
dvc gc -h
-

Remove all versions of data/Posts.xml file (but the latest one) from the local cache directory but keep it in the cloud storage:

-
dvc gc data/Posts.xml --keep-in-cloud
-

Target Command

-

This command sets the default target for the current DVC repository.

-
usage: dvc target [-h] [-q] [-v] [-G] [-u] [target_file]
-
-positional arguments:
-    target_file           Target data item.
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check and commits.
-    -u, --unset           Reset target.
-

Examples

-

Get help for Target command:

-
dvc target -h
-

Set data/Posts.xml file as a default target in the current DVC repository:

-
dvc target data/Posts.xml
-

ex Command

-

This command is designed for risky enthusiasts who would like to try the newest capabilities of DVC which are still under active development.

-

Note: It is provided for your reference and early try only. DVC development team does not provide any warranty as for this piece of DVC functionality to work in a stable manner, in your environment. We do not recommend you to use the experimental functionality in production mode unless you really clear as for what you are going to do.

-
usage: dvc ex [-h] [-q] [-v] [-G] {cloud} ...
-
-positional arguments:
-    {cloud}             Use dvc cloud CMD --help for command-specific help
-    cloud               Cloud manipulation
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check
-                          and commits.
-

Examples

-

Display help for experimental commands in DVC:

-
dvc ex -h
-

Config Command

-

This command is designed to overwrite some configuration options for just this session of DVC (as you remember, default configuration values are specified in dvc.conf , which is located in root of your DVC installation folder).

-
usage: dvc config [-h] [-q] [-v] [-G] [-u] name [value]
-
-positional arguments:
-    name                  Option name
-    value                 Option value
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check
-                          and commits.
-    -u, --unset           Unset option
-

Examples

-

Overwrite the value of DataDir configuration option with etc/data for a current dvc session only:

-
dvc config DataDir etc/data
-

Show Command

-

This command is used to display either pipeline or workflow image for your current ML project managed by DVC

-
usage: dvc show [-h] [-q] [-v] [-G] {pipeline,workflow} ...
-
-positional arguments:
-    {pipeline,workflow}   Use dvc show CMD --help for command-specific help
-    pipeline              Show pipeline image
-    workflow              Show workflow image
-
-optional arguments:
-    -h, --help            show this help message and exit
-    -q, --quiet           Be quiet.
-    -v, --verbose         Be verbose.
-    -G, --no-git-actions  Skip all git actions including reproducibility check
-                          and commits.
-

Examples

-

Show the workflow image for the ML project in your current DVC repository:

-
dvc show workflow
-

Show the pipeline image for the ML project in your current DVC repository:

-
dvc show pipeline
-

Notes on Optional Arguments to DVC Commands

-

Common Options

-

As you can see, there are four optional arguments that are applicable to any DVC command. These are

-
-h, --help            show this help message and exit
--q, --quiet           Be quiet.
--v, --verbose         Be verbose.
--G, --no-git-actions  Skip all git actions including reproducibility check and commits.
-

Although these optional arguments are pretty self-explanatory, there is a note on DVC and Git commands used together.

- -

Number of DVC Jobs

-

DVC can benefit from parallel processing and multiple processors/cores available on your machine. It can spin a number of jobs to run in parallel.

-

The number of DVC jobs is 5 by default. In case you like to change it to any other reasonable value, you use -j (--jobs) option in DVC commands where it is applicable.

-

Further Reading

-

If you are interested in more information about technical aspects of using DVC in your machine learning projects, you can review technical tutorials below

- -

If you are interested in a further reading about conceptual impact of DVC on data scientist productivity and data science-to-DevOps convergence, you are welcome to review the articles below

- - - -