Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #458 from dataversioncontrol/doc_update_1

Browse Source 
Update the doc structure
This commit is contained in:
Dmitry Petrov
2018-02-13 01:30:17 -08:00
committed by GitHub
parent 6cdfaffc0e d11f858cce
commit 71f7b9da0c
1 changed files with 252 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/doc.rst
+252 -138
View File
@@ -1,70 +1,199 @@
========================
Introduction
========================
========
Why DVC?
========
It is hardly possible in real life to develop a good machine learning model in a single pass. ML modeling is an iterative process and it is extremely important to keep track of your steps, dependencies between the steps, dependencies between your code and data files and all code running arguments. This becomes even more important and complicated in a team environment where data scientists’ collaboration takes a serious amount of the team’s effort.
Why? Really.
**Data Version Control** (aka DVC) is designed to help data scientists keep track of their ML processes and file dependencies in the simple form of git-like commands: "dvc run python train_model.py data/train_matrix.p data/model.p". Your existing ML processes can be easily transformed into reproducible DVC pipelines regardless of which programming language or tool was used.
=============================
What is DVC?
=============================
==================
DVC Basic
==================
Data Version Control or DVC is a new type of **experiment management software** that has built on top of source code version control system (currently - Git).
This gives the ability to **use the best engineering practices in data science projects**.
Traditional Makefile based reproducibility
__________________________________________
The underlying source control system **eliminates the need of using external services**.
Experiment sharing and data scientists collaboration can be done by a regular Git tools (such as commit messages, merges, pull requests, code comments) the same way as it works for software engineers.
Traditional approach in software enginering - reproduce everything from code.
This is nice and clean way to build software from source code and it is heavely utilised by Makefile and it's analogs.
DVC implements an **Git experimentation methodology** where each experiment with it's code as well as data is presented as a separate Git branch or commit.
The approach is based on some implicit assumptions:
DVC uses a few concepts:
1. Project consists of a large amount of small source code files.
2. Each code file can be processed (compiled) separately into some object file and this is a fast.
3. The final result (an application file) is a combination of these object files.
4. Building the the entire project form scratch is slow.
- **Experiment** is equal to Git branch. Each experiment (extract new features, change model hyperparameters, clean noise in data, add a new data source) should be performaed in a separate branch and then merged into the master branch if the experiment was successfull. DVC manages to NOT recompute the results after such merges.
Makefile and it's analogs optimize the entire project building stage (5) by identifying changed files and quickly processing only this small subset of files (4).
Because of (5)
- **Experiment state** or state is equal to Git snapshot (all commited files). Git checksum, a branch name or a tag can be used as a reference to an experiment state.
4. It is easy to derive what was changed from the last reproduction (last make run).
5. If only a few files were changed it is easy to rebuild only this subset of files and build a final result.
- **Reproducability** - an action to reproduce an experiment state. The action generates output files based on a set of input files and source code. This action usually changes experiment state.
Makefile tool and it's analogs do a good job in recognizing the small changes (step 4), rebuilding small parts of the project and constrcting them together into a single result.
- **Pipeline** - directed acyclic graph (DAG) of commands to reproduce an experiment state. The commands are connected by input and output files. Pipeline is defined by special **dvc-files** (like Makefiles).
- **Workflow** - set of experiments and relationships between them. Workflow corresponds to the entire Git repository.
- **Data files** - cached files (for large files). For data files the file content is stored outside of Git repository in a local hard drive but data file meta information is stored in Git for DVC needs (to maintain pipilines and reproducibility).
- **Data cache** is a directory with all data files content in a local hard drive or a cloud storage but not in Git repository.
- **Cloud storage** support is a compliment to the core DVC features. This is how a data scientist transfers large data files or shares a trained on GPU model to whom who does not have GPU.
Data science project reproducibility
____________________________________
=============
Core features
=============
Machine learning (ML) modleing process is slightly different from software engineering projects.
1. DVC works **on top of Git repositories** and has a similar command line interface and Git workflow.
1. [Dependencies tree is deep, not wide] To get a final result you need to go through a process with many dependent steps or stages. This is ML pipeline.
2. [Many steps are slow] Long processing time for many of operations: data cleaning is slow because of size of input raw files, model training is slow because of advanced ML algorithms.
3. []
2. It makes data science projects **reproductable** by creating lightweight pipelines (DAG).
IN PROGRESS...
3. **Large data file versioning** by creating pointers in your Git repository into a data cache in a local hard drive.
4. **Programming language agnostic**: Python, R, Julia, shell scripts, etc. ML library agnostic: Keras, Tensorflow, PyTorch, scipy, etc.
5. **Open sourced** and **Self-served**. DVC is free and does not require any additional services.
6. DVC supports cloud storges (AWS S3 and GCP storage) for **data and pretrained models sharing**.
Two reproducibility philosophies
________________________________
====================
Related technologies
====================
There are two different reproducibility "philosophies":
* Versioning only code.
* Versioning code and data.
Due to the the novelty of this approach DVC can be understood better in comparision to the existing technologies and ideas.
Makefile versions only code.
DVC combines numbers of existing technologies and ideas into a single product with a goal of bringing the best engineering practices into data sciance process. This comparision can give you good understanding of DVC if you are familliar with some of these technologies.
1. **Git** is an underlying technology for DVC. Difference:
- DVC extends Git by concept of *data files* - large files that should NOT be stored in Git repository but still have to be tracked and versioned. See more details in Git-annexand Git-LFS items below.
2. **Workflow management tools** (pipelines and DAGs): Apache Airflow, Luigi and etc. Differences:
- Comparing to these data engineering technologies, DVC is focused on data science and modeling. As a result, DVC pipelines are lightweight, easy to create and modify. However, DVC lacks of pipeline execution features like execution monitoring and execution error handling and recovering.
- DVC is a purly command line tool that does not have user interface and does not run any servers. Nevertheless, DVC can generate images with pipeline and experiment workflow visualization.
3. Today **experiment management** software are mostly designed for enterprise usage. Open sourced experimentation tool example: http://studio.ml/. Differences:
- DVC uses Git as underlying platform for experiment tracking instead of a web application.
- There is no need to run any services. No user interface as a result. Nevertheless, we expect some UI sevices will be created on top of DVC.
- DVC has transparent design: dvc-files, meta files, state file, cache dirs have a simple (usually text) format and can be easily reused by external tools.
4. **Git workflows** and Git usage methodologies such as Gitflow. Differences:
- DVC supports a new experimentation methodology (and Git workflow) when a separate branch should be created for each experiment with unpredictable results with subsequent merge of this branch if the result was successful. An experiment can have multiple sub experiments.
- One of the major DVC innovation is the ability to navigate through the experiments without recomputing them.
______________________
DVC is a command line tool that works on top of an existing Git repository.
5. DVC uses the **Makefile** (and it's analogues) idea of defining DAG by a special files. Differences:
Using traditional UNIX-tool terminology DVC might be treated as a Makefile for data projects which
1. DVC never rebuilds a target if it was already built
2. DVC
- The DAG definition:
- The DAG is defined by dvc-files with filenames *Dvcfile* or *<filename>.dvc*, not *Makefile*.
- One dvc-file defines one node in DAG. All dvc-files in a repository make up a single pipeline (think a single Makefile). All dvc-files (and corresponded pipeline commands) are imlicitly combined through theirs inputs and outputs. This file separation is done to simplify conflict resolving during Git merges.
- DVC provide a simple command *dvc run CMD* to generate dvc file automaticaly based on the provided command, dependencies and outputs.
- File tracking:
- DVC tracks files based on checksum (md5) instead of file timesteps. This helps to avoid of running heavy process like model re-training when you checkout a previous, already trained version of a modeling code (Makefile will retrain a model).
- Despite the existence of file checksum tracking, DVC still uses the files timestemps and inodes and keep them in a local state file for optimization. Otherwise, each reproduction command will require recomputing all dependency files checksum - unappropriate approach for large files (10Gb+).
========================
Getting Started with DVC
========================
6. **Git-annex**. DVC uses the idea of storing the content of large files (that you don't what to see in your Git repository) in a local key-value store and use file symlinks insted of the actual files. Difference:
- DVC uses hardlinks instead of symlinks to make user experience better.
- DVC optimizes checksum calculation (see Makefile differences from the above).
- DVC stores data file mata information in Git repository *.dvc/*, not in the Git tree *.git/annex/*. As a result, all meta information can be shared through any Git server like Github (Git-annex loses all meta information when shared by Git server).
7. **Git-LFS** (Large File Storage) interface is much user friendly comparing to Git-annex. It requires a very limited number of additional command. DVC uses this methodology - native Git like interface with a couple additional file related commands. Differences:
- DVC is fully compatable with Git. It does not required special Git servers like Git-LFS does.
- DVC does not add any hooks to Git by default. To checkout data files *dvc checkout* command has to be run after each *git checkout* and *git clone* command.
- *dvc checkout* command does not actually copy data files from cache to the working tree because of the file copying command is a heavy operation for large files (30Gb+). DVC creates hardlinks instead and changes data file permissions to read only.
==============
How it works?
==============
1. DVC is a command line tool that works on top of Git::
$ cd my_git_repo
$ dvc init
2. It helps define pipelines of your commands and keeps all the commands and the dependencies (DAG) in Git repository::
$ dvc run -d input.csv -o results.csv python cnn_train.py --seed 20180227 --epoch 20 input.csv result.csv
$ git add result.csv
$ git commit -m 'Train CNN. 20 epochs.'
3. DVC is programming language agnostic. R command example::
$ dvc run -d result.csv -o plots.jpg Rscript plot.R result.csv plots.jpg
$ git add plots.jpg
$ git commit -m 'CNN plots'
4. Reproduce a pipeline with respect to dependencies::
# The input dataset was changed
$ dvc repro plots.jpg
Reproducing 'output.p':
python cnn_train.py --seed 20180227 --epoch 20 input.csv output.p
Reproducing 'plots.jpg':
Rscript plot.R result.csv plots.jpg
5. DVC introduces data files concept for Git repository. DVC keeps data files outside of Git repository but meta data in Git::
$ git checkout a03_normbatch_vgg16 # checkout code and DVC meta data
$ dvc checkout # checkout data files from the local cache (not Git)
$ ls -l data/ # These LARGE files were copied from DVC cache, not from Git
total 1017488
-r-------- 2 501 staff 273M Jan 27 03:48 Posts-test.tsv
-r-------- 2 501 staff 12G Jan 27 03:48 Posts-train.tsv
6. Reproductable DVC repository with DVC meta information (DAG) can be easily shared by any Git server::
$ git clone https://github.com/dataversioncontrol/myrepo.git
$ cd myrepo
# Reproduce data files
$ dvc repro
Reproducing 'output.p':
python cnn_train.py --seed 20180227 --epoch 20 input.csv output.p
Reproducing 'plots.jpg':
Rscript plot.R result.csv plots.jpg
7. DVC local cache can be transfered to your collegues and parthners through AWS S3 or GCP Storage::
$ git push
$ dvc push # push the data cache to your cloud bucket
# On a collegue machine:
$ git clone https://github.com/dataversioncontrol/myrepo.git
$ cd myrepo
$ git pull # get the data cache from cloud
$ dvc checkout # checkout data files
$ ls -l data/ # You just got gigabites of data though Git and DVC:
total 1017488
-r-------- 2 501 staff 273M Jan 27 03:48 Posts-test.tsv
8. DVC works on Mac, Linux and Windows systems. Windows example::
$ dir
?????
????
==============================
Getting Started with DVC (old)
==============================
To show DVC in action let's play with an actual machine learning (ML) scenario.
This is going to be a natural language processing (NLP) problem of predicting tags for given
@@ -190,6 +319,86 @@ If you like to pull the latest version of DVC from the master branch in its repo
This command will automatically upgrade your DVC version in case it is behind the latest version in the master branch of the github repo.
=============
Configuration
=============
Once you install DVC, you should be able to start using it (in its local setup) immediately.
However, you can optionally proceed to further configure DVC (especially if you intend to use it in a Cloud-based scenario).
DVC Files and Directories
=========================
Once installed, dvc will populate its installation folder (hereinafter referred to as .dvc) with essential shared and internal files and folders will be stored
* **.dvc/config** - This is a configuration file.
The config file can be edited directly or indirectly using command **dvc config NAME VALUE**.
* **.dvc/cache** - the cache directory will contain your data files (the data directories of DVC repositories will only contain symlinks to the data files in the global cache).
**Note:** DVC includes the cache directory to **.gitignore** file during the initilization. And no data files (with actual content) will be pushed to Git repository,
only data file symlinks and commands to reproduce them.
* **.dvc/state** - this file is ceated for optimization. The file contains data files checksum and timestemps.
Working with Cloud Data Storages
======================================================
Using DVC with Cloud-based Data Storages is an optional feature.
By default, DVC is configured to use local data storage only (.dvc/cache directory),
and it enables basic DVC usage scenarios out of the box.
DVC can use cloud storages as a common file storage.
With cloud storage you might use models and data file which were created by your team members
without spending time and resources for re-building models and re-processing data files.
As of this version, DVC supports two types of cloud-based data storage providers:
* **AWS** - Amazon Web Services
* **GCP** - Google Cloud Provider
The subsections below explain how to configure DVC to use of the data cloud storages above.
Using AWS Cloud
---------------
For using AWS as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively
* **Cloud = AWS** in *Global* section.
* **StoragePath = /mybucket/dvc/tag_classifier** in **AWS** section - path to a cloud storage bucket and directory in the bucket.
* **CredentialPath = ~/aws/credentials** in **AWS** section - path to AWS credentials in your local machine (AWS cli command line tools creates this directory).
In Mac, default value is *~/.aws/credentials*, and it is *%USERPATH%/.aws/credentials* in Windows
**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to AWS"**
Instead of manual file modification we recommend to run corresponded commands::
$ dvc config Global.Cloud AWS # This step is not needed for new DVC repositories
$ dvc config AWS.StoragePath /mybucket/dvc/tag_classifier
$ dvc config AWS.CredentialPath ~/.aws/credentials # Not needed if aws cli is instelled to default path
$ dvc config AWS.CredentialSection default # Not needed if you have only one AWS account
$ git commit -am "Change cloud to AWS"
Using Google Cloud
------------------
For using GCP (Google Cloud Provider) as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively
* **Cloud = GCP** in *Global* section.
* **StoragePath = /mybucket/dvc/tag_classifier** in GCP section - this option has the same meaning as AWS one above. Run **dvc config GCP.StoragePath /my/path/to/a/bucket**
* **ProjectName = MyCloud** - a GCP specific project name.
**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to GCP"**
Instead of manual file modification we recommend to run corresponded commands::
$ dvc config Global.Cloud GCP
$ dvc config GCP.StoragePath /mybucket/dvc/tag_classifier
$ dvc config GCP.ProjectName MyCloud
$ git commit -am "Change cloud to AWS"
==================
Using DVC Commands
==================
@@ -755,98 +964,3 @@ DVC can benefit from parallel processing and multiple processors/cores available
The number of DVC jobs is 5 by default. In case you like to change it to any other reasonable value, you use *-j (--jobs)* option in DVC commands where it is applicable.
=============
Configuration
=============
Once you install DVC, you should be able to start using it (in its local setup) immediately.
However, you can optionally proceed to further configure DVC (especially if you intend to use it in a Cloud-based scenario).
DVC Files and Directories
=========================
Once installed, dvc will populate its installation folder (hereinafter referred to as .dvc) with essential shared and internal files and folders will be stored
* **.dvc/config** - This is a configuration file.
The config file can be edited directly or indirectly using command **dvc config NAME VALUE**.
* **.dvc/cache** - the cache directory will contain your data files (the data directories of DVC repositories will only contain symlinks to the data files in the global cache).
**Note:** DVC includes the cache directory to **.gitignore** file during the initilization. And no data files (with actual content) will be pushed to Git repository,
only data file symlinks and commands to reproduce them.
* **.dvc/state** - this file is ceated for optimization. The file contains data files checksum and timestemps.
Working with Cloud Data Storages
======================================================
Using DVC with Cloud-based Data Storages is an optional feature.
By default, DVC is configured to use local data storage only (.dvc/cache directory),
and it enables basic DVC usage scenarios out of the box.
DVC can use cloud storages as a common file storage.
With cloud storage you might use models and data file which were created by your team members
without spending time and resources for re-building models and re-processing data files.
As of this version, DVC supports two types of cloud-based data storage providers:
* **AWS** - Amazon Web Services
* **GCP** - Google Cloud Provider
The subsections below explain how to configure DVC to use of the data cloud storages above.
Using AWS Cloud
---------------
For using AWS as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively
* **Cloud = AWS** in *Global* section.
* **StoragePath = /mybucket/dvc/tag_classifier** in **AWS** section - path to a cloud storage bucket and directory in the bucket.
* **CredentialPath = ~/aws/credentials** in **AWS** section - path to AWS credentials in your local machine (AWS cli command line tools creates this directory).
In Mac, default value is *~/.aws/credentials*, and it is *%USERPATH%/.aws/credentials* in Windows
**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to AWS"**
Instead of manual file modification we recommend to run corresponded commands::
$ dvc config Global.Cloud AWS # This step is not needed for new DVC repositories
$ dvc config AWS.StoragePath /mybucket/dvc/tag_classifier
$ dvc config AWS.CredentialPath ~/.aws/credentials # Not needed if aws cli is instelled to default path
$ dvc config AWS.CredentialSection default # Not needed if you have only one AWS account
$ git commit -am "Change cloud to AWS"
Using Google Cloud
------------------
For using GCP (Google Cloud Provider) as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively
* **Cloud = GCP** in *Global* section.
* **StoragePath = /mybucket/dvc/tag_classifier** in GCP section - this option has the same meaning as AWS one above. Run **dvc config GCP.StoragePath /my/path/to/a/bucket**
* **ProjectName = MyCloud** - a GCP specific project name.
**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to GCP"**
Instead of manual file modification we recommend to run corresponded commands::
$ dvc config Global.Cloud GCP
$ dvc config GCP.StoragePath /mybucket/dvc/tag_classifier
$ dvc config GCP.ProjectName MyCloud
$ git commit -am "Change cloud to AWS"
===============
Further Reading
===============
If you are interested in more information about technical aspects of using DVC in your machine learning projects, you can review technical tutorials below
* Data Version Control beta release: iterative machine learning: https://blog.dataversioncontrol.com/data-version-control-beta-release-iterative-machine-learning-a7faf7c8be67 (it exemplifies a practical case study explaining how to use DVC in a Python-based machine learning project)
* R code and reproducible model development with DVC: https://blog.dataversioncontrol.com/r-code-and-reproducible-model-development-with-dvc-1507a0e3687b (it exemplifies a practical case study explaining how to use DVC in an R-based machine learning project)
* ML Model Ensembling with Fast Iterations: https://blog.dataversioncontrol.com/ml-model-ensembling-with-fast-iterations-91e8cad6a9b5
If you are interested in a further reading about conceptual impact of DVC on data scientist productivity and data science-to-DevOps convergence, you are welcome to review the articles below
* How A Data Scientist Can Improve His Productivity: https://blog.dataversioncontrol.com/how-a-data-scientist-can-improve-his-productivity-730425ba4aa0
* Data Version Control in Analytical DevOps Paradigm: https://blog.dataversioncontrol.com/data-version-control-in-analytics-devops-paradigm-35a880e99133