From a1c4f93a0208a1d8d9de30463ca6aa26fc651009 Mon Sep 17 00:00:00 2001 From: Tom Molinari Date: Sun, 18 Feb 2018 18:27:29 -0500 Subject: [PATCH] first pass edits for content, spelling, grammar to doc.rst --- docs/doc.rst | 434 +++++++++++++++++++++++++-------------------------- 1 file changed, 215 insertions(+), 219 deletions(-) diff --git a/docs/doc.rst b/docs/doc.rst index 077e1e9f6..aae14cb7b 100644 --- a/docs/doc.rst +++ b/docs/doc.rst @@ -2,84 +2,84 @@ Collaboration issues in data science ==================================== -Today with all the successes in machine learning (ML), deep learning specifically and its applications in business the data science community is still lacking good practices of organizing theirs ML projects and effectively collaborating within ML projects. -This is the biggest challenge for the community and the industry now when ML algorithms and methods are not "tribal knowledge" anymore but it is still not easy to implement, reuse and manage. +Even with all the successes today in machine learning (ML), specifically deep learning and its applications in business, the data science community is still lacking good practices for organizing their projects and effectively collaborating across their varied ML projects. +This is a massive challenge for the community and the industry now, when ML algorithms and methods are no longer simply "tribal knowledge" but are still difficult to implement, reuse, and manage. -To make a progress in this challange many areas of the ML modeling process have to be formalized and many common questions have to be answered in an unified way: +To make progress in this challenge, many areas of the ML experimentation process need to be formalized. Many common questions need to be answered in an unified, principled way: 1. Source code and data versioning. -- How to avoid the discrepancy between versions of source code files and versions of data files when the data files do not fit to a code repository? +- How do you avoid any discrepancies between versions of the source code and versions of the data files when the data cannot fit into a repository? 2. Experiment time log. - -- Which of the model hyperparameters changes contributed the most to the target metric? What was exact contribution value? + +- How do you track which of the hyperparameter changes contributed the most to producing your target metric? How do you monitor the extent of each change? 3. Navigating through experiments. -- How to get back to a ML model from the previous week without spending time for the model re-training? +- How do you recover a model from last week without wasting time waiting for the model to re-train? -- How to quickly switch back and forth between the large data source and a small data subset without modifying source code? +- How do you quickly switch between the large data source and a small data subset without modifying source code? 4. Reproducibility. -- How to rerun a model evaluation step without re-trining the model training and preprocessing a raw dataset? +- How do you rerun a model's evaluation without re-training the model and preprocessing a raw dataset? 5. Managing and sharing large data files. -- How to share trained on GPU model with collegues who does not have any GPU? +- How do you share models trained in a GPU environment with colleagues who do not have access to a GPU? -- How to share 147Gb of an entire ML project with all the data sources and the intermediate data files and models? +- How do you share the entire 147Gb of your project, with all of its data sources, intermediate data files, and models? Some of these questions are easy to answer individually. -Any data scientist, engineer and manager knows or can easily find answers. -However, the variety of answers makes data science collaboration process a nightmare. +Any data scientist, engineer, or manager knows or could easily find answers to some of them. +However, the variety of answers and approaches makes data science collaboration a nightmare. **A systematic approach is required.** ========================================= -Experiment management software and issues +Experiment management software and its issues ========================================= There is one common opinion regarding data science tooling. -Data scientists as engineers are supposed to use the best practices and collaboration software from software engineering area like source version control system (Git), continuous integration services (CI), unit test frameworks and others methods and tools. -But a comprehensive look at data science processes identifies that the software engineering toolset does not cover data scientists needs (try to answer all the questions from the above using only engineering tools). +Data scientists as engineers are supposed to use the best practices and collaboration software from software engineering. Source version control system (Git), continuous integration services (CI), and unit test frameworks are all expected to be utilized in the data science pipeline. +But a comprehensive look at data science processes shows that the software engineering toolset does not cover data science needs. Try to answer all the questions from the above using only engineering tools, and you are likely to be left wanting for more. -A new type of **experiment management software** aims to cover the gap between data scientists needs and the existing toolset. -This kind of software are usually designed for enterprises with the assumption that all collaborators uses the same cloud and the same technology stack. -Some of modern ML modeling frameworks and cloud services includes these data scientists collaboration elements. +A new type of **experiment management software** aims to cover the gap between data scientist needs and the existing toolset. +This kind of software is typically designed for enterprise use, with the assumption that all collaborators uses the same cloud and technology stack. +Some modern ML modeling frameworks and cloud services address a portion of these much needed data science collaboration elements. -This software are usually requires external services (usually paid services). -This might be a good solution for a particular company or a group of data scientists. -However, a more common tool is needed for a wider audience including open source community. +This style of software usually require external services, typically accompanied with a monthly bill. +This might be a good solution for a particular companies or groups of data scientists. +However a more accessible, free tool is needed for a wider audience. ============================= What is DVC? ============================= -Data Version Control or DVC is **a new type of experiment management software** that has built on top of source code version control system (currently - Git). -This gives the ability to **use the best engineering practices in data science projects**. +Data Version Control, or DVC, is **a new type of experiment management software** that has been built to operate on top of source code version control (currently - Git). +This enables data scientist to **use the best engineering practices in their projects and experiments**. -The underlying source control system **eliminates the need of using external services**. -Experiment sharing and data scientists collaboration can be done by a regular Git tools (such as commit messages, merges, pull requests, code comments) the same way as it works for software engineers. +The underlying source control system **eliminates the need to use external services**. +Experiment sharing and data scientist collaboration can be done through regular Git tools (commit messages, merges, pull requests, code comments), the same way it works for software engineers. -DVC implements an **Git experimentation methodology** where each experiment with it's code as well as data is presented as a separate Git branch or commit. +DVC implements a **Git experimentation methodology** where each experiment exists with its code as well as data, and can be represented as a separate Git branch or commit. -DVC uses a few concepts: +DVC uses a few core concepts: -- **Experiment** is equal to Git branch. Each experiment (extract new features, change model hyperparameters, clean noise in data, add a new data source) should be performaed in a separate branch and then merged into the master branch if the experiment was successfull. DVC manages to NOT recompute the results after such merges. +- **Experiment** is equivalent to a Git branch. Each experiment (extract new features, change model hyperparameters, data cleaning, add a new data source) should be performed in a separate branch and then merged into the master branch only if the experiment is successful. DVC allows experiments to be integrated into a project's history and NEVER needs to recompute the results after a successful merge. -- **Experiment state** or state is equal to Git snapshot (all commited files). Git checksum, a branch name or a tag can be used as a reference to an experiment state. +- **Experiment state** or state is equivalent to a Git snapshot (all committed files). Git checksum, branch name, or tag can be used as a reference to a experiment state. -- **Reproducability** - an action to reproduce an experiment state. The action generates output files based on a set of input files and source code. This action usually changes experiment state. +- **Reproducibility** - an action to reproduce an experiment state. This action generates output files based on a set of input files and source code. This action usually changes experiment state. -- **Pipeline** - directed acyclic graph (DAG) of commands to reproduce an experiment state. The commands are connected by input and output files. Pipeline is defined by special **dvc-files** (like Makefiles). +- **Pipeline** - directed acyclic graph (DAG) of commands to reproduce an experiment state. The commands are connected by input and output files. Pipeline is defined by special **dvc-files** (which act like Makefiles). -- **Workflow** - set of experiments and relationships between them. Workflow corresponds to the entire Git repository. +- **Workflow** - set of experiments and relationships among them. Workflow corresponds to the entire Git repository. -- **Data files** - cached files (for large files). For data files the file content is stored outside of Git repository in a local hard drive but data file meta information is stored in Git for DVC needs (to maintain pipilines and reproducibility). +- **Data files** - cached files (for large files). For data files the file content is stored outside of the Git repository on a local hard drive, but data file metadata is stored in Git for DVC needs (to maintain pipelines and reproducibility). -- **Data cache** is a directory with all data files content in a local hard drive or a cloud storage but not in Git repository. +- **Data cache** - directory with all data files on a local hard drive or in cloud storage, but not in the Git repository. - **Cloud storage** support is a compliment to the core DVC features. This is how a data scientist transfers large data files or shares a trained on GPU model to whom who does not have GPU. @@ -90,86 +90,88 @@ Core features 1. DVC works **on top of Git repositories** and has a similar command line interface and Git workflow. -2. It makes data science projects **reproductable** by creating lightweight pipelines (DAG). +2. It makes data science projects **reproducible** by creating lightweight pipelines of DAGs. -3. **Large data file versioning** by creating pointers in your Git repository into a data cache in a local hard drive. +3. **Large data file versioning** works by creating pointers in your Git repository to the data cache on a local hard drive. 4. **Programming language agnostic**: Python, R, Julia, shell scripts, etc. ML library agnostic: Keras, Tensorflow, PyTorch, scipy, etc. -5. **Open sourced** and **Self-served**. DVC is free and does not require any additional services. +5. **Open-sourced** and **Self-served**. DVC is free and does not require any additional services. -6. DVC supports cloud storges (AWS S3 and GCP storage) for **data and pretrained models sharing**. +6. DVC supports cloud storage (AWS S3 and GCP storage) for **data sources and pre-trained models sharing**. ==================== Related technologies ==================== -Due to the the novelty of this approach DVC can be understood better in comparision to the existing technologies and ideas. +Due to the the novelty of this approach, DVC can be better understood in comparison to existing technologies and ideas. -DVC combines numbers of existing technologies and ideas into a single product with a goal of bringing the best engineering practices into data sciance process. This comparision can give you good understanding of DVC if you are familliar with some of these technologies. +DVC combines a number of existing technologies and ideas into a single product with the goal of bringing the best engineering practices into the data science process. -1. **Git** is an underlying technology for DVC. Difference: +1. **Git**. The difference is: - - DVC extends Git by concept of *data files* - large files that should NOT be stored in Git repository but still have to be tracked and versioned. See more details in Git-annexand Git-LFS items below. + - DVC extends Git by introducing the concept of *data files* - large files that should NOT be stored in a Git repository but still need to be tracked and versioned. -2. **Workflow management tools** (pipelines and DAGs): Apache Airflow, Luigi and etc. Differences: - - - Comparing to these data engineering technologies, DVC is focused on data science and modeling. As a result, DVC pipelines are lightweight, easy to create and modify. However, DVC lacks of pipeline execution features like execution monitoring and execution error handling and recovering. +2. **Workflow management tools** (pipelines and DAGs): Apache Airflow, Luigi and etc. The differences are: - - DVC is a purly command line tool that does not have user interface and does not run any servers. Nevertheless, DVC can generate images with pipeline and experiment workflow visualization. + - DVC is focused on data science and modeling. As a result, DVC pipelines are lightweight, easy to create and modify. However, DVC lacks pipeline execution features like execution monitoring, execution error handling, and recovering. -3. Today **experiment management** software are mostly designed for enterprise usage. Open sourced experimentation tool example: http://studio.ml/. Differences: + - DVC is purely a command line tool that does not have a user interface and does not run any servers. Nevertheless, DVC can generate images with pipeline and experiment workflow visualization. - - DVC uses Git as underlying platform for experiment tracking instead of a web application. +3. **Experiment management** software today is mostly designed for enterprise usage. An open-sourced experimentation tool example: http://studio.ml/. The differences are: - - There is no need to run any services. No user interface as a result. Nevertheless, we expect some UI sevices will be created on top of DVC. + - DVC uses Git as the underlying platform for experiment tracking instead of a web application. - - DVC has transparent design: dvc-files, meta files, state file, cache dirs have a simple (usually text) format and can be easily reused by external tools. + - DVC does not need to run any services. No user interface as a result, but we expect some UI services will be created on top of DVC. -4. **Git workflows** and Git usage methodologies such as Gitflow. Differences: + - DVC has transparent design: DVC-files, meta files, state file, cache dirs have a simple format and can be easily reused by external tools. - - DVC supports a new experimentation methodology (and Git workflow) when a separate branch should be created for each experiment with unpredictable results with subsequent merge of this branch if the result was successful. An experiment can have multiple sub experiments. +4. **Git workflows** and Git usage methodologies such as Gitflow. The differences are: - - One of the major DVC innovation is the ability to navigate through the experiments without recomputing them. + - DVC supports a new experimentation methodology that integrates easily with a Git workflow. A separate branch should be created for each experiment, with a subsequent merge of this branch if it was successful. + + - DVC innovates by allowing experimenters the ability to easily navigate through past experiments without recomputing them. -5. DVC uses the **Makefile** (and it's analogues) idea of defining DAG by a special files. Differences: +5. **Makefile** (and it's analogues). The differences are: - - The DAG definition: + - DVC utilizes a DAG: - - The DAG is defined by dvc-files with filenames *Dvcfile* or *.dvc*, not *Makefile*. + - The DAG is defined by dvc-files with filenames *Dvcfile* or *.dvc*. - - One dvc-file defines one node in DAG. All dvc-files in a repository make up a single pipeline (think a single Makefile). All dvc-files (and corresponded pipeline commands) are imlicitly combined through theirs inputs and outputs. This file separation is done to simplify conflict resolving during Git merges. + - One dvc-file defines one node in the DAG. All dvc-files in a repository make up a single pipeline (think a single Makefile). All dvc-files (and corresponding pipeline commands) are implicitly combined through their inputs and outputs, to simplify conflict resolving during merges. - - DVC provide a simple command *dvc run CMD* to generate dvc file automaticaly based on the provided command, dependencies and outputs. + - DVC provides a simple command *dvc run CMD* to generate a dvc-file automatically based on the provided command, dependencies, and outputs. - File tracking: - - DVC tracks files based on checksum (md5) instead of file timesteps. This helps to avoid of running heavy process like model re-training when you checkout a previous, already trained version of a modeling code (Makefile will retrain a model). + - DVC tracks files based on checksum (md5) instead of file timestamps. This helps avoid running into heavy processes like model re-training when you checkout a previous, trained version of a modeling code (Makefile will retrain the model). - - Despite the existence of file checksum tracking, DVC still uses the files timestemps and inodes and keep them in a local state file for optimization. Otherwise, each reproduction command will require recomputing all dependency files checksum - unappropriate approach for large files (10Gb+). + - DVC uses the files timestamps and inodes for optimization. This allows DVC to avoid recomputing all dependency files checksum, which would be highly problematic when working with large files (10Gb+). -6. **Git-annex**. DVC uses the idea of storing the content of large files (that you don't what to see in your Git repository) in a local key-value store and use file symlinks insted of the actual files. Difference: +6. **Git-annex**. The differences are: + + - DVC uses the idea of storing the content of large files (that you don't what to see in your Git repository) in a local key-value store and use file symlinks instead of the actual files. - DVC uses hardlinks instead of symlinks to make user experience better. - - - DVC optimizes checksum calculation (see Makefile differences from the above). - - DVC stores data file mata information in Git repository *.dvc/*, not in the Git tree *.git/annex/*. As a result, all meta information can be shared through any Git server like Github (Git-annex loses all meta information when shared by Git server). + - DVC optimizes checksum calculation. -7. **Git-LFS** (Large File Storage) interface is much user friendly comparing to Git-annex. It requires a very limited number of additional command. DVC uses this methodology - native Git like interface with a couple additional file related commands. Differences: + - DVC stores data file metadata in Git repository *.dvc/*, not in the Git tree *.git/annex/*. As a result, all metadata can be shared through any Git server like Github (Git-annex loses all metadata when shared by Git server). - - DVC is fully compatable with Git. It does not required special Git servers like Git-LFS does. +7. **Git-LFS** (Large File Storage). The differences are: - - DVC does not add any hooks to Git by default. To checkout data files *dvc checkout* command has to be run after each *git checkout* and *git clone* command. + - DVC is fully compatible with Git. It does not require special Git servers like Git-LFS demands. - - *dvc checkout* command does not actually copy data files from cache to the working tree because of the file copying command is a heavy operation for large files (30Gb+). DVC creates hardlinks instead and changes data file permissions to read only. + - DVC does not add any hooks to Git by default. To checkout data files, the *dvc checkout* command has to be run after each *git checkout* and *git clone* command. + + - DVC creates hardlinks instead and changes data file permissions to read only. The *dvc checkout* command does not actually copy data files from cache to the working tree, as copying files is a heavy operation for large files (30Gb+). ============== -How it works? +How does it work? ============== 1. DVC is a command line tool that works on top of Git:: @@ -177,7 +179,7 @@ How it works? $ cd my_git_repo $ dvc init -2. It helps define pipelines of your commands and keeps all the commands and the dependencies (DAG) in Git repository:: +2. DVC helps define pipelines of your commands, and keeps all the commands and dependencies in a Git repository:: $ dvc run -d input.csv -o results.csv python cnn_train.py --seed 20180227 --epoch 20 input.csv result.csv $ git add result.csv @@ -189,7 +191,7 @@ How it works? $ git add plots.jpg $ git commit -m 'CNN plots' -4. Reproduce a pipeline with respect to dependencies:: +4. DVC can reproduce a pipeline with respect to the pipeline's dependencies:: # The input dataset was changed $ dvc repro plots.jpg @@ -198,7 +200,7 @@ How it works? Reproducing 'plots.jpg': Rscript plot.R result.csv plots.jpg -5. DVC introduces data files concept for Git repository. DVC keeps data files outside of Git repository but meta data in Git:: +5. DVC introduces the concept of data files to Git repositories. DVC keeps data files outside of the repository but retains the metadata in Git:: $ git checkout a03_normbatch_vgg16 # checkout code and DVC meta data $ dvc checkout # checkout data files from the local cache (not Git) @@ -208,7 +210,7 @@ How it works? -r-------- 2 501 staff 12G Jan 27 03:48 Posts-train.tsv -6. Reproductable DVC repository with DVC meta information (DAG) can be easily shared by any Git server:: +6. DVC makes repositories reproducible. DVC metadata can be easily shared through any Git server, and allows for experiments to be easily reproduced:: $ git clone https://github.com/dataversioncontrol/myrepo.git $ cd myrepo @@ -219,21 +221,21 @@ How it works? Reproducing 'plots.jpg': Rscript plot.R result.csv plots.jpg -7. DVC local cache can be transfered to your collegues and parthners through AWS S3 or GCP Storage:: +7. DVC's local cache can be transferred to your colleagues and partners through AWS S3 or GCP Storage:: $ git push $ dvc push # push the data cache to your cloud bucket - - # On a collegue machine: + + # On a colleague machine: $ git clone https://github.com/dataversioncontrol/myrepo.git $ cd myrepo $ git pull # get the data cache from cloud $ dvc checkout # checkout data files - $ ls -l data/ # You just got gigabites of data though Git and DVC: + $ ls -l data/ # You just got gigabytes of data through Git and DVC: total 1017488 -r-------- 2 501 staff 273M Jan 27 03:48 Posts-test.tsv -8. DVC works on Mac, Linux and Windows systems. Windows example:: +8. DVC works on Mac, Linux ,and Windows. A Windows example:: $ dir ????? @@ -241,15 +243,15 @@ How it works? ============================== -Getting Started with DVC (old) +Getting Started with DVC ============================== -To show DVC in action let's play with an actual machine learning (ML) scenario. -This is going to be a natural language processing (NLP) problem of predicting tags for given - stackoverflow question. -For instance, for the tag "Java" one classifier will be created which can predict a post that is about the Java language. +To show DVC in action, let's play with an actual machine learning scenario. +Let's explore the natural language processing (NLP) problem of predicting tags for a given + StackOverflow question. +For example, we want one classifier which can predict a post that is about the Java language by tagging it "Java". -First, let's download modeling code and set up Git repository:: +First, let's download the model code and set up the Git repository:: $ mkdir myrepo $ cd myrepo @@ -265,152 +267,148 @@ First, let's download modeling code and set up Git repository:: $ git add code/ $ git commit -m 'Download code' -The full pipeline could be built by running the code below:: +The full pipeline can be built by running the code below:: $ # Initialize DVC repository (in your Git repository) $ dvc init - - $ # Download a file and put to data/ directory. + + $ # Download a file and put it into the data/ directory. $ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz data/ - + $ # Extract XML from the archive. $ dvc run tar zxf data/Posts.xml.tgz -C data/ - - $ # Prepare data. + + $ # Prepare the data. $ dvc run python code/xml_to_tsv.py data/Posts.xml data/Posts.tsv python - + $ # Split training and testing dataset. Two output files. - $ # 0.33 is the test dataset splitting ratio. 20170426 is a seed for randomization. + $ # 0.33 is the test dataset split ratio. 20170426 is a seed for randomization. $ dvc run python code/split_train_test.py data/Posts.tsv 0.33 20170426 data/Posts-train.tsv data/Posts-test.tsv - - $ # Extract features from text data. Two TSV inputs and two pickle matrixes outputs. + + $ # Extract features from the data. Two TSV as inputs with two pickle matrices as outputs. $ dvc run python code/featurization.py data/Posts-train.tsv data/Posts-test.tsv data/matrix-train.p data/matrix-test.p - - $ # Train ML model out of the training dataset. 20170426 is another seed value. + + $ # Train ML model on the training dataset. 20170426 is another seed value. $ dvc run python code/train_model.py data/matrix-train.p 20170426 data/model.p - - # Evaluate the model by the testing dataset. + + # Evaluate the model on the test dataset. $ dvc run python code/evaluate.py data/model.p data/matrix-test.p data/evaluation.txt - + $ # The result. $ cat data/evaluation.txt AUC: 0.596182 - -DVC an open source tool for data science projects. DVC makes your data science projects reproducible by automatically building data dependency graph (DAG). Your code and the dependencies could be easily shared by Git, and data - through cloud storage (AWS S3, GCP) in a single DVC environment. - -Your code can be easily reproduced after modification:: +Your code can be easily reproduced after some minor modification:: $ # Improve feature extraction step. $ vi code/featurization.py - + $ # Commit all the changes. $ git commit -am "Add bigram features" [master 50b5a2a] Add bigram features 1 file changed, 5 insertion(+), 2 deletion(-) - + $ # Reproduce all required steps to get our target metrics file. $ dvc repro data/evaluation.txt Reproducing run command for data item data/matrix-train.p. Args: python code/featurization.py data/Posts-train.tsv data/Posts-test.tsv data/matrix-train.p data/matrix-test.p Reproducing run command for data item data/model.p. Args: python code/train_model.py data/matrix-train.p 20170426 data/model.p Reproducing run command for data item data/evaluation.txt. Args: python code/evaluate.py data/model.p data/matrix-test.p data/evaluation.txt Data item "data/evaluation.txt" was reproduced. - + $ # Take a look at the target metric improvement. $ cat data/evaluation.txt AUC: 0.627196 -It is quite easy to integrate DVC in your existing ML pipeline/processes without any significant effort to re-implement your ML code/application. +It's easy to integrate DVC into your existing ML pipeline/processes without any significant effort to re-implement your code/application. -The one thing to wrap your head around is that DVC automatically derives the dependencies between the steps and builds the dependency graph (DAG) transparently to the user. This graph is used for reproducing parts of your pipeline which were affected by recent changes. +The key step to note is that DVC automatically derives the dependencies between the experiment steps and builds the dependency graph (DAG) transparently. -Not only can DVC streamline your work into a single, reproducible environment, it also makes it easy to share this environment by Git including the dependencies (DAG) — an exciting collaboration feature which gives the ability to reproduce the research results in different computers. Moreover, you can share your data files through cloud storage services like AWS S3 or Google Cloud Project (GCP) Storage since DVC does not push data files to Git repositories. +Not only can DVC streamline your work into a single, reproducible environment, it also makes it easy to share this environment by Git including the dependencies  —  an exciting collaboration feature which gives the ability to reproduce the research easily in a myriad of environments. ============ Installation ============ -Operation system dependent packages is the recommended way of installing DVC. -Some other methods of installation are provided. +Operating system dependent packages are the recommended way to install DVC. +Some other methods of installation are available. OS packages =========== -DVC installation packages available for Mac OS, Linux and Windows platforms. -You can download the packages at https://github.com/dataversioncontrol/dvc/releases/ +DVC installation packages are available for Mac OS, Linux, and Windows. +You can download the packages here at https://github.com/dataversioncontrol/dvc/releases/ Python pip ========== -Another option to deploy DVC to your machine is to use its standard Python pip package:: +Another option is to use the standard Python pip package:: $ pip install dvc -**Note:** if you use *Anaconda*, you can use the above-mentioned command there as well. -It will work in *Anaconda’s* command prompt tool. -As of the moment, DVC does not provide a special installation package for a native *Anaconda* package manager (that is, *conda*). +**Note:** if you use *Anaconda*, it will work in *Anaconda’s* command prompt tool. +At the moment, DVC does not provide a special installation package for the native *Anaconda* package manager *conda*. Homebrew Cask ============= -Mac OS users can install DVC by **brew** command:: +Mac OS users can install DVC by using the **brew** command:: $ brew cask install dataversioncontrol/homebrew-dvc/dvc Development Version =================== -If you like to pull the latest version of DVC from the master branch in its repo at github, you execute the following command in your command prompt:: +If you would like to pull the latest version of DVC, you can do the following:: $ pip install git+git://github.com/dataversioncontrol/dvc -This command will automatically upgrade your DVC version in case it is behind the latest version in the master branch of the github repo. +Note: this will automatically upgrade your DVC version to the latest version. ============= Configuration ============= -Once you install DVC, you should be able to start using it (in its local setup) immediately. +Once you install DVC, you will be able to start using it (in its local setup) immediately. -However, you can optionally proceed to further configure DVC (especially if you intend to use it in a Cloud-based scenario). +However, you can proceed to configure DVC (especially if you intend to use it in a *cloud-based* scenario). DVC Files and Directories ========================= -Once installed, dvc will populate its installation folder (hereinafter referred to as .dvc) with essential shared and internal files and folders will be stored +Once installed, DVC will populate its installation folder (hereafter referred to as .dvc) * **.dvc/config** - This is a configuration file. - The config file can be edited directly or indirectly using command **dvc config NAME VALUE**. + The config file can be edited directly using command **dvc config NAME VALUE**. * **.dvc/cache** - the cache directory will contain your data files (the data directories of DVC repositories will only contain symlinks to the data files in the global cache). - **Note:** DVC includes the cache directory to **.gitignore** file during the initilization. And no data files (with actual content) will be pushed to Git repository, - only data file symlinks and commands to reproduce them. -* **.dvc/state** - this file is ceated for optimization. The file contains data files checksum and timestemps. + **Note:** DVC includes the cache directory to **.gitignore** file during the initialization. And no data files (with actual content) will ever be pushed to Git repository, + only data file symlinks and commands are needed to reproduce them. +* **.dvc/state** - this file is created for optimization. The file contains data files checksum and timestamps. Working with Cloud Data Storages ====================================================== -Using DVC with Cloud-based Data Storages is an optional feature. -By default, DVC is configured to use local data storage only (.dvc/cache directory), - and it enables basic DVC usage scenarios out of the box. +Using DVC with Cloud-based data storage is optional. +By default, DVC is configured to use a local data storage only (.dvc/cache directory), + which enables basic DVC usage scenarios out of the box. -DVC can use cloud storages as a common file storage. -With cloud storage you might use models and data file which were created by your team members - without spending time and resources for re-building models and re-processing data files. +DVC can use cloud storage as a common file storage. +With cloud storage, you might use models and data files which were created by your team members + without spending time and resources to re-build models and re-process data files. -As of this version, DVC supports two types of cloud-based data storage providers: +As of this version, DVC supports two types of cloud-based storage providers: * **AWS** - Amazon Web Services * **GCP** - Google Cloud Provider -The subsections below explain how to configure DVC to use of the data cloud storages above. +The subsections below explain how to configure DVC to use each of them. -Using AWS Cloud +Using AWS --------------- -For using AWS as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively +To use AWS as cloud storage for your DVC repositories, you should update these **.dvc/config** options * **Cloud = AWS** in *Global* section. * **StoragePath = /mybucket/dvc/tag_classifier** in **AWS** section - path to a cloud storage bucket and directory in the bucket. @@ -420,11 +418,11 @@ For using AWS as a data cloud storage for your DVC repositories, you should upda **Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to AWS"** -Instead of manual file modification we recommend to run corresponded commands:: +Instead of manual file modification, we recommend you run the following commands:: $ dvc config Global.Cloud AWS # This step is not needed for new DVC repositories - $ dvc config AWS.StoragePath /mybucket/dvc/tag_classifier - $ dvc config AWS.CredentialPath ~/.aws/credentials # Not needed if aws cli is instelled to default path + $ dvc config AWS.StoragePath /mybucket/dvc/tag_classifier + $ dvc config AWS.CredentialPath ~/.aws/credentials # Not needed if AWS CLI is installed to default path $ dvc config AWS.CredentialSection default # Not needed if you have only one AWS account $ git commit -am "Change cloud to AWS" @@ -432,18 +430,18 @@ Instead of manual file modification we recommend to run corresponded commands:: Using Google Cloud ------------------ -For using GCP (Google Cloud Provider) as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively +For using GCP (Google Cloud Provider) as cloud storage for your DVC repositories, you should update these **.dvc/config** options * **Cloud = GCP** in *Global* section. -* **StoragePath = /mybucket/dvc/tag_classifier** in GCP section - this option has the same meaning as AWS one above. Run **dvc config GCP.StoragePath /my/path/to/a/bucket** +* **StoragePath = /mybucket/dvc/tag_classifier** in GCP section - Run **dvc config GCP.StoragePath /my/path/to/a/bucket** * **ProjectName = MyCloud** - a GCP specific project name. **Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to GCP"** -Instead of manual file modification we recommend to run corresponded commands:: +Instead of manual file modification, we recommend you run the following commands:: $ dvc config Global.Cloud GCP - $ dvc config GCP.StoragePath /mybucket/dvc/tag_classifier + $ dvc config GCP.StoragePath /mybucket/dvc/tag_classifier $ dvc config GCP.ProjectName MyCloud $ git commit -am "Change cloud to AWS" @@ -453,26 +451,26 @@ Using DVC Commands ================== DVC is a command-line tool. -The typical method of use of DVC is as follows +The typical use case for DVC goes as follows -* In an existing Git repository initialize a DVC repository with **dvc init** command. -* Copy source files for modeling into the repository (without checking out to Git) and convert the files in DVC data files with **dvc add** command. -* Process source data files by your data processing and modeling code through **dvc run** command. In this command generated DVC files to describe these processing steps. -* Use **--outs** option to specify **dvc run** command outputs which should be to be converted to DVC data files after the code is completed. -* You clone a git repo with the code of your ML application pipeline. However, it does not copy DVC cache. Use cloud storage settings and **dvc push** command to share the cache (data). -* You use **dvc repro** command to quickly reproduce your pipeline on a new iteration, once either the data item files or the source code of your ML application are modified. +* In an existing Git repository, initialize a DVC repository with **dvc init**. +* Copy source files for modeling into the repository and convert the files into DVC data files with **dvc add** command. +* Process source data files through your data processing and modeling code using the **dvc run** command. +* Use **--outs** option to specify **dvc run** command outputs which will be converted to DVC data files after the code runs. +* Clone a git repo with the code of your ML application pipeline. However, this will not copy your DVC cache. Use cloud storage settings and **dvc push** to share the cache (data). +* Use **dvc repro** to quickly reproduce your pipeline on a new iteration, after your data item files or source code of your ML application are modified. ======================== DVC Commands Cheat Sheet ======================== -Below is the quick summary of the most important commands of DVC +Below is the quick summary of the most important commands * **dvc -h** - Show how to use DVC and show the list of commands. -* **dvc CMD -h** - Display help to use a specific DVC command (CMD) -* **dvc init** - Initialize a new DVC repository. +* **dvc CMD -h** - Display help to use a specific DVC command (CMD). +* **dvc init** - Initialize a new DVC repository. * **dvc add** - Add data file or data directory. The command converts regular files to DVC data files. -* **dvc checkout** - Checkout data files and dirs into working tree. The command should be executed after **git checkout** or cloning a repository. +* **dvc checkout** - Checkout data files and dirs into the working tree. The command should be executed after **git checkout** or cloning a repository. * **dvc run** - Generate a DVC file from a given command and execute the command. The command dependencies and outputs should be specified. * **dvc pull** - Pull data files from the cloud. Cloud settings for your DVC environment should be already configured prior to using this command. * **dvc push** - Push data files to the cloud. Cloud settings should be already configured. @@ -506,23 +504,23 @@ Example. Creating a new DVC repository:: $ mkdir tag_classifier $ cd tag_classifier - + $ git init Initialized empty Git repository in /Users/dmitry/src/tag_classifier/.git/ - + $ dvc init $ git status On branch master - + Initial commit - + Changes to be committed: - + (use "git rm --cached ..." to unstage) - + new file: .dvc/.gitignore new file: .dvc/config - + $ git commit -m 'Init DVC' [master (root-commit) 2db4618] Init DVC 2 files changed, 41 insertions(+) @@ -535,22 +533,21 @@ add Converts files and directories to DVC data files. -The command doe the convertation from a *regular file* to DVC data file in a few steps: +The command doe the conversion from a *regular file* to *DVC data file* in a few steps: 1. Calculate the file checksum. -2. Create a cache file in the cache dir *.dvc/cache* with the content of this file. -3. Create a corresponded DVC file. -4. Replace the file by a hardlink to the cache file. +2. Create a cache file in the cache dir *.dvc/cache*. +3. Create a corresponding DVC file. +4. Replace the file with a hardlink to the cache file. -Also, to reduce time on recomputing the file checksum in future DVC stores the file last modification time, inode and the checksum into a global state file *.dvc/state*. -Next time, then the file chacksum will be needed DVC will try to get it from the file if the file was not modified. +DVC stores the file's last modification timestamp, inode, and the checksum into a global state file *.dvc/state* to reduce time recomputing checksums later. -Note, this command does NOT copy any file content and run quickly even for a large files. +Note, this command does NOT copy any file contents and will run quickly even for a large files. Step (2) from the above is also made by hardlinks movement, not file content. -The only haavy step is (1) which requires checksum calculation. +The only heavy step is (1), which requires checksum calculation. -For directories the command does the same steps for each file recursively. -To keep information about the directory structure a corresponded directory will be created in the cache *.dvc/cache*. +For directories, the command does the same steps for each file recursively. +To retain information about the directory structure, a corresponding directory will be created in *.dvc/cache*. .. code-block:: shell :linenos: @@ -575,7 +572,7 @@ Convert files into data files:: Badges.xml PostLinks.xml Votes.xml Badges.xml.dvc PostLinks.xml.dvc Votes.xml.dvc -Note, DVC files were created. +Note, DVC files are created. checkout @@ -584,11 +581,11 @@ checkout Checkout data files from cache. This command has to be called after *git checkout* since Git does not handle DVC data files. -The command restores data files from cache to working tree and removes data files that are not belog to the current working tree anymore. +The command restores data files from cache to the working tree and removes data files that are no longer on the working tree. -Note, this command does NOT copy any files - DVC uses hardlinks to perform the data file restoring. -This is crucial for large files where checking out (copiyng) 50Gb file might take a few minutes. -For DVC it will take less than a second to restore 50Gb data file. +Note, this command does NOT copy any files - DVC uses hardlinks to perform data file restoration. +This is crucial for large files where checking out as a 50Gb file might take a few minutes. +For DVC, it will take less than a second to restore a 50Gb data file. .. code-block:: shell @@ -612,10 +609,10 @@ Checking out a branch example:: $ 'data/Posts-train.tsv': cache file not found DVC does not report in the output which data files were restored. -However, it reports removed files and files which DVC was unable to restore because of missing cache. -To restore file with missing cache reproduction command should be called or cache can be pulled from a cloud. +However, it reports removed files and files that DVC was unable to restore due to missing cache. +To restore a file with a missing cache, the reproduction command should be called or the cache can be pulled from the cloud. -It might be convinient to assign Git hook to *git checkout* comman:: +It might be convenient to assign Git hook to *git checkout*:: $ echo 'dvc checkout' > .git/hooks/post-checkout $ chmod +x .git/hooks/post-checkout @@ -630,20 +627,20 @@ run Generate a stage file from a given command and execute the command. The command dependencies and outputs should be specified. -By default stage file name is **.dvc** where **** is file name of a first output. +By default, stage file name is **.dvc** where **** is file name of the first output. For example, launch Python with a given python script and arguments. Or R script by Rscript command. .. code-block:: shell :linenos: - + usage: dvc run [-h] [-q] [-v] [-d DEPS] [-o OUTS] [-O OUTS_NO_CACHE] [-f FILE] [-c CWD] [--no-exec] ... - + positional arguments: command Command or command file to execute - + optional arguments: -h, --help show this help message and exit -q, --quiet Be quiet. @@ -659,18 +656,18 @@ For example, launch Python with a given python script and arguments. Or R script Examples: -Execute a Python script as a DVC ML pipeline step. Stage file was not specified. So, **model.p.dvc** stage file will be created:: +Execute a Python script as the DVC pipeline step. Stage file was not specified, so a **model.p.dvc** stage file will be created:: - $ # Train ML model out of the training dataset. 20180226 is a seed value. + $ # Train ML model on the training dataset. 20180226 is a seed value. $ dvc run -d matrix-train.p -d train_model.py -o model.p python train_model.py matrix-train.p 20180226 model.p -Execute an R script as a DVC ML pipeline step:: +Execute an R script as the DVC pipeline step:: $ dvc run -d parsingxml.R -d Posts.xml -o Posts.csv Rscript parsingxml.R Posts.xml Posts.csv -Extract an XML file from an archive to data subfolder:: +Extract an XML file from an archive to the data/ subfolder:: $ mkdir data $ dvc run -d Posts.xml.tgz -o data/Posts.xml tar zxf Posts.xml.tgz -C data/ @@ -679,9 +676,9 @@ Extract an XML file from an archive to data subfolder:: push ==== -This command pushes all data files caches related to the current Git branch to the cloud storage. +This command pushes all data file caches related to the current Git branch to cloud storage. Cloud storage settings need to be configured. -See cloud storage configuration. +See cloud storage configuration for more details on how to set up cloud storage. .. code-block:: shell :linenos: @@ -696,7 +693,7 @@ See cloud storage configuration. Examples: -Push all data files caches from the current Git branch to cloud:: +Push all data file caches from the current Git branch to cloud:: $ dvc push (1/8): [########################################] 100% 72271bebdf053178a5cce48b4 @@ -707,19 +704,19 @@ Push all data files caches from the current Git branch to cloud:: (6/8): [########################################] 100% 3f9c7c3ae51db2eed7ba99e6e (7/8): [########################################] 100% cfdaa4bba57fa07d81ff96685 (8/8): [####################### ] 57% 1de6178a9dd844e249ba05414 - + pull ==== -This command pulls all data files caches from the cloud storage. +This command pulls all data file caches from cloud storage. Cloud storage settings need to be configured. .. code-block:: shell :linenos: - + usage: dvc pull [-h] [-q] [-v] [-j JOBS] - + optional arguments: -h, --help show this help message and exit -q, --quiet Be quiet. @@ -749,7 +746,7 @@ Show mismatches between local cache and cloud cache. :linenos: usage: dvc status [-h] [-q] [-v] [-j JOBS] - + optional arguments: -h, --help show this help message and exit -q, --quiet Be quiet. @@ -768,7 +765,7 @@ repro Reproduce DVC file and all stages the file depends on (recursively). Default file name is **Dvcfile**. -However, usually DVC files have any name and **.dvc** suffix. +However, DVC files can have any name followed by the **.dvc** suffix. .. code-block:: shell :linenos: @@ -802,7 +799,7 @@ Reproduce default stage file:: Reproducing 'model.p.dvc': python code/train_model.py data/matrix-train.p 20170426 data/model.p -Reproduce the part of the pipeline (from above) where *Posts.tsv.dvc* is target DVC file:: +Reproduce the part of the pipeline where *Posts.tsv.dvc* is the target DVC file:: $ dvc repro Posts.tsv.dvc Reproducing 'Posts.xml.dvc': @@ -841,13 +838,13 @@ Remove *matrix-train.p* data file:: gc === -This command collects the garbage - removes unused cache files based on the current Git branch. -So, if a data file was created in a different branch then it is going to be removed by command. -If a data file has a few versions (and, correspondingly, caches) - all the chaches except the current one will be removed. +This command collects the garbage, removing unused cache files based on the current Git branch. +If a data file was created in a different branch, then it will be removed by gc. +If a data file has a few versions (and, of course. corresponding caches) - all caches except the current one will be removed. .. code-block:: shell :linenos: - + age: dvc gc [-h] [-q] [-v] optional arguments: @@ -876,12 +873,12 @@ Clean up example:: config ====== -Get or set config options. This command reads and owerwrites DVC config file *.dvc/config*. +Get or set config options. This command reads and overwrites the DVC config file *.dvc/config*. .. code-block:: shell :linenos: - + usage: dvc config [-h] [-q] [-v] [-u] name [value] positional arguments: @@ -896,7 +893,7 @@ Get or set config options. This command reads and owerwrites DVC config file *.d Examples: -Specify an option name to get the option value from config file:: +Specify an option name to get the option's value from config file:: $ dvc config config Global.Cloud AWS @@ -915,7 +912,7 @@ Generate pipeline image for your current project. .. code-block:: shell :linenos: - + usage: dvc show [-h] [-q] [-v] {pipeline} ... positional arguments: @@ -937,16 +934,16 @@ fsck ==== Data file consistency check. -By default the commands outputs statuses of all corrupted data files if any. -Use *--all* option to see statuses of all data files statuses. +By default the commands outputs statuses of all corrupted data files (if any). +Use *--all* option to see statuses of all data files. The command checks: -1. Cache file name which equals to the file content checksum on the time when DVC created the file. +1. Cache file name which is equal to the file content checksum when DVC created the file. 2. Checksum from local state file. 3. Checksum regarding DVC files. -4. Actual recomputed checksum. This is computation heavy command for large data file. Enabled only by *--physical* option. +4. The actual recomputed checksum. This is a computation heavy command for large data files. Enabled only by *--physical* option. -Data file is considered as corrupted if one of the checksum does not match all others. +Data file is considered to be corrupted if one of the checksums does not match all others. .. code-block:: shell @@ -991,25 +988,24 @@ Common Arguments Common Options -------------- -As you can see, there are four optional arguments that are applicable to any DVC command. These are +As you can see, there are four optional arguments that are applicable to any DVC command. They are .. code-block:: shell :linenos: - + -h, --help show this help message and exit -q, --quiet Be quiet. -v, --verbose Be verbose. -G, --no-git-actions Skip all git actions including reproducibility check and commits. -Although these optional arguments are pretty self-explanatory, there is a note on DVC and Git commands used together. +Although these optional arguments are pretty self-explanatory, there is a note for DVC and Git commands that are used together. -* If you specify *--no-git-action* option, DVC does not modify (add/commit to) Git repository, however, it can still read it - for example, run *git status* command etc. +* If you specify *--no-git-action*, DVC does not modify (add/commit to) the Git repository. However, it can still read it - for example, run *git status* command etc. * To see Git commands in DVC, you can set logging level to *Debug* (in **dvc.conf**) or run dvc with option *--verbose* Number of DVC Jobs ------------------ -DVC can benefit from parallel processing and multiple processors/cores available on your machine. It can spin a number of jobs to run in parallel. - -The number of DVC jobs is 5 by default. In case you like to change it to any other reasonable value, you use *-j (--jobs)* option in DVC commands where it is applicable. +DVC can benefit from parallel processing and multiple processors/cores. +The number of DVC jobs is 5 by default. If you would like to change it to any other reasonable value, you use *-j (--jobs)* option in DVC commands where applicable.