From 6595fffbbf4f976f3747262bed39cc3dfcbba39c Mon Sep 17 00:00:00 2001 From: Dmitry Date: Sat, 20 Jan 2018 23:45:23 -0800 Subject: [PATCH 1/4] Doc: raditional Makefile based reproducibility --- docs/doc.rst | 53 ++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/docs/doc.rst b/docs/doc.rst index f52dfa3b6..dd73568ee 100644 --- a/docs/doc.rst +++ b/docs/doc.rst @@ -6,6 +6,51 @@ It is hardly possible in real life to develop a good machine learning model in a **Data Version Control** (aka DVC) is designed to help data scientists keep track of their ML processes and file dependencies in the simple form of git-like commands: "dvc run python train_model.py data/train_matrix.p data/model.p". Your existing ML processes can be easily transformed into reproducible DVC pipelines regardless of which programming language or tool was used. +================== +DVC Basic +================== + +Traditional Makefile based reproducibility +__________________________________________ + +Traditional approach in software enginering - reproduce everything from code. +This is nice and clean way to build software from source code and it is heavely utilised by Makefile and it's analogs. +The approach is based on some implicit assumptions: + +* Project consists on a large amount of small source code files +* Each code file can be processed (compiled) separately into some object file +* The final result (an application file) is a combination of these object files +* It is easy derive what was changed from the last reproduction (last make run) +* If only a few files were changed it is easy to rebuild only this subset of files and build a final result. + +Makefile tool and it's analogs do a good job in recognizing the small changes (step 4), rebuilding small parts of the project and constrcting them together into a single result. + + +Data science project +____________________ + + +Two reproducibility philosophies +________________________________ + + + +There are two different reproducibility "philosophies": +* Versioning only code. +* Versioning code and data. + +Basic assumption: Data and object files can be easily derived from code. + + + +______________________ +DVC is a command line tool that works on top of an existing Git repository. + +Using traditional UNIX-tool terminology DVC might be treated as a Makefile for data projects which +1. DVC never rebuilds a target if it was already built +2. DVC + + ======================== Getting Started with DVC ======================== @@ -97,14 +142,14 @@ Not only can DVC streamline your work into a single, reproducible environment, i Installation ============ -Operatio system dependent packages is the recommended way of installing DVC. +Operation system dependent packages is the recommended way of installing DVC. However, you can install DVC from Python repositories using **pip** command or install development version from DVC git repository. Packages ======== DVC installation packages available for Mac OS, Linux and Windows platforms. -You can download the packages at https://github.com/dataversioncontrol/dvc/releases/tag/0.8.7.2-travis-macos-release-test +You can download the packages at https://github.com/dataversioncontrol/dvc/releases/ Python pip ========== @@ -141,11 +186,11 @@ DVC Files and Directories Once installed, dvc will populate its installation folder (hereinafter referred to as .dvc) with essential shared and internal files and folders will be stored * **.dvc/config** - This is a configuration file. - The config file can be edited directly or indirectly using command **dvc config name value**. -* **.dvc/state** - this directory contains DVC state files including DAG (direct acyclic graph) of all dependencies of the commands in your DVC repositories + The config file can be edited directly or indirectly using command **dvc config NAME VALUE**. * **.dvc/cache** - the cache directory will contain your data files (the data directories of DVC repositories will only contain symlinks to the data files in the global cache). **Note:** DVC includes the cache directory to **.gitignore** file during the initilization. And no data files (with actual content) will be pushed to Git repository, only data file symlinks and commands to reproduce them. +* **.dvc/state** - this file is ceated for optimization. The file contains data files checksum and timestemps. Working with Cloud Data Storages From dd1e9abfcadd7df275f0cc1b49d1c7705d06c536 Mon Sep 17 00:00:00 2001 From: Dmitry Date: Tue, 23 Jan 2018 09:31:32 -0800 Subject: [PATCH 2/4] doc: improve Intor --- docs/doc.rst | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/docs/doc.rst b/docs/doc.rst index dd73568ee..9083124c2 100644 --- a/docs/doc.rst +++ b/docs/doc.rst @@ -15,32 +15,43 @@ __________________________________________ Traditional approach in software enginering - reproduce everything from code. This is nice and clean way to build software from source code and it is heavely utilised by Makefile and it's analogs. + The approach is based on some implicit assumptions: -* Project consists on a large amount of small source code files -* Each code file can be processed (compiled) separately into some object file -* The final result (an application file) is a combination of these object files -* It is easy derive what was changed from the last reproduction (last make run) -* If only a few files were changed it is easy to rebuild only this subset of files and build a final result. +1. Project consists of a large amount of small source code files. +2. Each code file can be processed (compiled) separately into some object file and this is a fast. +3. The final result (an application file) is a combination of these object files. +4. Building the the entire project form scratch is slow. + +Makefile and it's analogs optimize the entire project building stage (5) by identifying changed files and quickly processing only this small subset of files (4). +Because of (5) + +4. It is easy to derive what was changed from the last reproduction (last make run). +5. If only a few files were changed it is easy to rebuild only this subset of files and build a final result. Makefile tool and it's analogs do a good job in recognizing the small changes (step 4), rebuilding small parts of the project and constrcting them together into a single result. -Data science project -____________________ +Data science project reproducibility +____________________________________ + +Machine learning (ML) modleing process is slightly different from software engineering projects. + +1. [Dependencies tree is deep, not wide] To get a final result you need to go through a process with many dependent steps or stages. This is ML pipeline. +2. [Many steps are slow] Long processing time for many of operations: data cleaning is slow because of size of input raw files, model training is slow because of advanced ML algorithms. +3. [] + +IN PROGRESS... Two reproducibility philosophies ________________________________ - - There are two different reproducibility "philosophies": * Versioning only code. * Versioning code and data. -Basic assumption: Data and object files can be easily derived from code. - +Makefile versions only code. ______________________ From 0fa2e158e0d1ca6cede1fb89e7d05b3fa235fffc Mon Sep 17 00:00:00 2001 From: Dmitry Date: Sun, 28 Jan 2018 00:38:14 -0800 Subject: [PATCH 3/4] Doc: update command reference section --- docs/doc.rst | 613 ++++++++++++++++++++++++++++----------------------- 1 file changed, 333 insertions(+), 280 deletions(-) diff --git a/docs/doc.rst b/docs/doc.rst index 9083124c2..0fe5b31ce 100644 --- a/docs/doc.rst +++ b/docs/doc.rst @@ -267,18 +267,15 @@ Instead of manual file modification we recommend to run corresponded commands:: Using DVC Commands ================== -Since DVC is a command-line application, the appropriate method of use of DVC commands is essential to properly utilize it to the benefit of your machine learning projects. - +DVC is a command-line tool. The typical method of use of DVC is as follows -* You initialize a local DVC repository with **dvc init** command -* You pull or import data files into DVC repository either with **dvc pull** command or via external process invoked by **dvc run** command -* You clone a git repo with the code of your ML application pipeline -* You execute the steps in your ML pipeline as needed (**dvc run** command is often used to run respective processes/steps of your ML pipeline) -* You use **dvc repro** command to quickly reproduce your ML pipeline on a new iteration, once either the data item files or the source code of your ML application are modified -* You push the results of calculations back to your data storage using **dvc push** command - -**Note:** please refer to “Further Reading” section to see in-depth articles and tutorials on the end-to-end ML pipeline setup with DVC for real-world ML projects in Python and R. +* In an existing Git repository initialize a DVC repository with **dvc init** command. +* Copy source files for modeling into the repository (without checking out to Git) and convert the files in DVC data files with **dvc add** command. +* Process source data files by your data processing and modeling code through **dvc run** command. In this command generated DVC files to describe these processing steps. +* Use **--outs** option to specify **dvc run** command outputs which should be to be converted to DVC data files after the code is completed. +* You clone a git repo with the code of your ML application pipeline. However, it does not copy DVC cache. Use cloud storage settings and **dvc push** command to share the cache (data). +* You use **dvc repro** command to quickly reproduce your pipeline on a new iteration, once either the data item files or the source code of your ML application are modified. ======================== DVC Commands Cheat Sheet @@ -288,20 +285,19 @@ Below is the quick summary of the most important commands of DVC * **dvc -h** - Show how to use DVC and show the list of commands. * **dvc CMD -h** - Display help to use a specific DVC command (CMD) -* **dvc init** - Initialize a new local DVC repository in a folder of your choice (this folder should be already initialized either as a local git repository or a clone of a remote git repository) -* **dvc run** - Run an external command (for example, launch Python runtime with a python script to execute as its argument) -* **dvc pull** - Pull data files from the cloud (cloud settings for your DVC environment should be already configured prior to using this command). -* **dvc push** - Push data files to the cloud (cloud settings for your DVC environment should be already configured prior to using this command). -* **dvc status** - Show status of a data file in the DVC repository -* **dvc repro** - Reproduce the entire ML pipeline (or its part) where affected changes relate to the arguments passed (for example, rerun machine learning models where a changed data file is used as an input) -* **dvc remove** - Remove data items (files or/and folders) from the local DVC repository storage -* **dvc import** - Import a data file into a local DVC repository -* **dvc lock** - Lock files in the DVC repository -* **dvc gc** - Do garbage collection and clear DVC cache -* **dvc target** - Set default target -* **dvc ex** - Execute experimental commands supported by DVC -* **dvc config** - Alter configuration settings of the DVC repository (as specified in dvc.conf) for the current session +* **dvc init** - Initialize a new DVC repository. +* **dvc add** - Add data file or data directory. The command converts regular files to DVC data files. +* **dvc checkout** - Checkout data files and dirs into working tree. The command should be executed after **git checkout** or cloning a repository. +* **dvc run** - Generate a DVC file from a given command and execute the command. The command dependencies and outputs should be specified. +* **dvc pull** - Pull data files from the cloud. Cloud settings for your DVC environment should be already configured prior to using this command. +* **dvc push** - Push data files to the cloud. Cloud settings should be already configured. +* **dvc status** - Show status of a data file in the DVC repository. +* **dvc repro** - Reproduce a stage of pipeline. Default stage file is **Dvcfile**. +* **dvc remove** - Remove data file (files or/and folders). +* **dvc gc** - Collect garbage by cleaning DVC cache. +* **dvc config** - Get or set configuration settings (as specified in dvc.conf). * **dvc show** - Show graphs. +* **dvc fsck** - Data file consistency check. ===================== DVC Command Reference @@ -310,392 +306,398 @@ DVC Command Reference init ==== -This command initializes a local DVC environment (repository) in a current Git repository. +This command initializes a DVC environment in a current Git repository. .. code-block:: shell :linenos: - usage: - - dvc init [-h] [-q] [-v] [-G] [--data-dir DATA_DIR] - [--cache-dir CACHE_DIR] [--state-dir STATE_DIR] - + usage: dvc init [-h] [-q] [-v] optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check and commits. - --data-dir DATA_DIR Data directory. - --cache-dir CACHE_DIR A well-formed path to the Cache directory. - --state-dir STATE_DIR A well-formed path to the State directory. + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. -Example. Creating a new Git repository and DVC:: +Example. Creating a new DVC repository:: $ mkdir tag_classifier $ cd tag_classifier $ git init - Initialized empty Git repository in /Users/dmitry/src/tag_classifier__3/.git/ + Initialized empty Git repository in /Users/dmitry/src/tag_classifier/.git/ $ dvc init - Directories .dvc/, data/, cache/, state/ were created - File .gitignore was created - Directory cache was added to .gitignore file + $ git status + On branch master + + Initial commit + + Changes to be committed: + + (use "git rm --cached ..." to unstage) + + new file: .dvc/.gitignore + new file: .dvc/config + + $ git commit -m 'Init DVC' + [master (root-commit) 2db4618] Init DVC + 2 files changed, 41 insertions(+) + create mode 100644 .dvc/.gitignore + create mode 100644 .dvc/config +add +==== + +Converts files and directories to DVC data files. + +The command doe the convertation from a *regular file* to DVC data file in a few steps: + +1. Calculate the file checksum. +2. Create a cache file in the cache dir *.dvc/cache* with the content of this file. +3. Create a corresponded DVC file. +4. Replace the file by a hardlink to the cache file. + +Also, to reduce time on recomputing the file checksum in future DVC stores the file last modification time, inode and the checksum into a global state file *.dvc/state*. +Next time, then the file chacksum will be needed DVC will try to get it from the file if the file was not modified. + +Note, this command does NOT copy any file content and run quickly even for a large files. +Step (2) from the above is also made by hardlinks movement, not file content. +The only haavy step is (1) which requires checksum calculation. + +For directories the command does the same steps for each file recursively. +To keep information about the directory structure a corresponded directory will be created in the cache *.dvc/cache*. + +.. code-block:: shell + :linenos: + + usage: dvc add [-h] [-q] [-v] targets [targets ...] + + optional arguments: + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + +Examples: + +Convert files into data files:: + + $ mkdir raw + $ cp ~/Downloads/dataset/* raw + $ ls raw + Badges.xml PostLinks.xml Votes.xml + $ dvc add raw/Badges.tsv raw/PostLinks.tsv raw/Votes.tsv + $ ls raw + Badges.xml PostLinks.xml Votes.xml + Badges.xml.dvc PostLinks.xml.dvc Votes.xml.dvc + +Note, DVC files were created. + + +checkout +======== + +Checkout data files from cache. +This command has to be called after *git checkout* since Git does not handle DVC data files. + +The command restores data files from cache to working tree and removes data files that are not belog to the current working tree anymore. + +Note, this command does NOT copy any files - DVC uses hardlinks to perform the data file restoring. +This is crucial for large files where checking out (copiyng) 50Gb file might take a few minutes. +For DVC it will take less than a second to restore 50Gb data file. + + +.. code-block:: shell + :linenos: + + usage: dvc checkout [-h] [-q] [-v] + + optional arguments: + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + +Examples. + +Checking out a branch example:: + + $ git checkout input_100K + $ dvc checkout + $ Remove 'data/model.p' + $ Remove 'data/matrix-train.p' + $ 'data/Posts-train.tsv': cache file not found + +DVC does not report in the output which data files were restored. +However, it reports removed files and files which DVC was unable to restore because of missing cache. +To restore file with missing cache reproduction command should be called or cache can be pulled from a cloud. + +It might be convinient to assign Git hook to *git checkout* comman:: + + $ echo 'dvc checkout' > .git/hooks/post-checkout + $ chmod +x .git/hooks/post-checkout + $ git checkout input_100K # dvc checkout is not needed anymore + $ Remove 'data/model.p' + $ Remove 'data/matrix-train.p' + $ 'data/Posts-train.tsv': cache file not found + run === -This command executes is used to execute the steps in your ML pipeline, for instance -* Running a python or R script -* Running a database SQL script -* Etc. +Generate a stage file from a given command and execute the command. +The command dependencies and outputs should be specified. + +By default stage file name is **.dvc** where **** is file name of a first output. + +For example, launch Python with a given python script and arguments. Or R script by Rscript command. .. code-block:: shell :linenos: - usage: dvc run [-h] [-q] [-v] [-G] [--stdout STDOUT] [--stderr STDERR] - [-i INPUT] [-o OUTPUT] [-c CODE] [--shell] [-l] - command [args] - + usage: dvc run [-h] [-q] [-v] [-d DEPS] [-o OUTS] [-O OUTS_NO_CACHE] [-f FILE] + [-c CWD] [--no-exec] + ... + positional arguments: - command Command to execute - args Arguments of a command (optional; it can be a list of the command-line arguments of command separated by spaces) - + command Command or command file to execute + optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check and commits. - --stdout STDOUT Output std output to a file. - --stderr STDERR Output std error to a file. - -i INPUT, --input INPUT Declare input data items for reproducible cmd. - -o OUTPUT, --output OUTPUT Declare output data items for reproducible cmd. - -c CODE, --code CODE Code dependencies which produce the output. - --shell Shell command - -l, --lock Lock data item - disable reproduction. + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + -d DEPS, --deps DEPS Declare dependencies for reproducible cmd. + -o OUTS, --outs OUTS Declare output data file or data directory. + -O OUTS_NO_CACHE, --outs-no-cache OUTS_NO_CACHE + Declare output regular file or directory (sync to Git, + not DVC cache). + -f FILE, --file FILE Specify name of the state file + -c CWD, --cwd CWD Directory to run your command and place state file in + --no-exec Only create stage file without actually running it Examples: -Execute a Python script as a DVC ML pipeline step:: +Execute a Python script as a DVC ML pipeline step. Stage file was not specified. So, **model.p.dvc** stage file will be created:: - $ # Train ML model out of the training dataset. 20170426 is another seed value. - $ dvc run python code/train_model.py data/matrix-train.p 20170426 data/model.p + $ # Train ML model out of the training dataset. 20180226 is a seed value. + $ dvc run -d matrix-train.p -d train_model.py -o model.p python train_model.py matrix-train.p 20180226 model.p Execute an R script as a DVC ML pipeline step:: - dvc run Rscript code/parsingxml.R data/Posts.xml data/Posts.csv + $ dvc run -d parsingxml.R -d Posts.xml -o Posts.csv Rscript parsingxml.R Posts.xml Posts.csv Extract an XML file from an archive to data subfolder:: - dvc run tar zxf data/Posts.xml.tgz -C data/ + $ mkdir data + $ dvc run -d Posts.xml.tgz -o data/Posts.xml tar zxf Posts.xml.tgz -C data/ push ==== -This command pushes data files to the cloud storage you configured for DVC. +This command pushes all data files caches related to the current Git branch to the cloud storage. +Cloud storage settings need to be configured. +See cloud storage configuration. .. code-block:: shell :linenos: - usage: dvc push [-h] [-q] [-v] [-G] [-j JOBS] targets [targets ...] - - positional arguments: - targets File or directory to sync. + usage: dvc push [-h] [-q] [-v] [-j JOBS] optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check - and commits. - -j JOBS, --jobs JOBS Number of jobs to run simultaneously. + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + -j JOBS, --jobs JOBS Number of jobs to run simultaneously. Examples: -Push all files from the current DVC snapshot to cloud:: +Push all data files caches from the current Git branch to cloud:: - $ dvc push data/ + $ dvc push + (1/8): [########################################] 100% 72271bebdf053178a5cce48b4 + (2/8): [########################################] 100% d7208b910d1a40fedc2da5a44 + (3/8): [########################################] 100% 7f6ed2919af9c9e94c32ea13d + (4/8): [########################################] 100% 5988519f8465218abb23ce0e0 + (5/8): [########################################] 100% 11de13709a78379d253a3d0f5 + (6/8): [########################################] 100% 3f9c7c3ae51db2eed7ba99e6e + (7/8): [########################################] 100% cfdaa4bba57fa07d81ff96685 + (8/8): [####################### ] 57% 1de6178a9dd844e249ba05414 + pull ==== -This command pulls data from the cloud storage you configured for DVC. +This command pulls all data files caches from the cloud storage. +Cloud storage settings need to be configured. .. code-block:: shell :linenos: - usage: dvc pull [-h] [-q] [-v] [-G] [-j JOBS] targets [targets ...] - - positional arguments: - targets File or directory to sync. - + usage: dvc pull [-h] [-q] [-v] [-j JOBS] + optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check and commits. - -j JOBS, --jobs JOBS Number of jobs to run simultaneously. - + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + -j JOBS, --jobs JOBS Number of jobs to run simultaneously. Examples: -Pull all files from the current DVC snapshot to cloud:: - - $ dvc pull data/ +Pull all files from the current Git branch:: + $ dvc pull + (1/8): [########################################] 100% 54a6f1787490ba13fb811a46b + (2/8): [########################################] 100% 5806dc797c08fb6ddd5d97d46 + (3/8): [########################################] 100% 5988519f8465218abb23ce0e0 + (4/8): [########################################] 100% 7f6ed2919af9c9e94c32ea13d + (5/8): [########################################] 100% 11de13709a78379d253a3d0f5 + (6/8): [########################################] 100% c6f5a256d628e144db4181de8 + (7/8): [########################################] 100% 3f9c7c3ae51db2eed7ba99e6e + (8/8): [########################################] 100% cfdaa4bba57fa07d81ff96685 status ====== -This command shows status for data files in the DVC repository +Show mismatches between local cache and cloud cache. .. code-block:: shell :linenos: - usage: dvc status [-h] [-q] [-v] [-G] [-j JOBS] targets [targets ...] - - positional arguments: - targets File or directory to sync. - + usage: dvc status [-h] [-q] [-v] [-j JOBS] + optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check - and commits. - -j JOBS, --jobs JOBS Number of jobs to run simultaneously. + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + -j JOBS, --jobs JOBS Number of jobs to run simultaneously. Examples: -Get status of data in *training.csv* file:: +Show statuses:: - $ dvc status data/training.csv + $ dvc status + new file: /Users/dmitry/src/myrepo_1/.dvc/cache/62f8c2ba93cfe5a6501136078f0336f9 repro ===== -This command reproduces the that part of the ML pipeline that is dependent on the data or code file targeted by it. +Reproduce DVC file and all stages the file depends on (recursively). +Default file name is **Dvcfile**. +However, usually DVC files have any name and **.dvc** suffix. .. code-block:: shell :linenos: - usage: dvc repro [-h] [-q] [-v] [-G] [-f] [-s] [target [target ...]] + usage: dvc repro [-h] [-q] [-v] [-f] [-s] [targets [targets ...]] positional arguments: - target Data items to reproduce. + target DVC file to reproduce. optional arguments: -h, --help show this help message and exit -q, --quiet Be quiet. -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check and commits. -f, --force Reproduce even if dependencies were not changed. -s, --single-item Reproduce only single data item without recursive dependencies check. Examples: -Reproduce the part of the pipeline where *training.csv* data file is involved:: +Reproduce default stage file:: - $ dvc repro data/training.csv + $ dvc repro + Verifying data sources in 'data/Posts.xml.tgz.dvc' + Reproducing 'Posts.xml.dvc': + tar zxf data/Posts.xml.tgz -C data/ + Reproducing 'Posts.tsv.dvc': + python code/xml_to_tsv.py data/Posts.xml data/Posts.tsv python + Reproducing 'Posts-train.tsv.dvc': + python code/split_train_test.py data/Posts.tsv 0.33 20170426 data/Posts-train.tsv data/Posts-test.tsv + Reproducing 'matrix-train.p.dvc': + python code/featurization.py data/Posts-train.tsv data/Posts-test.tsv data/matrix-train.p data/matrix-test.p + Reproducing 'model.p.dvc': + python code/train_model.py data/matrix-train.p 20170426 data/model.p + +Reproduce the part of the pipeline (from above) where *Posts.tsv.dvc* is target DVC file:: + + $ dvc repro Posts.tsv.dvc + Reproducing 'Posts.xml.dvc': + tar zxf data/Posts.xml.tgz -C data/ + Reproducing 'Posts.tsv.dvc': + python code/xml_to_tsv.py data/Posts.xml data/Posts.tsv python remove ====== -This command removes a data item from the data directory of a DVC repository. +Remove data file or data directory. .. code-block:: shell :linenos: - usage: dvc remove [-h] [-q] [-v] [-G] [-l] [-r] [-c] [target [target ...]] + usage: dvc remove [-h] [-q] [-v] targets [targets ...] positional arguments: - target Target to remove - file or directory. + targets Target to remove - file or directory. optional arguments: -h, --help show this help message and exit -q, --quiet Be quiet. -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check - and commits. - -l, --keep-in-cloud Do not remove data from cloud. - -r, --recursive Remove directory recursively. - -c, --keep-in-cache Do not remove data from cache. Examples: -Remove *training.csv* data file from the DVC repository:: +Remove *matrix-train.p* data file:: - $ dvc remove data/training.csv - -import -====== - -This command imports a new data file to the data directory of the DVC repository. - -.. code-block:: shell - :linenos: - - usage: dvc import [-h] [-q] [-v] [-G] [-l] [-j JOBS] [-c] - input [input ...] output - - positional arguments: - input Input file/files. - output Output file/directory. - - optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check and commits. - -l, --lock Lock data item - disable reproduction. - -j JOBS, --jobs JOBS Number of jobs to run simultaneously. - -c, --continue Resume downloading file from url - -Examples: - -Download a file and put to data/ directory:: - - $ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/25K/Posts.xml.tgz data/ - -lock -==== - -This command is used to - -* Lock the data item in the DVC repository, protecting it from further changes -* Unlock the data item locked earlier (switch -u is added in this case) - -.. code-block:: shell - :linenos: - - usage: dvc lock [-h] [-q] [-v] [-G] [-l] [-u] [files [files ...]] - - positional arguments: - files Data items to lock or unlock. - - optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check - and commits. - -l, --lock Lock data item - disable reproduction. - -u, --unlock Unlock data item - enable reproduction. - -Examples. + $ dvc remove matrix-train.p -Lock *data/Posts.xml* file:: - - $ dvc lock data/Posts.xml - -Unlock a previously locked *data/Posts.xml* file:: - - $ dvc lock -u data/Posts.xml - -**Notes** - -* If you invoke lock command with *-u* switch against a locked target file, it will be unlocked -* Adding *-l* switch to any other command where *-l* switch is enabled will automatically lock/unlock the target files (much like you do with a separate lock command against that target) gc === -This command collects the garbage in DVC environment. -It is especially important when you work with large data files. -Under such a condition, keeping previous versions of the large files may slow down performance/drain the disk quota thus swift removing of unnecessary files will be beneficial. + +This command collects the garbage - removes unused cache files based on the current Git branch. +So, if a data file was created in a different branch then it is going to be removed by command. +If a data file has a few versions (and, correspondingly, caches) - all the chaches except the current one will be removed. .. code-block:: shell :linenos: - usage: dvc gc [-h] [-q] [-v] [-G] [-l] [-r] [-c] [target [target ...]] - - positional arguments: - target Target to remove - file or directory. + age: dvc gc [-h] [-q] [-v] optional arguments: -h, --help show this help message and exit -q, --quiet Be quiet. -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check and commits. - -l, --keep-in-cloud Do not remove data from cloud. - -r, --recursive Remove directory recursively. - -c, --keep-in-cache Do not remove data from cache. -Examples: +Clean up example:: -Remove all cloud not in the current DVC snapshot:: - - $ dvc gc data/ - -Remove all versions of *data/Posts.xml* file (but the latest one) from the local cache directory but keep it in the cloud storage:: - - $ dvc gc data/Posts.xml --keep-in-cloud - -target -====== - -This command sets the default target for the current DVC repository. - -.. code-block:: shell - :linenos: - - usage: dvc target [-h] [-q] [-v] [-G] [-u] [target_file] - - positional arguments: - target_file Target data item. - - optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check and commits. - -u, --unset Reset target. - -Examples: - -Set *data/Posts.xml* file as a default target in the current DVC repository:: - - $ dvc target data/Posts.xml - -ex -== - -This command is designed for risky enthusiasts who would like to try the newest capabilities of DVC which are still under active development. - -**Note:** It is provided for your reference and early try only. DVC development team does not provide any warranty as for this piece of DVC functionality to work in a stable manner, in your environment. We do not recommend you to use the experimental functionality in production mode unless you really clear as for what you are going to do. - -.. code-block:: shell - :linenos: - - usage: dvc ex [-h] [-q] [-v] [-G] {cloud} ... - - positional arguments: - {cloud} Use dvc cloud CMD --help for command-specific help - cloud Cloud manipulation - - optional arguments: - -h, --help show this help message and exit - -q, --quiet Be quiet. - -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check - and commits. + $ du -sh .dvc/cache/ + 7.4G .dvc/cache/ + $ dvc gc + '.dvc/cache/27e30965256ed4d3e71c2bf0c4caad2e' was removed + '.dvc/cache/2e006be822767e8ba5d73ebad49ef082' was removed + '.dvc/cache/2f412200dc53fb97dcac0353b609d199' was removed + '.dvc/cache/541025db4da02fcab715ca2c2c8f4c19' was removed + '.dvc/cache/62f8c2ba93cfe5a6501136078f0336f9' was removed + '.dvc/cache/7c4521365288d69a03fa22ad3d399f32' was removed + '.dvc/cache/9ff7365a8256766be8c363fac47fc0d4' was removed + '.dvc/cache/a86ca87250ed8e54a9e2e8d6d34c252e' was removed + '.dvc/cache/f64d65d4ccef9ff9d37ea4cf70b18700' was removed + $ du -sh .dvc/cache/ + 3.1G .dvc/cache/ config ====== -This command is designed to overwrite some configuration options for just this session of DVC (as you remember, default configuration values are specified in **dvc.conf** , which is located in root of your DVC installation folder). +Get or set config options. This command reads and owerwrites DVC config file *.dvc/config*. + .. code-block:: shell :linenos: - usage: dvc config [-h] [-q] [-v] [-G] [-u] name [value] + usage: dvc config [-h] [-q] [-v] [-u] name [value] positional arguments: name Option name @@ -705,48 +707,99 @@ This command is designed to overwrite some configuration options for just this s -h, --help show this help message and exit -q, --quiet Be quiet. -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check - and commits. -u, --unset Unset option Examples: -Overwrite the value of DataDir configuration option with *etc/data* for a current dvc session only:: +Specify an option name to get the option value from config file:: + $ dvc config config Global.Cloud + AWS - $ dvc config DataDir etc/data +Overwrite the value:: + + $ dvc config Global.Cloud GCP + $ git add .dvc/config + $ git commit -m 'Change cloud to GCP' + [input_100K a4c985f] Change cloud to GCP + 1 file changed, 1 insertion(+), 1 deletion(-) show ==== -This command is used to display either pipeline or workflow image for your current ML project managed by DVC +Generate pipeline image for your current project. .. code-block:: shell :linenos: - usage: dvc show [-h] [-q] [-v] [-G] {pipeline,workflow} ... + usage: dvc show [-h] [-q] [-v] {pipeline} ... positional arguments: - {pipeline,workflow} Use dvc show CMD --help for command-specific help + {pipeline} Use `dvc show CMD` --help for command-specific help pipeline Show pipeline image - workflow Show workflow image optional arguments: -h, --help show this help message and exit -q, --quiet Be quiet. -v, --verbose Be verbose. - -G, --no-git-actions Skip all git actions including reproducibility check - and commits. Examples: -Show the workflow image for the ML project in your current DVC repository:: - - $ dvc show workflow - -Show the pipeline image for the ML project in your current DVC repository:: +Show the pipeline image:: $ dvc show pipeline +fsck +==== + +Data file consistency check. +By default the commands outputs statuses of all corrupted data files if any. +Use *--all* option to see statuses of all data files statuses. + +The command checks: +1. Cache file name which equals to the file content checksum on the time when DVC created the file. +2. Checksum from local state file. +3. Checksum regarding DVC files. +4. Actual recomputed checksum. This is computation heavy command for large data file. Enabled only by *--physical* option. + +Data file is considered as corrupted if one of the checksum does not match all others. + + +.. code-block:: shell + :linenos: + + dvc fsck [-h] [-q] [-v] [-p] [-a] [targets [targets ...]] + + positional arguments: + targets Data files to check + + optional arguments: + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + -p, --physical Compute actual md5 + -a, --all Show all data files including correct ones + +Examples. + + +Check list of corrupted data files:: + + $ dvc fsck --physical + File data/matrix-test.p: + Error status: Checksum missmatch!!! + Actual checksum: 7c4521365288d69a03fa22ad3d399f32 + Cache file name: 7c4521365288d69a03fa22ad3d399f32 + Local state checksum: 7c4521365288d69a03fa22ad3d399f32 + Local state mtime: 1517048086.0 + Actual mtime: 1517048086.0 + Stage file: eval_auc.txt.dvc + Checksum: 7c4521365288d69a03fa22ad3d399f32 + Type: Dependency + Stage file: matrix-train.p.dvc + Checksum: 7c4521365288d69a03fa22ad3d399f32 + Type: Output + Use cache: true + Common Arguments =========================================== From eb1dda44755b8ad7a7a2f81f2bfc966e600b8961 Mon Sep 17 00:00:00 2001 From: Dmitry Date: Sun, 28 Jan 2018 00:49:21 -0800 Subject: [PATCH 4/4] Doc: updata Installation and move Configuration to the end --- docs/doc.rst | 174 +++++++++++++++++++++++++++------------------------ 1 file changed, 91 insertions(+), 83 deletions(-) diff --git a/docs/doc.rst b/docs/doc.rst index 0fe5b31ce..0dc707270 100644 --- a/docs/doc.rst +++ b/docs/doc.rst @@ -154,10 +154,10 @@ Installation ============ Operation system dependent packages is the recommended way of installing DVC. -However, you can install DVC from Python repositories using **pip** command or install development version from DVC git repository. +Some other methods of installation are provided. -Packages -======== +OS packages +=========== DVC installation packages available for Mac OS, Linux and Windows platforms. You can download the packages at https://github.com/dataversioncontrol/dvc/releases/ @@ -173,6 +173,13 @@ Another option to deploy DVC to your machine is to use its standard Python pip p It will work in *Anaconda’s* command prompt tool. As of the moment, DVC does not provide a special installation package for a native *Anaconda* package manager (that is, *conda*). +Homebrew Cask +============= + +Mac OS users can install DVC by **brew** command:: + + $ brew cask install dataversioncontrol/homebrew-dvc/dvc + Development Version =================== @@ -183,86 +190,6 @@ If you like to pull the latest version of DVC from the master branch in its repo This command will automatically upgrade your DVC version in case it is behind the latest version in the master branch of the github repo. -============= -Configuration -============= - -Once you install DVC, you should be able to start using it (in its local setup) immediately. - -However, you can optionally proceed to further configure DVC (especially if you intend to use it in a Cloud-based scenario). - -DVC Files and Directories -========================= - -Once installed, dvc will populate its installation folder (hereinafter referred to as .dvc) with essential shared and internal files and folders will be stored - -* **.dvc/config** - This is a configuration file. - The config file can be edited directly or indirectly using command **dvc config NAME VALUE**. -* **.dvc/cache** - the cache directory will contain your data files (the data directories of DVC repositories will only contain symlinks to the data files in the global cache). - **Note:** DVC includes the cache directory to **.gitignore** file during the initilization. And no data files (with actual content) will be pushed to Git repository, - only data file symlinks and commands to reproduce them. -* **.dvc/state** - this file is ceated for optimization. The file contains data files checksum and timestemps. - - -Working with Cloud Data Storages -====================================================== - -Using DVC with Cloud-based Data Storages is an optional feature. -By default, DVC is configured to use local data storage only (.dvc/cache directory), - and it enables basic DVC usage scenarios out of the box. - -DVC can use cloud storages as a common file storage. -With cloud storage you might use models and data file which were created by your team members - without spending time and resources for re-building models and re-processing data files. - -As of this version, DVC supports two types of cloud-based data storage providers: - -* **AWS** - Amazon Web Services -* **GCP** - Google Cloud Provider - -The subsections below explain how to configure DVC to use of the data cloud storages above. - -Using AWS Cloud ---------------- - -For using AWS as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively - -* **Cloud = AWS** in *Global* section. -* **StoragePath = /mybucket/dvc/tag_classifier** in **AWS** section - path to a cloud storage bucket and directory in the bucket. -* **CredentialPath = ~/aws/credentials** in **AWS** section - path to AWS credentials in your local machine (AWS cli command line tools creates this directory). - In Mac, default value is *~/.aws/credentials*, and it is *%USERPATH%/.aws/credentials* in Windows - - -**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to AWS"** - -Instead of manual file modification we recommend to run corresponded commands:: - - $ dvc config Global.Cloud AWS # This step is not needed for new DVC repositories - $ dvc config AWS.StoragePath /mybucket/dvc/tag_classifier - $ dvc config AWS.CredentialPath ~/.aws/credentials # Not needed if aws cli is instelled to default path - $ dvc config AWS.CredentialSection default # Not needed if you have only one AWS account - $ git commit -am "Change cloud to AWS" - - -Using Google Cloud ------------------- - -For using GCP (Google Cloud Provider) as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively - -* **Cloud = GCP** in *Global* section. -* **StoragePath = /mybucket/dvc/tag_classifier** in GCP section - this option has the same meaning as AWS one above. Run **dvc config GCP.StoragePath /my/path/to/a/bucket** -* **ProjectName = MyCloud** - a GCP specific project name. - -**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to GCP"** - -Instead of manual file modification we recommend to run corresponded commands:: - - $ dvc config Global.Cloud GCP - $ dvc config GCP.StoragePath /mybucket/dvc/tag_classifier - $ dvc config GCP.ProjectName MyCloud - $ git commit -am "Change cloud to AWS" - - ================== Using DVC Commands ================== @@ -828,6 +755,87 @@ DVC can benefit from parallel processing and multiple processors/cores available The number of DVC jobs is 5 by default. In case you like to change it to any other reasonable value, you use *-j (--jobs)* option in DVC commands where it is applicable. + +============= +Configuration +============= + +Once you install DVC, you should be able to start using it (in its local setup) immediately. + +However, you can optionally proceed to further configure DVC (especially if you intend to use it in a Cloud-based scenario). + +DVC Files and Directories +========================= + +Once installed, dvc will populate its installation folder (hereinafter referred to as .dvc) with essential shared and internal files and folders will be stored + +* **.dvc/config** - This is a configuration file. + The config file can be edited directly or indirectly using command **dvc config NAME VALUE**. +* **.dvc/cache** - the cache directory will contain your data files (the data directories of DVC repositories will only contain symlinks to the data files in the global cache). + **Note:** DVC includes the cache directory to **.gitignore** file during the initilization. And no data files (with actual content) will be pushed to Git repository, + only data file symlinks and commands to reproduce them. +* **.dvc/state** - this file is ceated for optimization. The file contains data files checksum and timestemps. + + +Working with Cloud Data Storages +====================================================== + +Using DVC with Cloud-based Data Storages is an optional feature. +By default, DVC is configured to use local data storage only (.dvc/cache directory), + and it enables basic DVC usage scenarios out of the box. + +DVC can use cloud storages as a common file storage. +With cloud storage you might use models and data file which were created by your team members + without spending time and resources for re-building models and re-processing data files. + +As of this version, DVC supports two types of cloud-based data storage providers: + +* **AWS** - Amazon Web Services +* **GCP** - Google Cloud Provider + +The subsections below explain how to configure DVC to use of the data cloud storages above. + +Using AWS Cloud +--------------- + +For using AWS as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively + +* **Cloud = AWS** in *Global* section. +* **StoragePath = /mybucket/dvc/tag_classifier** in **AWS** section - path to a cloud storage bucket and directory in the bucket. +* **CredentialPath = ~/aws/credentials** in **AWS** section - path to AWS credentials in your local machine (AWS cli command line tools creates this directory). + In Mac, default value is *~/.aws/credentials*, and it is *%USERPATH%/.aws/credentials* in Windows + + +**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to AWS"** + +Instead of manual file modification we recommend to run corresponded commands:: + + $ dvc config Global.Cloud AWS # This step is not needed for new DVC repositories + $ dvc config AWS.StoragePath /mybucket/dvc/tag_classifier + $ dvc config AWS.CredentialPath ~/.aws/credentials # Not needed if aws cli is instelled to default path + $ dvc config AWS.CredentialSection default # Not needed if you have only one AWS account + $ git commit -am "Change cloud to AWS" + + +Using Google Cloud +------------------ + +For using GCP (Google Cloud Provider) as a data cloud storage for your DVC repositories, you should update **.dvc/config** options respectively + +* **Cloud = GCP** in *Global* section. +* **StoragePath = /mybucket/dvc/tag_classifier** in GCP section - this option has the same meaning as AWS one above. Run **dvc config GCP.StoragePath /my/path/to/a/bucket** +* **ProjectName = MyCloud** - a GCP specific project name. + +**Important:** do not forget to commit the config file change to Git: **git commit -am "Change cloud to GCP"** + +Instead of manual file modification we recommend to run corresponded commands:: + + $ dvc config Global.Cloud GCP + $ dvc config GCP.StoragePath /mybucket/dvc/tag_classifier + $ dvc config GCP.ProjectName MyCloud + $ git commit -am "Change cloud to AWS" + + =============== Further Reading ===============