mirror of
https://github.com/Stone-Red-Code/dvc.git
synced 2026-11-19 23:11:44 +01:00
Start new version of README
This commit is contained in:
@@ -5,299 +5,28 @@
|
||||
Git for data science projects. It streamlines your git code, your data (S3 and GCP) and dependencies to a single reproducible environment.
|
||||
|
||||
|
||||
# Tutorial
|
||||
# Introduction
|
||||
|
||||
## Install
|
||||
It is hardly possible in real life to develop a good machine learning model in a single pass. ML modeling is an iterative process and it is extremely important to keep track of your steps, dependencies between the steps, dependencies between your code and data files and all code running arguments. This becomes even more important and complicated in a team environment where data scientists’ collaboration takes a serious amount of the team’s effort.
|
||||
|
||||
Please download dvc from http://dataversioncontrol.com for your platform or
|
||||
install through source code from Github https://github.com/dmpetrov/dataversioncontrol.
|
||||

|
||||
|
||||
## Initialize dvc environment
|
||||
[Data Version Control](https://dataversioncontrol.com) or DVC is an open source tool which is designed to help data scientists keep track of their ML processes and file dependencies in the simple form of git-like commands: `dvc run python train_model.py data/train_matrix.p data/model.p`. Your existing ML processes can be easily transformed into reproducible DVC pipelines regardless of which programming language or tool was used.
|
||||
|
||||
DVC is an extension of the git command.
|
||||
It works on top of git command in existing git repositories.
|
||||
Let's create a new git repository for the tutorial perpose.
|
||||
This blog post walks you through an iterative process of building a machine learning model with DVC using [stackoverflow posts dataset](https://archive.org/details/stackexchange).
|
||||
The full pipeline could be built by running the bash code below.
|
||||
|
||||
```
|
||||
$ cd ~/src
|
||||
$ mkdir tag_classifier
|
||||
$ cd tag_classifier
|
||||
$ git init
|
||||
Initialized empty Git repository in /Users/dmitry/src/tag_classifier/.git/
|
||||
```
|
||||
# Installation
|
||||
|
||||
Initialize dvc environment:
|
||||
DVC could be installed via the Python Package Index (PyPI).
|
||||
|
||||
```
|
||||
$ dvc init
|
||||
Directories data/, .cache/ and .state/ were created
|
||||
File .gitignore was created
|
||||
Directory .cache was added to .gitignore file
|
||||
[Git] A new commit e7a4b0d was made in the current branch. Added files:
|
||||
[Git] A .gitignore
|
||||
[Git] A dvc.conf
|
||||
```
|
||||
|
||||
The output shows the actions that dvc has made.
|
||||
Special directories `data/`, `cache/` and `state/` were created to keep track of your external data which are not stored in git but have to be under users control.
|
||||
These three directories are mondatory for any dvc project.
|
||||
|
||||
Names of the directory might be changed by the `dvc init` command line parameters `--data-dir`, `--cache-dir` and `--state-dir` respectively.
|
||||
All these special directories names are going to be saved in `dvc.conf` config file as a result of
|
||||
the init command and should not be changed over dvc project life time.
|
||||
|
||||
As you might see from the output `.gitignore` file was created (or modified if you already had one).
|
||||
This is result of including cache directory to this file and also `.dvc.conf.lock` file.
|
||||
Meaning of these changes will be explain later in the tutorial.
|
||||
|
||||
DVC commits all the chenges to the git repository at the end of successful run.
|
||||
The commit step might be skipped by command line parameter `--skip-git-actions`.
|
||||
This parameter might be applied to any dvc command.
|
||||
|
||||
## Importing data files
|
||||
|
||||
Data file has to be imported in dvc `data/` directory before using it.
|
||||
|
||||
```
|
||||
$ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/small/Posts-100k.xml.tgz data/
|
||||
[Git] A new commit b69ab31 was made in the current branch. Added files:
|
||||
[Git] A .state/Posts.xml.tgz.state
|
||||
[Git] A data/Posts.xml.tgz
|
||||
```
|
||||
|
||||
The data file was downloaded and now it is available in the data directory.
|
||||
However, it is only a part of the work that dvc has done.
|
||||
|
||||
### Cache files
|
||||
|
||||
If you look carefully at the file you could see that this is actually a symlink to
|
||||
a cache directory file Badges.xml_a5ed2b9:
|
||||
```
|
||||
$ ls -l data/Posts-100k.xml.tgz
|
||||
lrwxr-xr-x 1 dmitry staff 31B Apr 21 17:56 data/Posts.xml.tgz@ -> ../.cache/Posts.xml.tgz_9185f92
|
||||
```
|
||||
|
||||
Instead of importing the file to `data/Badges.xml` the system imports the file in the cache directory
|
||||
and creates a symlink `data/Badges.xml`.
|
||||
The file in the cache directory `cache/Badges.xml_a5ed2b9` has the same name as
|
||||
the file in the data directory but extended by a suffix `_a5ed2b9` which
|
||||
correspondes to a git snapshot checksum on the time when the file was created.
|
||||
|
||||
### State files
|
||||
|
||||
In addition to the cache file dvc creates a state file in the `state/` directory for each data file:
|
||||
```
|
||||
$ ls -l .state/Posts-100k.xml.tgz.state
|
||||
-rw-r--r-- 1 dmitry staff 437B Apr 21 17:56 .state/Posts.xml.tgz.state
|
||||
```
|
||||
|
||||
The state file contains information for file reproducibility.
|
||||
|
||||
|
||||
## Structure data
|
||||
|
||||
|
||||
```
|
||||
$ dvc run tar zxf data/Posts-100k.xml.tgz -C data/
|
||||
[Git] A new commit fbff960 was made in the current branch. Added files:
|
||||
[Git] A .state/Posts.xml.state
|
||||
[Git] A data/Posts.xml
|
||||
$ cd ..
|
||||
```
|
||||
|
||||
```
|
||||
$ dvc run cp data/Posts-100k.xml data/Posts.xml
|
||||
```
|
||||
|
||||
Full file is here https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/posts_to_tsv.py
|
||||
|
||||
```
|
||||
$ mkdir code
|
||||
$ wget -nv -P code/ https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/posts_to_tsv.py
|
||||
2017-04-22 12:10:13 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/posts_to_tsv.py [1689/1689] -> "code/posts_to_tsv.py" [1]
|
||||
```
|
||||
|
||||
```
|
||||
$ git add code/posts_to_tsv.py
|
||||
$ git commit -m 'Post to tsv script'
|
||||
[master 4a581d9] Post to tsv script
|
||||
1 file changed, 51 insertions(+)
|
||||
create mode 100644 code/posts_to_tsv.py
|
||||
To install using pip:
|
||||
```bash
|
||||
pip install dvc
|
||||
```
|
||||
|
||||
|
||||
```python
|
||||
def process_posts(fd_in, fd_out):
|
||||
num = 1
|
||||
for line in fd_in:
|
||||
try:
|
||||
attr = xml.etree.ElementTree.fromstring(line).attrib
|
||||
items = (
|
||||
attr.get('Id', ''),
|
||||
attr.get('PostTypeId', ''),
|
||||
attr.get('CreationDate', ''),
|
||||
attr.get('Score', ''),
|
||||
attr.get('ViewCount', ''),
|
||||
attr.get('OwnerUserId', ''),
|
||||
attr.get('LastActivityDate', ''),
|
||||
attr.get('AnswerCount', ''),
|
||||
attr.get('CommentCount', ''),
|
||||
attr.get('FavoriteCount', ''),
|
||||
attr.get('Title', ''),
|
||||
attr.get('Body', '')
|
||||
)
|
||||
fmt = ('{}\t' * len(items))[:-1] + '\n'
|
||||
|
||||
fd_out.write(fmt.format(*items))
|
||||
|
||||
num += 1
|
||||
except Exception as ex:
|
||||
sys.stderr.write('Error in line {}: {}'.format(num, ex))
|
||||
```
|
||||
|
||||
```
|
||||
$ dvc run python code/posts_to_tsv.py data/Posts.xml data/Posts.tsv
|
||||
[Git] A new commit dc54e5a was made in the current branch. Added files:
|
||||
[Git] A .state/Posts.tsv.state
|
||||
[Git] A data/Posts.tsv
|
||||
```
|
||||
|
||||
```
|
||||
$ echo "cut -d$'\t' -f 1,11-13 \$@" > code/cut_fr.sh
|
||||
$ chmod a+x code/cut_fr.sh
|
||||
$ git add code/cut_fr.sh
|
||||
$ git commit -m 'Cut posts features'
|
||||
[master aa9c6c1] Cut posts features
|
||||
1 file changed, 1 insertion(+)
|
||||
create mode 100755 code/cut_fr.sh
|
||||
```
|
||||
|
||||
Reducing file size:
|
||||
```
|
||||
$ dvc run --stdout data/Posts-fr.tsv bash ./code/cut_fr.sh data/Posts.tsv
|
||||
[Git] A new commit 39a9544 was made in the current branch. Added files:
|
||||
[Git] A .state/Posts-fr.tsv.state
|
||||
[Git] A data/Posts-fr.tsv
|
||||
```
|
||||
|
||||
Looking for data items:
|
||||
```
|
||||
$ ls -l data
|
||||
total 32
|
||||
lrwxr-xr-x 1 dmitry staff 30B Apr 21 23:36 Posts-fr.tsv@ -> ../.cache/Posts-fr.tsv_e1233ed
|
||||
lrwxr-xr-x 1 dmitry staff 27B Apr 21 18:53 Posts.tsv@ -> ../.cache/Posts.tsv_b58a46d
|
||||
lrwxr-xr-x 1 dmitry staff 27B Apr 21 18:15 Posts.xml@ -> ../.cache/Posts.xml_4a581d9
|
||||
lrwxr-xr-x 1 dmitry staff 31B Apr 21 17:56 Posts.xml.tgz@ -> ../.cache/Posts.xml.tgz_9185f92
|
||||
```
|
||||
|
||||
Looking for the actual size:
|
||||
```
|
||||
$ ls -lL data
|
||||
total 7744240
|
||||
-rw-r--r-- 1 dmitry staff 927M Apr 21 23:36 Posts-fr.tsv
|
||||
-rw-r--r-- 1 dmitry staff 990M Apr 21 18:53 Posts.tsv
|
||||
-rw-r--r-- 1 dmitry staff 1.4G Apr 20 10:05 Posts.xml
|
||||
-rw-r--r-- 1 dmitry staff 405M Apr 21 17:56 Posts.xml.tgz
|
||||
```
|
||||
|
||||
Note, `ls -lL` doesn't show unresolved symlinks.
|
||||
|
||||
|
||||
### Extracting a binary feature
|
||||
|
||||
Binary feature - if <python> is present in tags.
|
||||
|
||||
|
||||
```python
|
||||
import sys
|
||||
|
||||
reload(sys)
|
||||
sys.setdefaultencoding('utf-8')
|
||||
|
||||
if len(sys.argv) != 4:
|
||||
sys.stderr.write('Argument error. Usage:\n')
|
||||
sys.stderr.write('\tpython single_fr.py POST_FEATURE INPUT_FILE OUTPUT_FILE\n')
|
||||
|
||||
with open(sys.argv[2]) as fd_in:
|
||||
with open(sys.argv[3], 'w') as fd_out:
|
||||
target = sys.argv[1]
|
||||
for line in fd_in:
|
||||
id, tags, title, body = line.split('\t')
|
||||
fr = 1 if target in tags else 0
|
||||
fd_out.write(u'{}\t{}\t{}\t{}\n'.format(id, fr, title, body))
|
||||
```
|
||||
|
||||
Get code:
|
||||
```
|
||||
$ wget -nv -P code https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/extract_binary_fr.py
|
||||
2017-04-22 13:01:43 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/extract_binary_fr.py [518/518] -> "code/extract_binary_fr.py" [1]
|
||||
$ git add code/extract_binary_fr.py
|
||||
$ git commit -m 'Extract binry feature'
|
||||
[master 28f8177] Extract binry feature
|
||||
1 file changed, 17 insertions(+)
|
||||
create mode 100644 code/extract_binary_fr.py
|
||||
```
|
||||
|
||||
```
|
||||
$ dvc run python code/extract_binary_fr.py '<python>' data/Posts-fr.tsv data/Posts-bin-fr.tsv
|
||||
[Git] A new commit dc78360 was made in the current branch. Added files:
|
||||
[Git] A .state/Posts-bin-fr.tsv.state
|
||||
[Git] A data/Posts-bin-fr.tsv
|
||||
```
|
||||
|
||||
|
||||
```
|
||||
$ wget -nv -P code https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/train_test_split.py
|
||||
2017-04-23 15:39:23 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/train_test_split.py [1615/1615] -> "code/train_test_split.py.1" [1]
|
||||
$ git add code/train_test_split.py
|
||||
$ git commit -m 'Train and test split code'
|
||||
[master ed6ddae] Train and test split code
|
||||
1 file changed, 50 insertions(+)
|
||||
create mode 100644 code/train_test_split.py
|
||||
```
|
||||
|
||||
```
|
||||
$ dvc run python code/train_test_split.py data/Posts-bin-fr.tsv 0.25 20170422 data/Posts-bin-fr-train.tsv data/Posts-bin-fr-test.tsv
|
||||
[Git] A new commit c4c7a86 was made in the current branch. Added files:
|
||||
[Git] A .state/Posts-bin-fr-test.tsv.state
|
||||
[Git] A .state/Posts-bin-fr-train.tsv.state
|
||||
[Git] A data/Posts-bin-fr-test.tsv
|
||||
[Git] A data/Posts-bin-fr-train.tsv
|
||||
```
|
||||
|
||||
```
|
||||
$ wget -nv -P code https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/df_to_matrix.py
|
||||
2017-04-23 15:38:56 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/df_to_matrix.py [1522/1522] -> "code/df_to_matrix.py.1" [1]
|
||||
$ git add code/df_to_matrix.py
|
||||
$ git commit -m 'Data frame to matrix script'
|
||||
[master 51779a5] Data frame to matrix script
|
||||
1 file changed, 55 insertions(+)
|
||||
create mode 100644 code/df_to_matrix.py
|
||||
```
|
||||
|
||||
```
|
||||
$ run python code/df_to_matrix.py data/Posts-bin-fr-train.tsv data/Posts-train-matrix.p
|
||||
[Git] A new commit 56d02ee was made in the current branch. Added files:
|
||||
[Git] A .state/Posts-train-matrix.p.state
|
||||
[Git] A data/Posts-train-matrix.p
|
||||
```
|
||||
|
||||
```
|
||||
$ dvc run python code/df_to_matrix.py data/Posts-bin-fr-test.tsv data/Posts-test-matrix.p
|
||||
[Git] A new commit 14772f7 was made in the current branch. Added files:
|
||||
[Git] A .state/Posts-test-matrix.p.state
|
||||
[Git] A data/Posts-test-matrix.p
|
||||
```
|
||||
|
||||
|
||||
|
||||
# Copyright
|
||||
|
||||
This project is distributed under the Apache license version 2.0 (see the LICENSE file in the project root).
|
||||
|
||||
By submitting a pull request for this project, you agree to license your contribution under the Apache license version 2.0 to this project.
|
||||
|
||||
# Setup / Configuration
|
||||
# Cloud configuration
|
||||
`dvc` supports both aws and google cloud; you will need accounts either from [aws](https://aws.amazon.com/) or [gcloud](https://cloud.google.com/).
|
||||
|
||||
Edit `dvc.conf`, and fill in `StoragePath` under `[Data]` with your preferred bucket and path. For example,
|
||||
@@ -364,3 +93,11 @@ pip install -r requirements.txt
|
||||
# creates dist/dvc
|
||||
./build_osx.sh
|
||||
```
|
||||
|
||||
|
||||
# Copyright
|
||||
|
||||
This project is distributed under the Apache license version 2.0 (see the LICENSE file in the project root).
|
||||
|
||||
By submitting a pull request for this project, you agree to license your contribution under the Apache license version 2.0 to this project.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user