[![Code Climate](https://codeclimate.com/github/dmpetrov/dataversioncontrol/badges/gpa.svg)](https://codeclimate.com/github/dmpetrov/dataversioncontrol) [![Test Coverage](https://codeclimate.com/github/dmpetrov/dataversioncontrol/badges/coverage.svg)](https://codeclimate.com/github/dmpetrov/dataversioncontrol/coverage) # Data version control Git for data science projects. It streamlines your git code, your data (S3 and GCP) and dependencies to a single reproducible environment. # Tutorial ## Install Please download dvc from http://dataversioncontrol.com for your platform or install through source code from Github https://github.com/dmpetrov/dataversioncontrol. ## Initialize dvc environment DVC is an extension of the git command. It works on top of git command in existing git repositories. Let's create a new git repository for the tutorial perpose. ``` $ cd ~/src $ mkdir tag_classifier $ cd tag_classifier $ git init Initialized empty Git repository in /Users/dmitry/src/tag_classifier/.git/ ``` Initialize dvc environment: ``` $ dvc init Directories data/, .cache/ and .state/ were created File .gitignore was created Directory .cache was added to .gitignore file [Git] A new commit e7a4b0d was made in the current branch. Added files: [Git] A .gitignore [Git] A dvc.conf ``` The output shows the actions that dvc has made. Special directories `data/`, `cache/` and `state/` were created to keep track of your external data which are not stored in git but have to be under users control. These three directories are mondatory for any dvc project. Names of the directory might be changed by the `dvc init` command line parameters `--data-dir`, `--cache-dir` and `--state-dir` respectively. All these special directories names are going to be saved in `dvc.conf` config file as a result of the init command and should not be changed over dvc project life time. As you might see from the output `.gitignore` file was created (or modified if you already had one). This is result of including cache directory to this file and also `.dvc.conf.lock` file. Meaning of these changes will be explain later in the tutorial. DVC commits all the chenges to the git repository at the end of successful run. The commit step might be skipped by command line parameter `--skip-git-actions`. This parameter might be applied to any dvc command. ## Importing data files Data file has to be imported in dvc `data/` directory before using it. ``` $ dvc import https://s3-us-west-2.amazonaws.com/dvc-share/so/small/Posts-100k.xml.tgz data/ [Git] A new commit b69ab31 was made in the current branch. Added files: [Git] A .state/Posts.xml.tgz.state [Git] A data/Posts.xml.tgz ``` The data file was downloaded and now it is available in the data directory. However, it is only a part of the work that dvc has done. ### Cache files If you look carefully at the file you could see that this is actually a symlink to a cache directory file Badges.xml_a5ed2b9: ``` $ ls -l data/Posts-100k.xml.tgz lrwxr-xr-x 1 dmitry staff 31B Apr 21 17:56 data/Posts.xml.tgz@ -> ../.cache/Posts.xml.tgz_9185f92 ``` Instead of importing the file to `data/Badges.xml` the system imports the file in the cache directory and creates a symlink `data/Badges.xml`. The file in the cache directory `cache/Badges.xml_a5ed2b9` has the same name as the file in the data directory but extended by a suffix `_a5ed2b9` which correspondes to a git snapshot checksum on the time when the file was created. ### State files In addition to the cache file dvc creates a state file in the `state/` directory for each data file: ``` $ ls -l .state/Posts-100k.xml.tgz.state -rw-r--r-- 1 dmitry staff 437B Apr 21 17:56 .state/Posts.xml.tgz.state ``` The state file contains information for file reproducibility. ## Structure data ``` $ dvc run tar zxf data/Posts-100k.xml.tgz -C data/ [Git] A new commit fbff960 was made in the current branch. Added files: [Git] A .state/Posts.xml.state [Git] A data/Posts.xml $ cd .. ``` ``` $ dvc run cp data/Posts-100k.xml data/Posts.xml ``` Full file is here https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/posts_to_tsv.py ``` $ mkdir code $ wget -nv -P code/ https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/posts_to_tsv.py 2017-04-22 12:10:13 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/posts_to_tsv.py [1689/1689] -> "code/posts_to_tsv.py" [1] ``` ``` $ git add code/posts_to_tsv.py $ git commit -m 'Post to tsv script' [master 4a581d9] Post to tsv script 1 file changed, 51 insertions(+) create mode 100644 code/posts_to_tsv.py ``` ```python def process_posts(fd_in, fd_out): num = 1 for line in fd_in: try: attr = xml.etree.ElementTree.fromstring(line).attrib items = ( attr.get('Id', ''), attr.get('PostTypeId', ''), attr.get('CreationDate', ''), attr.get('Score', ''), attr.get('ViewCount', ''), attr.get('OwnerUserId', ''), attr.get('LastActivityDate', ''), attr.get('AnswerCount', ''), attr.get('CommentCount', ''), attr.get('FavoriteCount', ''), attr.get('Title', ''), attr.get('Body', '') ) fmt = ('{}\t' * len(items))[:-1] + '\n' fd_out.write(fmt.format(*items)) num += 1 except Exception as ex: sys.stderr.write('Error in line {}: {}'.format(num, ex)) ``` ``` $ dvc run python code/posts_to_tsv.py data/Posts.xml data/Posts.tsv [Git] A new commit dc54e5a was made in the current branch. Added files: [Git] A .state/Posts.tsv.state [Git] A data/Posts.tsv ``` ``` $ echo "cut -d$'\t' -f 1,11-13 \$@" > code/cut_fr.sh $ chmod a+x code/cut_fr.sh $ git add code/cut_fr.sh $ git commit -m 'Cut posts features' [master aa9c6c1] Cut posts features 1 file changed, 1 insertion(+) create mode 100755 code/cut_fr.sh ``` Reducing file size: ``` $ dvc run --stdout data/Posts-fr.tsv bash ./code/cut_fr.sh data/Posts.tsv [Git] A new commit 39a9544 was made in the current branch. Added files: [Git] A .state/Posts-fr.tsv.state [Git] A data/Posts-fr.tsv ``` Looking for data items: ``` $ ls -l data total 32 lrwxr-xr-x 1 dmitry staff 30B Apr 21 23:36 Posts-fr.tsv@ -> ../.cache/Posts-fr.tsv_e1233ed lrwxr-xr-x 1 dmitry staff 27B Apr 21 18:53 Posts.tsv@ -> ../.cache/Posts.tsv_b58a46d lrwxr-xr-x 1 dmitry staff 27B Apr 21 18:15 Posts.xml@ -> ../.cache/Posts.xml_4a581d9 lrwxr-xr-x 1 dmitry staff 31B Apr 21 17:56 Posts.xml.tgz@ -> ../.cache/Posts.xml.tgz_9185f92 ``` Looking for the actual size: ``` $ ls -lL data total 7744240 -rw-r--r-- 1 dmitry staff 927M Apr 21 23:36 Posts-fr.tsv -rw-r--r-- 1 dmitry staff 990M Apr 21 18:53 Posts.tsv -rw-r--r-- 1 dmitry staff 1.4G Apr 20 10:05 Posts.xml -rw-r--r-- 1 dmitry staff 405M Apr 21 17:56 Posts.xml.tgz ``` Note, `ls -lL` doesn't show unresolved symlinks. ### Extracting a binary feature Binary feature - if is present in tags. ```python import sys reload(sys) sys.setdefaultencoding('utf-8') if len(sys.argv) != 4: sys.stderr.write('Argument error. Usage:\n') sys.stderr.write('\tpython single_fr.py POST_FEATURE INPUT_FILE OUTPUT_FILE\n') with open(sys.argv[2]) as fd_in: with open(sys.argv[3], 'w') as fd_out: target = sys.argv[1] for line in fd_in: id, tags, title, body = line.split('\t') fr = 1 if target in tags else 0 fd_out.write(u'{}\t{}\t{}\t{}\n'.format(id, fr, title, body)) ``` Get code: ``` $ wget -nv -P code https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/extract_binary_fr.py 2017-04-22 13:01:43 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/extract_binary_fr.py [518/518] -> "code/extract_binary_fr.py" [1] $ git add code/extract_binary_fr.py $ git commit -m 'Extract binry feature' [master 28f8177] Extract binry feature 1 file changed, 17 insertions(+) create mode 100644 code/extract_binary_fr.py ``` ``` $ dvc run python code/extract_binary_fr.py '' data/Posts-fr.tsv data/Posts-bin-fr.tsv [Git] A new commit dc78360 was made in the current branch. Added files: [Git] A .state/Posts-bin-fr.tsv.state [Git] A data/Posts-bin-fr.tsv ``` ``` $ wget -nv -P code https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/train_test_split.py 2017-04-23 15:39:23 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/train_test_split.py [1615/1615] -> "code/train_test_split.py.1" [1] $ git add code/train_test_split.py $ git commit -m 'Train and test split code' [master ed6ddae] Train and test split code 1 file changed, 50 insertions(+) create mode 100644 code/train_test_split.py ``` ``` $ dvc run python code/train_test_split.py data/Posts-bin-fr.tsv 0.25 20170422 data/Posts-bin-fr-train.tsv data/Posts-bin-fr-test.tsv [Git] A new commit c4c7a86 was made in the current branch. Added files: [Git] A .state/Posts-bin-fr-test.tsv.state [Git] A .state/Posts-bin-fr-train.tsv.state [Git] A data/Posts-bin-fr-test.tsv [Git] A data/Posts-bin-fr-train.tsv ``` ``` $ wget -nv -P code https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/df_to_matrix.py 2017-04-23 15:38:56 URL:https://s3-us-west-2.amazonaws.com/dvc-share/so/small/code/df_to_matrix.py [1522/1522] -> "code/df_to_matrix.py.1" [1] $ git add code/df_to_matrix.py $ git commit -m 'Data frame to matrix script' [master 51779a5] Data frame to matrix script 1 file changed, 55 insertions(+) create mode 100644 code/df_to_matrix.py ``` ``` $ run python code/df_to_matrix.py data/Posts-bin-fr-train.tsv data/Posts-train-matrix.p [Git] A new commit 56d02ee was made in the current branch. Added files: [Git] A .state/Posts-train-matrix.p.state [Git] A data/Posts-train-matrix.p ``` ``` $ dvc run python code/df_to_matrix.py data/Posts-bin-fr-test.tsv data/Posts-test-matrix.p [Git] A new commit 14772f7 was made in the current branch. Added files: [Git] A .state/Posts-test-matrix.p.state [Git] A data/Posts-test-matrix.p ``` # Copyright This project is distributed under the Apache license version 2.0 (see the LICENSE file in the project root). By submitting a pull request for this project, you agree to license your contribution under the Apache license version 2.0 to this project. # Setup / Configuration `dvc` supports both aws and google cloud; you will need accounts either from [aws](https://aws.amazon.com/) or [gcloud](https://cloud.google.com/). Edit `dvc.conf`, and fill in `StoragePath` under `[Data]` with your preferred bucket and path. For example, ``` [Data] StoragePath = gc://dvc-demo/dvcdata ``` ## Amazon AWS Setup NB: To see your available buckets, run `aws s3 ls` Configuration: * edit `dvc.conf`, and set `Cloud = amazon` * under `[AWS]`, fill in `AccessKeyId` and `SecretAccessKey` Test `dvc` is correctly configured by running `dvc test-aws` (TODO) ## Google Cloud Setup NB: To see your available buckets, run `gsutil ls`. Configuration: * edit `dvc.conf`, and set `Cloud = google` * under `[GC]`, set `ProjectName` to your preferred (or just your default) project name. Test `dvc` is correctly configured by running `dvc test-gcloud`. # Usage ``` mkdir t cd !$ git init . dvc init Info. Directories data/, cache/ and state/ were created Info. File .gitignore was created Info. Directory cache was added to .gitignore file Info. [Git] A new commit 34687b2 was made in the current branch. Added files: Info. [Git] A .gitignore Info. [Git] A dvc.conf ``` # Development Create a virtualenv, for example, at `~/.environments/dvc` by `mkdir -p ~/.environments/dvc; virtualenv ~/.environments/dvc` or use `virtualenvwrapper`. ``` # if you use virtualenvwrapper workon dvc # otherwise: source ~/.environments/dvc/bin/activate pip install -r requirements.txt # happy coding! ``` # Building ## OSX ``` # creates dist/dvc ./build_osx.sh ```