Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-11 15:28:27 +01:00
Code Issues Packages Projects Releases Wiki Activity

Tutorial

Browse Source
This commit is contained in:
Dmitry Petrov
2017-04-17 17:08:17 -07:00
parent a06ff8e8d4
commit 2c849da1d7
1 changed files with 93 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+93 -1
View File
@@ -2,7 +2,99 @@
[![Test Coverage](https://codeclimate.com/github/dmpetrov/dataversioncontrol/badges/coverage.svg)](https://codeclimate.com/github/dmpetrov/dataversioncontrol/coverage)
# Data version control
Git for data science projects. It streamlines your git code, your data (S3 and GCP) and the dependencies to a single reproducible environment.
Git for data science projects. It streamlines your git code, your data (S3 and GCP) and dependencies to a single reproducible environment.
# Tutorial
## Install
Please download dvc from http://dataversioncontrol.com for your platform or
install through source code from Github https://github.com/dmpetrov/dataversioncontrol.
## Initialize dvc environment
DVC is an extension of the git command.
It works on top of git command in existing git repositories.
Let's create a new git repository for the tutorial perpose.
```
$ cd ~/src
$ mkdir tag_classifier
$ cd tag_classifier
$ git init
Initialized empty Git repository in /Users/dmitry/src/tag_classifier/.git/
```
Initialize dvc environment:
```
$ dvc init
Info. Directories data/, cache/ and state/ were created
Info. File .gitignore was created
Info. Directory cache was added to .gitignore file
Info. [Git] A new commit a5ed2b9 was made in the current branch. Added files:
Info. [Git] A .gitignore
Info. [Git] A dvc.conf
```
The output shows the actions that dvc has made.
Special directories `data/`, `cache/` and `state/` were created to keep track of your external data which are not stored in git but have to be under users control.
These three directories are mondatory for any dvc project.
Names of the directory might be changed by the `dvc init` command line parameters `--data-dir`, `--cache-dir` and `--state-dir` respectively.
All these special directories names are going to be saved in `dvc.conf` config file as a result of
the init command and should not be changed over dvc project life time.
As you might see from the output `.gitignore` file was created (or modified if you already had one).
This is result of including cache directory to this file and also `.dvc.conf.lock` file.
Meaning of these changes will be explain later in the tutorial.
DVC commits all the chenges to the git repository at the end of successful run.
The commit step might be skipped by command line parameter `--skip-git-actions`.
This parameter might be applied to any dvc command.
## Importing data files
Data file has to be imported in dvc `data/` directory before using it.
```
$ dvc data-import https://s3-us-west-2.amazonaws.com/dataversioncontrol/functests/stackoverflow_raw_small/Badges.xml data/
$ dvc data-import https://s3-us-west-2.amazonaws.com/dvc-doc/small/Badges.xml data/
Info. [Git] A new commit 9f27943 was made in the current branch. Added files:
Info. [Git] A data/Badges.xml
Info. [Git] A state/Badges.xml.state
```
The data file was downloaded and now it is available in the data directory.
However, it is only a part of the work that dvc has done.
If you look carefully at the file you could see that this is actually a symlink to
a cache directory file Badges.xml_a5ed2b9:
```
$ ls -l data/Badges.xml
lrwxr-xr-x 1 dmitry staff 27B Apr 17 16:19 data/Badges.xml@ -> ../cache/Badges.xml_a5ed2b9
```
Instead of importing the file to `data/Badges.xml` the system imports the file in the cache directory
`cache/Badges.xml_a5ed2b9` and creates a symlink `data/Badges.xml`.
The file in the cache directory `cache/Badges.xml_a5ed2b9` has the same name as
the file in the data directory but extended by a suffix `_a5ed2b9` which
correspondes to a git snapshot checksum on the time when the file was created.
The system transparently moves the actual file to the cache directory and creates a symlink
1. `data/` directory was created to keep all your data files that not suppose to go to Git repository.
This is the directory users suppose to use for all the data files.
However, dvc transparently moves all actual files to the `cache` directory and keeps only
symlink in `data/`.
2. `cache/` directory is a directory the file content.
# Copyright