mirror of
https://github.com/Stone-Red-Code/dvc.git
synced 2026-11-01 23:08:43 +01:00
readme: update description, summary, links
This commit is contained in:
+38
-18
@@ -28,17 +28,18 @@
|
||||
|
|
||||
|
||||
**Data Science Version Control** or **DVC** is an **open-source** tool for data science and
|
||||
machine learning projects. With a simple and flexible Git-like architecture and interface it
|
||||
helps data scientists:
|
||||
machine learning projects. Key features:
|
||||
|
||||
#. manage **machine learning models** - versioning, including data sets and transformations (scripts) that were
|
||||
used to generate models;
|
||||
#. simple **command line** Git-like exprerience. Does not require installing and maintaining
|
||||
any databases. Does not depend on any proprietary online services;
|
||||
|
||||
#. make projects **reproducible**;
|
||||
#. it manages and versions **datasets** and **machine learning models**. Data is saved in
|
||||
S3, Google cloud, Azure, SSH server, HDFS or even local HDD RAID;
|
||||
|
||||
#. make projects **shareable**;
|
||||
#. it makes projects **reproducible** and **shareable**, it helps answering question how
|
||||
the model was build;
|
||||
|
||||
#. manage experiments with branching and **metrics** tracking;
|
||||
#. it helps manage experiments with Git tags or branches and **metrics** tracking;
|
||||
|
||||
It aims to replace tools like Excel and Docs that are being commonly used as a knowledge repo and
|
||||
a ledger for the team, ad-hoc scripts to track and move deploy different model versions, ad-hoc
|
||||
@@ -50,14 +51,30 @@ data file suffixes and prefixes.
|
||||
How DVC works
|
||||
=============
|
||||
|
||||
DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache.
|
||||
To store and share data files cache DVC supports remotes - any cloud (S3, Azure, Google Cloud, etc) or any on-premise
|
||||
network storage (via SSH, for example).
|
||||
We encourage you to read our `Get Started <https://dvc.org/doc/get-started>`_ to better understand what DVC
|
||||
is and how does it fit your scenarios.
|
||||
|
||||
The easiest (but not perfect!) way to describe it: DVC is Git (or Git-lfs to be precise) + ``makefiles``
|
||||
tailored specifically for ML and Data Science scenarios.
|
||||
|
||||
#. ``Git/Git-lfs`` part - DVC helps you storing and sharing data artifacts, models. It connects them with your
|
||||
Git repository.
|
||||
#. ``Makefiles`` part - DVC describes how one data or model artifact was build from another data.
|
||||
|
||||
DVC usually runs along with Git. Git is used as usual to store and version code and DVC meta-files. DVC helps
|
||||
to store data and model files seamlessly out of Git while preserving almost the same user experience as if they
|
||||
were stored in Git itself. To store and share data files cache DVC supports remotes - any cloud (S3, Azure,
|
||||
Google Cloud, etc) or any on-premise network storage (via SSH, for example). It does not require
|
||||
|
||||
.. image:: https://dvc.org/static/img/flow.gif
|
||||
:target: https://dvc.org/static/img/flow.gif
|
||||
:alt: how_dvc_works
|
||||
|
||||
DVC pipelines (aka computational graph) feature connects code and data together. In a very explicit way you can
|
||||
specify, run, and save information that a certain command with certain dependencies needs to be run to produce
|
||||
a model. See the quick start section below or check `Get Started <https://dvc.org/doc/get-started>`_ tutorial to
|
||||
learn more.
|
||||
|
||||
Quick start
|
||||
===========
|
||||
|
||||
@@ -66,7 +83,7 @@ Please read `Get Started <https://dvc.org/doc/get-started>`_ for the full versio
|
||||
+-----------------------------------+-------------------------------------------------------------------+
|
||||
| Step | Command |
|
||||
+===================================+===================================================================+
|
||||
| Track code and data together | | ``$ git add train.py`` |
|
||||
| Track data | | ``$ git add train.py`` |
|
||||
| | | ``$ dvc add images.zip`` |
|
||||
+-----------------------------------+-------------------------------------------------------------------+
|
||||
| Connect code and data by commands | | ``$ dvc run -d images.zip -o images/ unzip -q images.zip`` |
|
||||
@@ -79,27 +96,30 @@ Please read `Get Started <https://dvc.org/doc/get-started>`_ for the full versio
|
||||
| | | ``$ git commit -m 'The baseline model'`` |
|
||||
| | | ``$ git push`` |
|
||||
+-----------------------------------+-------------------------------------------------------------------+
|
||||
| Share data and ML models | | ``$ dvc remote add myremote s3://mybucket/image_cnn`` |
|
||||
| | | ``$ dvc config core.remote myremote`` |
|
||||
| Share data and ML models | | ``$ dvc remote add myremote -d s3://mybucket/image_cnn`` |
|
||||
| | | ``$ dvc push`` |
|
||||
+-----------------------------------+-------------------------------------------------------------------+
|
||||
|
||||
Installation
|
||||
============
|
||||
|
||||
There are three options to install DVC: ``pip``, Homebrew, or an OS-specific package:
|
||||
Read this `instruction <https://dvc.org/doc/get-started/install>`_ to get more details. There are three
|
||||
options to install DVC: ``pip``, Homebrew, or an OS-specific package:
|
||||
|
||||
pip (PyPI)
|
||||
----------
|
||||
|
||||
Stable
|
||||
^^^^^^
|
||||
.. code-block:: bash
|
||||
|
||||
pip install dvc
|
||||
|
||||
Development
|
||||
^^^^^^^^^^^
|
||||
Depending on the remote storage type you plan to use to keep and share your data you might need to specify
|
||||
one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``ssh``. Or ``all_remotes`` to include them all.
|
||||
The command should look like this: ``pip install dvc[s3]`` - it installs the ``boto3`` library along with
|
||||
DVC to support the AWS S3 storage.
|
||||
|
||||
To install the development version, run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install git+git://github.com/iterative/dvc
|
||||
|
||||
Reference in New Issue
Block a user