diff --git a/README.rst b/README.rst index 96db11056..137f458fe 100644 --- a/README.rst +++ b/README.rst @@ -28,17 +28,18 @@ | **Data Science Version Control** or **DVC** is an **open-source** tool for data science and -machine learning projects. With a simple and flexible Git-like architecture and interface it -helps data scientists: +machine learning projects. Key features: -#. manage **machine learning models** - versioning, including data sets and transformations (scripts) that were - used to generate models; +#. simple **command line** Git-like exprerience. Does not require installing and maintaining + any databases. Does not depend on any proprietary online services; -#. make projects **reproducible**; +#. it manages and versions **datasets** and **machine learning models**. Data is saved in + S3, Google cloud, Azure, SSH server, HDFS or even local HDD RAID; -#. make projects **shareable**; +#. it makes projects **reproducible** and **shareable**, it helps answering question how + the model was build; -#. manage experiments with branching and **metrics** tracking; +#. it helps manage experiments with Git tags or branches and **metrics** tracking; It aims to replace tools like Excel and Docs that are being commonly used as a knowledge repo and a ledger for the team, ad-hoc scripts to track and move deploy different model versions, ad-hoc @@ -50,14 +51,30 @@ data file suffixes and prefixes. How DVC works ============= -DVC is compatible with Git for storing code and the dependency graph (DAG), but not data files cache. -To store and share data files cache DVC supports remotes - any cloud (S3, Azure, Google Cloud, etc) or any on-premise -network storage (via SSH, for example). +We encourage you to read our `Get Started `_ to better understand what DVC +is and how does it fit your scenarios. + +The easiest (but not perfect!) way to describe it: DVC is Git (or Git-lfs to be precise) + ``makefiles`` +tailored specifically for ML and Data Science scenarios. + +#. ``Git/Git-lfs`` part - DVC helps you storing and sharing data artifacts, models. It connects them with your + Git repository. +#. ``Makefiles`` part - DVC describes how one data or model artifact was build from another data. + +DVC usually runs along with Git. Git is used as usual to store and version code and DVC meta-files. DVC helps +to store data and model files seamlessly out of Git while preserving almost the same user experience as if they +were stored in Git itself. To store and share data files cache DVC supports remotes - any cloud (S3, Azure, +Google Cloud, etc) or any on-premise network storage (via SSH, for example). It does not require .. image:: https://dvc.org/static/img/flow.gif :target: https://dvc.org/static/img/flow.gif :alt: how_dvc_works +DVC pipelines (aka computational graph) feature connects code and data together. In a very explicit way you can +specify, run, and save information that a certain command with certain dependencies needs to be run to produce +a model. See the quick start section below or check `Get Started `_ tutorial to +learn more. + Quick start =========== @@ -66,7 +83,7 @@ Please read `Get Started `_ for the full versio +-----------------------------------+-------------------------------------------------------------------+ | Step | Command | +===================================+===================================================================+ -| Track code and data together | | ``$ git add train.py`` | +| Track data | | ``$ git add train.py`` | | | | ``$ dvc add images.zip`` | +-----------------------------------+-------------------------------------------------------------------+ | Connect code and data by commands | | ``$ dvc run -d images.zip -o images/ unzip -q images.zip`` | @@ -79,27 +96,30 @@ Please read `Get Started `_ for the full versio | | | ``$ git commit -m 'The baseline model'`` | | | | ``$ git push`` | +-----------------------------------+-------------------------------------------------------------------+ -| Share data and ML models | | ``$ dvc remote add myremote s3://mybucket/image_cnn`` | -| | | ``$ dvc config core.remote myremote`` | +| Share data and ML models | | ``$ dvc remote add myremote -d s3://mybucket/image_cnn`` | | | | ``$ dvc push`` | +-----------------------------------+-------------------------------------------------------------------+ Installation ============ -There are three options to install DVC: ``pip``, Homebrew, or an OS-specific package: +Read this `instruction `_ to get more details. There are three +options to install DVC: ``pip``, Homebrew, or an OS-specific package: pip (PyPI) ---------- -Stable -^^^^^^ .. code-block:: bash pip install dvc -Development -^^^^^^^^^^^ +Depending on the remote storage type you plan to use to keep and share your data you might need to specify +one of the optional dependencies: ``s3``, ``gs``, ``azure``, ``ssh``. Or ``all_remotes`` to include them all. +The command should look like this: ``pip install dvc[s3]`` - it installs the ``boto3`` library along with +DVC to support the AWS S3 storage. + +To install the development version, run: + .. code-block:: bash pip install git+git://github.com/iterative/dvc