AirFlow integration link (#6077)

* Update README.rst
* typo fix
This commit is contained in:
Dmitry Petrov
2021-05-28 19:43:36 -07:00
committed by GitHub
parent efda015de4
commit a55c47e89f
+8 -1
View File
@@ -182,6 +182,12 @@ Fedora / CentOS (rpm)
Comparison to related technologies
==================================
#. *Data Engineering* tools such as `AirFlow <https://airflow.apache.org/>`_, `Luigi <https://github.com/spotify/luigi>`_ and
others - in DVC data, model and ML pipelines represent a single ML project and focuses on data scientist' experience while
data engineering pipeline orchestrates multiple data projects and focuses on efficent execution. DVC project can be used
from data pipeline as a single execution step. `AirFlow DVC <https://github.com/covid-genomics/airflow-dvc>`_ is an example
of such integration.
#. `Git-annex <https://git-annex.branchable.com/>`_ - DVC uses the idea of storing the content of large files (which should
not be in a Git repository) in a local key-value store, and uses file hardlinks/symlinks instead of
copying/duplicating files.
@@ -195,7 +201,8 @@ Comparison to related technologies
#. `Workflow Management Systems <https://en.wikipedia.org/wiki/Workflow_management_system>`_ - DVC is a workflow
management system designed specifically to manage machine learning experiments. DVC is built on top of Git.
#. `DAGsHub <https://dagshub.com/>`_ - This is a Github equivalent for DVC. Pushing Git+DVC based repositories to DAGsHub will produce in a high level project dashboard; including DVC pipelines and metrics visualizations, as well as links to any DVC-managed files present in cloud storage.
#. `DAGsHub <https://dagshub.com/>`_ - online service of Git+DVC repositories with pipeline and metrics visualization and
DVC-specific cloud storage.
Contributing
============