From a55c47e89fecc447ea4eb0e9b526426a17342029 Mon Sep 17 00:00:00 2001 From: Dmitry Petrov Date: Fri, 28 May 2021 19:43:36 -0700 Subject: [PATCH] AirFlow integration link (#6077) * Update README.rst * typo fix --- README.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index ef53728e9..0f51228e7 100644 --- a/README.rst +++ b/README.rst @@ -182,6 +182,12 @@ Fedora / CentOS (rpm) Comparison to related technologies ================================== +#. *Data Engineering* tools such as `AirFlow `_, `Luigi `_ and + others - in DVC data, model and ML pipelines represent a single ML project and focuses on data scientist' experience while + data engineering pipeline orchestrates multiple data projects and focuses on efficent execution. DVC project can be used + from data pipeline as a single execution step. `AirFlow DVC `_ is an example + of such integration. + #. `Git-annex `_ - DVC uses the idea of storing the content of large files (which should not be in a Git repository) in a local key-value store, and uses file hardlinks/symlinks instead of copying/duplicating files. @@ -195,7 +201,8 @@ Comparison to related technologies #. `Workflow Management Systems `_ - DVC is a workflow management system designed specifically to manage machine learning experiments. DVC is built on top of Git. -#. `DAGsHub `_ - This is a Github equivalent for DVC. Pushing Git+DVC based repositories to DAGsHub will produce in a high level project dashboard; including DVC pipelines and metrics visualizations, as well as links to any DVC-managed files present in cloud storage. +#. `DAGsHub `_ - online service of Git+DVC repositories with pipeline and metrics visualization and + DVC-specific cloud storage. Contributing ============