This adds support for custom metadata in outputs under `meta` key.
Also adds `--meta` flag to `add`/`import`/`import-url` that supports
`key=value` values.
add --no-download flag to dvc import-url/dvc update to only create/update .dvc
files without downloading the associated data.
Created .dvc files can be fetched using `dvc pull` or can be
updated using `dvc update --no-download`.
* Implement export-to-remote functionality
* Redistrubition of functionality from tree to remote
* Pass chunk size into the open_url
* Move functionality away from remote to cache
* Basic progress bars for both transfer types
* Proper job calculation
* Redesign .open() usage in order to work with different trees
* Align the signatures of other open() functions to accept **kwargs
* Implement chunked uploading to the ssh
* Use local cache to compute directory hash (md5)
* S3: multipart upload test
* Wrap local cache's hash_to_path with PathInfo
* Use .isdir() instead of a try/except statement for calculating the from_infos
* New CLI layout (add/import-url)
* Adjust CLI tests to accept new arguments
* tests: import_url/add func/unit tests
* Introduce cache.transfer()
* Invalid CLI combination tests
* Store dir_info key in the transfer_directory
* LocalTree.upload_fobj
* Tests for utils
* Lazy imports on some places
* --to-remote migration (and repo._transfer)
* Use proper temporary location, more efficient pbar calc
* Directory transfer logic rewrite
* Use shutil.copyfileobj()
* Rebase...
* Better tests
* Inline transfer operation to .add() / .transfer()
* Open in the 'rb' mode for the tests
* More tests, better invalid option handling at repo level
* .read() certain amount of bytes to validate the content
* Better invalid argument handling
* Better progress bars by using .getsize() API
* For fallback upload, use the from_info.name instead of temporary naem
* introduce --jobs for add/import-url
Outs desc will be used by viewer right away. Stage desc is for people to
start using now, but we won't be using it it viewer until DAG support is
added there.
* remote: drop not working http resume
Closes#2136
* remote: drop @transfer_context()
It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.
* remote: drop upload/download chunking
Ruslan Kuprieiev
Dave Berenbaum
skshetry
Oded Messer
Peter Rowlands
Saugat Pachhai (सौगात)
Daniele Trifirò
Batuhan Taskaya
karajan1001andRuslan Kuprieiev
pawel
Ruslan Kuprieiev
Jorge Orpinel
Alexander Schepanovski