Commit Graph
11 Commits
Author SHA1 Message Date
Batuhan Taskaya 51ada6d272 Implement straight-to-remote functionality (#5198)
* Implement export-to-remote functionality

* Redistrubition of functionality from tree to remote

* Pass chunk size into the open_url

* Move functionality away from remote to cache

* Basic progress bars for both transfer types

* Proper job calculation

* Redesign .open() usage in order to work with different trees

* Align the signatures of other open() functions to accept **kwargs

* Implement chunked uploading to the ssh

* Use local cache to compute directory hash (md5)

* S3: multipart upload test

* Wrap local cache's hash_to_path with PathInfo

* Use .isdir() instead of a try/except statement for calculating the from_infos

* New CLI layout (add/import-url)

* Adjust CLI tests to accept new arguments

* tests: import_url/add func/unit tests

* Introduce cache.transfer()

* Invalid CLI combination tests

* Store dir_info key in the transfer_directory

* LocalTree.upload_fobj

* Tests for utils

* Lazy imports on some places

* --to-remote migration (and repo._transfer)

* Use proper temporary location, more efficient pbar calc

* Directory transfer logic rewrite

* Use shutil.copyfileobj()

* Rebase...

* Better tests

* Inline transfer operation to .add() / .transfer()

* Open in the 'rb' mode for the tests

* More tests, better invalid option handling at repo level

* .read() certain amount of bytes to validate the content

* Better invalid argument handling

* Better progress bars by using .getsize() API

* For fallback upload, use the from_info.name instead of temporary naem

* introduce --jobs for add/import-url
2021-01-19 12:53:32 +02:00
Ruslan Kuprieiev 251a3269c2 dvcfile: add desc field for outs/stages (#4901)
Outs desc will be used by viewer right away. Stage desc is for people to
start using now, but we won't be using it it viewer until DAG support is
added there.
2020-11-18 22:06:52 +02:00
karajan1001andRuslan Kuprieiev 74930c01ec add --no-exec mode to import url (#4075)
* add --no-exec mode to import url

fix #3513

* Update tests/func/test_import_url.py

* Update tests/func/test_import_url.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-06-20 13:29:09 +03:00
Saugat Pachhai d2734dfe1e tests: remove use of mocker as context-manager (#3480) 2020-03-16 08:25:22 +05:45
pawel e1176bf42d imp/imp_url cmd test: remove unnecessary dvc fixture 2020-01-06 16:40:15 +01:00
pawel a620658884 http/ imp tests - migrate do dir helpers 2020-01-06 16:18:01 +01:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Jorge Orpinel a48481b01f dvc: change ` for ' in dynamic parts of output messages (#2644)
* dvc: change ` for ' in dinamic parts of DVC outputs

per https://github.com/iterative/dvc/pull/2644#issuecomment-545151398

* dvc: use ' for dynamic output and ` for DVC names

per https://github.com/iterative/dvc/pull/2644#discussion_r338417698
2019-10-28 22:00:53 +02:00
Alexander Schepanovski db19957b8a Download refactor (#2275)
* remote: drop not working http resume

Closes #2136

* remote: drop @transfer_context()

It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.

* remote: drop upload/download chunking
2019-07-16 16:27:44 +03:00
Ruslan Kuprieiev 94fe86afe8 tests: import-url: rewrite and move error msg test (#2262)
This one was failing randomly.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-14 03:09:08 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-21 01:16:00 +03:00