9 Commits

Include renames

Author	SHA1	Message	Date
3 people	b13aec0f2b	[6478] Fix 401 unauthorized for directories in basic auth protected http remote (#6479) ... * added auth to iter_url and created custom open for http fs * added default value to auth * Update dvc/utils/http.py Co-authored-by: Ruslan Kuprieiev <[email protected]> * Update dvc/fs/http.py Co-authored-by: Ruslan Kuprieiev <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pr fixes Co-authored-by: Ruslan Kuprieiev <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2021-08-24 18:39:29 +03:00
Batuhan Taskaya	51ada6d272	Implement straight-to-remote functionality (#5198) ... * Implement export-to-remote functionality * Redistrubition of functionality from tree to remote * Pass chunk size into the open_url * Move functionality away from remote to cache * Basic progress bars for both transfer types * Proper job calculation * Redesign .open() usage in order to work with different trees * Align the signatures of other open() functions to accept **kwargs * Implement chunked uploading to the ssh * Use local cache to compute directory hash (md5) * S3: multipart upload test * Wrap local cache's hash_to_path with PathInfo * Use .isdir() instead of a try/except statement for calculating the from_infos * New CLI layout (add/import-url) * Adjust CLI tests to accept new arguments * tests: import_url/add func/unit tests * Introduce cache.transfer() * Invalid CLI combination tests * Store dir_info key in the transfer_directory * LocalTree.upload_fobj * Tests for utils * Lazy imports on some places * --to-remote migration (and repo._transfer) * Use proper temporary location, more efficient pbar calc * Directory transfer logic rewrite * Use shutil.copyfileobj() * Rebase... * Better tests * Inline transfer operation to .add() / .transfer() * Open in the 'rb' mode for the tests * More tests, better invalid option handling at repo level * .read() certain amount of bytes to validate the content * Better invalid argument handling * Better progress bars by using .getsize() API * For fallback upload, use the from_info.name instead of temporary naem * introduce --jobs for add/import-url	2021-01-19 12:53:32 +02:00
Casper da Costa-Luis	4d140a9944	gdrive: add open (#3916) ... * gdrive: add open Fixes #3408 Related #2865 Fixes #3897 * dependency: add gdrive * test: api: open: gdrive * rollback pydrive dependency * Revert "dependency: add gdrive" This reverts commit fd333261d2e9fe1364cc278138c79890a1f5fe7a. * tests: remotes: GDrive: fix get_url * tests: api: fully test GDrive * minor typo fix * attempt gdrive tests fix * fix gdrive credentials config * replace GDrivePathNotFound => FileMissingError * fix test_open_external[GDrive] * add ensure_dir_scm * minor exception fix	2020-06-13 23:53:25 +03:00
Saugat Pachhai	b8aa8ad951	pyupgrade: 3.6+ (#3805) ... * pyupgrade: 3.6+ * tests: pyupgrade 3.6+	2020-05-15 16:48:02 +03:00
Mr. Outis	27f581cf4e	py3: remove imports from utils.compat	2019-12-31 23:16:11 -06:00
Ruslan Kuprieiev	95fe1578bb	dvc: automatically reorder imports	2019-11-05 02:34:40 +02:00
Alexander Schepanovski	cb5ff6e085	perf: import some things later ... Saves on startup time	2019-09-20 16:45:02 +07:00
Alexander Schepanovski	85289ad00b	api: wrap not found errors ... api.open()/read() will now raise OutputFileMissingError and Remote*.open() will raise FileNotFoundError if target file is absent.	2019-08-15 23:20:48 +07:00
Alexander Schepanovski	e5e29065c8	api: make .open() stream data for Azure ... This employs generating temporary signed public url and then streaming from it. The bulk of the code might be reused for other remotes. Url streaming does resume on connection error using "Range" header and regenerates public url too. If remote class doesn't provide .open() then this falls back to pulling a file and opening it locally.	2019-08-09 21:51:43 +07:00