This employs generating temporary signed public url and then streaming
from it. The bulk of the code might be reused for other remotes.
Url streaming does resume on connection error using "Range" header and
regenerates public url too.
If remote class doesn't provide .open() then this falls back to pulling
a file and opening it locally.
Since this is multiplexing over the same TCP connection no system
resources are withheld, so this should be a no-brainer optimization.
As a side-effect this should fix "open max - close all - try open -
Administratively prohobited" error @darabi experiencing in #2280.
* remote: drop not working http resume
Closes#2136
* remote: drop @transfer_context()
It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.
* remote: drop upload/download chunking
* remote: support older linuxes as ssh remotes
Work around md5sum not having --tag in Red Hat 6.3 and friends.
Fixes#2242.
* remote: optimize consequent ssh md5 calls
Bugs fixed along the way:
- unable to copy between buckets in GS
- unneeded connection init in RemoteGS.copy()
- connection not closed on some errors in RemoteSSH.download()
- inconsistent progress bar handling
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.
Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous
Both families might had caused undefined surprising behavior when local
and remote paths clash.
As a part of this:
- DataCloud._cloud was refactored into `.get_remote()`
- data stopped using internal _cloud
- added a parametrization for remote tests
Now simple `py.test` command won't use DEBUG log level and `py.test ...
--log-level=LEVEL` will do what it's supposed to.
Staging continues to use DEBUG log level.
This is part of #1888 resolution.
Note we call main() to spare process overhead, this leads to main()
inheriting log level and later tests inheriting whatever main() run
left. This makes main to behave.
* test: clean up test dirs
Closes#1770. Partially does #1888.
* dvc: close anything opened
* test: finalize git repo explicitly to remove test dir later
This was previously an issue on Windows.
* test: remove never used copy of load_stage_file()
* dvc: save comments and custom formatting in stage files
* Add a couple of TODOs
* test: test repro doesn't clear comments and formatting in .dvc
As part of this I started writing pytest-like tests and added commonly
used fixtures.
* test: simplify and prettify dvc formatting retained test
Also test that line comments after md5 fields are retained.
* test: fix tests
* dvc: fix stage and metrics opens not closed
* dvc: always use utf-8 in stage files
It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.
* test: unit test apply_diff()
* test: remove TODOs and ASK
* dvc: polish dvc-formatting PR
* dvc: move stage load/dump utils to a new stage utils file
* dvc: clean up apply_diff()
* test: test stage md5 ignores comments
* test: remove pytests starting marker
