- SSH number of jobs increased from 4 to 8.
- `batch_exists` now opens all the SFTP sessions available to
distribute the work between them. Number of sessions is restricted
by the `MaxSessions` option configurable by the `sshd_config`.
- Fix#2104
With the optimizations mentioned above, we increased the speed by 8x
(using the default `MaxSessions` configuration -- 10).
Shout-outs to @efiop, @shcheklein, and @dmpetrov for suggesting the
optimization techniques.
Bugs fixed along the way:
- unable to copy between buckets in GS
- unneeded connection init in RemoteGS.copy()
- connection not closed on some errors in RemoteSSH.download()
- inconsistent progress bar handling
Fixes#2039
Required by #2012 to be able to fetch data through the API without
interuption from updater.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
------------------------------------------------------------
Traceback (most recent call last):
File "/Users/efiop/git/dvc/dvc/remote/local/__init__.py", line 280, in upload
no_progress_bar=no_progress_bar,
File "/Users/efiop/git/dvc/dvc/utils/__init__.py", line 129, in copyfile
System.reflink(src, dest)
File "/Users/efiop/git/dvc/dvc/system.py", line 125, in reflink
ret = System._reflink_darwin(source, link_name)
File "/Users/efiop/git/dvc/dvc/system.py", line 89, in _reflink_darwin
ctypes.c_char_p(src.encode("utf-8")),
AttributeError: 'PosixPathInfo' object has no attribute 'encode'
------------------------------------------------------------
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.
Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous
Both families might had caused undefined surprising behavior when local
and remote paths clash.
Mr. Outis
Ruslan Kuprieiev
Alexander Schepanovski
Ruslan Kuprieiev
Sanidhya Mangal
Aman Sharma
pawel
Andrew Grigorev
dlg227