It replaces POOL_SIZE constant and by default equals to the
number of cpus available.
map_progress is a helper for 'sync' and 'import' commands to
run in a multi-threaded mode.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Currently progress bar is only supported for downloading
from web(aws,http* etc), but now it is also supported for
local files(i.e. LOCAL for 'data sync' and just local files
for 'import')
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This one is a bit like the one you see in yum/dnf.
It has one single line updating current progress
for different targets and once they are finished
prints them as standalone line. So it looks smth
like this:
$ dvc import https://foo.bar/one.data \
https://foo.bar/two.data \
https://foo.bar/three.data
(1/3): [########################################] 100% one.data
(2/3): [######################### ] 60% three.data
Signed-off-by: Ruslan Kuprieiev <[email protected]>
1) use io for StringIO in python3
2) use builtins instead of __builtin__
3) use StringIO instead of BytesIO when mocking aws config
Othervise this test fails on python3.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Our tests assume that user has his default credentials at
~/.aws/credentials, but this is not the case with travis.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This allows us to download files in a few threads.
Also includes RFC, as we can now get rid of import-bulk.py.
v2: use POOL_SIZE=4
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Recent changes for building standalone dvc binary on Linux
made it possible to do the same thing on Windows. So now
we are getting rid of WinPython and instead building
standalone dvc binary using PyInstall and packaing it
into an installer with InnoSetup.
One of the benefits is dramatically decreased installer size.
From 125M we got to only 13M!
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Pyinstall can't resolve all dependencies on it's own, so
we need to help it with hoolks. From now on, we could
try to use pyinstall on Windows and MacOS to see if will
build standalone binaries.
Dmitry Petrov
Ruslan Kuprieiev