This one is a bit like the one you see in yum/dnf.
It has one single line updating current progress
for different targets and once they are finished
prints them as standalone line. So it looks smth
like this:
$ dvc import https://foo.bar/one.data \
https://foo.bar/two.data \
https://foo.bar/three.data
(1/3): [########################################] 100% one.data
(2/3): [######################### ] 60% three.data
Signed-off-by: Ruslan Kuprieiev <[email protected]>
1) use io for StringIO in python3
2) use builtins instead of __builtin__
3) use StringIO instead of BytesIO when mocking aws config
Othervise this test fails on python3.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Our tests assume that user has his default credentials at
~/.aws/credentials, but this is not the case with travis.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This allows us to download files in a few threads.
Also includes RFC, as we can now get rid of import-bulk.py.
v2: use POOL_SIZE=4
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Recent changes for building standalone dvc binary on Linux
made it possible to do the same thing on Windows. So now
we are getting rid of WinPython and instead building
standalone dvc binary using PyInstall and packaing it
into an installer with InnoSetup.
One of the benefits is dramatically decreased installer size.
From 125M we got to only 13M!
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Pyinstall can't resolve all dependencies on it's own, so
we need to help it with hoolks. From now on, we could
try to use pyinstall on Windows and MacOS to see if will
build standalone binaries.
This patch introduces DataCloud class that used config to
determine proper cloud class(DataCloudAWS and DataCloudGCP
are awailable) to use in "data_sync" and "remove" commands.
This patch focuses on introducing the common class itself
saving original code and definetly needs some RFC later.
It also adds simple DataCloudLOCAL class that uses directory
as a StoragePath and cp as a sync method. Which makes testing
so much easier and covers local scenarios.
And it also introduces ability to run data sync in parallel
by collecting files first and then running sync in a number
of threads.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Dmitry Petrov
Ruslan Kuprieiev
Ruslan Kuprieiev