1) use io for StringIO in python3
2) use builtins instead of __builtin__
3) use StringIO instead of BytesIO when mocking aws config
Othervise this test fails on python3.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Our tests assume that user has his default credentials at
~/.aws/credentials, but this is not the case with travis.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This allows us to download files in a few threads.
Also includes RFC, as we can now get rid of import-bulk.py.
v2: use POOL_SIZE=4
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Recent changes for building standalone dvc binary on Linux
made it possible to do the same thing on Windows. So now
we are getting rid of WinPython and instead building
standalone dvc binary using PyInstall and packaing it
into an installer with InnoSetup.
One of the benefits is dramatically decreased installer size.
From 125M we got to only 13M!
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Pyinstall can't resolve all dependencies on it's own, so
we need to help it with hoolks. From now on, we could
try to use pyinstall on Windows and MacOS to see if will
build standalone binaries.
This patch introduces DataCloud class that used config to
determine proper cloud class(DataCloudAWS and DataCloudGCP
are awailable) to use in "data_sync" and "remove" commands.
This patch focuses on introducing the common class itself
saving original code and definetly needs some RFC later.
It also adds simple DataCloudLOCAL class that uses directory
as a StoragePath and cp as a sync method. Which makes testing
so much easier and covers local scenarios.
And it also introduces ability to run data sync in parallel
by collecting files first and then running sync in a number
of threads.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
These are cmdline shortcuts to change dvc's verbosity
without fiddling around with loglevel in dvc.conf.
Both are added and handled as a common options for
all dvc commands in base cmd class.
There is no need to keep our own implementation, as we have
more versatile cross-platform standard module for it.
For now just replacing Logger class guts with redirects to
'logging' module, and writing everything into single logging
instance.
Currently dvc prints "Unimplemented or unrecognized command: --help"
and returns error code when calling dvc --help.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This will fix CreateSymLinkPermission modification in systems with
locale that is other than english.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This will allow us to make our installer ~4 times smaller(120MB ->
30MB). This patch also has fix for PATH bug and installs dvc to whatever
direstory user specified.
Ruslan Kuprieiev
Ruslan Kuprieiev
Dmitry Petrov