In all operations we're dealing with a single stream,
so there is no good reason to process targets in parallel
as they all use the same resource to pull\push data to.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Number of targets from command line doesn't have to match
number of actual files that we've processed, as it may as
well be a directory.
Fixes#204
Signed-off-by: Ruslan Kuprieiev <[email protected]>
These wasn't caught in pull-request as it doesn't
have secure env vars to test sync/pull/push/status.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This command shows status for data files. I.e.
$ dvc status data/
new file: .cache/Posts-train.tsv_2ccd8e7
new file: data/llss
modified: .cache/Posts.xml_512a612
new file: .cache/empty_0000000
new file: data/oss
new file: .cache/matrix-test.p_69b924d
new file: .cache/dir21111/Posts.xml.tgz_cd71d01
so it will tell us if there are any new/modified/deleted data
files that you might want to push/pull/etc.
Fixes#158
Signed-off-by: Ruslan Kuprieiev <[email protected]>
We already have 'sync' command, but users report it to be confusing
when both local and remote files exist, so lets introduce two direct
commands to implicitly push and pull from cloud.
Fixes#157.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Cloud is determined from config and stays the same from that moment,
so it makes sense to parse it once and store in settings, so whoever
needs it, could just use self.settings.cloud. This is going to be
more useful in the few next commits, where push/pull/status are introduced.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This will make the structure of dvc more linear
making it easier to read. Plus, these changes
are needed in preparation for libdvc, as they
move all cmdline parsing to one common place.
This patch also refactors our settings system
also make it more linear and remove abstraction
levels that are not needed and only cause confusion.
There is a lot more to be done in preparation for
dvc lib and RFC overall, but this a first step
in that long journey.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
It replaces POOL_SIZE constant and by default equals to the
number of cpus available.
map_progress is a helper for 'sync' and 'import' commands to
run in a multi-threaded mode.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This allows us to download files in a few threads.
Also includes RFC, as we can now get rid of import-bulk.py.
v2: use POOL_SIZE=4
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This patch introduces DataCloud class that used config to
determine proper cloud class(DataCloudAWS and DataCloudGCP
are awailable) to use in "data_sync" and "remove" commands.
This patch focuses on introducing the common class itself
saving original code and definetly needs some RFC later.
It also adds simple DataCloudLOCAL class that uses directory
as a StoragePath and cp as a sync method. Which makes testing
so much easier and covers local scenarios.
And it also introduces ability to run data sync in parallel
by collecting files first and then running sync in a number
of threads.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
remove aws creds from dvc.conf; instead, look in: look for creds path in dvc.conf, default to ~/.aws/credentials
warn if aws creds are specified but we fall back anyway
clean up config.get_cloud -> config.cloud
sanity check the config: complain in nicer way if cloud, aws creds, or various other requirements aren't set
Ruslan Kuprieiev
Ruslan Kuprieiev
Dmitry
Earl