Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-04 23:09:44 +01:00
Code Issues Packages Projects Releases Wiki Activity
2,373 Commits 7 Branches 579 Tags
1a5a840aeed3300a557faef72fa1eebefd6e34a2
Commit Graph
90 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8 
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-01-30 03:48:36 +02:00
Mr. Outis 0d46921ec7 py2: add unicode literals 
Fix #1512
 2019-01-27 20:40:21 -06:00
Paweł Redzyński 64a4883ba1 state: cast file size to string 
Fixes #1485
 2019-01-13 12:09:13 +02:00
pared 6f7e2095d3 project: support recursive sync operations 2019-01-10 18:02:23 +01:00
Paweł Redzyński 61e0ea9478 [WIP] 1445 warning on outdated stage (#1466) 
* prepare test structure

* add warning on empty out info, test

* reformat line lengtha according to pep8

* change warning message
 2019-01-05 14:03:13 +02:00
Ruslan Kuprieiev 484f318c27 remote: local: support relative paths 
Fixes #1421

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-01-04 21:14:03 +02:00
Ruslan Kuprieiev d3de148fc4 azure: fix backward compatibility 
Fixes #1458

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-12-28 19:58:25 +02:00
Ruslan Kuprieiev 3840b8fa75 test: gc: skip if key file does not exist 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-12-22 04:49:20 +02:00
Ruslan Kuprieiev 4029116b85 remote: RFC: unify push/pull 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-12-21 18:18:33 +02:00
Ruslan Kuprieiev d5dd0fd804 test: don't use sleep 
Since we now use file size to help detect file changes on filesystems
with bad mtime resolution, we no longer need these explicit sleep()
calls in our tests for mac os.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-12-17 12:42:35 +02:00
Ruslan Kuprieiev eaa2d559db azure: support prefix 
Fixes #1336

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-12-12 19:54:41 +02:00
Ruslan Kuprieiev a67988a80e test: enable flake8 checks for tests 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-12-06 06:40:55 +02:00
Ruslan Kuprieiev 75ab3074a1 dvc: dynamically load default cloud 
Fixes #1390

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-12-01 01:45:18 +02:00
Mr. Outis 1e3d752917 remote: HTTP implemented (support only for downloading) (#1350) 
* remote: HTTP implemented (support only for downloading)

* Remove duplicated content in the schema

* Chomp trailing spaces in test_remote

* remote http: Implement pull/fetch

* add some tests

* Update dvc/remote/http.py

Co-Authored-By: mroutis <[email protected]>

* Update dvc/remote/http.py

Co-Authored-By: mroutis <[email protected]>

* correct test

* rehprase error message

* add __init__ for utils

* try to fix windows test
 2018-11-24 21:22:05 +02:00
Ruslan Kuprieiev 20517e44b8 test: save cache backup 2018-11-08 14:26:52 +02:00
Ruslan Kuprieiev 683f2c9a57 tests: debug travis failure 2018-11-07 23:05:58 +02:00
Ruslan Kuprieiev aa8b52908e test: sleep before checkout 2018-11-07 21:23:48 +02:00
Ruslan Kuprieiev 49cadb68da test: don't forget to sleep() 2018-11-06 17:39:57 +02:00
Ruslan Kuprieiev f00d08d46d remote: properly collect dir cache 
Fixes #1293

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-11-06 14:19:38 +02:00
Ruslan Kuprieiev 78a20a3895 remote: properly check reqs for a legacy config 
Fixes #1231

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-10-18 03:12:47 +03:00
Ruslan Kuprieiev 31fed211f6 test: use suffixes and prefixes in tmpdir names 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-10-14 00:37:02 +03:00
Ruslan Kuprieiev edf73dc52f test: skip hdfs test if can't connect 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-10-01 16:29:55 +03:00
Ruslan Kuprieiev 59dcdb02ea state: optimize operations 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-09-17 10:47:16 +03:00
Ruslan Kuprieiev c6f642fd6b dvc: add --recursive 
Fixes #1026

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-08-21 10:41:19 +03:00
Ruslan Kuprieiev 7e4f39d8eb dvc: show filenames in push/pull output 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-26 16:50:25 +03:00
Ruslan Kuprieiev 66f1777d9c test: ssh: don't try to ask for a password 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-25 17:28:58 +03:00
Ruslan Kuprieiev bc4a5d78e3 tests: data cloud: don't forget to test fetch 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-19 08:24:55 +03:00
Ruslan Kuprieiev f54428d5ed test: properly check for hadoop availability 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-18 10:12:49 +03:00
Ruslan Kuprieiev c636581699 test: gc: catch OSError 2018-07-12 10:41:34 +03:00
Ruslan Kuprieiev 4668c10e6e test: gs: catch WinError 2018-07-12 10:32:34 +03:00
Ruslan Kuprieiev 9be161de54 test: use IOError instead of FileNotFoundError 2018-07-12 10:25:40 +03:00
Ruslan Kuprieiev 79826b21a4 test: gcp: handle FileNotFoundError 2018-07-12 10:14:15 +03:00
Ruslan Kuprieiev f476143c52 test: check if we need to test RemoteAzure with _should_test_azure 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-12 09:13:07 +03:00
Clemens Wolff 97dfc88647 Flag skipped tests during test run 2018-07-10 16:44:28 -04:00
Clemens Wolff fcbc6d115c Wire-up Azure in TestDataCloud 2018-07-10 16:33:52 -04:00
Clemens Wolff 1b1ca68b64 Fix inspections 2018-07-10 16:33:52 -04:00
Clemens Wolff 9724bd94aa Replace os.system with subprocess.check_output 
It is generally recommended to use the subprocess module instead of the
os module to run subshell commands given since the former makes it
easier to comunicate with the process, has an improved security model,
etc. More information is available in [1].

Note that this change uses `check_output` instead of `check_call` to
prevent the command output from being printed to stdout.

This change also introduces two additional enhancements:
- Skip the hadoop test if the hadoop binary is not available.
- Skip the ssh test if the ssh binary is not available.

[1] https://docs.python.org/3/library/subprocess.html#replacing-os-system
 2018-07-10 16:33:14 -04:00
Clemens Wolff 581132bbe0 Fix environment variable name 2018-07-10 13:54:49 -04:00
Clemens Wolff 6882a781a4 Add Azure Blob Storage remote 
This change adds a new cloud remote based on Azure Blob Storage [1] and
the azure-storage-blob SDK [2].

The implementation closely follows the patterns laid out by the existing
cloud remotes (AWS and GCP). Notable differences include:

- No use of a lock in the progress-bar callback and no requirement to
  look up total transfer size ahead of time: the Azure Storage SDK
  provides the total transfer bytes and current transfer offset in its
  progress callback.

- When `exists` is called, check the existence of every blob instead of
  listing the entire container. Container list operations can be
  expensive so unless the number of checked files is very large, making
  separate requests is likely going to be more efficient.

- For increased flexibility, access credentials for the Azure Storage
  backend can be provided either connection-string style via the remote
  URL or via environment variables. The Azure Storage backend can be
  emulated via Azurite [3].

[1] https://azure.microsoft.com/en-us/services/storage/blobs/
[2] https://github.com/Azure/azure-storage-python
[3] https://github.com/Azure/azurite

This change was created together with @EricSchles.
 2018-07-09 14:47:40 -04:00
Ruslan Kuprieiev a392289cbf travis: run hdfs test only on Linux 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 10:02:43 +03:00
Ruslan Kuprieiev 591cd90583 travis: enable hdfs test 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-07-01 09:44:42 +03:00
Ruslan Kuprieiev fa9a79008b test: use gauth 2018-07-01 03:01:16 +03:00
Ruslan Kuprieiev 12edcbaa04 remote: support HDFS for push/pull/status from local repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-30 22:54:49 +03:00
Ruslan Kuprieiev 5d85b19e77 test: enable ssh test on osx 
Fixes #798

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-30 16:56:17 +03:00
Ruslan Kuprieiev 555e802ed2 RFC/perf: data cloud logic 
Properly splitting jobs between thread workers and removing unnecessary logic.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-30 07:44:11 +03:00
Ruslan Kuprieiev db7251b3ea dvc: support ssh deps/outs/cache 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-23 18:47:52 +03:00
Ruslan Kuprieiev c274bfee3e RFC: remove unused code 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-22 18:46:09 +03:00
Ruslan Kuprieiev d0b730f62f RFC: data_cloud and remotes 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-22 18:21:50 +03:00
Ruslan Kuprieiev 0a951f5ab3 dvc: move cloud/ drivers to remote/ 
Fixes #741

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-20 19:42:33 +03:00
Ruslan Kuprieiev 88397ef430 dvc: use Remote concept in existing code 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2018-06-04 22:28:14 +03:00