Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-05 15:28:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,322 Commits 7 Branches 579 Tags
a4e461fc0211885b16ca0903accfbfa8bcbb998f
Commit Graph
806 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 68bebe7944 remote: hdfs: use pyarrow and connection pool 
Related to #1629

Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-20 01:12:17 +03:00
Ruslan Kuprieiev 958a4fbd3c tests: s3: use moto for multipart test 
Fixes #1889

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-18 20:51:49 +03:00
Ruslan Kuprieiev 6598b88b77 Merge pull request #2290 from pared/2221 
data_cloud: remove backward compatibilty on get_remote
 2019-07-18 14:10:50 +03:00
pared e39be6d1aa data_cloud: remove compat methods and tests 2019-07-18 10:29:40 +02:00
Alexander Schepanovski ed387d824d remote: do not swallow exceptions by ssh pool 2019-07-17 22:20:43 +07:00
Georgy Dyuldin 840318dd22 git: Fix submodule names issue (#2274) 
GitPython creates submodules nodes during tree iterate without explicit
passing name attribute. This path replaces such submodules with new
instances with filled name attribute.

Fixes: #1898
 2019-07-16 16:38:42 +03:00
Alexander Schepanovski db19957b8a Download refactor (#2275) 
* remote: drop not working http resume

Closes #2136

* remote: drop @transfer_context()

It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.

* remote: drop upload/download chunking
 2019-07-16 16:27:44 +03:00
Ruslan Kuprieiev e215364e6d test: http: use random port (#2269) 
Fixes #1878

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-15 13:18:10 +03:00
Ruslan Kuprieiev 10121411e0 import: don't forget to add outs to gitignore (#2266) 
Fixes #2263

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-14 21:35:23 +03:00
Ruslan Kuprieiev 94fe86afe8 tests: import-url: rewrite and move error msg test (#2262) 
This one was failing randomly.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-14 03:09:08 +03:00
Paweł Redzyński e8623a0c69 run: rerun deterministic callback (#2254) 
* run: rerun deterministic callback

* test_run: refactor deterministic run tests to use pytest
 2019-07-14 02:25:13 +03:00
Paweł RedzyńskiandRuslan Kuprieiev 393d926b74 state: use dvcignore when accessing metadata (#2161) 
* state: use ignore file handler when accessing metadata

* fs: don't add dir size to overall size for dir

* ignore: use posix-like path on windows

* ignore: remove tree from dvc_walk arguments

* ignore: handle DvcIgnoreFilter creation outside of DVC repo

* ignore: throw on dvcignore file under output dir, but not dependency dir

* ignore: move ignore initialization to Repo init

* ignore: work in progress

* ignore: work in progress 2

* ignore: throw on .dvcignore upon collecting dir

* ignore: refactor

* ignore: loading upper level ignores in stages()

* Update dvc/repo/__init__.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* dvcignore review refactor
 2019-07-12 11:50:22 +03:00
Paweł Redzyński 217f2c4dbd imp_url: normalize path for default out name (#2248) 2019-07-12 09:20:47 +03:00
Alexander Schepanovski 732fd01cc5 test: drop a reimport (#2247) 2019-07-10 03:24:59 +03:00
Alexander Schepanovski 90b1a80240 test: test ssh dir output (#2246) 2019-07-10 03:24:35 +03:00
Ruslan Kuprieiev fcc8938b33 test: repro: fix formatting 2019-07-08 21:27:32 +03:00
Ruslan Kuprieiev e8b0d2d987 test: disable repro s3 test on windows 2019-07-08 21:24:23 +03:00
Ruslan Kuprieiev f3c4aff6c5 test: don't use python to call awscli 2019-07-08 17:44:18 +03:00
Ruslan Kuprieiev af34867d96 test: try calling awscli through python 2019-07-06 02:49:08 +03:00
Ruslan Kuprieiev 932b0d7af9 introduce dvc update for ext repos (#2218) 
* erepo: dvc: implement `update` for imports from ext repos

Fixes #1774
Fixes #2139
Fixes #2201

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* deps: repo: make  private

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* update: only support repo imports

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-06 02:28:35 +03:00
Ruslan Kuprieiev fb06306ea8 Merge pull request #2231 from efiop/2146 
checkout: safe_remove output if no checksum is found
 2019-07-06 02:25:38 +03:00
Ivan Shcheklein 8fd18f5017 url info and repo grap optimization: address review comments 2019-07-04 23:35:58 -07:00
Ivan Shcheklein 0abf530bef tests: add tests for some previously failing calls in url info 2019-07-04 19:34:43 -07:00
Ruslan Kuprieiev 2013e21aaf checkout: safe_remove output if no checksum is found 
Fixes #2146

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-05 02:59:57 +03:00
Alexander Schepanovski 9ba54c1f56 dvc: add rev param to api.open()/read()/get_url() 2019-07-01 14:40:48 +07:00
Alexander Schepanovski 3187040fe8 dvc: raise when Repo.open() fails 2019-07-01 14:40:48 +07:00
pared 7f8da34cf2 utils: extend to_chunks functionality, can specify number of chunks or length of chunk 2019-06-28 13:52:59 +02:00
pared 3d5c42e2fe state: test: refactor existing state test to pytest 2019-06-28 13:52:59 +02:00
pared ff1809b060 state: bulk sql for unused links deletion 2019-06-28 13:52:59 +02:00
Jorge Orpinel 91bd0faa87 dvc: feedback to #2112 round 1 2019-06-26 16:25:56 -05:00
Jorge Orpinel f004a84e7c tests: fix failing test_add (due to extra 's' in word) 
To close #2112
 2019-06-26 14:05:18 -05:00
Jorge Orpinel fba7d85cbd dvc: Use "DVC-file" term as much as possible 
Keep "stage file" in the context of pipelines

No issue
 2019-06-26 14:05:18 -05:00
Ruslan Kuprieiev bbd31cdb19 Merge pull request #2189 from Suor/some-fixes 
Fix relpath() with PathInfo
 2019-06-26 16:45:20 +03:00
Alexander Schepanovski bdd5ca48da dvc: fix relpath() 
- accept Path-like object as start in Python3.5-
- always return str

Closes #2188.
 2019-06-26 15:19:22 +07:00
Alexander Schepanovski 6e761c7e7a dvc: check stage name before running command 
Closes #2150.
 2019-06-25 19:21:16 +07:00
Ruslan Kuprieiev dd6ed7e7e9 test: s3: remove debug marker 2019-06-25 13:13:26 +03:00
Ruslan Kuprieiev d58341b57c Merge pull request #2162 from Suor/api-remote-repo 
dvc: add repo as url support to API
 2019-06-25 11:47:33 +03:00
Alexander Schepanovski 4d248b4183 dvc: support external repos in api functions 2019-06-25 13:08:03 +07:00
Ruslan Kuprieiev dceca73f00 tests: debug appveyor s3 failure 2019-06-25 05:26:45 +03:00
Ruslan Kuprieiev 5802efdfdb Merge pull request #2177 from shcheklein/fix-2170 
Fix 2170: pipeline show command does not work
 2019-06-24 00:20:22 +03:00
Ivan Shcheklein 478d6aec2f test: ls/dir depends on platform, dir is not supported by Mac 2019-06-23 14:09:22 -07:00
Ruslan Kuprieiev 1cb70fc631 tests: win: don't cleanup processes on Popen 
Workaround for two bugs:

https://bugs.python.org/issue37380

and

https://github.com/gitpython-developers/GitPython/issues/546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-23 22:35:50 +03:00
Ruslan Kuprieiev 918ef646dc appveyor: test py2 and py3 
Related to #2166

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-23 13:33:47 +03:00
Ruslan Kuprieiev 81fd0b3415 dvc: destroy: don't remove data 
Fix #1011

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-21 13:13:29 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-21 01:16:00 +03:00
Ruslan Kuprieiev 85fd990849 Merge pull request #2153 from vasinkd/SIGINT_support_2 
dvc: handle sigint when running a command with `dvc run`
 2019-06-20 19:57:24 +03:00
Kirill Vasin 1734e25827 added old signal persistance and fixed tests accordingly 2019-06-20 19:21:53 +03:00
Kirill Vasin 5457c16e76 replace exception catch with signal 2019-06-20 13:39:22 +03:00
pared c9b3bdc927 ignore: remove dulwich in favour of pathspec 2019-06-20 12:15:42 +02:00
Ruslan Kuprieiev 676021d563 Merge pull request #2149 from efiop/1772 
pkg: handle install/version errors
 2019-06-19 17:27:05 +03:00