Commit Graph
3327 Commits
Author SHA1 Message Date
Ivan Shcheklein 164c2e8dc1 url info: fix infinite recustion on deepcopy 2019-07-20 15:26:00 -07:00
Ruslan Kuprieiev df28309403 Merge pull request #2300 from efiop/moto
tests: use moto pre-release
2019-07-20 17:12:38 +03:00
Ruslan Kuprieiev f901ba5c10 tests: use moto pre-release
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-20 16:09:44 +03:00
Ruslan Kuprieiev c2d25f455f travis: hadoop: fix typo 2019-07-20 03:00:39 +03:00
Ruslan Kuprieiev 3ccb80466a travis: hadoop: add openjdk-7 ppa 2019-07-20 02:45:38 +03:00
Ruslan Kuprieiev a4e461fc02 travis: hadoop: use openjdk-7 2019-07-20 02:32:49 +03:00
Ruslan Kuprieiev 3becd5c20b Merge pull request #2297 from efiop/hdfs
remote: hdfs: use pyarrow and connection pool
2019-07-20 02:14:56 +03:00
Ruslan Kuprieiev 68bebe7944 remote: hdfs: use pyarrow and connection pool
Related to #1629

Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-20 01:12:17 +03:00
Ruslan Kuprieiev 1e9f1d0619 Merge pull request #2281 from jorgeorpinel/master
help: finish pluralizing "the pipeline" term in output
2019-07-19 12:55:10 +03:00
Jorge Orpinel 6f5449fb83 term: remove "(s)" and "(are) related to the previous commit
PR #2281 feedback
2019-07-18 17:51:20 -06:00
Jorge Orpinel d4f4126ddd help: finish singularizing "pipeline" term...
(in status and repro command help outputs)

Per #2158
2019-07-18 17:51:20 -06:00
Ruslan Kuprieiev 993b1df8ef Merge pull request #2282 from efiop/moto
tests: s3: use moto for multipart test
2019-07-18 23:09:50 +03:00
Ruslan Kuprieiev 958a4fbd3c tests: s3: use moto for multipart test
Fixes #1889

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-18 20:51:49 +03:00
Ruslan Kuprieiev 36c07b8abe Merge pull request #2278 from Suor/ssh-md5-jobs
remote: speed up ssh dir checksum
2019-07-18 19:28:23 +03:00
Alexander Schepanovski 67e50cd71e remote: speed up ssh dir checksum 2019-07-18 22:13:42 +07:00
Ruslan Kuprieiev 432373fb42 stage: properly quote command in exception 2019-07-18 16:10:38 +03:00
Ruslan Kuprieiev 6598b88b77 Merge pull request #2290 from pared/2221
data_cloud: remove backward compatibilty on get_remote
2019-07-18 14:10:50 +03:00
Ruslan Kuprieiev 598ef02558 Merge pull request #2277 from pared/1900
reproduce: use kwargs instead of explicit arguments
2019-07-18 13:57:15 +03:00
pared e39be6d1aa data_cloud: remove compat methods and tests 2019-07-18 10:29:40 +02:00
pawel 14c4a9c52f data_cloud: remove backward compatibilty on get_remote 2019-07-17 23:02:13 +02:00
pared 11cfd663ac reproduce: use kwargs instead of explicit arguments 2019-07-17 21:47:46 +02:00
Ruslan Kuprieiev 9b37c6b021 Merge pull request #2288 from Suor/pool-fixes
remote: pool fixes
2019-07-17 19:56:04 +03:00
Alexander Schepanovski ed387d824d remote: do not swallow exceptions by ssh pool 2019-07-17 22:20:43 +07:00
Alexander Schepanovski 6010ce54cd remote: manually close all ssh pools at the end
Python 2 fails to do it automatically occasionally and shows weird error
messages.

This was harmless but confusing.
2019-07-17 22:20:43 +07:00
Andrew Grigorev 7d3e9808b5 ci: unify pip cache location for all platforms (#2286) 2019-07-17 15:42:38 +03:00
Nabanita Dash 89541b73fa Autocompletion of commands and files (#2226)
* changed " to '

* autocompletion of commands and files

* Revert "autocompletion of commands and files"

This reverts commit 0b394ec841c68357cec6eb1cfa2cb20b636d3739.

revert the changes made

* changed

* bash complete

* modified

* shows subcommands but not files

* modified
2019-07-17 00:29:09 +03:00
Georgy Dyuldin 840318dd22 git: Fix submodule names issue (#2274)
GitPython creates submodules nodes during tree iterate without explicit
passing name attribute. This path replaces such submodules with new
instances with filled name attribute.

Fixes: #1898
2019-07-16 16:38:42 +03:00
Alexander Schepanovski db19957b8a Download refactor (#2275)
* remote: drop not working http resume

Closes #2136

* remote: drop @transfer_context()

It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.

* remote: drop upload/download chunking
2019-07-16 16:27:44 +03:00
Alexander Schepanovski 9766d34cec remote: support older linuxes as ssh remotes (#2276)
* remote: support older linuxes as ssh remotes

Work around md5sum not having --tag in Red Hat 6.3 and friends.
Fixes #2242.

* remote: optimize consequent ssh md5 calls
2019-07-16 14:15:28 +03:00
Ruslan Kuprieiev fc5d3c182c dvc: bump to 0.52.1 0.52.1 2019-07-15 20:33:54 +03:00
Ruslan Kuprieiev a59fdfdd35 stage: don't ignore SIGINT in the child (#2271)
Signal mask is inherited by the child, so when we set it up before
Popen, our child ignores SIGINT, which is wrong. What we should've
done ideally, is to setup the signal handler after fork() but before
exec(). That is pretty much what preexec_fn is used for in Popen, but
unfortunately for us, it is not available on windows, so we can't use
it. What we will do instead is just setup signal handler after Popen()
but before communicate(), which would leave a slight gap where we might
also get affected by SIGINTS meant for the child, but that is the best
we can do.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-15 20:17:10 +03:00
Alexander Schepanovski b5a26e5853 remote: fix collect dir progress (#2270)
We were resolving futures after ThreadPoolExecutor exited and finished
everything.
2019-07-15 18:31:19 +03:00
Paweł Redzyński 8641773992 run: use kwargs instead list of explicit arguments (#2260) 2019-07-15 16:19:11 +03:00
Ruslan Kuprieiev e215364e6d test: http: use random port (#2269)
Fixes #1878

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-15 13:18:10 +03:00
Ruslan Kuprieiev 8c303512e8 dvc: bump to 0.52.0 0.52.0 2019-07-15 11:41:30 +03:00
Ivan Shcheklein 587687fda7 readme: add blog link, fix Twitter link 2019-07-14 14:29:38 -07:00
Alexander Schepanovski b9a3c7c038 remote: use ssh connections pool (#2264)
* remote: use ssh connections pool

* dvc: never use paramiko DEBUG mode

It's ridiculously verbose, makes logs unreadable.
2019-07-14 23:30:24 +03:00
mindsbackyard 5976b5e466 disable analytics warning message on init if disabled globally/system… (#2268)
* disable analytics warning message on init if disabled globally/system-wide

* rename Analytics._is_enabled
Analytics.is_enabled

* refactor trying reduce cognitive complexity metric

* fix missing imports

* fix mistaken reference of self
2019-07-14 23:29:30 +03:00
Ruslan Kuprieiev 10121411e0 import: don't forget to add outs to gitignore (#2266)
Fixes #2263

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-14 21:35:23 +03:00
Ruslan Kuprieiev 94fe86afe8 tests: import-url: rewrite and move error msg test (#2262)
This one was failing randomly.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-14 03:09:08 +03:00
Paweł Redzyński e8623a0c69 run: rerun deterministic callback (#2254)
* run: rerun deterministic callback

* test_run: refactor deterministic run tests to use pytest
2019-07-14 02:25:13 +03:00
Ruslan Kuprieiev 2c2f573897 import: remove beta warnings (#2257)
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-07-13 23:29:36 +03:00
Paweł RedzyńskiandRuslan Kuprieiev 393d926b74 state: use dvcignore when accessing metadata (#2161)
* state: use ignore file handler when accessing metadata

* fs: don't add dir size to overall size for dir

* ignore: use posix-like path on windows

* ignore: remove tree from dvc_walk arguments

* ignore: handle DvcIgnoreFilter creation outside of DVC repo

* ignore: throw on dvcignore file under output dir, but not dependency dir

* ignore: move ignore initialization to Repo init

* ignore: work in progress

* ignore: work in progress 2

* ignore: throw on .dvcignore upon collecting dir

* ignore: refactor

* ignore: loading upper level ignores in stages()

* Update dvc/repo/__init__.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* dvcignore review refactor
2019-07-12 11:50:22 +03:00
Paweł Redzyński 217f2c4dbd imp_url: normalize path for default out name (#2248) 2019-07-12 09:20:47 +03:00
Ruslan Kuprieiev 95110a865c dvc: bump to 0.51.2 0.51.2 2019-07-11 12:36:35 +03:00
Ruslan Kuprieiev eaab968036 build: windows: don't forget to pull images 2019-07-11 08:52:22 +03:00
Ruslan Kuprieiev 72a61b5599 build: windows: don't use abs path for iscc 2019-07-10 20:59:04 +03:00
Ruslan Kuprieiev 375dc261d0 build: windows: use call to call refreshenv 2019-07-10 18:03:15 +03:00
Ruslan Kuprieiev ccb0d7d1b3 travis: build_package: use which 2019-07-10 15:46:51 +03:00
Ruslan Kuprieiev d5c473acd4 travis: fix build_package.sh 2019-07-10 15:41:52 +03:00