Ruslan Kuprieiev
a0d364e78a
version: fix warning
2019-10-22 23:21:22 +03:00
Ruslan Kuprieiev
803ffaec57
remote: local: don't auto-create directory in __init__
...
Imagine the scenario of shared cache. To set it up, user would do:
```
dvc config cache.dir /path
dvc config cache.shared group
```
Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.
The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.
Fixes #2563
2019-10-14 15:50:49 +03:00
Ruslan Kuprieiev
7ed8cea12a
dvc: move local/__init__.py -> local.py
...
No need to keep that subdir for 1 file.
2019-10-10 23:34:02 +03:00
Ruslan Kuprieiev
cba8ac7a8f
dvc: load state when taking repo lock in @locked
...
We are always loading state when taking a repo lock, so doing it right
in @locked decorator simplifies the code for Repo methods.
2019-10-10 01:08:28 +03:00
Ruslan Kuprieiev
3b8c48793b
scripts: don't include completion scripts into mac pkg
...
Fixes https://github.com/iterative/dvc/issues/2585
2019-10-08 16:48:36 +03:00
Ruslan Kuprieiev
41552d0af7
deb/rpm/pkg: add zsh completion scripts
...
Fixes #2181
2019-10-03 22:02:33 +03:00
Ruslan Kuprieiev
132de572af
run: improve pyenv PATH modification criteria
...
https://discordapp.com/channels/485586884165107732/485596304961962003/628766062472724510
Related to #2506
2019-10-02 06:33:01 +03:00
Ruslan Kuprieiev
ee976a1c1a
run: make shell not load config files
...
Config files like `~/.zshenv`, `~/.bashenv` etc, are loaded on each
shell invocation, which might overwrite the environment breaking the
commands we are trying to wrap. To fix that, we need to provide
appropriate flags to the shell we are using to avoid loading those
configs.
Fixes #2506
2019-09-30 19:34:55 +03:00
Ruslan Kuprieiev
fd14ca3269
dvc: remote: optimize imports
...
Decreases `time dvc --version` from ~1.4 sec to ~0.3 sec.
Related #2495
Related #2445
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-09-23 11:30:45 +03:00
Ruslan Kuprieiev
43975c8618
dvc: remove PATH prefixes appended by pyenv
...
Unlike PyInstaller, pyenv doesn't leave backups of original env vars
when it modifies them. If we look into the shim, pyenv and pyenv-exec,
we can figure out that the PATH is modified like this:
PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH
where
PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
bin_path - might not start with $PYENV_ROOT as it runs realpath on
it, see pyenv source code.
plugin_bin - might contain more than 1 entry, which start with
$PYENV_ROOT, see pyenv source code.
So having this, we can make a rightful assumption about what parts of the PATH
we need to remove in order to get the original PATH.
Fixes #2506
2019-09-23 04:25:42 +03:00
Ruslan Kuprieiev
31b40816b5
daemon: use sys.argv[0] instead of -m dvc
...
This way dvc will actually launch the same script that it is running
from and not installed dvc module, that might not match.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-09-22 05:18:51 +03:00
Ruslan Kuprieiev
9d6ca1ef6c
dvc: move repo lock from CLI to API
...
For the longest time we've had `repo.lock`, but were only using it from
CLI when running specific command, instead of simply properly taking it
on API operations. This patch takes care of that.
2019-09-21 05:10:06 +03:00
Ruslan Kuprieiev
f0c43b4e02
path_info: don't forget to override as_posix method
...
Unlike original implementation [1] that uses `str()` we actually need
to use `fspath`, because we've overriden `__str__` method to return
relative paths, which will break original `as_posix`.
[1] https://github.com/python/cpython/blob/v3.7.0/Lib/pathlib.py#L692
Fixes #2483
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-09-13 04:52:03 +03:00
Ruslan Kuprieiev
384fd4cd9a
test: repro: fix link types
2019-09-10 02:12:28 +03:00
Ruslan Kuprieiev
ce27444dd2
scm: remove pkg install leftovers
...
Leftover code was causing problems when working with symlinks, because
it tried to use `realpath` on symlinks to further use it to determine
parent directory.
Fixes #2476
2019-09-09 19:43:10 +03:00
Ruslan Kuprieiev
357ce82f28
run: introduce --always-changed
...
This is an extension of our callback stages (no deps, so always
considered as changed) for stages with dependencies, so that these
stages could be used properly in the middle of the DAG. See attached
issue for more info.
Related to #2378
2019-09-09 18:03:28 +03:00
Ruslan Kuprieiev
60d8e8fbec
erepo: fix environment before running git
...
Fixes #2471
2019-09-06 23:42:46 +03:00
Ruslan Kuprieiev
096a043e21
dvc: don't include gssapi
...
If user needs it, he can install it by himself through pip.
https://github.com/iterative/dvc/pull/2424#issuecomment-523990535
https://github.com/iterative/dvc.org/pull/582
2019-08-24 07:20:06 +03:00
Ruslan Kuprieiev
89d3ccae0d
test: run: refactor signal handler tests
...
Current test is testing for signal handler restoration in an obscure
way. As a result, the test wasn't actually testing anything, because
KeyboardInterrupt is raised by default_int_handler, and so raising it
in a mock won't be caught by signal handlers, because they are dealing
with real singnals.
Fix #2406
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-19 14:49:19 +03:00
Ruslan Kuprieiev
06d347178f
remote: base: remove whitespace
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-13 02:25:15 +03:00
Ruslan Kuprieiev
c308ad8a58
remote: base: don't forget about the progress bar
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-13 02:08:29 +03:00
Ruslan Kuprieiev
baa548eec9
remote: base: call makedirs in move
...
Remotes like s3/gs/etc don't need to create dirs in advance.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-10 04:53:50 +03:00
Ruslan Kuprieiev
919a49a231
travis: windows: cache chocolatey downloads
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-10 04:16:11 +03:00
Ruslan Kuprieiev
eb3b3f9f4e
travis: windows: skip install stage
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-10 04:15:22 +03:00
Ruslan Kuprieiev
3f19036646
travis: osx: speedup install stage
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-10 04:15:22 +03:00
Ruslan Kuprieiev
2b36d25f49
remote: base: remove unused argument
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev
9955f1c64f
utils: merge isinstance checks
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev
640e737570
remote: base: remove unused argument
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev
24428e47c3
remote: local: cache_exists: match parameters from the BASE
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev
713e288adc
remote: base: checkout: remove unnecessary return
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev
b2fe3f7a71
utils: remove unnecessary else
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev
9c5fd0c76c
dvc: support group cache mode
...
In this mode, dvc will set appropriate file/dir permissions to support
sharing cache between different users within the same group.
Fixes #2287
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 17:10:40 +03:00
Ruslan Kuprieiev
5aae88b316
local: don't try to protect file if it is already protected
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-09 16:46:24 +03:00
Ruslan Kuprieiev
4dd221dab9
reproduce: properly handle --single-item case
...
Fix #2354
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-08-01 22:36:47 +03:00
Ruslan Kuprieiev
56b162d2d0
readme: fixed formatting
2019-07-31 21:09:46 +03:00
Ruslan Kuprieiev
3bdcaba39f
travis: osx: retry brew update
...
Fixes #2347
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-31 20:49:08 +03:00
Ruslan Kuprieiev
79052ddd5c
readme: fix message about the number of install options
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-31 20:37:30 +03:00
Ruslan Kuprieiev
5e02a90e13
version: don't skip link info and skip if there is no psutil
...
Fixes #2336
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-29 03:33:07 +03:00
Ruslan Kuprieiev
4a4ddf078c
version: temporarily make psutil optional
...
Workaround for https://github.com/iterative/dvc/issues/2284
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-25 01:52:11 +03:00
Ruslan Kuprieiev
641edacef6
import-url: lock by-default
...
Fixes #2238
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-22 20:24:00 +03:00
Ruslan Kuprieiev
4a0b3a807d
import: only donwload specific output
...
Fixes #2252
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-22 15:04:10 +03:00
Ruslan Kuprieiev
f901ba5c10
tests: use moto pre-release
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-20 16:09:44 +03:00
Ruslan Kuprieiev
68bebe7944
remote: hdfs: use pyarrow and connection pool
...
Related to #1629
Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-20 01:12:17 +03:00
Ruslan Kuprieiev
2013e21aaf
checkout: safe_remove output if no checksum is found
...
Fixes #2146
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-05 02:59:57 +03:00
Ruslan Kuprieiev
da63c0ffdc
travis: enable windows build and ditch appveyor
...
Fixes #2176
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-07-03 21:09:23 +03:00
Ruslan Kuprieiev
babc4a7f7e
appveyor: require py3 to pass
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-25 04:48:51 +03:00
Ruslan Kuprieiev
1123a303bf
import: require 'path' argument
...
`dvc import` doesn't yet support importing the whole package.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-25 01:54:32 +03:00
Ruslan Kuprieiev
2141a5ac65
import: add "beta feature" warning
2019-06-25 01:54:32 +03:00
Ruslan Kuprieiev
12c7689a74
dvc: update paramiko requirements
...
Fixes #2155
No longer need to explicitly ignore paramiko warnings.
Big thanks to @cowlinator for the heads-up!
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-24 00:24:06 +03:00
Ruslan Kuprieiev
9bca1c9bff
dvc: use remove() instead of shutil.rmtree()
...
Our `remove()` is able to handle read-only files which we or git
sometimes leave behind.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-23 18:11:30 +03:00
Ruslan Kuprieiev
918ef646dc
appveyor: test py2 and py3
...
Related to #2166
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-23 13:33:47 +03:00
Ruslan Kuprieiev
e890c0558a
dvc: fix help messages for get and import-url
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-22 00:50:22 +03:00
Ruslan Kuprieiev
81fd0b3415
dvc: destroy: don't remove data
...
Fix #1011
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-21 13:13:29 +03:00
Ruslan Kuprieiev
d795cdf1ba
dvc: address review from #2160
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-21 12:27:24 +03:00
Ruslan Kuprieiev
1ba8e0171d
dvc: replace pkg concept with external repo
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-21 01:16:00 +03:00
Ruslan Kuprieiev
be255ba4b0
http: enable progress bar
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-19 14:22:52 +03:00
Ruslan Kuprieiev
d3bdca3f13
tests: use shortuuid for shorter paths
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-18 16:11:39 +03:00
Ruslan Kuprieiev
14dc0c0115
pkg: handle install/version errors
...
Fixes #1772
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-18 16:09:06 +03:00
Ruslan Kuprieiev
4947069c4f
pkg: introduce -f|--force for pkg install
...
Fixes #1773
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-17 21:06:10 +03:00
Ruslan Kuprieiev
914a587123
pkg: implement dvc pkg get command
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-17 18:12:36 +03:00
Ruslan Kuprieiev
5941ace270
dvc: initial 'dvc pkg' implementation
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-17 18:12:36 +03:00
Ruslan Kuprieiev
0d330c1b9b
dvc: move Updater out of Repo()
...
Fixes #2039
Required by #2012 to be able to fetch data through the API without
interuption from updater.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-14 00:54:36 +03:00
Ruslan Kuprieiev
b1eebe2f86
diff: don't use self
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-14 00:20:56 +03:00
Ruslan Kuprieiev
bfdb990b3b
utils: dont use str() for dvc_walk
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-13 11:51:14 +03:00
Ruslan Kuprieiev
39a8cab49d
test: fix s3 test with extra args
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-11 16:28:11 +03:00
Ruslan Kuprieiev
8ef9133194
remote: use RemoteConfig class
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-06 16:00:44 +03:00
Ruslan Kuprieiev
f69836c2b3
tests: remove fixme msg
...
https://github.com/iterative/dvc/issues/2086
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-06 14:41:02 +03:00
Ruslan Kuprieiev
9e9e078de3
config: move logic from CLI to API
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-06 14:41:02 +03:00
Ruslan Kuprieiev
122b94129f
Revert "tests: test hdfs on cron and releases"
...
This reverts commit 76c4755865 .
2019-06-02 15:07:36 +03:00
Ruslan Kuprieiev
76c4755865
tests: test hdfs on cron and releases
...
Related to
https://github.com/iterative/dvc/issues/1629#issuecomment-498023424
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-06-02 14:44:56 +03:00
Ruslan Kuprieiev
31848d5c7f
remote: support remote:// notation for push/pull/gc
...
Fixes #2066
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-29 12:54:31 +03:00
Ruslan Kuprieiev
4e9bc48f2a
travis: build binary on python 3
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-25 02:40:16 +03:00
Ruslan Kuprieiev
095221ef7d
appveyor: build windows binary on python 3.7
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-25 02:40:16 +03:00
Ruslan Kuprieiev
abb6725a40
appveyor: adjust artifact regex
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-20 17:09:12 +03:00
Ruslan Kuprieiev
362866ba97
appveyor: specify artifact for specific provider
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-20 16:25:32 +03:00
Ruslan Kuprieiev
00bb09c3fe
appveyor: don't deploy to pypi
...
No need to deploy it again, since we have universal wheels anyways.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-20 15:36:17 +03:00
Ruslan Kuprieiev
5e49797264
travis: don't forget to install twine
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-20 15:26:16 +03:00
Ruslan Kuprieiev
7675888a83
appveyor: deploy to pypi
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-20 02:21:35 +03:00
Ruslan Kuprieiev
b8fd41e923
travis: deploy to pypi
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-20 01:49:00 +03:00
Ruslan Kuprieiev
1d5d608c21
test: make sure checksums are not recomputed on checkout
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-13 21:07:12 +03:00
Ruslan Kuprieiev
dd6217120f
remote: checkout: save checksums without re-computation
...
We do a similar thing on save, where we set cache checksum from already
computated ones for actual data. This patch does the same thing but in
revers: on checkout we know cache checksums, so we can set data
checksums from it without having to re-calculate it.
Fixes #1991
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-12 23:42:29 +03:00
Ruslan Kuprieiev
b4143b19dd
remote: unify class names
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-12 17:29:38 +03:00
Ruslan Kuprieiev
a593a9ce2e
dvc: refactor PathInfos
...
Following same conventions from remotes/outsputs/dependencies.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-12 17:18:46 +03:00
Ruslan Kuprieiev
d1881fd1b1
remote: separate http and https
...
Fixes #1988
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-12 16:39:40 +03:00
Ruslan Kuprieiev
797b356e3a
pyinstaller: remove hooks that are already released
...
We've contributed these hooks a long time ago and they got finally
released, so we can remove our copies.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-11 21:44:55 +03:00
Ruslan Kuprieiev
aacb0b4c98
pyinstaller: fix build for oss2
...
This version added support for oss as a remote, which has caused some
issues when building the binary package. To fix those, we need two
pyinstaller hools:
1) hook-Crypto.py
This one has been added to PyInstaller's upstream, but wasn't
released yet, so we are using a copy of it for now.
2) hook-aliyunsdkcore.py
This one is a new one. Sent it to PyInstaller's team in a PR [1].
[1] https://github.com/pyinstaller/pyinstaller/pull/4228
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-11 21:35:41 +03:00
Ruslan Kuprieiev
d4ef461fce
version: use CmdBaseNoRepo
...
Currently `dvc version` outside of dvc repo raises "not dvc repo" error,
even though it is not really using repo in any way.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-11 17:20:30 +03:00
Ruslan Kuprieiev
cc2f28c12b
cmd: use CmdBaseNoRepo for commands without repo
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-05-11 17:09:19 +03:00
Ruslan Kuprieiev
9c77f2818c
stage: commit cached stage to avoid data duplication
...
Fixes #1655
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-24 21:14:08 +03:00
Ruslan Kuprieiev
37ee55b1f8
test: test ssh
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev
49f87a5bff
use ValueError instead of JSONDecodeError
...
JSONDecodeError is only available in never python 3 versions and still
inherits from ValueError, so we could just use it.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev
b97b99cf15
test: func: use pytest for outputs/dependencies
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev
44b95f23a3
remote: ssh: support directories
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev
00cfb04fea
remote: generalize file/dir handling
...
Fixes #1654
Fixes #1655
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev
41b67c3730
move: move directory into an existing directory
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-24 04:56:33 +03:00
Ruslan Kuprieiev
9618b32fe8
pkg: comply with design of other dvc commands
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-17 23:24:27 +03:00
Ruslan Kuprieiev
6f11f1f51f
test: move func tests to tests/func
...
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-16 20:19:22 +03:00
Ruslan Kuprieiev
2460dc255e
repo: scm_context: re-raise error directly
...
This way we preserve original traceback.
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-09 14:09:54 +03:00
Ruslan Kuprieiev
56b52cc9bf
checkout/remove: properly handle broken symlinks
...
Fixes #1856
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-09 14:09:54 +03:00
Ruslan Kuprieiev
54f7b43aac
dvc: version: detect release
...
See failed release at [1]. Previously we used to look at TRAVIS_TAG env
var, but now we will try to detect release by matching current tag with
base version.
[1] https://travis-ci.com/iterative/dvc/builds/107365596
Signed-off-by: Ruslan Kuprieiev <[email protected] >
2019-04-07 22:55:20 +03:00