Imagine the scenario of shared cache. To set it up, user would do:
```
dvc config cache.dir /path
dvc config cache.shared group
```
Now, when he runs any dvc operation that involves the Repo instance,
RemoteLOCAL will also be initialized and it will currently auto-create
the directory with a regular `os.mkdir`, which is not using group
permissions. To solve that, we can't simply use `self.makdirs` instead,
as it will cause same issues if you run any dvc command before
`dvc config cache.shared group`. So what we should do is create
directories on demand, instead of creating it in advance, same as we do
in other remotes.
The change itself is pretty trivial and involved removing that `mkdir`
line, adjusting tests to not rely on cache dir existence and also
adjusting `dvc version` to not rely on cache directory existance.
Fixes#2563
Config files like `~/.zshenv`, `~/.bashenv` etc, are loaded on each
shell invocation, which might overwrite the environment breaking the
commands we are trying to wrap. To fix that, we need to provide
appropriate flags to the shell we are using to avoid loading those
configs.
Fixes#2506
Unlike PyInstaller, pyenv doesn't leave backups of original env vars
when it modifies them. If we look into the shim, pyenv and pyenv-exec,
we can figure out that the PATH is modified like this:
PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH
where
PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
bin_path - might not start with $PYENV_ROOT as it runs realpath on
it, see pyenv source code.
plugin_bin - might contain more than 1 entry, which start with
$PYENV_ROOT, see pyenv source code.
So having this, we can make a rightful assumption about what parts of the PATH
we need to remove in order to get the original PATH.
Fixes#2506
This way dvc will actually launch the same script that it is running
from and not installed dvc module, that might not match.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
For the longest time we've had `repo.lock`, but were only using it from
CLI when running specific command, instead of simply properly taking it
on API operations. This patch takes care of that.
Leftover code was causing problems when working with symlinks, because
it tried to use `realpath` on symlinks to further use it to determine
parent directory.
Fixes#2476
This is an extension of our callback stages (no deps, so always
considered as changed) for stages with dependencies, so that these
stages could be used properly in the middle of the DAG. See attached
issue for more info.
Related to #2378
Current test is testing for signal handler restoration in an obscure
way. As a result, the test wasn't actually testing anything, because
KeyboardInterrupt is raised by default_int_handler, and so raising it
in a mock won't be caught by signal handlers, because they are dealing
with real singnals.
Fix#2406
Signed-off-by: Ruslan Kuprieiev <[email protected]>
In this mode, dvc will set appropriate file/dir permissions to support
sharing cache between different users within the same group.
Fixes#2287
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Related to #1629
Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Fixes#2155
No longer need to explicitly ignore paramiko warnings.
Big thanks to @cowlinator for the heads-up!
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Fixes#2039
Required by #2012 to be able to fetch data through the API without
interuption from updater.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
We do a similar thing on save, where we set cache checksum from already
computated ones for actual data. This patch does the same thing but in
revers: on checkout we know cache checksums, so we can set data
checksums from it without having to re-calculate it.
Fixes#1991
Signed-off-by: Ruslan Kuprieiev <[email protected]>
We've contributed these hooks a long time ago and they got finally
released, so we can remove our copies.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
This version added support for oss as a remote, which has caused some
issues when building the binary package. To fix those, we need two
pyinstaller hools:
1) hook-Crypto.py
This one has been added to PyInstaller's upstream, but wasn't
released yet, so we are using a copy of it for now.
2) hook-aliyunsdkcore.py
This one is a new one. Sent it to PyInstaller's team in a PR [1].
[1] https://github.com/pyinstaller/pyinstaller/pull/4228
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Currently `dvc version` outside of dvc repo raises "not dvc repo" error,
even though it is not really using repo in any way.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
JSONDecodeError is only available in never python 3 versions and still
inherits from ValueError, so we could just use it.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
Ruslan Kuprieiev