This employs generating temporary signed public url and then streaming
from it. The bulk of the code might be reused for other remotes.
Url streaming does resume on connection error using "Range" header and
regenerates public url too.
If remote class doesn't provide .open() then this falls back to pulling
a file and opening it locally.
In this mode, dvc will set appropriate file/dir permissions to support
sharing cache between different users within the same group.
Fixes#2287
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* dvc: hooks only act in branches where dvc active
In a branch where DVC hasn't been initialized (as detected by
`git ls-files .dvc` not listing any files), don't invoke dvc in the git hooks.
This avoids trouble when switching between branches with and without dvc
initialized.
Fixes#2208
* dvc: simplify hooks per PR comment
Related to #1629
Speeds up HDFS tests significantly. E.g. ExternalHDFS test goes from 600
sec to 190sec. test_open_external[HDFS] from 160sec to 3sec.
Signed-off-by: Ruslan Kuprieiev <[email protected]>
GitPython creates submodules nodes during tree iterate without explicit
passing name attribute. This path replaces such submodules with new
instances with filled name attribute.
Fixes: #1898
* remote: drop not working http resume
Closes#2136
* remote: drop @transfer_context()
It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.
* remote: drop upload/download chunking
* state: use ignore file handler when accessing metadata
* fs: don't add dir size to overall size for dir
* ignore: use posix-like path on windows
* ignore: remove tree from dvc_walk arguments
* ignore: handle DvcIgnoreFilter creation outside of DVC repo
* ignore: throw on dvcignore file under output dir, but not dependency dir
* ignore: move ignore initialization to Repo init
* ignore: work in progress
* ignore: work in progress 2
* ignore: throw on .dvcignore upon collecting dir
* ignore: refactor
* ignore: loading upper level ignores in stages()
* Update dvc/repo/__init__.py
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
* dvcignore review refactor
Ruslan Kuprieiev
Mr. Outis
Alexander Schepanovski
Ruslan Kuprieiev
alex maslakoff
RahulR19
kurianbenoy
Gunnlaugur Thor Briem
pawel
Ivan Shcheklein
Jorge Orpinel
Georgy Dyuldin