* s3/gs: do not try to make directory on top level
* tests: unittest makedirs for s3 and gs
* remove setattr and use patch.object
* Update tests/unit/remote/test_remote.py
* remote: Optimize traverse/no_traverse behavior
- estimate remote file count by fetching a single parent cache dir, then
determine whether or not to use no_traverse method for checking
remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
one parent cache dir per thread)
* Use arbitrary prefix length for traverse remote cache queries
- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)
* Add unit tests for new traverse/no_traverse behavior
* Fix deepsource warning
* Fix review issues
* fix CI
* Fix review issues
* obsolete no_traverse config option
- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
RemoteBASE.CAN_TRAVERSE (True for all remotes except http)
* review issues - use 'object exists' instead of 'no_traverse'
* fix obsolete no_traverse test issues
* fix SSHMocked func tests
* pull: support pulling git-only imports
* pull: also add git imports from external dvc repo
* fetch: add logging
* pull: don't save link to state
When `checkout` or `pull` is done, it'll try to cleanup unused links.
So, when we add from erepo to cache, it creates a link, which DVC will try
to cleanup. Also, it'll provide unwanted checkout change details. So, skipping those
* tests: refactor/rewrite
* flake8: why it's complaining?
* tests: do not leak file descriptors
* tests: importing from bare repo
* external_repo: don't try too hard to set upstream repo especially if url is not a dvc repo.
* Update tests/func/test_import.py
* Add a comment regarding bare-repo clone
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* Restyled by black
Co-authored-by: elgehelge <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
Since our remotes have optional dependencies, it is really useful to be
able to see what is currently supported. Also helps debug issues with
packages such as brew/conda/etc.
* py38: make wheels
- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows
* use realpath to resolve symbolic links
* do not build pkgs for Windows and Linux on Python3.8
* reorder
* pyarrow: document regression fix
Co-authored-by: Casper da Costa-Luis <[email protected]>
* Initial design of using pre-commit for Git hooks
* Refactor to reuse existing code and provide merge capabilities
* Linter fixes
* More cleanup
* scm: make use_precommit_tool optional
* dvc: install our own hooks
* install: fix typo
* install: fix bugs in pre-commit integration
* install: don't forget 'always_run' for post-checkout
* install: no need to run pre-commit install if conf file doesn't exist
* install: fix bugs
* install: use python hooks
* add .pre-commit-hooks.yaml
Co-authored-by: Andrew Hare <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* checkout: implement --show-changes flag to show detailed checkout details
Show summary by default on checkout
checkout: show directory modified even when files are removed from inside it
checkout: refactor template logic
checkout: change output format
checkout: do not show summary if nothing's changed
checkout: do not show summary if show_changes is specified
checkout: show '/' at the end if it's a directory
rename variable name
test: cover stats for checkout
checkout: show relink as not modified
checkout: remove unwanted message during empty checkout
pull: show summary on checkout
import/get: change external_repo to catch CheckoutError
fix tests and check if path is url
checkout: support external deps/outputs
tests: checkout on external outputs
* fix tests
* checkout: make show changes default and introduce summary flag to show summary
* address suggestions
* exceptions: remove extra spacing on erorr
* Update dvc/command/checkout.py
Co-Authored-By: Jorge Orpinel <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* erepo: fix relative urls for remotes (#2756)
* erepo: add test case for relative path remotes (#2756)
* Remove unused argument
* erepo: move handling of relative remotes to a separate method (#2756)
* erepo: don't manually build paths on test (#2756)
* erepo: refactor handling of missing or relative upstream (#2756)
* erepo: move handling of source repo resource to `_fix_upstream` (#2756)
* erepo: improve relative remote tests (#2756)
- use `tmp_dir` fixture
- commit config file changes
* erepo: Remove comment (#2756)
* erepo: use scm_add to commit too, no need for separate scm.commit call (#2756)
* erepo: minor refactoring in `_fix_upstream` (#2756)
* erepo: add comment explaining that old relative paths are restored (#2756)
* gc: make it safer by only activating by use of certain flags
Previous plain `gc` is now not supported. The same behavior can be trigger by
`gc -w` or `gc --workspace`. When `--all-{tags, commits, branches}` are
specified, it will work as if `-w` is also specified (same as before).
* tests: adjust gc to use workspace on test_repro
* remote.http: support _upload() via HTTP POST
- uploaded files are sent as chunked encoding POST data
Fixes#3247
* remote.http: add support for HTTP basic and digest auth
- username, password, ask_password options work the same way as for
ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
remotes
- digest_auth takes precedence over basic_auth if both are enabled
* remote.http: auth_method() unit tests
* tests: Add functional tests for HTTP remotes
- HTTP remotes now tested locally using a SimpleHTTPServer instance that
allows reading/writing to a temp directory
* tests: fix http tests on py3.5, 3.6
* remote.http: add custom auth method
* s/custom_header/custom_auth_header/
* Apply suggestions from code review
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
* add: auto convert absolute_path => relative
- fixes#2955
* use path_isin
* catch-all absolute paths in repo
* unneeded leftover
* revert realtive imports for later
* output: move abs2relpath from Base to Local
* output:local:fix parse_path
* remove path_info
* test:local versus base abs2relpath
* test:fix remaining abs2relpath test
* output: local path relative to wdir
* output: local: modify path before inheritance
* output: local: minor inheritance fix
* test: local: split output test
* test: local: positively explicit
* made '_ignored' check if the given file is already ignored by the root .gitignore. Added corresponding functional test
* code formatting
* Changed method input, adjusted tests, formatted code
* typo
* tests: use git logger fix in conftest
See https://github.com/iterative/dvc/issues/3167
* scm: git: remove redundant comments
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Saugat Pachhai
Paweł Redzyński
Ruslan Kuprieiev
Peter Rowlands (변기호)
Guro Bokum
Max Risuhin
Bruno Deferrari
Ivan Shcheklein
Mr. Outis
Alexander Schepanovski
Aljoscha SteffensandRuslan Kuprieiev