Currently `dvc run -p angles` is broken when you use it with `params.yaml`
that looks like this:
```
angles:
- 2
- 3
- 100
- 3.2
- 36.2
- 3.6
```
because we are incorrectly using `flatten` instead of simply using some
xpath-like library to access params that are specified.
* remote: Optimize traverse/no_traverse behavior
- estimate remote file count by fetching a single parent cache dir, then
determine whether or not to use no_traverse method for checking
remainder of cache entries in `cache_exists`
- thread requests when traversing full remote file lists (by fetching
one parent cache dir per thread)
* Use arbitrary prefix length for traverse remote cache queries
- default to 3
- for remotes that only support per-directory query use 2 (gdrive, hdfs)
* Add unit tests for new traverse/no_traverse behavior
* Fix deepsource warning
* Fix review issues
* fix CI
* Fix review issues
* obsolete no_traverse config option
- remove RemoteBASE.DEFAULT_NO_TRAVERSE and replace it with
RemoteBASE.CAN_TRAVERSE (True for all remotes except http)
* review issues - use 'object exists' instead of 'no_traverse'
* fix obsolete no_traverse test issues
* fix SSHMocked func tests
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* fix formatting
* dep: param: fix schema
Signed-off-by: Ruslan Kuprieiev <[email protected]>
* run: check if --params is specified
* dvc: rework param handling
* params: fix exc formatting
* params: support basic nested configs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* pull: support pulling git-only imports
* pull: also add git imports from external dvc repo
* fetch: add logging
* pull: don't save link to state
When `checkout` or `pull` is done, it'll try to cleanup unused links.
So, when we add from erepo to cache, it creates a link, which DVC will try
to cleanup. Also, it'll provide unwanted checkout change details. So, skipping those
* tests: refactor/rewrite
* flake8: why it's complaining?
* tests: do not leak file descriptors
* dvc: add Repo._skip_graph_checks flag
Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.
The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.
* dvc: explain skip_graphs_checks flag in a comment
* test: that skip_graph_checks works
* repo: improve hack description and test
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: importing from bare repo
* external_repo: don't try too hard to set upstream repo especially if url is not a dvc repo.
* Update tests/func/test_import.py
* Add a comment regarding bare-repo clone
* Work in progress
* added file parsing and name validation + adjust schema
* Exceptions on bad input
* Support multiple parameters
* Support multi 's in
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
* Restyled by black
Co-authored-by: elgehelge <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
Since our remotes have optional dependencies, it is really useful to be
able to see what is currently supported. Also helps debug issues with
packages such as brew/conda/etc.
Unfortunately overriding `format` is not enough in order to use our own
custom formatting in logger. `Record` uses percent formatting that is
hardcoded inside, so we would also have to override it, which is just
way too much effort that will inevitably break as we discover other
methods that are not ready for non-% formatting. So we are just giving
up and sticking to default %-formatting for lazy log messages.
Fixes#3397
* py38: make wheels
- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows
* use realpath to resolve symbolic links
* do not build pkgs for Windows and Linux on Python3.8
* reorder
* pyarrow: document regression fix
Co-authored-by: Casper da Costa-Luis <[email protected]>
* Initial design of using pre-commit for Git hooks
* Refactor to reuse existing code and provide merge capabilities
* Linter fixes
* More cleanup
* scm: make use_precommit_tool optional
* dvc: install our own hooks
* install: fix typo
* install: fix bugs in pre-commit integration
* install: don't forget 'always_run' for post-checkout
* install: no need to run pre-commit install if conf file doesn't exist
* install: fix bugs
* install: use python hooks
* add .pre-commit-hooks.yaml
Co-authored-by: Andrew Hare <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* checkout: implement --show-changes flag to show detailed checkout details
Show summary by default on checkout
checkout: show directory modified even when files are removed from inside it
checkout: refactor template logic
checkout: change output format
checkout: do not show summary if nothing's changed
checkout: do not show summary if show_changes is specified
checkout: show '/' at the end if it's a directory
rename variable name
test: cover stats for checkout
checkout: show relink as not modified
checkout: remove unwanted message during empty checkout
pull: show summary on checkout
import/get: change external_repo to catch CheckoutError
fix tests and check if path is url
checkout: support external deps/outputs
tests: checkout on external outputs
* fix tests
* checkout: make show changes default and introduce summary flag to show summary
* address suggestions
* exceptions: remove extra spacing on erorr
* Update dvc/command/checkout.py
Co-Authored-By: Jorge Orpinel <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
Casper da Costa-Luis
Ruslan Kuprieiev
Max Hora
Peter Rowlands (변기호)
Helge Munk JacobsenandRuslan Kuprieiev
Saugat Pachhai
Alexander SchepanovskiandRuslan Kuprieiev
Ramón Valles
Jorge Orpinel
Ivan Shcheklein
Paweł Redzyński
Guro Bokum