* dvc: include cache urls in info output.
Show cache urls for external and remote cache in the info output
to assist in diagnosis of issues.
Fixes#4574.
* Remove unnecessary constant check for generator expression.
* Add remotes and caches in version command for better diagnosis.
* Fix flaky unit test to be runtime independent by using regex search.
* Use Cache.by_scheme() for enumerating supported caches.
* Add get_cloud_schemes routine for retrieval of configured remote schemes.
Add tests for local remote and remote:// scheme for info.
* Cleanup unused code.
* info: refactor remote parsing
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Outs desc will be used by viewer right away. Stage desc is for people to
start using now, but we won't be using it it viewer until DAG support is
added there.
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tree: use file_mode in upload() for trees that support modes
* experiments: preserve file_mode for files in the workspace
* skip permissions tests on windows
* LocalTree: separate chmod from protect
* fix rebase issue
* experiments: Allow branching from checkpoints with modified hyper params
* experiments: initial checkpoints tree functionality for `exp show`
* go back to flat/sortable view
* add test for resume w/modified params
* tests: move experiments fixtures into conftest
* add show checkpoints test
* test checkpoint table rendering
* remove dead code
* test show for checkpoint branches
This is a hidden flag (no docs needed) for a handy shortcut to pdb when
we need it. E.g. binaries built by pyinstaller are much easier to debug
with it.
* Job count expected in a core config
Besides defaulted 4 * core_count number of workers sourced in the
"tree", the configuration file based value expected and prioritised.
* Multiply the configured value by cpu cores count.
* Don't multiply by the core count twice.
* A couple more usages and a question note.
* Don't multiply a number of jobs to a cpu count
Number of jobs set in a config file directly used to spawn workers.
* Remove the obsolete note.
* Disallow 0 jobs set in a config
* `jobs` config-based value is optional
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* CPU count based default
* Update dvc/tree/base.py
* Happy path unit test
* Update tests/unit/remote/test_remote.py
* Fix the accidentally removed blank line
* Update dvc/tree/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* api: add support for simple wildcards
Related to #4816.
Signed-off-by: Ioana Grigoropol <[email protected]>
* api: make wildcard interpretation optional
Adds a new argument for the add command `glob` that is disabled by default and when enabled it passes
the input targets through glob filtering.
Related: #4816
Signed-off-by: Ioana Grigoropol <[email protected]>
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: refactor regular/checkpoint experiment separation
* fix scm_checkout pull bug
* experiments: return stages from executor.reproduce
* since executor has to move to threadpool execution anyways, we no
longer have to work around the pickle repro result limitation
* refactor new/resume behavior
* refactor checkpoints tests
* use -f to force checkpoints reset/restart
* checkpoints: use 'exp run' and 'exp res[ume]' commands
* stage: all outs can be missing for checkpoint stages
* regular outputs may not be written at the time a checkpoint is
generated, but subsequent checkouts of that stage should not fail due
to the missing outs
* commit stage on checkpoint generation otherwise non-persist cached outs
are lost
* checkpoints: wait for stage proc to end after killing via terminate()
* stage: fix issues with killing checkpoint stage
* refactor signaling (use threading.Event instead of Condition)
* fix windows bug (kill all child processes via taskkill)
* experiments: refactor executor collection
* tests: move checkpoint tests into separate module
* refactor experiment tests to use stage fixture
We wrap every data to one of the `Node` object. But, since we were
passing `self.data` instead of `self`, it was skipping the conversion
and instead was just trying to set data as-is, which would have make
our latter assumptions incorrect, as other codebases assume it to be
a `Node`.
I have added asserts in a few places. Also, I noticed that, on a few
test assumptions, when `resolving`, we were not converting values
back to it's original form (`CtxDict` -> `dict`, `CtxList` -> `list`
and `Value` -> _). So, I have added `node.value` property, as it is
a complement of the above fix.
This caused a ripple effect for the changes in the API as `foreach`
now resolved and changed the data to the original form,
which prevents us from tracking the data later in the "field".
So, `resolve`/`select` got a real `unwrap` option that should
reliably work for all kinds of `Node` objects.
Currently we are converting dir_info to/from lists all the time.
The reason is that dir_info is stored as list of dicts in *.dir files,
but that makes it hard to work with. In addition to that, we will likely
be changing .dir file format in the near future #829, so we need to
abstract away dir_info into something that we won't care how it will be
stored on disk.
Related #3256
Related #4847
* http: added check for HTTP status code 404
Fixes#4288
* http: add unit test and minor change
In tests_exists, changed to raise a HTTPError from dvc instead.
Fixes#4288
* http: removed fixtures in test for HTTPTree.exists
Fixes#4288
* test: add comments on why `raw` is required on the mock
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* command: add dvc push instruction after dvc repro
Adds a logger.info() statement to remind the user to run dvc push after
running dvc repro.
Fixes#2359
* command: change dvc push instruction for dvc repro
* command: Use backticks and else in dvc repro log
* add: warn on cache link failure instead of error
* add test case
* fix linted string
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
* metrics: include revision information when collecting metrics files
* plots: test warning message on no plot on revision
* plots/metrics: move to one package, introduce common tests
* tests: fix utils placement
In my tests `Git._ignore` is taking around ~65seconds for the whole test
suit(~50K invocations). The reason is that `gitpython` is launching a real
`git check-ignore` process, that takes time and resources.
Switching to `pygit2` or `dulwich` makes it drop to ~5seconds, saving us
almost a minute of the test run time and a lot of resources by not
launching any processes. I've chosen `dulwich` over `pygit2` here
because it is much more pythonic and as long as the performance is
on-par with `pygit2` it is better for us to use `dulwich`.
Arun MahapatraandRuslan Kuprieiev
Ruslan Kuprieiev
Paweł Redzyński
Tomás Farías SantanaandRuslan Kuprieiev
Peter Rowlands (변기호)
Ilya PekelnyandRuslan Kuprieiev
danielfleischer
Guro Bokum
Saugat Pachhai
Sami Jawhar
Ioana GrigoropolandRuslan Kuprieiev
LucaButeraandLuca Butera
Michal NowikowskiandRuslan Kuprieiev
Liew XunandSaugat Pachhai
simonlousky
Kevin Lloyd Bernal
João Vitor