* parametrization: prevent dumping parametrized stages
This is a bit hackish, but does the job done.
It feels like we need some unified way to be able to load
and dump, but not sure how. Till then, this dataclass
will be helpful in communicating between those.
* s/Meta/RawData
* Update dvc/stage/loader.py
Some of the tests are run by default, other requires
`--enable-{remote}` to run. The remotes that are run by default
also have a `--disable-{remote}` flag. All of the remotes can be
enabled using `--all` flag.
To use this functionality in the fixture,
`test_config` fixture has to be used, and requires to call
`test_config.requires(remote)`.
In an individual test, you can simply use `pytest.mark.remote`
to make them skip or run.
The schema is changed of `vars` to accomodate the both use of
`imports` and setting variables locally.
Now:
```
vars:
- params.yaml
- foo: foo
bar: bar
```
Also, same distinction occurs for the stages.
* dvc: include cache urls in info output.
Show cache urls for external and remote cache in the info output
to assist in diagnosis of issues.
Fixes#4574.
* Remove unnecessary constant check for generator expression.
* Add remotes and caches in version command for better diagnosis.
* Fix flaky unit test to be runtime independent by using regex search.
* Use Cache.by_scheme() for enumerating supported caches.
* Add get_cloud_schemes routine for retrieval of configured remote schemes.
Add tests for local remote and remote:// scheme for info.
* Cleanup unused code.
* info: refactor remote parsing
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Outs desc will be used by viewer right away. Stage desc is for people to
start using now, but we won't be using it it viewer until DAG support is
added there.
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tree: use file_mode in upload() for trees that support modes
* experiments: preserve file_mode for files in the workspace
* skip permissions tests on windows
* LocalTree: separate chmod from protect
* fix rebase issue
* experiments: Allow branching from checkpoints with modified hyper params
* experiments: initial checkpoints tree functionality for `exp show`
* go back to flat/sortable view
* add test for resume w/modified params
* tests: move experiments fixtures into conftest
* add show checkpoints test
* test checkpoint table rendering
* remove dead code
* test show for checkpoint branches
* api: add support for simple wildcards
Related to #4816.
Signed-off-by: Ioana Grigoropol <[email protected]>
* api: make wildcard interpretation optional
Adds a new argument for the add command `glob` that is disabled by default and when enabled it passes
the input targets through glob filtering.
Related: #4816
Signed-off-by: Ioana Grigoropol <[email protected]>
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: refactor regular/checkpoint experiment separation
* fix scm_checkout pull bug
* experiments: return stages from executor.reproduce
* since executor has to move to threadpool execution anyways, we no
longer have to work around the pickle repro result limitation
* refactor new/resume behavior
* refactor checkpoints tests
* use -f to force checkpoints reset/restart
* checkpoints: use 'exp run' and 'exp res[ume]' commands
* stage: all outs can be missing for checkpoint stages
* regular outputs may not be written at the time a checkpoint is
generated, but subsequent checkouts of that stage should not fail due
to the missing outs
* commit stage on checkpoint generation otherwise non-persist cached outs
are lost
* checkpoints: wait for stage proc to end after killing via terminate()
* stage: fix issues with killing checkpoint stage
* refactor signaling (use threading.Event instead of Condition)
* fix windows bug (kill all child processes via taskkill)
* experiments: refactor executor collection
* tests: move checkpoint tests into separate module
* refactor experiment tests to use stage fixture
We wrap every data to one of the `Node` object. But, since we were
passing `self.data` instead of `self`, it was skipping the conversion
and instead was just trying to set data as-is, which would have make
our latter assumptions incorrect, as other codebases assume it to be
a `Node`.
I have added asserts in a few places. Also, I noticed that, on a few
test assumptions, when `resolving`, we were not converting values
back to it's original form (`CtxDict` -> `dict`, `CtxList` -> `list`
and `Value` -> _). So, I have added `node.value` property, as it is
a complement of the above fix.
This caused a ripple effect for the changes in the API as `foreach`
now resolved and changed the data to the original form,
which prevents us from tracking the data later in the "field".
So, `resolve`/`select` got a real `unwrap` option that should
reliably work for all kinds of `Node` objects.
Currently we are converting dir_info to/from lists all the time.
The reason is that dir_info is stored as list of dicts in *.dir files,
but that makes it hard to work with. In addition to that, we will likely
be changing .dir file format in the near future #829, so we need to
abstract away dir_info into something that we won't care how it will be
stored on disk.
Related #3256
Related #4847
* add: warn on cache link failure instead of error
* add test case
* fix linted string
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
* metrics: include revision information when collecting metrics files
* plots: test warning message on no plot on revision
* plots/metrics: move to one package, introduce common tests
* tests: fix utils placement
* Allow setting of list parameteres
Check if dict is Mapping or list before using .get()
* Added test for dvc exp run with list paramater
* Add more tests for string parameter modification
Saugat Pachhai
Sami Jawhar
Arun MahapatraandRuslan Kuprieiev
Ruslan Kuprieiev
Paweł Redzyński
Tomás Farías SantanaandRuslan Kuprieiev
Peter Rowlands (변기호)
Ioana GrigoropolandRuslan Kuprieiev
Michal NowikowskiandRuslan Kuprieiev
Kevin Lloyd Bernal
Juliano Efson Sales
Tomás Farías Santana