astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.
Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).
Example errors that we were getting because of this:
```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
Pre-requisite for removing fs.path_info and switching to regular strings
as paths in filesystems.
With new filesystems we no longer need the zoo of different dependency
and output types, as most of them are now just handled by the underlying
filesystem. Though deps/outs are still due for a refactor in other
aspects.
* Decoupling dvcignore and fs
[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check
* Solve all of the arugments error.
* add dvcignore to dvc list
* Solve some test_ignore cases
* add dvcignore to checkout.
1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs
* Solve two tests
* Get rid of ignore_filter
* Add ignore examine to download
* Solve a test issue
* Solve dvcignore and subrepo problems
* Solve sub repo error
* Revert some changes
* pass all tests
* Remove debug codes
* Some final exams
* dvcignore trie tree only update when needed
* reuse of the dir result in path walk.
* Some change after code review.
* Recover something from false `--force` push
* add abspath trans to dvcginore
* pass pylint
* pylint related
* Update dvc/fs/repo.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Changes related to reviews.
* dvcignore call api change
from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)
* Get rid of PathInfo in test_ignore.py
* fs.walk follow the arguments of os.walk
* Dvcignore as parameters
* pass dvcignore as arguments to checkouts
* Decoupling state and repo
* Update dvc/dvcfile.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Update dvc/objects/stage.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Solve some small problems
* Add tests to verify that dependency now not ignore
fix#5256
* Update dvc/ignore.py
* Update dvc/fs/git.py
* Update dvc/repo/__init__.py
* Some small improvements
* pass black
* Refactor is_ignored
* Update dvc/output/base.py
* repofs: use dvcignore in exists/isdir/isfile
Per https://github.com/iterative/dvc/pull/5812/files#r630604524
Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
We used to allow changing default output and error stream
but that was problematic for the testing. Now it just uses
sys, so capsys should work for the testing.
The proper way to change sys.stdout and sys.stderr is
by using 'redirect_stdout' now.
* try to close dvc.state on exit
I have a feeling that it is causing the issues with
shutil.copytree and some PermissionError during 'init',
as we are not closing it between multiple 'main()' calls.
But I am not quite sure, and this seems like a good thing to
do anyway.
* fixup
* close dvc on exit
* close cache on tests and on init
* azure: use a custom event loop for authentication
* add test_temp_event_loop
* wrap open() too!
* ignore aiohttp warnings for normal level use-cases
* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
we just need to mark them with needs_internet marker.
Unfortunately, it's not possible to do that automatically
with the fixtures as fixtures are per-session and we need
something on more higher level (which the marker is not passed
through in the pytest).
This is because, the repo root is used from a realpath, and the stage.wdir is abspath.
We might need to use same kinds of path, but this patch seems to be the easiest/fastest
way to fix the flaky tests (if all the tests agree).
* dvcfile: add check for gitignored when validating filenames
* update tests
* update exception using in tests
* review fixes
- clarify method name
- use FileIsGitIgnored
I tried increasing max_run to very high number: 1000. Even in those
cases, it failed 947 times. So I don't really see a reason to mark
it as flaky. Anyway, we are treating it as xfail by keeping it failing
in the CI, so this PR does not change anything for us. :)
I found that it's harder to style a header with the list using index,
they are to be applied by header name. Otherwise, we might end up using
a different header for coloring, or worse throw some error because the
header name and style don't match.
Row does have it's property of index, so it makes sense there.
* apply: allow applying exp with multiple possible branches
* default to new checkpoint branch after applying ambiguous commit
* output shas in addition to exp names by default
* update show tests
* git: allow force checkout in detach_head context
* exp run: error when trying to resume from a non-head checkpoint without
modifications
* add test for apply/resume from intermediate checkpoint
* list some possible heads when multiple matches
* use new table data structure for diff and show
* fix params diff showing changes
* metrics show: use default precision
* skip precision format for exp diff params
* exp diff: separate tables in --quiet mode
* change precision to signify significants digits
changed f specifier to g, so that we show significant digits
instead of fractional digits. This does changes how precision
worked for metrics/params diff/show.
* stage,commit: add support for only committing data stages from
experiments
* exp run: `dvc commit` data deps when stashing exp changes
* add test for modified DVC-tracked data when generating experiment
* test with and without params change
* handle run without target
* objects: merge transfer() with stage()
* objects: drop streamfile
* objects: preserve the signature of get_file_hash
* objects: override hash_jobs if jobs= passed
* objects: support no progress bar directive
* tests: spy on _get_file_hash
* objects: update=... directive
* update: only transfer missing caches with --to-remote
* tests: remove local_remote from the primary test (due to it moves)
* objects: add move=True/False option
* objects: implement _get_file_obj on top of get_file_hash
* objects: use threading on save()
* objects: pass jobs= on to-cache as well
* objects: use in place tqdm
* objects: check for FileExistsError
* Update dvc/objects/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Due to https://about.codecov.io/security-update/ , there is a potential that we've leaked these. This PR takes a more granular approach when setting secret env vars so that this doesn't happen in the future.
We'll change these secrets after this PR is merged.
* exp: add util method for getting baseline HEAD during run
* exp: use baseline for dvc commands which compare to HEAD during exp run
* add test for new util method
* update plots diff test mock
* Add support for kerberos token based auth
* Added kerb_ticket as attribute to HDFSFileSystem
* Lint fix for config schema
* Updated black style changes
* Removed staticmethod decorator from hdfs
This part of logic used to be only used by external repo for cloned
repos, but since we now handle local repos without cloning, we don't
always pass scm instance. This PR automatically creates scm instance
for such cases.
Fixes#5590
* params: read from ParamsDependency if possible
* params: introduce include_untracked flag
* params: include untracked upon targets
* experiments: add option to show tracked params
* fixing test
* params: rename tracked to deps
* params: roll back to default params.yaml behavior
* experiments: add option to show only used params on diff
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/experiments.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/params.py
Co-authored-by: Jorge Orpinel <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* exp show: Allow for wildcard patterns in include and exclude params/metrics
Fixes#5642
* Add test case for nested metrics/params
* match full string in _filter_name
* pre-commit
* Update dvc/command/experiments.py
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Added SSL verification option
* Changed param name to avoid conflicts
* verify_ssl now accepts strings
* Added simple unit tests
* Changed parameter name fron verify_ssl to ssl_verify so it's consistent with HTTP remotes
Forget to change 1 test
* add processing on ssl_verify param
typo
* revert back to bool param
* Update dvc/config_schema.py
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
This is achieved by using `lock_repo`
context manager that can be used in a recursive depth
that only resets stages when it has left any lock_repo
blocks.
Also added some ui related tests for the status which I found
missing
Before, we used to set it during initialization. But this
made it hard to test using capsys as capsys changes the
reference to sys.stdout and sys.stderr with the mocked one
during the test. But since we are using dvc.ui globally and
setting it to a real sys.stdout and sys.stderr before capsys
comes into effect, testing was not possible with it.
Now, by using property hack, we will be using the values of
sys.stdout and sys.stderr during the time of write, so the mocking
would work and would be easier to test.
* dvc: fix breakage with rich==10.0.0
Change in the arguments of __rich_measure__.
Fixes#5722
* dvc: Fix linting breakage from rich==10.0.0
Fixes: #5733
* dvc: set lower bound for rich dependency to 10.0.0
Fixes#5722
lxml wheels are built with an old SDK, which breaks osxpkg notarization:
```
* error for path "dvc-2.0.7.pkg/dvc-2.0.7.pkg Contents/Payload/usr/local/lib/dvc/lxml/etree.cpython-37m-darwin.so": The binary uses an SDK older than the 10.9 SDK.
```
This commit fixes 3 things:
1. The path printed on the terminal is properly is quoted. This was
problematic when the output is piped to `xdg-open` if the path contained
spaces or quotes or special characters.
2. The path is resolved. This is not an issue at all, but makes the
output nicer.
3. The path with which the browser is opened with uses relative path.
This is done to solve issue in WSL, that the absolute path is not
accessible outside the WSL for host programs as the linux filesystem
is available on `/wsl$` path.
* diff: clear calculated renamed files from inverted hashmap
We were trying to reuse the first encounter of file from inverted hashmap again and again, which was throwing ValueError
* refactor
* remove ordering
* add tests
* build: add pyinstaller hook for googleapiclient
The Google API Discovery Service JSON files from
googleapiclient/discovery_cache/documents were being
excluded due to a bug and, particularly, drive.v2.json
is essential for our gdrive backend. Now we're explicitly
asking PyInstaller to embed these files on the generated packages.
Fixes#5618
* Suppress linter error for unrecognized imports
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* Apply black style after manual code review
Co-authored-by: Peter Rowlands (변기호) <[email protected]>
* import-url: support importing empty directories on object storage services
Empty directories are basically files with a trailing slash at the end of their names,
with empty contents on object service providers (such as azure, google storage). This
patch implements the `is_empty()` method for the `FSSpecWrapper` to determining whether
a path is empty (directory or a file) and also tweaks `.isdir()` in order to see empty
directories.
The base class is only used by azure and gcsfs for now, but if we go on the route of
migrating normal providers (like proper filesystems, not just object storages), we might
need to do these tweaks on a subclass (like `ObjectStorageFSSpecWrapper` etc), but I
don't think it is neccesary as of now.
The `is_empty` method is already there, and also used in the importing part to
warn users that they are importing something empty (https://github.com/iterative/dvc/blob/e385e59a098cd661ea1ea4257c9a5735c5c32bc6/dvc/output/base.py#L252-L254)
so this is not a new method. Just for the reference, as far as I can locate there
is no variant of this method in the fsspec.
* tests: revert custom mkdir for cloud storages
* tests: allign assingments
* add some comments
* exp run: gracefully handle duplicate workspace runs
* exp run: fix issue where errors could be logged twice
* move ref conflict check into its own function
* dvc: get rid of DirInfo
DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.
As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.
* objects: move Tree into tree.py
The `GitPythonBackend.checkout()` method already implicitly supported
this by forwarding `**kwargs` to the backend. `pygit2` didn't support
this yet.
This patch fixes this inconsistency.
* pygit2: disable set_ref with reflog message
* git: implement is_dirty in dulwich
* pygit2: implement _stash_apply
* pygit: update to 1.5.0
* enable set_ref(message=...) for pygit2
* git: replace backend reflog_delete in favor of _stash_drop
* pygit: support stash_push with custom refs
* update tests for backend stash_drop behavior
* pygit: raise MergeConflictError on failed stash apply
* pygit: reload index from disk before stash apply
* update stash_apply checkout strategy
* mark flaky test
* gdrive: sync with pydrive's save backend changes
* bump pydrive version
* better temporary credentials managements for service accounts
* proper temp var management
* repro: skip DVC deps in SCM track stage
(DVC deps will be tracked when their own stage is repro'd)
* scm: use git status when reminding to git-track files
* sort names in reminder instead of random set order
* update unit test
* live integration: get rid of dvc from live
* live: remove from api, add test for html generation during run
* stage: run: convert monitors to class context managers
* run: make monitors run in single thread
* run: move monitor logic out of run
delete is a direct alias to rm(), though apparently it is kind of
broken on the latest release of gcsfs. This doesn't change any behavior
but only uses rm() directly instead of the given alias.
Switch from `md5` to `etag` as the checksum parameter. Due to `etag`
is not preserved while doing server side copies, this patch drops support
for external gs outputs.
* state: only load when needed
* state: don't use separate load/dump
* checkout: group link removal logic together
* state: use diskcache.Cache
* staging: use pbar and save to state
* state: don't pass the whole repo to __init__
* staging: explicitly pass state object
* fs: get rid of odb property
* state: don't close the connection to cache
* repo: close: close state
* stage:list: update output and help text
* Replaced 'Produces' with 'Outputs'
* Replaced 'Stage depends on' with just 'Depends on'
* Changes help text of --names-only and --fail
* Update dvc/command/stage.py
Co-authored-by: Dave Berenbaum <[email protected]>
* s/Generates/Reports
Co-authored-by: Dave Berenbaum <[email protected]>
* git: return backend-agnostic object for resolve_commit
* use int timestamps instead of datetime
* make branch_revs Git method instead of backend
* pygit: implement resolve_commit
* dulwich: implement add with --update
* add git backend test for add()
* fix list comprehension in dulwich status()
* allow adding single string path instead of list
* handle windows paths in add(update=...)
When trying to publish to pypi, we get:
```
Invalid value for requires_dist. Error: Can't have direct dependency:
"gdrivefs @ git+https://github.com/intake/gdrivefs.git ; extra ==
'tests'"
```
We used to think this way, but state is not really an fs concept
(fsspec's dircache is though), as it is a cache of hashes of objects
that we are staging, and the hash doesn't depend on fs, as it is usually
computed by us by reading from fs.
Pre-requisite for diskcache-based state.
This makes all cache objects go through the same logic, so that all of
the appropriate rules are applied to them (e.g. proper permissions).
Pre-requisite for #5337
* Make minor change that fixes things for some reason
While working on vscode-dvc, both mattseddon and I found that this change allows
us to run `dvc exp run`, and we cannot otherwise.
I don't exactly know what this change does, and I'm hoping someone more
knowledgable does- after which we can give this PR a better name.
* test relpaths in git backend funcs which take path lists
* gitpython: handle relpaths in reset and checkout_index
* use / for git relpaths on windows
Co-authored-by: Peter Rowlands <[email protected]>
Checkout is an operation that builds object from cache(aka object
database) in a particular tree(aka filesystem). Thus it shouldn't be
part of the object database itself nor of the filesystem.
Pre-requisite for #5337
* s3: faster directory hashing
* Introduce Tree.info() API
* tests for s3
* Implement support for azure
* New .ls() API
* remove _iter_hashes from other trees
* assert recursive
* yield strings instead of file_infos
* Add GSTree.ls()
* Add DETAIL_FIELDS to the GSTree
* WebDAVTree.ls()
* proper webdav root directory initalization
* disable arguments-differ
* webhdfs: don't use self.path_info
* azure: don't use self.path_info
* webdav: don't use self.path_info
* oss: don't use self.path_info
* http: don't use self.path_info
* gdrive: don't use self.path_info
Per https://github.com/iterative/dvc.org/pull/2064
Not changing the internal terminology because we stick to Repo in a lot
of places there and adjusting all of it would be wasteful. Maybe in the
future.
* exp show: add -n/--num to display the last N commits from HEAD
* exp run: fix empty commit with no changes
* add test for exp show with multiple commits
* allow -n greater than num commits in repo
* review fix
The `--glob` flag is parsed from the command line, but in the
`CmdDataPull` class in the invocation the parameter (`self.args.glob`)
is not passed to the `pull` method implementation. Hence, always the
default value (False) is used. This commit fixes this issue and sets the
`glob` parameter properly.
* don't allow loading stages without `cmd`
This was loosened when introducing parametrization.
But, `foreach`..`do` should be considered first
before stage's regular structure. And, cmd is made
required (as it should have been)
Related: #5371, #5370, #5312
* fix schema
* add: implement saving straight to cache
* Adjust CLI error cases to handle -o
* make tests pass on windows
* support explicit --jobs
* simplify the implementation a bit
* polish the tests
* tree: Implement upload_fobj protocol to azure
* tree: Implement upload_fobj protocol to google storage
* tree: Implement upload_fobj protocol to google drive
* tree: Implement upload_fobj protocol to hdfs
* tree: Implement upload_fobj protocol to oss (alibaba cloud)
* tree: Implement upload_fobj protocol to webhdfs (also some cleanup)
* tree: Implement upload_fobj protocol to http/https
* Add _generate_download_url() to the HTTP (for tests)
* tree: Implement a temporary upload_fobj protocol to webdav/webdavs
* Unify progress bars
* Lazily evaluate the file size, for http uploads
* Remove copy_fobj protocol and _transfer_file_as_whole
* show progress bar on cache transfer
* Simplify s3 upload
* Remove CHUNK_SIZE (except for some special cases)
* remove concurrency restrictions
* Update dvc/tree/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
When I opened it, I got confused for a second and had to double-check if it was really empty.
The empty file gives a false assurance. We do need to add one, which I'll add later, but for now,
we better remove it.
* dvcignore: Fix incorrect ignored output computation
* dvcignore: Fixed unignored patterns
Unignored patterns were shown in the dvc status output and caused Error.
Also made a bit of a refactoring in `matches` and `ignore`.
* dvcignore: Fix functional tests
Not sure if I follow, but tests assume that unignored files should trigger `check-ignore`. What was the intention?
* dvcignore: Fix stage.wdir being MagicMock
* Use `is_ignored` instead of `check_ignore`
1. use is_ignored to judge DVC ignore status.
2. use check_ignore to show ignore patterns.
* Solve tests fails
* Add a new tests
* Return to the old check_ignore logic.
* Pass the pylint
* Simplify ignore tests
Co-authored-by: karajan1001 <[email protected]>
This happens because KeyError quotes the error message.
We'd still like to keep StageNotFound being a KeyError
semantics, but workaround the __str__ implementation.
Mypy was having some problem with the previous `__str__`
override.
* Implement export-to-remote functionality
* Redistrubition of functionality from tree to remote
* Pass chunk size into the open_url
* Move functionality away from remote to cache
* Basic progress bars for both transfer types
* Proper job calculation
* Redesign .open() usage in order to work with different trees
* Align the signatures of other open() functions to accept **kwargs
* Implement chunked uploading to the ssh
* Use local cache to compute directory hash (md5)
* S3: multipart upload test
* Wrap local cache's hash_to_path with PathInfo
* Use .isdir() instead of a try/except statement for calculating the from_infos
* New CLI layout (add/import-url)
* Adjust CLI tests to accept new arguments
* tests: import_url/add func/unit tests
* Introduce cache.transfer()
* Invalid CLI combination tests
* Store dir_info key in the transfer_directory
* LocalTree.upload_fobj
* Tests for utils
* Lazy imports on some places
* --to-remote migration (and repo._transfer)
* Use proper temporary location, more efficient pbar calc
* Directory transfer logic rewrite
* Use shutil.copyfileobj()
* Rebase...
* Better tests
* Inline transfer operation to .add() / .transfer()
* Open in the 'rb' mode for the tests
* More tests, better invalid option handling at repo level
* .read() certain amount of bytes to validate the content
* Better invalid argument handling
* Better progress bars by using .getsize() API
* For fallback upload, use the from_info.name instead of temporary naem
* introduce --jobs for add/import-url
* cli: remove: fix error message for the case when file is not under control (#4497)
* cli: remove: fix error message for the case when file is not under control (#4497)
fixes after code review
* cli: remove: fix error message for the case when file is not under control (#4497)
2nd round of fixes after code review
* cli: remove: fix error message for the case when file is not under control (#4497)
3rd round of fixes after code review
* pygit2: implement get/set/remove refs
* pygit: implement iter_refs
* pygit: implement resolve_rev
* pygit2: implement working tree checkout
* pygit2: implement reset, explicitly read/write index when used
* pygit2: checkout does not need to reset index
* pygit2: handle "-" branch shorthand in checkout
* tests: install post-checkout hook should be run via gitpython
* exp apply: use stricter exception handling for merge conflict
* pygit: fix index read/write usage
* gitpython: handle untracked file conflicts in stash apply
* fix flaky/race index issue
* tests: upgrade azurite and enable loose mode
Azure tests started failing recently. They still work with real azure, so this is on azurite side.
* tests: use cloud instead of workspace
also gdrive doesn't support it yet :(
* dvc repro: fail on checkpoint stages
* exp res: check for moved HEAD/baseline before resuming
* exp run/res: always consider last checkpoint as applied for workspace runs
Looking at the log is confusing, as the version is stuck in `1.11.0` whereas the recent one is `1.11.10`.
As the updater is also dependent on that version, it prints an ugly update message while running too.
https://github.com/iterative/dvc/issues/5023#issuecomment-758450519
* scm: implement gitignore; use `pygit2` backend
* Add is_ignored implementation on `pygit2` and `gitpython` backends.
* Also implement `._reset()` on `repo.scm` which is most of the changes.
* fix tests
* use _backend_func
* minor fixups
* S3Tree._list_paths now recognizes the max_items limit
* Use modern approach + add a test
* refactor tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* lint
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Moved exception handling imports wherever they occur
* Move dvc.tree and dvc.external_repo imports to it's
nearest use, as it is expensive for `--version`.
* Move analytics import to when it's used
* Move format_link to place where error occurs
A great benefit to this is that, if the dvc.yaml file
is never interpolated, the parser is never built, which
reduces the gap even further between 1.0 and 2.0 files
* Fix partial file checkouts in the same directory
* unittest => pytest
* Try to rationalize changed() to take filter_info
* Use path.read_text() to verify the directory structure
* Try to use specialized hash, fall back to general one if doesn't exist
* Cover subdirectories in hash specialization
* coverage: add a test for empty subdirectories
* Move out the dir_info filtering block
* hash_origin => path_info
* cleanup
* Move hashing functionality after the existence checks
* Simplify it a bit
* Update dvc/cache/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Apply suggestions
* simplify tests
* change filenames
* Some visual changes for consistency
Co-authored-by: Ruslan Kuprieiev <[email protected]>
I had to keep providing default value before when profiling
simple functions as an API. I added a default now, and
also made `debugtools` customizable so that you could
edit the code to, for example, have `pdb` enabled by default.
* experiments: use relpaths when tracking changed files
* experiments: suppress SCM hints in tempdir executors
* experiments: move multiprocessing error logging into child
* exp run: clarify help text for --temp
* Add pyinstrument: a statistical profiler
It is easier than cprofile, in that it can show stacktrace
of where things are slow.
Also refactored/moved debugging related utils to a separate
module
* use ipdb first, then fallback to pdb
* use ipdb first, then fallback to pdb
* plots: allow dir plots
* first scenario
* separate dvclive tests
* rename index to step
* another test
* refactor
* its ok to have step in data
* sort plots
* introduce --logs
* add logs to run commands
* variable name refactoring
* fix logs option
* fixup
* cmd: run: fix unit tests
* plots: modify: fix tests
* improve dir handling
* fixup
* dvclive: logs summary
* api: introduce logs summarization method
* handle no repo case
* fix granular dir plots
* api: dvclive: simplify
* dvclive: roll back logs flag
* cmd: introduce logs
* api: adjust to logs
* test: dvclive fix
* rename logs to dvclive
* live: command: add uninitialized
* outs: dvclive type
* dvclive: support providing config via evn variables
* dvclive: treat as metrics
* dvclive integration: add support for summary on/off
* dvc integration: add temporary test for configuration export
* remove requirements
* cmd: introduce logs
* rename logs to dvclive
* cmd: introduce logs
* remove logs leftowvers
* dvclive: env: adjust to chekpoints env variables
* command: live: remove unnecessary help
* fix linter suggestions
* type hinting
* review refactor
* replace dvclive with live
* stage: loading: move live loading to separate method
* stage: serialize: allow only single live output
* live: add show/diff
* stage: serialize: roll back single live enforcing
* fixup
* rebase fixups
* output: live: force single instance
* live: command: add unit tests
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/__init__.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/stage/serialize.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* review v2
* live: fix summary tracking by experiments
* naming fixups
* api: live: add tests
* tests: api: separate live tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* cli: config: show merged config with --show-origin
Builds on #5126 and #5184 by showing the full merged config when
listing or getting config options.
References #5126
References #5184
References iterative/dvc.org#2028Fixes#5119
* Add additional tests.
* Add test for merged config.
* config: rename methods
Co-authored-by: OLOLO ALALA <[email protected]>
* cli: config: add support for --show-origin
Support the --show-origin option for config, which, similar to git,
prefixes each config option with the source file it originated from.
Fixes#5119
* Reformat with black.
* Simplify logic for local config files.
* Add test for --show-config output.
* Update tests/func/test_config.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* stage: support explicit allow_missing in checkout
* exp: explicitly clear EXEC_BRANCH and checkout with allow_missing on
starting experiment run
* git: wrap stash apply errors with SCMError
* git: add `git status` to backends
* exp: reset staged git changes before workspace exp runs to avoid merge
conflicts
* test untracked (and staged) pipeline file and deps for both workspace
and tmpdir runs
* experiments: explicitly track pipeline files and uncached deps for exp
commmits
* fix exp typing errors
* dvc/tree/local: fix it for no repo/cache case
fixes#5069
* Update dvc/output/base.py
* Update dvc/output/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* exp apply: show passed exp name in CLI output instead of resolved git sha
* exp res: warn that --name will be ignored when resuming without
modifications
* exp gc: don't re-use gc parameter check message
* exp: update help strings
* exp: warn if exp dir contains untracked files after tmpdir execution
* #4578 added is_user_executable to dvc file
* Addressed PR feedback and introduced test_add_executable
Also fixed all tests in test_add.py
* Added a test to checkout executable file
* Pop isexec in dvc/output/__init__.py:loadd_from
* Implement test that file is executable after checkout
* Address PR feedback
* Address PR feedback
* Changed BaseTree.isexec to use path_info as an argument
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* Update tests/func/test_add.py
* fix formatting
* add: preserve exec
* checkout: fix setting exec
* tests: refactor checkout exec test
* tests: fix windows cases
* tests: add: fix exec test on windows
* Update dvc/tree/local.py
* Update dvc/output/base.py
* Update dvc/output/base.py
* Update dvc/cache/base.py
* Update tests/func/test_add.py
* tests: add: skip whole exec test on windows
* save isexec to lockfiles
* fix formatting
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: move executor into a module
* experiments: separate executor into base/local
* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default
* git: restore HEAD using reset not checkout in detached_head()
* experiments: support calling BaseExecutor.reproduce directly for
workspace runs
* experiments: run single experiments in local workspace by default
* update experiments tests for workspace changes
* experiments: update checkpoints for workspace runs
* update checkpoints tests
* update test_show
* add force param for apply
* exp apply: add --force option
* exp run/res: add --temp option to force tempdir execution for single
experiments
* force apply in non-apply specific tests
* Cleanup log messages when loading stages
1. Simplified callback stage debug message
2. Reduced no lock entry message to trace
If lockfile does not have some entry, it will simply complain
that lockfile is not uptodate once.
If there is no lockfile or empty lockfile, it will complain that
lockfile not found.
3. Annoying parametrization related wdir-interpolated debug message
removed.
* Update dvc/stage/__init__.py
* black
* Skip paramdependency for vars
Previously, `configs` which was a list of PathInfo was being used
to check if the `vars` should be skipped or not.
But, the file is stored as a string that made that check redundant
which might have caused #5117 as they were being stored twice,
once as a expanded paramdependency and next as a flattened dict
from vars, which later during `flatten`, it might have panicked
from that duplication.
* Use defaultdict
* change add_vars to collect_vars
Currently, we iterate over targets manually in CLI, which results in a
complicated code and a DAG collection/check overhead. With this PR we
collect the DAG once, figure out all the needed targets within it and
just reproduce them in one pass.
* fix making too many ssh connections when pushing run cache (#4732)
* reformat
* generator should exit on GeneratorExit exception
* Update dvc/tree/ssh/__init__.py
* pool: close on Exception and not BaseException
Co-authored-by: Tomasz Łakota <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
minor refactors, code comments
Rename interim to definitions
provide location of entry during init for definition
small fixups to set_temporarily
Refactor loop to use dict comprehension
Provide same interface for definitions
Make it work for composite lists
Check for sequence rather than a list
refactor tests
Fixes#4955Fixes#4957Fixes#4963
Currently we don't handle bad name well:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: unexpected error - not enough values to unpack (expected 2, got 1)
```
With this PR, we check name to conform to either `remote.name.option` or
`section.option`, so we'll give a more friendly:
```
(dvc-3.8.3) ➜ dvc git:(config-bug) ✗ dvc config section
ERROR: argument name: name argument should look like remote.name.option or section.option
usage: dvc config [-h] [--global | --system | --local] [-q | -v] [-u] [-l] [name] [value]
Get or set config options.
Documentation: <https://man.dvc.org/config>
positional arguments:
name Option name (remote.name.option or section.option).
value Option value.
optional arguments:
-h, --help show this help message and exit
--global Use global config.
--system Use system config.
--local Use local config.
-q, --quiet Be quiet.
-v, --verbose Be verbose.
-u, --unset Unset option.
-l, --list List all defined config values.
```
We try to optimize `tree.exists` calls and probably few others
in that they either look directly into the workspace or,
to the cache without running graph checks. It does not seem
to be possible just to run graph checks on `find_outs_by_path`
due to those optimizations.
So, that's why, the `collect` also does a graph check for this
reason.
Fixes#5027Fixes#4010
* experiments: add utils.exp_commits for iterating over experiment commits
* repo: add support for all_experiments in brancher and used_cache
* git: make all_commits return commits in standard refs instead of --all
* gc: add --all-experiments to preserve cache for experiments
* update gc tests
* repo: support pushing and fetching specific revs internally
* exp push: push cache for experiment by default
* exp pull: pull cache for experiment by default
* fix empty git repo bug
* [WIP] git: use dulwich for add/commit
* dulwich: add is_tracked implementation
* experiments: try to workaround windows cleanup() bug on py < 3.8
* tests: experiments: convert CRLF to LF before comparing params
* trie: updated message for OverlappingOutputsError
the message given by the exception was not verbose enough why this exception is thrown.
now gives the user more information as to why the overlapping paths should be changed.
* test_run_single_stage: fixed test TestShouldRaiseOnOverlappingOutputPaths
the test is now updated to reflect the exception message changes.
Signed-off-by: metalglove <[email protected]>
* git: remove scm from backends, make backends configurable
* git: implement get/set/remove ref in gitpython
* git: support reflog --rewrite for stash
* tests: support testing individual backends in unit/scm/test_git
* stash: create reflog if it does not exist for gitpython backend
* git: fix Stash.push args typing
* experiments: handle case where args file was inadvertently committed to
git
* add test for args file being committed to git
* run/repro: show command on dry-run, add ">" to script output
This rollbacks the earlier commit to not show command during
the dry-run, and also prepends ">" in the command script
instead of "$" sign.
* fix tests
* Allow list type in STAGE_DEFINITION scheme and join(cmd) if list
* Remove join in Stage.__init__ and update test
* Loop over commands in cmd_run
* Improve logging of stage.run with detailed by command
* Fix test based on log message
* Update command logging with >
* Add tests for multi lines command
* Update > to $ in stage command log to prevent confusion with > operator
* Fix PR review about test: single quote and read_text
* tests: unify/clarify tests
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* WIP fixes#4621
* changes to all_branch if condition
* flatten_dict changes
* fix one test
* minor changes
* minor changes
* add precision
* tests and added precision to arguments
Co-authored-by: indh8717 <[email protected]>
* experiments: commit to refs/exps/... instead of branch heads
* update exp show
* restore checkpoint_resume kwarg
* refactor new() to use experiments stash
* experiments: use refspecs to populate and collect executors
* experiments: use new stash and merge on exp checkout
* experiments: fix executor chdir
* experiments: update run and show
* skip checkout after reproduce
* checkpoints: use experiments refspecs
* check if workspace was stashed before applying
* exp show: unify checkpoint/standalone experiment collection
* git: move iter_refs to backend
* update for scm changes
* experiments: replace exp checkout in favor of exp apply
* git: handle rev_parse errors in resolve_commit
* update refs containing and baseline check
* fix sigint handling for checkpoints
* update checkpoint exists behavior
* fix checkpoint baseline behavior
* update tests
* experiments: update gc
* update show tests
* fix type checking errors
* update diff
* scm: add diff, describe
* experiments: generate human readable names
* exp run/resume: allow user specified exp name with -n/--name
* update tests
* experiments: add `exp branch` for promoting experiment to full branch
* fix test
* fix missing file
* experiments: explicitly close scm instances before cleaning up tmp dir
* git: add support for dulwich Stash
* git: start adding support for custom refs
* git: add Stash and helper context managers for preserving
workspace state
* scm: cleanup refs/stash and add push_refspec
* tests: add tests for git refs and stash
* fix push_refspec
* tests: add tests for git push_refspec
* scm: add git fetch-refspecs
* add test for fetch refspecs
* scm: add test for git refs containing
* remove extraneous log msgs
* fix typing issues
* stash: only try to push if workspace is dirty
* update dulwich to 0.20.14
* scm: move dulwich into scm.backend
* move stash internals to backends
* scm: move git code into dvc/scm/git/... module
* git: use ABC for BaseGitBackend
* fix rebase issue
* git: fix refspec tests on windows
- git file:// URLs need posix style paths
* git: move get_refs_containing to backend
* scm: add callback for handling diverged refs during fetch
Having that logic in `create_stage` is unnatural and also has some
side-effects for unsuspecting users. For example, it will check the
filesystem for existing dvcfile and will try to load a corresponding old
stage. I've adjusted it with minimal changes to better reflect the logic
behind it.
Pre-requisite for #4907
* repro: support regex/foreach-group to run at once
Fixes#4912Fixes#4886Fixes#4958
* Use `tree.isdir` rather than `os.path.isdir`
* Use glob rather than regex
* Update dvc/command/repro.py
* s/regex/glob
* disable glob on `collect_granular`
There's no need for a glob here
* add tests for `collect` and `collect_granular`
* s3: use mocked s3 instead of the real one
Not removing the fixtures just yet
* Update moto
Tests was failing, because of a change in `responses` package, which
`moto` was mocking. This was fixed on
https://github.com/spulec/moto/pull/3466 which is only available on
`dev122` release.
* Use mocked s3 in checkout tests
`Azure` tests are flaky for some reason. When I investigated one
of the job [1], something seemed wrong with the Azurite or
the client lib.
On `AzureTree.blob_service`, we do the following to check if the
container exists:
```python
try:
container_client.get_container_properties()
except ResourceNotFoundError:
container_client.create_container()
```
When looking at the test log, I found out that the
`get_container_properties` was failing with
`ErrorCode:ContainerNotFound` whereas `create_container`
successively was failing with `ErrorCode:ContainerAlreadyExists`.
`create_container` is correct, but `get_container_properties` is
incorrect, as many of the Azure tests have already run successfully
before.
So, either the Azurite has/had bug or the client lib has a bug.
Anyway, we'll find out after updating the Azurite.
[1]: https://github.com/iterative/dvc/runs/1448864706#step:7:4029
* parametrization: prevent dumping parametrized stages
This is a bit hackish, but does the job done.
It feels like we need some unified way to be able to load
and dump, but not sure how. Till then, this dataclass
will be helpful in communicating between those.
* s/Meta/RawData
* Update dvc/stage/loader.py
Some of the tests are run by default, other requires
`--enable-{remote}` to run. The remotes that are run by default
also have a `--disable-{remote}` flag. All of the remotes can be
enabled using `--all` flag.
To use this functionality in the fixture,
`test_config` fixture has to be used, and requires to call
`test_config.requires(remote)`.
In an individual test, you can simply use `pytest.mark.remote`
to make them skip or run.
Via pyparsing. The error message is not that great, as it's greedy.
But, it does point out where the mistakes are, correctly most of the
time. :)
Also raises appropriate error messages for ParseError
The schema is changed of `vars` to accomodate the both use of
`imports` and setting variables locally.
Now:
```
vars:
- params.yaml
- foo: foo
bar: bar
```
Also, same distinction occurs for the stages.
* dvc: include cache urls in info output.
Show cache urls for external and remote cache in the info output
to assist in diagnosis of issues.
Fixes#4574.
* Remove unnecessary constant check for generator expression.
* Add remotes and caches in version command for better diagnosis.
* Fix flaky unit test to be runtime independent by using regex search.
* Use Cache.by_scheme() for enumerating supported caches.
* Add get_cloud_schemes routine for retrieval of configured remote schemes.
Add tests for local remote and remote:// scheme for info.
* Cleanup unused code.
* info: refactor remote parsing
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Outs desc will be used by viewer right away. Stage desc is for people to
start using now, but we won't be using it it viewer until DAG support is
added there.
* Add support for WebHDFS
* Style and linting fixes
* Implement abstract methods and disable no-member pylint errors
* Add init unit test
* Yield client and fix redefinition of hdfs
* Get dynamic hdfs Docker port
* Add support for open in WebHDFSTree
* Remove hdfscli_config fallbacks in WebHDFSTree
* Use port 50070 for webhdfs
* WebHDFSTree supports http URL and proper walking
* Fix failing unit test after refactoring
* Add overwrite argument to upload
* Add remaining func tests for webhdfs
* Remove print statements
* Allow dynamic attributes in pylint for hdfs library
* Comment error on instantiating HDFS client from config
* Remove depth argument from call to walk
* Update dvc/tree/webhdfs.py
* Remove unnecessary deque
* Check exception messages hdfs client failure from config
* Return client directly
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Split long line
* Add size to HashInfo
* Implement move via hdfs.rename
* Add missing line
* Make parent directories before moving file in WebHDFS
* Add total to progress bar
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tree: use file_mode in upload() for trees that support modes
* experiments: preserve file_mode for files in the workspace
* skip permissions tests on windows
* LocalTree: separate chmod from protect
* fix rebase issue
* experiments: Allow branching from checkpoints with modified hyper params
* experiments: initial checkpoints tree functionality for `exp show`
* go back to flat/sortable view
* add test for resume w/modified params
* tests: move experiments fixtures into conftest
* add show checkpoints test
* test checkpoint table rendering
* remove dead code
* test show for checkpoint branches
This is a hidden flag (no docs needed) for a handy shortcut to pdb when
we need it. E.g. binaries built by pyinstaller are much easier to debug
with it.
* Job count expected in a core config
Besides defaulted 4 * core_count number of workers sourced in the
"tree", the configuration file based value expected and prioritised.
* Multiply the configured value by cpu cores count.
* Don't multiply by the core count twice.
* A couple more usages and a question note.
* Don't multiply a number of jobs to a cpu count
Number of jobs set in a config file directly used to spawn workers.
* Remove the obsolete note.
* Disallow 0 jobs set in a config
* `jobs` config-based value is optional
* Apply suggestions from code review
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* CPU count based default
* Update dvc/tree/base.py
* Happy path unit test
* Update tests/unit/remote/test_remote.py
* Fix the accidentally removed blank line
* Update dvc/tree/base.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* api: add support for simple wildcards
Related to #4816.
Signed-off-by: Ioana Grigoropol <[email protected]>
* api: make wildcard interpretation optional
Adds a new argument for the add command `glob` that is disabled by default and when enabled it passes
the input targets through glob filtering.
Related: #4816
Signed-off-by: Ioana Grigoropol <[email protected]>
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
* Update dvc/repo/add.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
```
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\moto\core\models.py:17: in <module>
import responses
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\responses.py:29: in <module>
from urllib3.connection import HTTPHeaderDict
E ImportError: cannot import name 'HTTPHeaderDict'
```
* experiments: refactor regular/checkpoint experiment separation
* fix scm_checkout pull bug
* experiments: return stages from executor.reproduce
* since executor has to move to threadpool execution anyways, we no
longer have to work around the pickle repro result limitation
* refactor new/resume behavior
* refactor checkpoints tests
* use -f to force checkpoints reset/restart
* checkpoints: use 'exp run' and 'exp res[ume]' commands
* stage: all outs can be missing for checkpoint stages
* regular outputs may not be written at the time a checkpoint is
generated, but subsequent checkouts of that stage should not fail due
to the missing outs
* commit stage on checkpoint generation otherwise non-persist cached outs
are lost
* checkpoints: wait for stage proc to end after killing via terminate()
* stage: fix issues with killing checkpoint stage
* refactor signaling (use threading.Event instead of Condition)
* fix windows bug (kill all child processes via taskkill)
* experiments: refactor executor collection
* tests: move checkpoint tests into separate module
* refactor experiment tests to use stage fixture
We wrap every data to one of the `Node` object. But, since we were
passing `self.data` instead of `self`, it was skipping the conversion
and instead was just trying to set data as-is, which would have make
our latter assumptions incorrect, as other codebases assume it to be
a `Node`.
I have added asserts in a few places. Also, I noticed that, on a few
test assumptions, when `resolving`, we were not converting values
back to it's original form (`CtxDict` -> `dict`, `CtxList` -> `list`
and `Value` -> _). So, I have added `node.value` property, as it is
a complement of the above fix.
This caused a ripple effect for the changes in the API as `foreach`
now resolved and changed the data to the original form,
which prevents us from tracking the data later in the "field".
So, `resolve`/`select` got a real `unwrap` option that should
reliably work for all kinds of `Node` objects.
Currently we are converting dir_info to/from lists all the time.
The reason is that dir_info is stored as list of dicts in *.dir files,
but that makes it hard to work with. In addition to that, we will likely
be changing .dir file format in the near future #829, so we need to
abstract away dir_info into something that we won't care how it will be
stored on disk.
Related #3256
Related #4847
moto has aws-sam-translator as an indirect dependency that since 1.29.0
started to have a public package
that causes conflicts in flufl.lock when it tries to use atpublic
package.
```
tests/func/test_add.py:21: in <module>
from dvc.main import main
dvc/main.py:7: in <module>
from dvc import analytics
dvc/analytics.py:16: in <module>
from dvc.lock import Lock, LockError
dvc/lock.py:8: in <module>
import flufl.lock
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/__init__.py:3: in <module>
from flufl.lock._lockfile import (
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/flufl/lock/_lockfile.py:42: in <module>
from public import public
E ImportError: cannot import name 'public'
```
Oddly enough, this seems to also been causing pylint issues. public is a
terrible name for a package SMH
* http: added check for HTTP status code 404
Fixes#4288
* http: add unit test and minor change
In tests_exists, changed to raise a HTTPError from dvc instead.
Fixes#4288
* http: removed fixtures in test for HTTPTree.exists
Fixes#4288
* test: add comments on why `raw` is required on the mock
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
* command: add dvc push instruction after dvc repro
Adds a logger.info() statement to remind the user to run dvc push after
running dvc repro.
Fixes#2359
* command: change dvc push instruction for dvc repro
* command: Use backticks and else in dvc repro log
* add: warn on cache link failure instead of error
* add test case
* fix linted string
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
* metrics: include revision information when collecting metrics files
* plots: test warning message on no plot on revision
* plots/metrics: move to one package, introduce common tests
* tests: fix utils placement
In my tests `Git._ignore` is taking around ~65seconds for the whole test
suit(~50K invocations). The reason is that `gitpython` is launching a real
`git check-ignore` process, that takes time and resources.
Switching to `pygit2` or `dulwich` makes it drop to ~5seconds, saving us
almost a minute of the test run time and a lot of resources by not
launching any processes. I've chosen `dulwich` over `pygit2` here
because it is much more pythonic and as long as the performance is
on-par with `pygit2` it is better for us to use `dulwich`.
* Allow setting of list parameteres
Check if dict is Mapping or list before using .get()
* Added test for dvc exp run with list paramater
* Add more tests for string parameter modification
* Escape file paths for calculation of md5 hashes on ssh remote
* Add tests for hardlink and improved md5 test with PropertyMock
* Add additional special characters to check for fix of issue #4628 and fix formatting
* Fix linting errors
* Move tests to correct file and use injected ssh_connection instead of setting it up from scratch
* Add test, and fix resolving of paths for params
Improve merging of vars and imported params, improve tracking
Resolve wdir correctly
* Fix windows relpath
* isort imports
* Fixes#4637
* use PathInfo instead of Path
* catch ValueError with PathInfo.relative_to and fallback to dvc.utils.relpath
* cleanup
* add test
Co-authored-by: Peter Rowlands <[email protected]>
* repo: make Repo.scm a cached attribute
* checkpoints: create final checkpoint after repro when needed
* use correct exception rev in _collect_and_commit
* tests/remote: adding tests for real oss
the new test suggests #4633 is a bug in the emulator.
* Update tests/func/test_data_cloud.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update tests/remotes/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: correcting small lint issue
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* command: print the current cache directory when no cmd flags are provided
tests: corrected the test for cache dir cmomand
comamnd 'dvc cache dir' now prints the current cache dir
when no cmd flags are provided and modified existing test for
the same.
Fixes#1998
Signed-off-by: Girish Joshi <[email protected]>
* command: updated validation for args.value and args.unset flag
according to current convention in dvc/command/cache.py
Signed-off-by: Girish Joshi <[email protected]>
* Update dvc/command/cache.py
* dont forget logger
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
When executing `dvc import-url remote://some-webdav/data`, the name of
the WebDAV dependency in the created stage for data used to be `None`.
This resulted in the following error:
```
Traceback (most recent call last):
File "/home/schubert/projects/dvc/dvc/main.py", line 76, in main
ret = cmd.run()
File "/home/schubert/projects/dvc/dvc/command/imp_url.py", line 14, in run
self.repo.imp_url(
File "/home/schubert/projects/dvc/dvc/repo/__init__.py", line 51, in wrapper
return f(repo, *args, **kwargs)
File "/home/schubert/projects/dvc/dvc/repo/scm_context.py", line 4, in run
result = method(repo, *args, **kw)
File "/home/schubert/projects/dvc/dvc/repo/imp_url.py", line 54, in imp_url
stage.run()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 39, in wrapper
return deco(call, *dargs, **dkwargs)
File "/home/schubert/projects/dvc/dvc/stage/decorators.py", line 36, in rwlocked
return call()
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/site-packages/funcy/decorators.py", line 60, in __call__
return self._func(*self._args, **self._kwargs)
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 450, in run
self.save()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 395, in save
self.md5 = self.compute_md5()
File "/home/schubert/projects/dvc/dvc/stage/__init__.py", line 388, in compute_md5
m = compute_md5(self)
File "/home/schubert/projects/dvc/dvc/stage/utils.py", line 167, in compute_md5
return dict_md5(
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 118, in dict_md5
return dict_hash(d, "md5", **kwargs)
File "/home/schubert/projects/dvc/dvc/utils/__init__.py", line 113, in dict_hash
byts = json.dumps(filtered, sort_keys=True).encode("utf-8")
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/home/schubert/miniconda3/tmp/envs/some-project/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
TypeError: '<' not supported between instances of 'str' and 'NoneType'
```
* state: add simple reentrance
* state: pass Repo instead of LocalCache
* state: use granularly
Users could still load state preemptively if they want to bulk up
the operations, but other than that tree/cache will just load it
when they need it.
* stage: make runs checkpointable
* api: add make_checkpoint API call
* tests: add test for checkpointed stage
* experiments: implement checkpoints callback chain
* initial working state
* use .dvc/tmp for checkpoint signal file location
* force repro for checkpoint experiments
* checkout: add flag to allow missing persistent outputs
(i.e. not yet created)
* use allow_persist_missing on checkpoint experiment repro
* executor: force checkout before checkpoints experiment repro
* move checkpoint commands to `dvc exp run`
* support resuming checkpoint runs with --continue
* fix git bugs
* experiments: include checkpoint commits in `exp show`
* use 1sec sleep timers
* fix apply_workspace test conflict
* tests: add test for `dvc exp run`
* cleanup signal file behavior
* fix bool conversion
* tests: add tests for checkpoint and checkpoint_continue
* fix styling
* fix checkpoint stage test
* fix checkpoint monitor thread condition/notification
* repro: support --pull
This option allows run-cache to pull the cache for the outputs
when restoring a stage instead of pulling everything at once during
`dvc pull --run-cache`.
Fixes#4223
* repro: adjust --pull help
* yaml: wrap ruamel's DuplicateKeyError
Unfortunately ruamel's exception doesn't contain much info that we could
reuse, so we can only reasonably extract the exc.problem and have
something like:
```
ERROR: failed to obtain data status - unable to read: 'dvc.yaml', found
duplicate key "metrics" with value "[]" (original value: "[]")
```
Fixes#4664
* status: remove unneeded error prefix
* metrics: diff: make table more intuitive
Currently we have a "diff not supported" marker in the table, that is
used whenever dvc is not able to diff the values (e.g. no old one, no
new one or format error). It is quite ugly and it is much more intuitive
to just show both old and new values and now show that marker at all, as
if there is no old/new value it is intuitive that we can't calculate the
change. Next step is to add old/new "bad format"/etc markers for
unsupported types or malformed/missing metric files.
E.g., before:
```
Path Metric Value Change
metrics.json foo 1.2 diff not supported
```
after this PR:
```
Path Metric Old New Change
metrics.json foo None 1.2
```
* metrics/params: use em dash in diff table
* dvctree: use string paths for performance reasons in
_get_granular_checksum()
* dvctree: cache most recently accessed metadata
* dvctree: cache output dir cache during walk() calls
* diff: use get_file_hash directly instead of get_hash when appropriate
* dvctree: cache most recen dir_cache hashes in _get_granular_checksum
* revert metadata cache changes
- mostly unnecessary after granular checksum changes, will be better
addressed by the future exception handling updates
* fix windows path issue
* HashInfo: store HashInfo.dir_info as a dict instead of list
* DvcTree: use hash_info.dir_info for getting granular file hashes
* use str relpath
* use correct dir_info hash type
* fix windows paths
* Run CI for Python3.9, skips hdfs tests for 3.9
Also, `dvc[hdfs]` install will be a no-op till we have
a pyarrow wheel for 3.9.
Note that this does not add an entry to the classifier
claiming it supports 3.9, that will be added when 3.9
gets released (or, wait for pyarrow to be released).
* Skip HDFS related tests on Python3.9
Also, fix the python version regex for `3.9.0b5` like versions.
* Exclude python3.9 for Windows
libxml does not have wheels for Python39.
We do compile them on Linux and Mac
* Update tests/unit/test_info.py
* Use realpath for gitignore
Windows tests for Python3.8 was failing
intermittently.
Note: Also removed the realpath wrap around self.root_dir as it's already a
realpath when `Git` is instantiated.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* status: Show friendly message for empty project.
When there are no stages in the repo, show a friendly message to that
effect and display a getting-started URL.
Fixes#4395.
* Fix style check failing due to line too long. (E501)
* Use format_link(…) as suggested by @skshetry, vs {blue}/{nc}.
Since I'd pulled the old syntax out of repo/init.py, I've fixed that up
as well so no one makes that mistake again. Note that this adds <>s around
the link text, unlike before:
+---------------------------------------------------------------------+
| |
| DVC has enabled anonymous aggregate usage analytics. |
| Read the analytics documentation (and how to opt-out) here: |
| <https://dvc.org/doc/user-guide/analytics> |
| |
+---------------------------------------------------------------------+
This is probably fine, since if we're going to have a standard link format
we might as well stick with it consistently. If we don't like it, we can
change format_link() I guess.
* Update dvc/command/status.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Paramiko has recently added support for `Match`
https://github.com/paramiko/paramiko/issues/717
which resulted in some users getting an import error about `invoke`
module, as it is not a default dependecy for paramiko.
@Pseudomanifold reported that installing `invoke` #4589 worked for
him, which means that this newly added paramiko feature works in
his case. That feature seems to not affect configs without `Match`,
so we could give this a try and install it by default.
Our conda package will be updated during the next release.
Fixes#4589Fixes#4616
* Do not require account-level permissions for Azure tree
* Only allow the existence check to pass without error if it failed due to authorization issues
* Update dvc/tree/azure.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/tree/azure.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Restyled by isort
* Update azure.py
Co-authored-by: Matt Hollingsworth <[email protected]>
Co-authored-by: Matt Hollingsworth <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Restyled.io <[email protected]>
* Fix error during external outputs checkout
User in discord complained that external outputs' checkout
was failing. Looks like, during refactor of `HashInfo`,
this was forgotten, and as no tests existed before, we had
a regression from 1.6.5 onwards.
* fix tests
* Restyled by black (#4571)
Co-authored-by: Restyled.io <[email protected]>
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
Co-authored-by: Restyled.io <[email protected]>
* `check-ignore` show sub-repo patterns
fix#4555
1. Sub-repo pattern now would be considered in `check-ignore` command.
2. New outputs for sub-repo patterns in a `detail`
3. Related tests.
* message change
* tree: validate resolved remote:// sections
* dvc: use checksum_jobs instead of hash_jobs
This is a regression introduced during one of the refactorings.
* Print dvc version info for debugging
fix#4095
1. seperate info from dvc version.
2. print dvc version info when some unexpected error occured.
* Psutil NoneType error
* Update tests/func/test_version.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Move `package` outside dvc/utils/pkg.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
The API of `Dvcfile.dump` has been changed to
`dump(stage, update_pipeline=True, update_lock=True)`,
which will make it to always dump by default to both files,
and provide flags to suppress one or the other.
Fixes#4235
* allow for a config option for http method
* add tests
* fix tests with method
* Update dvc/tree/http.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
We used to use that logic in DvcTree back before we had proper trees.
Now we have a RepoTree, that combines git/workspace tree with the
DvcTree and dirty state clearly belongs to the git/workspace and not
the dvc itself, as dvc state only depends on dvc-files and lock-files.
Prerequisite for #4523
* diff: support missing cache status for workspace diff
* diff: add --hide-missing option to hide cache missing status
* update command tests
* update diff func tests
* only show missing cache status when diffing local workspace
* suppress all output if no changes and --hide-missing
* use hide_missing instead of show_missing
* use is_dvc
* dvc: override replace in HTTPURLInfo to include extra_parts
The objective is to passtrough any parameter configured in http remotes.
fixes#4508
* add unit test for #4508
* Use HTTPURLInfo directly instead of decorator
* diff: use RepoTree to get granular results for dir outs
* update tests for dir file granularity
* diff: prefer repo_tree.get_hash when possible
* DvcTree: handle extra files in dirty dvc out dirs
* support subrepos
This adds support for get/list/api/import/ls on a subrepo.
Also fixes#3180, by adding a granular url support for `dvc.api.get_url`
And, of course it fixes#3369.
* Add tests
* Fix tests, started to get lock_error
* Fix tests for windows
* try to fix mac builds
* Fix comments, readd removed comments
* Fix test
* address review suggestions:
* use `spy` instead of `wraps`
* remove dvc fixture in a test
* set `cache_types` on Git repo
* version: improve link type detection fall back
If failed to determine cache type print link to related web page.
Fixes#2788
* Update dvc/command/version.py
* fix formatting
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
By itself `self.info` is quite confusing, as it is not clear what it is
about. Using `hash_info` makes much more sense and is required to
support alternative hash types.
Related to #4144, #3069, #1676
fix#4482
According to the discussion. Make `DVC status [outputs]` output format the same with `DVC status [stages]` instead of `DVC status -c [outputs]`
* experiments: cleanup show UI
* show git tag/branch names instead of SHAs by default
* make timestamp formatting more like the viewer
* update tests
* place timestamps in their own column
* update command test
* exp: remove unused regex
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: use cache state when saving downloaded cache
Related to #4485
* tree: dont compute hash from scratch for output dirs
Fixes#4485
* tree: fix get_hash
* plots: add -e/--experiment option for plots diff
* experiments: add method to get baseline for a given experiment
* plots: default to diff against baseline instead of workspace/head for experiments
* use exact match for experiments RE since name-rev may return relative
commits
* diff: Add sort logic to sort diff ouput irrespective of output format(default/json/md)
* diff: test stub using actual value, not pass by reference for (dict)
* Speed up dvc files load
See `._load()` implementation. Alternative is not using 3 io calls, but
simply use open and catch exceptions.
Reading is slow for old git revisions.
* tree: git: cache tree and objects
* external: ignore PermissionError on Windows
This is the same thing we have in tests.
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Outputs as target supporting for `dvc status`
fix#4191
1. Add a related test which would fail on current version.
* Pass the tests
1. add deps to the tests.
2. make change to pass the tests.
* Update tests/func/test_status.py
Co-authored-by: Saugat Pachhai <[email protected]>
* Solve some change request.
Co-authored-by: Saugat Pachhai <[email protected]>
* executor: abstract tmpdir population using tree upload
* experiments: unmark flaky test
* fix wrong tree download
* LocalTree: use os.replace instead of os.rename in _upload
* Also fixes plots not getting dumped on flow style
* Make serialization utils aware of trees
* Directly load yamls/tomls using utils, rather than using parser
* use yaml 1.2 everywhere
This commit:
* gets rid of pyyaml
* uses ruamel everywhere
* tries to unite toml and yaml
However, this will require a few more changes that will come soonish.
Eg: Current `parse_yaml` and `parse_toml` behavior could be error-prone,
so, it's better to introduce `tree` inside these serialization methods
(which I will try to tackle next).
Another is, yaml load can be slow for json. We could speed it up just by
using `json.load(s)` here (also in upcoming patch).
* Fix tests
* experiments: use RepoTree for executor output collection
* experiments: use tree.download for output collection
* cleanup docstring
* use string name for tqdm
* Imports are no longer moved to top.
* `seed-isort-config` is no longer needed.
* Has a `black`-compatible profile/config.
* Also `isorts` the imports anywhere in the code
(not just at the top).
Also bumps `restyled-isort` to newest version
Currently we kinda assume that whatever is returned by `get_file_hash`
is of type self.PARAM_CHECKSUM, which is not actually true. E.g. for
http it might return `etag` or `md5`, but we don't distinguish between
those and call both `etag`. This is becoming more relevant for dir
hashes that are computed a few different ways (e.g. in-memory md5 or
upload to remote and get etag for the dir file).
Prerequisite for #4144 and #3069
Fix#4333, as `is_ignored_file` and `is_ignored_dir` and `__call__`
share the same call `DvcIgnorePatterns.matches`, we only need to focus
on the out of repo situation.
1. Add tests for the `is_ignored_file` and `is_ignored_dir` for the out
of repo condition.
2. Now `is_ignored_file` and `is_ignored_dir` returns false for those
files outside the repo.
* experiments: add options to include/exclude specific metrics/params columns in
`dvc exp show`
* allow param/metric file to be provided to keep consistency with other
dvc commands
* match nested params/metrics names
* Remove dvcignore reset in `repo._reset`
Repo `add`,`checkout`,etc will not change `.dvcignore`, and can remove dvcignore reset after these operations.
* Reset dvcignore manually after a subrepo created.
1. subrepo created during session open time.
* tests: add test to ensure dir hashes are not pushed for incomplete
directories
* push: do not push .dir file for directory with missing files
* update test
* Error message when no match found.
1. remove error message when no match found.
* Test update.
1.Test update cover return 0 and 1
* Remove unused arguments
* Error message show in dvc check-ignore
fix#4361
1. Remove error message when input a empty string
2. Remove error message when read input from stream
2. Remove error message when interrupt by `CTRL-C`
* Allow ignoring subrepos in dvcignore
I implemented this inside dvcignore, and it
should not require "internal" APIs hack as we discussed before.
Right now, `ignore_subrepos` will only work for the paths of subrepo,
not anything inside it. And the API user is more or less only
RepoTree, so should be safe (we switch DvcTree and repo.tree based
on path prefixes, so this is only required during `walk`, so as it
shows the repo that were previously dvcignored and so that we could
switch the trees later during walk.
So, if dir is `dir -> repo`
So, `tree.walk("dir")` will return `[]`,
whereas `tree.walk("dir", ignore_subrepos=False)` will return `["repo"]`
But, successive walk will just return `("repo", [], [])`, and could
just be ignored as a side-effect.
* Fix tests
* Two new mode `-all` and `--stdin` for check-ignore
Fix#4321
Introduce new arguments --all for dvc check-ignore
Add tests for dvc check-ignore --all
* `--stdin` and targets only one
1. `--stdin` and targets can and only can have one
2. more tests
* New tests for stdin mode and solve a problem
1. Add a new tests with mock stdin for interactive mode
2. Solve the problem of `assert "" in caplog.text` in tests always `True`.
* Remove path_input interface
1. remove path_input and make _ask to a public method instead
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* Update dvc/command/check_ignore.py
* fix styling
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* tests: test importing into subdirectory
* import: fix bug when output subdir doesnt already exist
* use urlparse to determine if output path is local, not os.path.exists
* account for windows drive letters
* import: error out if output subdirectory does not exist
Tests had checks for python version number, expecting it to be only
single digits, which was failing for me on 3.6.10 and might fail
in the future with Python 3.10.
I adjusted it to expect minor and patch version to be of multiple digits.
* restore Experiments.chdir
* used in experiments.show
* experiments: make checkout work without `git reset --hard`
* experiments: remove --force parameter
* experiments: reject checkout if current workspace commit doesnt match
expected baseline
* Update some tests first
fix#3736
* first edition
* solve failure in Windows
* For Windows ci
* Some help ducuments issue.
* Update dvc/ignore.py
abspath
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Refactor with OutOfWorkSpaceError
* Solve a bug
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
Co-authored-by: Jorge Orpinel <[email protected]>
* Update dvc/command/check_ignore.py
* Revert "Refactor with OutOfWorkSpaceError"
This reverts commit 27eec49ed67346c89157edea998b6b5fa81d88f5.
* Two change request
1. Argument `targets`'s description.
2. Error handling of `_get_normalize_path`
* Update dvc/main.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* `check_ignore` now only accept one path a time
1. Add a new test for the out side repo cases
2. check ignore now only check one file not file lists
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
* tree,remote: add support for webdav
Webdav support is based on https://pypi.org/project/webdavclient3/ and
supports basic download/upload operation, directory creation as well as
existence, file hash and isdir query. Copy, move and remove are also
implemented, though probably not used yet.
WebdavURLInfo is taken from https://github.com/shizacat/dvc/tree/remote-webdav
Fixes iterative#1153
* tree,remote: add further webdavclient3 options
Webdav token auth, certificate and key path and connection timeout are
configurable. Webdav username might be specified or extracted from URL.
Refs iterative#1153
* tree,remote: validate webdav client configuration
Refs iterative#1153
* tree,remote: WebDAV is written with capitalized 'DAV'...
Refs iterative#1153
* tree,remote: terminate WebDAV makedirs at self.path_info.path
This enables the WebDAV api location (e.g. '/public.php/webdav') to be
part of the remote 'url' configuration instead of beeing specified
separately via the 'root' option. The 'root' option may then be used to
specify real directories at the WebDAV storage, although using it to
set the api location is still possible.
Refs iterative#1153
* tree,remote: use >=3.14.5 for WebDAV dependency webdavclient3
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458490609
Refs iterative#1153
* tree,remote: get rid of WebDAV 'root' option and add connection check
The WebDAV 'root' option was rather confusing and should be handled by
the initial 'path_info' from the config 'url' option.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458754500
While stripping the path/root from the hostname the port got lost, which
is fixed now by simply using the URLInfo 'replace' method as suggested.
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458842853
The WebDAV client connection is tested by probing the existence of the
root (self.path_info.path).
Refs iterative#1153
* tree,remote: implement walk_files for WebDAV
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs: iterative#1153
* tree,remote: let WebDAV client list query file info in walk_files
Context: https://github.com/iterative/dvc/pull/4256#discussion_r458751391
Refs iterative#1153
* tree,remote: add some unit tests for WebDAVTree
Refs iterative#1153
* tree,remote: use ConfigError for WebDAVTree and move Error to webdav.py
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460591937
Refs iterative#1153
* tree,remote: remove/change some (unnecessary) comments
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460592181
* tree,remote: uploading to WebDAV only create directories if not exist
* tree,remote: add BaseTree parameter use_dvcignore to WebDAVTree exists
Refs iterative/#1153
* tree,remote: remove WebDAVTree copy method as proposed
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460892725
Refs iterative#1153
* tree,remote: add progress bar to WebDAV _download and _upload method
Context: https://github.com/iterative/dvc/pull/4256#discussion_r460593130
Refs iterative#1153
Co-authored-by: Christoph Berganski <[email protected]>
* experiments: add initial local tmpdir executor
* fix detached head pull
* experiments: include unchanged (unrepro'd) stages in experiment hash
* experiments: repro single experiment using tmpdir executor by default
* stage experiments as stash commits before running
* include repro args/kwargs in stashed experiments
* support running arbitrary experiment commits (including stash commits)
* experiments: use ProcessPoolExecutor
* experiments: add `dvc exp` alias for `dvc experiments`
* experiments: add `dvc repro --experiment --queue`
* `--queue` can be used to stage an experiment for future execution
* experiments: show queued (unexecuted) experiments in `dvc exp show`
* revert ProcessPoolExecutor change
* experiments: add `dvc repro --run-all [--jobs]`
* `--run-all` can be used to run all queued experiments in parallel
* cleanup output
* experiments: use ProcessPoolExecutor
* fix returning unpicklable objects error
* update tests
* fix windows cleanup issue
* on windows tempdir cannot be removed if we are chdir'd into that
directory
https://github.com/pytest-dev/pytest/issues/7558
pylint throws error for `pytest.mark.*` functions in pytest6.
```
tests/unit/repo/test_repo.py:15:1: E1102: pytest.mark.parametrize is not callable (not-callable)
tests/unit/repo/test_repo.py:33:1: E1102: pytest.mark.parametrize is not callable (not-callable)
************* Module tests.unit.remote.ssh.test_connection
tests/unit/remote/ssh/test_connection.py:87:1: E1102: pytest.mark.skipif is not callable (not-callable)
tests/unit/remote/ssh/test_connection.py:100:1: E1102: pytest.mark.skipif is not callable (not-callable)
```
https://github.com/iterative/dvc/pull/4257#issuecomment-665474467
* [WIP] gdrive: add workaround for downloading empty files
Better to solve it on PyDrive's side:
https://github.com/iterative/dvc/issues/4286Fixes#4286
* deps: limit flatten_json
```
pylint...................................................................Failed
- hook id: pylint
- exit code: 2
Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.command.experiments
dvc/command/experiments.py:14:4: E0401: Unable to import 'flatten_json'
(import-error)
dvc/command/experiments.py:39:4: E0401: Unable to import 'flatten_json'
(import-error)
-----------------------------------
Your code has been rated at 9.99/10
Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.utils.diff
dvc/utils/diff.py:38:8: E0401: Unable to import 'flatten_json'
(import-error)
************* Module dvc.command.metrics
dvc/command/metrics.py:17:4: E0401: Unable to import 'flatten_json'
(import-error)
-----------------------------------
```
* Version: Redesign output format #3499
Redesigned the output format for the command: dvc version
* version: reformat output format #3499
Done some suggested fixes in code for the dvc command output format issue.
* version: redesign output format #3499
* version: redesign output format for dvc command #3499
* version: redesign output format #3499
* version: redesign output format #3499
* Redesign output format for the dvc version command #3499
* Redesign output format for the dvc version command #3499
* Update dvc/command/version.py
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
```
pylint...................................................................Failed
- hook id: pylint
- exit code: 2
Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.command.experiments
dvc/command/experiments.py:14:4: E0401: Unable to import 'flatten_json'
(import-error)
dvc/command/experiments.py:39:4: E0401: Unable to import 'flatten_json'
(import-error)
-----------------------------------
Your code has been rated at 9.99/10
Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.utils.diff
dvc/utils/diff.py:38:8: E0401: Unable to import 'flatten_json'
(import-error)
************* Module dvc.command.metrics
dvc/command/metrics.py:17:4: E0401: Unable to import 'flatten_json'
(import-error)
-----------------------------------
```
This allows us avoid collecting dvcignore for the whole repo if we only
care about particular paths. As a result, in a repo with 2 datasets
(2M + 0.5M files), creating a defunct stage takes ~4sec on 1.2.0, but
~1sec(most of it is actually dvc module initialization) with this PR.
This is also a pre-requisite for dynamic dvcignore and subrepo
collection (https://github.com/iterative/dvc/pull/4247) while walking
the tree.
Also, it is important to clarify that regular `dvc status`(without
arguments) has the same performance after this PR, because when we check
dataset for changes, we call things like `tree.exists()`, which call
dvcignore and make it collect dvcignore in the dataset itself, so we
still endup collecting dvcignore for the whole repo (including walking
into the datasets). This should be solved soon by telling dvcignore that
it shouldn't walk into the datasets searching for `.dvcignore`s.
We used `Popen` just to provide a general way of launching our daemons
on both Windows and *nix, but it is causing us issues when dvc is
launched from some exotic places (like nix package) or through `main()`
helper from a script that is not dvc.
This patch makes us just use `main()` directly in our double-forked
process.
Windows implementation has to use the `Popen` mechanism, because this is
the only way to daemonize a process on Windows. This means that dvc on
Windows is still susceptible to this problem, but there is nothing we
can do about it right now :(
Fixes#4206Fixes#4165
* Update
1. unit test update
2. two might cause bugs
* dvc ignore
* Some refactoring
* clean
* Config tree use
* Solve sub dir repo problem. Add more tests
* typo error
* Change request
Co-authored-by: karajan1001 <[email protected]>
* erepo: use shallow clones when rev is branch or tag name
* tests: test erepo shallow clone behavior
* erepo: fix detached head after shallow cloning a tag
* move unshallow logic into its own function
* erepo: default rev should be origin HEAD not local HEAD
* use functools.partial
* review fixes
* comment shallow clone behavior in erepo
* don't duplicate unnecessary fetch/pull call
* dvc: get rid of is_walking_tree
No longer needed. Some applications of `isinstance()` will be refactored
in the future, as trees will just provide the needed functionality.
* dvc: move GitTree to dvc/tree/
* dvc: rename *RemoteTree to *Tree
* key api
* Restyled by black
* test
* fixes
* test
* Restyled by black
* Trigger CI
* default val
* remove redundant default
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update tests/unit/remote/test_s3.py
* rename fields
* update test
* key_id->secret_key_id
Co-authored-by: Restyled.io <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* experiments: add initial experiments.show
* only includes metrics + params (no code/data information)
* experiments: basic `dvc experiments show` functionality
* experiments: clone and run repro inside clone workspace
* repro: add -e/--experiment option
* experiments: hash experiments to identify duplicates
* experiments: update show
* experiments: checkout experiment after running repro
* experiments: add `dvc experiments checkout`
* experiments: add simple `dvc experiments diff` command
* just shows combined output from `metrics diff` and `plots diff` for
now
* add very simple tests
* add some useful output messages
* experiments: suppress help messages while feature is in development
* experiment related commands will be accessible but hidden from the
default command help messages
* add simple experiments command tests
* experiments: disable feature by default
* experiments only enabled in test environment (DVC_TEST) or when
core.experiments config option is true
* use fetch on checkout missing revs
* dvcignore: move file/dir/parent checks into DvcIgnoreFilter
* dvc: get rid of CleanTree
CleanTree is a very awkward wrapper that spreads the context across
the codebase and results in unexpected behaviour when using it.
This PR starts moving dvcignore-related logic into the trees themselves
(it makes a lot of sense, kinda like `state`) so they could deal with
it however they like.
There are at least two temporary ugly parts about this PR:
1) dvcignore is used by individual trees and not packed into
tree/base.py yet;
2) `dvcignore_root` argument. This one is caused by the dvcignore trying
to collect everything topdown starting from the certain root dir. What
it should do instead is for a certain path that is being checked look up
the tree through the parents until it finds repo root (.dvc dir) and
then stop. That would handle subrepos as well. At the same time we need
to leverage existing dvcignore trie structure to cache those results.
Working tree is really just a regular local tree and should be
used by outputs when trying to compute a hash for themselves.
We didn't use it previously because local tree was embeded into
the local remote class.
Related to #4050
Outputs and dependencies don't really care about the remotes, as they
don't do any pushing or pulling. What they really need are the trees,
so let's use them directly.
Part of #4050
* Remove Duplicate ignore Match
* Continue Optimize Dvcignore
fix#3869
* Add a new test with multi ignore files
* Solve merging two dvc files
* Solve Code Climate
* For Windows
* Complete addition of patterns. Add one test
* Systematic test
* Change request
* Change request
* Seperate path sepcification math
* Rename and add comment
* rename change_dirname to private
* Update dvc/pathspec_math.py
list comprehension
Co-authored-by: Alexander Schepanovski <[email protected]>
* Change request
* Update dvc/ignore.py
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Alexander Schepanovski <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* remove: remove dvc.yaml, dvc.lock if they are empty
* tests: remove: simplified dvc.yaml in test_remove_stage_removes_dvcfiles_if_no_stages_left
* tests: remove: minor fixes
* tests: remove: simplified test_remove_stage_removes_dvcfiles_if_no_stages_left
* remove: independent removal of dvc.lock and dvc.yaml
The falsy values were being skipped when merging params values
from dvc.yaml and dvc.lock, showing the values as always changed.
These values were never added to `info`, but were found in
`self.params` making them appear as changed.
Fixes#4184
* Add more tests according to .gitignore
* Test update
* Update test
* Solve question of path
* update one test
* Update 2 tests
* Add one tests
* comment those failed tests caused by other package
* parametrize some tests
Co-authored-by: karajan1001 <[email protected]>
Usually we need it to parametrize tests with, so there is no
reason to duplicate the code. I've left `local_remote` fixture
just because it is so convenient and often used in a lot of tests.
* [RFC] tests: make read_text() work with directories
This PR is just a showcase to see how we like it. I've converted a few
tests to it, but there will be much more to come.
```
assert (tmp_dir / "file") == "contents"
assert (tmp_dir / "dir").read_text() == {"subdir": {"file": "hello"}}
```
This approach is nicer because it allows us to use a declaratory
asserts instead of relying on fs files. The problem becomes much more
apparent once you get into remote workspaces, where we `trees_equal`
doesn't work.
* status: pass tree as a kwarg
Fixes#4150
Depends on https://github.com/iterative/dvc/pull/4155
This PR is just a showcase to see how we like it. I've converted a few
tests to it, but there will be much more to come.
```
assert (tmp_dir / "file") == "contents"
assert (tmp_dir / "dir").read_text() == {"subdir": {"file": "hello"}}
```
This approach is nicer because it allows us to use a declaratory
asserts instead of relying on fs files. The problem becomes much more
apparent once you get into remote workspaces, where we `trees_equal`
doesn't work.
* s3,gs: don't append "/" to prefixed path when listing objects
* remote: use optional prefix kwarg for walk_files
* if prefix is True, path will be walked as a prefix
* if prefix is False, path will be treated as a directory (trailing
slash will be appended for S3-like remotes)
* tests: add unit test for BaseRemoteTree.list_paths
flufl.lock released 4.0 in which there is no finalize method anymore, hence
```
dvc/lock.py:143:16: E1101: Instance of 'HardlinkLock' has no 'finalize' member (no-member)
```
2020-07-01 01:55:30 +03:00
527 changed files with 42852 additions and 14925 deletions
Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.
Example: `repro: doesn't detect input changes`
-->
## Description
<!--
A clear and concise description of what the bug is.
-->
### Reproduce
<!--
Step list of how to reproduce the bug
-->
<!--
Example:
1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->
### Expected
<!--
A clear and concise description of what you expect to happen.
-->
### Environment information
<!--
This is required to ensure that we can reproduce the bug.
-->
**Output of `dvc doctor`:**
```console
$ dvc version
$ dvc doctor
```
**Additional Information (if any):**
<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.
If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
* [ ] 📖 If this PR requires [documentation](https://dvc.org/doc) updates, I have created a separate PR (or issue, at least) in [dvc.org](https://github.com/iterative/dvc.org) and linked it here.
* [ ] ❌ I will check DeepSource, CodeClimate, and other sanity checks below. (We consider them recommendatory and don't expect everything to be addressed. Please fix things that actually improve code or fix bugs.)
Thank you for the contribution - we'll try to review it as soon as possible. 🙏
#.*Data Engineering* tools such as `AirFlow <https://airflow.apache.org/>`_, `Luigi <https://github.com/spotify/luigi>`_ and
others - in DVC data, model and ML pipelines represent a single ML project and focuses on data scientist' experience while
data engineering pipeline orchestrates multiple data projects and focuses on efficent execution. DVC project can be used
from data pipeline as a single execution step. `AirFlow DVC <https://github.com/covid-genomics/airflow-dvc>`_ is an example
of such integration.
#.`Git-annex <https://git-annex.branchable.com/>`_ - DVC uses the idea of storing the content of large files (which should
not be in a Git repository) in a local key-value store, and uses file hardlinks/symlinks instead of
copying/duplicating files.
@@ -193,7 +201,8 @@ Comparison to related technologies
#.`Workflow Management Systems <https://en.wikipedia.org/wiki/Workflow_management_system>`_ - DVC is a workflow
management system designed specifically to manage machine learning experiments. DVC is built on top of Git.
#.`DAGsHub <https://dagshub.com/>`_ - This is a Github equivalent for DVC. Pushing Git+DVC based repositories to DAGsHub will produce in a high level project dashboard; including DVC pipelines and metrics visualizations, as well as links to any DVC-managed files present in cloud storage.
#.`DAGsHub <https://dagshub.com/>`_ - online service of Git+DVC repositories with pipeline and metrics visualization and
DVC-specific cloud storage.
Contributing
============
@@ -203,38 +212,6 @@ Contributing
Contributions are welcome! Please see our `Contributing Guide <https://dvc.org/doc/user-guide/contributing/core>`_ for more
Barrak, A., Eghan, E.E. and Adams, B. `On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects <https://mcis.cs.queensu.ca/publications/2021/saner.pdf>`_ , in Proceedings of the 28th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2021. Hawaii, USA.
Description: Git for data scientists - manage your code and data together
Data Version Control or DVC is an open-source tool for data science and machine learning projects.
.
Key features:
.
1. Simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services.
.
2. Management and versioning of datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS, or even local HDD RAID.
.
3. Makes projects reproducible and shareable; helping to answer questions about how a model was built.
.
4. Helps manage experiments with Git tags/branches and metrics tracking.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ruslan Kuprieiev
Saugat Pachhai
Dmitry Petrov
dependabot[bot]
Batuhan Taskaya
Peter Rowlands (변기호)
Ivan Shcheklein
ciaochiaociaoandChiao-Wei Hsu
Jan
Naveen Sinha
Jorge Orpinel
Bhagawan Subedi
David de la Iglesia CastroandPeter Rowlands
Alex Lopez
Alex Ford
Helio MachadoandPeter Rowlands
nik123
Laurens Duijvesteijn
Laurens Duijvesteijn
Corentin Hembise
Sami Jawhar
Pavel Ditenbir
RogerandPeter Rowlands
Piotr Bielak
Timofey Yefimovandkarajan1001
Asif Saif UddinandRuslan Kuprieiev
Dmitry Kropachev
Yihao JiangandSaugat Pachhai
mbiesekandRuslan Kuprieiev
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి)
Michael StegemanandOLOLO ALALA
Ioana Grigoropol
devramx
João Rafael
Sandeep MistryandRuslan Kuprieiev
Johnny Chen
metalglove
OrangeCubeandGary Yuan
Victor
Clément WalterandRuslan Kuprieiev
jwokatyandRuslan Kuprieiev
alex lustenberg
Indhu Kumarandindh8717
Arun MahapatraandRuslan Kuprieiev
Tomás Farías SantanaandRuslan Kuprieiev
Ilya PekelnyandRuslan Kuprieiev
danielfleischer
Guro Bokum
LucaButeraandLuca Butera
Michal NowikowskiandRuslan Kuprieiev
Liew XunandSaugat Pachhai
simonlousky
Kevin Lloyd Bernal
João Vitor
Juliano Efson Sales
Tomás Farías Santana
Samuel Neugber
Harsh Patel
Alexei Tenitski
Musa AtlıhanandPeter Rowlands
Jonny Browningandbrowningjp
omar.zhouandzwh
Frederik Schubert
Guillermo Blanco
Casper da Costa-Luis
Raymond GasperandRuslan Kuprieiev
gsvoltandRestyled.io
Guillaume COTER
David RobertsandRuslan Kuprieiev
Robert Lowry
Alexanderandaandrusenko
David Roberts
Robert F. DickersonandRuslan Kuprieiev
Alexander Schepanovski
Puneetha Pai
Christoph Berganski
AnatolyandRuslan Kuprieiev
Sarthak khandelwalandRuslan Kuprieiev
Arthur Colombini Gusmão
dtrifiroandDaniele Trifirò
faucct