* exp: allow fetching from arbitrary executor by git URL
* exp: write info for running executors to .dvc/tmp/exp/run
currently only includes PID and git fetch URL
* exp: refactor stash/queue cleanup
* exp: add ability to collect dict of currently running experiments
* exp show: display run/queue state in the table
* exp run: unlock main repo while running tmpdir executors
* add tests for new exp show behavior
* fix workspace checkpoint run handling
* fix issue where running stash revs were shown as queued
* use "running"/"queued"
* add executor (location) column
* compare: add TabularData.is_empty
* use td.is_empty to drop state/executor
* use "workspace" and "temp" as executor names
* fs: use fsspecwrapper in memfs
* odb: add memfs staging ODB in ODBManager
* odb: add read_only attribute for odb instances
* objects: handle memfs paths when staging trees
* odb: remove git ODB
* output: use staging ODB
* RepoDependency: use staging odb for imports
* dvcfs: handle case where dir cache is unavailable
* fix circular import check
* odb: use temp local ODB for staging
* objects: use staging ODB in stage() and tree load
* objects: add typing for add/check/load/stage
* output/dep: use new staging
* add unit tests for local odb staging
* localfs: account for string paths in move()
* include local in odb.by_scheme
* tests: count dir cache like any other object in test_pull_git_imports
* odb: clear staging on gc()
* objects: add tree.get/filter/insert
* output: use obj.filter/get for granular dir commit
* squash filter logic output
* dependency: don't filter imports at the dep level
* don't init staging ODB for external cache
* use staging ODB when computing tree hash for external outs
* remove unneeded force in test granular commit
* use unique-per-manager memfs URLs for staging
* odb: make staging per-ODBManager
* objects: move state dependent staging into ODBManager
* objects: support checking multiple odbs
- staging dependent checks can be done via ODBManager.check
* checkout: do state lookup before explicit stage() in _changed()
* output: use odbmanager based staging
* RepoDependency: use odbmanager based staging
* update obj staging unit tests
* checkout: remove check(), handle individual file exceptions on link()
* diff: use odbmanager based staging
* objects: add ObjectPermissionError for read/write perm errors
* odb: move staging out of odbmanager
* objects: handle staging ODB in objects.stage()
* objects.tree: don't stage anything in tree methods
* odb: clear staged objects on gc()
* remote: don't filter objs by scheme, objs are already grouped by ODB
* diff: update objects.stage usage
* output/dep: update stage() usage
* update ODB tests
* update func tests
* remove memfs from FS_MAP
* revert unneeded objects.check() changes
* remove unused tree.insert()
* revert unused odbmanager changes
* add separate odb error classes
* stage external outs directly in ODB
* check for obj existence in main ODB before staging
* use .dvc/tmp/staging instead of per-odb staging
* gc: only gc loal staging once
* Fixed diff summary to only show states with values.
* Removed redundant comma by using join.
* Swithced for loop to generator for better readability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* add: do not delete stage files before add
Because of the way we collect stages and cache them, we were not able to
collect them for the `add` without removing them from the workspace.
As doing so, we'd have two same/similar stages - one collected from the
workspace and the other just created from the `dvc add` in-memory.
This would raise errors during graph checks, so we started to delete them
and reset them (which is very recently, see #2886 and #3349).
By deleting the file before we even do any checks, we are making DVC
fragile, and results in data loss for the users with even simple mistakes.
This should make it more reliable and robust.
And, recently, we have started to keep state of a lot of things, that by
resetting them on each stage, we waste a lot of performance, especially
on gitignores. We cache the dulwich's IgnoreManager, which when resetted
too many times, will waste a lot of our time just collecting them again
next time (see #6227).
It's hard to say how much this improves, as this very much depends on
no. of gitignores in the repo (which can be assumed to be quite in number
for a dvc repo) and the amount of files that we are adding
(eg: `-R` adding a large directory).
On a directory with 10,000 files (in a datadet-registry repo),
creating stages on `dvc add -R` went from 64 files/sec to 1.1k files/sec.
* add tests
* make the test more specific
* separate validation and transformation of args from add func
* make tests pass for Python3.6
* cleanup
* use dict for attr-access and patch boundarguments.arguments
* Keep trailing slash when checking if directory is gitignored
* Calling is_ignored with correct type (could be Path-like)
* Fix test to comply to annotated types
* Fix separator issues
* Relaxed typing in is_ignored
* Added test
* Test scm
* Fix test on Windows
* Expanded test_ignored_dir_unignored_subdirs
* Even more assertions
* objects: add ExternalRepoFile object type
* RepoDependency: implement get_obj (use ExternalRepoFile)
* drop distinction between used external and used objects
* update tests
* remove unneeded fetch from ExternalRepoFile
* wrap erepo errors as an object error
* update cloud/remote functions for new used_objs behavior
* fix update() rev/locked behavior
* update import tests
* remove ExternalRepoFile
* move get_obj and get_used_objs implementations back into repo dependency
* return objects plus associated ODB (remote) in get_used_objs
* odb: add tmp_dir field and support creating Remote from an odb
* return dict mapping odb -> objects in get_used_objs
* use dummy git odb for git imports
* fetch: support pulling from specific odb
* update cloud funcs for get_used_objs usage
* update tests for used_objs behavior
* use fetch in dep.download
* catch circular imports from local fs repos
* add tests for chained and circular imports
* move tmp_dir into odb config
* update return docstring for repo.used_objs
* add: prevent crash while getting relpath
Since python 3.8 os.path.realpath has resolved windows network shares
to UNC paths. This breaks the logic for relpath as the UNC and mapped
drive paths don't have an obvious common prefix. As it is not easy to go
back to the mapped drive path from UNC path, this is resolved by always
using the UNC paths for calculating relative paths in windows.
Fixes#5832
* relpath: Update changes following review
Two changes:
- use pytest.mark.skipif rather than if statement inside test to make
it clearer that test is windows specific
- Use tail rather than upper_path to denote rest of path after drive
name taken off (to match python documentation)
Pull request: #5837
* tests, linter: update to add test case and apply linter though pre-commit
Co-authored-by: Tom Rutter <[email protected]>
* objects: make HashInfo & HashFile hashable
* stage/output: collect objects (HashFiles) instead of NamedCache
* remote/data_cloud: use objs instead of NamedCache
* repo: use objects instead of NamedCache
* rerun black after rebase
* remove NamedCache usage from run-cache
* drop NamedCache
* update HashInfo.__hash__
* update unit tests
* revert remote_name change
* start cleaning up test failures
* fix tree obj expansion
* update missing cache logging
* fix external dict handling
* update naive odb.gc
* update state save behavior for pull
* update test_data_cloud
* update test_remote
* fix gc with multiple repos
* objects.tree: support filtering w/preserved hash_info
* fix stage cache restore
* drop fs.path_info from HashFile.__hash__
* objects: add optional name attribute
* revert missing cache logging name/suffix changes
* revert name changes in remote tests
* review fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update dvc/remote/base.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* config, remote: Made S3 CA bundle customizable
botocore allows a path to a custom CA bundle either by passing a path
to the CA bundle file into the verify argument of
boto3.session.Session.client or passing None (the default) which will
fall back to the AWS config. Previously, the DVC config only accepted a
boolean into the ssl_verify option in the remote S3 config. This changes
the DVC config to accept both string and None in addition to boolean and
defaults to None. I also changed the default for ssl_verfiy to None in
BaseS3FileSystem. Thus, if ssl_verify is not provided, botocore will
fall back to the AWS config.
Testing
Unit tests to cover the changes to the config schema and addition
ssl_verify types that will be passed into S3FileSystem. Also, ran
dvc push -r object-store data/cifar-10-python.tar.gz
in my work environment that has a private S3 endpoint that requires a
custom CA bundle, both with and without ssl_verify specified in the
config. This was successful, showing that communication could be
established. And I ran
dvc remote modify object-store ssl_verify "$HOME/.aws/cabundle.pem"
and confirmed that the custom CA bundle path was added to the config.
Fixes#6012
* Removed default None on ssl_verify
Responding to PR comment, removed the Optional, default None on
ssl_verify since the config keys are optional by default. Rather
than a missing ssl_verify producing a None that eventually gets
filtered, it doesn't appear in the parsed config in the first place.
Co-authored-by: Robert Van Wesep <[email protected]>
astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.
Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).
Example errors that we were getting because of this:
```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
Pre-requisite for removing fs.path_info and switching to regular strings
as paths in filesystems.
With new filesystems we no longer need the zoo of different dependency
and output types, as most of them are now just handled by the underlying
filesystem. Though deps/outs are still due for a refactor in other
aspects.
Peter Rowlands (변기호)
Saugat Pachhai
Batuhan Taskaya
Ruslan Kuprieiev
Christian ClaussandSaugat Pachhai
Hadas Arikandpre-commit-ci[bot]
Miguel Méndez
Tomasz Pietruszka
yehiyam
Laurens Duijvesteijn
tomrutterandTom Rutter
rgvanwesepandRobert Van Wesep