Currently we use both absolute and relative (depend on cwd) paths, which is
terrible for an api, as their interpretation depends on external factors.
This PR makes dvcfs accept only paths relative to the repo root for regular
outputs and absolute paths for external outputs.
Pre-requisite for making repofs use same relative paths.
Introducing simple repo tree that combines all workspaces into one structure,
and allows us to detach data representation from pipelines.
Pre-requisite for dvc-data extraction and using relpaths in dvcfs/repofs.
metadata was introduced before our migration to fsspec and no longer needed
now. This is a pre-requisite to using data index in dvcfs/repofs.
Note that a bunch of awkward flags like isdvc/isout/etc are still kept, and
will be removed in followups.
Initial simple separation to get the process of extracting dvc-objects
and dvc-data packages out of dvc.
dvc-objects is for:
* defining base Object (aka HashFile) and ObjectDB
* providing tools to work with those objects (check/transfer/etc)
* etc
dvc-data is for:
* defining higher level objects (e.g. Tree)
* bulding objects from the workspace (aka staging)
* projecting objects back into the workspace (aka checkout)
* comparing Tree objects (aka diff)
* defining misc object databases (e.g LocalObjectDB)
* higher-level transfer (e.g. with Tree indexes)
* etc
For example, the objects we use in run-cache are based on dvc-objects
and not on dvc-data, as they have absolutely nothing to do with data
management.
There are still some mixups in the code that will be addressed in the
followups.
* test: fix test failure in case of different default branch
When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.
This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.
Set default git branch to main for test CI
Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.
Introduce isolation for git configuration
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
With introduction of https://github.com/iterative/dvc-plugins,
all important tests are being ran there, so there is no need
for dvc repo to run them again.
This PR also removes some redundant/broken remote tests.
Intermediate PR in removing path_info from our code base.
It is completely removed from the main codebase and is still
being used in tests, which will be migrated in the followups.
For fs-specific simple path manipulation, we've introduced
`fs.path` helper (similar to `os.path`).
Fixes#6774
* objects.status: use object IDs instead of naive objects
* objects.transfer: use object IDs instead of naive objects
* objects: add ReferenceHashFile
* odb: add ReferenceObjectDB
* repo: collect used object IDs instead of naive objs
* support basic/naive fs serialization
* objects.stage: stage file objects in addition to trees
* objects: remove save(), update transfer()
* output/dep: update for osave removal
* path_info: support both url and path attributes
* status/gc: use object IDs instead of naive objs
* diff: update staging usage
* commit: update staging usage
* tests: clean staging when needed
* tests: update for save/transfer/stage changes
* dvcfs: don't allow config serialization (repofs should be used instead)
* utils: return mtime as str, size as int
* tests: update unit tests for save/stage/transfer changes
* objects: use Repo.open when needed in ReferenceHashFile
* fix ls test staging
* objects.stage: don't skip hash check when loading from state
- individual hashes will still be loaded from state if they are still
valid
* ReferenceHashFile: serialize repo fs paths relative to repo root
* preserve state when staging
* objects.tree: skip hash computation in digest() when we have already
loaded hash from state
* always check file existence in referencehashfile
* update repofs test since stage no longer allows nonexistent files
* odb: only protect on check() if hash was verified
* staging: use unique memfs namespace for staging per dest ODB
* tests: clean full staging namespace
* refodb: preserve repofs cache dir
* repodependency: stage imports to local ODB and strict check circular
imports
* fetch: fetch staging last
* staging: make dest odb mandatory and preserve state
* refodb: cache nonlocal fs's
* fix rebase errors
* repofs: support full instantiation from fs config
* repofs: preserve subrepo factory for erepos
* tests: clarify repofs dirty file behavior
* config, remote: Made S3 CA bundle customizable
botocore allows a path to a custom CA bundle either by passing a path
to the CA bundle file into the verify argument of
boto3.session.Session.client or passing None (the default) which will
fall back to the AWS config. Previously, the DVC config only accepted a
boolean into the ssl_verify option in the remote S3 config. This changes
the DVC config to accept both string and None in addition to boolean and
defaults to None. I also changed the default for ssl_verfiy to None in
BaseS3FileSystem. Thus, if ssl_verify is not provided, botocore will
fall back to the AWS config.
Testing
Unit tests to cover the changes to the config schema and addition
ssl_verify types that will be passed into S3FileSystem. Also, ran
dvc push -r object-store data/cifar-10-python.tar.gz
in my work environment that has a private S3 endpoint that requires a
custom CA bundle, both with and without ssl_verify specified in the
config. This was successful, showing that communication could be
established. And I ran
dvc remote modify object-store ssl_verify "$HOME/.aws/cabundle.pem"
and confirmed that the custom CA bundle path was added to the config.
Fixes#6012
* Removed default None on ssl_verify
Responding to PR comment, removed the Optional, default None on
ssl_verify since the config keys are optional by default. Rather
than a missing ssl_verify producing a None that eventually gets
filtered, it doesn't appear in the parsed config in the first place.
Co-authored-by: Robert Van Wesep <[email protected]>
Pre-requisite for removing fs.path_info and switching to regular strings
as paths in filesystems.
With new filesystems we no longer need the zoo of different dependency
and output types, as most of them are now just handled by the underlying
filesystem. Though deps/outs are still due for a refactor in other
aspects.
* Decoupling dvcignore and fs
[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check
* Solve all of the arugments error.
* add dvcignore to dvc list
* Solve some test_ignore cases
* add dvcignore to checkout.
1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs
* Solve two tests
* Get rid of ignore_filter
* Add ignore examine to download
* Solve a test issue
* Solve dvcignore and subrepo problems
* Solve sub repo error
* Revert some changes
* pass all tests
* Remove debug codes
* Some final exams
* dvcignore trie tree only update when needed
* reuse of the dir result in path walk.
* Some change after code review.
* Recover something from false `--force` push
* add abspath trans to dvcginore
* pass pylint
* pylint related
* Update dvc/fs/repo.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Changes related to reviews.
* dvcignore call api change
from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)
* Get rid of PathInfo in test_ignore.py
* fs.walk follow the arguments of os.walk
* Dvcignore as parameters
* pass dvcignore as arguments to checkouts
* Decoupling state and repo
* Update dvc/dvcfile.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Update dvc/objects/stage.py
Co-authored-by: Paweł Redzyński <[email protected]>
* Solve some small problems
* Add tests to verify that dependency now not ignore
fix#5256
* Update dvc/ignore.py
* Update dvc/fs/git.py
* Update dvc/repo/__init__.py
* Some small improvements
* pass black
* Refactor is_ignored
* Update dvc/output/base.py
* repofs: use dvcignore in exists/isdir/isfile
Per https://github.com/iterative/dvc/pull/5812/files#r630604524
Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* azure: use a custom event loop for authentication
* add test_temp_event_loop
* wrap open() too!
* ignore aiohttp warnings for normal level use-cases
* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
* Added SSL verification option
* Changed param name to avoid conflicts
* verify_ssl now accepts strings
* Added simple unit tests
* Changed parameter name fron verify_ssl to ssl_verify so it's consistent with HTTP remotes
Forget to change 1 test
* add processing on ssl_verify param
typo
* revert back to bool param
* Update dvc/config_schema.py
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Nigel <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dvc: get rid of DirInfo
DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.
As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.
* objects: move Tree into tree.py
* state: only load when needed
* state: don't use separate load/dump
* checkout: group link removal logic together
* state: use diskcache.Cache
* staging: use pbar and save to state
* state: don't pass the whole repo to __init__
* staging: explicitly pass state object
* fs: get rid of odb property
* state: don't close the connection to cache
* repo: close: close state
Ruslan Kuprieiev
Jon Burdo
Amrit GhimireandSaugat Pachhai
Batuhan Taskaya
Nikita Kodenko
Peter Rowlands (변기호)
Batuhan Taskaya
rgvanwesepandRobert Van Wesep
