objects: use separate staging ODB for staging trees (#6195)

* fs: use fsspecwrapper in memfs

* odb: add memfs staging ODB in ODBManager

* odb: add read_only attribute for odb instances

* objects: handle memfs paths when staging trees

* odb: remove git ODB

* output: use staging ODB

* RepoDependency: use staging odb for imports

* dvcfs: handle case where dir cache is unavailable

* fix circular import check

* odb: use temp local ODB for staging

* objects: use staging ODB in stage() and tree load

* objects: add typing for add/check/load/stage

* output/dep: use new staging

* add unit tests for local odb staging

* localfs: account for string paths in move()

* include local in odb.by_scheme

* tests: count dir cache like any other object in test_pull_git_imports

* odb: clear staging on gc()

* objects: add tree.get/filter/insert

* output: use obj.filter/get for granular dir commit

* squash filter logic output

* dependency: don't filter imports at the dep level

* don't init staging ODB for external cache

* use staging ODB when computing tree hash for external outs

* remove unneeded force in test granular commit

* use unique-per-manager memfs URLs for staging

* odb: make staging per-ODBManager

* objects: move state dependent staging into ODBManager

* objects: support checking multiple odbs

- staging dependent checks can be done via ODBManager.check

* checkout: do state lookup before explicit stage() in _changed()

* output: use odbmanager based staging

* RepoDependency: use odbmanager based staging

* update obj staging unit tests

* checkout: remove check(), handle individual file exceptions on link()

* diff: use odbmanager based staging

* objects: add ObjectPermissionError for read/write perm errors

* odb: move staging out of odbmanager

* objects: handle staging ODB in objects.stage()

* objects.tree: don't stage anything in tree methods

* odb: clear staged objects on gc()

* remote: don't filter objs by scheme, objs are already grouped by ODB

* diff: update objects.stage usage

* output/dep: update stage() usage

* update ODB tests

* update func tests

* remove memfs from FS_MAP

* revert unneeded objects.check() changes

* remove unused tree.insert()

* revert unused odbmanager changes

* add separate odb error classes

* stage external outs directly in ODB

* check for obj existence in main ODB before staging

* use .dvc/tmp/staging instead of per-odb staging

* gc: only gc loal staging once

tests/func/test_commit.py

@@ -123,6 +123,8 @@ def test_commit_granular_output_dir(tmp_dir, dvc):
 
 
 def test_commit_granular_dir(tmp_dir, dvc):
+    from dvc.objects.stage import _STAGING_DIR
+
     tmp_dir.gen(
         {
             "data": {
@@ -135,8 +137,11 @@ def test_commit_granular_dir(tmp_dir, dvc):
     dvc.add("data", no_commit=True)
 
     cache = tmp_dir / ".dvc" / "cache"
-    assert set(cache.glob("*/*")) == {
-        cache / "1a" / "ca2c799df82929bbdd976557975546.dir"
+    staging = tmp_dir / ".dvc" / "tmp" / _STAGING_DIR
+
+    assert set(cache.glob("*/*")) == set()
+    assert set(staging.glob("*/*")) == {
+        staging / "1a" / "ca2c799df82929bbdd976557975546.dir"
     }
 
     dvc.commit(os.path.join("data", "foo"))
@@ -144,15 +149,26 @@ def test_commit_granular_dir(tmp_dir, dvc):
         cache / "1a" / "ca2c799df82929bbdd976557975546.dir",
         cache / "ac" / "bd18db4cc2f85cedef654fccc4a4d8",
     }
+    assert set(staging.glob("*/*")) == set()
 
     dvc.commit(os.path.join("data", "subdir"))
     assert set(cache.glob("*/*")) == {
         cache / "1a" / "ca2c799df82929bbdd976557975546.dir",
-        cache / "26" / "d6b64d96a660707412f523e8184b5f.dir",
         cache / "ac" / "bd18db4cc2f85cedef654fccc4a4d8",
         cache / "4c" / "e8d2a2cf314a52fa7f315ca37ca445",
         cache / "68" / "dde2c3c4e7953c2290f176bbdc9a54",
     }
+    assert set(staging.glob("*/*")) == set()
+
+    dvc.commit(os.path.join("data"))
+    assert set(cache.glob("*/*")) == {
+        cache / "1a" / "ca2c799df82929bbdd976557975546.dir",
+        cache / "ac" / "bd18db4cc2f85cedef654fccc4a4d8",
+        cache / "4c" / "e8d2a2cf314a52fa7f315ca37ca445",
+        cache / "68" / "dde2c3c4e7953c2290f176bbdc9a54",
+        cache / "37" / "b51d194a7513e45b56f6524f2d51f2",
+    }
+    assert set(staging.glob("*/*")) == set()
 
 
 def test_commit_no_exec_missing_dep(tmp_dir, dvc):