Commit Graph
25 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 3e4e143bee deps: update dvc-data to >=3 2023-12-15 08:02:40 +02:00
Ruslan Kuprieiev c827654fe7 tests: use mocker instead of unittest.mock where possible
Had the "pleasure" of finding a mocker.patch leak into other tests and one
of the suspicions is that mixing unittest.mock and mocker might be somehow
related.

In any case, we should use mocker where possible anyway.
2023-06-18 17:10:45 +03:00
Gao 4459d6ac57 Add back an dvc ignore test
Original from
https://github.com/iterative/dvc/pull/8767#discussion_r1070778749
2023-03-08 12:03:07 +02:00
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
karajan1001 fc1a192c90 Bump pathspec to 0.10.3
fix: #8217
1. Replace group name to avoid conflict.
2. Disable failed tests.
3. Bump pathspec to 0.10.3
2023-01-13 22:47:36 +08:00
skshetry 5441964fdb add ruff (#8793)
* add ruff linter

* enable few more checks
2023-01-11 15:20:38 +05:45
Saugat Pachhai (सौगात) 570e3a5359 dvc.fs: move fs to dvc.objects.fs
Some of the APIs/FileSystems are reimported inside dvc.fs module.
2022-05-16 13:15:19 +03:00
Ruslan Kuprieiev a80a85e6bd dvc: detach gitfs from local paths
Our gitfs paths are currently tied to the local path in which the cloned
repository resides in, which results in a lot of hacky assumptions throughout
the code.

This PR detaches gitfs from local paths completely, making dvc work with a
proper fsspec gitfs implementation.

Part of #7543
2022-05-11 18:17:39 +03:00
Ruslan Kuprieiev fd440df5b3 dvcignore: use fs.path instead of os.path 2022-04-20 12:27:26 +03:00
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
Ruslan Kuprieiev a5b080645c dvcignore: refactor from_file(s) (#6002) 2021-05-13 12:23:58 +02:00
Ruslan Kuprieiev 6d941e6cb5 fs: get rid of fs_root (#6000)
* dvcignore: don't use fs_root

* fs: get rid of fs_root
2021-05-13 04:15:56 +03:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
karajan1001andkarajan1001 67896aa45d Optimize dvcignore (#4242)
* Update

1. unit test update
2. two might cause bugs

* dvc ignore

* Some refactoring

* clean

* Config tree use

* Solve sub dir repo problem. Add more tests

* typo error

* Change request

Co-authored-by: karajan1001 <[email protected]>
2020-07-23 00:01:09 +03:00
58f65345ff Optimize ignore performance (#4120)
* Remove Duplicate ignore Match

* Continue Optimize Dvcignore

fix #3869

* Add a new test with multi ignore files

* Solve merging two dvc files

* Solve Code Climate

* For Windows

* Complete addition of patterns. Add one test

* Systematic test

* Change request

* Change request

* Seperate path sepcification math

* Rename and add comment

* rename change_dirname to private

* Update dvc/pathspec_math.py

list comprehension

Co-authored-by: Alexander Schepanovski <[email protected]>

* Change request

* Update dvc/ignore.py

Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Alexander Schepanovski <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-07-15 02:28:18 +03:00
karajan1001andkarajan1001 253a0eae13 Add more tests according to gitignore (#4166)
* Add more tests according to .gitignore

* Test update

* Update test

* Solve question of path

* update one test

* Update 2 tests

* Add one tests

* comment those failed tests caused by other package

* parametrize some tests

Co-authored-by: karajan1001 <[email protected]>
2020-07-07 23:04:52 +03:00
karajan1001 a59f90fac9 Performance improvement of dvcignore (#3967)
fix#3869
1.Use big regex.

* Solve windows

* add rule order test

* Solve ignore order

* remove list comprehensions
2020-06-15 11:37:01 +02:00
Ruslan Kuprieiev d77af8196e optimize local status (#3867)
* remote: remove unpacked dir

We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.

* local: save us a stat call in is_protected

* local: speedup checksum_to_path ~x5.5

* tree: walk: don't use os.path.join

It is ~5.5 times slower than joining by hand.

* dvcignore: optimize matching x10

On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.

To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.

It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
2020-05-25 03:20:11 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
pawel f14aca630f ignore: make DvcIgnoreFilter use tree 2019-11-07 15:37:24 +01:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Paweł RedzyńskiandRuslan Kuprieiev 393d926b74 state: use dvcignore when accessing metadata (#2161)
* state: use ignore file handler when accessing metadata

* fs: don't add dir size to overall size for dir

* ignore: use posix-like path on windows

* ignore: remove tree from dvc_walk arguments

* ignore: handle DvcIgnoreFilter creation outside of DVC repo

* ignore: throw on dvcignore file under output dir, but not dependency dir

* ignore: move ignore initialization to Repo init

* ignore: work in progress

* ignore: work in progress 2

* ignore: throw on .dvcignore upon collecting dir

* ignore: refactor

* ignore: loading upper level ignores in stages()

* Update dvc/repo/__init__.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* dvcignore review refactor
2019-07-12 11:50:22 +03:00
pared c9b3bdc927 ignore: remove dulwich in favour of pathspec 2019-06-20 12:15:42 +02:00
pared c467ccebcb ignore: introduce .dvcignore, apply when searching for stage files 2019-04-15 14:31:34 +02:00