Commit Graph
11 Commits
Author SHA1 Message Date
karajan1001andkarajan1001 67896aa45d Optimize dvcignore (#4242)
* Update

1. unit test update
2. two might cause bugs

* dvc ignore

* Some refactoring

* clean

* Config tree use

* Solve sub dir repo problem. Add more tests

* typo error

* Change request

Co-authored-by: karajan1001 <[email protected]>
2020-07-23 00:01:09 +03:00
58f65345ff Optimize ignore performance (#4120)
* Remove Duplicate ignore Match

* Continue Optimize Dvcignore

fix #3869

* Add a new test with multi ignore files

* Solve merging two dvc files

* Solve Code Climate

* For Windows

* Complete addition of patterns. Add one test

* Systematic test

* Change request

* Change request

* Seperate path sepcification math

* Rename and add comment

* rename change_dirname to private

* Update dvc/pathspec_math.py

list comprehension

Co-authored-by: Alexander Schepanovski <[email protected]>

* Change request

* Update dvc/ignore.py

Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Alexander Schepanovski <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-07-15 02:28:18 +03:00
karajan1001andkarajan1001 253a0eae13 Add more tests according to gitignore (#4166)
* Add more tests according to .gitignore

* Test update

* Update test

* Solve question of path

* update one test

* Update 2 tests

* Add one tests

* comment those failed tests caused by other package

* parametrize some tests

Co-authored-by: karajan1001 <[email protected]>
2020-07-07 23:04:52 +03:00
karajan1001 a59f90fac9 Performance improvement of dvcignore (#3967)
fix#3869
1.Use big regex.

* Solve windows

* add rule order test

* Solve ignore order

* remove list comprehensions
2020-06-15 11:37:01 +02:00
Ruslan Kuprieiev d77af8196e optimize local status (#3867)
* remote: remove unpacked dir

We've done a lot of optimizations lately, which made unpacked dir trick
obsoleted.

* local: save us a stat call in is_protected

* local: speedup checksum_to_path ~x5.5

* tree: walk: don't use os.path.join

It is ~5.5 times slower than joining by hand.

* dvcignore: optimize matching x10

On a repo with a dvcignore with 1 pattern and a directory with 400K
files, `dvc status` now takes ~8 sec instead of ~30 sec.

To achieve that, we make some assumptions about the paths formats that
we are dealing with, so we could use simpler logic instead of using
very slow `relpath`, `abspath` etc on every entry in a directory.

It is also clear that CleanTree behavior is inconsistent (even tests
expect very different outputs from it), so we will need to look into
this later.
2020-05-25 03:20:11 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
pawel f14aca630f ignore: make DvcIgnoreFilter use tree 2019-11-07 15:37:24 +01:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Paweł RedzyńskiandRuslan Kuprieiev 393d926b74 state: use dvcignore when accessing metadata (#2161)
* state: use ignore file handler when accessing metadata

* fs: don't add dir size to overall size for dir

* ignore: use posix-like path on windows

* ignore: remove tree from dvc_walk arguments

* ignore: handle DvcIgnoreFilter creation outside of DVC repo

* ignore: throw on dvcignore file under output dir, but not dependency dir

* ignore: move ignore initialization to Repo init

* ignore: work in progress

* ignore: work in progress 2

* ignore: throw on .dvcignore upon collecting dir

* ignore: refactor

* ignore: loading upper level ignores in stages()

* Update dvc/repo/__init__.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* dvcignore review refactor
2019-07-12 11:50:22 +03:00
pared c9b3bdc927 ignore: remove dulwich in favour of pathspec 2019-06-20 12:15:42 +02:00
pared c467ccebcb ignore: introduce .dvcignore, apply when searching for stage files 2019-04-15 14:31:34 +02:00