Commit Graph
13 Commits
Author SHA1 Message Date
Batuhan Taskaya 6275f5d0d7 fsspec: create directories on upload/upload_fobj (#6309) 2021-07-14 15:42:51 +03:00
Batuhan Taskaya 87b49fd9f9 staging: optimize the number of info() calls (#6281)
* staging: optimize the number of info() calls

* fs: make info() give details about the object
2021-07-07 22:53:43 +03:00
Ruslan Kuprieiev 07f5f6b633 ssh: don't use path_info for connections (#6259) 2021-07-03 21:54:14 +03:00
Saugat Pachhai ce3a3ab100 pass size to upload_fobj (#6187)
This allows us to select between chunked and non-chunked uploading
in Webdav, and hopefully in the http itself.
2021-06-22 13:16:15 +03:00
Ruslan Kuprieiev 5fb181fcf8 odb: don't use fs.path_info (#6059)
* odb: don't use fs.path_info

* fs: get rid of fs.path_info
2021-06-02 20:43:51 +03:00
Saugat Pachhai a172067c7d fix errors raised by pylint (#6081)
astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.

Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).

Example errors that we were getting because of this:

```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
2021-05-31 05:46:18 +00:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032)
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
2021-05-18 15:51:15 +00:00
Ruslan Kuprieiev 89b40afee7 fs: make constructor closer to fsspec (#5911)
* fs: don't use repo for jobs/hash_jobs

* fs: local: exists: use fsspec's LocalFileSystem

* fs: make constructor comply with fsspec
2021-05-12 22:03:09 +03:00
b349b39d62 Decoupling dvcignore and fs (#5812)
* Decoupling dvcignore and fs

[x] remove dvcignore from fs objects
[x] attach dvcignore to repo
[x] add ignore check to functions(add, checkout, output, pipeline, etc)
[x] pass all tests
[x] final check

* Solve all of the arugments error.

* add dvcignore to dvc list

* Solve some test_ignore cases

* add dvcignore to checkout.

1. add dvcignore to 'checkout', 'add'
2. add ignore_filter to fs

* Solve two tests

* Get rid of ignore_filter

* Add ignore examine to download

* Solve a test issue

* Solve dvcignore and subrepo problems

* Solve sub repo error

* Revert some changes

* pass all tests

* Remove debug codes

* Some final exams

* dvcignore trie tree only update when needed

* reuse of the dir result in path walk.

* Some change after code review.

* Recover something from false `--force` push

* add abspath trans to dvcginore

* pass pylint

* pylint related

* Update dvc/fs/repo.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Changes related to reviews.

* dvcignore call api change

from dvcignore.walk_files(fs.walk(path_info)) to dvcignore.walk_files(fs, path_info)

* Get rid of PathInfo in test_ignore.py

* fs.walk follow the arguments of os.walk

* Dvcignore as parameters

* pass dvcignore as arguments to checkouts

* Decoupling state and repo

* Update dvc/dvcfile.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Update dvc/objects/stage.py

Co-authored-by: Paweł Redzyński <[email protected]>

* Solve some small problems

* Add tests to verify that dependency now not ignore

fix#5256

* Update dvc/ignore.py

* Update dvc/fs/git.py

* Update dvc/repo/__init__.py

* Some small improvements

* pass black

* Refactor is_ignored

* Update dvc/output/base.py

* repofs: use dvcignore in exists/isdir/isfile

Per https://github.com/iterative/dvc/pull/5812/files#r630604524

Co-authored-by: myname <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-05-12 05:29:04 +03:00
Paweł Redzyński eccc320606 remote: use configured jobs number (#5747) 2021-04-08 17:52:42 +03:00
Saugat Pachhai 0a2e9f98d5 dvc: update precommit hooks (#5716)
* update pre-commit hooks

* fix mypy errors

1. Casting `columns` to the custom Column with collapse.
2. Ignoring typedef difference for RichTable._collapse_widths.
3. Suppressing error with conditional paramiko imports
   https://github.com/python/mypy/issues/1153
2021-03-26 22:41:56 +05:45
Laurens Duijvesteijn e0a0342ab3 fs: ssh: pass timeout to paramiko's connect() (#5569)
Prior to this commit, DVC could hang indefinitely when connecting to
remote SSH filesystems. This should address this.
2021-03-08 22:59:26 +02:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00