Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-06 23:09:55 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,204 Commits 7 Branches 579 Tags
6b05cc5e24788975a3357504bc2b3ab9f3a7a941
Commit Graph
38 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 7f0537038c dvc: preserve desc on add/run/imp (#5003) 2020-12-01 11:52:31 +02:00
Ruslan Kuprieiev 7c45711e34 dvc: preserve comments/meta on add/imp/run (#5001) 2020-12-01 10:39:51 +02:00
Tomás Farías SantanaandRuslan Kuprieiev ca2e6a004c Support for WebHDFS (#4747) 
* Add support for WebHDFS

* Style and linting fixes

* Implement abstract methods and disable no-member pylint errors

* Add init unit test

* Yield client and fix redefinition of hdfs

* Get dynamic hdfs Docker port

* Add support for open in WebHDFSTree

* Remove hdfscli_config fallbacks in WebHDFSTree

* Use port 50070 for webhdfs

* WebHDFSTree supports http URL and proper walking

* Fix failing unit test after refactoring

* Add overwrite argument to upload

* Add remaining func tests for webhdfs

* Remove print statements

* Allow dynamic attributes in pylint for hdfs library

* Comment error on instantiating HDFS client from config

* Remove depth argument from call to walk

* Update dvc/tree/webhdfs.py

* Remove unnecessary deque

* Check exception messages hdfs client failure from config

* Return client directly

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Split long line

* Add size to HashInfo

* Implement move via hdfs.rename

* Add missing line

* Make parent directories before moving file in WebHDFS

* Add total to progress bar

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-11-17 20:02:02 +02:00
Ruslan Kuprieiev e1b82c5222 dvc: add size/nfiles for deps/outs (#4836) 
* dvc: add size for deps/outs

Related to #3256

* dvc: add nfiles for deps/outs

* dvc: put size/nfiles into the hash_info
 2020-11-04 19:22:57 +02:00
Ruslan Kuprieiev d7a35b9852 gha: enable hdfs tests (#4644) 2020-09-30 05:43:34 +03:00
Ruslan Kuprieiev 38dd1c9fe7 tests: allow failure for hdfs tests (#4600) 2020-09-23 00:53:32 +03:00
Paweł Redzyński b31bd1cb30 github actions tests: initial (#4469) 
* github actions tests: initial

* build: tests: add codecov action
 2020-09-22 23:05:06 +03:00
Ruslan Kuprieiev 61bf1194c3 tests: add gs etag 
Kudos to @skshetry
 2020-09-08 09:29:56 +03:00
Ruslan Kuprieiev 636a019289 dvc: compute dir hash without external cache (#4528) 
Fixes #4144
 2020-09-05 16:31:34 +03:00
Ruslan Kuprieiev 6c352eeaf0 hdfs: implement isdir and isfile (#4460) 
Fixes #4332
 2020-08-25 02:02:33 +03:00
Ruslan Kuprieiev 07da9f1e5c tests: remotes: use TmpDir-like fixtures (#4140) 2020-07-02 04:46:40 +03:00
karajan1001andRuslan Kuprieiev 74930c01ec add --no-exec mode to import url (#4075) 
* add --no-exec mode to import url

fix #3513

* Update tests/func/test_import_url.py

* Update tests/func/test_import_url.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-06-20 13:29:09 +03:00
Ruslan Kuprieiev 613ca0bdb5 get-url/import-url: save dependency before downloading (#4049) 
This allows us to properly process the errors like missing target.

Fixes #3897
 2020-06-16 13:05:21 +03:00
Ruslan Kuprieiev 91a7194e4f add/run/import-url: don't use -f as --file (#4004) 
If anything, `-f` is closer to `--force` than to `--file`. So let's set
this straight.
 2020-06-10 05:04:55 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792) 
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
 2020-05-14 01:46:42 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749) 
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
 2020-05-06 23:56:07 +03:00
Paweł Redzyński 5b737a0a0b imp_url: make default stage fname accompany import target (#3312) 2020-02-14 17:05:08 +02:00
Paweł Redzyński 48f8f2402a import-url: test: prevent in-repo testing 2020-01-23 12:40:21 +01:00
Fábio Santos fc92266114 address comments 2020-01-22 17:31:46 +00:00
Fábio Santos e41aa00c20 convert tests to pytest function tests 2020-01-22 14:04:52 +00:00
Fábio Santos bdf665a34a don't use patch.object() 2020-01-21 20:52:48 +00:00
Fábio Santos 5dfcda54f9 tests: replace spy() with pytest-mock spy 
Fixes #3198
 2020-01-21 14:47:49 +00:00
Ruslan Kuprieiev 095464d851 Merge pull request #3093 from iterative/move-fs-utilities 
Move fs utilities to fs.py from __init__.py
 2020-01-09 17:25:58 +02:00
Paweł RedzyńskiandRuslan Kuprieiev ad1450635b Update tests/func/test_import_url.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2020-01-09 10:00:02 +01:00
Aman Sharma eca5d2dde4 Move makedirs to fs.py 2020-01-09 04:30:51 +05:30
Paweł RedzyńskiandRuslan Kuprieiev b9e700105e Update tests/func/test_import_url.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2020-01-08 22:21:02 +01:00
Paweł RedzyńskiandRuslan Kuprieiev 7e12c7104d Update tests/func/test_import_url.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2020-01-08 22:20:50 +01:00
pawel 62d6f4dbdb import url: test: migrate to dir helpers 2020-01-08 10:30:33 +01:00
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev 0bfb3c0ab2 get/import/url: handle existing output directory (#2610) 
* get/import/url: handle existing output directory

Currently if you run

```
dvc get https://github.com/iterative/example-get-started model.pkl --out .
```

it will nuke your cwd. To fix that we need to properly handle existing
directories, same as usual unix cli utilities do that.

* travis: specify python version when installing from chocolatey

Fixes #2609
 2019-10-15 22:42:52 +03:00
Alexander Schepanovski db19957b8a Download refactor (#2275) 
* remote: drop not working http resume

Closes #2136

* remote: drop @transfer_context()

It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.

* remote: drop upload/download chunking
 2019-07-16 16:27:44 +03:00
Ruslan Kuprieiev e215364e6d test: http: use random port (#2269) 
Fixes #1878

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-15 13:18:10 +03:00
Ruslan Kuprieiev 94fe86afe8 tests: import-url: rewrite and move error msg test (#2262) 
This one was failing randomly.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-07-14 03:09:08 +03:00
Paweł Redzyński 217f2c4dbd imp_url: normalize path for default out name (#2248) 2019-07-12 09:20:47 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo 
Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-06-21 01:16:00 +03:00