Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,949 Commits 7 Branches 579 Tags
73f95ee6b4e3e451940fa2211816fac287af555c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruslan Kuprieiev 5e2fa3742b Merge pull request #2741 from algomaster99/remove-get-parent-dirs 
Remove `get_parent_dirs_up_to`
 2019-11-05 23:01:31 +02:00
Ruslan Kuprieiev 9145dcd6a8 Merge pull request #2740 from algomaster99/test-arg-types-symlink 
Ensure `contains_symlink_up_to` accept both str and Path-like objects
 2019-11-05 22:42:06 +02:00
Ruslan Kuprieiev a4c57bbb55 Merge pull request #2739 from mroutis/sftp-move-fix 
sftp: only remove src after an atomic copy during move
 2019-11-05 19:01:35 +02:00
Ruslan Kuprieiev 62c5e12d43 dvc: bump to 0.66.6 2019-11-05 03:20:24 +02:00
Ruslan Kuprieiev ee70822bd4 Merge pull request #2736 from efiop/master 
dvc: automatically reorder imports
 2019-11-05 03:02:32 +02:00
Ruslan Kuprieiev ba8360d4fd Merge pull request #2735 from efiop/win 
system: win: use win32file for _getdirinfo
 2019-11-05 02:19:23 +02:00
Ruslan Kuprieiev 197a802c61 Merge pull request #2734 from shcheklein/fix-get-npe 
fix NPE when we do get that does not provide progress callback
 2019-11-05 00:17:41 +02:00
Ruslan Kuprieiev 522579be1e dvc: bump to 0.66.5 2019-11-04 22:55:08 +02:00
Ruslan Kuprieiev a5d317eaa2 Merge pull request #2679 from n3hrox/enhance/conda_detection 
updater: detect conda
 2019-11-04 22:54:05 +02:00
Ruslan Kuprieiev 35507dfc61 Update .restyled.yaml 2019-11-04 20:31:20 +02:00
Ruslan Kuprieiev 8b4cc01110 add restyled.io 2019-11-04 20:20:22 +02:00
Ruslan Kuprieiev 3974862f87 github: add config for release-drafter bot 2019-11-04 19:13:39 +02:00
Ruslan Kuprieiev e0fe8a8f94 Merge pull request #2723 from shcheklein/fix-pr-template 
fix PR template to mention recommendatory nature of CC and DS
 2019-11-04 11:36:03 +02:00
Ruslan Kuprieiev 5227fc58d5 dvc: bump to 0.66.4 2019-11-04 00:38:10 +02:00
Ruslan Kuprieiev 2df7eda159 Merge pull request #2674 from n3hrox/enhance/footnote_deduplication 
logger: output footer once for multithreaded push/pull errors
 2019-11-03 23:37:13 +02:00
Ruslan Kuprieiev 46d221fade test: s3: try using boto3.client 2019-11-03 22:38:48 +02:00
Ruslan Kuprieiev 17512feb52 Merge pull request #2722 from kss682/2672_1 
Added package manager to dvc version output
 2019-11-03 21:56:01 +02:00
Ruslan Kuprieiev e75c52a170 setup: freeze awscli version 2019-11-03 15:09:29 +02:00
Ruslan Kuprieiev 3514cc5b5a test: freeze moto version 2019-11-03 14:49:36 +02:00
Ruslan Kuprieiev 0fece74d00 test: get-url: don't use get_aws_url 2019-11-03 14:18:25 +02:00
Ruslan Kuprieiev 1b54c7fa6a Merge pull request #2715 from ATR-oCiTy/master 
Fixed issue #2680 and Simple typo rectified
 2019-11-03 12:51:43 +02:00
Ruslan Kuprieiev 6400eba83f Merge pull request #2716 from eshaan-jeckyll/master 
typos corrected
 2019-11-03 12:48:55 +02:00
Ruslan Kuprieiev 6559bae0c9 Merge pull request #2717 from johntharian/master 
typo changed for existance->existence
 2019-11-03 12:48:50 +02:00
Ruslan Kuprieiev a65e5c37fb Merge pull request #2714 from Pandemic179/master 
Rectified simple typos
 2019-11-03 12:44:56 +02:00
Ruslan Kuprieiev c71e5562bb Merge pull request #2713 from GeorgeSabu-mec/master 
fix typo(correct spelling of existence)
 2019-11-03 12:44:03 +02:00
Ruslan Kuprieiev 601a77ed4b Merge pull request #2712 from the-thirteenth-star/master 
fixed typo
 2019-11-03 12:42:37 +02:00
Ruslan Kuprieiev 372e1070ac Merge pull request #2703 from kurianbenoy/master 
Remove multi-threaded prefix before progress bar(#2686)
 2019-11-03 00:34:19 +02:00
Ruslan Kuprieiev e21fc34c84 Merge pull request #2698 from algomaster99/test-argument-types-get_mtime_and_size 
Verify if `get_mtime_and_size` can take both Path objects and strings
 2019-11-03 00:25:47 +02:00
Ruslan Kuprieiev 2e6485f24b Merge pull request #2709 from mroutis/fix-2704 
sftp: make move across drives possible
 2019-11-02 14:07:46 +02:00
Ruslan Kuprieiev 91a7af03e7 Merge pull request #2707 from Suor/urlinfo 
perf: refactor URLInfo to not use ParseResult inside
 2019-11-02 13:49:49 +02:00
Ruslan Kuprieiev 809a85a340 dvc: bump to 0.66.3 2019-11-01 23:58:17 +02:00
Ruslan Kuprieiev ce8e019dd1 Merge pull request #2705 from shcheklein/fix-2702 
get-url: fix NPE when src non local and dst not absolute
 2019-11-01 23:54:53 +02:00
Ruslan Kuprieiev 7ecbdab564 Merge pull request #2693 from sadielbartholomew/i2685-pull-cmd-output 
pull, at import stage: replace warning with info report
 2019-11-01 23:15:23 +02:00
Ruslan Kuprieiev d932405ee1 Merge pull request #2695 from mroutis/test-s3-etag 
tests: preserve the same ETag across buckets during copy
 2019-10-31 01:55:17 +02:00
Ruslan Kuprieiev e623c4faf7 dvc: bump to 0.66.2 2019-10-30 17:20:05 +02:00
Ruslan Kuprieiev 877e38551e lock: add a workaround for a flufl.lock bug (#2688) 
This bug doesn't affect us during regular dvc CLI commands, but is not
visualy pleasing during API usage.

Fixes #2573
 2019-10-29 20:54:56 +02:00
Ruslan Kuprieiev b9c7d88fcc dvc: bump to 0.66.1 2019-10-26 23:36:13 +03:00
Ruslan Kuprieiev 64f78c164d Revert "Added path.lstrip for / in imports (#2651)" (#2675) 
This reverts commit bfc0a3596f.
 2019-10-26 23:35:22 +03:00
Ruslan Kuprieiev 1baf020de7 s3: don't use walk_files for list_cache_paths 
No need to convert back and forth from string to PathInfo
 2019-10-26 00:29:23 +03:00
Ruslan Kuprieiev 1ab9319682 gs: fix typo 2019-10-25 23:24:43 +03:00
Ruslan Kuprieiev 32ba773aee s3: don't forget to convert to regular path strings 2019-10-25 23:07:17 +03:00
Ruslan Kuprieiev fc485581f9 dvc: bump to 0.66.0 2019-10-25 22:46:50 +03:00
Ruslan Kuprieiev f3bacb7c28 gs: retry upload with smaller chunk size (#2661) 
Default chunk size for gs is 100M, which might be too much for
particular network. So if we are getting ConnectionError, we
should try lowering the chunk size until we reach the minimum
allowed chunk size of 256K. Also note that `chunk_size` must be a
multiple of 256K per the API specification.

Fixes #2572
 2019-10-25 22:45:54 +03:00
Ruslan Kuprieiev 35499faa7d readme: remove brew cask and update brew install cmd 2019-10-25 01:41:46 +03:00
Ruslan Kuprieiev 3667830c2a dvc: bump to 0.65.0 2019-10-24 23:41:43 +03:00
Ruslan Kuprieiev 8d9e4ca934 graph: use find_cycle instead of simple_cycle (#2668) 
We are incorrectly using `simple_cycle` here, `find_cycle` is much more
suitable for this application and much faster too. For example:

```
$ cat test_networkx.py
import networkx as nx
from dvc.repo.graph import check_acyclic

G = nx.scale_free_graph(1200)

list(nx.find_cycle(G, orientation='ignore'))

list(nx.simple_cycles(G))

$ python -mcProfile -scumtime test_networkx.py &> log.log
```

shows ~100s for `simple_cycles` and ~0s for `find_cycle`.

Fixes #2666
 2019-10-24 23:41:11 +03:00
Ruslan Kuprieiev 24ed639901 setup: add upper limit for tqdm version 
https://github.com/iterative/dvc/issues/2659#issuecomment-545940739
 2019-10-24 17:29:53 +03:00
Ruslan Kuprieiev b8d1ec204c dvc: bump to 0.64.1 2019-10-24 17:15:30 +03:00
Ruslan Kuprieiev 8d067c8805 setup: update project url 2019-10-24 15:19:04 +03:00
Ruslan Kuprieiev bd5a8a5cef dvc: bump to 0.64.0 2019-10-23 22:48:33 +03:00
Ruslan Kuprieiev 480a612da4 output: base: don't forget to return empty cache (#2656) 
* output: base: don't forget to return empty cache

CC @Suor

Fixes https://gist.github.com/leimbernon/21924e02a998e3f1604990f8a7824c37

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* gc: add test for missing .dir cache file
 2019-10-23 22:47:50 +03:00
Ruslan Kuprieiev 878998433f dvc: bump to 0.63.4 2019-10-22 01:26:02 +03:00
Ruslan Kuprieiev 763bfb4734 http: reuse requests.Session (#2646) 
This way we are able to properly utilize automatic connection pools and
not create new fds for each request, which overflows ulimit for max fds
very quickly on mac and windows. Kudos @pared for investigating 🎉

Fixes #2600

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-10-22 00:54:29 +03:00
Ruslan Kuprieiev 2274261808 progress: enable progress bar if it is a TTY 
Fixes #2645
 2019-10-21 23:35:47 +03:00
Ruslan Kuprieiev 3043ed2cb9 lock: skip using slow getaddrinfo (#2642) 
This is basically Lock.__init__ copy-paste, except that
instead of using `socket.getfqdn()` we use `socket.gethostname()`
to speed this up. We've seen [1] `getfqdn()` take ~5sec to return
anything, which is way too slow. `gethostname()` is actually a
fallback for `getfqdn()` when it is not able to resolve a
canonical hostname through network. The claimfile that uses
`self._hostname` is still usable, as it uses `pid` and random
number to generate the resulting lock file name, which is unique
enough for our application.

[1] https://github.com/iterative/dvc/issues/2582

Fixes #2582
 2019-10-21 19:20:08 +03:00
Ruslan Kuprieiev 59e62a7227 checkout: accept multiple targets (#2640) 
Make `Repo.checkout` accept a list of targets, instead of having to feed
it targets one-by-one. This allows us to properly handle checkout
errors.

Fixes #2580
 2019-10-21 17:06:21 +03:00
Ruslan Kuprieiev 638aaa254e dvc: bump to 0.63.3 2019-10-18 14:32:16 +03:00
Ruslan Kuprieiev 566d7b48d0 lock: monkeypatch socket.getfqdn used by flufl.lock (#2628) 
Workaround for slow and obsoleted gethostbyaddr used in
`socket.getfqdn()`. See [1], [2] and [3] for more info.

[1] https://bugs.python.org/issue5004
[2] https://github.com/iterative/dvc/issues/2582
[3] https://gitlab.com/warsaw/flufl.lock/merge_requests/12

Flufl.lock guys seem unresponsive, so we have to monkeypatch
this by ourselves, at least for now.

Fixes #2582
 2019-10-18 12:32:22 +03:00
Ruslan Kuprieiev 25ff08fdc8 dvc: bump to 0.63.2 2019-10-17 23:13:02 +03:00
Ruslan Kuprieiev ccd39ef6cf dvc: bump to 0.63.1 2019-10-17 21:35:48 +03:00
Ruslan Kuprieiev 1bc076904d travis: use ruby 2.4.3 for mac 2019-10-17 21:12:38 +03:00
Ruslan Kuprieiev 32a9194eab hdfs: don't set default user (#2622) 
https://opendatascience.slack.com/archives/CGGLZJ119/p1571253702008100?thread_ts=1571240388.006500&cid=CGGLZJ119
 2019-10-17 20:09:21 +03:00
Ruslan Kuprieiev e2b40d5764 dvc: bump to 0.63.0 2019-10-17 15:52:17 +03:00
Ruslan Kuprieiev e7c5ea6bb4 setup: use more flexible networkx reqs 
https://github.com/iterative/dvc/commit/50fc721811a015142b476e90f8d198a99e88892f
 2019-10-17 15:13:12 +03:00
Ruslan Kuprieiev 50fc721811 setup: fix networkx reqs 2019-10-17 14:22:28 +03:00
Ruslan Kuprieiev 4171aac029 pylint: disable bad-continuation error 
https://github.com/PyCQA/pylint/issues/289
 2019-10-16 18:50:36 +03:00
Ruslan Kuprieiev 04105036c2 pylint: disable missing-docstring error 2019-10-16 18:46:53 +03:00
Ruslan Kuprieiev 2df6555ef0 setup: update pyinstaller to 3.5 (#2615) 
As a part of the research for https://github.com/iterative/dvc/issues/2600
 2019-10-16 17:22:25 +03:00
Ruslan Kuprieiev 0bfb3c0ab2 get/import/url: handle existing output directory (#2610) 
* get/import/url: handle existing output directory

Currently if you run

```
dvc get https://github.com/iterative/example-get-started model.pkl --out .
```

it will nuke your cwd. To fix that we need to properly handle existing
directories, same as usual unix cli utilities do that.

* travis: specify python version when installing from chocolatey

Fixes #2609
 2019-10-15 22:42:52 +03:00
Ruslan Kuprieiev f4739b60ae Merge pull request #2607 from algomaster99/reduce-verbosity 
Reduce verbosity of `dvc add`
 2019-10-15 00:07:57 +03:00
Ruslan Kuprieiev 2491143d7a Merge pull request #2597 from efiop/2563 
remote: local: don't auto-create directory in `__init__`
 2019-10-14 23:34:21 +03:00
Ruslan Kuprieiev 2b6446a8d9 Merge pull request #2604 from Suor/reset-dvcignore 
dvc: reset dvcignore
 2019-10-13 18:07:43 +03:00
Ruslan Kuprieiev 5cabb6d115 Merge pull request #2595 from efiop/2563 
dvc: move `local/__init__.py` -> `local.py`
 2019-10-11 00:10:13 +03:00
Ruslan Kuprieiev 4535d626bb Merge pull request #2590 from efiop/state 
dvc: load state when taking repo lock in @locked
 2019-10-10 22:49:16 +03:00
Ruslan Kuprieiev 33bd3d63c5 dvc: bump to 0.62.1 2019-10-10 14:40:04 +03:00
Ruslan Kuprieiev f254004fae dvc: bump to 0.62.0 2019-10-09 11:47:52 +03:00
Ruslan Kuprieiev e34af30353 Merge pull request #2586 from efiop/2585 
scripts: don't include completion scripts into mac pkg
 2019-10-08 18:02:37 +03:00
Ruslan Kuprieiev 449f43f3cc requirements: require gitdb2 >= 2.0.6 
Fixes https://github.com/iterative/dvc/issues/1880
 2019-10-04 12:58:59 +03:00
Ruslan Kuprieiev 96eb7197c1 Merge pull request #2552 from mroutis/fix-2043 
git: escape filename before ignoring it
 2019-10-04 11:33:48 +03:00
Ruslan Kuprieiev f2cf5930aa Merge pull request #2571 from jimmy1134/master 
Update to Readme
 2019-10-04 11:31:23 +03:00
Ruslan Kuprieiev 092c498baa dvc: bump to 0.61.2 2019-10-03 22:53:33 +03:00
Ruslan Kuprieiev f7188f564e Merge pull request #2568 from efiop/2181 
deb/rpm/pkg: add zsh completion scripts
 2019-10-03 22:52:50 +03:00
Ruslan Kuprieiev 06f00b904a Merge pull request #2564 from nik123/master 
scripts: make completion scripts part of the debian package
 2019-10-03 21:50:51 +03:00
Ruslan Kuprieiev 8787ac0380 dvc: bump to 0.61.1 2019-10-02 10:17:22 +03:00
Ruslan Kuprieiev fdf0a8e99a Merge pull request #2562 from efiop/pyenv 
run: improve pyenv PATH modification criteria
 2019-10-02 10:16:52 +03:00
Ruslan Kuprieiev 8fb3eaaa08 dvc: bump to 0.61.0 2019-10-01 10:56:00 +03:00
Ruslan Kuprieiev 39104a1dee Merge pull request #2557 from efiop/2506 
run: make shell not load config files
 2019-10-01 07:59:07 +03:00
Ruslan Kuprieiev f0de362e0d Merge pull request #2548 from iterative/py27 
fix py27 compat
 2019-09-30 08:35:29 +03:00
Ruslan Kuprieiev beea7e56f2 Merge pull request #2543 from casperdcl/checksum 
remove unneeded TqdmThreadPoolExecutor
 2019-09-29 09:15:01 +03:00
Ruslan Kuprieiev e3450257c6 Merge pull request #2544 from yfarjoun/patch-1 
Added a space in error message
 2019-09-29 09:13:10 +03:00
Ruslan Kuprieiev 5c76b46b46 Merge pull request #2538 from mroutis/fix-2524 
py2: display friendly error when raising a UnicodeError
 2019-09-28 03:35:14 +03:00
Ruslan Kuprieiev 22afc48d9b Merge pull request #2540 from Suor/repo-imports 
perf: import dvc.repo dynamically
 2019-09-27 12:02:21 +03:00
Ruslan Kuprieiev 4061d4aff1 dvc: bump to 0.60.1 2019-09-26 18:34:06 +03:00
Ruslan Kuprieiev 3eb23870d2 build: posix: don't forget to install optional psutil 
Fixes https://github.com/iterative/dvc/issues/2537
 2019-09-26 18:33:36 +03:00
Ruslan Kuprieiev 381601f4ee dvc: cache graph/stages/pipelines (#2529) 
* dvc: cache graph/stages/pipelines

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: check_dag: don't reset the cache

https://github.com/iterative/dvc/pull/2529#discussion_r327711928

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: rename check_dag(stages) -> check_modified_graph(new_stages)

https://github.com/iterative/dvc/pull/2529#discussion_r327708439

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: refactor "active" logic and remove it from Repo class

* repo: use tree.setter to automate cache reset

As noted by @Suor's , this is much less prone to errors.

* repo: move graph-related staticmethods into graph.py

As per @Suor's request.

* dvc: remove unused from_directory

https://github.com/iterative/dvc/pull/2529#discussion_r328079258

* repo: use _tree instead of __tree

https://github.com/iterative/dvc/pull/2529#discussion_r328071115

* repo: add a note about cache invalidation on repo.lock release

https://github.com/iterative/dvc/pull/2529#discussion_r328071635

* repo: rename reset() -> _reset()

https://github.com/iterative/dvc/pull/2529#discussion_r328079899

* pipeline: use graph helpers

https://github.com/iterative/dvc/pull/2529#discussion_r327502293

* repo: use list comprehension for collect()

https://github.com/iterative/dvc/pull/2529#discussion_r328076651

* repo: graph: only use first cycle

https://github.com/iterative/dvc/pull/2529#discussion_r328081346

* dvc: don't use G.nodes()

https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
 2019-09-26 07:42:29 +03:00
Ruslan Kuprieiev fea05b3924 Merge pull request #2536 from mroutis/contributing 
move .github/CONTRIBUTING.md -> CONTRIBUTING.md
 2019-09-26 06:27:30 +03:00
Ruslan Kuprieiev 987cf883c5 dvc: bump to 0.60.0 2019-09-25 11:12:20 +03:00
Ruslan Kuprieiev dcce70f4b9 output: base: checkout: make returns consistent 
As reported by deep source: Either all return statements in a function should return an expression, or none of them should.
 2019-09-24 16:33:15 +03:00
Ruslan Kuprieiev 7409fadb84 Merge pull request #2527 from Suor/delay-cache 
perf: delay cache creation
 2019-09-24 08:11:45 +03:00
Ruslan Kuprieiev 33f1cdfab8 Merge pull request #2525 from efiop/2506 
dvc: remove PATH prefixes appended by pyenv
 2019-09-23 19:35:46 +03:00