Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,727 Commits 7 Branches 579 Tags
80cb1c919a2c627c18728ed4c35bc95a1eb312c7
Commit Graph
3727 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Mr. Outis 80cb1c919a git: escape question mark character 2019-09-30 11:27:53 -05:00
Mr. Outis a9201f8237 git: add a note referencing gitignore pattern 2019-09-30 11:21:39 -05:00
Mr. Outis c9c36c2049 git: escape filename before ignoring it 
Fix #2043
 2019-09-29 23:16:46 -05:00
Ruslan Kuprieiev 4061d4aff1 dvc: bump to 0.60.1 0.60.1 2019-09-26 18:34:06 +03:00
Ruslan Kuprieiev 3eb23870d2 build: posix: don't forget to install optional psutil 
Fixes https://github.com/iterative/dvc/issues/2537
 2019-09-26 18:33:36 +03:00
Ruslan Kuprieiev 381601f4ee dvc: cache graph/stages/pipelines (#2529) 
* dvc: cache graph/stages/pipelines

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: check_dag: don't reset the cache

https://github.com/iterative/dvc/pull/2529#discussion_r327711928

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: rename check_dag(stages) -> check_modified_graph(new_stages)

https://github.com/iterative/dvc/pull/2529#discussion_r327708439

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: refactor "active" logic and remove it from Repo class

* repo: use tree.setter to automate cache reset

As noted by @Suor's , this is much less prone to errors.

* repo: move graph-related staticmethods into graph.py

As per @Suor's request.

* dvc: remove unused from_directory

https://github.com/iterative/dvc/pull/2529#discussion_r328079258

* repo: use _tree instead of __tree

https://github.com/iterative/dvc/pull/2529#discussion_r328071115

* repo: add a note about cache invalidation on repo.lock release

https://github.com/iterative/dvc/pull/2529#discussion_r328071635

* repo: rename reset() -> _reset()

https://github.com/iterative/dvc/pull/2529#discussion_r328079899

* pipeline: use graph helpers

https://github.com/iterative/dvc/pull/2529#discussion_r327502293

* repo: use list comprehension for collect()

https://github.com/iterative/dvc/pull/2529#discussion_r328076651

* repo: graph: only use first cycle

https://github.com/iterative/dvc/pull/2529#discussion_r328081346

* dvc: don't use G.nodes()

https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
 2019-09-26 07:42:29 +03:00
Ruslan Kuprieiev fea05b3924 Merge pull request #2536 from mroutis/contributing 
move .github/CONTRIBUTING.md -> CONTRIBUTING.md
 2019-09-26 06:27:30 +03:00
Mr. Outis 5c5dfed68f .github/CONTRIBUTING.md -> CONTRIBUTING.md 2019-09-25 21:33:46 -05:00
Ruslan Kuprieiev 987cf883c5 dvc: bump to 0.60.0 0.60.0 2019-09-25 11:12:20 +03:00
Ruslan Kuprieiev dcce70f4b9 output: base: checkout: make returns consistent 
As reported by deep source: Either all return statements in a function should return an expression, or none of them should.
 2019-09-24 16:33:15 +03:00
Paweł Redzyński be012e70cf checkout: raise on non existing dir cache (#2514) 
* checkout: raise on non existing dir cache

* checkout: don't throw on failed load_dir_cache

* checkout: raise on failed checkouts, only after all checkouts are perfromed

* remote: base: raise on failed dir cache loading

* rename exception for failed checkout
 2019-09-24 16:31:36 +03:00
Ruslan Kuprieiev 7409fadb84 Merge pull request #2527 from Suor/delay-cache 
perf: delay cache creation
 2019-09-24 08:11:45 +03:00
Alexander Schepanovski e2eb4ae0c3 perf: delay cache creation 
The older code instantiates non-local cache remotes on Repo creation if
any of those are configured. And remote instantiation loads their API
libs.
 2019-09-24 09:03:03 +07:00
Ruslan Kuprieiev 33f1cdfab8 Merge pull request #2525 from efiop/2506 
dvc: remove PATH prefixes appended by pyenv
 2019-09-23 19:35:46 +03:00
Ruslan Kuprieiev 313fdce5fa Merge pull request #2528 from Suor/datacloud-dead-code 
dvc: remove dead code in DataCloud
 2019-09-23 16:26:44 +03:00
Alexander Schepanovski 4006c6434e dvc: remove dead code in DataCloud 2019-09-23 20:00:19 +07:00
Ruslan Kuprieiev bd2b5900a1 Merge pull request #2526 from efiop/2495 
dvc: remote: optimize imports
 2019-09-23 12:44:09 +03:00
Ruslan Kuprieiev fd14ca3269 dvc: remote: optimize imports 
Decreases `time dvc --version` from ~1.4 sec to ~0.3 sec.

Related #2495
Related #2445

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-09-23 11:30:45 +03:00
Ruslan Kuprieiev 43975c8618 dvc: remove PATH prefixes appended by pyenv 
Unlike PyInstaller, pyenv doesn't leave backups of original env vars
when it modifies them. If we look into the shim, pyenv and pyenv-exec,
we can figure out that the PATH is modified like this:

    PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH

where

    PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
    bin_path - might not start with $PYENV_ROOT as it runs realpath on
        it, see pyenv source code.
    plugin_bin - might contain more than 1 entry, which start with
        $PYENV_ROOT, see pyenv source code.

So having this, we can make a rightful assumption about what parts of the PATH
we need to remove in order to get the original PATH.

Fixes #2506
 2019-09-23 04:25:42 +03:00
Ruslan Kuprieiev 70efa6c52a Merge pull request #2523 from efiop/daemon 
daemon: use `sys.argv[0]` instead of `-m dvc`
 2019-09-22 05:42:56 +03:00
Ruslan Kuprieiev 0347cfb891 gitignore: don't forget to add /tmp 2019-09-22 05:22:55 +03:00
Ruslan Kuprieiev 31b40816b5 daemon: use sys.argv[0] instead of -m dvc 
This way dvc will actually launch the same script that it is running
from and not installed dvc module, that might not match.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-09-22 05:18:51 +03:00
Mr. OutisandRuslan Kuprieiev ae90904fb3 pull: download data from imported stages (#2499) 
* wip - pull is working

* use fetch instead of reproduce

* test pull and fetch

* style changes

* fetch: catch config error on fetch instead of get_remote

* output: remove unused guard clause

* fetch: define a more specific error instead of ConfigError

* fetch: display number of errors correctly

* exceptions: remove "Failed" prefix to download/upload

* fetch: catch other exceptions that might halt the download

* fetch: log catched exceptions

* fetch: reformat exception

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* fix formatting
 2019-09-22 01:28:12 +03:00
Ruslan Kuprieiev 23fc2662d0 Merge pull request #2519 from efiop/755 
dvc: move repo lock from CLI to API
 2019-09-21 19:12:34 +03:00
Ruslan Kuprieiev 9d6ca1ef6c dvc: move repo lock from CLI to API 
For the longest time we've had `repo.lock`, but were only using it from
CLI when running specific command, instead of simply properly taking it
on API operations. This patch takes care of that.
 2019-09-21 05:10:06 +03:00
Ruslan Kuprieiev 64c3973522 Merge pull request #2517 from Suor/later-imports 
perf: import some things later
 2019-09-20 18:00:01 +03:00
Ruslan Kuprieiev f1d3f4344e Merge pull request #2494 from Suor/lazy 
perf: do not use pkg_resources
 2019-09-20 13:47:47 +03:00
Alexander Schepanovski cb5ff6e085 perf: import some things later 
Saves on startup time
 2019-09-20 16:45:02 +07:00
Paweł RedzyńskiandRuslan Kuprieiev d16178ab61 checkout: raise error on failed dir cache loading (#2503) 
* remote: base: load_dir_cache: raise exception on failed download

* output: base: raise error on failed get_files_number

* remote: base: log failed download as warning

* Update dvc/output/base.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* remote: base: dont download dir cache on load_dir_cache
 2019-09-19 16:35:37 +03:00
Paweł RedzyńskiandRuslan Kuprieiev 349b4af3ae install: verify no hooks colliding with dvc exist (#2505) 
* install: verify no hooks colliding with dvc exist

* test: install: remove obsolete tests

* Update tests/func/test_install.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* test: install: import GitHookAlreadyExistsError
 2019-09-19 15:40:58 +03:00
Ruslan Kuprieiev 67e31eaa2f Merge pull request #2504 from Suor/http-compat 
perf: do not import http.server each run
 2019-09-18 22:27:52 +03:00
Alexander Schepanovski ffb52b256b perf: do not import http.server each run 
It is only used in tests.
 2019-09-17 18:41:05 +07:00
Alexander Schepanovski eac0aa13b2 perf: use packaging.version instead of pkg_resources 
This is the same functionality without a cost of loading a huge package.
 2019-09-17 14:32:31 +07:00
Ruslan Kuprieiev e36f242f66 Merge pull request #2493 from pared/2345 
output: prevent stage file from being used as output
 2019-09-14 21:42:06 +03:00
Paweł RedzyńskiandRuslan Kuprieiev 53f6aead95 Update dvc/dependency/base.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-09-14 19:40:59 +02:00
Paweł RedzyńskiandRuslan Kuprieiev 74a0cef3e5 Update dvc/output/base.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-09-13 16:02:49 +02:00
pawel 0935930f46 output: prevent stage file from being used as output 2019-09-13 13:31:50 +02:00
Ruslan Kuprieiev 6d16227bc1 Merge pull request #2496 from efiop/2483 
path_info: don't forget to override as_posix method
 2019-09-13 05:12:39 +03:00
Ruslan Kuprieiev f0c43b4e02 path_info: don't forget to override as_posix method 
Unlike original implementation [1] that uses `str()` we actually need
to use `fspath`, because we've overriden `__str__` method to return
relative paths, which will break original `as_posix`.

[1] https://github.com/python/cpython/blob/v3.7.0/Lib/pathlib.py#L692

Fixes #2483

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-09-13 04:52:03 +03:00
Alexander Schepanovski 293a50d75c perf: use fast entry points on non-wheel install 
This avoids importing `pkg_resources` on startup, which saves around 200
ms.

NOTE: wheel installer uses its own entry points template, which is the same
fastentrypoints.py script provides, so no gain there.
 2019-09-12 22:33:59 +07:00
Paweł RedzyńskiandRuslan Kuprieiev 69f0de0090 google-resumable-media build problems (#2449) 
* google-resumable-media hook for pyinstaller

* working wtih g-r-m build with -e

* update google-cloud-storage

* google-cloud-storage hook: remove hiddenimports

* Update setup.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-09-12 12:12:37 +03:00
Ruslan Kuprieiev da6a0ca933 lock: remove unused lifetime argument 2019-09-11 16:45:09 +03:00
Ruslan Kuprieiev 05adcf9a4f Merge pull request #2461 from Suor/nfs 
dvc: switch to flufl lock
 2019-09-11 16:43:40 +03:00
Ruslan Kuprieiev 8338d4507c Merge pull request #2488 from pared/2444 
remote: checkout: reuse _checkout_file functionality
 2019-09-11 15:22:18 +03:00
pawel d4c9c71a2f remote: checkout: reuse _checkout_file functionality 2019-09-11 11:26:30 +02:00
Alexander Schepanovski 3c3378bdca dvc: make lock lifetime long by default 2019-09-11 14:21:12 +07:00
Alexander Schepanovski d9d85f3cb0 dvc: fix locks in Python2.7/Windows 
Return to zc.lockfile lock in Python 2, NFS is not supported there
anyway since sqlite3 doesn't support URIs.
 2019-09-11 14:17:21 +07:00
Alexander Schepanovski f29f07ca88 dvc: do not lock sqlite db 2019-09-11 14:17:16 +07:00
Ruslan Kuprieiev 09f6b1c7b9 Merge pull request #2485 from pared/2477 
move: use relpath to find target path
 2019-09-10 16:23:27 +03:00
pawel a16dc58ae3 move: use relpath to find target path 2019-09-10 14:03:55 +02:00