Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-06 15:28:28 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,744 Commits 7 Branches 579 Tags
8fb3eaaa08fc8bc50dcbe357c482bd7f62c2e67a
Commit Graph
917 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 381601f4ee dvc: cache graph/stages/pipelines (#2529) 
* dvc: cache graph/stages/pipelines

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: check_dag: don't reset the cache

https://github.com/iterative/dvc/pull/2529#discussion_r327711928

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: rename check_dag(stages) -> check_modified_graph(new_stages)

https://github.com/iterative/dvc/pull/2529#discussion_r327708439

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* repo: refactor "active" logic and remove it from Repo class

* repo: use tree.setter to automate cache reset

As noted by @Suor's , this is much less prone to errors.

* repo: move graph-related staticmethods into graph.py

As per @Suor's request.

* dvc: remove unused from_directory

https://github.com/iterative/dvc/pull/2529#discussion_r328079258

* repo: use _tree instead of __tree

https://github.com/iterative/dvc/pull/2529#discussion_r328071115

* repo: add a note about cache invalidation on repo.lock release

https://github.com/iterative/dvc/pull/2529#discussion_r328071635

* repo: rename reset() -> _reset()

https://github.com/iterative/dvc/pull/2529#discussion_r328079899

* pipeline: use graph helpers

https://github.com/iterative/dvc/pull/2529#discussion_r327502293

* repo: use list comprehension for collect()

https://github.com/iterative/dvc/pull/2529#discussion_r328076651

* repo: graph: only use first cycle

https://github.com/iterative/dvc/pull/2529#discussion_r328081346

* dvc: don't use G.nodes()

https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
 2019-09-26 07:42:29 +03:00
Paweł Redzyński be012e70cf checkout: raise on non existing dir cache (#2514) 
* checkout: raise on non existing dir cache

* checkout: don't throw on failed load_dir_cache

* checkout: raise on failed checkouts, only after all checkouts are perfromed

* remote: base: raise on failed dir cache loading

* rename exception for failed checkout
 2019-09-24 16:31:36 +03:00
Ruslan Kuprieiev 33f1cdfab8 Merge pull request #2525 from efiop/2506 
dvc: remove PATH prefixes appended by pyenv
 2019-09-23 19:35:46 +03:00
Alexander Schepanovski 4006c6434e dvc: remove dead code in DataCloud 2019-09-23 20:00:19 +07:00
Ruslan Kuprieiev fd14ca3269 dvc: remote: optimize imports 
Decreases `time dvc --version` from ~1.4 sec to ~0.3 sec.

Related #2495
Related #2445

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-09-23 11:30:45 +03:00
Ruslan Kuprieiev 43975c8618 dvc: remove PATH prefixes appended by pyenv 
Unlike PyInstaller, pyenv doesn't leave backups of original env vars
when it modifies them. If we look into the shim, pyenv and pyenv-exec,
we can figure out that the PATH is modified like this:

    PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH

where

    PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
    bin_path - might not start with $PYENV_ROOT as it runs realpath on
        it, see pyenv source code.
    plugin_bin - might contain more than 1 entry, which start with
        $PYENV_ROOT, see pyenv source code.

So having this, we can make a rightful assumption about what parts of the PATH
we need to remove in order to get the original PATH.

Fixes #2506
 2019-09-23 04:25:42 +03:00
Mr. OutisandRuslan Kuprieiev ae90904fb3 pull: download data from imported stages (#2499) 
* wip - pull is working

* use fetch instead of reproduce

* test pull and fetch

* style changes

* fetch: catch config error on fetch instead of get_remote

* output: remove unused guard clause

* fetch: define a more specific error instead of ConfigError

* fetch: display number of errors correctly

* exceptions: remove "Failed" prefix to download/upload

* fetch: catch other exceptions that might halt the download

* fetch: log catched exceptions

* fetch: reformat exception

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* fix formatting
 2019-09-22 01:28:12 +03:00
Paweł RedzyńskiandRuslan Kuprieiev d16178ab61 checkout: raise error on failed dir cache loading (#2503) 
* remote: base: load_dir_cache: raise exception on failed download

* output: base: raise error on failed get_files_number

* remote: base: log failed download as warning

* Update dvc/output/base.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* remote: base: dont download dir cache on load_dir_cache
 2019-09-19 16:35:37 +03:00
Paweł RedzyńskiandRuslan Kuprieiev 349b4af3ae install: verify no hooks colliding with dvc exist (#2505) 
* install: verify no hooks colliding with dvc exist

* test: install: remove obsolete tests

* Update tests/func/test_install.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* test: install: import GitHookAlreadyExistsError
 2019-09-19 15:40:58 +03:00
Alexander Schepanovski ffb52b256b perf: do not import http.server each run 
It is only used in tests.
 2019-09-17 18:41:05 +07:00
Ruslan Kuprieiev e36f242f66 Merge pull request #2493 from pared/2345 
output: prevent stage file from being used as output
 2019-09-14 21:42:06 +03:00
Paweł RedzyńskiandRuslan Kuprieiev 53f6aead95 Update dvc/dependency/base.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-09-14 19:40:59 +02:00
pawel 0935930f46 output: prevent stage file from being used as output 2019-09-13 13:31:50 +02:00
Ruslan Kuprieiev f0c43b4e02 path_info: don't forget to override as_posix method 
Unlike original implementation [1] that uses `str()` we actually need
to use `fspath`, because we've overriden `__str__` method to return
relative paths, which will break original `as_posix`.

[1] https://github.com/python/cpython/blob/v3.7.0/Lib/pathlib.py#L692

Fixes #2483

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-09-13 04:52:03 +03:00
Ruslan Kuprieiev 05adcf9a4f Merge pull request #2461 from Suor/nfs 
dvc: switch to flufl lock
 2019-09-11 16:43:40 +03:00
pawel d4c9c71a2f remote: checkout: reuse _checkout_file functionality 2019-09-11 11:26:30 +02:00
Ruslan Kuprieiev 09f6b1c7b9 Merge pull request #2485 from pared/2477 
move: use relpath to find target path
 2019-09-10 16:23:27 +03:00
pawel a16dc58ae3 move: use relpath to find target path 2019-09-10 14:03:55 +02:00
Ruslan Kuprieiev 384fd4cd9a test: repro: fix link types 2019-09-10 02:12:28 +03:00
Ruslan Kuprieiev 2c003fdc04 Merge pull request #2426 from mroutis/close-1652 
ssh: support linking
 2019-09-09 20:48:22 +03:00
Ruslan Kuprieiev b0b958e14b Merge pull request #2481 from efiop/2476 
scm: remove `pkg install` leftovers
 2019-09-09 20:16:01 +03:00
Ruslan Kuprieiev a529f7d8ad Merge pull request #2479 from efiop/always_reproduce 
run: introduce --always-changed
 2019-09-09 19:45:35 +03:00
Ruslan Kuprieiev ce27444dd2 scm: remove pkg install leftovers 
Leftover code was causing problems when working with symlinks, because
it tried to use `realpath` on symlinks to further use it to determine
parent directory.

Fixes #2476
 2019-09-09 19:43:10 +03:00
Mr. Outis 2a847ef25c tests: ignore ssh's hardlink and copy tests on windows 2019-09-09 11:20:29 -05:00
Ruslan Kuprieiev 357ce82f28 run: introduce --always-changed 
This is an extension of our callback stages (no deps, so always
considered as changed) for stages with dependencies, so that these
stages could be used properly in the middle of the DAG. See attached
issue for more info.

Related to #2378
 2019-09-09 18:03:28 +03:00
pawel 90738fab9c stage: prevent writing default wdir into stage file 2019-09-09 16:20:33 +02:00
Ruslan Kuprieiev b3e4ff7cba test: install: don't forget to adjust hook tests 2019-09-09 05:10:30 +03:00
Mr. OutisandRuslan Kuprieiev 5103dfd4aa tests: fix skipif on ssh/test_connection.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-09-07 21:38:50 -05:00
Mr. Outis b4507444d5 version: adjust get_fs_type to discover all partitions 2019-09-06 19:27:23 -05:00
Paweł Redzyński 0a214ae15a test: add: remove TestDvcGit (#2467) 2019-09-06 23:15:26 +03:00
Mr. Outis 05cbe0c922 tests: support windows 2019-09-06 14:17:44 -05:00
Mr. Outis 737f1e141d tests: use repo_dir instead of custom fixtures 2019-09-06 13:28:48 -05:00
Mr. Outis b903fe8023 get: show error message when path is a DVC-file (#2470) 
* get: show error message when path is a DVC-file

Fix #2468

* exceptions: rename output for data artifact

* get: use method to indetify if file is a DVC-file

* exceptions: change GetDVCFileError message
 2019-09-06 21:11:23 +03:00
Mr. Outis dd1d5bdbb0 tests: PosixPath to string in chdir_tmp fixture 2019-09-06 12:57:06 -05:00
Mr. Outis 73133e8c7a tests: [re] fix hardlink stuff with HDFS 2019-09-06 12:35:38 -05:00
Mr. Outis f97c33178f tests: use copy as cache type for HDFS 2019-09-06 12:05:53 -05:00
Mr. Outis 6eb26de856 fix tests 2019-09-06 11:32:31 -05:00
Mr. Outis 42168d34f3 Merge branch 'master' into close-1652 2019-09-06 09:29:54 -05:00
Paweł RedzyńskiAlexander SchepanovskiRuslan Kuprieiev
 		5ec1d274ae remote: base: don't checkout existing file (#2358) 
* remote: base: don't checkout existing file

* remote: _checkout_file: split to smaller methods

* state: local: verify if file needs checkout considering link type and access rights

* tests: checkout: test avoiding unnecessary copying

* efiop review refactor

* remote: local: refactor _needs_checkout NOTE

* remote: local: remove bool cast for `protected`, wrap linking in try

* test: checkout: test relinking upon existing cache

* stage: remove obsolete unprotecing from Stage.create

* test: checkout: refactor setting link type

* test: checkout: parametrize relinking test

* remote: local: don't remove cache test file on cache_copy test

* remote: base: remove force from _needs_checkout arguments

* Update dvc/remote/local/__init__.py

Co-Authored-By: Alexander Schepanovski <[email protected]>

* suor review refactor

* remote: base: re-add note why do we unprotect on _checkout_file

* test: checkout: tests for relinking and resetting target protection

* remove checkout behavior change

* remove unused imports

* remote: base: reuse __checkout_file in __checkout_dir

* initial dir

* checkout: test: should relink on repeated add

* remote: cleanup commented lines

* remote: base/local: add save_link flag to _checkout_file

* remote: local: add allow_copy flag to unprotect method

* Update dvc/remote/local/__init__.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* remote: local: remove unnecessary removal warning on _checkout_file

* remote: local: proper cache link type detection

* test: checkout: move repeated add tests

* Update dvc/remote/local/__init__.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* remote: base: remove _needs_checkout

* Update dvc/remote/base.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* remote: base: remove wrappers from _checkout_file
 2019-09-06 17:20:32 +03:00
Alexander Schepanovski 1bdaca0d6c dvc: switch to flufl lock 
This locks employs hardlink trick, which works on NFS and alike unlike
fcntl/open locks.

Closes #1918.
 2019-09-06 16:29:09 +07:00
Mr. Outis 369146a926 Merge branch 'master' into close-1652 2019-09-05 10:47:18 -05:00
Mr. Outis 0c231b68d7 fix formatting 2019-09-04 22:07:02 -05:00
Mr. Outis 2fe6fc1ba6 refactor: use a module for slow link detection 2019-09-04 19:32:55 -05:00
pawel 651d925578 stage: don't unprotect outs on stage create 2019-09-04 12:54:32 +02:00
pawel 4236f18e0c repo: get: intercept and convert NotDvcRepoErro to UrlNotDvcRepoError 2019-09-03 13:59:04 +02:00
Mr. Outis 3c69dc9fd5 tests: unit ssh symlink 2019-09-03 00:04:44 -05:00
Mr. Outis da65f8fe9a tests: functional ssh symlink 2019-08-31 13:34:48 -05:00
Mr. Outis 7b4584f86c ssh: stat -> lstat 2019-08-31 13:02:17 -05:00
Mr. Outis 0fc83be9ff tests: ssh hardlink optimization 2019-08-31 03:23:35 -05:00
Mr. Outis 0361b1b9c0 tests: ssh connection link methods 2019-08-31 01:56:00 -05:00