Ruslan Kuprieiev
381601f4ee
dvc: cache graph/stages/pipelines ( #2529 )
...
* dvc: cache graph/stages/pipelines
Signed-off-by: Ruslan Kuprieiev <[email protected] https://github.com/iterative/dvc/pull/2529#discussion_r327711928
Signed-off-by: Ruslan Kuprieiev <[email protected] https://github.com/iterative/dvc/pull/2529#discussion_r327708439
Signed-off-by: Ruslan Kuprieiev <[email protected] https://github.com/iterative/dvc/pull/2529#discussion_r328079258
* repo: use _tree instead of __tree
https://github.com/iterative/dvc/pull/2529#discussion_r328071115
* repo: add a note about cache invalidation on repo.lock release
https://github.com/iterative/dvc/pull/2529#discussion_r328071635
* repo: rename reset() -> _reset()
https://github.com/iterative/dvc/pull/2529#discussion_r328079899
* pipeline: use graph helpers
https://github.com/iterative/dvc/pull/2529#discussion_r327502293
* repo: use list comprehension for collect()
https://github.com/iterative/dvc/pull/2529#discussion_r328076651
* repo: graph: only use first cycle
https://github.com/iterative/dvc/pull/2529#discussion_r328081346
* dvc: don't use G.nodes()
https://github.com/iterative/dvc/pull/2529#discussion_r328089671
https://github.com/iterative/dvc/pull/2529#discussion_r328089878
2019-09-26 07:42:29 +03:00
Paweł Redzyński
be012e70cf
checkout: raise on non existing dir cache ( #2514 )
...
* checkout: raise on non existing dir cache
* checkout: don't throw on failed load_dir_cache
* checkout: raise on failed checkouts, only after all checkouts are perfromed
* remote: base: raise on failed dir cache loading
* rename exception for failed checkout
2019-09-24 16:31:36 +03:00
Alexander Schepanovski
4006c6434e
dvc: remove dead code in DataCloud
2019-09-23 20:00:19 +07:00
Mr. Outis and Ruslan Kuprieiev
ae90904fb3
pull: download data from imported stages ( #2499 )
...
* wip - pull is working
* use fetch instead of reproduce
* test pull and fetch
* style changes
* fetch: catch config error on fetch instead of get_remote
* output: remove unused guard clause
* fetch: define a more specific error instead of ConfigError
* fetch: display number of errors correctly
* exceptions: remove "Failed" prefix to download/upload
* fetch: catch other exceptions that might halt the download
* fetch: log catched exceptions
* fetch: reformat exception
Co-Authored-By: Ruslan Kuprieiev <[email protected]
2019-09-22 01:28:12 +03:00
Paweł Redzyński and Ruslan Kuprieiev
d16178ab61
checkout: raise error on failed dir cache loading ( #2503 )
...
* remote: base: load_dir_cache: raise exception on failed download
* output: base: raise error on failed get_files_number
* remote: base: log failed download as warning
* Update dvc/output/base.py
Co-Authored-By: Ruslan Kuprieiev <[email protected]
2019-09-19 16:35:37 +03:00
Paweł Redzyński and Ruslan Kuprieiev
349b4af3ae
install: verify no hooks colliding with dvc exist ( #2505 )
...
* install: verify no hooks colliding with dvc exist
* test: install: remove obsolete tests
* Update tests/func/test_install.py
Co-Authored-By: Ruslan Kuprieiev <[email protected]
2019-09-19 15:40:58 +03:00
Paweł Redzyński and Ruslan Kuprieiev
53f6aead95
Update dvc/dependency/base.py
...
Co-Authored-By: Ruslan Kuprieiev <[email protected]
2019-09-14 19:40:59 +02:00
pawel
0935930f46
output: prevent stage file from being used as output
2019-09-13 13:31:50 +02:00
Ruslan Kuprieiev
05adcf9a4f
Merge pull request #2461 from Suor/nfs
...
dvc: switch to flufl lock
2019-09-11 16:43:40 +03:00
pawel
d4c9c71a2f
remote: checkout: reuse _checkout_file functionality
2019-09-11 11:26:30 +02:00
Ruslan Kuprieiev
09f6b1c7b9
Merge pull request #2485 from pared/2477
...
move: use relpath to find target path
2019-09-10 16:23:27 +03:00
pawel
a16dc58ae3
move: use relpath to find target path
2019-09-10 14:03:55 +02:00
Ruslan Kuprieiev
384fd4cd9a
test: repro: fix link types
2019-09-10 02:12:28 +03:00
Ruslan Kuprieiev
2c003fdc04
Merge pull request #2426 from mroutis/close-1652
...
ssh: support linking
2019-09-09 20:48:22 +03:00
Ruslan Kuprieiev
ce27444dd2
scm: remove pkg install leftovers
...
Leftover code was causing problems when working with symlinks, because
it tried to use `realpath` on symlinks to further use it to determine
parent directory.
Fixes #2476
2019-09-09 19:43:10 +03:00
pawel
90738fab9c
stage: prevent writing default wdir into stage file
2019-09-09 16:20:33 +02:00
Ruslan Kuprieiev
b3e4ff7cba
test: install: don't forget to adjust hook tests
2019-09-09 05:10:30 +03:00
Paweł Redzyński
0a214ae15a
test: add: remove TestDvcGit ( #2467 )
2019-09-06 23:15:26 +03:00
Mr. Outis
05cbe0c922
tests: support windows
2019-09-06 14:17:44 -05:00
Mr. Outis
b903fe8023
get: show error message when path is a DVC-file ( #2470 )
...
* get: show error message when path is a DVC-file
Fix #2468
* exceptions: rename output for data artifact
* get: use method to indetify if file is a DVC-file
* exceptions: change GetDVCFileError message
2019-09-06 21:11:23 +03:00
Mr. Outis
73133e8c7a
tests: [re] fix hardlink stuff with HDFS
2019-09-06 12:35:38 -05:00
Mr. Outis
f97c33178f
tests: use copy as cache type for HDFS
2019-09-06 12:05:53 -05:00
Mr. Outis
6eb26de856
fix tests
2019-09-06 11:32:31 -05:00
Mr. Outis
42168d34f3
Merge branch 'master' into close-1652
2019-09-06 09:29:54 -05:00
5ec1d274ae
remote: base: don't checkout existing file ( #2358 )
...
* remote: base: don't checkout existing file
* remote: _checkout_file: split to smaller methods
* state: local: verify if file needs checkout considering link type and access rights
* tests: checkout: test avoiding unnecessary copying
* efiop review refactor
* remote: local: refactor _needs_checkout NOTE
* remote: local: remove bool cast for `protected`, wrap linking in try
* test: checkout: test relinking upon existing cache
* stage: remove obsolete unprotecing from Stage.create
* test: checkout: refactor setting link type
* test: checkout: parametrize relinking test
* remote: local: don't remove cache test file on cache_copy test
* remote: base: remove force from _needs_checkout arguments
* Update dvc/remote/local/__init__.py
Co-Authored-By: Alexander Schepanovski <[email protected] [email protected] [email protected] [email protected]
2019-09-06 17:20:32 +03:00
Alexander Schepanovski
1bdaca0d6c
dvc: switch to flufl lock
...
This locks employs hardlink trick, which works on NFS and alike unlike
fcntl/open locks.
Closes #1918 .
2019-09-06 16:29:09 +07:00
Mr. Outis
369146a926
Merge branch 'master' into close-1652
2019-09-05 10:47:18 -05:00
pawel
651d925578
stage: don't unprotect outs on stage create
2019-09-04 12:54:32 +02:00
pawel
4236f18e0c
repo: get: intercept and convert NotDvcRepoErro to UrlNotDvcRepoError
2019-09-03 13:59:04 +02:00
Mr. Outis
da65f8fe9a
tests: functional ssh symlink
2019-08-31 13:34:48 -05:00
Alexander Schepanovski
8984e9fed4
dvc: use wdir == "." on add
...
Fixes #2404
2019-08-24 10:23:41 +07:00
Ruslan Kuprieiev
2d9497c524
Merge pull request #2421 from pared/2328
...
repo: move: save stage after moving
2019-08-21 16:33:13 +03:00
Ruslan Kuprieiev
5aa9a2f501
Merge pull request #2333 from casperdcl/tqdm
...
change progress bar backend to tqdm
2019-08-21 15:38:51 +03:00
pared
c4de07b2bd
repo: move: save stage after moving
2019-08-21 14:34:23 +02:00
Mr. Outis and Ruslan Kuprieiev
8918bdc383
http: add exception when trying to traverse remote ( #2411 )
...
* http: add exception when trying to traverse remote
Fix #2380
* http: exception raise on `list_cache_paths` instead of `all`
`NotImplementedError` -> `RemoteActionNotImplemented`
* http: raise exception on `__init__`
* set default no_traverse to True on remote base
* tests: explicitly use traverse on data cloud and no traverse with HTTP
* remote: define a new RemoteConfigError
* remote: reuse ConfigError instead of RemoteConfigError
* tests: set no_traverse to False with SSHMocked
* deepsource: fix minor issue with spacing
* remove blankspace
Co-Authored-By: Ruslan Kuprieiev <[email protected]
2019-08-20 00:47:15 +03:00
Ruslan Kuprieiev
f0c12ea162
Merge pull request #2407 from efiop/master
...
test: run: refactor signal handler tests
2019-08-19 15:22:25 +03:00
Ruslan Kuprieiev
89d3ccae0d
test: run: refactor signal handler tests
...
Current test is testing for signal handler restoration in an obscure
way. As a result, the test wasn't actually testing anything, because
KeyboardInterrupt is raised by default_int_handler, and so raising it
in a mock won't be caught by signal handlers, because they are dealing
with real singnals.
Fix #2406
Signed-off-by: Ruslan Kuprieiev <[email protected]
2019-08-19 14:49:19 +03:00
Casper da Costa-Luis
1c8bf11238
Merge remote-tracking branch 'upstream/master' into tqdm
2019-08-19 11:15:13 +01:00
Casper da Costa-Luis
093ad20a84
flake8 unused imports
2019-08-17 16:15:15 +01:00
Casper da Costa-Luis
534d6fdb0c
remove defunct test, extraneous comments
...
- https://github.com/iterative/dvc/pull/2333#discussion_r314927966
2019-08-17 15:43:54 +01:00
Gunnlaugur Thor Briem
e1dd75edcd
fix: don't dvc checkout on git file checkout
...
Closes #2400
2019-08-16 22:11:16 +00:00
Casper da Costa-Luis
60566b380e
minor test update
2019-08-16 22:33:00 +01:00
Casper da Costa-Luis
adf05e6ff0
minor rename
2019-08-16 22:32:47 +01:00
Casper da Costa-Luis
b018933b5a
fix py2 MagicMock(sys.stderr).encoding in tests
2019-08-16 22:26:19 +01:00
Alexander Schepanovski
85289ad00b
api: wrap not found errors
...
api.open()/read() will now raise OutputFileMissingError and
Remote*.open() will raise FileNotFoundError if target file is absent.
2019-08-15 23:20:48 +07:00
Casper da Costa-Luis
f88c2fe5fc
remove old test
2019-08-14 00:19:44 +01:00
Casper da Costa-Luis
3f8caeccc4
Merge remote-tracking branch 'upstream/master' into tqdm
...
major updates across dvc/remote/
2019-08-13 23:29:59 +01:00
Alexander Schepanovski
60a983b63f
api: stream api.open() for local, SSH and HDFS
2019-08-13 18:19:52 +07:00
Casper da Costa-Luis
120dfff05a
naive test update
2019-08-11 01:21:50 +01:00
Casper da Costa-Luis
ee4248df92
fix and update more tests
2019-08-11 01:17:42 +01:00
Ruslan Kuprieiev
Paweł Redzyński
Alexander Schepanovski
Mr. OutisandRuslan Kuprieiev
Ruslan Kuprieiev
Casper da Costa-Luis
Gunnlaugur Thor Briem