Ruslan Kuprieiev
e88856dc16
Merge pull request #2943 from efiop/2942
...
remote: make `copy` links atomic for local/hdfs/ssh
2019-12-13 20:57:46 +02:00
Ruslan Kuprieiev
da49badbfe
Merge branch 'master' into relink
2019-12-13 20:35:02 +02:00
Ruslan Kuprieiev
e384685715
dvc: release locks when running a command
...
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:
1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;
We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.
We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.
Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.
Fixes #755
2019-12-13 10:38:10 +02:00
Ruslan Kuprieiev
b8bcff5e06
remote: make copy links atomic for local/hdfs/ssh
...
Fixes #2942
2019-12-13 09:37:47 +02:00
Ruslan Kuprieiev
fd62ba840d
Merge pull request #2934 from pared/2896_checkout
...
checkout: test: refactor to dir helpers
2019-12-12 04:42:31 +02:00
pawel
5f826dd2e6
commit: test: refactor to dir helpers
2019-12-11 14:23:44 +01:00
pawel
26ef77cf96
checkout: test: refactor to dir helpers
2019-12-11 13:54:44 +01:00
Ruslan Kuprieiev
d2f40703cd
Merge pull request #2926 from pared/2896_test_move
...
test: move: convert to dir helpers
2019-12-11 02:44:14 +02:00
Ruslan Kuprieiev
6bf8d487da
Merge pull request #2921 from pared/2896_add_3
...
add: test: convert to dir helpers 3
2019-12-10 19:58:15 +02:00
Paweł Redzyński
267ac9b3ff
test: move: convert to dir helpers
2019-12-09 19:27:28 +01:00
Alexander Schepanovski
9607e5f873
get: refactor, clean up and fix dvc get implementation
...
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway
Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
2019-12-09 21:02:49 +03:00
Ruslan Kuprieiev
26a9702a4a
Merge pull request #2837 from danihodovic/feat/2515
...
get: copy/download files tracked by Git
2019-12-09 18:57:52 +02:00
Ruslan Kuprieiev
8890daf703
Merge pull request #2826 from mroutis/refactor-analytics
...
analytics: refactor into a module
2019-12-09 17:59:51 +02:00
pawel
c4565ce2ac
add: test: convert to dir helpers 3
...
TestAddUnsupportedFile
TestAdd
TestAddDirectory
2019-12-09 16:44:56 +01:00
Ruslan Kuprieiev
24fc8beaeb
Update test_ignore.py
2019-12-09 15:27:54 +02:00
Ruslan Kuprieiev
302b016d65
Merge pull request #2913 from Suor/migrate-test-ignore
...
test: migrate test_ignore
2019-12-09 15:25:46 +02:00
Alexander Schepanovski
2f79e362f4
dvc: add checkout --relink
2019-12-09 13:25:32 +03:00
Mr. Outis
5dd6300878
💅 black
2019-12-07 21:19:54 -06:00
Mr. Outis
c50bd17bb4
analytics: collect execution info only when available
2019-12-07 21:14:31 -06:00
Alexander Schepanovski
6c12a2e555
test: fix unicode test_ignore issue under Windows Py2
2019-12-07 17:17:18 +01:00
Alexander Schepanovski
79902988fc
test: fix files set comparisons in dvcignore tests under windows
2019-12-07 13:17:08 +01:00
Alexander Schepanovski
42096c5bfc
test: migrate test_ignore to new helpers
2019-12-07 11:09:52 +01:00
Mr. Outis
2145daa130
analytics: define private methods
2019-12-06 16:26:10 -06:00
Dani Hodovic
29b34bacc2
fixup! fixes
2019-12-06 21:55:54 +01:00
Dani Hodovic
d59bceb71a
fixes
2019-12-06 20:20:25 +01:00
Mr. Outis
da064e2ede
analytics: remove the report after sending it
2019-12-06 12:28:06 -06:00
Dani Hodovic
811855ade2
fixes
2019-12-06 18:56:19 +01:00
pawel
b39ebc8591
add: test: convert to dir helpers
...
test_should_relink_on_repeated_add
test_should_protect_on_repeated_add
2019-12-06 15:35:32 +01:00
pawel
551f5af561
add: test: convert to dir helpers
...
test_add_file_in_dir
test_readding_dir_should_not_unprotect_all
test_should_not_checkout_when_adding_cached_copy
2019-12-06 09:07:50 +01:00
Dani Hodovic
7debaca22e
fixup! fix error message for git files
2019-12-05 21:05:38 +01:00
Dani Hodovic
9ae4ab88bf
fix error message for git files
2019-12-05 21:05:04 +01:00
Dani Hodovic
b17d1c5b04
fixup
2019-12-05 21:05:03 +01:00
Dani Hodovic
3c2633e33f
Pass all tests
2019-12-05 21:05:03 +01:00
Dani Hodovic
38bed227c6
fixup
2019-12-05 21:05:03 +01:00
Dani Hodovic
9ece4ed8ad
ensure absolute paths for dvc files work but not for git
2019-12-05 21:05:02 +01:00
Dani Hodovic
ed10497854
Remove unused var
2019-12-05 21:05:02 +01:00
Dani Hodovic
cadbf10e64
dont use absolute paths in tests
2019-12-05 21:05:02 +01:00
Dani Hodovic
74f3851117
Disallow absolute paths in dvc get
2019-12-05 21:05:02 +01:00
Dani Hodovic
7e1423d13c
fix tests
2019-12-05 21:05:01 +01:00
Dani Hodovic
8c3ffe816d
fixup! Make tree.walk() output deterministic
2019-12-05 21:05:01 +01:00
Dani Hodovic
97862a3d5f
Make tree.walk() output deterministic
2019-12-05 21:05:01 +01:00
Dani Hodovic
4d14256995
fixup! get: copy regular files
2019-12-05 21:05:01 +01:00
Dani Hodovic
af2f216c5e
get: copy regular files
...
Allows `dvc get` to copy regular files or directories.
fixes : #2515
2019-12-05 21:05:00 +01:00
Ruslan Kuprieiev
73e69be067
Merge pull request #2882 from Suor/revert-checkout
...
dvc: revert checkout to be silly and fast, relink on dvc add though
2019-12-05 16:07:52 +02:00
Ruslan Kuprieiev
c328075c8b
Merge pull request #2868 from Suor/better-fixtures
...
test: refactor tmp dir helper fixtures
2019-12-05 16:00:37 +02:00
Mr. Outis
71ead3f62e
analytics: use a tempfile for inter-process communication
2019-12-04 19:47:06 -06:00
pawel
ca852170c1
get: test: from non-dvc master: refactor
2019-12-04 14:34:50 +01:00
Paweł Redzyński
f1d59dd64d
conftest: empty dir is tmp_path now
2019-12-04 13:10:06 +01:00
pawel
5e5c82652b
test: get: inline erepo with non-dvc master
2019-12-04 13:10:06 +01:00
Paweł Redzyński
b9d5933f5d
external repo: checkout rev before initializing repo
2019-12-04 13:10:06 +01:00
Ruslan Kuprieiev
pawel
Alexander Schepanovski
Mr. Outis
Dani Hodovic