Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-07 07:18:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
9,569 Commits 7 Branches 579 Tags
dependabot/github_actions/github/codeql-action-4.38.2
Commit Graph
1894 Commits
Author SHA1 Message Date
Wojciech Baranowski c8c285536f os.path.isdir(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 5ceaf5a6f0 os.path.isfile(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 0ee6998a29 New tests use tmp_dir 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c04f5a0f9d import: allow downloading regular files/dirs tracked by git 
Fixes #2862
 2019-12-14 14:08:38 +02:00
Mr. Outis 299fb9022f 💅 black 2019-12-13 18:04:15 -06:00
Mr. Outis 37a2844a74 tests/repo: move .stages to unit 2019-12-13 18:01:11 -06:00
Mr. Outis b3a52515a3 tests/repo: move .collect to unit 2019-12-13 17:54:05 -06:00
Mr. Outis 655b79bf0c tests/repo: move .destroy to unit test 2019-12-13 17:51:54 -06:00
Ruslan Kuprieiev e88856dc16 Merge pull request #2943 from efiop/2942 
remote: make `copy` links atomic for local/hdfs/ssh
 2019-12-13 20:57:46 +02:00
Ruslan Kuprieiev da49badbfe Merge branch 'master' into relink 2019-12-13 20:35:02 +02:00
Ruslan Kuprieiev e384685715 dvc: release locks when running a command 
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:

1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;

We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.

We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.

Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.

Fixes #755
 2019-12-13 10:38:10 +02:00
Ruslan Kuprieiev b8bcff5e06 remote: make copy links atomic for local/hdfs/ssh 
Fixes #2942
 2019-12-13 09:37:47 +02:00
Mr. Outis c09c3aec0c remove empty file 2019-12-12 19:05:31 -06:00
Mr. Outis d0f3559d96 tests: updater, use dir helpers 2019-12-12 18:59:52 -06:00
Mr. Outis 7426a62baa tests: analytics, use dir helpers 2019-12-12 18:16:10 -06:00
Ruslan Kuprieiev fd62ba840d Merge pull request #2934 from pared/2896_checkout 
checkout: test: refactor to dir helpers
 2019-12-12 04:42:31 +02:00
pawel 5f826dd2e6 commit: test: refactor to dir helpers 2019-12-11 14:23:44 +01:00
pawel 26ef77cf96 checkout: test: refactor to dir helpers 2019-12-11 13:54:44 +01:00
Ruslan Kuprieiev d2f40703cd Merge pull request #2926 from pared/2896_test_move 
test: move: convert to dir helpers
 2019-12-11 02:44:14 +02:00
Ruslan Kuprieiev 6bf8d487da Merge pull request #2921 from pared/2896_add_3 
add: test: convert to dir helpers 3
 2019-12-10 19:58:15 +02:00
Paweł Redzyński 267ac9b3ff test: move: convert to dir helpers 2019-12-09 19:27:28 +01:00
Alexander Schepanovski 9607e5f873 get: refactor, clean up and fix dvc get implementation 
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway

Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
 2019-12-09 21:02:49 +03:00
Ruslan Kuprieiev 26a9702a4a Merge pull request #2837 from danihodovic/feat/2515 
get: copy/download files tracked by Git
 2019-12-09 18:57:52 +02:00
Ruslan Kuprieiev 8890daf703 Merge pull request #2826 from mroutis/refactor-analytics 
analytics: refactor into a module
 2019-12-09 17:59:51 +02:00
pawel c4565ce2ac add: test: convert to dir helpers 3 
TestAddUnsupportedFile
TestAdd
TestAddDirectory
 2019-12-09 16:44:56 +01:00
Ruslan Kuprieiev 24fc8beaeb Update test_ignore.py 2019-12-09 15:27:54 +02:00
Ruslan Kuprieiev 302b016d65 Merge pull request #2913 from Suor/migrate-test-ignore 
test: migrate test_ignore
 2019-12-09 15:25:46 +02:00
Alexander Schepanovski 2f79e362f4 dvc: add checkout --relink 2019-12-09 13:25:32 +03:00
Mr. Outis 5dd6300878 💅 black 2019-12-07 21:19:54 -06:00
Mr. Outis c50bd17bb4 analytics: collect execution info only when available 2019-12-07 21:14:31 -06:00
Alexander Schepanovski 6c12a2e555 test: fix unicode test_ignore issue under Windows Py2 2019-12-07 17:17:18 +01:00
Alexander Schepanovski 79902988fc test: fix files set comparisons in dvcignore tests under windows 2019-12-07 13:17:08 +01:00
Alexander Schepanovski 42096c5bfc test: migrate test_ignore to new helpers 2019-12-07 11:09:52 +01:00
Mr. Outis 2145daa130 analytics: define private methods 2019-12-06 16:26:10 -06:00
Dani Hodovic 29b34bacc2 fixup! fixes 2019-12-06 21:55:54 +01:00
Dani Hodovic d59bceb71a fixes 2019-12-06 20:20:25 +01:00
Mr. Outis da064e2ede analytics: remove the report after sending it 2019-12-06 12:28:06 -06:00
Dani Hodovic 811855ade2 fixes 2019-12-06 18:56:19 +01:00
pawel b39ebc8591 add: test: convert to dir helpers 
test_should_relink_on_repeated_add
test_should_protect_on_repeated_add
 2019-12-06 15:35:32 +01:00
pawel 551f5af561 add: test: convert to dir helpers 
test_add_file_in_dir
test_readding_dir_should_not_unprotect_all
test_should_not_checkout_when_adding_cached_copy
 2019-12-06 09:07:50 +01:00
Dani Hodovic 7debaca22e fixup! fix error message for git files 2019-12-05 21:05:38 +01:00
Dani Hodovic 9ae4ab88bf fix error message for git files 2019-12-05 21:05:04 +01:00
Dani Hodovic b17d1c5b04 fixup 2019-12-05 21:05:03 +01:00
Dani Hodovic 3c2633e33f Pass all tests 2019-12-05 21:05:03 +01:00
Dani Hodovic 38bed227c6 fixup 2019-12-05 21:05:03 +01:00
Dani Hodovic 9ece4ed8ad ensure absolute paths for dvc files work but not for git 2019-12-05 21:05:02 +01:00
Dani Hodovic ed10497854 Remove unused var 2019-12-05 21:05:02 +01:00
Dani Hodovic cadbf10e64 dont use absolute paths in tests 2019-12-05 21:05:02 +01:00
Dani Hodovic 74f3851117 Disallow absolute paths in dvc get 2019-12-05 21:05:02 +01:00
Dani Hodovic 7e1423d13c fix tests 2019-12-05 21:05:01 +01:00