Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-08 15:28:26 +01:00
Code Issues Packages Projects Releases Wiki Activity
4,456 Commits 7 Branches 579 Tags
3d133186d4ec0095f103dc2a9fdcf0ad7881800b
Commit Graph
4456 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Mr. Outis 3d133186d4 summon: ugly hack 2019-12-18 00:48:10 -06:00
Ruslan Kuprieiev 7531694f1b Merge pull request #2935 from pared/2896_import 
import: test: refactor to dir helpers
 2019-12-17 15:38:45 +02:00
Ruslan Kuprieiev 8ee8b9ff14 Merge pull request #2966 from Suor/relink 
ui: reorder checkout options to go from common to rare
 2019-12-17 15:10:57 +02:00
pawel ffe0071647 import: test: refactor to dir helpers 2019-12-17 13:57:46 +01:00
Ruslan Kuprieiev 85b1721de8 Merge pull request #2866 from pared/2473 
remote: s3: adjust jobs number basing on file descriptors number
 2019-12-17 14:20:36 +02:00
Ruslan Kuprieiev 032d094239 Merge pull request #2889 from Baranowski/import_regulars.squashed 
import: allow downloading regular files/dirs tracked by git
 2019-12-17 14:09:58 +02:00
Alexander Schepanovski 6c0cda8c7b ui: reorder checkout options to go from common to rare 
This is the order we use in docs. It also makes sense.
 2019-12-17 18:40:26 +07:00
Paweł RedzyńskiandRuslan Kuprieiev ab548470f8 Update dvc/main.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-12-17 11:56:22 +01:00
Paweł Redzyński 53163366b6 remote: s3: revert jobs adjustment, pass OSError to main 2019-12-17 11:41:10 +01:00
pawel 6ba4aba419 remote: base: [down/up]load: raise on too many open files 2019-12-17 11:41:03 +01:00
pawel d253f1319d remote: upload/download: handle OSError 2019-12-17 11:41:03 +01:00
pawel a0eda4c884 remote: s3: reading ulimit/handles limit: support windows 2019-12-17 11:41:03 +01:00
pawel 710c4bd32e s3: test: mocker instead of mock annotations 2019-12-17 11:41:03 +01:00
pawel c167eb2675 remote: s3: adjust jobs only on default 2019-12-17 11:41:03 +01:00
pawel 0fdb88a54c remote: s3: adjust jobs number basing on file descriptors number 2019-12-17 11:41:03 +01:00
Ruslan Kuprieiev bdfeba8f9b Merge pull request #2958 from MaxRis/gdrive-tests 
Standardize gdrive tests with others remotes
 2019-12-16 20:52:46 +02:00
Ruslan Kuprieiev f51c222b1c Merge pull request #2937 from pared/2896_add_windows 
temporary_windows_drive fixture: move to dir_helpers
 2019-12-16 15:55:46 +02:00
Ruslan Kuprieiev 39a9fd8d79 Merge pull request #2951 from mroutis/tests-analytics 
tests: use dir helpers on analytics
 2019-12-16 14:41:28 +02:00
prd d40588aefa temporary_windows_drive fixture: move to dir_helpers 2019-12-16 13:15:53 +01:00
Mr. Outis f209e1eb6f tests/analytics: explicitly set content for foo 2019-12-15 13:11:40 -06:00
Wojciech Baranowski ad98f0fb50 Use fspath() in tests 2019-12-15 18:21:11 +02:00
Wojciech Baranowski c02c0169cf Revert "DependencyREPO.status: handle git-tracked files" 
This reverts commit 48bf4490d5.
 2019-12-15 18:14:47 +02:00
Wojciech Baranowski 7461f14845 Revert "_git_status(): use different cache dirs" 
This reverts commit 8985759d66.
 2019-12-15 18:14:38 +02:00
Max Risuhin 4d4a3fc716 Standardize gdrive tests with others remotes 2019-12-15 05:31:52 -08:00
Wojciech Baranowski 59156021de Revert "test_status_imported_git_file" 
This reverts commit 4d59bcddfd.
 2019-12-14 19:39:05 +02:00
Wojciech Baranowski 4d59bcddfd test_status_imported_git_file 2019-12-14 19:38:57 +02:00
Wojciech Baranowski 8985759d66 _git_status(): use different cache dirs 2019-12-14 14:42:51 +02:00
Wojciech Baranowski 48bf4490d5 DependencyREPO.status: handle git-tracked files 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 11d890a4fa Wrap tmp_dir in str() 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c8c285536f os.path.isdir(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 5ceaf5a6f0 os.path.isfile(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski a36ac0c4ea dependency/repo: fix py2 support 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 0ee6998a29 New tests use tmp_dir 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c04f5a0f9d import: allow downloading regular files/dirs tracked by git 
Fixes #2862
 2019-12-14 14:08:38 +02:00
Ruslan Kuprieiev 6e22769f3d dvc: bump to 0.77.3 0.77.3 2019-12-14 00:18:04 +02:00
Ruslan Kuprieiev 62495f4f06 Merge pull request #2949 from iterative/pull-2948-restyled 
Restyle remote: use absolute from root for gdrive credentials file
 2019-12-14 00:08:54 +02:00
Restyled.io a2b398255e Restyled by black 2019-12-13 22:04:20 +00:00
Eamon O'Dea e823828d6c remote: use absolute from root for gdrive credentials file 
Fixes #2947
 2019-12-13 16:58:11 -05:00
Ruslan Kuprieiev 1ed1a89b9f dvc: bump to 0.77.2 0.77.2 2019-12-13 20:58:13 +02:00
Ruslan Kuprieiev e88856dc16 Merge pull request #2943 from efiop/2942 
remote: make `copy` links atomic for local/hdfs/ssh
 2019-12-13 20:57:46 +02:00
Ruslan Kuprieiev 8de5286448 Merge pull request #2922 from Suor/relink 
dvc: add checkout --relink
 2019-12-13 20:35:10 +02:00
Ruslan Kuprieiev da49badbfe Merge branch 'master' into relink 2019-12-13 20:35:02 +02:00
Ruslan Kuprieiev 75ecc9960d dvc: bump to 0.77.1 0.77.1 2019-12-13 20:28:55 +02:00
Ruslan Kuprieiev 3c835909f6 Merge pull request #2931 from rbvesb/dvc-s3-acl 
remote: Extend S3 remote support for custom ACL
 2019-12-13 19:31:44 +02:00
Ruslan Kuprieiev d823f3b6c3 dvc: bump to 0.77.0 0.77.0 2019-12-13 17:43:26 +02:00
Ruslan Kuprieiev 3ed07cc448 daemon: fix typo 
Thanks @pared for catching this one https://github.com/iterative/dvc/pull/2918#discussion_r357574263 . Will add a test on top, need to re-release ASAP
 2019-12-13 17:42:08 +02:00
Ruslan Kuprieiev 8d7f20f688 Merge pull request #2584 from efiop/755 
dvc: release locks when running a command
 2019-12-13 17:35:34 +02:00
Alexander Schepanovski 5d1f01d58f checkout: iron out --relink related messages 2019-12-13 19:00:06 +07:00
Veselin Borisov cd461aec4f fixup! remote: Extend S3 remote support for custom ACL 2019-12-13 10:43:38 +02:00
Ruslan Kuprieiev e384685715 dvc: release locks when running a command 
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:

1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;

We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.

We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.

Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.

Fixes #755
 2019-12-13 10:38:10 +02:00