Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-09 23:10:24 +01:00
Code Issues Packages Projects Releases Wiki Activity
4,452 Commits 7 Branches 579 Tags
ffe007164777a8d752fe5901b15ca3ad542e9de3
Commit Graph
1178 Commits
Author SHA1 Message Date
pawel ffe0071647 import: test: refactor to dir helpers 2019-12-17 13:57:46 +01:00
Ruslan Kuprieiev 85b1721de8 Merge pull request #2866 from pared/2473 
remote: s3: adjust jobs number basing on file descriptors number
 2019-12-17 14:20:36 +02:00
Ruslan Kuprieiev 032d094239 Merge pull request #2889 from Baranowski/import_regulars.squashed 
import: allow downloading regular files/dirs tracked by git
 2019-12-17 14:09:58 +02:00
Paweł Redzyński 53163366b6 remote: s3: revert jobs adjustment, pass OSError to main 2019-12-17 11:41:10 +01:00
pawel 6ba4aba419 remote: base: [down/up]load: raise on too many open files 2019-12-17 11:41:03 +01:00
pawel d253f1319d remote: upload/download: handle OSError 2019-12-17 11:41:03 +01:00
pawel a0eda4c884 remote: s3: reading ulimit/handles limit: support windows 2019-12-17 11:41:03 +01:00
pawel 710c4bd32e s3: test: mocker instead of mock annotations 2019-12-17 11:41:03 +01:00
pawel c167eb2675 remote: s3: adjust jobs only on default 2019-12-17 11:41:03 +01:00
Ruslan Kuprieiev bdfeba8f9b Merge pull request #2958 from MaxRis/gdrive-tests 
Standardize gdrive tests with others remotes
 2019-12-16 20:52:46 +02:00
prd d40588aefa temporary_windows_drive fixture: move to dir_helpers 2019-12-16 13:15:53 +01:00
Ruslan Kuprieiev 39a9fd8d79 Merge pull request #2951 from mroutis/tests-analytics 
tests: use dir helpers on analytics
 2019-12-16 14:41:28 +02:00
Mr. Outis f209e1eb6f tests/analytics: explicitly set content for foo 2019-12-15 13:11:40 -06:00
Wojciech Baranowski ad98f0fb50 Use fspath() in tests 2019-12-15 18:21:11 +02:00
Max Risuhin 4d4a3fc716 Standardize gdrive tests with others remotes 2019-12-15 05:31:52 -08:00
Wojciech Baranowski 59156021de Revert "test_status_imported_git_file" 
This reverts commit 4d59bcddfd.
 2019-12-14 19:39:05 +02:00
Wojciech Baranowski 4d59bcddfd test_status_imported_git_file 2019-12-14 19:38:57 +02:00
Wojciech Baranowski 11d890a4fa Wrap tmp_dir in str() 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c8c285536f os.path.isdir(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 5ceaf5a6f0 os.path.isfile(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 0ee6998a29 New tests use tmp_dir 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c04f5a0f9d import: allow downloading regular files/dirs tracked by git 
Fixes #2862
 2019-12-14 14:08:38 +02:00
Ruslan Kuprieiev e88856dc16 Merge pull request #2943 from efiop/2942 
remote: make `copy` links atomic for local/hdfs/ssh
 2019-12-13 20:57:46 +02:00
Ruslan Kuprieiev da49badbfe Merge branch 'master' into relink 2019-12-13 20:35:02 +02:00
Ruslan Kuprieiev 3c835909f6 Merge pull request #2931 from rbvesb/dvc-s3-acl 
remote: Extend S3 remote support for custom ACL
 2019-12-13 19:31:44 +02:00
Veselin Borisov cd461aec4f fixup! remote: Extend S3 remote support for custom ACL 2019-12-13 10:43:38 +02:00
Ruslan Kuprieiev e384685715 dvc: release locks when running a command 
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:

1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;

We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.

We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.

Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.

Fixes #755
 2019-12-13 10:38:10 +02:00
Ruslan Kuprieiev b8bcff5e06 remote: make copy links atomic for local/hdfs/ssh 
Fixes #2942
 2019-12-13 09:37:47 +02:00
Mr. Outis 7426a62baa tests: analytics, use dir helpers 2019-12-12 18:16:10 -06:00
Ruslan Kuprieiev 6ab2322cde Merge pull request #2938 from Suor/fix-dir-helpers 
test: a couple of dir helpers fixes
 2019-12-12 23:20:46 +02:00
Alexander Schepanovski 1143a8c57f test: add checkout parse-cli-api test 2019-12-12 20:16:17 +07:00
Veselin Borisov 4c2bc6bf48 fixup! remote: Extend S3 remote support for custom ACL 2019-12-12 12:41:38 +02:00
Ruslan Kuprieiev fd62ba840d Merge pull request #2934 from pared/2896_checkout 
checkout: test: refactor to dir helpers
 2019-12-12 04:42:31 +02:00
pawel 5f826dd2e6 commit: test: refactor to dir helpers 2019-12-11 14:23:44 +01:00
Alexander Schepanovski b8cf906edb test: stop erepo_dir from undoing parent chdir 2019-12-11 20:19:38 +07:00
Alexander Schepanovski a1643d576d test: fix run_copy fixture when used with scm 2019-12-11 20:19:02 +07:00
pawel 26ef77cf96 checkout: test: refactor to dir helpers 2019-12-11 13:54:44 +01:00
Ruslan Kuprieiev 53c33a22d9 Merge pull request #2915 from maykulkarni/import-external-repo-fix 
use cache dir as a default remote when importing from local repo
 2019-12-11 08:45:22 +02:00
Ruslan Kuprieiev d2f40703cd Merge pull request #2926 from pared/2896_test_move 
test: move: convert to dir helpers
 2019-12-11 02:44:14 +02:00
Ruslan Kuprieiev 6bf8d487da Merge pull request #2921 from pared/2896_add_3 
add: test: convert to dir helpers 3
 2019-12-10 19:58:15 +02:00
Mayur Kularni a1ae3578c0 Fix caplog test case and move call to CmdRemoteAdd 2019-12-10 11:49:19 +05:30
Paweł Redzyński 267ac9b3ff test: move: convert to dir helpers 2019-12-09 19:27:28 +01:00
Alexander Schepanovski 9607e5f873 get: refactor, clean up and fix dvc get implementation 
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway

Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
 2019-12-09 21:02:49 +03:00
Ruslan Kuprieiev 26a9702a4a Merge pull request #2837 from danihodovic/feat/2515 
get: copy/download files tracked by Git
 2019-12-09 18:57:52 +02:00
Ruslan Kuprieiev 237704ef2b Merge pull request #2908 from Suor/better-fixtures 
test: write a usage text for dir_helpers
 2019-12-09 18:01:33 +02:00
Ruslan Kuprieiev 8890daf703 Merge pull request #2826 from mroutis/refactor-analytics 
analytics: refactor into a module
 2019-12-09 17:59:51 +02:00
pawel c4565ce2ac add: test: convert to dir helpers 3 
TestAddUnsupportedFile
TestAdd
TestAddDirectory
 2019-12-09 16:44:56 +01:00
Ruslan Kuprieiev 24fc8beaeb Update test_ignore.py 2019-12-09 15:27:54 +02:00
Ruslan Kuprieiev 302b016d65 Merge pull request #2913 from Suor/migrate-test-ignore 
test: migrate test_ignore
 2019-12-09 15:25:46 +02:00
Alexander Schepanovski 2f79e362f4 dvc: add checkout --relink 2019-12-09 13:25:32 +03:00