Mr. Outis
4a25612282
💅 refactor
2019-12-25 01:04:52 -06:00
Mr. Outis
75ad14aaa2
💅 black
2019-12-25 00:43:33 -06:00
Mr. Outis
0658622ca2
summon: pair-programming adjustments
2019-12-25 00:42:55 -06:00
Mr. Outis
1e65c76ef1
summon: raise SummonError appropiately
2019-12-24 20:20:03 -06:00
Mr. Outis
fbb7aebd31
tests/summon: use different args for non-canon.yml
2019-12-24 19:50:43 -06:00
Mr. Outis
1c22cd5af4
summon: adjust comments
2019-12-24 02:20:08 -06:00
Mr. Outis
addc56c232
summon: allow different YAML file names
2019-12-18 23:38:52 -06:00
Mr. Outis
c852d3316d
💅 black
2019-12-18 23:34:43 -06:00
Mr. Outis
4273ff49b3
summon: params -> args; use summon key in yaml
2019-12-18 23:32:28 -06:00
Mr. Outis
29339d29a2
summon: file & method -> call
2019-12-18 23:25:45 -06:00
Mr. Outis
1b3c771df5
summon: artifacs.yaml -> dvcsummon.yaml; artifact -> object
2019-12-18 22:42:55 -06:00
Mr. Outis
32f35ea686
summon: remove scm.add, *_gen already handles it
2019-12-18 22:29:44 -06:00
Mr. Outis
5d66a572d4
summon: support overriding parameters
2019-12-18 17:20:07 -06:00
Mr. Outis
d0344ce975
summon: grimoire/spells -> artifacts
2019-12-18 17:00:39 -06:00
Mr. Outis
789db9169a
summon: support checking out dependencies
2019-12-18 16:41:03 -06:00
Mr. Outis
b921730f37
summon: safely get a module; change naming
2019-12-18 13:32:23 -06:00
Mr. Outis
3d133186d4
summon: ugly hack
2019-12-18 00:48:10 -06:00
pawel
ffe0071647
import: test: refactor to dir helpers
2019-12-17 13:57:46 +01:00
Ruslan Kuprieiev
85b1721de8
Merge pull request #2866 from pared/2473
...
remote: s3: adjust jobs number basing on file descriptors number
2019-12-17 14:20:36 +02:00
Ruslan Kuprieiev
032d094239
Merge pull request #2889 from Baranowski/import_regulars.squashed
...
import: allow downloading regular files/dirs tracked by git
2019-12-17 14:09:58 +02:00
Paweł Redzyński
53163366b6
remote: s3: revert jobs adjustment, pass OSError to main
2019-12-17 11:41:10 +01:00
pawel
6ba4aba419
remote: base: [down/up]load: raise on too many open files
2019-12-17 11:41:03 +01:00
pawel
d253f1319d
remote: upload/download: handle OSError
2019-12-17 11:41:03 +01:00
pawel
a0eda4c884
remote: s3: reading ulimit/handles limit: support windows
2019-12-17 11:41:03 +01:00
pawel
710c4bd32e
s3: test: mocker instead of mock annotations
2019-12-17 11:41:03 +01:00
pawel
c167eb2675
remote: s3: adjust jobs only on default
2019-12-17 11:41:03 +01:00
Ruslan Kuprieiev
bdfeba8f9b
Merge pull request #2958 from MaxRis/gdrive-tests
...
Standardize gdrive tests with others remotes
2019-12-16 20:52:46 +02:00
prd
d40588aefa
temporary_windows_drive fixture: move to dir_helpers
2019-12-16 13:15:53 +01:00
Ruslan Kuprieiev
39a9fd8d79
Merge pull request #2951 from mroutis/tests-analytics
...
tests: use dir helpers on analytics
2019-12-16 14:41:28 +02:00
Mr. Outis
f209e1eb6f
tests/analytics: explicitly set content for foo
2019-12-15 13:11:40 -06:00
Wojciech Baranowski
ad98f0fb50
Use fspath() in tests
2019-12-15 18:21:11 +02:00
Max Risuhin
4d4a3fc716
Standardize gdrive tests with others remotes
2019-12-15 05:31:52 -08:00
Wojciech Baranowski
59156021de
Revert "test_status_imported_git_file"
...
This reverts commit 4d59bcddfd
2019-12-14 19:39:05 +02:00
Wojciech Baranowski
4d59bcddfd
test_status_imported_git_file
2019-12-14 19:38:57 +02:00
Wojciech Baranowski
11d890a4fa
Wrap tmp_dir in str()
2019-12-14 14:08:38 +02:00
Wojciech Baranowski
c8c285536f
os.path.isdir(str(...))
2019-12-14 14:08:38 +02:00
Wojciech Baranowski
5ceaf5a6f0
os.path.isfile(str(...))
2019-12-14 14:08:38 +02:00
Wojciech Baranowski
0ee6998a29
New tests use tmp_dir
2019-12-14 14:08:38 +02:00
Wojciech Baranowski
c04f5a0f9d
import: allow downloading regular files/dirs tracked by git
...
Fixes #2862
2019-12-14 14:08:38 +02:00
Ruslan Kuprieiev
e88856dc16
Merge pull request #2943 from efiop/2942
...
remote: make `copy` links atomic for local/hdfs/ssh
2019-12-13 20:57:46 +02:00
Ruslan Kuprieiev
da49badbfe
Merge branch 'master' into relink
2019-12-13 20:35:02 +02:00
Ruslan Kuprieiev
3c835909f6
Merge pull request #2931 from rbvesb/dvc-s3-acl
...
remote: Extend S3 remote support for custom ACL
2019-12-13 19:31:44 +02:00
Veselin Borisov
cd461aec4f
fixup! remote: Extend S3 remote support for custom ACL
2019-12-13 10:43:38 +02:00
Ruslan Kuprieiev
e384685715
dvc: release locks when running a command
...
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:
1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;
We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.
We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.
Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.
Fixes #755
2019-12-13 10:38:10 +02:00
Ruslan Kuprieiev
b8bcff5e06
remote: make copy links atomic for local/hdfs/ssh
...
Fixes #2942
2019-12-13 09:37:47 +02:00
Mr. Outis
7426a62baa
tests: analytics, use dir helpers
2019-12-12 18:16:10 -06:00
Ruslan Kuprieiev
6ab2322cde
Merge pull request #2938 from Suor/fix-dir-helpers
...
test: a couple of dir helpers fixes
2019-12-12 23:20:46 +02:00
Alexander Schepanovski
1143a8c57f
test: add checkout parse-cli-api test
2019-12-12 20:16:17 +07:00
Veselin Borisov
4c2bc6bf48
fixup! remote: Extend S3 remote support for custom ACL
2019-12-12 12:41:38 +02:00
Ruslan Kuprieiev
fd62ba840d
Merge pull request #2934 from pared/2896_checkout
...
checkout: test: refactor to dir helpers
2019-12-12 04:42:31 +02:00
Mr. Outis
pawel
Ruslan Kuprieiev
Wojciech Baranowski
Max Risuhin
Veselin Borisov
Alexander Schepanovski