Ruslan Kuprieiev
e88856dc16
Merge pull request #2943 from efiop/2942
...
remote: make `copy` links atomic for local/hdfs/ssh
2019-12-13 20:57:46 +02:00
Ruslan Kuprieiev
da49badbfe
Merge branch 'master' into relink
2019-12-13 20:35:02 +02:00
Ruslan Kuprieiev
3c835909f6
Merge pull request #2931 from rbvesb/dvc-s3-acl
...
remote: Extend S3 remote support for custom ACL
2019-12-13 19:31:44 +02:00
Veselin Borisov
cd461aec4f
fixup! remote: Extend S3 remote support for custom ACL
2019-12-13 10:43:38 +02:00
Ruslan Kuprieiev
e384685715
dvc: release locks when running a command
...
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:
1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;
We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.
We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.
Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.
Fixes #755
2019-12-13 10:38:10 +02:00
Ruslan Kuprieiev
b8bcff5e06
remote: make copy links atomic for local/hdfs/ssh
...
Fixes #2942
2019-12-13 09:37:47 +02:00
Ruslan Kuprieiev
6ab2322cde
Merge pull request #2938 from Suor/fix-dir-helpers
...
test: a couple of dir helpers fixes
2019-12-12 23:20:46 +02:00
Alexander Schepanovski
1143a8c57f
test: add checkout parse-cli-api test
2019-12-12 20:16:17 +07:00
Veselin Borisov
4c2bc6bf48
fixup! remote: Extend S3 remote support for custom ACL
2019-12-12 12:41:38 +02:00
Ruslan Kuprieiev
fd62ba840d
Merge pull request #2934 from pared/2896_checkout
...
checkout: test: refactor to dir helpers
2019-12-12 04:42:31 +02:00
pawel
5f826dd2e6
commit: test: refactor to dir helpers
2019-12-11 14:23:44 +01:00
Alexander Schepanovski
b8cf906edb
test: stop erepo_dir from undoing parent chdir
2019-12-11 20:19:38 +07:00
Alexander Schepanovski
a1643d576d
test: fix run_copy fixture when used with scm
2019-12-11 20:19:02 +07:00
pawel
26ef77cf96
checkout: test: refactor to dir helpers
2019-12-11 13:54:44 +01:00
Ruslan Kuprieiev
53c33a22d9
Merge pull request #2915 from maykulkarni/import-external-repo-fix
...
use cache dir as a default remote when importing from local repo
2019-12-11 08:45:22 +02:00
Ruslan Kuprieiev
d2f40703cd
Merge pull request #2926 from pared/2896_test_move
...
test: move: convert to dir helpers
2019-12-11 02:44:14 +02:00
Ruslan Kuprieiev
6bf8d487da
Merge pull request #2921 from pared/2896_add_3
...
add: test: convert to dir helpers 3
2019-12-10 19:58:15 +02:00
Mayur Kularni
a1ae3578c0
Fix caplog test case and move call to CmdRemoteAdd
2019-12-10 11:49:19 +05:30
Paweł Redzyński
267ac9b3ff
test: move: convert to dir helpers
2019-12-09 19:27:28 +01:00
Alexander Schepanovski
9607e5f873
get: refactor, clean up and fix dvc get implementation
...
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway
Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
2019-12-09 21:02:49 +03:00
Ruslan Kuprieiev
26a9702a4a
Merge pull request #2837 from danihodovic/feat/2515
...
get: copy/download files tracked by Git
2019-12-09 18:57:52 +02:00
Ruslan Kuprieiev
237704ef2b
Merge pull request #2908 from Suor/better-fixtures
...
test: write a usage text for dir_helpers
2019-12-09 18:01:33 +02:00
Ruslan Kuprieiev
8890daf703
Merge pull request #2826 from mroutis/refactor-analytics
...
analytics: refactor into a module
2019-12-09 17:59:51 +02:00
pawel
c4565ce2ac
add: test: convert to dir helpers 3
...
TestAddUnsupportedFile
TestAdd
TestAddDirectory
2019-12-09 16:44:56 +01:00
Ruslan Kuprieiev
24fc8beaeb
Update test_ignore.py
2019-12-09 15:27:54 +02:00
Ruslan Kuprieiev
302b016d65
Merge pull request #2913 from Suor/migrate-test-ignore
...
test: migrate test_ignore
2019-12-09 15:25:46 +02:00
Alexander Schepanovski
2f79e362f4
dvc: add checkout --relink
2019-12-09 13:25:32 +03:00
Alexander Schepanovski and Mr. Outis
1da133b9ed
test: update dir_helpers usage
...
Co-Authored-By: Mr. Outis <[email protected]
2019-12-09 11:34:10 +03:00
Alexander Schepanovski and Mr. Outis
a59041aeab
test: update dir_helpers usage
...
Co-Authored-By: Mr. Outis <[email protected]
2019-12-09 11:33:35 +03:00
Alexander Schepanovski and Mr. Outis
0cd6106189
Update tests/dir_helpers.py
...
Co-Authored-By: Mr. Outis <[email protected]
2019-12-09 11:24:41 +03:00
Mayur Kularni
5b9daea2b3
fixed test case
2019-12-09 12:11:49 +05:30
Mayur Kularni
c6050efbe3
codeclimate recommendation fix
2019-12-08 18:57:19 +05:30
Mayur Kularni
ca20a1c93a
incorporated feedback
2019-12-08 18:56:38 +05:30
Mayur Kulkarni
e1c5f6c5c0
external_repo: fix for local file import fail
...
If the URL is local dvc repo, the import/get should fetch from the source project cache.
Fixes #2599
2019-12-08 18:52:35 +05:30
Ruslan Kuprieiev
3796c4182b
Merge pull request #2906 from pared/2896_add_2
...
add: test: convert to dir helpers 2
2019-12-08 05:46:17 +02:00
Mr. Outis
5dd6300878
💅 black
2019-12-07 21:19:54 -06:00
Mr. Outis
6e4c3d6acd
tests: require keys on analytics report schema
2019-12-07 21:18:28 -06:00
Mr. Outis
c50bd17bb4
analytics: collect execution info only when available
2019-12-07 21:14:31 -06:00
Alexander Schepanovski
6c12a2e555
test: fix unicode test_ignore issue under Windows Py2
2019-12-07 17:17:18 +01:00
Alexander Schepanovski
79902988fc
test: fix files set comparisons in dvcignore tests under windows
2019-12-07 13:17:08 +01:00
Alexander Schepanovski
77fcd75b7d
test: use utf-8 in tmp_dir.gen()
2019-12-07 13:16:42 +01:00
Alexander Schepanovski
42096c5bfc
test: migrate test_ignore to new helpers
2019-12-07 11:09:52 +01:00
Alexander Schepanovski
c6c67ace20
test: create empty dirs via tmp_dir.gen()
2019-12-07 10:32:37 +01:00
Ruslan Kuprieiev
82dd1ac7c1
Merge pull request #2815 from xliiv/2807-use-pager
...
dvc/dagascii: Use pager instead of AsciiCanvas._do_draw
2019-12-07 01:16:16 +02:00
Tymoteusz Jankowski
44df59f96c
Fix test_dagascii tests
2019-12-06 23:36:21 +01:00
Mr. Outis
2145daa130
analytics: define private methods
2019-12-06 16:26:10 -06:00
Tymoteusz Jankowski
ffefab152e
Remove make_pager.cmd attribute
2019-12-06 22:17:44 +01:00
Tymoteusz Jankowski
ed4e951f22
Use monkeypatch in test_dagascii.py
2019-12-06 21:58:56 +01:00
Dani Hodovic
29b34bacc2
fixup! fixes
2019-12-06 21:55:54 +01:00
Tymoteusz Jankowski
6cfc831879
Use mocker fixture in test_dagascii.py
2019-12-06 21:54:40 +01:00
Ruslan Kuprieiev
Veselin Borisov
Alexander Schepanovski
pawel
Mayur Kularni
Tymoteusz Jankowski
Dani Hodovic