Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-10 07:18:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
4,427 Commits 7 Branches 579 Tags
11d890a4fa85c77f8cebbbdab42eccb9a4b5e51b
Commit Graph
4427 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Wojciech Baranowski 11d890a4fa Wrap tmp_dir in str() 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c8c285536f os.path.isdir(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 5ceaf5a6f0 os.path.isfile(str(...)) 2019-12-14 14:08:38 +02:00
Wojciech Baranowski a36ac0c4ea dependency/repo: fix py2 support 2019-12-14 14:08:38 +02:00
Wojciech Baranowski 0ee6998a29 New tests use tmp_dir 2019-12-14 14:08:38 +02:00
Wojciech Baranowski c04f5a0f9d import: allow downloading regular files/dirs tracked by git 
Fixes #2862
 2019-12-14 14:08:38 +02:00
Ruslan Kuprieiev 6e22769f3d dvc: bump to 0.77.3 0.77.3 2019-12-14 00:18:04 +02:00
Ruslan Kuprieiev 62495f4f06 Merge pull request #2949 from iterative/pull-2948-restyled 
Restyle remote: use absolute from root for gdrive credentials file
 2019-12-14 00:08:54 +02:00
Restyled.io a2b398255e Restyled by black 2019-12-13 22:04:20 +00:00
Eamon O'Dea e823828d6c remote: use absolute from root for gdrive credentials file 
Fixes #2947
 2019-12-13 16:58:11 -05:00
Ruslan Kuprieiev 1ed1a89b9f dvc: bump to 0.77.2 0.77.2 2019-12-13 20:58:13 +02:00
Ruslan Kuprieiev e88856dc16 Merge pull request #2943 from efiop/2942 
remote: make `copy` links atomic for local/hdfs/ssh
 2019-12-13 20:57:46 +02:00
Ruslan Kuprieiev 8de5286448 Merge pull request #2922 from Suor/relink 
dvc: add checkout --relink
 2019-12-13 20:35:10 +02:00
Ruslan Kuprieiev da49badbfe Merge branch 'master' into relink 2019-12-13 20:35:02 +02:00
Ruslan Kuprieiev 75ecc9960d dvc: bump to 0.77.1 0.77.1 2019-12-13 20:28:55 +02:00
Ruslan Kuprieiev 3c835909f6 Merge pull request #2931 from rbvesb/dvc-s3-acl 
remote: Extend S3 remote support for custom ACL
 2019-12-13 19:31:44 +02:00
Ruslan Kuprieiev d823f3b6c3 dvc: bump to 0.77.0 0.77.0 2019-12-13 17:43:26 +02:00
Ruslan Kuprieiev 3ed07cc448 daemon: fix typo 
Thanks @pared for catching this one https://github.com/iterative/dvc/pull/2918#discussion_r357574263 . Will add a test on top, need to re-release ASAP
 2019-12-13 17:42:08 +02:00
Ruslan Kuprieiev 8d7f20f688 Merge pull request #2584 from efiop/755 
dvc: release locks when running a command
 2019-12-13 17:35:34 +02:00
Alexander Schepanovski 5d1f01d58f checkout: iron out --relink related messages 2019-12-13 19:00:06 +07:00
Veselin Borisov cd461aec4f fixup! remote: Extend S3 remote support for custom ACL 2019-12-13 10:43:38 +02:00
Ruslan Kuprieiev e384685715 dvc: release locks when running a command 
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:

1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;

We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.

We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.

Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.

Fixes #755
 2019-12-13 10:38:10 +02:00
Ruslan Kuprieiev b8bcff5e06 remote: make copy links atomic for local/hdfs/ssh 
Fixes #2942
 2019-12-13 09:37:47 +02:00
Ruslan Kuprieiev ddfa46aedb Merge pull request #2941 from efiop/2559 
run: provide a hint if we get output duplication error
 2019-12-13 04:19:51 +02:00
Ruslan Kuprieiev 173388b2f8 run: provide a hint if we get output duplication error 
Related to #2559
 2019-12-13 03:36:23 +02:00
Ruslan Kuprieiev 30f4551283 dvc: bump to 0.76.0 0.76.0 2019-12-13 03:20:43 +02:00
Ruslan Kuprieiev 602a34dad2 Merge pull request #2918 from efiop/lock 
dvc: make flufl.lock opt-in and use zc.lockfile
 2019-12-13 03:17:25 +02:00
Ruslan Kuprieiev 25807f2690 dvc: make flufl.lock opt-in and use zc.lockfile 
As it turned out (see issue numbers down below), we can't really take
hardlinks for granted, so `flufl.lock` is not a panacea for all
filesystems. Considering that the vast majority of filesystems that our
users use support `zc.lockfile`(flock-based) and it has benefits like
more reliable mechanism, auto-delete when process exits, more sturdy
implementation, etc, it makes more sense to bring it back and use by
default again. For filesystems that don't support `flock()`, users will
be able to manually enable `flufl.lock` use through the config option.
It would be ideal if we could auto-detect that flock is not supported,
but in the real world, it turned out to be non-trivial, as it might hang
forever in a kernel context, which makes the implementation way too
complex for our purposes. So what we're doing instead is showing a
message before locking with `zc.lockfile` that, under normal
circumstances will disappear once the lock is taken or failed, otherwise
it will point users to the related documentation where they can learn
about how to opt-in for `flufl.lock`.

Fixes #2831
Fixes #2897
Related #2860
 2019-12-13 02:25:57 +02:00
Ruslan Kuprieiev 6ab2322cde Merge pull request #2938 from Suor/fix-dir-helpers 
test: a couple of dir helpers fixes
 2019-12-12 23:20:46 +02:00
Alexander Schepanovski 1143a8c57f test: add checkout parse-cli-api test 2019-12-12 20:16:17 +07:00
Veselin Borisov 4c2bc6bf48 fixup! remote: Extend S3 remote support for custom ACL 2019-12-12 12:41:38 +02:00
Ruslan Kuprieiev fd62ba840d Merge pull request #2934 from pared/2896_checkout 
checkout: test: refactor to dir helpers
 2019-12-12 04:42:31 +02:00
Ruslan Kuprieiev e2d783aafd Merge pull request #2932 from pared/2896_commit 
commit: test: refactor to dir helpers
 2019-12-11 20:18:30 +02:00
Ruslan Kuprieiev 08347c06ee Merge pull request #2939 from casperdcl/snap 
snap: remove plugs (incompatible with classic)
 2019-12-11 20:04:51 +02:00
Casper da Costa-Luis ed95000bb4 remove plugs (incompatible with classic) 
Details: <https://launchpad.net/bugs/1655369>
 2019-12-11 17:38:09 +00:00
pawel 5f826dd2e6 commit: test: refactor to dir helpers 2019-12-11 14:23:44 +01:00
Alexander Schepanovski b8cf906edb test: stop erepo_dir from undoing parent chdir 2019-12-11 20:19:38 +07:00
Alexander Schepanovski a1643d576d test: fix run_copy fixture when used with scm 2019-12-11 20:19:02 +07:00
pawel 26ef77cf96 checkout: test: refactor to dir helpers 2019-12-11 13:54:44 +01:00
Ruslan Kuprieiev 53c33a22d9 Merge pull request #2915 from maykulkarni/import-external-repo-fix 
use cache dir as a default remote when importing from local repo
 2019-12-11 08:45:22 +02:00
Mayur Kularni e1e0cbd038 moved default check out of RemoteConfig 2019-12-11 10:54:59 +05:30
Ruslan Kuprieiev 6296e12f48 setup: update pyarrow version 
https://github.com/apache/arrow/pull/4907
 2019-12-11 06:24:49 +02:00
Ruslan Kuprieiev 5cc92a9b4c Merge pull request #2907 from efiop/import_get 
import-url: add directories to examples
 2019-12-11 03:23:50 +02:00
Ruslan Kuprieiev d2f40703cd Merge pull request #2926 from pared/2896_test_move 
test: move: convert to dir helpers
 2019-12-11 02:44:14 +02:00
Ruslan Kuprieiev 8a7cf747e9 dvc: bump to 0.75.1 0.75.1 2019-12-10 23:04:39 +02:00
Ruslan Kuprieiev 6bf8d487da Merge pull request #2921 from pared/2896_add_3 
add: test: convert to dir helpers 3
 2019-12-10 19:58:15 +02:00
Ruslan Kuprieiev a9e829ae4e Merge pull request #2925 from Suor/dani-cleaup 
get: refactor, clean up and fix `dvc get` implementation
 2019-12-10 19:20:54 +02:00
Veselin Borisov a160a8ea7a fixup! remote: Extend S3 remote support for custom ACL 2019-12-10 13:44:13 +02:00
Veselin Borisov 9eb1ac2d95 remote: Extend S3 remote support for custom ACL 
Besides AWS boto3 extra_arg `ACL` (`x-amz-acl` header), there four more
that could be used to grant specific permissions to specific grantees on
the uploaded object.
These headers are:
  - `x-amz-grant-full-control` with boto3's extra_arg `GrantFullControl`
  - `x-amz-grant-read` with boto3's extra_arg `GrantRead`
  - `x-amz-grant-read-acp` with boto3's extra_arg `GrantReadACP`
  - `x-amz-grant-write-acp` with boto3's extra_arg `GrantWriteACP`

As `ACL` (`x-amz-acl` header) is limited to be used with predefined canned ACLs,
adding support for the aformentioned headers will help in applying
more customized persmissions solutions according to different project
needs.

References:
  - https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#permissions
  - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer.ALLOWED_UPLOAD_ARGS
  - https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html

Fixes #(2930).
 2019-12-10 12:50:34 +02:00
Mayur Kularni e6285f70c5 typo in doc fix 2019-12-10 15:38:54 +05:30