Commit Graph
131 Commits
Author SHA1 Message Date
pawel 36427387c0 get_mtime_and_size: move working tree assertion after veryfing dir 2020-01-03 10:32:32 +01:00
pawel 1e1a9cf774 tree: introduce is_working_tree 2020-01-03 10:03:47 +01:00
pawel eb53c030c6 local-related methods should verify that WorkingTree is in use 2020-01-03 09:46:04 +01:00
Mr. Outis 5611ccb43a exceptions: remove cause 2020-01-02 20:43:24 -06:00
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Mr. Outis 086acf6404 py3: dvc/utils/compat.py -> dvc/compat.py 2019-12-31 23:42:07 -06:00
Mr. Outis 27f581cf4e py3: remove imports from utils.compat 2019-12-31 23:16:11 -06:00
Mr. Outis 9c42809587 py3: _makedirs -> os.makedirs 2019-12-31 22:19:07 -06:00
Mr. Outis 999130b029 py3: remove compat csv_reader 2019-12-31 22:17:02 -06:00
Mr. Outis 1ef466b849 py3: remove convert_to_unicode noop 2019-12-31 22:14:54 -06:00
Mr. Outis b844aea42c py3: remove cast_bytes method 2019-12-31 20:14:25 -06:00
Ruslan Kuprieiev cbc7e21593 utils: don't import colorama again 2019-12-31 13:06:36 +02:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
pawel e1eab9405c tree: check that repo.tree is CleanTree when using walk_files 2019-12-30 14:19:43 +01:00
pawel e225601031 reintroduce dvcignore collection test 2019-12-30 14:19:43 +01:00
pawel d8813dc5ca remove dvc_walk, skip .dvcignore gathering test 2019-12-30 14:19:43 +01:00
pawel d8c0ae6909 remove most occurences of dvc_walk 2019-12-30 14:19:43 +01:00
Jorge Orpinel 06822a060e dvc: use ' for dynamic output (through dvc/util/) 2019-12-20 19:00:09 -07:00
Wojciech Baranowski c04f5a0f9d import: allow downloading regular files/dirs tracked by git
Fixes #2862
2019-12-14 14:08:38 +02:00
Ruslan Kuprieiev e384685715 dvc: release locks when running a command
Currently only one instance of dvc could run in a particular repo
because of the `.dvc/lock`. That gets furstrating when you are running
some long-running command and blocked from doing anything else.
This PR solved that issue for `dvc run` by releasing the repo lock
while running the command. To protect against conflicts, we now use
rwlock, which keeps the list of readers and writers for particular
paths. The process looks like this:

1) aquire read locks for deps;
2) aquire write locks for outs;
3) release repo lock;
4) run cmd;
5) aquire repo lock;
6) release write locks for outs;
7) release read locks for outs;

We also take rwlocks for operations such as `import`, `add` and `checkout`
to ensure that we won't hit the same files that are being used by
another `dvc run` in the background.

We don't aquire rwlocks for the stage file itself, as we don't really
care about anyone overwriting a stage file, as there is a very little
difference here between parallel runs and sequential runs that overwrite
the same stage file.

Next steps might be to release repo lock on `import`(when dowloading
stuff) and when computing checksums or copying files.

Fixes #755
2019-12-13 10:38:10 +02:00
Ruslan Kuprieiev 25807f2690 dvc: make flufl.lock opt-in and use zc.lockfile
As it turned out (see issue numbers down below), we can't really take
hardlinks for granted, so `flufl.lock` is not a panacea for all
filesystems. Considering that the vast majority of filesystems that our
users use support `zc.lockfile`(flock-based) and it has benefits like
more reliable mechanism, auto-delete when process exits, more sturdy
implementation, etc, it makes more sense to bring it back and use by
default again. For filesystems that don't support `flock()`, users will
be able to manually enable `flufl.lock` use through the config option.
It would be ideal if we could auto-detect that flock is not supported,
but in the real world, it turned out to be non-trivial, as it might hang
forever in a kernel context, which makes the implementation way too
complex for our purposes. So what we're doing instead is showing a
message before locking with `zc.lockfile` that, under normal
circumstances will disappear once the lock is taken or failed, otherwise
it will point users to the related documentation where they can learn
about how to opt-in for `flufl.lock`.

Fixes #2831
Fixes #2897
Related #2860
2019-12-13 02:25:57 +02:00
Saugat Pachhai 461a79e1e8 Fix typos 2019-11-26 20:07:20 +05:45
Ruslan Kuprieiev 91ac75d023 Merge pull request #2798 from skshetry/fix-str-rel-repo
Display path relative to temporary repo when get/import-ing files
2019-11-20 11:34:31 +02:00
Saugat Pachhai 58624e0230 utils/fs: use fspath_py35 instead of fspath 2019-11-19 21:55:01 +05:45
Ruslan Kuprieiev af0d913760 Merge pull request #2775 from Suor/opt-yaml
perf: optimize yaml parsing for stages
2019-11-19 15:51:34 +02:00
Ruslan Kuprieiev 689dd95817 Merge pull request #2790 from shcheklein/fix-2789
makedirs: fix mode flag is being ignored starting from Python 3.7
2019-11-19 15:47:37 +02:00
Ivan Shcheklein aa6a77285d makedirs: add comment on why do have to use umask 2019-11-19 00:17:27 -08:00
Saugat Pachhai 68c7366ef0 utils: implement path_isin to compare child and parent paths 2019-11-19 05:34:33 +05:45
Ruslan Kuprieiev 38c210068b Merge pull request #2795 from algomaster99/test-arg-types-remove
Ensure `remove` accepts str and Path-like objects
2019-11-16 21:02:04 +02:00
Aman Sharma d9cd9ece8b Move _chmod to fs.py 2019-11-16 00:44:20 +05:30
Alexander Schepanovski 3943b0eb57 perf: a small stage validation optimization 2019-11-15 16:31:46 +01:00
Ivan Shcheklein ecfbc3a6de compat: fix py2 mkdirs implementation, bytes do not exist on py2 2019-11-14 14:33:36 -08:00
Aman Sharma 636d850c84 Ensure remove accepts str and Path-like objects 2019-11-15 01:54:40 +05:30
Ivan Shcheklein 2b934470b4 makedirs: fix mode flag is being ignored starting from Python 3.7 2019-11-13 14:12:01 -08:00
Alexander Schepanovski 329d072c22 perf: explain why two yaml parsers are used 2019-11-13 16:35:05 +01:00
Alexander Schepanovski 58c9e5557c perf: optimize yaml parsing for stages 2019-11-11 13:41:50 +01:00
n3hrox 53f4ad2943 add build cleanup 2019-11-10 23:56:28 +01:00
n3hrox 3ba4b8bd4d remove unnecessary pip check 2019-11-10 22:10:19 +01:00
n3hrox 1156763e84 add pip fallback 2019-11-10 21:40:46 +01:00
n3hrox 36f75dd519 Remove unnecessary file 2019-11-10 19:48:15 +01:00
n3hrox e1dfb75055 Refactor get_package_manager 2019-11-10 19:46:48 +01:00
n3hrox f4ee4b0434 Add better heuristic for build detection 2019-11-10 18:04:41 +01:00
Aman Sharma 9427bbe8d0 Move move to filesystem utility 2019-11-09 23:57:57 +05:30
Aman Sharma d4aa27d6ab Ensure copyfile accepts str and Path-like objects 2019-11-08 01:51:49 +05:30
Aman Sharma 34a868b420 Ensure file_md5 accept str and Path-like objects 2019-11-06 09:25:35 +05:30
Ruslan Kuprieiev 5e2fa3742b Merge pull request #2741 from algomaster99/remove-get-parent-dirs
Remove `get_parent_dirs_up_to`
2019-11-05 23:01:31 +02:00
Aman Sharma 8491ddf43f Remove get_parent_dirs_up_to 2019-11-06 02:03:47 +05:30
Aman Sharma f642ff76a5 Ensure contains_symlink_up_to accept both str and Path-like objects 2019-11-06 00:24:10 +05:30
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00