Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-07 23:09:21 +01:00
Code Issues Packages Projects Releases Wiki Activity
4,097 Commits 7 Branches 579 Tags
0.71.0
Commit Graph
186 Commits
Author SHA1 Message Date
Saugat Pachhai 461a79e1e8 Fix typos 2019-11-26 20:07:20 +05:45
Ruslan Kuprieiev 7bc7a572f2 Merge pull request #2809 from casperdcl/gs-progress 
GS progress for push & pull
 2019-11-21 05:38:35 +02:00
Ruslan Kuprieiev 91ac75d023 Merge pull request #2798 from skshetry/fix-str-rel-repo 
Display path relative to temporary repo when get/import-ing files
 2019-11-20 11:34:31 +02:00
Casper da Costa-Luis b811a773c6 update dynamic chunk tests 2019-11-19 17:44:01 +00:00
Ruslan Kuprieiev 404620e267 test: update: use dvc_repo fixture 
To fix test failure.
 2019-11-19 19:12:49 +02:00
Saugat Pachhai 83c8ab5c26 test: Refactor tests 2019-11-19 21:55:13 +05:45
Saugat Pachhai f07a70a052 get/import: display path relative to temporary repo 2019-11-19 05:36:11 +05:45
Saugat Pachhai 68c7366ef0 utils: implement path_isin to compare child and parent paths 2019-11-19 05:34:33 +05:45
Aman Sharma 3edad21b41 Write test for tmp_fname to check if it accepts Path-like and str objects 2019-11-18 11:49:40 +05:30
Aman Sharma 636d850c84 Ensure remove accepts str and Path-like objects 2019-11-15 01:54:40 +05:30
Aman Sharma 9427bbe8d0 Move move to filesystem utility 2019-11-09 23:57:57 +05:30
Aman Sharma 44268251b4 Ensure move accepts str and Path-like objects 2019-11-09 18:47:01 +05:30
Aman Sharma 204cfcb879 Ensure makedirs accepts str and Path-like objects 2019-11-08 20:37:32 +05:30
Ruslan Kuprieiev ec44a614b0 Merge pull request #2753 from efiop/analytics 
analytics: don't verify config from repo
 2019-11-08 14:38:02 +02:00
Ruslan Kuprieiev 1058df0f77 analytics: don't forget to convert to bool 2019-11-08 05:21:47 +02:00
Ruslan Kuprieiev 910a67b35f analytics: don't verify config from repo 
Otherwise this will raise an exception from analytics worker:
```
dvc config cache.type sometypo
```
But running that same command with `--system` or `--global` flag outside
of repo won't raise that.

Also need to better handle ConfigObjError, but that is a separate issue.
 2019-11-08 02:37:11 +02:00
Aman Sharma d4aa27d6ab Ensure copyfile accepts str and Path-like objects 2019-11-08 01:51:49 +05:30
pawel f14aca630f ignore: make DvcIgnoreFilter use tree 2019-11-07 15:37:24 +01:00
Aman Sharma 34a868b420 Ensure file_md5 accept str and Path-like objects 2019-11-06 09:25:35 +05:30
Ruslan Kuprieiev 5e2fa3742b Merge pull request #2741 from algomaster99/remove-get-parent-dirs 
Remove `get_parent_dirs_up_to`
 2019-11-05 23:01:31 +02:00
Aman Sharma 8491ddf43f Remove get_parent_dirs_up_to 2019-11-06 02:03:47 +05:30
Aman Sharma f642ff76a5 Ensure contains_symlink_up_to accept both str and Path-like objects 2019-11-06 00:24:10 +05:30
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev 2df7eda159 Merge pull request #2674 from n3hrox/enhance/footnote_deduplication 
logger: output footer once for multithreaded push/pull errors
 2019-11-03 23:37:13 +02:00
Ruslan Kuprieiev c71e5562bb Merge pull request #2713 from GeorgeSabu-mec/master 
fix typo(correct spelling of existence)
 2019-11-03 12:44:03 +02:00
GeorgeSabu-MEC 1e22fabece fix typo(correct spelling of existence) 2019-11-03 15:42:07 +05:30
Ruslan Kuprieiev e21fc34c84 Merge pull request #2698 from algomaster99/test-argument-types-get_mtime_and_size 
Verify if `get_mtime_and_size` can take both Path objects and strings
 2019-11-03 00:25:47 +02:00
Aman Sharma d454cb8637 Modify test to check for directory as well 2019-11-03 02:23:46 +05:30
Mr. Outis 283f848822 sftp: make move an atomic operation 2019-11-01 13:17:22 -06:00
n3hrox 89502bee88 fix tests 2019-11-01 18:05:27 +01:00
Ruslan Kuprieiev d932405ee1 Merge pull request #2695 from mroutis/test-s3-etag 
tests: preserve the same ETag across buckets during copy
 2019-10-31 01:55:17 +02:00
Aman Sharma b9a17752e3 Test acceptable path types of get_mtime_and_size 2019-10-31 00:30:54 +05:30
Mr. Outis 732ed904ce tests: same ETag across buckets 2019-10-30 11:42:39 -06:00
Jorge Orpinel a48481b01f dvc: change ` for ' in dynamic parts of output messages (#2644) 
* dvc: change ` for ' in dinamic parts of DVC outputs

per https://github.com/iterative/dvc/pull/2644#issuecomment-545151398

* dvc: use ' for dynamic output and ` for DVC names

per https://github.com/iterative/dvc/pull/2644#discussion_r338417698
 2019-10-28 22:00:53 +02:00
Aman Sharma 5f644745c7 Write test for get_inode (#2673) 2019-10-28 16:43:47 +02:00
Ruslan Kuprieiev f3bacb7c28 gs: retry upload with smaller chunk size (#2661) 
Default chunk size for gs is 100M, which might be too much for
particular network. So if we are getting ConnectionError, we
should try lowering the chunk size until we reach the minimum
allowed chunk size of 256K. Also note that `chunk_size` must be a
multiple of 256K per the API specification.

Fixes #2572
 2019-10-25 22:45:54 +03:00
Mr. Outis 27b63e2674 s3: support for directories (#2619) 
* s3: list_objects -> list_objects_api

* s3: use path_info for _list_paths

* s3: separate _list_paths from _list_objects

* s3: support max items for listing objects/paths

* s3: take into account directories on `exists`

* s3: overwrite `isdir` method

* s3: custom collect directory entry for s3

* s3: strip checksum when collecting directories

* remote: change walk implementation to walk_files

* s3: add comment about isdir logic

* s3: use `dir_path` correctly on `isdir`

* tests: add a few unit tests for S3

* s3: use path_info join instead of posixpath

* tests: use parametrize for s3 tests

* remote: use posixpath accordingly

* tests: remove undeeded AWS variables from S3 test

* tests: rearrange parametrize values for readability

* tests: add encoding, use a single s3 instance, for loop instead of parameterize

* py2: fix encoding
 2019-10-25 03:41:04 +03:00
Casper da Costa-Luis 9ce3b0c50c add: output improvements (#2546) 
* some doc improvements

* add: intermediate progress

* update after rebase

* linting

* fix checksum progress cleanup by ensuring gc

* progress-aware logger.info

* re-fix checksum pbar closure

* multi-file add

* make stage.dump() less verbose.

TODO: check all instances of stage.dump and possibly make verbose

* decrease verbosity of checksum calculations

* leave dvc.add stats

* update/fix logger tests

* minor doc reword

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* tidy API target_;list => targets

- as per https://github.com/iterative/dvc/pull/2546#discussion_r337042535

* tidy RecursiveAddingWhileUsingFilename message duplication

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337044670

* minor logging uncomment

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337051097

* minor doc improvements

- https://github.com/iterative/dvc/pull/2546#discussion_r337116255
- https://github.com/iterative/dvc/pull/2546#discussion_r337116809

* tidy exception

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337434142

* minor doc update

- fixes https://github.com/iterative/dvc/pull/2546/files#r337294491
- related to https://github.com/iterative/dvc.org/issues/719

* Adding => add

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337037137

* various review fixes

- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r337927837
- revert warning style fix
  https://github.com/iterative/dvc/pull/2546#discussion_r338046035
- Abstract error consistency
  https://github.com/iterative/dvc/pull/2546#discussion_r337931070
- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r338254958

* remove diabling progress option

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r338314050
 2019-10-25 01:09:19 +03:00
Alexander Schepanovski 33b0eefec0 dvc: dedup used cache (#2630) 
* dvc: dedup used cache

This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts

* dvc: delay checksum names concatenation
 2019-10-21 12:12:56 +03:00
Casper da Costa-Luis 1e01ce3ac9 isatty (#2550) 
* suppress Tqdm when isatty

- fixes #2549

* fix tests

* unconditionally disable Tqdm if not isatty

- fixes https://github.com/iterative/dvc/pull/2550#discussion_r329386950

* add TQDM_DISABLE env var

- update tests
- add documentation

* TQDM_DISABLE => TQDM_ISATTY

* tests: mock.patch sys.stderr

* TQDM_ISATTY => DVC_ISATTY

as per https://github.com/iterative/dvc/pull/2550#discussion_r334565120

* dvc.progress.Tqdm file kwarg

- fixes https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335706955

* dynamic env DVC_ISATTY

- fixes
  https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335811454

* DVC_ISATTY in tests

- fixes https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335811099

* re-add and abstract `env2bool` to `dvc.utils`

- partially reverts commit 524a0df188f566fb3e103f6965f1b6ac9c530886
- fixes
  https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335811639

* env2bool tidy

* rename DVC_ISATTY => DVC_IGNORE_ISATTY

- fixes https://github.com/iterative/dvc/pull/2550#discussion_r335947382
 2019-10-20 00:54:06 +03:00
Mr. Outis 13a7e0f978 logger: send warning messages to stderr (#2612) 
* logger: send warning messages to stderr

Fix #2561

* logger: filter warnings for stdout
 2019-10-16 03:12:00 +03:00
Ruslan Kuprieiev 132de572af run: improve pyenv PATH modification criteria 
https://discordapp.com/channels/485586884165107732/485596304961962003/628766062472724510

Related to #2506
 2019-10-02 06:33:01 +03:00
Ruslan Kuprieiev 33f1cdfab8 Merge pull request #2525 from efiop/2506 
dvc: remove PATH prefixes appended by pyenv
 2019-09-23 19:35:46 +03:00
Ruslan Kuprieiev fd14ca3269 dvc: remote: optimize imports 
Decreases `time dvc --version` from ~1.4 sec to ~0.3 sec.

Related #2495
Related #2445

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-09-23 11:30:45 +03:00
Ruslan Kuprieiev 43975c8618 dvc: remove PATH prefixes appended by pyenv 
Unlike PyInstaller, pyenv doesn't leave backups of original env vars
when it modifies them. If we look into the shim, pyenv and pyenv-exec,
we can figure out that the PATH is modified like this:

    PATH=$PYENV_BIN_PATH:${bin_path}:${plugin_bin}:$PATH

where

    PYENV_BIN_PATH - starts with $PYENV_ROOT, see pyenv-exec source code.
    bin_path - might not start with $PYENV_ROOT as it runs realpath on
        it, see pyenv source code.
    plugin_bin - might contain more than 1 entry, which start with
        $PYENV_ROOT, see pyenv source code.

So having this, we can make a rightful assumption about what parts of the PATH
we need to remove in order to get the original PATH.

Fixes #2506
 2019-09-23 04:25:42 +03:00
Ruslan Kuprieiev e36f242f66 Merge pull request #2493 from pared/2345 
output: prevent stage file from being used as output
 2019-09-14 21:42:06 +03:00
pawel 0935930f46 output: prevent stage file from being used as output 2019-09-13 13:31:50 +02:00
Ruslan Kuprieiev f0c43b4e02 path_info: don't forget to override as_posix method 
Unlike original implementation [1] that uses `str()` we actually need
to use `fspath`, because we've overriden `__str__` method to return
relative paths, which will break original `as_posix`.

[1] https://github.com/python/cpython/blob/v3.7.0/Lib/pathlib.py#L692

Fixes #2483

Signed-off-by: Ruslan Kuprieiev <[email protected]>
 2019-09-13 04:52:03 +03:00
Ruslan Kuprieiev 2c003fdc04 Merge pull request #2426 from mroutis/close-1652 
ssh: support linking
 2019-09-09 20:48:22 +03:00
Mr. Outis 2a847ef25c tests: ignore ssh's hardlink and copy tests on windows 2019-09-09 11:20:29 -05:00