Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,869 Commits 7 Branches 579 Tags
440b323e39b82cb646a8efd5f6ef1f867152b264
Commit Graph
3869 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Casper da Costa-Luis 440b323e39 TqdmThreadPoolExecutor => ThreadPoolExecutor 
- Fixes https://github.com/iterative/dvc/pull/2703#discussion_r341763150
 2019-11-02 15:22:46 +00:00
Casper da Costa-Luis 82dae6c25b remove special TqdmThreadPoolExecutor logic 
- fixes #2686
- closes #2703
 2019-11-02 15:22:46 +00:00
kurianbenoy 306c916626 Change import order in progress file 2019-11-02 15:22:46 +00:00
kurianbenoy 21bafccc96 Remove multi-threaded prefix before progress bar(#2686) 2019-11-02 15:22:46 +00:00
Ruslan Kuprieiev 2e6485f24b Merge pull request #2709 from mroutis/fix-2704 
sftp: make move across drives possible
 2019-11-02 14:07:46 +02:00
Ruslan Kuprieiev 91a7af03e7 Merge pull request #2707 from Suor/urlinfo 
perf: refactor URLInfo to not use ParseResult inside
 2019-11-02 13:49:49 +02:00
Mr. Outis 16164e65ad sftp: move -> use atomic copying 2019-11-01 17:01:50 -06:00
Ruslan Kuprieiev 809a85a340 dvc: bump to 0.66.3 0.66.3 2019-11-01 23:58:17 +02:00
Ruslan Kuprieiev ce8e019dd1 Merge pull request #2705 from shcheklein/fix-2702 
get-url: fix NPE when src non local and dst not absolute
 2019-11-01 23:54:53 +02:00
Ruslan Kuprieiev 7ecbdab564 Merge pull request #2693 from sadielbartholomew/i2685-pull-cmd-output 
pull, at import stage: replace warning with info report
 2019-11-01 23:15:23 +02:00
Mr. Outis 8f7000056b sftp: move -> fallback to copy when sftp.rename fails 2019-11-01 13:26:03 -06:00
Mr. Outis 283f848822 sftp: make move an atomic operation 2019-11-01 13:17:22 -06:00
Mr. Outis ab1ead881b sftp: make move across drives possible 
Fix #2704
 2019-11-01 12:40:16 -06:00
Ivan Shcheklein 3660d9eac4 local output: address PR comment, fix style a bit 2019-11-01 09:49:56 -07:00
Alexander Schepanovski b83564d80e perf: refactor URLInfo to not use ParseResult inside 
This permites URLInfo construction from parts without building/reparsing
string, which makes it around 20% faster. Code is also simpler and a bit
shorter now.

Other improvements:
- added URLInfo.replace(path=...)
- URLInfo._path now stringifies properly
- protected against passing path not starting with /
- do not allow query, params, fragments and passwords in urls (they were
silently ignored previously)
 2019-11-01 17:33:00 +01:00
Ruslan Kuprieiev c999b5d270 remove info message 2019-11-01 16:45:41 +02:00
Ruslan Kuprieiev 5c8a312002 fix order 2019-11-01 16:41:22 +02:00
Ruslan Kuprieiev 0dd4d2361e fix formatting 2019-11-01 16:40:06 +02:00
Sadie Bartholomew 31551e6e2e CLI: change return output for 'checkout' and 'pull' commands 
Change messages output after return of some commands:

* Remove warning for 'dvc checkout' & commands calling it under the hood
* Add an informational message output for 'dvc pull' command

Fixes #2685.
 2019-10-31 23:01:24 +00:00
Ivan Shcheklein 98750c95c5 get-url: fix NPE when src non local and dst not absolute 2019-10-31 09:58:07 -07:00
Ruslan Kuprieiev d932405ee1 Merge pull request #2695 from mroutis/test-s3-etag 
tests: preserve the same ETag across buckets during copy
 2019-10-31 01:55:17 +02:00
Mr. Outis 732ed904ce tests: same ETag across buckets 2019-10-30 11:42:39 -06:00
Ruslan Kuprieiev e623c4faf7 dvc: bump to 0.66.2 0.66.2 2019-10-30 17:20:05 +02:00
Ivan ShchekleinandRuslan Kuprieiev 2ec0aa0c37 failing while pulling imported directories (#2692) 
* pull: fix-2684 - failing while pulling imported directories

* pull: fix-2684 - fix code indentation

* minor imported directories pull test update

Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-10-30 17:18:07 +02:00
Alexander Schepanovski 77fcd49695 dvc: fix tqdm dep (#2694) 
tqdm==4.35.0 was throwing KeyError: 'percentage'.
 2019-10-30 16:43:18 +02:00
Ruslan Kuprieiev 877e38551e lock: add a workaround for a flufl.lock bug (#2688) 
This bug doesn't affect us during regular dvc CLI commands, but is not
visualy pleasing during API usage.

Fixes #2573
 2019-10-29 20:54:56 +02:00
Jorge Orpinel a48481b01f dvc: change ` for ' in dynamic parts of output messages (#2644) 
* dvc: change ` for ' in dinamic parts of DVC outputs

per https://github.com/iterative/dvc/pull/2644#issuecomment-545151398

* dvc: use ' for dynamic output and ` for DVC names

per https://github.com/iterative/dvc/pull/2644#discussion_r338417698
 2019-10-28 22:00:53 +02:00
Sagun Karanjit 5d6918a04c Remove --all-commits options from usage output (#2682) 
* Comment out --all-commits optional arg

* Suppress --all-commits optional arg
 2019-10-28 17:27:34 +02:00
Aman Sharma 5f644745c7 Write test for get_inode (#2673) 2019-10-28 16:43:47 +02:00
Ruslan Kuprieiev b9c7d88fcc dvc: bump to 0.66.1 0.66.1 2019-10-26 23:36:13 +03:00
Ruslan Kuprieiev 64f78c164d Revert "Added path.lstrip for / in imports (#2651)" (#2675) 
This reverts commit bfc0a3596f.
 2019-10-26 23:35:22 +03:00
Ruslan Kuprieiev 1baf020de7 s3: don't use walk_files for list_cache_paths 
No need to convert back and forth from string to PathInfo
 2019-10-26 00:29:23 +03:00
Ruslan Kuprieiev 1ab9319682 gs: fix typo 0.66.0 2019-10-25 23:24:43 +03:00
Ruslan Kuprieiev 32ba773aee s3: don't forget to convert to regular path strings 2019-10-25 23:07:17 +03:00
Ruslan Kuprieiev fc485581f9 dvc: bump to 0.66.0 2019-10-25 22:46:50 +03:00
Ruslan Kuprieiev f3bacb7c28 gs: retry upload with smaller chunk size (#2661) 
Default chunk size for gs is 100M, which might be too much for
particular network. So if we are getting ConnectionError, we
should try lowering the chunk size until we reach the minimum
allowed chunk size of 256K. Also note that `chunk_size` must be a
multiple of 256K per the API specification.

Fixes #2572
 2019-10-25 22:45:54 +03:00
Sanket Saurav 88f45c2348 Exclude compat files from DeepSource analysis (#2670) 
Hi team,

I just [noticed](https://github.com/iterative/dvc/pull/2548#issuecomment-536255251) that there are could be quite a number of false-positives in `dvc/utils/compat.py` because of, well, the things we have to do for compatibility. I would recommend adding that file to `exclude_patterns` to prevent the noise. I've proposed that in this PR.

We've seen some other projects report this as well, and we're working on proactively suggesting this via the product UI to make it easier.

Hope this helps!

Best,
Sanket
Founder, DeepSource
 2019-10-25 15:28:09 +03:00
Mr. Outis 27b63e2674 s3: support for directories (#2619) 
* s3: list_objects -> list_objects_api

* s3: use path_info for _list_paths

* s3: separate _list_paths from _list_objects

* s3: support max items for listing objects/paths

* s3: take into account directories on `exists`

* s3: overwrite `isdir` method

* s3: custom collect directory entry for s3

* s3: strip checksum when collecting directories

* remote: change walk implementation to walk_files

* s3: add comment about isdir logic

* s3: use `dir_path` correctly on `isdir`

* tests: add a few unit tests for S3

* s3: use path_info join instead of posixpath

* tests: use parametrize for s3 tests

* remote: use posixpath accordingly

* tests: remove undeeded AWS variables from S3 test

* tests: rearrange parametrize values for readability

* tests: add encoding, use a single s3 instance, for loop instead of parameterize

* py2: fix encoding
 2019-10-25 03:41:04 +03:00
Ruslan Kuprieiev 35499faa7d readme: remove brew cask and update brew install cmd 2019-10-25 01:41:46 +03:00
Casper da Costa-Luis 9ce3b0c50c add: output improvements (#2546) 
* some doc improvements

* add: intermediate progress

* update after rebase

* linting

* fix checksum progress cleanup by ensuring gc

* progress-aware logger.info

* re-fix checksum pbar closure

* multi-file add

* make stage.dump() less verbose.

TODO: check all instances of stage.dump and possibly make verbose

* decrease verbosity of checksum calculations

* leave dvc.add stats

* update/fix logger tests

* minor doc reword

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* tidy API target_;list => targets

- as per https://github.com/iterative/dvc/pull/2546#discussion_r337042535

* tidy RecursiveAddingWhileUsingFilename message duplication

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337044670

* minor logging uncomment

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337051097

* minor doc improvements

- https://github.com/iterative/dvc/pull/2546#discussion_r337116255
- https://github.com/iterative/dvc/pull/2546#discussion_r337116809

* tidy exception

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337434142

* minor doc update

- fixes https://github.com/iterative/dvc/pull/2546/files#r337294491
- related to https://github.com/iterative/dvc.org/issues/719

* Adding => add

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337037137

* various review fixes

- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r337927837
- revert warning style fix
  https://github.com/iterative/dvc/pull/2546#discussion_r338046035
- Abstract error consistency
  https://github.com/iterative/dvc/pull/2546#discussion_r337931070
- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r338254958

* remove diabling progress option

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r338314050
 2019-10-25 01:09:19 +03:00
Ruslan Kuprieiev 3667830c2a dvc: bump to 0.65.0 0.65.0 2019-10-24 23:41:43 +03:00
Ruslan Kuprieiev 8d9e4ca934 graph: use find_cycle instead of simple_cycle (#2668) 
We are incorrectly using `simple_cycle` here, `find_cycle` is much more
suitable for this application and much faster too. For example:

```
$ cat test_networkx.py
import networkx as nx
from dvc.repo.graph import check_acyclic

G = nx.scale_free_graph(1200)

list(nx.find_cycle(G, orientation='ignore'))

list(nx.simple_cycles(G))

$ python -mcProfile -scumtime test_networkx.py &> log.log
```

shows ~100s for `simple_cycles` and ~0s for `find_cycle`.

Fixes #2666
 2019-10-24 23:41:11 +03:00
Casper da Costa-Luis ed1f643a6e ssh credential prompt before progress (#2660) 
* ssh credential prompt before progress

* clean up ssh cred handling

- fixes https://github.com/iterative/dvc/pull/2660#discussion_r338536169
 2019-10-24 19:14:07 +03:00
Ruslan Kuprieiev 24ed639901 setup: add upper limit for tqdm version 
https://github.com/iterative/dvc/issues/2659#issuecomment-545940739
 2019-10-24 17:29:53 +03:00
Ruslan Kuprieiev b8d1ec204c dvc: bump to 0.64.1 0.64.1 2019-10-24 17:15:30 +03:00
Casper da Costa-Luis 6f068857cb Desc fix (#2663) 
* fix zero-width desc again

- fixes #2659

* correctly use tqdm API
 2019-10-24 17:05:52 +03:00
Ruslan Kuprieiev 8d067c8805 setup: update project url 2019-10-24 15:19:04 +03:00
Dashamir Hoxha 7879c7cb31 Update URLs by replacing /contributing with /contributing/code (#2657) 
* Update URLs by replacing /contributing with /contributing/code

* Rename contributing/code to contributing/core
 2019-10-24 02:12:39 +03:00
Ruslan Kuprieiev bd5a8a5cef dvc: bump to 0.64.0 0.64.0 2019-10-23 22:48:33 +03:00
Ruslan Kuprieiev 480a612da4 output: base: don't forget to return empty cache (#2656) 
* output: base: don't forget to return empty cache

CC @Suor

Fixes https://gist.github.com/leimbernon/21924e02a998e3f1604990f8a7824c37

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* gc: add test for missing .dir cache file
 2019-10-23 22:47:50 +03:00