Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,950 Commits 7 Branches 579 Tags
c25715dc729f49e5885564c28d0bdb01b0e1ff02
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Mr. Outis 0103f3330f sftp: only remove src after an atomic copy during move 
Shout-out to @Suor for pointing this out
 2019-11-05 10:02:41 -06:00
Mr. Outis 16164e65ad sftp: move -> use atomic copying 2019-11-01 17:01:50 -06:00
Mr. Outis 8f7000056b sftp: move -> fallback to copy when sftp.rename fails 2019-11-01 13:26:03 -06:00
Mr. Outis 283f848822 sftp: make move an atomic operation 2019-11-01 13:17:22 -06:00
Mr. Outis ab1ead881b sftp: make move across drives possible 
Fix #2704
 2019-11-01 12:40:16 -06:00
Mr. Outis 732ed904ce tests: same ETag across buckets 2019-10-30 11:42:39 -06:00
Mr. Outis 27b63e2674 s3: support for directories (#2619) 
* s3: list_objects -> list_objects_api

* s3: use path_info for _list_paths

* s3: separate _list_paths from _list_objects

* s3: support max items for listing objects/paths

* s3: take into account directories on `exists`

* s3: overwrite `isdir` method

* s3: custom collect directory entry for s3

* s3: strip checksum when collecting directories

* remote: change walk implementation to walk_files

* s3: add comment about isdir logic

* s3: use `dir_path` correctly on `isdir`

* tests: add a few unit tests for S3

* s3: use path_info join instead of posixpath

* tests: use parametrize for s3 tests

* remote: use posixpath accordingly

* tests: remove undeeded AWS variables from S3 test

* tests: rearrange parametrize values for readability

* tests: add encoding, use a single s3 instance, for loop instead of parameterize

* py2: fix encoding
 2019-10-25 03:41:04 +03:00
Mr. Outis 13a7e0f978 logger: send warning messages to stderr (#2612) 
* logger: send warning messages to stderr

Fix #2561

* logger: filter warnings for stdout
 2019-10-16 03:12:00 +03:00
Mr. Outis cfecb75c4c setup: pathspec@1d321d -> pathspec>=0.6.0 2019-10-03 21:55:21 -05:00
Mr. Outis e6ccf32189 scripts: upgrade pip before installing dependencies 2019-10-03 17:26:29 -05:00
Mr. Outis 1f6d8964b5 setup: pathspec@master -> pathspec@1d321d 2019-10-03 12:25:12 -05:00
Mr. Outis a47ea23561 git: use escape method from GitWildMatchPattern 2019-10-03 12:10:52 -05:00
Mr. Outis 9496f34f45 tests: Windows specific filename for gitignore 2019-09-30 11:28:38 -05:00
Mr. Outis 80cb1c919a git: escape question mark character 2019-09-30 11:27:53 -05:00
Mr. Outis e692be2dc9 sftp: add a more descriptive error on mkdirs (#2554) 
* sftp: add a more descriptive error on mkdirs

Fix #2535

* sftp: raise exception instead of logging one
 2019-09-30 19:24:31 +03:00
Mr. Outis a9201f8237 git: add a note referencing gitignore pattern 2019-09-30 11:21:39 -05:00
Mr. Outis c9c36c2049 git: escape filename before ignoring it 
Fix #2043
 2019-09-29 23:16:46 -05:00
Mr. Outis 39dc79b418 main: fix UnicodeError misplacement 2019-09-27 18:20:00 -05:00
Mr. Outis 0bdad65f04 main: make sure to raise UinocedeError for py3 2019-09-27 14:18:08 -05:00
Mr. Outis 1763607a88 py2: display friendly error when raising a UnicodeError 2019-09-26 19:50:15 -05:00
Mr. Outis 5c5dfed68f .github/CONTRIBUTING.md -> CONTRIBUTING.md 2019-09-25 21:33:46 -05:00
Mr. OutisandRuslan Kuprieiev ae90904fb3 pull: download data from imported stages (#2499) 
* wip - pull is working

* use fetch instead of reproduce

* test pull and fetch

* style changes

* fetch: catch config error on fetch instead of get_remote

* output: remove unused guard clause

* fetch: define a more specific error instead of ConfigError

* fetch: display number of errors correctly

* exceptions: remove "Failed" prefix to download/upload

* fetch: catch other exceptions that might halt the download

* fetch: log catched exceptions

* fetch: reformat exception

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* fix formatting
 2019-09-22 01:28:12 +03:00
Mr. Outis 3c5bcbd463 local: mark static methods 2019-09-09 12:26:48 -05:00
Mr. Outis 2a847ef25c tests: ignore ssh's hardlink and copy tests on windows 2019-09-09 11:20:29 -05:00
Mr. Outis e0fd808ab0 setup: iterative/mock-ssh-server -> pypi mock-ssh-server 2019-09-09 11:15:49 -05:00
Mr. OutisandRuslan Kuprieiev 5103dfd4aa tests: fix skipif on ssh/test_connection.py 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-09-07 21:38:50 -05:00
Mr. Outis b4507444d5 version: adjust get_fs_type to discover all partitions 2019-09-06 19:27:23 -05:00
Mr. Outis 190fbaa63b get: no relpath on output not found (#2435) 
* get: no relpath on output not found

related #2430

* get: catch output not found
 2019-09-06 23:14:07 +03:00
Mr. Outis 05cbe0c922 tests: support windows 2019-09-06 14:17:44 -05:00
Mr. Outis 737f1e141d tests: use repo_dir instead of custom fixtures 2019-09-06 13:28:48 -05:00
Mr. Outis b66996e0a1 ssh: support reflinks in Darwin 2019-09-06 13:24:25 -05:00
Mr. Outis b903fe8023 get: show error message when path is a DVC-file (#2470) 
* get: show error message when path is a DVC-file

Fix #2468

* exceptions: rename output for data artifact

* get: use method to indetify if file is a DVC-file

* exceptions: change GetDVCFileError message
 2019-09-06 21:11:23 +03:00
Mr. Outis dd1d5bdbb0 tests: PosixPath to string in chdir_tmp fixture 2019-09-06 12:57:06 -05:00
Mr. Outis 73133e8c7a tests: [re] fix hardlink stuff with HDFS 2019-09-06 12:35:38 -05:00
Mr. Outis 23f2650b24 ssh: remove write_through in TextIOWrapper 2019-09-06 12:07:22 -05:00
Mr. Outis f97c33178f tests: use copy as cache type for HDFS 2019-09-06 12:05:53 -05:00
Mr. Outis 6eb26de856 fix tests 2019-09-06 11:32:31 -05:00
Mr. Outis 38d4cbf5c1 fix broken merge 2019-09-06 09:42:56 -05:00
Mr. Outis 42168d34f3 Merge branch 'master' into close-1652 2019-09-06 09:29:54 -05:00
Mr. Outis 739236eab0 ssh: accept only wb mode for open 2019-09-06 08:48:22 -05:00
Mr. Outis d7da4fcd37 fix formatting 2019-09-05 10:52:15 -05:00
Mr. Outis 369146a926 Merge branch 'master' into close-1652 2019-09-05 10:47:18 -05:00
Mr. Outis 0c231b68d7 fix formatting 2019-09-04 22:07:02 -05:00
Mr. Outis 079d589b59 local: use path_info for link operations 2019-09-04 19:38:16 -05:00
Mr. Outis 2fe6fc1ba6 refactor: use a module for slow link detection 2019-09-04 19:32:55 -05:00
Mr. Outis 9295bbd160 ssh: support opening files for writing 2019-09-04 15:13:26 -05:00
Mr. Outis 1ae9078fee revert remove assertions from open 2019-09-04 12:35:54 -05:00
Mr. Outis 1f0a57d4af remote: remove unnecessary variable in _try_links 2019-09-04 09:48:46 -05:00
Mr. Outis cdee18a7a6 fix formatting 2019-09-03 00:27:10 -05:00
Mr. Outis 3c69dc9fd5 tests: unit ssh symlink 2019-09-03 00:04:44 -05:00
Mr. Outis 88574f300e remote: remove unecessary else clause 2019-09-03 00:01:38 -05:00
Mr. Outis 69638527cb setup: mroutis/mock-ssh-server -> iterative/mock-ssh-server 2019-09-03 00:01:14 -05:00
Mr. Outis bbfaf3a22b setup: use mock-ssh-server version supporting lstat 2019-09-02 23:21:20 -05:00
Mr. Outis da65f8fe9a tests: functional ssh symlink 2019-08-31 13:34:48 -05:00
Mr. Outis 7b4584f86c ssh: stat -> lstat 2019-08-31 13:02:17 -05:00
Mr. Outis 70010bbb31 remote: remove unecessary makedirs on checkout 2019-08-31 12:59:34 -05:00
Mr. Outis 8d41833c7c path_info: remove urljoin leftover 2019-08-31 03:32:23 -05:00
Mr. Outis 0fc83be9ff tests: ssh hardlink optimization 2019-08-31 03:23:35 -05:00
Mr. Outis 0361b1b9c0 tests: ssh connection link methods 2019-08-31 01:56:00 -05:00
Mr. Outis 5f89242791 Merge branch 'master' into close-1652 2019-08-31 00:27:28 -05:00
Mr. Outis 40c61b3aaa remote: abstract link logic to base 2019-08-30 23:51:20 -05:00
Mr. Outis 83a9a6362b remote: remove unecessary + on open mode 2019-08-24 11:45:43 -05:00
Mr. Outis 9c9079753b remote: use getattr instead of dict to get link methods 2019-08-24 11:44:50 -05:00
Mr. Outis 745b2e13a0 [wip] remote: abstract link logic to base 2019-08-24 01:47:58 -05:00
Mr. Outis 84872d231a docs: explain potential of hitting link limit 2019-08-24 01:20:26 -05:00
Mr. Outis 1be63dd93e ssh: implement interface for fs related operations 2019-08-24 00:16:37 -05:00
Mr. Outis d13b5a14f8 local: use PathInfo for link operations 2019-08-23 22:45:54 -05:00
Mr. Outis 02cef7b9a3 ssh: makedirs before symlink 2019-08-23 22:18:48 -05:00
Mr. Outis 521647e47e ssh: implement slow_link_detection 2019-08-23 21:54:53 -05:00
Mr. Outis f4563bdd85 ssh: support reflinks and hardlinks 2019-08-23 21:20:50 -05:00
Mr. Outis 110da0c366 ssh: adapt link logic and support symlinks 2019-08-22 01:00:53 -05:00
Mr. OutisandRuslan Kuprieiev 8918bdc383 http: add exception when trying to traverse remote (#2411) 
* http: add exception when trying to traverse remote

Fix #2380

* http: exception raise on `list_cache_paths` instead of `all`

`NotImplementedError` -> `RemoteActionNotImplemented`

* http: raise exception on `__init__`

* set default no_traverse to True on remote base

* tests: explicitly use traverse on data cloud and no traverse with HTTP

* remote: define a new RemoteConfigError

* remote: reuse ConfigError instead of RemoteConfigError

* tests: set no_traverse to False with SSHMocked

* deepsource: fix minor issue with spacing

* remove blankspace

Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-08-20 00:47:15 +03:00
Mr. Outis a17caefd50 integrate @efiop 's suggestions 2019-08-10 11:37:35 -05:00
Mr. Outis 163d872c92 fix tests for windows 2019-08-09 22:46:25 -05:00
Mr. Outis fa04c27580 add tests for recursive operations and file not found 2019-08-09 22:06:32 -05:00
Mr. Outis 0e46edd9bb fail silently but with a more sound error message 
* Add a `_missing` key to the return value of `repo.metrics.show` to
include targets not recognized as metrics or not found.
 2019-08-09 21:55:19 -05:00
Mr. Outis d4ef110794 exit with 1 if targets were not found 2019-08-06 21:10:48 -05:00
Mr. OutisandRuslan Kuprieiev 21a2d3c6ff remove no-op 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-08-05 15:30:28 -05:00
Mr. Outis 44f1cc6200 bring back debug message 2019-08-05 15:27:17 -05:00
Mr. OutisandRuslan Kuprieiev dc5dd1e443 change set.update to set.add 
created a set with `{}` but resulted in a dictionary, making me think about `set` not supporting the `add` method.

Thanks for the correction, @efiop !

Co-Authored-By: Ruslan Kuprieiev <[email protected]>
 2019-07-27 17:44:45 -05:00
Mr. Outis 405dfde23f use set instead of list for deduplication 2019-07-27 16:34:22 -05:00
Mr. Outis 9ca5452603 scm: reset tracked files after every operation 
Fix #1475
 2019-07-23 15:10:48 -05:00
Mr. Outis dda8368f20 ssh: increase jobs and parallelize batch_exists 
- SSH number of jobs increased from 4 to 8.

- `batch_exists` now opens all the SFTP sessions available to
distribute the work between them. Number of sessions is restricted
by the `MaxSessions` option configurable by the `sshd_config`.

- Fix #2104

With the optimizations mentioned above, we increased the speed by 8x
(using the default `MaxSessions` configuration -- 10).

Shout-outs to @efiop, @shcheklein, and @dmpetrov for suggesting the
optimization techniques.
 2019-06-19 17:45:59 -05:00
Mr. Outis 223f0f6c0f status: show a progress bar while gathering info 
Fix #1762
Fix #1825
 2019-06-14 16:39:30 -05:00
Mr. Outis d4f0d311fc remote: add other way to perform cache_exists 
Close #1762
 2019-06-11 08:55:48 -05:00
Mr. Outis e3b3f3f371 progress: bug fix with --quiet 
Related to #1377
 2019-05-29 16:45:10 -07:00
Mr. Outis 037f32c8cc docs: add some docstrings about computing status (#2048) 2019-05-23 16:16:02 -07:00
Mr. Outis 7c0c89a688 upgpkg: requests >= 2.22.0 
Close #2008
 2019-05-16 13:28:36 -07:00
Mr. Outis 15f074fd96 refactor: rely on RemoteBasic._changed_dir_cache 
Defining `_changed_cache_dir` for the sole purpose of executing
code before `_changed_dir_cache` was not that clear.

Now, it overwrites `_changed_dir_cache` but reference the
base method.
 2019-05-15 19:02:58 -07:00
Mr. Outis a91b198954 status: optimize the _fill_statuses method 
Close #1975
 2019-05-10 19:20:13 -07:00
Mr. Outis c39c2f087a install: append commands if hook already exists 
Close #1964
 2019-05-10 16:20:01 -07:00
Mr. Outis 9978bc87a4 logger: fix bug that ignored the progress bar 
Close #1973
 2019-05-10 00:17:32 -07:00
Mr. Outis 30e38c91dc tests: use UTF-8 when creating metrics files (#1956) 2019-05-01 20:07:37 -07:00
Mr. Outis 60e83285e1 config: implement bool_schema 💅 
Extracted from https://github.com/iterative/dvc/pull/1804/
 2019-05-01 17:37:47 -07:00
Mr. Outis 29f58bda9b status: optimization for already downloaded files 
Fix #1839
 2019-04-30 17:09:04 -07:00
Mr. Outis d3502ebfe3 add: warn when recursively adding large files 
Close #1758
 2019-04-30 12:33:28 -07:00
Mr. Outis c0104e2e61 repro: implement downstream option (#1926) 
* repro: implement downstream option

Close #1555

* Update scripts/completion/dvc.zsh

Co-Authored-By: mroutis <[email protected]>

* Update dvc/command/repro.py

Co-Authored-By: mroutis <[email protected]>
 2019-04-25 04:50:44 +03:00
Mr. Outis d42c8fbd14 repro: persisting outputs invalidates build cache 
Fix #1852
 2019-04-24 16:45:03 -07:00
Mr. Outis 1cac81430e tests: be paranoic with caplog 2019-04-11 14:09:02 -05:00
Mr. Outis f417f49c6e dvc: deprecate --remove-outs 
Close #1853
 2019-04-10 18:58:32 -05:00