Commit Graph
2855 Commits
Author SHA1 Message Date
Ivan Shcheklein 27bf6e2928 readme: add donate patreon badge 2019-04-26 12:33:45 -07:00
Ruslan Kuprieiev 360e459161 Merge pull request #1939 from Suor/gc-optimize
dvc: some minor optimizations
2019-04-26 19:02:05 +03:00
Alexander Schepanovski cdba077b61 dvc: some minor optimizations 2019-04-26 19:10:08 +07:00
Ruslan Kuprieiev 3a183f5303 Merge pull request #1911 from ag613915cao/enable_ssh_test_on_windows
Mocked SSH server and enable ssh test on windows
2019-04-26 14:24:00 +03:00
ag613915cao fd385c913b This commit intends to:
- Keep current SSH remote connection test on real system
- Faked and Mocked SSH server which running on both Linux and Windows
- Add corresponding tests

Fixed #1704
2019-04-26 18:07:16 +07:00
Ruslan Kuprieiev edefb5470f ssh: respect no_progress_bar on download/upload
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-26 04:45:52 +03:00
Ruslan Kuprieiev 65d31db8db state: remove redundant lines 2019-04-25 16:53:24 +03:00
Alexander Schepanovski 7478e7d4b0 test: clean up test dirs (#1904)
* test: clean up test dirs

Closes #1770. Partially does #1888.

* dvc: close anything opened

* test: finalize git repo explicitly to remove test dir later

This was previously an issue on Windows.
2019-04-25 14:55:39 +03:00
Mr. Outis c0104e2e61 repro: implement downstream option (#1926)
* repro: implement downstream option

Close #1555

* Update scripts/completion/dvc.zsh

Co-Authored-By: mroutis <[email protected]>

* Update dvc/command/repro.py

Co-Authored-By: mroutis <[email protected]>
2019-04-25 04:50:44 +03:00
Ruslan Kuprieiev 44e20f1aa8 Merge pull request #1877 from mroutis/fix-1852
repro: persisting outputs invalidates build cache
2019-04-25 02:56:27 +03:00
Mr. Outis d42c8fbd14 repro: persisting outputs invalidates build cache
Fix #1852
2019-04-24 16:45:03 -07:00
Ruslan Kuprieiev 165f6472ca Merge pull request #1925 from efiop/1655
stage: commit cached stage to avoid data duplication
2019-04-24 22:22:48 +03:00
Ruslan Kuprieiev 9c77f2818c stage: commit cached stage to avoid data duplication
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 21:14:08 +03:00
Ruslan Kuprieiev b8018c16a7 Merge pull request #1892 from efiop/1654
support external ssh directories as dependencies and outputs
2019-04-24 05:57:06 +03:00
Ruslan Kuprieiev 37ee55b1f8 test: test ssh
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 49f87a5bff use ValueError instead of JSONDecodeError
JSONDecodeError is only available in never python 3 versions and still
inherits from ValueError, so we could just use it.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev b97b99cf15 test: func: use pytest for outputs/dependencies
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 44b95f23a3 remote: ssh: support directories
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 00cfb04fea remote: generalize file/dir handling
Fixes #1654
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 41b67c3730 move: move directory into an existing directory
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 04:56:33 +03:00
Alexander Schepanovski ee8d9a1312 dvc: improve apply_diff() docstring
- explain how it works and why is it needed
- fix typo
2019-04-24 02:45:02 +03:00
Alexander Schepanovski 277b9e566c test: use caplog.at_level() to restore log level after test 2019-04-24 02:45:02 +03:00
pared dd8598e53e tree: walk: operate on tree absolute root 2019-04-23 02:38:23 +03:00
Alexander Schepanovski 8ef52f92ee dvc: add meta key to stages to store user data 2019-04-22 16:26:57 +03:00
ag613915cao 47fb2d3a3b Add test for ssh connection
This commit intends to mock SSH server that not depends on
OS which the test running on for testing purpose.
2019-04-21 04:22:59 +03:00
Alexander Schepanovski 0c867399bb dvc: retain formatting and comments in .dvc files (#1885)
* test: remove never used copy of load_stage_file()

* dvc: save comments and custom formatting in stage files

* Add a couple of TODOs

* test: test repro doesn't clear comments and formatting in .dvc

As part of this I started writing pytest-like tests and added commonly
used fixtures.

* test: simplify and prettify dvc formatting retained test

Also test that line comments after md5 fields are retained.

* test: fix tests

* dvc: fix stage and metrics opens not closed

* dvc: always use utf-8 in stage files

It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.

* test: unit test apply_diff()

* test: remove TODOs and ASK

* dvc: polish dvc-formatting PR

* dvc: move stage load/dump utils to a new stage utils file

* dvc: clean up apply_diff()

* test: test stage md5 ignores comments

* test: remove pytests starting marker
2019-04-20 01:12:53 +03:00
Ruslan Kuprieiev b057c814ea Merge pull request #1895 from vyloy/master
import: support the '--file' option
2019-04-18 14:18:17 +03:00
vyloy 60c89adeb5 import: use os.path.join instead of hardcode path in the test 2019-04-18 09:25:38 +08:00
vyloy 01a28f2de9 import: improve some details about the '--file' option. 2019-04-18 08:56:14 +08:00
vyloy b07c386d02 import: support the '--file' option 2019-04-18 08:56:14 +08:00
Ruslan Kuprieiev 30573c4546 Merge pull request #1899 from efiop/pkg
pkg: comply with design of other dvc commands
2019-04-18 03:31:21 +03:00
Ruslan Kuprieiev 9618b32fe8 pkg: comply with design of other dvc commands
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-17 23:24:27 +03:00
Ivan Shcheklein 817c8a9fe9 Merge pull request #1897 from shcheklein/m
readme: new logo header
2019-04-16 17:00:42 -07:00
Ivan Shcheklein 0180e7c95b readme: new logo header 2019-04-16 16:57:30 -07:00
Ruslan Kuprieiev 6f11f1f51f test: move func tests to tests/func
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-16 20:19:22 +03:00
Yao Adzaku 6c730dd5cb pipeline: print locked DVC stages (#1882)
* pipeline: add option to display locked DVC stages.

The '--locked' option xplicitly displays only locked DVC stages.

Fixes #1864

* pipeline: support short option '-l' for displaying locked stages

* pipeline: minor cosmetic tweak

* pipeline: reformat code with black

* pipeline: test printing of locked stages

* pipeline: use caplog to ensure command outputs everything as expected

* pipeline: cleanup test

* pipeline: more strict comparison of test output
2019-04-16 16:46:50 +03:00
Paulo Olveira 75b4d0e483 s3: fix for corrupted cache on multipart uploads (#1867)
* s3: fixed wrong etag when copying multipart objects

The etag of multipart objects depends of the number of parts, when copying to the cache we should do so in the same number of parts that the original object was moved/uploaded in.

Fixes part of #1410

* s3: added check on copy for equal etag

* s3: added specific exception for ETag mismatch

* s3: use multipart copy to preserve etags

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: add tests for etag preservation on s3

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: requirements: use dev version moto

Specifically because of this https://github.com/spulec/moto/issues/1941

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: requirements: install dev moto without -e

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: stop using moto

Turned out to be quite buggy.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-15 21:03:22 +03:00
Ruslan Kuprieiev ec9e2c63f2 Update PULL_REQUEST_TEMPLATE.md 2019-04-15 20:59:23 +03:00
Ruslan Kuprieiev dee04ad647 Create PULL_REQUEST_TEMPLATE.md 2019-04-15 20:24:35 +03:00
Ruslan Kuprieiev 30235b3d45 Delete pull_request_template.md 2019-04-15 20:23:41 +03:00
Ruslan Kuprieiev 64e481ec48 github: add PR template with a small checklist 2019-04-15 20:18:04 +03:00
Ruslan Kuprieiev a74e4226a1 Merge pull request #1820 from pared/1499
ignore: introduce .dvcignore
2019-04-15 17:43:21 +03:00
Ruslan Kuprieiev bcd9373ebc Merge pull request #1841 from pared/1257
cache: maker reflink, copy, default links
2019-04-15 17:42:59 +03:00
pared c467ccebcb ignore: introduce .dvcignore, apply when searching for stage files 2019-04-15 14:31:34 +02:00
Ruslan Kuprieiev f20818cd73 Merge pull request #1886 from pared/fix_file_naming_add
add: dont use 'file' as var name during iteration
2019-04-15 14:34:36 +03:00
pared 35360cfecd add: dont use 'file' as var name during iteration 2019-04-15 13:25:06 +02:00
pared 074b6eaa9e test: move corrupted hardlink cache test test_repro -> test_run 2019-04-15 12:55:50 +02:00
pared 9049475412 test_repro: make cache use hardlink 2019-04-15 11:27:44 +02:00
pared e505c38f20 cache: make reflink, copy default links 2019-04-15 11:27:44 +02:00
Ruslan Kuprieiev 8f27c14a8c Merge pull request #1881 from shcheklein/m
diff: cli: add documentation link
2019-04-13 05:07:39 +03:00