Commit Graph
2845 Commits
Author SHA1 Message Date
Mr. Outis d42c8fbd14 repro: persisting outputs invalidates build cache
Fix #1852
2019-04-24 16:45:03 -07:00
Ruslan Kuprieiev 165f6472ca Merge pull request #1925 from efiop/1655
stage: commit cached stage to avoid data duplication
2019-04-24 22:22:48 +03:00
Ruslan Kuprieiev 9c77f2818c stage: commit cached stage to avoid data duplication
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 21:14:08 +03:00
Ruslan Kuprieiev b8018c16a7 Merge pull request #1892 from efiop/1654
support external ssh directories as dependencies and outputs
2019-04-24 05:57:06 +03:00
Ruslan Kuprieiev 37ee55b1f8 test: test ssh
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 49f87a5bff use ValueError instead of JSONDecodeError
JSONDecodeError is only available in never python 3 versions and still
inherits from ValueError, so we could just use it.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev b97b99cf15 test: func: use pytest for outputs/dependencies
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 44b95f23a3 remote: ssh: support directories
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 00cfb04fea remote: generalize file/dir handling
Fixes #1654
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 41b67c3730 move: move directory into an existing directory
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 04:56:33 +03:00
Alexander Schepanovski ee8d9a1312 dvc: improve apply_diff() docstring
- explain how it works and why is it needed
- fix typo
2019-04-24 02:45:02 +03:00
Alexander Schepanovski 277b9e566c test: use caplog.at_level() to restore log level after test 2019-04-24 02:45:02 +03:00
pared dd8598e53e tree: walk: operate on tree absolute root 2019-04-23 02:38:23 +03:00
Alexander Schepanovski 8ef52f92ee dvc: add meta key to stages to store user data 2019-04-22 16:26:57 +03:00
ag613915cao 47fb2d3a3b Add test for ssh connection
This commit intends to mock SSH server that not depends on
OS which the test running on for testing purpose.
2019-04-21 04:22:59 +03:00
Alexander Schepanovski 0c867399bb dvc: retain formatting and comments in .dvc files (#1885)
* test: remove never used copy of load_stage_file()

* dvc: save comments and custom formatting in stage files

* Add a couple of TODOs

* test: test repro doesn't clear comments and formatting in .dvc

As part of this I started writing pytest-like tests and added commonly
used fixtures.

* test: simplify and prettify dvc formatting retained test

Also test that line comments after md5 fields are retained.

* test: fix tests

* dvc: fix stage and metrics opens not closed

* dvc: always use utf-8 in stage files

It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.

* test: unit test apply_diff()

* test: remove TODOs and ASK

* dvc: polish dvc-formatting PR

* dvc: move stage load/dump utils to a new stage utils file

* dvc: clean up apply_diff()

* test: test stage md5 ignores comments

* test: remove pytests starting marker
2019-04-20 01:12:53 +03:00
Ruslan Kuprieiev b057c814ea Merge pull request #1895 from vyloy/master
import: support the '--file' option
2019-04-18 14:18:17 +03:00
vyloy 60c89adeb5 import: use os.path.join instead of hardcode path in the test 2019-04-18 09:25:38 +08:00
vyloy 01a28f2de9 import: improve some details about the '--file' option. 2019-04-18 08:56:14 +08:00
vyloy b07c386d02 import: support the '--file' option 2019-04-18 08:56:14 +08:00
Ruslan Kuprieiev 30573c4546 Merge pull request #1899 from efiop/pkg
pkg: comply with design of other dvc commands
2019-04-18 03:31:21 +03:00
Ruslan Kuprieiev 9618b32fe8 pkg: comply with design of other dvc commands
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-17 23:24:27 +03:00
Ivan Shcheklein 817c8a9fe9 Merge pull request #1897 from shcheklein/m
readme: new logo header
2019-04-16 17:00:42 -07:00
Ivan Shcheklein 0180e7c95b readme: new logo header 2019-04-16 16:57:30 -07:00
Ruslan Kuprieiev 6f11f1f51f test: move func tests to tests/func
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-16 20:19:22 +03:00
Yao Adzaku 6c730dd5cb pipeline: print locked DVC stages (#1882)
* pipeline: add option to display locked DVC stages.

The '--locked' option xplicitly displays only locked DVC stages.

Fixes #1864

* pipeline: support short option '-l' for displaying locked stages

* pipeline: minor cosmetic tweak

* pipeline: reformat code with black

* pipeline: test printing of locked stages

* pipeline: use caplog to ensure command outputs everything as expected

* pipeline: cleanup test

* pipeline: more strict comparison of test output
2019-04-16 16:46:50 +03:00
Paulo Olveira 75b4d0e483 s3: fix for corrupted cache on multipart uploads (#1867)
* s3: fixed wrong etag when copying multipart objects

The etag of multipart objects depends of the number of parts, when copying to the cache we should do so in the same number of parts that the original object was moved/uploaded in.

Fixes part of #1410

* s3: added check on copy for equal etag

* s3: added specific exception for ETag mismatch

* s3: use multipart copy to preserve etags

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: add tests for etag preservation on s3

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: requirements: use dev version moto

Specifically because of this https://github.com/spulec/moto/issues/1941

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: requirements: install dev moto without -e

Signed-off-by: Ruslan Kuprieiev <[email protected]>

* test: stop using moto

Turned out to be quite buggy.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-15 21:03:22 +03:00
Ruslan Kuprieiev ec9e2c63f2 Update PULL_REQUEST_TEMPLATE.md 2019-04-15 20:59:23 +03:00
Ruslan Kuprieiev dee04ad647 Create PULL_REQUEST_TEMPLATE.md 2019-04-15 20:24:35 +03:00
Ruslan Kuprieiev 30235b3d45 Delete pull_request_template.md 2019-04-15 20:23:41 +03:00
Ruslan Kuprieiev 64e481ec48 github: add PR template with a small checklist 2019-04-15 20:18:04 +03:00
Ruslan Kuprieiev a74e4226a1 Merge pull request #1820 from pared/1499
ignore: introduce .dvcignore
2019-04-15 17:43:21 +03:00
Ruslan Kuprieiev bcd9373ebc Merge pull request #1841 from pared/1257
cache: maker reflink, copy, default links
2019-04-15 17:42:59 +03:00
pared c467ccebcb ignore: introduce .dvcignore, apply when searching for stage files 2019-04-15 14:31:34 +02:00
Ruslan Kuprieiev f20818cd73 Merge pull request #1886 from pared/fix_file_naming_add
add: dont use 'file' as var name during iteration
2019-04-15 14:34:36 +03:00
pared 35360cfecd add: dont use 'file' as var name during iteration 2019-04-15 13:25:06 +02:00
pared 074b6eaa9e test: move corrupted hardlink cache test test_repro -> test_run 2019-04-15 12:55:50 +02:00
pared 9049475412 test_repro: make cache use hardlink 2019-04-15 11:27:44 +02:00
pared e505c38f20 cache: make reflink, copy default links 2019-04-15 11:27:44 +02:00
Ruslan Kuprieiev 8f27c14a8c Merge pull request #1881 from shcheklein/m
diff: cli: add documentation link
2019-04-13 05:07:39 +03:00
Ivan Shcheklein 7785e48d67 diff: cli: add documentation link 2019-04-12 17:37:12 -07:00
Ruslan Kuprieiev c5cb02012f Merge pull request #1879 from maggyero/patch-1
Update README.rst
2019-04-12 18:55:11 +03:00
Géry Ogam a09863afbe Update README.rst 2019-04-12 17:44:03 +02:00
Ruslan Kuprieiev 700199edc0 Merge pull request #1875 from mroutis/try-to-fix-logger-test
tests: be paranoic with caplog
2019-04-12 14:07:47 +03:00
Mr. Outis 1cac81430e tests: be paranoic with caplog 2019-04-11 14:09:02 -05:00
Ruslan Kuprieiev e4290a72e5 dvc: bump to 0.35.7 0.35.7 2019-04-11 18:12:13 +03:00
Ruslan Kuprieiev 7fa869a7f8 Merge pull request #1872 from mroutis/fix-1853
dvc: deprecate --remove-outs
2019-04-11 07:01:22 +03:00
Mr. Outis f417f49c6e dvc: deprecate --remove-outs
Close #1853
2019-04-10 18:58:32 -05:00
Ruslan Kuprieiev d5a3e5db1a Merge pull request #1870 from mroutis/fix-1869
logger: use exception instead of error
2019-04-11 00:35:50 +03:00
Mr. Outis 059aef3ef0 logger: use exception instead of error
The problem was happening because `logger.error` doesn't include any
`exc_info`, and the `parse_exc` logic was previously to rely on the log
record's `exc_info` instead of the global stack trace.

There's still use cases when you want to just `log` an error without
printing the stack trace, those are left untouched.
You can search for them with: `grep -R 'logger.error' dvc/`

---

Fix #1869
2019-04-10 16:26:14 -05:00