Commit Graph
57 Commits
Author SHA1 Message Date
ag613915cao 47fb2d3a3b Add test for ssh connection
This commit intends to mock SSH server that not depends on
OS which the test running on for testing purpose.
2019-04-21 04:22:59 +03:00
Alexander Schepanovski 0c867399bb dvc: retain formatting and comments in .dvc files (#1885)
* test: remove never used copy of load_stage_file()

* dvc: save comments and custom formatting in stage files

* Add a couple of TODOs

* test: test repro doesn't clear comments and formatting in .dvc

As part of this I started writing pytest-like tests and added commonly
used fixtures.

* test: simplify and prettify dvc formatting retained test

Also test that line comments after md5 fields are retained.

* test: fix tests

* dvc: fix stage and metrics opens not closed

* dvc: always use utf-8 in stage files

It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.

* test: unit test apply_diff()

* test: remove TODOs and ASK

* dvc: polish dvc-formatting PR

* dvc: move stage load/dump utils to a new stage utils file

* dvc: clean up apply_diff()

* test: test stage md5 ignores comments

* test: remove pytests starting marker
2019-04-20 01:12:53 +03:00
pared c467ccebcb ignore: introduce .dvcignore, apply when searching for stage files 2019-04-15 14:31:34 +02:00
Mr. Outis 1cac81430e tests: be paranoic with caplog 2019-04-11 14:09:02 -05:00
Mr. Outis 727bfd955e version: privatize version.py methods and vars 2019-04-07 16:48:34 -05:00
Ruslan Kuprieiev 54f7b43aac dvc: version: detect release
See failed release at [1]. Previously we used to look at TRAVIS_TAG env
var, but now we will try to detect release by matching current tag with
base version.

[1] https://travis-ci.com/iterative/dvc/builds/107365596

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-07 22:55:20 +03:00
Ruslan Kuprieiev 7c540da5c4 daemon: don't spawn recursively
Fixes #1803

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-07 20:12:23 +03:00
Mr. Outis 921b39dbfe tests: use pytest caplog instead of custom mock
Close #1753
2019-04-06 20:35:30 -06:00
Ruslan Kuprieiev 6124a23138 Merge pull request #1831 from mroutis/fix-1824
dvc: refactor package versioning logic
2019-04-07 03:16:39 +03:00
Mr. Outis b4d0d37ac8 dvc: refactor package versioning logic
- Use `version.py` to compute a dynamic version stored in `__version__`.
  Version information was scattered across the source code
  (`VERSION_BASE`, `VERSION`, `__version__`, and `version.py`).

- Read `__version__` from `setup.py` and **pin** it using custom
  `build_py` command class.
  This way, we can maintain consistancy between the build version
  and the runtime one.
  Also, this gives us the ability to run `setup.py` without depending on
  on `dvc` itself.

Close #1824
2019-04-06 18:06:11 -06:00
Ruslan Kuprieiev 8ef5976ac8 add: check if output is local when generating stage name
When adding a remote file with

```
$ dvc add s3://bucket/path
```

it results in

```
Error: failed to add file - is local is not supported for s3
```

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-06 03:25:53 +03:00
pared 0f5216cb14 checkout: reset progress bar on checkout 2019-03-31 21:48:21 +02:00
Andrew Grigorev 7d79043bbb dvc.scm.scm_context -> dvc.repo.scm_context.scm_context
The `dvc/scm/__init__.py:scm_context` works with `Repo` objects and
is used mostly in `dvc/repo/...`, so it is better to put it in
`dvc/repo/scm_context.py`.

Fix #1769
2019-03-22 19:53:41 +03:00
Andrew Grigorev 9e6c52079b Abstract access to file objects
Abstract access to filesystem and its implementation to access files and
particulary Stage objects in different git branches without `git
checkout` execution.
2019-03-21 19:47:42 +03:00
Paweł Redzyński ee16409495 checkout: progress bar (#1713) 2019-03-20 13:39:10 +02:00
Ivan Shcheklein f085e01717 try to fix progess bar test in a parallel pytest environment 2019-03-17 19:33:09 -07:00
Ivan Shcheklein 75421f0f21 migrate to pytest 2019-03-17 14:49:49 -07:00
Mr. Outis 21778f6719 stage: convert wdir path to posix
Fix #1721
2019-03-14 14:40:28 -06:00
pared 4897a6a44f add: prevent adding git files 2019-03-13 14:11:59 +01:00
pared f8711b3faa repo/scm: move tracking reminder from repo to scm 2019-03-11 23:13:34 +01:00
Ivan Shcheklein ee7cb41fcb add tests to properly test wdir in checksum calculation 2019-03-02 16:37:00 -08:00
Ruslan Kuprieiev cce25db97e Merge pull request #1672 from shcheklein/add_wdir_tests
Add more tests for wdir/fname new logic
2019-03-01 16:15:27 +02:00
Ruslan Kuprieiev 0913a65661 Merge pull request #1668 from prihoda/patch-1
Log computed stage MD5 in verbose mode
2019-03-01 15:27:47 +02:00
prihoda d63db35100 Log computed stage MD5 in verbose mode
Sometimes it is handy to manually update the stage MD5 checksum after some non-breaking changes were made (e.g. renaming an output file). Logging the computed md5 hash in verbose mode can help experienced users to fix this problem.
2019-03-01 14:11:29 +02:00
Ivan Shcheklein e26db30f60 add more tests for wdir/fname new logic 2019-02-28 21:43:14 -08:00
pared f8ef28c4c6 stage: check if local path contains symlink and, if so, treat added files as relpath 2019-02-28 15:00:05 +01:00
Ruslan Kuprieiev 8ff27454e6 Merge pull request #1660 from mroutis/fix-1602
logger: Format progress bar interruption
2019-02-28 14:52:18 +02:00
Mr. Outis 93920baaec logger: Format progress bar interruption
Fix #1602
2019-02-27 21:24:22 -06:00
pared 4464e29c15 remote local/state : add check for cache dir metadata changes 2019-02-25 18:35:02 +01:00
Ruslan Kuprieiev 63f87ba9fe Merge pull request #1653 from efiop/output
output/deps: raise proper exception when file/dir doesn't exist
2019-02-23 09:32:00 +02:00
Ruslan Kuprieiev 9ca00adde8 dvc: modify flake8 configuration
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-22 19:36:19 +02:00
Ruslan Kuprieiev 9961956c01 output/deps: raise proper exception when file/dir doesn't exist
From discussion on discord
https://discordapp.com/channels/485586884165107732/485596304961962003/548083580052701205

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-22 14:59:01 +02:00
Ruslan Kuprieiev e5107b1da3 ssh: external-outs: check if cache exists before copying
Fixes #1635

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-19 21:33:47 +02:00
Mr. Outis 5e7a152b53 ssh: use sftp when possible
Related #830 [1/2]
2019-02-18 19:03:06 -06:00
Ruslan Kuprieiev 01a120f9fc dvc: introduce dvc commit
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-16 21:19:36 +02:00
Ruslan Kuprieiev 70c21f7f3e ssh: handle url without path
https://discordapp.com/channels/485586884165107732/485596304961962003/545702950350356502

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-15 04:38:45 +02:00
Ruslan Kuprieiev ba8d83e9b1 dvc: refactor dvc project class
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-13 03:31:23 +02:00
Ruslan Kuprieiev 91ed40fbfa updater: don't use io.open when working with json
Otherwise we get this on python2:
```
======================================================================
ERROR: test_fetch (tests.unit.updater.TestUpdater)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/efiop/.pyenv/versions/2.7.15/envs/2.7.15-dvc/local/lib/
python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/efiop/git/dvc/tests/unit/updater.py", line 34, in test_fetch
    self.dvc.updater.fetch(detach=False)
  File "/home/efiop/git/dvc/dvc/updater.py", line 77, in fetch
    self._with_lock(self._get_latest_version, "fetching")
  File "/home/efiop/git/dvc/dvc/updater.py", line 39, in _with_lock
    func()
  File "/home/efiop/git/dvc/dvc/updater.py", line 91, in _get_latest_version
    json.dump(convert_to_unicode(info), fobj)
  File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
TypeError: write() argument 1 must be unicode, not str
-------------------- >> begin captured logging << --------------------
```

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-12 01:40:07 +02:00
Mr. Outis ec77bfa1af remote: define command error on the base
fix: #1484
2019-02-09 12:03:17 -06:00
Thibault de Boissiere 2c67b59fc0 checkout: Suggest git checkout when dcv checkout fails (#1580) 2019-02-05 23:46:13 +02:00
Mehdi Hamoumi ffc94cc471 Add GCP credentials environment variable to config (#1567) 2019-02-05 01:33:04 +02:00
Ruslan Kuprieiev 4896f08ddf dvc: comply with both flake8 and black
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-04 00:10:36 +02:00
Ruslan Kuprieiev ba091a6d34 checkout: handle EOFError when running in a git hook
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-31 22:56:48 +02:00
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-30 03:48:36 +02:00
Mr. Outis 0d46921ec7 py2: add unicode literals
Fix #1512
2019-01-27 20:40:21 -06:00
Ruslan Kuprieiev 720d5d675f Merge pull request #1530 from efiop/master
remote: gs: don't forget to set projectname
2019-01-26 14:08:07 +02:00
Ruslan Kuprieiev e3304e14fa daemon: set PYTHONPATH
Fixes #1525

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-26 01:54:45 +02:00
Ruslan Kuprieiev a314209add remote: gs: don't forget to set projectname
Fixes #1529

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-26 00:30:08 +02:00
Ruslan Kuprieiev f5aa0da0ae stage: don't exclude md5s for deps/outs
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-24 17:02:07 +02:00
Ruslan Kuprieiev a108249db3 RFC: state: comply with pylint
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-23 03:35:46 +02:00