Commit Graph
1364 Commits
Author SHA1 Message Date
Ruslan Kuprieiev 435095e35f Merge pull request #1914 from pared/1890
repo: different cache link types info on long lasting operations
2019-04-29 19:14:27 +03:00
pared f496ba4f03 remote: local: different cache link types info on long lasting linking 2019-04-26 10:17:16 -07:00
ag613915cao fd385c913b This commit intends to:
- Keep current SSH remote connection test on real system
- Faked and Mocked SSH server which running on both Linux and Windows
- Add corresponding tests

Fixed #1704
2019-04-26 18:07:16 +07:00
Mr. Outis c0104e2e61 repro: implement downstream option (#1926)
* repro: implement downstream option

Close #1555

* Update scripts/completion/dvc.zsh

Co-Authored-By: mroutis <[email protected]>

* Update dvc/command/repro.py

Co-Authored-By: mroutis <[email protected]>
2019-04-25 04:50:44 +03:00
Ruslan Kuprieiev 37ee55b1f8 test: test ssh
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 44b95f23a3 remote: ssh: support directories
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
Ruslan Kuprieiev 00cfb04fea remote: generalize file/dir handling
Fixes #1654
Fixes #1655

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-24 05:09:48 +03:00
ag613915cao 47fb2d3a3b Add test for ssh connection
This commit intends to mock SSH server that not depends on
OS which the test running on for testing purpose.
2019-04-21 04:22:59 +03:00
Alexander Schepanovski 0c867399bb dvc: retain formatting and comments in .dvc files (#1885)
* test: remove never used copy of load_stage_file()

* dvc: save comments and custom formatting in stage files

* Add a couple of TODOs

* test: test repro doesn't clear comments and formatting in .dvc

As part of this I started writing pytest-like tests and added commonly
used fixtures.

* test: simplify and prettify dvc formatting retained test

Also test that line comments after md5 fields are retained.

* test: fix tests

* dvc: fix stage and metrics opens not closed

* dvc: always use utf-8 in stage files

It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.

* test: unit test apply_diff()

* test: remove TODOs and ASK

* dvc: polish dvc-formatting PR

* dvc: move stage load/dump utils to a new stage utils file

* dvc: clean up apply_diff()

* test: test stage md5 ignores comments

* test: remove pytests starting marker
2019-04-20 01:12:53 +03:00
pared c467ccebcb ignore: introduce .dvcignore, apply when searching for stage files 2019-04-15 14:31:34 +02:00
Mr. Outis 1cac81430e tests: be paranoic with caplog 2019-04-11 14:09:02 -05:00
Mr. Outis 727bfd955e version: privatize version.py methods and vars 2019-04-07 16:48:34 -05:00
Ruslan Kuprieiev 54f7b43aac dvc: version: detect release
See failed release at [1]. Previously we used to look at TRAVIS_TAG env
var, but now we will try to detect release by matching current tag with
base version.

[1] https://travis-ci.com/iterative/dvc/builds/107365596

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-07 22:55:20 +03:00
Ruslan Kuprieiev 7c540da5c4 daemon: don't spawn recursively
Fixes #1803

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-07 20:12:23 +03:00
Mr. Outis 921b39dbfe tests: use pytest caplog instead of custom mock
Close #1753
2019-04-06 20:35:30 -06:00
Ruslan Kuprieiev 6124a23138 Merge pull request #1831 from mroutis/fix-1824
dvc: refactor package versioning logic
2019-04-07 03:16:39 +03:00
Mr. Outis b4d0d37ac8 dvc: refactor package versioning logic
- Use `version.py` to compute a dynamic version stored in `__version__`.
  Version information was scattered across the source code
  (`VERSION_BASE`, `VERSION`, `__version__`, and `version.py`).

- Read `__version__` from `setup.py` and **pin** it using custom
  `build_py` command class.
  This way, we can maintain consistancy between the build version
  and the runtime one.
  Also, this gives us the ability to run `setup.py` without depending on
  on `dvc` itself.

Close #1824
2019-04-06 18:06:11 -06:00
Ruslan Kuprieiev 8ef5976ac8 add: check if output is local when generating stage name
When adding a remote file with

```
$ dvc add s3://bucket/path
```

it results in

```
Error: failed to add file - is local is not supported for s3
```

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-04-06 03:25:53 +03:00
pared 0f5216cb14 checkout: reset progress bar on checkout 2019-03-31 21:48:21 +02:00
Andrew Grigorev 7d79043bbb dvc.scm.scm_context -> dvc.repo.scm_context.scm_context
The `dvc/scm/__init__.py:scm_context` works with `Repo` objects and
is used mostly in `dvc/repo/...`, so it is better to put it in
`dvc/repo/scm_context.py`.

Fix #1769
2019-03-22 19:53:41 +03:00
Andrew Grigorev 9e6c52079b Abstract access to file objects
Abstract access to filesystem and its implementation to access files and
particulary Stage objects in different git branches without `git
checkout` execution.
2019-03-21 19:47:42 +03:00
Paweł Redzyński ee16409495 checkout: progress bar (#1713) 2019-03-20 13:39:10 +02:00
Ivan Shcheklein f085e01717 try to fix progess bar test in a parallel pytest environment 2019-03-17 19:33:09 -07:00
Ivan Shcheklein 75421f0f21 migrate to pytest 2019-03-17 14:49:49 -07:00
Mr. Outis 21778f6719 stage: convert wdir path to posix
Fix #1721
2019-03-14 14:40:28 -06:00
pared 4897a6a44f add: prevent adding git files 2019-03-13 14:11:59 +01:00
pared f8711b3faa repo/scm: move tracking reminder from repo to scm 2019-03-11 23:13:34 +01:00
Ivan Shcheklein ee7cb41fcb add tests to properly test wdir in checksum calculation 2019-03-02 16:37:00 -08:00
Ruslan Kuprieiev cce25db97e Merge pull request #1672 from shcheklein/add_wdir_tests
Add more tests for wdir/fname new logic
2019-03-01 16:15:27 +02:00
Ruslan Kuprieiev 0913a65661 Merge pull request #1668 from prihoda/patch-1
Log computed stage MD5 in verbose mode
2019-03-01 15:27:47 +02:00
prihoda d63db35100 Log computed stage MD5 in verbose mode
Sometimes it is handy to manually update the stage MD5 checksum after some non-breaking changes were made (e.g. renaming an output file). Logging the computed md5 hash in verbose mode can help experienced users to fix this problem.
2019-03-01 14:11:29 +02:00
Ivan Shcheklein e26db30f60 add more tests for wdir/fname new logic 2019-02-28 21:43:14 -08:00
pared f8ef28c4c6 stage: check if local path contains symlink and, if so, treat added files as relpath 2019-02-28 15:00:05 +01:00
Ruslan Kuprieiev 8ff27454e6 Merge pull request #1660 from mroutis/fix-1602
logger: Format progress bar interruption
2019-02-28 14:52:18 +02:00
Mr. Outis 93920baaec logger: Format progress bar interruption
Fix #1602
2019-02-27 21:24:22 -06:00
pared 4464e29c15 remote local/state : add check for cache dir metadata changes 2019-02-25 18:35:02 +01:00
Ruslan Kuprieiev 63f87ba9fe Merge pull request #1653 from efiop/output
output/deps: raise proper exception when file/dir doesn't exist
2019-02-23 09:32:00 +02:00
Ruslan Kuprieiev 9ca00adde8 dvc: modify flake8 configuration
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-22 19:36:19 +02:00
Ruslan Kuprieiev 9961956c01 output/deps: raise proper exception when file/dir doesn't exist
From discussion on discord
https://discordapp.com/channels/485586884165107732/485596304961962003/548083580052701205

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-22 14:59:01 +02:00
Ruslan Kuprieiev e5107b1da3 ssh: external-outs: check if cache exists before copying
Fixes #1635

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-19 21:33:47 +02:00
Mr. Outis 5e7a152b53 ssh: use sftp when possible
Related #830 [1/2]
2019-02-18 19:03:06 -06:00
Ruslan Kuprieiev 01a120f9fc dvc: introduce dvc commit
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-16 21:19:36 +02:00
Ruslan Kuprieiev 70c21f7f3e ssh: handle url without path
https://discordapp.com/channels/485586884165107732/485596304961962003/545702950350356502

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-15 04:38:45 +02:00
Ruslan Kuprieiev ba8d83e9b1 dvc: refactor dvc project class
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-13 03:31:23 +02:00
Ruslan Kuprieiev 91ed40fbfa updater: don't use io.open when working with json
Otherwise we get this on python2:
```
======================================================================
ERROR: test_fetch (tests.unit.updater.TestUpdater)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/efiop/.pyenv/versions/2.7.15/envs/2.7.15-dvc/local/lib/
python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/efiop/git/dvc/tests/unit/updater.py", line 34, in test_fetch
    self.dvc.updater.fetch(detach=False)
  File "/home/efiop/git/dvc/dvc/updater.py", line 77, in fetch
    self._with_lock(self._get_latest_version, "fetching")
  File "/home/efiop/git/dvc/dvc/updater.py", line 39, in _with_lock
    func()
  File "/home/efiop/git/dvc/dvc/updater.py", line 91, in _get_latest_version
    json.dump(convert_to_unicode(info), fobj)
  File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
TypeError: write() argument 1 must be unicode, not str
-------------------- >> begin captured logging << --------------------
```

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-12 01:40:07 +02:00
Mr. Outis ec77bfa1af remote: define command error on the base
fix: #1484
2019-02-09 12:03:17 -06:00
Thibault de Boissiere 2c67b59fc0 checkout: Suggest git checkout when dcv checkout fails (#1580) 2019-02-05 23:46:13 +02:00
Mehdi Hamoumi ffc94cc471 Add GCP credentials environment variable to config (#1567) 2019-02-05 01:33:04 +02:00
Ruslan Kuprieiev 4896f08ddf dvc: comply with both flake8 and black
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-04 00:10:36 +02:00
Ruslan Kuprieiev ba091a6d34 checkout: handle EOFError when running in a git hook
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-31 22:56:48 +02:00