Commit Graph
5737 Commits
Author SHA1 Message Date
Paweł Redzyński 6ae0be2179 metrics: show: debug info on parsing problems (#4370) 2020-08-17 17:37:15 +03:00
Peter Rowlands (변기호) 70ddb7c931 experiments: include experiment timestamps in dvc exp show output (#4389)
* experiments: include experiment commit times in show

* --no-timestamp can be used to hide timestamps

* update tests

* clean up kwargs
2020-08-17 17:35:43 +03:00
Saugat Pachhai c4bc4c83e8 upgrade to isort5 (#4399)
* Imports are no longer moved to top.
* `seed-isort-config` is no longer needed.
* Has a `black`-compatible profile/config.
* Also `isorts` the imports anywhere in the code
  (not just at the top).

Also bumps `restyled-isort` to newest version
2020-08-17 19:59:40 +05:45
Saugat Pachhai 31e9cc2e40 cleanup RepoTree (#4405)
1. Rename `_get_tree_pairs` to `_get_tree_pair`.
2. Add comments, docstrings and type-annotations.
3. Rename variable: `_discovered_subrepos` to `_subrepos_trie`.
4. Merge some nested ifs.
2020-08-17 19:42:45 +05:45
karajan1001 b8e40f5091 Solve check-ignore broken in a Linux pipeline (#4409)
fix #4407
1. Update test, now it can simulate a pipeline situation.
2. Solve remove `isatty` check in `dvc check-ignore --stdin` command.
2020-08-17 16:17:45 +03:00
Saugat Pachhai 1cfccef22e update flake8 to latest version (#4403) 2020-08-17 14:26:10 +05:45
nik123 44392941af ui: '-j' option now mentions docs (#4382)
* ui: '-j' option now mentions docs

* ui: copied '-j' default value desc. from cmd ref
2020-08-15 21:41:42 +03:00
Ruslan Kuprieiev 023dec4aa6 tree: make get_hash return type and hash pair (#4397)
Currently we kinda assume that whatever is returned by `get_file_hash`
is of type self.PARAM_CHECKSUM, which is not actually true. E.g. for
http it might return `etag` or `md5`, but we don't distinguish between
those and call both `etag`. This is becoming more relevant for dir
hashes that are computed a few different ways (e.g. in-memory md5 or
upload to remote and get etag for the dir file).

Prerequisite for #4144 and #3069
2020-08-15 20:35:39 +03:00
Saugat Pachhai b773ba73cf tests: add .dvcignore on the tests expectations (#4396)
Since #4229, we have started creating .dvcignore file after `dvc init`
This commit only adds that in assertions
2020-08-15 21:24:29 +05:45
Saugat Pachhai 0b1d4f10d6 RepoTree: add support for subrepo traversal (#4381)
Right now, no actual "functionality" is surfaced as
`subrepos` is always set to False by default for now.
2020-08-15 17:27:00 +03:00
Christoph Berganski 8b41655894 Derive WebDAVURLInfo from URLInfo instead of HTTPURLInfo (#4391)
Unused, empty http extra parts (params,query,fragment) cause problems
with pushing/pulling - lead to violation of the assumption where to look
for hash values in file/path names.

Context:
    https://discordapp.com/channels/485586884165107732/485596304961962003/743454659733094520
    https://discordapp.com/channels/485586884165107732/485596304961962003/743497749290287215
    https://discordapp.com/channels/485586884165107732/485596304961962003/743571921622401136
2020-08-14 19:46:03 +03:00
Paweł Redzyński 35f125f3df outputs: validate that output is not ignored (#4386) 2020-08-14 14:30:12 +03:00
Robert Lowry 00925029b4 exception text: link to troubleshooting page (#4339) 2020-08-13 13:09:58 +03:00
Paweł Redzyński 9c772f5f6c get-url: allow weak etags (#4372) 2020-08-13 12:03:47 +03:00
karajan1001 e3cfa7348c Result inconsistency of the dvcignore interfaces (#4378)
Fix #4333, as `is_ignored_file` and `is_ignored_dir` and `__call__`
share the same call `DvcIgnorePatterns.matches`, we only need to focus
on the out of repo situation.
1. Add tests for the `is_ignored_file` and `is_ignored_dir` for the out
of repo condition.
2. Now `is_ignored_file` and `is_ignored_dir` returns false for those
files outside the repo.
2020-08-13 01:22:03 +03:00
Peter Rowlands (변기호) 9b5740555c experiments: add options to include/exclude columns in exp show (#4375)
* experiments: add options to include/exclude specific metrics/params columns in
`dvc exp show`

* allow param/metric file to be provided to keep consistency with other
dvc commands

* match nested params/metrics names
2020-08-13 01:21:12 +03:00
Jorge OrpinelandRuslan Kuprieiev ce9205a3cc term: review "URL" term usage in remote cmds, and get/import-url (#4364)
* term: review upper case URL usage, and remote add url arg desc

* get/imp-url: better url arg help out

* Update dvc/command/get_url.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-13 01:12:45 +03:00
AnatolyandRuslan Kuprieiev 5cb36fc983 Special dir warning #3436 (#4229)
* add special folder warning

* add special folder warning

* remove special folder warning, start working on default dvcignore

* add default .dvcignote and templates

* remove dvcignore templates

* change text and add tests

* refactor

* dvcignore: ignore comments

* adjust tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-13 01:03:34 +03:00
Ruslan Kuprieiev 64f038fda8 dvc: bump to 1.5.0 1.5.0 2020-08-12 19:08:38 +03:00
Ruslan Kuprieiev a8f0c5241c dvc: migrate to azure-storage-blob >= 12.0 (#4379)
There has been some pretty drastic changes in 12.0, so migration
is a little bit more involved.

Fixes #3546
2020-08-12 19:07:28 +03:00
karajan1001 12c7013062 Remove dvcignore reset in repo._reset (#4371)
* Remove dvcignore reset in `repo._reset`

Repo `add`,`checkout`,etc will not change `.dvcignore`, and can remove dvcignore reset after these operations.

* Reset dvcignore manually after a subrepo created.

1. subrepo created during session open time.
2020-08-11 10:53:49 +03:00
Jorge OrpinelandRuslan Kuprieiev 9cb32b5d29 check-ignore: update help output (#4358)
* check_ignore: update help output
per https://github.com/iterative/dvc/pull/4282#pullrequestreview-460270636
and https://github.com/iterative/dvc/pull/4282#pullrequestreview-460272287

* check-ignore: more output string updates
per #4323 et al.

* check-ignore: min text update
to match iterative/dvc.org/pull/1673

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-11 10:38:23 +03:00
Saugat Pachhai 765d2ed61b Update pylint-pytest to >=0.3.0
Older versions had issues with loading pylint plugins on Windows.
Close #4357
2020-08-11 11:26:59 +05:45
Peter Rowlands (변기호) 78458ee5a1 push: do not push .dir file for directory with missing files (#4353)
* tests: add test to ensure dir hashes are not pushed for incomplete
directories

* push: do not push .dir file for directory with missing files

* update test
2020-08-10 15:09:00 +03:00
karajan1001 07dcf6953c Error message when no match found. (#4354)
* Error message when no match found.

1. remove error message when no match found.

* Test update.

1.Test update cover return 0 and 1

* Remove unused arguments

* Error message show in dvc check-ignore
2020-08-10 15:00:11 +03:00
Puneetha Pai 172032d3b3 Config to enable/disable updater (#4334)
* Updater: Add method to check dvc core config check_update

* Updater: check config before calling update._check

Fixes #1894
2020-08-10 11:59:25 +03:00
karajan1001 4e5a81402d Remove error message when exiting --stdin mode (#4368)
fix #4361
1. Remove error message when input a empty string
2. Remove error message when read input from stream
2. Remove error message when interrupt by `CTRL-C`
2020-08-10 11:33:56 +03:00
Peter Rowlands (변기호) fef635c6f0 pager: use --R/--RAW-CONTROL-CHARS with less (#4369) 2020-08-10 11:16:56 +03:00
Saugat Pachhai 8d1934d494 dvcignore: api to unhide subrepos directory (#4324)
* Allow ignoring subrepos in dvcignore

I implemented this inside dvcignore, and it
should not require "internal" APIs hack as we discussed before.

Right now, `ignore_subrepos` will only work for the paths of subrepo,
not anything inside it. And the API user is more or less only
RepoTree, so should be safe (we switch DvcTree and repo.tree based
on path prefixes, so this is only required during `walk`, so as it
shows the repo that were previously dvcignored and so that we could
switch the trees later during walk.

So, if dir is `dir -> repo`
So, `tree.walk("dir")` will return `[]`,
whereas `tree.walk("dir", ignore_subrepos=False)` will return `["repo"]`
But, successive walk will just return `("repo", [], [])`, and could
just be ignored as a side-effect.

* Fix tests
2020-08-10 09:58:16 +03:00
Ruslan Kuprieiev dea1950ba0 dvc: bump to 1.4.0 1.4.0 2020-08-07 04:31:02 +03:00
e2a5741374 Two new modes --all and --stdin for check-ignore (#4323)
* Two new mode `-all` and `--stdin` for check-ignore

Fix #4321

Introduce new arguments --all for dvc check-ignore
Add tests for dvc check-ignore --all

* `--stdin` and targets only one

1. `--stdin` and targets can and only can have one
2. more tests

* New tests for stdin mode and solve a problem

1. Add a new tests with mock stdin for interactive mode
2. Solve the problem of `assert "" in caplog.text` in tests always `True`.

* Remove path_input interface

1. remove path_input and make _ask to a public method instead

* Update dvc/command/check_ignore.py

* Update dvc/command/check_ignore.py

* Update dvc/command/check_ignore.py

* Update dvc/command/check_ignore.py

* Update dvc/command/check_ignore.py

* Update dvc/command/check_ignore.py

* Update dvc/command/check_ignore.py

* Update dvc/command/check_ignore.py

* fix styling

Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-06 22:30:28 +03:00
Peter Rowlands (변기호) ab23bcd2e8 experiments: support dvc repro --exp command line params (#4331)
* experiments: support passing params values instead of reading from user's workspace

* repro: add --params option for experiments

* support toml params

* update invalid param error message

* update tests
2020-08-07 00:04:02 +09:00
Sarthak khandelwalandRuslan Kuprieiev 4759517c8d repro: update status message for dvc-added files (#4317)
* stage/__init__.py: Updated output for dvc repro

* Delete config.ini

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-08-06 17:48:09 +03:00
Ruslan Kuprieiev 7afff1ef29 config: use wtree for non-repo configs (#4341)
For example, when `dvc import`ing something, dvc won't see stuff
declared in system and global configs, because GitTree doesn't see stuff
that is outside of the git repo.

Discord context:
https://discordapp.com/channels/485586884165107732/563406153334128681/740543492417126521
2020-08-06 17:37:21 +03:00
Peter Rowlands (변기호) d128d51a2b import: fix importing into subdirectory bug (#4340)
* tests: test importing into subdirectory

* import: fix bug when output subdir doesnt already exist

* use urlparse to determine if output path is local, not os.path.exists

* account for windows drive letters

* import: error out if output subdirectory does not exist
2020-08-06 23:26:52 +09:00
Casper da Costa-Luis afa0a30a73 deps: bump shtab (#4337)
Depends on https://github.com/iterative/shtab/pull/19
Fixes bash not completing filenames & dirs
2020-08-06 14:51:22 +03:00
Peter Rowlands (변기호) 6bb654750a experiments: add --no-pager option for exp show (#4345) 2020-08-06 15:48:40 +09:00
Saugat Pachhai 35dd1fda9c Fix version tests (#4329)
Tests had checks for python version number, expecting it to be only
single digits, which was failing for me on 3.6.10 and might fail
in the future with Python 3.10.

I adjusted it to expect minor and patch version to be of multiple digits.
2020-08-04 11:51:05 +05:45
Saugat Pachhai d8b03737ee Update updater.py (#4328) 2020-08-04 06:27:08 +03:00
Peter Rowlands (변기호) 73750df73f experiments: more intelligent checkout behavior (#4320)
* restore Experiments.chdir

* used in experiments.show

* experiments: make checkout work without `git reset --hard`

* experiments: remove --force parameter

* experiments: reject checkout if current workspace commit doesnt match
expected baseline
2020-08-04 04:44:51 +03:00
11d906ee51 dvc check-ignore command (#4282)
* Update some tests first

fix #3736

* first edition

* solve failure in Windows

* For Windows ci

* Some help ducuments issue.

* Update dvc/ignore.py

abspath

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Refactor with OutOfWorkSpaceError

* Solve a bug

* Update dvc/command/check_ignore.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/check_ignore.py

Co-authored-by: Jorge Orpinel <[email protected]>

* Update dvc/command/check_ignore.py

* Revert "Refactor with OutOfWorkSpaceError"

This reverts commit 27eec49ed67346c89157edea998b6b5fa81d88f5.

* Two change request

1. Argument `targets`'s description.
2. Error handling of `_get_normalize_path`

* Update dvc/main.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* `check_ignore` now only accept one path a time

1. Add a new test for the out side repo cases
2. check ignore now only check one file not file lists

Co-authored-by: Ruslan Kuprieiev <[email protected]>
Co-authored-by: karajan1001 <[email protected]>
Co-authored-by: Jorge Orpinel <[email protected]>
2020-08-03 08:27:50 +03:00
Ruslan Kuprieiev b823ce48e1 tests: fix version check on tags 2020-08-01 20:42:16 +03:00
Ruslan Kuprieiev 4061b74cf3 dvc: bump to 1.3.1 1.3.1 2020-08-01 20:18:09 +03:00
Ruslan Kuprieiev baa9a5701f tree: don't save hash twice (#4314)
We already do this in
https://github.com/iterative/dvc/blob/00de04c82d844b3bdf5162b2c78560f1b5656488/dvc/tree/base.py#L271
for both files and directories.
2020-07-31 23:44:44 +03:00
Arthur Colombini Gusmão 00de04c82d Fix typo in dvc remote modify output (#4311) 2020-07-31 22:04:25 +03:00
Ruslan KuprieievandSaugat Pachhai ed1c3a7a15 remote: allow local options for repo remotes (#4309)
* remote: allow local options for repo remotes

Fixes #4276

* Update tests/func/test_remote.py

Co-authored-by: Saugat Pachhai <[email protected]>

* add more tests

Co-authored-by: Saugat Pachhai <[email protected]>
2020-07-31 09:30:07 +03:00
Ruslan Kuprieiev fcf2e3522c tree: get rid of explicit tree arguments (#4310)
* cache: don't make save() compute the hash

This is incorrect and leads to weird bugs like #4305

* tree: get rid of explicit tree arguments
2020-07-31 08:38:36 +03:00
Ruslan Kuprieiev dfdb24d89d dvc: bump to 1.3.0 1.3.0 2020-07-31 05:01:37 +03:00
Saugat Pachhai 99ffa7c6fd Use pytest>=6.0.1 (#4308)
pytest 6.0.0 had issue with `pylint`, complaining for `pytest.mark.*` being `not-callable`.
2020-07-30 19:41:42 +05:45
Ruslan Kuprieiev 9592246746 commit: make sure deps/outs exist (#4305)
Fixes #4266
2020-07-30 16:09:00 +03:00