Commit Graph
6285 Commits
Author SHA1 Message Date
Peter Rowlands (변기호) 79dafe8310 exp gc: prune any exp/exec refs which point to gc'd experiments (#5408) 2021-02-04 17:50:04 +09:00
Peter Rowlands (변기호) 982802effc experiments: bug fixes (#5407)
* exp run: display metrics on -m/--metrics

* exp show: fix sort handling
2021-02-04 16:42:22 +09:00
Ruslan Kuprieiev b87e76aa84 tree: stop using self.path_info (#5403)
* webhdfs: don't use self.path_info

* azure: don't use self.path_info

* webdav: don't use self.path_info

* oss: don't use self.path_info

* http: don't use self.path_info

* gdrive: don't use self.path_info
2021-02-03 20:50:30 +02:00
Ruslan Kuprieiev 2c81bf1748 api: use Repo.open (#5401) 2021-02-03 18:39:40 +02:00
Batuhan Taskaya adedd19c52 gha: switch to pre-commit/action for caching (#5389) 2021-02-03 17:14:46 +02:00
Ruslan Kuprieiev 985aef883e cli: config: rename --repo to --project (#5399)
Per https://github.com/iterative/dvc.org/pull/2064

Not changing the internal terminology because we stick to Repo in a lot
of places there and adjusting all of it would be wasteful. Maybe in the
future.
2021-02-03 17:13:53 +02:00
Ruslan Kuprieiev ffaac0f088 repo: remove unneeded filter (#5393)
Leftover from #5385
2021-02-03 17:13:36 +02:00
Batuhan Taskaya fc77618ad1 update: implement --to-remote flag (#5343)
* upload: implement --to-remote flag

* unit tests

* preserve frozen= for stage
2021-02-03 12:31:16 +02:00
Saugat Pachhai b2c2929869 pyarrow support for Python3.9 (#5391) 2021-02-02 17:31:11 +02:00
Batuhan Taskaya a92bfd272d get, get-url: add --jobs option (#5362) 2021-02-02 15:49:58 +02:00
Paweł Redzyński 31e4a3325d live integration: rename report to html (#5366) 2021-02-02 13:44:38 +01:00
Saugat Pachhai 67980de44d tests: mock global config dir for Updater (#5386) 2021-02-02 17:08:09 +05:45
Ruslan Kuprieiev 8932e0be84 dvc: get rid of can_be_skipped logic (#5385) 2021-02-02 13:13:48 +02:00
Batuhan Taskaya 9ad1e0e159 ssh: show total-size when uploading/downloading on ssh remote (#5383) 2021-02-02 11:27:19 +02:00
Saugat Pachhai 4348832b44 keep stage.dump and dvcfile.dump in sync (#5382) 2021-02-02 15:11:13 +05:45
Peter Rowlands (변기호) d05be36694 exp show: cleanup UI issues (#5361)
* exp show: add -n/--num to display the last N commits from HEAD

* exp run: fix empty commit with no changes

* add test for exp show with multiple commits

* allow -n greater than num commits in repo

* review fix
2021-02-02 11:35:02 +09:00
Peter Rowlands (변기호) 2ea5ef07dd exp apply: force apply by default (#5360)
* exp apply: force apply by default

* update tests
2021-02-02 11:07:19 +09:00
Batuhan Taskaya 8205c84077 pull, fetch: optimize memory consuption by ~%35 (#5346) 2021-02-01 18:46:02 +02:00
Saugat Pachhai b020918f5d stage:add: use remainder as a command script (#5350)
* stage:add use remainder as a command

* use -c for checkpoints

* use -n for stage name, make it required, fix tests
2021-02-01 17:50:04 +02:00
Batuhan Taskaya c84833bab1 params: support unicode characters (#5363)
* params: support unicode characters

* open remove() calls with utf-8
2021-02-01 15:56:53 +02:00
Paweł Redzyński 13db267a5d collect: unify message for no file in workspace (#5339) 2021-02-01 14:25:22 +01:00
Batuhan Taskaya b337396144 cache pip directory on github actions (#5365) 2021-02-01 13:48:03 +01:00
Piotr Bielak 3feaa86696 Pass missing glob flag to pull command (#5354)
The `--glob` flag is parsed from the command line, but in the
`CmdDataPull` class in the invocation the parameter (`self.args.glob`)
is not passed to the `pull` method implementation. Hence, always the
default value (False) is used. This commit fixes this issue and sets the
`glob` parameter properly.
2021-02-01 14:21:21 +02:00
Saugat Pachhai 131eb7b3a1 don't allow loading stages without cmd (#5373)
* don't allow loading stages without `cmd`

This was loosened when introducing parametrization.
But, `foreach`..`do` should be considered first
before stage's regular structure. And, cmd is made
required (as it should have been)

Related: #5371, #5370, #5312

* fix schema
2021-02-01 13:23:40 +05:45
Peter Rowlands (변기호) b771b06b25 tests: reduce checkpoint experiment iterations (#5352)
* exp tests: reduce default checkpoint run iterations from 5 to 2

* api: reduce make_checkpoint sleep interval
2021-02-01 14:20:51 +09:00
Peter Rowlands (변기호) 08300d3bcf experiments: drop exp resume in favor of auto-resuming in exp run (#5351)
* exp: auto determine checkpoint resume rev based on workspace state

* exp: add 'dvc exp reset' for reset/restarting checkpoints

* update tests

* fix reset usage
2021-01-29 13:18:20 +09:00
Batuhan Taskaya bb46043689 add: implement saving straight to cache (#5301)
* add: implement saving straight to cache

* Adjust CLI error cases to handle -o

* make tests pass on windows

* support explicit --jobs

* simplify the implementation a bit

* polish the tests
2021-01-28 14:42:58 +02:00
Ruslan Kuprieiev fe2d913821 list: don't clone local repos (#5341)
* repo: introduce Repo.open

* ls: don't clone local repos
2021-01-28 03:05:05 +02:00
Ruslan Kuprieiev 969a85efb6 RepoTree: get rid if implicit fetching (#5324) 2021-01-27 02:01:09 +02:00
Saugat Pachhai b1d6918c5a stage collection: do not collect .gitignored dvcfiles (#5265)
* Do not collect .gitignored dvcfiles

* fix exc

* consistent error message

* fix imports

* add tests
2021-01-26 14:51:02 +02:00
Batuhan TaskayaandRuslan Kuprieiev a5e19ecd6a tree: Implement upload_fobj protocol to all trees (#5307)
* tree: Implement upload_fobj protocol to azure

* tree: Implement upload_fobj protocol to google storage

* tree: Implement upload_fobj protocol to google drive

* tree: Implement upload_fobj protocol to hdfs

* tree: Implement upload_fobj protocol to oss (alibaba cloud)

* tree: Implement upload_fobj protocol to webhdfs (also some cleanup)

* tree: Implement upload_fobj protocol to http/https

* Add _generate_download_url() to the HTTP (for tests)

* tree: Implement a temporary upload_fobj protocol to webdav/webdavs

* Unify progress bars

* Lazily evaluate the file size, for http uploads

* Remove copy_fobj protocol and _transfer_file_as_whole

* show progress bar on cache transfer

* Simplify s3 upload

* Remove CHUNK_SIZE (except for some special cases)

* remove concurrency restrictions

* Update dvc/tree/oss.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-26 13:57:51 +02:00
Ruslan Kuprieiev 434cd305e9 dvc: ignore errors on protect/set_exec (#5335)
Both of these are not fatal and could safely be ignored.

Fixes #5255
2021-01-26 03:46:14 +02:00
Ruslan Kuprieiev 8112ce36d1 dvc: move 'verify' property from tree to cache (#5334) 2021-01-26 02:00:35 +02:00
Ruslan Kuprieiev 6a1f3659f5 tests: limit pytest version by <6.2.2 (#5333) 2021-01-25 22:01:54 +02:00
Batuhan Taskaya ec82c3fff5 tests: azure directory generation (#5328) 2021-01-25 15:41:29 +02:00
Peter Rowlands (변기호) d4757120c9 exp show: cleanup data column handling (#5329)
* account for identical param/metric names in columns (disable filtering)

* update filtering
2021-01-25 22:19:20 +09:00
Saugat Pachhai 9f0b86f221 unit.scm.test_git: use pytest instead of unittest (#5325) 2021-01-25 10:23:46 +05:45
Ruslan Kuprieiev 48dbcf9163 repo: remove unused find_out_by_relpath (#5323) 2021-01-25 01:24:32 +02:00
Saugat Pachhai 3741f730e0 show git-add hint on stage:add, refactor scm_context (#5290)
* Refactor scm_context, show git-add hint on stage:add

* move tests
2021-01-24 22:43:38 +02:00
Ruslan Kuprieiev 58c97b963c hash_info: add __str__ (#5319) 2021-01-24 22:40:38 +02:00
Paweł Redzyński 6a9c0c5ce5 live/checkpoints: test continue (#5300)
* live: test resuming

* initial test, not working

* experiments: introduce DVCLIVE_RESUME env variable on resume
2021-01-24 22:06:37 +02:00
Saugat Pachhai 1d8761e0c9 Remove empty file tests/unit/command/test_commit.py (#5316)
When I opened it, I got confused for a second and had to double-check if it was really empty.
The empty file gives a false assurance. We do need to add one, which I'll add later, but for now,
we better remove it.
2021-01-23 18:27:43 +05:45
Paweł Redzyński 2277a1a655 metrics/plots/params: return ordered (#5297)
* metrics/plots/params: return ordered

* metrics/params/plots: add tests for return order
2021-01-22 20:57:43 +02:00
Ruslan Kuprieiev cbd7472079 dvc: refactor cache saving (#5314) 2021-01-22 19:07:36 +02:00
Batuhan Taskaya 7b59108379 push: Gradual task allocation for pushing big datasets (#5285) 2021-01-22 15:29:06 +02:00
Ruslan Kuprieiev c4888cbd7d dvc: get rid of ExternalRepo (#5309) 2021-01-21 23:03:53 +02:00
Ruslan Kuprieiev 9203d33a3c reqs: tests: remove awscli (#5306) 2021-01-21 18:09:35 +02:00
Timofey Yefimovandkarajan1001 60d83a74d7 dvcignore: Fix incorrect ignored output computation (#4986)
* dvcignore: Fix incorrect ignored output computation

* dvcignore: Fixed unignored patterns

Unignored patterns were shown in the dvc status output and caused Error.

Also made a bit of a refactoring in `matches` and `ignore`.

* dvcignore: Fix functional tests

Not sure if I follow, but tests assume that unignored files should trigger `check-ignore`. What was the intention?

* dvcignore: Fix stage.wdir being MagicMock

* Use `is_ignored` instead of `check_ignore`

1. use is_ignored to judge DVC ignore status.
2. use check_ignore to show ignore patterns.

* Solve tests fails

* Add a new tests

* Return to the old check_ignore logic.

* Pass the pylint

* Simplify ignore tests

Co-authored-by: karajan1001 <[email protected]>
2021-01-21 14:28:08 +02:00
Saugat Pachhai 689ae22fb4 Fix code-style issues in the parsing module (#5299) 2021-01-20 17:21:57 +05:45
Batuhan Taskaya b5068a9200 checkout: don't filter hash's if it is a file (#5298) 2021-01-20 12:36:47 +02:00