Commit Graph
6243 Commits
Author SHA1 Message Date
Paweł Redzyński 2277a1a655 metrics/plots/params: return ordered (#5297)
* metrics/plots/params: return ordered

* metrics/params/plots: add tests for return order
2021-01-22 20:57:43 +02:00
Ruslan Kuprieiev cbd7472079 dvc: refactor cache saving (#5314) 2021-01-22 19:07:36 +02:00
Batuhan Taskaya 7b59108379 push: Gradual task allocation for pushing big datasets (#5285) 2021-01-22 15:29:06 +02:00
Ruslan Kuprieiev c4888cbd7d dvc: get rid of ExternalRepo (#5309) 2021-01-21 23:03:53 +02:00
Ruslan Kuprieiev 9203d33a3c reqs: tests: remove awscli (#5306) 2021-01-21 18:09:35 +02:00
Timofey Yefimovandkarajan1001 60d83a74d7 dvcignore: Fix incorrect ignored output computation (#4986)
* dvcignore: Fix incorrect ignored output computation

* dvcignore: Fixed unignored patterns

Unignored patterns were shown in the dvc status output and caused Error.

Also made a bit of a refactoring in `matches` and `ignore`.

* dvcignore: Fix functional tests

Not sure if I follow, but tests assume that unignored files should trigger `check-ignore`. What was the intention?

* dvcignore: Fix stage.wdir being MagicMock

* Use `is_ignored` instead of `check_ignore`

1. use is_ignored to judge DVC ignore status.
2. use check_ignore to show ignore patterns.

* Solve tests fails

* Add a new tests

* Return to the old check_ignore logic.

* Pass the pylint

* Simplify ignore tests

Co-authored-by: karajan1001 <[email protected]>
2021-01-21 14:28:08 +02:00
Saugat Pachhai 689ae22fb4 Fix code-style issues in the parsing module (#5299) 2021-01-20 17:21:57 +05:45
Batuhan Taskaya b5068a9200 checkout: don't filter hash's if it is a file (#5298) 2021-01-20 12:36:47 +02:00
Ruslan Kuprieiev c6ce69dd30 erepo: move misc stuff out of erepo (#5296)
* erepo: move get_rev to repo

* erepo: move fetch defaults to external_repo
2021-01-20 07:08:00 +02:00
Ruslan Kuprieiev 7cbcd4a7be erepo: separate caching and repo (#5295)
* erepo: don't pass for_write

* erepo: move get_cache_dir out of ExternalRepo
2021-01-20 00:25:15 +02:00
Ruslan Kuprieiev 03fc531a5c pylint: enable no-else-return (#5294)
From https://github.com/iterative/dvc/pull/5270#discussion_r559440586
2021-01-19 22:12:46 +02:00
Ruslan Kuprieiev d038bca05e tests: add basic [web]hdfs mockers (#5279) 2021-01-19 20:03:16 +02:00
Saugat Pachhai d8982c6db7 Override __str__ implementation for StageNotFound (#5289)
This happens because KeyError quotes the error message.
We'd still like to keep StageNotFound being a KeyError
semantics, but workaround the __str__ implementation.

Mypy was having some problem with the previous `__str__`
override.
2021-01-19 18:33:18 +05:45
Saugat Pachhai 372d6f045b Introduce stage:list command with autocompletions (#5282)
* Implement stage:list

* change targets help message
2021-01-19 17:35:55 +05:45
Batuhan Taskaya 51ada6d272 Implement straight-to-remote functionality (#5198)
* Implement export-to-remote functionality

* Redistrubition of functionality from tree to remote

* Pass chunk size into the open_url

* Move functionality away from remote to cache

* Basic progress bars for both transfer types

* Proper job calculation

* Redesign .open() usage in order to work with different trees

* Align the signatures of other open() functions to accept **kwargs

* Implement chunked uploading to the ssh

* Use local cache to compute directory hash (md5)

* S3: multipart upload test

* Wrap local cache's hash_to_path with PathInfo

* Use .isdir() instead of a try/except statement for calculating the from_infos

* New CLI layout (add/import-url)

* Adjust CLI tests to accept new arguments

* tests: import_url/add func/unit tests

* Introduce cache.transfer()

* Invalid CLI combination tests

* Store dir_info key in the transfer_directory

* LocalTree.upload_fobj

* Tests for utils

* Lazy imports on some places

* --to-remote migration (and repo._transfer)

* Use proper temporary location, more efficient pbar calc

* Directory transfer logic rewrite

* Use shutil.copyfileobj()

* Rebase...

* Better tests

* Inline transfer operation to .add() / .transfer()

* Open in the 'rb' mode for the tests

* More tests, better invalid option handling at repo level

* .read() certain amount of bytes to validate the content

* Better invalid argument handling

* Better progress bars by using .getsize() API

* For fallback upload, use the from_info.name instead of temporary naem

* introduce --jobs for add/import-url
2021-01-19 12:53:32 +02:00
Saugat Pachhai 55df6b8a6a cli: remove: Show traceback on verbose (#5287) 2021-01-19 15:10:56 +05:45
Pavel Ditenbir 8b0e0b995e cli: remove: fix error message for the case when file is not under control (#4497) (#5236)
* cli: remove: fix error message for the case when file is not under control (#4497)

* cli: remove: fix error message for the case when file is not under control (#4497)

fixes after code review

* cli: remove: fix error message for the case when file is not under control (#4497)

2nd round of fixes after code review

* cli: remove: fix error message for the case when file is not under control (#4497)

3rd round of fixes after code review
2021-01-19 15:04:19 +05:45
Peter Rowlands (변기호) 407f5607a2 scm: implement pygit2 resolve rev and native checkout (#5270)
* pygit2: implement get/set/remove refs

* pygit: implement iter_refs

* pygit: implement resolve_rev

* pygit2: implement working tree checkout

* pygit2: implement reset, explicitly read/write index when used

* pygit2: checkout does not need to reset index

* pygit2: handle "-" branch shorthand in checkout

* tests: install post-checkout hook should be run via gitpython

* exp apply: use stricter exception handling for merge conflict

* pygit: fix index read/write usage

* gitpython: handle untracked file conflicts in stash apply

* fix flaky/race index issue
2021-01-18 17:41:25 +02:00
Saugat Pachhai 78e12327bc use repo.stage.create_from_cli to create stages (#5281) 2021-01-18 18:55:19 +05:45
Ruslan Kuprieiev 4f2de7a3ef tests: remove psutil checks (#5283)
Leftover from https://github.com/iterative/dvc/pull/5232
2021-01-18 13:50:21 +02:00
Asif Saif UddinandRuslan Kuprieiev f8c9797f2a add psutil to install requirement (#5232)
* add  psutil to install requirement

* info: remove unneded checks

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-18 12:30:38 +02:00
Paweł Redzyński 28f6beaf78 checkpoints/live: move execution env to stage (#5257)
* checkpoints/live: move execution env to stage

* stage: env: provide DVC_ROOT for all repo runs
2021-01-18 10:35:07 +02:00
Saugat Pachhai 628f87845c introduce stage:add command (#5263)
* Implement stage:add

* fix tests

* Add short flag -C for checkpoints
2021-01-18 10:05:36 +02:00
Ruslan Kuprieiev 273518afab hdfs: copy: don't read everything into memory (#5277)
Regression from #4973 and #4747
2021-01-16 00:57:13 +02:00
Ruslan Kuprieiev b027760e12 dvc: move protect from tree to cache (#5271) 2021-01-14 21:10:53 +02:00
Ruslan Kuprieiev a2cac8b423 tests: upgrade azurite and enable loose mode (#5272)
* tests: upgrade azurite and enable loose mode

Azure tests started failing recently. They still work with real azure, so this is on azurite side.

* tests: use cloud instead of workspace

also gdrive doesn't support it yet :(
2021-01-14 20:07:24 +02:00
Ruslan Kuprieiev 3e42e015d6 dvc: redefine is_callback as cmd and no outs/deps (#5187) 2021-01-14 17:23:20 +02:00
Saugat Pachhai 257f58450a refactor run; move to stage.utils (#5262)
* refactor run; move to stage.utils

* Update dvc/repo/run.py

* fix pylint

* skip on run-cache
2021-01-13 21:19:36 +02:00
Batuhan Taskaya c840cecc91 tree: implement getsize() (#5261)
* Implement Tree.getsize() API

* Remove azure from tests

* strip support for recursive size calculation

* _getsize() -> getsize()
2021-01-13 18:35:57 +02:00
Batuhan Taskaya e5f91f5c7f cache: don't initalize cloud cache until its needed (#5264) 2021-01-13 17:01:06 +02:00
Paweł RedzyńskiandSaugat Pachhai 2ee68b01b0 system: prevent mypy from checking windows-related method (#5260)
* system: prevent mypy from checking windows-related method

* Update dvc/system.py

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
2021-01-13 13:48:57 +02:00
Saugat Pachhai f8200ca2b8 move parsing params to utils (#5259)
* move parsing params to utils

* fix tests

* move chunk_dict and test it

* update test
2021-01-13 13:36:49 +02:00
Peter Rowlands (변기호) 228b3b16e5 experiments: exp res bug fixes (#5258)
* dvc repro: fail on checkpoint stages

* exp res: check for moved HEAD/baseline before resuming

* exp run/res: always consider last checkpoint as applied for workspace runs
2021-01-13 17:49:46 +09:00
Peter Rowlands (변기호) 1ab1abf62a git: implement merge/checkout via pygit2 (#5245)
* git: implement merge (gitpython, pygit2)

* pylint: add pygit2 as C extension

* git: s/checkout_paths/checkout_index/ and add pygit implementation

* git: support basic checkout/merge conflict strategies in checkout_index

* fix relpath issue in pygit merge

* git: add stubs for add --update

* experiments: drop remaining gitpython specific scm usage
2021-01-12 13:26:27 +02:00
Saugat Pachhai 49974c075c dvc: bump to 2.0.0a0 (#5250)
Looking at the log is confusing, as the version is stuck in `1.11.0` whereas the recent one is `1.11.10`.

As the updater is also dependent on that version, it prints an ugly update message while running too.
https://github.com/iterative/dvc/issues/5023#issuecomment-758450519
2021-01-12 09:24:57 +02:00
Saugat Pachhai a2c91d2058 scm: implement gitignore (#5243)
* scm: implement gitignore; use `pygit2` backend

* Add is_ignored implementation on `pygit2` and `gitpython` backends.
* Also implement `._reset()` on `repo.scm` which is most of the changes.

* fix tests

* use _backend_func

* minor fixups
2021-01-11 12:31:26 +02:00
Batuhan TaskayaandRuslan Kuprieiev 7fd191126e S3Tree._list_paths now recognizes the max_items limit (#5239)
* S3Tree._list_paths now recognizes the max_items limit

* Use modern approach + add a test

* refactor tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* lint

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-11 11:54:42 +02:00
Batuhan Taskaya 5374880220 s3: Allow authentication using session token (#5240)
* S3: Allow authentication using session token

* Add a little test
2021-01-11 10:48:57 +02:00
Ruslan Kuprieiev c83a4ea406 erepo: refactor download/fetch logic (#5234)
* erepo: don't use _get_tree_for

* erepo: refactor download/fetch logic

* dvc: unify handling existing output
2021-01-10 10:48:36 +02:00
Dmitry Kropachev 6b05cc5e24 dvc.command.diff: add renamed detection feature (#5160)
fixes #5150
2021-01-09 13:49:03 +09:00
Saugat Pachhai 388eccad13 fix mypy errors; add mypy precommit hooks (#5237) 2021-01-08 16:05:52 +02:00
Saugat Pachhai ccc17917cf Make few imports lazy on dvc.main (#5233)
* Moved exception handling imports wherever they occur
* Move dvc.tree and dvc.external_repo imports to it's
  nearest use, as it is expensive for `--version`.
* Move analytics import to when it's used
* Move format_link to place where error occurs
2021-01-07 15:55:34 +02:00
Paweł Redzyński 386d0c463a live: add report switch (#5223) 2021-01-07 13:52:21 +02:00
Saugat Pachhai c04cdaf70f Split config and schema (#5229)
so as not to compile a schema unless needed
2021-01-07 13:47:14 +02:00
Yihao JiangandSaugat Pachhai fa5487ebfe show more info when successfully init dvc repo (#5190)
* show more info when successfully init dvc repo

* Ignore 'line too long' linter error

* simpler solution

* black reformat

* don't break dvc init --no-scm case

* Update dvc/repo/init.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update dvc/repo/init.py

Co-authored-by: Saugat Pachhai <[email protected]>
2021-01-07 16:05:35 +05:45
Saugat Pachhai 9331ad23e4 Misc. imports lazy on utils,stage.cache and dvc.* (#5228) 2021-01-07 15:03:34 +05:45
Saugat Pachhai 66fc988ebf Make imports on dvc.repo and dvc.repo.plots lazy (#5227) 2021-01-07 15:02:31 +05:45
Saugat Pachhai 40122cefa5 parsing: make imports lazy (#5225)
* parsing: make imports lazy

* Locally declare default params file
2021-01-07 15:00:54 +05:45
Saugat Pachhai a9ccf70439 cmd: imports lazy (#5226)
They are not a major pain point though
2021-01-07 11:45:58 +05:45
Saugat Pachhai a9d620beda utils.serialize: lazy imports (#5224) 2021-01-07 11:44:37 +05:45