Commit Graph
6222 Commits
Author SHA1 Message Date
Paweł Redzyński 28f6beaf78 checkpoints/live: move execution env to stage (#5257)
* checkpoints/live: move execution env to stage

* stage: env: provide DVC_ROOT for all repo runs
2021-01-18 10:35:07 +02:00
Saugat Pachhai 628f87845c introduce stage:add command (#5263)
* Implement stage:add

* fix tests

* Add short flag -C for checkpoints
2021-01-18 10:05:36 +02:00
Ruslan Kuprieiev 273518afab hdfs: copy: don't read everything into memory (#5277)
Regression from #4973 and #4747
2021-01-16 00:57:13 +02:00
Ruslan Kuprieiev b027760e12 dvc: move protect from tree to cache (#5271) 2021-01-14 21:10:53 +02:00
Ruslan Kuprieiev a2cac8b423 tests: upgrade azurite and enable loose mode (#5272)
* tests: upgrade azurite and enable loose mode

Azure tests started failing recently. They still work with real azure, so this is on azurite side.

* tests: use cloud instead of workspace

also gdrive doesn't support it yet :(
2021-01-14 20:07:24 +02:00
Ruslan Kuprieiev 3e42e015d6 dvc: redefine is_callback as cmd and no outs/deps (#5187) 2021-01-14 17:23:20 +02:00
Saugat Pachhai 257f58450a refactor run; move to stage.utils (#5262)
* refactor run; move to stage.utils

* Update dvc/repo/run.py

* fix pylint

* skip on run-cache
2021-01-13 21:19:36 +02:00
Batuhan Taskaya c840cecc91 tree: implement getsize() (#5261)
* Implement Tree.getsize() API

* Remove azure from tests

* strip support for recursive size calculation

* _getsize() -> getsize()
2021-01-13 18:35:57 +02:00
Batuhan Taskaya e5f91f5c7f cache: don't initalize cloud cache until its needed (#5264) 2021-01-13 17:01:06 +02:00
Paweł RedzyńskiandSaugat Pachhai 2ee68b01b0 system: prevent mypy from checking windows-related method (#5260)
* system: prevent mypy from checking windows-related method

* Update dvc/system.py

Co-authored-by: Saugat Pachhai <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
2021-01-13 13:48:57 +02:00
Saugat Pachhai f8200ca2b8 move parsing params to utils (#5259)
* move parsing params to utils

* fix tests

* move chunk_dict and test it

* update test
2021-01-13 13:36:49 +02:00
Peter Rowlands (변기호) 228b3b16e5 experiments: exp res bug fixes (#5258)
* dvc repro: fail on checkpoint stages

* exp res: check for moved HEAD/baseline before resuming

* exp run/res: always consider last checkpoint as applied for workspace runs
2021-01-13 17:49:46 +09:00
Peter Rowlands (변기호) 1ab1abf62a git: implement merge/checkout via pygit2 (#5245)
* git: implement merge (gitpython, pygit2)

* pylint: add pygit2 as C extension

* git: s/checkout_paths/checkout_index/ and add pygit implementation

* git: support basic checkout/merge conflict strategies in checkout_index

* fix relpath issue in pygit merge

* git: add stubs for add --update

* experiments: drop remaining gitpython specific scm usage
2021-01-12 13:26:27 +02:00
Saugat Pachhai 49974c075c dvc: bump to 2.0.0a0 (#5250)
Looking at the log is confusing, as the version is stuck in `1.11.0` whereas the recent one is `1.11.10`.

As the updater is also dependent on that version, it prints an ugly update message while running too.
https://github.com/iterative/dvc/issues/5023#issuecomment-758450519
2021-01-12 09:24:57 +02:00
Saugat Pachhai a2c91d2058 scm: implement gitignore (#5243)
* scm: implement gitignore; use `pygit2` backend

* Add is_ignored implementation on `pygit2` and `gitpython` backends.
* Also implement `._reset()` on `repo.scm` which is most of the changes.

* fix tests

* use _backend_func

* minor fixups
2021-01-11 12:31:26 +02:00
Batuhan TaskayaandRuslan Kuprieiev 7fd191126e S3Tree._list_paths now recognizes the max_items limit (#5239)
* S3Tree._list_paths now recognizes the max_items limit

* Use modern approach + add a test

* refactor tests

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* lint

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-11 11:54:42 +02:00
Batuhan Taskaya 5374880220 s3: Allow authentication using session token (#5240)
* S3: Allow authentication using session token

* Add a little test
2021-01-11 10:48:57 +02:00
Ruslan Kuprieiev c83a4ea406 erepo: refactor download/fetch logic (#5234)
* erepo: don't use _get_tree_for

* erepo: refactor download/fetch logic

* dvc: unify handling existing output
2021-01-10 10:48:36 +02:00
Dmitry Kropachev 6b05cc5e24 dvc.command.diff: add renamed detection feature (#5160)
fixes #5150
2021-01-09 13:49:03 +09:00
Saugat Pachhai 388eccad13 fix mypy errors; add mypy precommit hooks (#5237) 2021-01-08 16:05:52 +02:00
Saugat Pachhai ccc17917cf Make few imports lazy on dvc.main (#5233)
* Moved exception handling imports wherever they occur
* Move dvc.tree and dvc.external_repo imports to it's
  nearest use, as it is expensive for `--version`.
* Move analytics import to when it's used
* Move format_link to place where error occurs
2021-01-07 15:55:34 +02:00
Paweł Redzyński 386d0c463a live: add report switch (#5223) 2021-01-07 13:52:21 +02:00
Saugat Pachhai c04cdaf70f Split config and schema (#5229)
so as not to compile a schema unless needed
2021-01-07 13:47:14 +02:00
Yihao JiangandSaugat Pachhai fa5487ebfe show more info when successfully init dvc repo (#5190)
* show more info when successfully init dvc repo

* Ignore 'line too long' linter error

* simpler solution

* black reformat

* don't break dvc init --no-scm case

* Update dvc/repo/init.py

Co-authored-by: Saugat Pachhai <[email protected]>

* Update dvc/repo/init.py

Co-authored-by: Saugat Pachhai <[email protected]>
2021-01-07 16:05:35 +05:45
Saugat Pachhai 9331ad23e4 Misc. imports lazy on utils,stage.cache and dvc.* (#5228) 2021-01-07 15:03:34 +05:45
Saugat Pachhai 66fc988ebf Make imports on dvc.repo and dvc.repo.plots lazy (#5227) 2021-01-07 15:02:31 +05:45
Saugat Pachhai 40122cefa5 parsing: make imports lazy (#5225)
* parsing: make imports lazy

* Locally declare default params file
2021-01-07 15:00:54 +05:45
Saugat Pachhai a9ccf70439 cmd: imports lazy (#5226)
They are not a major pain point though
2021-01-07 11:45:58 +05:45
Saugat Pachhai a9d620beda utils.serialize: lazy imports (#5224) 2021-01-07 11:44:37 +05:45
Saugat Pachhai af2a12a188 Use 'schema' top-level element in the lockfile for version (#5222) 2021-01-06 20:08:14 +02:00
mbiesekandRuslan Kuprieiev 00461bb5df commit: support granularity (#4297) (#5088)
* commit: support granularity (#4297)

* commit: support granularity

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-06 19:39:04 +02:00
Saugat Pachhai 0db41b94fc parametrization: do not load parser on import (#5217)
A great benefit to this is that, if the dvc.yaml file
is never interpolated, the parser is never built, which
reduces the gap even further between 1.0 and 2.0 files
2021-01-06 21:40:19 +05:45
Saugat Pachhai 2646c0d6bc Make some imports lazy for plots (#5219) 2021-01-06 17:41:30 +02:00
Saugat Pachhai 1cc646614f Misc. imports made lazy: scm,experiments,stage (#5218) 2021-01-06 17:41:12 +02:00
Saugat Pachhai 84e9c1b0de lazy imports on repo command (#5221) 2021-01-06 17:41:01 +02:00
Saugat Pachhai 4f692b0b69 Make repotree lazy (#5220)
Also removed dvcfile related imports on dvc.tree.repo.
This was making imports slower on dvc.main
2021-01-06 17:40:31 +02:00
Batuhan TaskayaandRuslan Kuprieiev 6fb84f641c checkout: fix partial file checkouts in the same directory (#5163)
* Fix partial file checkouts in the same directory

* unittest => pytest

* Try to rationalize changed() to take filter_info

* Use path.read_text() to verify the directory structure

* Try to use specialized hash, fall back to general one if doesn't exist

* Cover subdirectories in hash specialization

* coverage: add a test for empty subdirectories

* Move out the dir_info filtering block

* hash_origin => path_info

* cleanup

* Move hashing functionality after the existence checks

* Simplify it a bit

* Update dvc/cache/base.py

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Apply suggestions from code review

Co-authored-by: Ruslan Kuprieiev <[email protected]>

* Apply suggestions

* simplify tests

* change filenames

* Some visual changes for consistency

Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-01-06 14:21:00 +02:00
Saugat Pachhai 9b73a745b4 Misc imports lazy (#5215) 2021-01-06 17:50:24 +05:45
Saugat Pachhai 4c53e636bc Move misc. imports from command, make them lazy (#5214) 2021-01-06 17:44:02 +05:45
Saugat Pachhai 40bd838c11 config command: lazily import (#5211) 2021-01-06 16:03:06 +05:45
Saugat Pachhai 709e121b31 package: Increment shtab's version (#5212) 2021-01-06 15:48:33 +05:45
Saugat Pachhai 0cc5227f47 Add versioning scheme in the dvc.lock (#5128)
* Add versioning scheme in the dvc.lock

* remove whitespace

* use actual size in tests assumptions
2021-01-06 15:44:19 +05:45
Saugat Pachhai 4f89db8e0d analytics: lazily import (#5210) 2021-01-06 15:32:27 +05:45
Peter Rowlands (변기호) 37f890444a diff: account for deleted workspace data (#5205)
* diff: add test for deleted data in addition to .dvc file

* diff: handle deleted data when checking for missing cache
2021-01-05 17:52:24 +09:00
Ruslan Kuprieiev b779a117ab add: handle long filenames (#5201) 2021-01-05 10:33:07 +02:00
Saugat Pachhai c0fbdb3531 Lazily load git backends (#5203) 2021-01-05 09:46:24 +02:00
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి) a017b87db5 Remove redundant comment (#5199) 2021-01-04 16:35:29 +02:00
Saugat Pachhai c4970a016f debug functions: have a good default value (#5202)
I had to keep providing default value before when profiling
simple functions as an API. I added a default now, and
also made `debugtools` customizable so that you could
edit the code to, for example, have `pdb` enabled by default.
2021-01-04 20:16:42 +05:45
Peter Rowlands (변기호) cffc019e1f experiments: fix logging bugs (#5195)
* experiments: use relpaths when tracking changed files

* experiments: suppress SCM hints in tempdir executors

* experiments: move multiprocessing error logging into child

* exp run: clarify help text for --temp
2021-01-04 19:25:13 +09:00
Saugat Pachhai 8010005364 Reraise exception after pdb.post_mortem() (#5197)
Otherwise, `continue` etc. would fail
2021-01-04 15:34:04 +05:45