Commit Graph
64 Commits
Author SHA1 Message Date
Peter Rowlands (변기호) 75e795c491 remote: support saving RepoTree objects directly to cache (#3825)
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache

* RepoTree: support filtering out dvcfiles during walk()

* RepoTree: support recursive isdvc()

* remote: support computing checksums for tree objects

* RepoTree: pull dir contents on walk() if in `fetch` mode

* tests: add func test for DVC dir pull on RepoTree.walk()

* DvcTree: support granular open()

* RepoTree: add get_file_checksum()

* remote: revert checksum changes

* remote: add tree support for cache.save()

* tests: test saving directly to cache from RepoTree

* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree

* RepoTree: state context should be entered by open()/walk() caller

* RepoTree: only pull dir contents when cache has changed

* tree: update tree tests

* fix ds warnings, update comments

* fix windows tests
2020-05-21 06:31:28 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
Paweł Redzyński 451c7aac04 tests: erepo_dir: extract remote setup to separate method (#3748) 2020-05-12 02:05:22 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
Ruslan Kuprieiev bc7f177c7a run: use multistage dvcfiles by default (#3740)
Adding hidden `--single-stage` to simplify the test migration.
2020-05-04 19:55:39 +03:00
Ruslan Kuprieiev 6559b9d222 dvc: use protected mode by default (#3472)
* dvc: use protected mode by default

Fixes #3261
Fixes #2041

* unprotect: adjust help message

https://github.com/iterative/dvc.org/pull/1058/files/21aab371a487acf6f6e6201b29cd832e7c55ed23#r393324668
2020-03-17 19:16:21 +02:00
Alexander Schepanovski 6ccae711fe dvc: refactor config (#3298)
* dvc: refactor config

Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:

    with config.edit("local") as conf:
        conf["core"]["remote"] = "new-default"

Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict

* test: do not set cache.type for remotes that doesn't support it

* dvc: fix windows absolute paths validation in config

* test: fix windows separate drive test

* dvc: stylistic fixes in config
2020-02-13 19:35:37 +02:00
Alexander Schepanovski 0eb4e7e15f api: move summon out
Will be added to dvcx instead.
2020-01-31 00:02:28 +07:00
Dmitry Petrov e6d0d291e2 Code review feedback and change schema to dict of dvc-objects 2020-01-28 18:44:36 -08:00
Dmitry Petrov d5a3b6f4f6 Summon dataframe support 2020-01-26 02:50:42 -08:00
Dmitry Petrov 8f799210d3 Extract Summon class 2020-01-24 16:35:17 -08:00
Saugat Pachhai 003b5c392a api: fallback to external_repo on error 2020-01-23 14:44:38 +05:45
Jorge Orpinel ec0c39a28e term: review capitalization and backquotes around "dvc"
rel: https://github.com/iterative/dvc/pull/3156#discussion_r369897073
2020-01-22 20:27:01 -06:00
Alexander Schepanovski 72812b519e api: fix get_url() non-dvc repo error message 2020-01-19 23:01:53 +07:00
Alexander Schepanovski ee022eada0 api: adjust get_url requiring dvc repo to the new reality 2020-01-19 21:38:37 +07:00
Saugat Pachhai 50206d44dc test: fix test related to path on windows
On windows, `str(exc_info)` is returning escaped path, eg:
'<ExceptionInfo UrlNotDvcRepoError("URL \'C:\\\\Users\\\\travis\\\\AppData\\\\Local\\\\Temp\\\\pytest-of-travis\\\\pytest-0\\\\popen-gw2\\\\test_get_url_git_only_repo_thr0\' is not a dvc repository.") tblen=2>'

This commit just replaces the double slashes with single one. Bit of a hack, I know.
2020-01-16 21:06:52 +05:45
Saugat Pachhai aa7119a741 get: test get_url raises exception if not a dvc repo 2020-01-16 19:28:14 +05:45
Ruslan Kuprieiev 8041d70552 Merge pull request #3044 from pared/2896_api
tests: api: convert to dir helpers
2020-01-06 16:21:31 +02:00
Alexander Schepanovski 5dd2d7b192 test: do not use repo template in erepo_dir 2020-01-04 10:47:13 +07:00
Alexander Schepanovski b194bfa1ff test: do not bootstrap versions in erepo_dir 2020-01-04 10:47:13 +07:00
pawel af425318aa tests: api: roll back erepo usage 2020-01-04 03:10:43 +01:00
pawel 020c0496a6 tests: api: convert to dir helpers 2020-01-04 03:08:51 +01:00
pawel d9eaabe9bf some fixmes 2020-01-04 03:08:51 +01:00
pawel 39f010e50a api: open: fix open scm controlled test 2020-01-04 03:08:51 +01:00
Alexander Schepanovski 1e5fb102a2 test: add @TmpDir.chdir() 2020-01-03 16:58:32 +07:00
Mr. Outis 086acf6404 py3: dvc/utils/compat.py -> dvc/compat.py 2019-12-31 23:42:07 -06:00
Mr. Outis 49264b55ac py2: fix problem with ruamel.yaml and unicode 2019-12-25 13:07:40 -06:00
Mr. Outis 4a25612282 💅 refactor 2019-12-25 01:04:52 -06:00
Mr. Outis 75ad14aaa2 💅 black 2019-12-25 00:43:33 -06:00
Mr. Outis 0658622ca2 summon: pair-programming adjustments 2019-12-25 00:42:55 -06:00
Mr. Outis 1e65c76ef1 summon: raise SummonError appropiately 2019-12-24 20:20:03 -06:00
Mr. Outis fbb7aebd31 tests/summon: use different args for non-canon.yml 2019-12-24 19:50:43 -06:00
Mr. Outis 1c22cd5af4 summon: adjust comments 2019-12-24 02:20:08 -06:00
Mr. Outis addc56c232 summon: allow different YAML file names 2019-12-18 23:38:52 -06:00
Mr. Outis c852d3316d 💅 black 2019-12-18 23:34:43 -06:00
Mr. Outis 4273ff49b3 summon: params -> args; use summon key in yaml 2019-12-18 23:32:28 -06:00
Mr. Outis 29339d29a2 summon: file & method -> call 2019-12-18 23:25:45 -06:00
Mr. Outis 1b3c771df5 summon: artifacs.yaml -> dvcsummon.yaml; artifact -> object 2019-12-18 22:42:55 -06:00
Mr. Outis 32f35ea686 summon: remove scm.add, *_gen already handles it 2019-12-18 22:29:44 -06:00
Mr. Outis 5d66a572d4 summon: support overriding parameters 2019-12-18 17:20:07 -06:00
Mr. Outis d0344ce975 summon: grimoire/spells -> artifacts 2019-12-18 17:00:39 -06:00
Mr. Outis 789db9169a summon: support checking out dependencies 2019-12-18 16:41:03 -06:00
Mr. Outis b921730f37 summon: safely get a module; change naming 2019-12-18 13:32:23 -06:00
Mr. Outis 3d133186d4 summon: ugly hack 2019-12-18 00:48:10 -06:00
c27988a042 test: minor fixture refactor fixes from @pared and @efiop
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-Authored-By: Paweł Redzyński <[email protected]>
2019-12-02 23:33:44 +01:00
Alexander Schepanovski 0385810662 test: fix python 2, 3.5 and mac issues with new fixtures 2019-12-02 23:32:11 +01:00
Alexander Schepanovski fcebad821e test: refactor tmp dir helper fixtures
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
2019-12-02 23:31:54 +01:00
Saugat Pachhai 22c0a644e5 address @efiop's suggestions 2019-12-01 16:08:53 +05:45
Saugat Pachhai f4f836e17c Merge branch 'master' into gs-external-deps
* master:
  travis: windows: use python 3.7.5
  test: skip non supported remotes fast in api tests
2019-12-01 09:01:47 +05:45