Commit Graph
60 Commits
Author SHA1 Message Date
skshetry e5eec59cb8 test on 3.11 (#8196)
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.

Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.

See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.

dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.

dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
2022-08-29 19:58:17 +05:45
Paweł Redzyński 700b648da9 plots: tests: integration test for vscode
Closes: #7549
2022-04-26 21:32:56 +02:00
David de la Iglesia Castro ddd6c97f71 render: Extract to dvc_render
Create and use separate package for rendering logic.
Closes #6944
2022-03-28 19:34:05 +02:00
Peter Rowlands 59b8498905 fs: remove unused connection pool module 2022-03-01 15:36:49 +09:00
Saugat Pachhai (सौगात) 672f079cc6 tests: clean repositories before each test runs 2022-01-19 20:21:48 +05:45
Amrit GhimireandSaugat Pachhai 44a1146915 test: fix test failure in case of different default branch (#6857)
* test: fix test failure in case of different default branch

When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.

This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.

Set default git branch to main for test CI

Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.

Introduce isolation for git configuration

Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
2022-01-07 11:13:04 +05:45
Ruslan Kuprieiev 09f48a5aaa tests: introduce dvc.testing
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
2021-12-02 09:08:40 +02:00
Saugat Pachhai (सौगात) cab60b54f8 live: import webbrowser lazily; only patch webbrowser.open 2021-11-23 22:08:21 +05:45
David de la Iglesia Castro fc7caa487f tests: Patch webbrowser.open on session level
Closes #7014
2021-11-23 22:08:21 +05:45
Paweł Redzyński a1b146669c plots: load templates from package (#6550)
* plots: vega: make interactive linear default plot

* plots: stop dumping default templates

* plots: move templates to package initial

* plots: update default plots schema

* plots: write defaults on init

* templates: migrate to importlib.resources

* fixup

* fixup
2021-10-14 08:40:31 +02:00
Batuhan Taskaya d7489bb65f hdfs: enable mocked hdfs tests by default (#6766) 2021-10-08 12:42:41 +02:00
Batuhan Taskaya 6d3045320b ssh: use a docker-based testing environment (#6677) 2021-09-23 16:00:48 +03:00
Saugat Pachhai 198cdeec06 tests: scriptify function into module (#6619)
so that we can write tests having larger side-effects than
the run_copy that we use as a hack
2021-09-15 22:01:10 +03:00
Paweł Redzyński ab08db5065 plots: introduce support for images (#6431)
* plots: support images initial

* plots: data: unify PlotsData

* plots: data conversion refactor

* plots: renderers: abstract functionality

* plots: uncommenting tests

* plots: self implement grouping

* plots: rename group

* plots: extract rendering tests

* plots: move rendering tests to unit

* plots: add image rendering tests

* plots: dont use dpath to find vega plot

* plots: test find_vega

* plots: test render function

* plots: move rendering out of repo package

* plots: refactoring

* plots: self review refactor
2021-08-26 20:00:31 +03:00
Peter Rowlands (변기호) 9dda6bd32c git: update dulwich to 0.20.22 (#6049)
* git: update dulwich to 0.20.22

* git: disable reading system config in CI
2021-05-24 17:40:14 +09:00
Batuhan Taskaya d4a3df73df s3: fsspec migration (#5683)
* s3: migrate to s3fs

* s3: strip etag

* s3: disable instance cache

* tests: use the proper remote:// form in test_checkout

* tests: move/drop some tests

* setup: depend on s3fs

* tests: normalize s3 tests with some cleanups

* Use _download/_upload with old implementation

* fsspec: cache .exists() results

* setup: pin the boto version

* setup: install s3fs[boto3]

* s3: proper exception management

* s3: config improvements

* setup: reference without quotes

* setup: install boto separately

* setup: install moto server

* s3: mocking adjustments for moto_server

* s3: mark some fixtures with scope=session

* s3: s3_fake_creds_file scopes to func

* fsspec: add some docs regarding exists() optimization

* fsspec: exists() fallback

* fsspec: revert exists() optimization

* tests: ssl_verify => verify

* setup: temporarily fetch from master

* s3: add boto as a requirement

* tests: do an extra check regarding race conditions

* tests: use motoserver docker image

* tests: wrap config deletion with try/finally

* tests: disable s3 by default

* setup: upgrade fsspec
2021-05-05 11:36:10 +03:00
Saugat Pachhai ff56379f8a tests: rerun gdrive and gs on failures (#5959)
we just need to mark them with needs_internet marker.
Unfortunately, it's not possible to do that automatically
with the fixtures as fixtures are per-session and we need
something on more higher level (which the marker is not passed
through in the pytest).
2021-05-04 17:59:50 +03:00
Saugat Pachhai b5f31daab0 ui: Use sys.stdout and sys.stderr dynamically (#5729)
Before, we used to set it during initialization. But this
made it hard to test using capsys as capsys changes the
reference to sys.stdout and sys.stderr with the mocked one
during the test. But since we are using dvc.ui globally and
setting it to a real sys.stdout and sys.stderr before capsys
comes into effect, testing was not possible with it.

Now, by using property hack, we will be using the values of
sys.stdout and sys.stderr during the time of write, so the mocking
would work and would be easier to test.
2021-03-30 23:44:46 +03:00
Ruslan Kuprieiev b0fd40f12c dvc: rename tree -> fs (#5461) 2021-02-13 15:40:44 +02:00
Saugat Pachhai 02a0d1af52 stage: implement add/create methods in stage loader (#5397)
* stage: add/create methods in stage loader

* minor fixups
2021-02-08 18:30:10 +05:45
Saugat Pachhai 131eb7b3a1 don't allow loading stages without cmd (#5373)
* don't allow loading stages without `cmd`

This was loosened when introducing parametrization.
But, `foreach`..`do` should be considered first
before stage's regular structure. And, cmd is made
required (as it should have been)

Related: #5371, #5370, #5312

* fix schema
2021-02-01 13:23:40 +05:45
Paweł Redzyński 2277a1a655 metrics/plots/params: return ordered (#5297)
* metrics/plots/params: return ordered

* metrics/params/plots: add tests for return order
2021-01-22 20:57:43 +02:00
Saugat Pachhai e33274723d Move flags from tests to config (#4943) 2020-11-23 16:21:24 +05:45
Saugat Pachhai 5faae2a6e9 test: allow selectively running tests for remotes (#4920)
Some of the tests are run by default, other requires
`--enable-{remote}` to run. The remotes that are run by default
also have a `--disable-{remote}` flag. All of the remotes can be
enabled using `--all` flag.

To use this functionality in the fixture,
`test_config` fixture has to be used, and requires to call
`test_config.requires(remote)`.
In an individual test, you can simply use `pytest.mark.remote`
to make them skip or run.
2020-11-23 11:43:39 +05:45
Ruslan Kuprieiev 76228a2118 dvc: split remotes from trees (#4212)
* dvc: remove tree-specific wrappers from remote

This helps combat the remote/tree/cache confusion.

* dvc: split remotes from trees

Related to #4050
2020-07-15 23:44:09 +03:00
Saugat Pachhai ad306d7bf3 Enable most of the pylint checks (#4111) 2020-06-25 18:10:14 +03:00
Saugat Pachhai b81ace54df Enable pylint checks (#4092)
* pylint: enable checks for imports

* ci: add github action for pylint

* enable pylint on pre-commit

* Install requirements

* use local pylint installation rather than of pre-commit's

* fix import errors

* more checks

* Silence unused imports

* Make a copy of dict
2020-06-24 18:02:39 +03:00
Ruslan Kuprieiev 51a8c782ae tests: create remote fixtures (#4063)
* tests: use remote fixtures

* tests: use remote fixtures for data cloud tests

* tests: treat remotes as tmp dirs
2020-06-18 01:36:13 +03:00
Ruslan Kuprieiev 02a52d038a tests: make ssh_server session scoped (#3898)
Fixes #3888
2020-05-28 16:09:58 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749)
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
2020-05-06 23:56:07 +03:00
68e543f7e1 remotes.http: Support dvc push for http remotes (#3343)
* remote.http: support _upload() via HTTP POST

- uploaded files are sent as chunked encoding POST data

Fixes #3247

* remote.http: add support for HTTP basic and digest auth

- username, password, ask_password options work the same way as for
  ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
  remotes
- digest_auth takes precedence over basic_auth if both are enabled

* remote.http: auth_method() unit tests

* tests: Add functional tests for HTTP remotes

- HTTP remotes now tested locally using a SimpleHTTPServer instance that
  allows reading/writing to a temp directory

* tests: fix http tests on py3.5, 3.6

* remote.http: add custom auth method

* s/custom_header/custom_auth_header/

* Apply suggestions from code review

Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>

Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2020-02-21 12:05:55 +02:00
pawel c2f5e35f1d conftest: remove repo_dir 2020-01-16 10:43:48 +01:00
pawel a4b55a00ec conftest: remove dvc_repo, erepo, git, git_erepo 2020-01-16 10:43:48 +01:00
Mr. Outis b844aea42c py3: remove cast_bytes method 2019-12-31 20:14:25 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
prd d40588aefa temporary_windows_drive fixture: move to dir_helpers 2019-12-16 13:15:53 +01:00
Mayur Kularni ca20a1c93a incorporated feedback 2019-12-08 18:56:38 +05:30
Alexander Schepanovski fcebad821e test: refactor tmp dir helper fixtures
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
2019-12-02 23:31:54 +01:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Casper da Costa-Luis 1e01ce3ac9 isatty (#2550)
* suppress Tqdm when isatty

- fixes #2549

* fix tests

* unconditionally disable Tqdm if not isatty

- fixes https://github.com/iterative/dvc/pull/2550#discussion_r329386950

* add TQDM_DISABLE env var

- update tests
- add documentation

* TQDM_DISABLE => TQDM_ISATTY

* tests: mock.patch sys.stderr

* TQDM_ISATTY => DVC_ISATTY

as per https://github.com/iterative/dvc/pull/2550#discussion_r334565120

* dvc.progress.Tqdm file kwarg

- fixes https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335706955

* dynamic env DVC_ISATTY

- fixes
  https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335811454

* DVC_ISATTY in tests

- fixes https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335811099

* re-add and abstract `env2bool` to `dvc.utils`

- partially reverts commit 524a0df188f566fb3e103f6965f1b6ac9c530886
- fixes
  https://github.com/iterative/dvc/pull/2550/files/ad1f245107c405697c4b61fe58a92fc0265c6588#r335811639

* env2bool tidy

* rename DVC_ISATTY => DVC_IGNORE_ISATTY

- fixes https://github.com/iterative/dvc/pull/2550#discussion_r335947382
2019-10-20 00:54:06 +03:00
pawel 4236f18e0c repo: get: intercept and convert NotDvcRepoErro to UrlNotDvcRepoError 2019-09-03 13:59:04 +02:00
RahulR19 371c57ac24 Renamed git to repo in dvc/scm/git along with changes in some tests. 2019-08-04 23:26:45 +05:30
Alexander Schepanovski 6afc2a38ba test: close pools on tests cleanup
This avoids weird errors as pools are collected by gc in broken env.
2019-07-21 13:40:29 +07:00
Alexander Schepanovski ed387d824d remote: do not swallow exceptions by ssh pool 2019-07-17 22:20:43 +07:00
Ruslan Kuprieiev 918ef646dc appveyor: test py2 and py3
Related to #2166

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-23 13:33:47 +03:00
Ruslan Kuprieiev 1ba8e0171d dvc: replace pkg concept with external repo
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-21 01:16:00 +03:00
prd 0615ff8730 pathinfo: extract relative path method to standalone method 2019-06-18 12:08:19 +02:00
Ruslan Kuprieiev 5941ace270 dvc: initial 'dvc pkg' implementation
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-17 18:12:36 +03:00
Ruslan Kuprieiev f41dcea248 tests: don't use var before assignment
```
repo_dir = <tests.basic_env.TestDirFixture object at 0x7f36c3f3d650>                  
git = <git.Repo "/tmp/dvc-test.2894.0LMoaT.eefefa48-172d-44e4-8c3d-4f65f76c78ee/.git">
                                                                                      
    @pytest.fixture                                                                   
    def dvc_repo(repo_dir, git):                                                      
        try:                                                                          
            dvc = DvcRepo.init(repo_dir._root_dir)                                    
            dvc.scm.commit("init dvc")                                                
            yield dvc                                                                 
        finally:                                                                      
>           dvc.scm.git.close()                                                       
E           UnboundLocalError: local variable 'dvc' referenced before assignment      
                                                                                      
/home/efiop/git/dvc/tests/conftest.py:76: UnboundLocalError                           
```
2019-06-06 13:44:13 +03:00