* tests: isolate dvc config and directories
by introducing envvars to change default values. This lets pytest
handle the deletion of tmpdirs, instead of filling /var/tmp with
hundreds of thousands of directories. Also isolates test run from
faulty global/system config files.
* move to dvc/env and sort
* tests: fix logic to gc.collect() on Windows + 3.11
* dvc.cli: cache get_main_parser()
* tests: isolate environment on a session-level
* bring back gc.collect()
* cleanup linters: get rid of pylint-pytest, run pylint only on dvc/ directory, etc.
* remove pylint direcitves
* Apply suggestions from code review
* Update pyproject.toml
There seems to be a regression in Python 3.11, where the sqlite
connections are not deallocated, due to some internal changes in
Python 3.11, where they are now using LRU cache. They are not deallocated
until `gc.collect()` is not called.
See https://github.com/python/cpython/issues/97641.
This affects only Windows, because when we try to remove the tempdir for
the exp run, the sqlite connection is open which prevents us from
deleting that folder.
Although this may happen in real scenario in `exp run`, I am only fixing
the tests by mocking `dvc.close()` and extending it by calling
`gc.collect()` after it. We could also mock `State.close()` but didnot
want to mock something that is not in dvc itself.
The `diskcache` uses threadlocal for connections, so they are expected
to be garbage collected, and therefore does not provide a good way to
close the connections. The only API it offers is `self.close()` and that
only closes main thread's connection. If we had access to connection,
an easier way would have been to explicitly call `conn.close()`.
But we don''t have such option at the moment.
Related: https://github.com/iterative/dvc/pull/8404#issuecomment-1310024661
GHA Failure: https://github.com/iterative/dvc/actions/runs/3437324559/jobs/5731929385#step:5:57
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.
Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.
See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.
dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.
dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
* test: fix test failure in case of different default branch
When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.
This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.
Set default git branch to main for test CI
Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.
Introduce isolation for git configuration
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
This is a very early version of test utils for dvc plugins. This module
consists of existing tests with minimal changes and is going to evolve
a lot as our plugins are getting filled in and pytest-test-utils matures.
we just need to mark them with needs_internet marker.
Unfortunately, it's not possible to do that automatically
with the fixtures as fixtures are per-session and we need
something on more higher level (which the marker is not passed
through in the pytest).
Before, we used to set it during initialization. But this
made it hard to test using capsys as capsys changes the
reference to sys.stdout and sys.stderr with the mocked one
during the test. But since we are using dvc.ui globally and
setting it to a real sys.stdout and sys.stderr before capsys
comes into effect, testing was not possible with it.
Now, by using property hack, we will be using the values of
sys.stdout and sys.stderr during the time of write, so the mocking
would work and would be easier to test.
* don't allow loading stages without `cmd`
This was loosened when introducing parametrization.
But, `foreach`..`do` should be considered first
before stage's regular structure. And, cmd is made
required (as it should have been)
Related: #5371, #5370, #5312
* fix schema
Some of the tests are run by default, other requires
`--enable-{remote}` to run. The remotes that are run by default
also have a `--disable-{remote}` flag. All of the remotes can be
enabled using `--all` flag.
To use this functionality in the fixture,
`test_config` fixture has to be used, and requires to call
`test_config.requires(remote)`.
In an individual test, you can simply use `pytest.mark.remote`
to make them skip or run.
* pylint: enable checks for imports
* ci: add github action for pylint
* enable pylint on pre-commit
* Install requirements
* use local pylint installation rather than of pre-commit's
* fix import errors
* more checks
* Silence unused imports
* Make a copy of dict
* remote.http: support _upload() via HTTP POST
- uploaded files are sent as chunked encoding POST data
Fixes#3247
* remote.http: add support for HTTP basic and digest auth
- username, password, ask_password options work the same way as for
ssh remotes
- basic_auth (bool) and digest_auth (bool) options added for http(s)
remotes
- digest_auth takes precedence over basic_auth if both are enabled
* remote.http: auth_method() unit tests
* tests: Add functional tests for HTTP remotes
- HTTP remotes now tested locally using a SimpleHTTPServer instance that
allows reading/writing to a temp directory
* tests: fix http tests on py3.5, 3.6
* remote.http: add custom auth method
* s/custom_header/custom_auth_header/
* Apply suggestions from code review
Co-Authored-By: Saugat Pachhai <[email protected]>
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-authored-by: Saugat Pachhai <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
skshetry
Ruslan Kuprieiev
Saugat Pachhai (सौगात)
Paweł Redzyński
David de la Iglesia Castro
Peter Rowlands
Amrit GhimireandSaugat Pachhai
Batuhan Taskaya
Batuhan Taskaya
Casper da Costa-Luis
Mr. Outis
Mayur Kularni
Alexander Schepanovski
Ruslan Kuprieiev