* move config to pyproject.toml
* ci: use pyproject.toml for cache key
* pyproject.toml: use recursive dependencies for dev and all
* Update pyproject.toml
We don't have aiohttp support yet, due to which we have to install
without cythonized extension.
Similarly, there has been some changes in how dataclasses disallows some
mutable arguments to be set as a class property, due to which
`hydra.conf` fails to import in 3.11.
See https://github.com/python/cpython/pull/29867. For now, I have
patched the module and avoid importing as a pytest-plugin.
dvc-hdfs/pyarrow is not there yet, we'll usually see a release 2-3
months after official reelase.
dvc-hdfs is skipped in 3.11 in extras_requires. `--set-params` will
return a nice error message if hydra cannot be imported in 3.11. I
decided to only patch that in tests.
* test: fix test failure in case of different default branch
When the git config has set global `init.defaultBranch` different than master,
a few test cases were failing.
Currently, the test case is assuming the default branch to be set as
master.
This commit will introduce **initial_branch** option to git init call
which will ensure *master* branch is used as default for out test cases.
Set default git branch to main for test CI
Since main is going to be new default branch, it will let the test case
handle the scenario where default branch is set other than master.
Introduce isolation for git configuration
Co-authored-by: Saugat Pachhai (सौगात) <[email protected]>
pygit2 wheels should be now available.
Had to skip pytest-docker for Windows + Python 3.10 due to
a dependency issue with it's dependency, see:
https://github.com/docker/docker-py/issues/2902.
Linux/Windows machines have only 2 cores, whereas macOS machines have 3 cores. Maybe we are stretching available resources by 2x, let's see if `-n=auto` improves test freezes.
* git: support client kwargs for auth in remote methods
* tests: use docker for git/ssh server
* move dulwich backend into subdir
* add base sync<->async support
* dulwich: add initial asyncssh SSH vendor
* dulwich: use asyncssh as the default SSH vendor
* add test for git over ssh
* test read (fetch) over ssh
* ci: re-enable ssh tests
temporary workaround for #6788, #6789
* move async stubs into dvc.scm.asyn
* tests: use sshfs and not dvc.fs.ssh
* allow explicit 0-length read in asyncssh vendor
* dvc: drop 3.6 support
Considering that 3.6 has ~2 months until EOL, and our deps are
dropping support for it already (e.g. pygit2), we are droppipping
it as well. This will reduce the load on our test CI as well, since
we've started supporting 3.10 recently.
* setup.cfg: adjust requirements
* daemon: remove 3.6 logic
* tests: utils: remove pre 3.7 logic
* use setuptools_scm and build_requires
* remove kwargs from setup.py
* fix tests; fetch in depth in github actions
* use build to build sdist/wheel
* try updating pip and setuptools
* update pip
* remove or unpin wheel
* bring back the version in pyinstaller
* Update scripts/build.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* advertise Python 3.10 support
Pygit2 does not have windows wheels yet and no pyarrow
for 3.10. But these platform-dependent things and
remote-dependent things should not stop us from
advertising support as we are pure-python based
package and we cannot claim support for all of our
dependencies.
* set pygit2 for Python 3.6 to <1.7
* Set timeout for jobs in Github Actions
Sometimes tests are getting frozen(most likely an issue with `pytest-xdist`), and sometimes the pip resolver is slow due to some weird constraints set by us or one of our dependencies.
The default timeout is 360 minutes (6hrs), but our test completes on ~18 minutes at maximum, so I think 30mins should be enough to account for occasional CI slowdown.
Also keeping the total job timeout at 45 mins, we usually require just 4 mins of extra time to do installation and setup but giving it 11 more minutes just in case.
Similarly, `lint` completes in <5 mins usually, but giving it a 10 mins timeout.
This should give us early feedback on the issue.
* Update setup.cfg
* Update setup.cfg
* add support for Python 3.10
pyarrow is not available, so we are skipping tests for it.
And, restricting `hdfs` extras to be not installed in Python3.10.
Pathlib removed `_init` method and `init` parameter, so just working
around that. It's not a big change.
* skip windows on 3.10; update pip/wheel/setuptools during install
pywin32 is not yet available for 3.10 Python, so skipping those for now
* install libgit2 with Python3.10 as pygit2 wheel is not available
* install libgit2
* only run on macOS, libgit2-1.1 not available on ubuntu
* Migrate to ossfs
fix 6177
* Add verification to object stored on oss
* Change the oss remote's dependency.
pass all of the tests
* Performance optimaztion and ossfs version update
* Remove the argument to set the resumable get/put
1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
* Use a new api for performance optimization
* Without filesize progress bar version
* With progress bar version
* Bump and add comments
1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
* Change the name of kwrags following parent class.
* Bump to new ossfs and rewrite upload_fobj
Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
* Optimization on rm_files
* Some optimization follow the old version.
* Update dvc/fs/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* Update dvc/objects/db/oss.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* bump into ossfs 2021.7.2
* Bump into ossfs 2021.7.3OC
* Switch to real_oss_server
* Switch to a real oss server
* Use the oss bucket located in US.
* Remove some emulator related code
* Add envs to ci
* remove some oss pytest XFAIL
Because we didn't use emulator, we can remove some pytest XFAILs
Co-authored-by: Ruslan Kuprieiev <[email protected]>
skshetry
dependabot[bot]
Ruslan Kuprieiev
Amrit GhimireandSaugat Pachhai
Paweł Redzyński
Peter Rowlands (변기호)
GaoandRuslan Kuprieiev