Commit Graph
6520 Commits
Author SHA1 Message Date
Ruslan Kuprieiev f2ba67aece build: upgrade pyinstaller to 4.2 (#5682)
* build: upgrade pyinstaller to 4.2

* pyinstaller: hooks: remove hook-googleapiclient.model.py

https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-googleapiclient.model.py

* gha: use python 3.8 for pyinstaller
2021-03-24 09:43:05 +02:00
Ruslan Kuprieiev 19cdef1675 build: separate building and packaging (#5675)
* build: separate pyinstaller and fpm/innosetup

* gha: upload deb/rpm/osxpkg/exe artifacts

* build: dynamically generate dvc version
2021-03-23 19:55:44 +02:00
Saugat Pachhai dc3fe6fbec Update pylint-pytest to 1.0.3 (#5673) 2021-03-23 15:28:33 +05:45
Ruslan Kuprieiev 619c55d399 dvc: bump to 2.0.7 2.0.7 2021-03-22 15:29:52 +02:00
Alex Ford b3bfa346c3 Using existing logging level if defined in addLoggingLevel (#5632)
* Use existing level if defined in addLoggingLevel

Fixes https://github.com/iterative/dvc/issues/4546

* Add addLoggingLevel test

* Format fix.
2021-03-22 15:15:16 +02:00
Saugat Pachhai 0a8927e3d8 diff: clear calculated renamed files from inverted hashmap (#5652)
* diff: clear calculated renamed files from inverted hashmap

We were trying to reuse the first encounter of file from inverted hashmap again and again, which was throwing ValueError

* refactor

* remove ordering

* add tests
2021-03-22 15:13:52 +02:00
Batuhan Taskaya 35597ed83d fsspec: skip instance cache on all filesystems (#5668) 2021-03-22 14:36:21 +02:00
Batuhan Taskaya 21fc06e947 fsspec: support raw buckets on path splitting (#5658) 2021-03-22 02:01:06 +02:00
Saugat Pachhai f2b216261e move Repo.diff related test to diff (#5654) 2021-03-19 16:58:21 +05:45
Batuhan Taskaya 6026573860 update: use unresolved path with --to-remote calls (#5631) 2021-03-17 16:48:00 +02:00
Ruslan Kuprieiev dcc55aa645 dvc: bump to 2.0.6 2.0.6 2021-03-17 00:01:10 +02:00
Batuhan Taskaya 862e18488c gs: process etag when it is available (#5634) 2021-03-16 14:08:59 +02:00
Saugat Pachhai 0b810fb256 Advertise Python 3.9 support in classifiers 2021-03-16 13:37:37 +05:45
Helio MachadoandPeter Rowlands 4ed55d0051 build: add pyinstaller hook for googleapiclient (#5619)
* build: add pyinstaller hook for googleapiclient

The Google API Discovery Service JSON files from
googleapiclient/discovery_cache/documents were being
excluded due to a bug and, particularly, drive.v2.json
is essential for our gdrive backend. Now we're explicitly
asking PyInstaller to embed these files on the generated packages.

Fixes #5618

* Suppress linter error for unrecognized imports

Co-authored-by: Peter Rowlands (변기호) <[email protected]>

* Apply black style after manual code review

Co-authored-by: Peter Rowlands (변기호) <[email protected]>
2021-03-14 12:56:49 +02:00
nik123 6ac7def6f1 setup: remove duplicate 'pytest-cov' dependency (#5617)
* setup: remove duplicate 'pytest-cov' dependency

* setup: returned pytest-cov version
2021-03-14 14:08:11 +05:45
Peter Rowlands (변기호) 8c1d46e502 checkpoints: completely remove checkpoint outs on exp run --reset (#5586)
* checkpoints: remove checkpoint outs on --reset

* exp run: no longer prune/reset lockfiles

* update tests

* make --queue imply --reset unless --rev is provided
2021-03-13 01:29:50 +09:00
Batuhan Taskaya 11b0581536 import-url: support importing empty directories on object storage services (#5536)
* import-url: support importing empty directories on object storage services

Empty directories are basically files with a trailing slash at the end of their names,
with empty contents on object service providers (such as azure, google storage). This
patch implements the `is_empty()` method for the `FSSpecWrapper` to determining whether
a path is empty (directory or a file) and also tweaks `.isdir()` in order to see empty
directories.

The base class is only used by azure and gcsfs for now, but if we go on the route of
migrating normal providers (like proper filesystems, not just object storages), we might
need to do these tweaks on a subclass (like `ObjectStorageFSSpecWrapper` etc), but I
don't think it is neccesary as of now.

The `is_empty` method is already there, and also used in the importing part to
warn users that they are importing something empty (https://github.com/iterative/dvc/blob/e385e59a098cd661ea1ea4257c9a5735c5c32bc6/dvc/output/base.py#L252-L254)
so this is not a new method. Just for the reference, as far as I can locate there
is no variant of this method in the fsspec.

* tests: revert custom mkdir for cloud storages

* tests: allign assingments

* add some comments
2021-03-12 18:11:08 +02:00
Batuhan Taskaya abcb5b2f6f gs: disable consistency checker (#5613) 2021-03-12 14:04:54 +02:00
Peter Rowlands (변기호) d365e6d5b4 exp run: fix issue where duplicate workspace runs would incorrectly conflict (#5611)
* exp run: gracefully handle duplicate workspace runs

* exp run: fix issue where errors could be logged twice

* move ref conflict check into its own function
2021-03-12 20:13:49 +09:00
Peter Rowlands (변기호) bd470e9586 stage add: unsuppress -c/--checkpoints help text (#5599)
* stage add: unsuppress -c/--checkpoints help text

* review fix
2021-03-12 14:47:12 +09:00
Peter Rowlands (변기호) fe4cb9c439 exp run: do nothing if result matches parent (HEAD) (#5600) 2021-03-12 14:09:03 +09:00
Ruslan Kuprieiev 2db55032c4 objects: file: remove unused load() method (#5610) 2021-03-12 05:35:48 +02:00
Peter Rowlands (변기호) 686898240c Revert "repro/exp run: use git status when reminding to git-track files (#5544)" (#5596)
This reverts commit ce2d7c65db.
2021-03-12 11:50:13 +09:00
Saugat Pachhai d245008987 plots: support opening plots file directly in the browser (#5584)
* plots: support opening plots file directly in the browser

* add tests

* fix path on windows tests
2021-03-12 04:23:07 +02:00
Ruslan Kuprieiev 29ad87ebe2 objects: split into separate files (#5609) 2021-03-12 04:12:46 +02:00
Ruslan Kuprieiev 4af3b3d20d dvc: refactor objects (#5607)
* dvc: get rid of DirInfo

DirInfo was created before we had proper objects, as a way to have a
structure that represents a staged directory. Right now we have a
dedicated Tree object that is specifically meant for that.

As a result, HashInfo no longer needs to have dir_info and we no longer
need `get_hash/get_dir_hash` helpers.

* objects: move Tree into tree.py
2021-03-12 01:10:34 +02:00
Saugat Pachhai b691b15007 Add a link to help debugging more information regarding the issue (#5601)
* Add a link to help debugging more information regarding the issue

* Update .github/ISSUE_TEMPLATE/bug_report.md
2021-03-11 16:02:53 +02:00
Saugat Pachhai fd66ef6bc1 http: fix info() if there is no content-length header (#5602) 2021-03-11 16:01:24 +02:00
Saugat Pachhai e709437cd2 update pylint and pin dependencies (#5603)
Started having some issues with pylint-pytest, pinning it as well
2021-03-11 15:24:27 +05:45
Saugat Pachhai 6c00fedb46 remove flaky-service
Looks like the project is dead: https://github.com/GoogleCloudPlatform/flaky-service.

And, the site https://flaky.dev seems to be dead.
2021-03-11 15:21:12 +05:45
Peter Rowlands (변기호) cc9ded2081 exp: subclass tempfile.TemporaryDirectory to handle cleanup (#5598)
* exp: subclass tempfile.TemporaryDirectory to handle cleanup

* override cleanup() since cls._rmtree does not exist in <3.8
2021-03-11 14:54:36 +09:00
Dmitry Petrov 5b300d8aae Add a link to Barrak&Adams citation (#5592) 2021-03-11 00:01:05 +02:00
Ruslan Kuprieiev cf1070a531 dvc: bump to 2.0.5 2.0.5 2021-03-10 11:33:47 +02:00
Laurens Duijvesteijn 9f224ecde3 scm: add force parameter to checkout() (#5582)
The `GitPythonBackend.checkout()` method already implicitly supported
this by forwarding `**kwargs` to the backend. `pygit2` didn't support
this yet.

This patch fixes this inconsistency.
2021-03-10 14:42:48 +09:00
Ruslan Kuprieiev 8433b36266 dvc: bump to 2.0.4 2.0.4 2021-03-09 17:59:26 +02:00
Batuhan Taskaya 3e75fe99e3 gdrive: change the temporary directory location to .dvc/tmp (#5583) 2021-03-09 17:44:48 +02:00
Batuhan Taskaya b221c16c89 pull: proper path join for partial pulls (#5579)
* pull: proper path join for partial pulls

* use os.path.join in the tests for windows
2021-03-09 15:36:44 +02:00
Dmitry Petrov 7ae2d37116 Add Barrak, Eghan & Adam citation (#5573) 2021-03-09 14:50:42 +02:00
Laurens Duijvesteijn e0a0342ab3 fs: ssh: pass timeout to paramiko's connect() (#5569)
Prior to this commit, DVC could hang indefinitely when connecting to
remote SSH filesystems. This should address this.
2021-03-08 22:59:26 +02:00
Peter Rowlands (변기호) d68f6d3696 pygit/dulwich: implement stash functionality (#5286)
* pygit2: disable set_ref with reflog message

* git: implement is_dirty in dulwich

* pygit2: implement _stash_apply

* pygit: update to 1.5.0

* enable set_ref(message=...) for pygit2

* git: replace backend reflog_delete in favor of _stash_drop

* pygit: support stash_push with custom refs

* update tests for backend stash_drop behavior

* pygit: raise MergeConflictError on failed stash apply

* pygit: reload index from disk before stash apply

* update stash_apply checkout strategy

* mark flaky test
2021-03-08 17:24:35 +02:00
Ruslan Kuprieiev b4232bdc7c state: use pickle v4 (#5571)
Kudos @skshetry for the discovery, research and fix.

Fixes #5563
2021-03-08 17:00:40 +02:00
Batuhan Taskaya f7d8761734 gdrive: use the normal file backend for saved credentials (#5554)
* gdrive: sync with pydrive's save backend changes

* bump pydrive version

* better temporary credentials managements for service accounts

* proper temp var management
2021-03-08 16:30:05 +02:00
Ruslan Kuprieiev 70c445d1ee utils: makedirs: ignore chmod errors (#5570)
Same as with `protect()`, these errors might happen if fs is immutable
or doesn't support chmod()-ing. We can safely ignore these.

Fixes #5552
2021-03-08 16:28:02 +02:00
Ruslan Kuprieiev 47bcb02ef1 fs: git: use pygit2 (#5559) 2021-03-06 02:54:13 +02:00
Ruslan Kuprieiev 4a75067f4c github: template: use dvc doctor instead of dvc version 2021-03-05 16:08:46 +02:00
Ruslan Kuprieiev 70f4c44ec6 dvc: bump to 2.0.3 2.0.3 2021-03-05 13:08:22 +02:00
af37527560 setup: rename azure.indentity to azure-identity (#5548)
* simple like that

* Update dvc/version.py

Co-authored-by: Marc-Antoine Belanger <[email protected]>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
2021-03-05 01:43:32 +02:00
Ruslan Kuprieiev 71f8ffe13d dvc: bump to 2.0.2 2.0.2 2021-03-05 01:04:00 +02:00
Ruslan Kuprieiev 06aad3e2b3 dependency: repo: try to fetch before staging (#5549)
Related to https://github.com/iterative/dvc-bench/issues/234
2021-03-05 01:03:33 +02:00
Peter Rowlands (변기호) 1e7df0c27e dulwich: update to 0.20.20 (#5545) 2021-03-04 12:37:19 +02:00