Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-05 15:28:27 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,671 Commits 7 Branches 579 Tags
2.5.0
Commit Graph
6671 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kurian Benoyandpre-commit-ci[bot] 2485779d59 run: update -h message (#6197) 
* update dvc exp run -h message

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2.5.0 		2021-07-03 23:02:19 +03:00
Ruslan Kuprieiev 247b9c8be8 dvc: bump to 2.5.0 2021-07-03 22:34:11 +03:00
Saugat Pachhai 8d5f9ab6e5 add: do not delete stage files before add (#6239) 
* add: do not delete stage files before add

Because of the way we collect stages and cache them, we were not able to
collect them for the `add` without removing them from the workspace.
As doing so, we'd have two same/similar stages - one collected from the
workspace and the other just created from the `dvc add` in-memory.

This would raise errors during graph checks, so we started to delete them
and reset them (which is very recently, see #2886 and #3349).

By deleting the file before we even do any checks, we are making DVC
fragile, and results in data loss for the users with even simple mistakes.
This should make it more reliable and robust.

And, recently, we have started to keep state of a lot of things, that by
resetting them on each stage, we waste a lot of performance, especially
on gitignores. We cache the dulwich's IgnoreManager, which when resetted
too many times, will waste a lot of our time just collecting them again
next time (see #6227).

It's hard to say how much this improves, as this very much depends on
no. of gitignores in the repo (which can be assumed to be quite in number
for a dvc repo) and the amount of files that we are adding
(eg: `-R` adding a large directory).

On a directory with 10,000 files (in a datadet-registry repo),
creating stages on `dvc add -R` went from 64 files/sec to 1.1k files/sec.

* add tests

* make the test more specific
 2021-07-03 22:02:06 +03:00
Ruslan Kuprieiev 07f5f6b633 ssh: don't use path_info for connections (#6259) 2021-07-03 21:54:14 +03:00
Ruslan Kuprieiev aa255a1102 output: optimize obj collection (#6277) 
* output: str only once

Calling `__str__` takes a long time and if we are dealing with a large
dataset, it will result in a very substantial time. For example,
locally, for 700K objects, it took 20sec more.

Related to #6276

* output: don't use os.path.join

Takes around 2sec for 700K objects.

* output: use fs.sep

* output: remove unused sep
 2021-07-03 21:53:55 +03:00
Peter Rowlands (변기호) cfcb7e1294 pygit: handle annotated tags in resolve_refish (#6274) 2021-07-02 18:00:47 +03:00
Saugat Pachhai b8d9800e3c separate validation and transformation of args from the add func (#6220) 
* separate validation and transformation of args from add func

* make tests pass for Python3.6

* cleanup

* use dict for attr-access and patch boundarguments.arguments
 2021-07-01 21:15:02 +05:45
dependabot[bot] fe46d50847 build(deps): bump google-cloud-storage from 1.39.0 to 1.40.0 (#6261) 
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.39.0 to 1.40.0.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.39.0...v1.40.0)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-01 21:14:20 +05:45
Batuhan Taskaya e3ff6d54a9 fsspec: loosen the prefix check to cover both None and False (#6246) 
* fsspec: loosen the prefix check to cover both None and False

* gs: disable prefix-based search
 2021-07-01 14:59:35 +03:00
Saugat Pachhai ce45ec8a28 align metrics to right and params to left in exp table (#6254) 
Made a mistake during refactoring on styling before
 2021-06-30 16:05:17 +05:45
Miguel Méndez cc5ba985b0 import: error message when imported file does not exist (#6240) 
* Fix exception handling

* Replace exceptions with PathMissingError
 2021-06-30 15:49:15 +09:00
Ruslan Kuprieiev 87cdbf8ace dvc: bump to 2.4.3 2.4.3 2021-06-29 17:27:21 +03:00
Tomasz Pietruszka eda8cd5846 git: Keep trailing slash when checking if directory is gitignored (#6232) 
* Keep trailing slash when checking if directory is gitignored

* Calling is_ignored with correct type (could be Path-like)

* Fix test to comply to annotated types

* Fix separator issues

* Relaxed typing in is_ignored

* Added test

* Test scm

* Fix test on Windows

* Expanded test_ignored_dir_unignored_subdirs

* Even more assertions
 2021-06-29 21:49:29 +09:00
pre-commit-ci[bot] 61835f6511 [pre-commit.ci] pre-commit autoupdate (#6243) 
updates:
- [github.com/pre-commit/mirrors-mypy: v0.902 → v0.910](https://github.com/pre-commit/mirrors-mypy/compare/v0.902...v0.910)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-06-29 07:24:12 +05:45
Jon Crall e72354e98e Loosen versions for fsspec and pygtrie (#6242) 2021-06-28 23:57:49 +03:00
Ruslan Kuprieiev d1b36e36ca dvc: bump to 2.4.2 2.4.2 2021-06-28 20:19:42 +03:00
Batuhan Taskaya 6ec66ab3de output: don't lower the etag field on the schema when deserializing (#6238) 2021-06-28 20:15:58 +03:00
Peter Rowlands (변기호) 7c13897674 utils: normalize case in path_isin functions (#6236) 
* utils: normalize case in path_isin

* utils: normalize case in contains_symlink_up_to

* split platform test
 2021-06-28 19:21:44 +09:00
dependabot[bot] 1db4637f2d build(deps): bump crc32c from 2.2 to 2.2.post0 (#6234) 
Bumps [crc32c](https://github.com/ICRAR/crc32c) from 2.2 to 2.2.post0.
- [Release notes](https://github.com/ICRAR/crc32c/releases)
- [Changelog](https://github.com/ICRAR/crc32c/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ICRAR/crc32c/compare/v2.2...v2.2.post0)

---
updated-dependencies:
- dependency-name: crc32c
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-28 12:17:14 +03:00
dependabot[bot] ca3757b45b build(deps): bump urllib3 from 1.26.5 to 1.26.6 (#6235) 
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.5 to 1.26.6.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.6/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.5...1.26.6)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-28 08:43:13 +05:45
Ruslan Kuprieiev 4e792ae61c setup: update fsspec to 2021.6.1 (#6219) 
* setup: update fsspec to 2021.6.1

* fs: local: remove temp workaround
 2021-06-24 10:06:42 +05:45
dependabot[bot] 6b13f76fdb build(deps): bump mypy from 0.902 to 0.910 (#6212) 
Bumps [mypy](https://github.com/python/mypy) from 0.902 to 0.910.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.902...v0.910)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-24 10:17:52 +09:00
Ruslan Kuprieiev 53e9460c65 staging: remove redundant exists() call (#6217) 2021-06-24 00:19:06 +03:00
dependabot[bot] dafe6e991c build(deps): bump google-cloud-storage from 1.38.0 to 1.39.0 (#6211) 
Bumps [google-cloud-storage](https://github.com/googleapis/python-storage) from 1.38.0 to 1.39.0.
- [Release notes](https://github.com/googleapis/python-storage/releases)
- [Changelog](https://github.com/googleapis/python-storage/blob/master/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-storage/compare/v1.38.0...v1.39.0)

---
updated-dependencies:
- dependency-name: google-cloud-storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-23 19:07:43 +00:00
Ruslan Kuprieiev 09946847d2 dvc: bump to 2.4.1 2.4.1 2021-06-23 16:08:10 +03:00
Saugat PachhaiandRuslan Kuprieiev 6e5e62857b Bump fsspec version (#6214) 
* Bump fsspec version

* local: use fsspec 2021.06.0 with a temp hack

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-06-23 16:07:32 +03:00
Saugat Pachhai b7c6df0ece Create CODEOWNERS (#6213) 2021-06-23 11:56:37 +05:45
Ruslan Kuprieiev 6779d3d431 dvc: bump to 2.4.0 2.4.0 2021-06-22 20:14:46 +03:00
pre-commit-ci[bot]andRuslan Kuprieiev 5844a0cd5b [pre-commit.ci] pre-commit autoupdate (#6167) 
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/ambv/black: 21.5b2 → 21.6b0](https://github.com/ambv/black/compare/21.5b2...21.6b0)
- [github.com/timothycrosley/isort: 5.8.0 → 5.9.1](https://github.com/timothycrosley/isort/compare/5.8.0...5.9.1)
- [github.com/pre-commit/mirrors-mypy: v0.812 → v0.902](https://github.com/pre-commit/mirrors-mypy/compare/v0.812...v0.902)

* test_requirements: add types packages

* pre-commit: add additional dependencies for mypy

* pre-commit: fix formatting

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2021-06-22 13:59:11 +03:00
Saugat Pachhai ce3a3ab100 pass size to upload_fobj (#6187) 
This allows us to select between chunked and non-chunked uploading
in Webdav, and hopefully in the http itself.
 2021-06-22 13:16:15 +03:00
yehiyam d1a219df0a repo: preserve meta field on move (#6201) 
fixes #6200
 2021-06-22 13:12:20 +03:00
Saugat Pachhai 476af100de ui: migrating rest to the ui.write (#6176) 
* complete migration to ui.write

* fix small bugs
 2021-06-21 22:14:59 +03:00
Saugat Pachhai f2998630a3 env: track DVC_IGNORE_ISATTY as a env that we support (#6202) 2021-06-21 05:35:39 +00:00
Laurens Duijvesteijn 9a7b6d2a1a serialize: Wrap UnicodeDecodeErrors (#6193) 
Also expose `encoding` and `path` as attributes, so programmatic
consumers can access it easily.
 2021-06-19 11:35:50 +05:45
Peter Rowlands (변기호) 1d3524f144 odb: treat external repo dependencies as objects (#6109) 
* objects: add ExternalRepoFile object type

* RepoDependency: implement get_obj (use ExternalRepoFile)

* drop distinction between used external and used objects

* update tests

* remove unneeded fetch from ExternalRepoFile

* wrap erepo errors as an object error

* update cloud/remote functions for new used_objs behavior

* fix update() rev/locked behavior

* update import tests

* remove ExternalRepoFile

* move get_obj and get_used_objs implementations back into repo dependency

* return objects plus associated ODB (remote) in get_used_objs

* odb: add tmp_dir field and support creating Remote from an odb

* return dict mapping odb -> objects in get_used_objs

* use dummy git odb for git imports

* fetch: support pulling from specific odb

* update cloud funcs for get_used_objs usage

* update tests for used_objs behavior

* use fetch in dep.download

* catch circular imports from local fs repos

* add tests for chained and circular imports

* move tmp_dir into odb config

* update return docstring for repo.used_objs
 2021-06-17 14:41:33 +03:00
Batuhan Taskaya 1a5fa228ab azure: add allow_anonymous_login option to the schema (#6190) 2021-06-17 14:24:07 +03:00
Saugat Pachhai adeb9ef63e fix webdav tests and then add some (#6189) 2021-06-17 13:10:53 +03:00
dependabot[bot] 3e93bc286f build(deps): bump pytest-xdist from 2.2.1 to 2.3.0 (#6188) 
Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 2.2.1 to 2.3.0.
- [Release notes](https://github.com/pytest-dev/pytest-xdist/releases)
- [Changelog](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest-xdist/compare/v2.2.1...v2.3.0)

---
updated-dependencies:
- dependency-name: pytest-xdist
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-17 09:37:36 +05:45
Batuhan Taskaya ea89f4a102 add, import-url: preserve size for to-remote/to-cache (#6162) 2021-06-16 18:42:51 +03:00
dependabot[bot] 78299ad345 build(deps): bump pytest-docker from 0.10.1 to 0.10.3 (#6172) 
Bumps [pytest-docker](https://github.com/avast/pytest-docker) from 0.10.1 to 0.10.3.
- [Release notes](https://github.com/avast/pytest-docker/releases)
- [Changelog](https://github.com/avast/pytest-docker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/avast/pytest-docker/compare/v0.10.1...v0.10.3)

---
updated-dependencies:
- dependency-name: pytest-docker
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-15 14:00:38 +05:45
Batuhan Taskaya b9df4451b1 s3, gs, azure: re-enable prefix-based search optimization (#6151) 2021-06-13 16:55:36 +03:00
Saugat Pachhai df67dc2db8 webdav: bring back support for user in url (#6155) 
Also added some optimizations, and merged Webdavs into the same file.
 2021-06-11 19:52:42 +05:45
dependabot[bot] 37fb73606c build(deps): bump mypy from 0.901 to 0.902 (#6153) 
Bumps [mypy](https://github.com/python/mypy) from 0.901 to 0.902.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.901...v0.902)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-11 15:17:38 +05:45
Batuhan Taskaya 0deddf81cf setup: bump fsspec/s3fs/gcsfs to 2021.06.0 (#6137) 
* setup: bump fsspec/s3fs/gcsfs to 2021.06.0

* bump fsspec to upstream
 2021-06-10 15:17:43 +03:00
Ruslan Kuprieiev 429d54e146 stage: monitor: don't forget to unprotect outs (#6134) 
Fixes #6083
 2021-06-10 10:53:51 +02:00
dependabot[bot] 32859763be build(deps): bump mypy from 0.812 to 0.901 (#6140) 
Bumps [mypy](https://github.com/python/mypy) from 0.812 to 0.901.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.812...v0.901)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-10 10:56:39 +05:45
Saugat Pachhai 5012494213 http/webdav: support specifying certs in ssl (#6142) 2021-06-10 02:24:56 +03:00
dependabot[bot] 57c728269d build(deps): bump codecov/codecov-action from 1.5.0 to 1.5.2 (#6139) 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1.5.0 to 1.5.2.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.5.0...v1.5.2)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-06-09 23:30:31 +05:45
pre-commit-ci[bot] 3ca18cb400 [pre-commit.ci] pre-commit autoupdate (#6135) 
updates:
- [github.com/ambv/black: 21.5b1 → 21.5b2](https://github.com/ambv/black/compare/21.5b1...21.5b2)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2021-06-08 12:29:17 +05:45
Dmitry PetrovandIvan Shcheklein 1335c7f5e4 Add Studio to the related technologies list (#6131) 
* Add Studio to the related technologies list

* Add CML to DVC Studio description

* fix k8s names a bit

Co-authored-by: Ivan Shcheklein <[email protected]>
 2021-06-07 23:48:35 +03:00