Casper da Costa-Luis
4d140a9944
gdrive: add open ( #3916 )
...
* gdrive: add open
Fixes #3408
Related #2865
Fixes #3897
* dependency: add gdrive
* test: api: open: gdrive
* rollback pydrive dependency
* Revert "dependency: add gdrive"
This reverts commit fd333261d2e9fe1364cc278138c79890a1f5fe7a.
* tests: remotes: GDrive: fix get_url
* tests: api: fully test GDrive
* minor typo fix
* attempt gdrive tests fix
* fix gdrive credentials config
* replace GDrivePathNotFound => FileMissingError
* fix test_open_external[GDrive]
* add ensure_dir_scm
* minor exception fix
2020-06-13 23:53:25 +03:00
Casper da Costa-Luis
2ba73daedc
snap: make future major version upgrades easier ( #4038 )
...
Related: #3999
Related: #4037
2020-06-13 23:27:03 +03:00
Ruslan Kuprieiev
7a31a35cbf
dvc: bump to 1.0.0b0
2020-06-13 00:56:14 +03:00
Saugat Pachhai
24a14563cc
run: preserve order when -f/--force overwrite stage ( #4035 )
...
This commit preserves comments, meta and order inside a dvc.yaml
when `-f/--force` is used. But, it does not maintain comments/meta
for .dvc files.
Related: #4014
2020-06-13 00:55:15 +03:00
Ruslan Kuprieiev
a1d3440f85
metrics: diff: introduce --precision ( #4026 )
...
Rounding to 5 digits by default.
Fixes #3904
2020-06-13 00:54:42 +03:00
Saugat Pachhai
45dcb7c27d
Update dag.py ( #4036 )
2020-06-12 20:33:19 +03:00
Saugat Pachhai
9ff6fef53a
bugfix: get dump folder contents in respective output folder ( #4034 )
...
The relative path for external git directory was computed relative to
the root of the repo itself, which created a relative path which is
current working directory. So, all of the contents of the erepo to download
was dumped in the pwd rather than inside of it's own directory.
Fix #4033
2020-06-12 20:31:18 +03:00
Saugat Pachhai and Ruslan Kuprieiev
9e7dc2640e
schema: provide a jsonschema for dvc.yaml ( #4028 )
...
* schema: provide a jsonschema for dvc.yaml
* Update scripts/schema/dvc-yaml.json
Co-authored-by: Ruslan Kuprieiev <[email protected] [email protected]
2020-06-12 20:09:59 +05:45
Paweł Redzyński
584c97a58d
plots: fix handling of multiple plots from single revision ( #4031 )
2020-06-12 16:08:08 +03:00
Saugat Pachhai
b94a18cfb9
tests: tests for dvcfile ( #4032 )
2020-06-12 15:54:05 +03:00
Peter Rowlands (변기호)
9ead641c95
remote: separate cloud remote and cloud cache classes ( #4019 )
...
* remote: move get_file_checksum() into tree
* makes RemoteTree and RepoTree consistent with regard to checksum
calculation
* remote: save() now takes explicit tree parameter
* when tree is remote.tree, save will be a move + link operation
(same as default existing behavior)
* when saving path from a different tree, save will be a copy operation
* tests: update for moved remote/tree functions
* remote: move get_checksum into tree
* remote: separate cloud remote and cache classes
* tests: update unit tests for remote/cache separation
* remote: cloud cache should extend cloud remote
* remote: move LocalRemote.get
* tests: update func tests
* dependency: update for moved get_checksum
* remote: fix state lookup bug
2020-06-12 15:52:47 +03:00
Peter Rowlands (변기호)
652f5ab439
remote: use separate working tree instance for local cache ( #3991 )
...
* dvcignore: ignore paths outside of CleanTree root
* remote: local remote/cache should always use its own working tree
* repo tree (CleanTree) instance should not be relevant to local cache
operations. When the dvcignore CleanTree is relevant, repo.tree should
be used. For local remote/cache operations, the local remote work_tree
should be used.
* state: tie state to local cache working tree
* tests: update tests
* state: use os.path.exists instead of WorkingTree.exists in some cases
2020-06-12 12:52:38 +03:00
bjonnh and Jonathan Bisson
28a123e727
Remove a remnant umask to solve #4021 ( #4022 )
...
Co-authored-by: Jonathan Bisson <[email protected]
2020-06-11 19:47:25 +03:00
Ruslan Kuprieiev
566472f088
dvc: bump to 1.0.0a11
2020-06-11 19:09:51 +03:00
Saugat Pachhai and Ruslan Kuprieiev
2e12a31c5a
tests: cover run cache with functional tests ( #4018 )
...
* tests: cover run cache with functional tests
* run-cache: use _log_exceptions wrapper
Co-authored-by: Ruslan Kuprieiev <[email protected]
2020-06-11 18:44:21 +03:00
Alexander Schepanovski
d7788887f0
plots: fix a couple of bugs ( #4017 )
...
- running .plots.collect() not from repo root used to skip plot files
- make Plots.render() be more convenient to be called from outside
2020-06-11 18:20:12 +07:00
Ruslan Kuprieiev
87276faef2
RepoTree: work with dirty repos ( #4005 )
...
Currently `RepoTree.open` will throw FileNotFoundError if you have a
file that is tracked by dvc, but which has no md5 or a cache for which
is missing, even though you have the file itself in your workspace.
This corresponds very nicely to the `--no-commit` or `--no-exec`
scenario, where you might not want to commit your artifacts to cache
just yet.
Fixes #3974
2020-06-11 09:50:27 +03:00
Saugat Pachhai
4042d5232d
run: use -f/--force to overwrite, fix msg ( #4011 )
...
Also, makes fixes for `--overwrite` rename to `--force`
and also remove `remove_with_prompt` which was only being
used with `force\=True` anyway.
Fix #4003
2020-06-10 20:17:27 +03:00
Saugat Pachhai
fda14c82d3
plots modify: do not convert output types to plot on unset ( #4009 )
...
Fix #4006
2020-06-10 15:31:17 +03:00
Saugat Pachhai
64be262d18
dvc.yaml: preserve meta ( #4008 )
2020-06-10 14:37:21 +03:00
Saugat Pachhai
1d6c80001d
plots modify: complain if template does not exist ( #3995 )
2020-06-10 16:12:56 +05:45
Ruslan Kuprieiev
91a7194e4f
add/run/import-url: don't use -f as --file ( #4004 )
...
If anything, `-f` is closer to `--force` than to `--file`. So let's set
this straight.
2020-06-10 05:04:55 +03:00
Ruslan Kuprieiev
afe539aaba
plots: rename --no-csv-header to --no-header ( #4001 )
...
* plots: rename --no-csv-header to --no-header
Fixes #3996
* fix typo
2020-06-10 03:34:25 +03:00
Ruslan Kuprieiev
f259bc52cd
config/remote/cache: make --system/--global/--local mutually exclusive ( #4000 )
2020-06-10 02:32:18 +03:00
Ruslan Kuprieiev
db69def834
status: introduce --show-json ( #3998 )
...
Fixes #1406
2020-06-10 01:51:30 +03:00
Casper da Costa-Luis
5efc3bbee2
snap: build fixes ( #3990 )
...
* snap: drop sudo
Fixes #3989
* snap: ci: fix builds
* snap: ci: developer notifications
* snap: minimise size
* debug commit
* Revert "snap: minimise size"
This reverts commit 77b41a96f68b2c8df833803d0a644172b7db4779.
* fix image info
* Revert "debug commit"
This reverts commit 232baf0ad78335b0f3806456a3f20b82cdf0f24c.
* snap: v1 channels
Related: #3872
2020-06-10 00:00:15 +03:00
Alexander Schepanovski
7230d87137
Refactor plots ( #3994 )
...
- separate data collection and rendering
- provide a way to use custom template loader
- drop hard-coding datafiles in template feature
2020-06-09 22:47:17 +03:00
Paweł Redzyński
56833387ce
plots: find metric data in yaml dictionary ( #3992 )
2020-06-09 16:42:28 +03:00
Saugat Pachhai
593e67318b
do not consider stage cached if checksum/cmd has changed ( #3993 )
2020-06-09 18:46:49 +05:45
Saugat Pachhai
afc183f4e7
tests serializing of stage to lockfile ( #3988 )
...
* tests: serialize stage for lockfile
* config: make isort/black compatible
* convert class based tests to pytest
2020-06-09 15:48:17 +03:00
Saugat Pachhai
d201fad1e3
tests: serialize pipeline file ( #3985 )
...
serialize: make params ordered
2020-06-09 14:22:55 +03:00
Ruslan Kuprieiev
5be12b5aac
plots: fix help message ( #3983 )
...
Per https://github.com/iterative/dvc/pull/3911#discussion_r436238442
2020-06-09 14:15:15 +03:00
Saugat Pachhai
c1e5b10cbd
remove unnecessary dependencies ( #3987 )
...
Fix #3971
2020-06-09 12:43:34 +03:00
Saugat Pachhai
b47b2c5673
tests: update mock-ssh-server ( #3986 )
2020-06-09 12:39:10 +03:00
Ruslan Kuprieiev
939f78948a
remote: tree: don't ignore exceptions ( #3977 )
...
Moving ignore logic to the remote itself, so that tree could behave
properly.
2020-06-09 11:25:13 +03:00
Saugat Pachhai
215ea4944c
log: show modifying if entry already exists in dvcfile ( #3978 )
2020-06-08 18:56:00 +03:00
Saugat Pachhai
4e12e76e19
tests: unit testing for StageLoader for Pipelinefile ( #3976 )
2020-06-08 18:55:16 +03:00
Saugat Pachhai
33092e8614
issue template: do not auto-add labels on new issues ( #3979 )
2020-06-08 19:47:42 +05:45
Saugat Pachhai
ae7b604531
Fix loading of params from stage cache ( #3972 )
2020-06-08 12:43:13 +03:00
Ruslan Kuprieiev
b7c174641e
tests: mark s3 test as flaky
2020-06-08 12:27:25 +03:00
Ruslan Kuprieiev
cb43f9cdda
dvc: bump to 1.0.0a10
2020-06-08 11:56:58 +03:00
Ruslan Kuprieiev
791ce22759
pyinstaller: add hook for google-api-python-client ( #3965 )
...
Fixes #3913
2020-06-08 11:53:12 +03:00
Casper da Costa-Luis
c70adebc75
snap: restrict site-package checks ( #3878 )
...
* snap: restrict site-package checks
Fixes #3514
* add comment
2020-06-07 07:50:15 +03:00
2c9cb87679
Subcommand help ( #3956 )
...
* change DvcParser to print subcommand help
* add tests for subcommand help
* docstring refactoring
* remote: finish separating RemoteTree methods (#3946 )
* remote: move upload()/download() into tree
* remote: move path_info/path_cls into tree
* remote.azure: finish moving methods into tree
* remote.gs: finish moving methods into tree
* remote.hdfs: finish moving methods to tree
* remote.http: finish moving methods into tree
* remote.oss: finish moving methods into tree
* remote.s3: finish moving methods into tree
* remote.ssh: finish moving methods into tree
* remote.gdrive: finish moving methods into tree
* tests: update remote unit tests
* tests: update func tests
* remote: use walk_files() for all remotes
* list_cache_paths() now uses tree.walk_files() for all remotes except
local/ssh
* fix DS warnings
* bugfixes
* Unify metrics behavior with dirs (#3935 )
* run: disable directories for -m/-M flags
* run: reformatted string literals
* run: simplified metrics check
* tests: func: run: dirs as metrics outs
* tests: func: run: reformatted string literals
* tests: use pytest
Co-authored-by: Ruslan Kuprieiev <[email protected] #3949 )
* refactor multistage load for params and outputs
* tests: load params
* tests: output loading from pipeline file
* fix test
* fix typo in name
* split params load
* rename params func s/inject_values/fill_values
* fix tests
* simplify loads_params and output.load_from_pipeline
* address @pared's suggestions for tests
* remote: fix build errors (#3957 )
* remote: fix remaining gdrive build issues (#3959 )
* tests: fix gdrive build
* remote.gdrive: fix walk_files prefix relpath
* change DvcParser to print subcommand help
* add tests for subcommand help
* docstring refactoring
* change tests to pytest-style
* remove gettext
Co-authored-by: Peter Rowlands (변기호) <[email protected] [email protected] [email protected] [email protected]
2020-06-05 23:44:02 +03:00
Peter Rowlands (변기호)
b57077af11
remote: fix remaining gdrive build issues ( #3959 )
...
* tests: fix gdrive build
* remote.gdrive: fix walk_files prefix relpath
2020-06-05 15:36:47 +09:00
Peter Rowlands (변기호)
803bb3eb61
remote: fix build errors ( #3957 )
2020-06-05 05:14:34 +03:00
Saugat Pachhai
99b58d3d05
refactor/test multistage load for params and outputs ( #3949 )
...
* refactor multistage load for params and outputs
* tests: load params
* tests: output loading from pipeline file
* fix test
* fix typo in name
* split params load
* rename params func s/inject_values/fill_values
* fix tests
* simplify loads_params and output.load_from_pipeline
* address @pared's suggestions for tests
2020-06-04 23:35:07 +03:00
nik123 and Ruslan Kuprieiev
5bbaa28cae
Unify metrics behavior with dirs ( #3935 )
...
* run: disable directories for -m/-M flags
* run: reformatted string literals
* run: simplified metrics check
* tests: func: run: dirs as metrics outs
* tests: func: run: reformatted string literals
* tests: use pytest
Co-authored-by: Ruslan Kuprieiev <[email protected]
2020-06-04 23:32:00 +03:00
Peter Rowlands (변기호)
74bd8137ba
remote: finish separating RemoteTree methods ( #3946 )
...
* remote: move upload()/download() into tree
* remote: move path_info/path_cls into tree
* remote.azure: finish moving methods into tree
* remote.gs: finish moving methods into tree
* remote.hdfs: finish moving methods to tree
* remote.http: finish moving methods into tree
* remote.oss: finish moving methods into tree
* remote.s3: finish moving methods into tree
* remote.ssh: finish moving methods into tree
* remote.gdrive: finish moving methods into tree
* tests: update remote unit tests
* tests: update func tests
* remote: use walk_files() for all remotes
* list_cache_paths() now uses tree.walk_files() for all remotes except
local/ssh
* fix DS warnings
* bugfixes
2020-06-04 23:22:31 +03:00
Ruslan Kuprieiev
72a576cefe
dvc: bump to 1.0.0a9
2020-06-04 06:06:25 +03:00
Casper da Costa-Luis
Ruslan Kuprieiev
Saugat Pachhai
Paweł Redzyński
Peter Rowlands (변기호)
bjonnhandJonathan Bisson
Alexander Schepanovski
