Commit Graph
45 Commits
Author SHA1 Message Date
Casper da Costa-Luis 9ce3b0c50c add: output improvements (#2546)
* some doc improvements

* add: intermediate progress

* update after rebase

* linting

* fix checksum progress cleanup by ensuring gc

* progress-aware logger.info

* re-fix checksum pbar closure

* multi-file add

* make stage.dump() less verbose.

TODO: check all instances of stage.dump and possibly make verbose

* decrease verbosity of checksum calculations

* leave dvc.add stats

* update/fix logger tests

* minor doc reword

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* tidy API target_;list => targets

- as per https://github.com/iterative/dvc/pull/2546#discussion_r337042535

* tidy RecursiveAddingWhileUsingFilename message duplication

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337044670

* minor logging uncomment

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337051097

* minor doc improvements

- https://github.com/iterative/dvc/pull/2546#discussion_r337116255
- https://github.com/iterative/dvc/pull/2546#discussion_r337116809

* tidy exception

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337434142

* minor doc update

- fixes https://github.com/iterative/dvc/pull/2546/files#r337294491
- related to https://github.com/iterative/dvc.org/issues/719

* Adding => add

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337037137

* various review fixes

- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r337927837
- revert warning style fix
  https://github.com/iterative/dvc/pull/2546#discussion_r338046035
- Abstract error consistency
  https://github.com/iterative/dvc/pull/2546#discussion_r337931070
- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r338254958

* remove diabling progress option

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r338314050
2019-10-25 01:09:19 +03:00
Ruslan Kuprieiev 9d6ca1ef6c dvc: move repo lock from CLI to API
For the longest time we've had `repo.lock`, but were only using it from
CLI when running specific command, instead of simply properly taking it
on API operations. This patch takes care of that.
2019-09-21 05:10:06 +03:00
Ruslan Kuprieiev 0d330c1b9b dvc: move Updater out of Repo()
Fixes #2039

Required by #2012 to be able to fetch data through the API without
interuption from updater.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-14 00:54:36 +03:00
Ruslan Kuprieiev cc2f28c12b cmd: use CmdBaseNoRepo for commands without repo
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-05-11 17:09:19 +03:00
Mr. Outis 85cd344152 logger: Use a more _pythonic_ logging
- Uses a formatter for coloring output
- Move the `progress_aware` decorator logic to the logger itself
- Refactor modules so each one uses their own instance of a logger

Close #1753
2019-04-06 18:19:32 -06:00
Ivan Shcheklein 4a9188c0b5 Move man doc links to descriptions, fix some help copies (#1805)
* move man doc links to descriptions, fix some help copies

* minor init force message fix

* deal with ugly inheritance in lock base

* remove unnecessary super calls

* add return back to python lock/unlock run functions
2019-03-29 03:56:41 +02:00
Ruslan Kuprieiev ba8d83e9b1 dvc: refactor dvc project class
1) split Project's methods into separate files;
2) rename Project into Repo(similar to gitpython's Repo)

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-13 03:31:23 +02:00
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-30 03:48:36 +02:00
Mr. Outis 0d46921ec7 py2: add unicode literals
Fix #1512
2019-01-27 20:40:21 -06:00
Ruslan Kuprieiev 6fec8b16cd RFC: conform to pylint
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-16 14:12:19 +02:00
Mr. Outis 4c420d5f4d logger [rfc]: Use a module instead of a class
- Remove deprecation warnings
- Format error output and remove duplicated info

Close #1086
2018-12-23 21:06:39 -06:00
Ruslan Kuprieiev e02a33f934 logger: shortcut logger in quiet and normal modes
In scenarios with millions of files, late shorcutting of
debug msgs results in massive overhead. In order to avoid
that, just shortcut logger messages ASAP if verbose mode
is not enabled.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-11 12:07:28 +02:00
Ruslan Kuprieiev f04ae2f74f dvc: optimize imports
Fixes #903

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-02 04:54:41 +02:00
Ruslan Kuprieiev 22c6ad5ba3 analytics: send user events
Fixes #693

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-01 06:02:28 +02:00
Mr. Outis 7ae97009fa cli: Move mutual exclusion of --quiet and --verbose to the parser level (#1271)
Close #1270
2018-10-28 22:14:59 -06:00
Ruslan Kuprieiev 0e9774a3f8 RFC: follow flake8 rules
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-25 17:28:58 +03:00
hfchong f7a2377779 Fix _find_root in base.py 2018-07-23 17:33:06 +08:00
Ruslan Kuprieiev 2cab8e03be command: remove unused code
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-07-19 07:00:39 +03:00
Dmitry 1349589ec7 Move CmdBase class to common dir 2017-11-19 03:12:37 -08:00
Dmitry 847a5e2716 Add branch and new-branch options 2017-11-19 02:41:59 -08:00
Ruslan Kuprieiev 17f0c0de10 RFC: find and fix errors and some warning reported by pylint
Fixes #219

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2017-09-17 19:25:03 +03:00
Ruslan Kuprieiev db43e6bc91 dvc: add pull/push commands
We already have 'sync' command, but users report it to be confusing
when both local and remote files exist, so lets introduce two direct
commands to implicitly push and pull from cloud.

Fixes #157.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2017-08-12 22:44:10 +03:00
Ruslan Kuprieiev ee34fdfa31 RFC: move cmdline parsing to cli.py and settings RFC
This will make the structure of dvc more linear
making it easier to read. Plus, these changes
are needed in preparation for libdvc, as they
move all cmdline parsing to one common place.

This patch also refactors our settings system
also make it more linear and remove abstraction
levels that are not needed and only cause confusion.
There is a lot more to be done in preparation for
dvc lib and RFC overall, but this a first step
in that long journey.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2017-07-31 01:39:32 +03:00
Dmitry Petrov 4bb569c16d Refactoring 2017-07-25 03:27:49 -07:00
Ruslan Kuprieiev 76db9db5c2 dvc: introduce --jobs option and use map_progress
It replaces POOL_SIZE constant and by default equals to the
number of cpus available.

map_progress is a helper for 'sync' and 'import' commands to
run in a multi-threaded mode.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2017-06-25 04:04:19 +03:00
Dmitry Petrov f2bac9aac1 GC command 2017-06-14 23:48:58 -07:00
Ruslan Kuprieiev a01ef3197c dvc: misc changes for DataCloud, 'data sync' and 'remove'
This patch introduces DataCloud class that used config to
determine proper cloud class(DataCloudAWS and DataCloudGCP
are awailable) to use in "data_sync" and "remove" commands.

This patch focuses on introducing the common class itself
saving original code and definetly needs some RFC later.

It also adds simple DataCloudLOCAL class that uses directory
as a StoragePath and cp as a sync method. Which makes testing
so much easier and covers local scenarios.

And it also introduces ability to run data sync in parallel
by collecting files first and then running sync in a number
of threads.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2017-05-28 22:28:58 +03:00
Ruslan Kuprieiev 57a30764b7 dvc: add -v|--verbose and -q|-quiet
These are cmdline shortcuts to change dvc's verbosity
without fiddling around with loglevel in dvc.conf.
Both are added and handled as a common options for
all dvc commands in base cmd class.
2017-05-22 04:14:37 +03:00
Dmitry Petrov 43e80bec4a Extract DvcLock logic to a separate class 2017-05-15 19:44:56 -07:00
Dmitry Petrov f6ded2fa72 Lock cmd: small changes 2017-04-29 18:01:24 -07:00
Dmitry Petrov 223380d299 New command: lock 2017-04-29 15:12:31 -07:00
Dmitry Petrov 4697473cf4 Merge branch 'master' of https://github.com/dataversioncontrol/dvc 2017-04-24 19:22:05 -07:00
Dmitry Petrov 7c9a52e879 Repro fixes. Close #43. Close #42 2017-04-24 19:21:13 -07:00
Earl Hathaway a70746c594 most commands dont set locker yet 2017-04-24 15:15:38 -07:00
Dmitry Petrov b5dd8c1d71 Fix repro command. Close #31 2017-04-19 03:07:04 -07:00
Dmitry Petrov 378b56a303 Add --no-locker param 2017-04-18 09:08:40 -07:00
Dmitry Petrov 3e5c7611aa Force reproduction 2017-04-18 09:01:54 -07:00
Earl Hathaway 6fa96f5406 more docs;
remove debugging
2017-04-16 22:18:57 -07:00
Earl Hathaway f84cbd0d17 google cloud installs ALL THE THINGS
fixes to single file commands
got data import to work
get path_factory from settings
cloud selector
data sync -> s3 works again
blank dvc.conf
2017-04-16 19:51:58 -07:00
Dmitry Petrov 4578b6aeb4 Windows symlink: extract all system depended code to System static class 2017-04-10 21:04:40 -07:00
Dmitry Petrov 5f3c53603f Refactoring: extract Settings from BaseCmd; move path_factory to Settings 2017-04-05 23:08:40 -07:00
Dmitry Petrov c3c2116220 Close #1 - repro cmd and close #2 repro unit tese 2017-04-05 20:41:35 -07:00
Dmitry Petrov 84218deccb Finalize ReproCodeDependencyTest 2017-04-03 15:02:49 -07:00
Dmitry Petrov dccc15c0d6 Repro cmd first unttest 2017-04-03 03:56:08 -07:00
Dmitry Petrov a536424141 Extract commands to a separate module: remove cmd_ file prefix 2017-03-25 21:40:58 -07:00