Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-12 07:18:23 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,535 Commits 7 Branches 579 Tags
8ed9b911db73bb034b240d24233eadeef2a59bf7
Commit Graph
107 Commits
Author SHA1 Message Date
skshetry 1de37556d5 bump ruff to v0.0.247 (#9050) 
* bump ruff to v0.0.247

* ruff: enable pep8-naming check

rename some camelCased APIs, especially in logger
rename G variables to graph
ignore some variable namings and help messages in dvc.commands
 2023-02-18 16:50:35 +05:45
skshetry ddeb32ed33 ui: log footer support message to standard error (#9034) 2023-02-16 11:31:31 +05:45
skshetry 00e61e18ae logging: do not disable other loggers (#8995) 
Closes https://github.com/iterative/dvc/issues/8631
 2023-02-09 15:04:29 +05:45
skshetry 5490fc00d1 logger: check isatty stream to print colored logs or not instead of depending on colorama (#8993 
logger: check isatty stream to print colored logs or not
 2023-02-09 13:50:38 +05:45
skshetry f1043676da [WIP] logging: change from dictConfig to equivalent code for creating logge… (#8985 
logging: change from dictConfig to equivalent code for creating loggers/handlers/formatters

dictConfig clears all handlers, and potentially disables existing loggers (we disabled this functionality
though). There's much more code than necessary in dictConfig than what we need.

I have tried to keep the setup equivalent to what we had in the config, so I have not tried to cleanup
the code.
 2023-02-09 09:34:14 +05:45
skshetry ae80889477 logging: defer calling colorama.init() in jupyter, otherwise initialize immediately (#8969)Closes https://github.com/iterative/dvc/issues/8957 
logging: init colorama before logging if outside jupyter env

Otherwise, tty detection does not work as `colorama.init()` mutates
sys.stdout/sys.stderr, and after `dictConfig()`, the handlers will hold
on to the stream.
 2023-02-07 10:05:05 +05:45
skshetry a86470ccbc update black and change line-length to 88 (#8953) 2023-02-04 05:28:07 +02:00
skshetry 8208846003 bump ruff to 0.0.238 (#8915) 2023-01-31 11:24:03 +05:45
skshetry a137b2a6bc update ruff and enable flake8-logging-format and flake8-type-checking (#8894) 
* update ruff and enable flake8-logging-format and flake8-type-checking check

* Update dvc/external_repo.py

* Update dvc/fs/dvc.py

* Update dvc/repo/experiments/__init__.py

* Update dvc/repo/experiments/queue/base.py

* Update pyproject.toml
 2023-01-27 19:14:19 +05:45
skshetry 309ae4d995 mypy: enable check-untyped-defs (#8815) 2023-01-15 21:40:40 +05:45
Peter Rowlands 7393ebdcb4 logger: init logging config before colorama 2022-10-04 21:27:24 +03:00
Saugat Pachhai (सौगात) fd7dcae8aa cli/log: handle SIGPIPE errors 
`dvc <command> | head` for example could fail with SIGPIPE and
would print a traceback with BrokenPipeError exception. Now we
redirect do /dev/null in case of error as suggested in Python's
docs: https://docs.python.org/3/library/signal.html\#note-on-sigpipe.
 2022-07-25 16:38:09 +05:45
Saugat Pachhai (सौगात) e34767ca90 logging: show logs from dvc-data 2022-05-20 15:13:50 +05:45
Saugat Pachhai (सौगात) 572b6dfca9 logging: show logs from dvc-objects 2022-05-20 14:02:08 +05:45
Saugat Pachhai 6a417885b8 exceptional exceptions (#6654) 
* exceptional exceptions

* backward compatibility with previous exceptions

* make linters happy :)
 2021-09-21 11:27:57 +05:45
Christian Clauss d8b67f6f78 Upgrade to modern Python syntax (#6296) 2021-07-19 01:30:37 +03:00
Saugat Pachhai a172067c7d fix errors raised by pylint (#6081) 
astroid==2.5.7 has improved inference and was throwing
errors that were false negatives before. Fixing these
and ignoring some.

Also added `funcy.decorator` to the `signature-mutators`,
as it changes function signature (i.e. provides `Call` as
a first argument which is not required to pass).

Example errors that we were getting because of this:

```
tests/remotes/gdrive.py:40:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
************* Module dvc.stage.serialize
dvc/stage/serialize.py:38:1: E1120: No value for argument 'func' in
function call (no-value-for-parameter)
************* Module dvc.fs.http
dvc/fs/http.py:19:1: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
************* Module dvc.fs.gdrive
dvc/fs/gdrive.py:66:11: E1120: No value for argument 'tries' in function call (no-value-for-parameter)
dvc/fs/gdrive.py:459:5: E1120: No value for argument 'ctx' in function call (no-value-for-parameter)
```
 2021-05-31 05:46:18 +00:00
Saugat Pachhai b9000ed926 pre-commit: update black to 21.5b1 (#6032) 
* pre-commit: update black to 21.5b1

also updated others, but black is more noticeable

* remove magic trailing comma
 2021-05-18 15:51:15 +00:00
Batuhan Taskaya a234f53b76 azure: use a custom event loop for authentication (#5958) 
* azure: use a custom event loop for authentication

* add test_temp_event_loop

* wrap open() too!

* ignore aiohttp warnings for normal level use-cases

* rename tests/unit/remote/test_azure to tests/unit/fs/test_azure
 2021-05-05 15:41:55 +03:00
Batuhan Taskaya d4a3df73df s3: fsspec migration (#5683) 
* s3: migrate to s3fs

* s3: strip etag

* s3: disable instance cache

* tests: use the proper remote:// form in test_checkout

* tests: move/drop some tests

* setup: depend on s3fs

* tests: normalize s3 tests with some cleanups

* Use _download/_upload with old implementation

* fsspec: cache .exists() results

* setup: pin the boto version

* setup: install s3fs[boto3]

* s3: proper exception management

* s3: config improvements

* setup: reference without quotes

* setup: install boto separately

* setup: install moto server

* s3: mocking adjustments for moto_server

* s3: mark some fixtures with scope=session

* s3: s3_fake_creds_file scopes to func

* fsspec: add some docs regarding exists() optimization

* fsspec: exists() fallback

* fsspec: revert exists() optimization

* tests: ssl_verify => verify

* setup: temporarily fetch from master

* s3: add boto as a requirement

* tests: do an extra check regarding race conditions

* tests: use motoserver docker image

* tests: wrap config deletion with try/finally

* tests: disable s3 by default

* setup: upgrade fsspec
 2021-05-05 11:36:10 +03:00
Alex Ford b3bfa346c3 Using existing logging level if defined in addLoggingLevel (#5632) 
* Use existing level if defined in addLoggingLevel

Fixes https://github.com/iterative/dvc/issues/4546

* Add addLoggingLevel test

* Format fix.
 2021-03-22 15:15:16 +02:00
Saugat Pachhai 1c0f5cdf4b pylint: enable additional checks (#4122) 
Created a separate pylint plugin to disable some checks for tests.
 2020-06-29 19:15:05 +03:00
Saugat Pachhai ad306d7bf3 Enable most of the pylint checks (#4111) 2020-06-25 18:10:14 +03:00
Casper da Costa-Luis 184028f176 logger: add trace, count --verbose/--quiet (#3806) 
* logger: initial verbosity/quiet counts

Fixes #2332

* actually use TRACE

* logger: properly patch in trace

* FATAL -> CRITICAL

* only change log level if not info

Makes testing easier

* tests: logger: fix and update

* state: logger: debug => trace

* logger: filter duplicate debug messages
 2020-05-20 18:23:31 +03:00
Casper da Costa-Luis 2bb7257a06 lint: isort pre-commit & misc improvements (#3749) 
* test: clean flake8 config root clutter

* test: pre-commit: add isort

Compatible with black (see https://github.com/timothycrosley/isort/issues/694)

* lint: reorder imports

* lint: more complex imports

* lint: fix deepsource

* config: isort: add first party dvc

* tests: restyled: add isort

* temp commit to revert before merge

* Revert "temp commit to revert before merge"

This reverts commit 11db0fd8e95fe891597cbb9bf804480909b19e24.
 2020-05-06 23:56:07 +03:00
Ivan ShchekleinandRuslan Kuprieiev 7b1dbd6109 CLI: redirect warnings to logger (#3591) 
* redirect warnings module messages to logger to silent them in gdrive

* minor cleanup: remove unused var assignment

* dvc: don't ignore warnings manually

We use logger.captureWarnings() instead for CLI and for API this is harmful.

Co-authored-by: Ruslan Kuprieiev <[email protected]>
 2020-04-04 22:18:14 +03:00
Ruslan Kuprieiev 2ffebb0b5e logger: use standard lazy formatting (#3487) 
Unfortunately overriding `format` is not enough in order to use our own
custom formatting in logger. `Record` uses percent formatting that is
hardcoded inside, so we would also have to override it, which is just
way too much effort that will inevitably break as we discover other
methods that are not ready for non-% formatting. So we are just giving
up and sticking to default %-formatting for lazy log messages.

Fixes #3397
 2020-03-17 00:38:00 +02:00
ttekampeandGuro Bokum de453ab06d dont disable existing loggers when dvc is set up (#3345) 
* dont disable existing loggers when dvc is set up

* corrected code style

* dont disable existing loggers, fixes

Co-authored-by: Guro Bokum <[email protected]>
 2020-02-27 07:02:06 +02:00
Mr. Outis 8492786ae4 logger: refactor, remove dead code (#3306) 
* Remove dead code
* Rename _walk_exc to _causes, simplify it
* Split up stack trace and remove extra \n
* Import traceback only when needed
 2020-02-17 03:19:37 +02:00
Saugat Pachhai 17c7b72cb0 logger: log datetime when on debug mode (#3325) 2020-02-14 13:58:15 +02:00
Ruslan Kuprieiev 2786a4e59b logger: allow exc_info in all record types (#3290) 
This better reflects the original logger formatter.

This also fixes annoying redundant newline between traceback and "need
any help" footer.
 2020-02-09 22:22:20 +02:00
Jorge Orpinel ec0c39a28e term: review capitalization and backquotes around "dvc" 
rel: https://github.com/iterative/dvc/pull/3156#discussion_r369897073
 2020-01-22 20:27:01 -06:00
Ruslan Kuprieiev 6f8ac0c1b1 logger: support {} formatting 2020-01-19 11:53:29 +02:00
Mr. Outis f942e26453 py3: remove tb_list 2020-01-07 00:13:51 -06:00
Mr. Outis 7ed4fd3840 py3: use format_exc instead of print_exception 2020-01-06 23:40:00 -06:00
Ruslan Kuprieiev 6411a3bb32 ui: don't show exception for "Too many files" error 
https://github.com/iterative/dvc.org/pull/875#discussion_r362920851

Stops showing the exception in non-verbose mode, but shows full
traceback in verbose mode.

Introducing our custom parameter `tb_only` through standard `extra`
logging parameter.
 2020-01-06 01:02:59 +02:00
Mr. Outis 5611ccb43a exceptions: remove cause 2020-01-02 20:43:24 -06:00
Mr. Outis fd82402d34 py3: use new magic from PEP 3135 2020-01-02 16:24:31 -06:00
Mr. Outis b844aea42c py3: remove cast_bytes method 2019-12-31 20:14:25 -06:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Saugat Pachhai 461a79e1e8 Fix typos 2019-11-26 20:07:20 +05:45
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
n3hrox 2893c9acad move footer to better place and change logging to stderr 2019-11-03 22:07:27 +01:00
n3hrox 3bf810a832 move footer from logger to main 2019-11-01 16:36:31 +01:00
Casper da Costa-Luis 9ce3b0c50c add: output improvements (#2546) 
* some doc improvements

* add: intermediate progress

* update after rebase

* linting

* fix checksum progress cleanup by ensuring gc

* progress-aware logger.info

* re-fix checksum pbar closure

* multi-file add

* make stage.dump() less verbose.

TODO: check all instances of stage.dump and possibly make verbose

* decrease verbosity of checksum calculations

* leave dvc.add stats

* update/fix logger tests

* minor doc reword

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* tidy API target_;list => targets

- as per https://github.com/iterative/dvc/pull/2546#discussion_r337042535

* tidy RecursiveAddingWhileUsingFilename message duplication

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337044670

* minor logging uncomment

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337051097

* minor doc improvements

- https://github.com/iterative/dvc/pull/2546#discussion_r337116255
- https://github.com/iterative/dvc/pull/2546#discussion_r337116809

* tidy exception

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337434142

* minor doc update

- fixes https://github.com/iterative/dvc/pull/2546/files#r337294491
- related to https://github.com/iterative/dvc.org/issues/719

* Adding => add

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r337037137

* various review fixes

- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r337927837
- revert warning style fix
  https://github.com/iterative/dvc/pull/2546#discussion_r338046035
- Abstract error consistency
  https://github.com/iterative/dvc/pull/2546#discussion_r337931070
- minor comment as per
  https://github.com/iterative/dvc/pull/2546#discussion_r338254958

* remove diabling progress option

- fixes https://github.com/iterative/dvc/pull/2546#discussion_r338314050
 2019-10-25 01:09:19 +03:00
Mr. Outis 13a7e0f978 logger: send warning messages to stderr (#2612) 
* logger: send warning messages to stderr

Fix #2561

* logger: filter warnings for stdout
 2019-10-16 03:12:00 +03:00
Casper da Costa-Luis 83f55de7a9 move compatibility helper 2019-09-29 16:43:37 +01:00
Casper da Costa-Luis 4c01a86302 fix py27 compat 2019-09-28 23:12:55 +01:00
Paweł RedzyńskiandRuslan Kuprieiev d16178ab61 checkout: raise error on failed dir cache loading (#2503) 
* remote: base: load_dir_cache: raise exception on failed download

* output: base: raise error on failed get_files_number

* remote: base: log failed download as warning

* Update dvc/output/base.py

Co-Authored-By: Ruslan Kuprieiev <[email protected]>

* remote: base: dont download dir cache on load_dir_cache
 2019-09-19 16:35:37 +03:00