Commit Graph
71 Commits
Author SHA1 Message Date
Saugat Pachhai c4bc4c83e8 upgrade to isort5 (#4399)
* Imports are no longer moved to top.
* `seed-isort-config` is no longer needed.
* Has a `black`-compatible profile/config.
* Also `isorts` the imports anywhere in the code
  (not just at the top).

Also bumps `restyled-isort` to newest version
2020-08-17 19:59:40 +05:45
Saugat Pachhai 1c0f5cdf4b pylint: enable additional checks (#4122)
Created a separate pylint plugin to disable some checks for tests.
2020-06-29 19:15:05 +03:00
Saugat Pachhai b81ace54df Enable pylint checks (#4092)
* pylint: enable checks for imports

* ci: add github action for pylint

* enable pylint on pre-commit

* Install requirements

* use local pylint installation rather than of pre-commit's

* fix import errors

* more checks

* Silence unused imports

* Make a copy of dict
2020-06-24 18:02:39 +03:00
Saugat Pachhai b8aa8ad951 pyupgrade: 3.6+ (#3805)
* pyupgrade: 3.6+

* tests: pyupgrade 3.6+
2020-05-15 16:48:02 +03:00
Peter Rowlands (변기호) 833e82a7ea Drop Python 3.5 support (#3792)
* drop python 3.5 support

* dvc: remove fspath() use inside os.path calls

* dvc: replace dvc.compat.fspath with os.fspath

* tests: replace dvc.compat.fspath with os.fspath

* utils: relpath() should use fspath/str

* utils: os.readlink doesn't support path-like obj on Windows until 3.8
2020-05-14 01:46:42 +03:00
charlesbaynhamandCharles Baynham f4bec650ed Only import speedcopy on Win > 7 (#3581)
* system: Only import speedcopy for Windows > 7

Fixes #3259

* Allow for Win 8

Co-authored-by: Charles Baynham <[email protected]>
2020-04-02 17:13:46 +03:00
Saugat PachhaiandCasper da Costa-Luis 109ded4068 dvc: support Python3.8 (#3463)
* py38: make wheels

- ci: set ARROW_LIBHDFS_DIR to load native library
- deps: pin pyarrow to 0.15.1 for py<=3.7
- pin deps for python<3.8
- pin deps for python==3.8 as well
- add windows Py38 on travis
- do not use speedcopy for Python3.8 and Windows

* use realpath to resolve symbolic links

* do not build pkgs for Windows and Linux on Python3.8

* reorder

* pyarrow: document regression fix

Co-authored-by: Casper da Costa-Luis <[email protected]>
2020-03-16 19:18:33 +00:00
rxxg d7d5d8fcca Remove Linux pyfastcopy (#3412)
Fixes #3395
2020-02-26 22:35:48 +02:00
rxxg 083a630b44 Fallback to standard Python code in case of missing packages 2020-01-16 17:08:07 +01:00
rxxg 674541f922 Use pyfastcopy for non-Windows systems using Python < 3.8 2020-01-14 09:32:09 +01:00
rxxg 09223442f5 Use speedcopy module for faster copies across network boundaries 2020-01-14 09:32:09 +01:00
Mr. Outis df97a2a7de don't reimport DvcException 2020-01-10 18:32:34 -06:00
Mr. Outis b8dd4225d2 py3: fspath source and link_name 2020-01-10 17:40:59 -06:00
Mr. Outis 6eac40d226 system: catch OSError instead of Exception 2020-01-10 17:35:54 -06:00
Mr. Outis e5bee243a3 py3: use links from os module 2020-01-08 14:09:22 -06:00
Mr. Outis 5611ccb43a exceptions: remove cause 2020-01-02 20:43:24 -06:00
Mr. Outis 086acf6404 py3: dvc/utils/compat.py -> dvc/compat.py 2019-12-31 23:42:07 -06:00
Mr. Outis 27f581cf4e py3: remove imports from utils.compat 2019-12-31 23:16:11 -06:00
Mr. Outis 6d36b85736 py3: remove string compat related code 2019-12-30 23:49:19 -06:00
Mr. Outis 94a0914bba py3: get rid of future imports 2019-12-30 19:35:48 -06:00
Tymoteusz Jankowski c17cf2b2f7 dvc/dagascii: Use less instead of AsciiCanvas._do_draw
Uses less CLI tool to paging the output in the interactive mode while
doing e.g.  `dvc pipeline show ..`.

Fixes #2807
2019-12-01 13:23:09 +01:00
Ruslan Kuprieiev 21b7d1f7ec system: use logging instead of dvc.logger
A leftover from an old logger refactoring.
2019-11-06 23:24:46 +02:00
Ruslan Kuprieiev ee70822bd4 Merge pull request #2736 from efiop/master
dvc: automatically reorder imports
2019-11-05 03:02:32 +02:00
Ruslan Kuprieiev 95fe1578bb dvc: automatically reorder imports 2019-11-05 02:34:40 +02:00
Ruslan Kuprieiev b718b839bf system: win: use win32file for _getdirinfo
Current implementation is triggering some non-obvious gc bug, that
causes memory leaks. Using win32file simplifies the code and gets rid of
this bug.

To reproduce it, run:

```
from dvc.system import System

while True:
    System.inode(__file__)
```

and see your Windows machine run out of memory pretty quickly.
2019-11-05 01:46:21 +02:00
Nabanita Dash a0cac4a3c8 changed order of imports (#2636) 2019-10-19 01:14:24 +03:00
Mr. Outis 6eb26de856 fix tests 2019-09-06 11:32:31 -05:00
Ruslan Kuprieiev 16e162941e system: handle PurePath
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/efiop/git/dvc/dvc/remote/local/__init__.py", line 280, in upload
    no_progress_bar=no_progress_bar,
  File "/Users/efiop/git/dvc/dvc/utils/__init__.py", line 129, in copyfile
    System.reflink(src, dest)
  File "/Users/efiop/git/dvc/dvc/system.py", line 125, in reflink
    ret = System._reflink_darwin(source, link_name)
  File "/Users/efiop/git/dvc/dvc/system.py", line 89, in _reflink_darwin
    ctypes.c_char_p(src.encode("utf-8")),
AttributeError: 'PosixPathInfo' object has no attribute 'encode'
------------------------------------------------------------

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-06-13 11:51:14 +03:00
prd 1cd883822f system: use windows constants 2019-05-14 17:03:41 -06:00
Alexander Schepanovski 7478e7d4b0 test: clean up test dirs (#1904)
* test: clean up test dirs

Closes #1770. Partially does #1888.

* dvc: close anything opened

* test: finalize git repo explicitly to remove test dir later

This was previously an issue on Windows.
2019-04-25 14:55:39 +03:00
Ruslan Kuprieiev 5109d759f1 system: reflink: workaround SIP error
Fixes #1515

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-03-30 20:53:06 +02:00
pared f8ef28c4c6 stage: check if local path contains symlink and, if so, treat added files as relpath 2019-02-28 15:00:05 +01:00
Mr. Outis 2be7232d1a py2: Add unicode support for open and os.walk 2019-02-07 23:33:30 -06:00
Ruslan Kuprieiev 4896f08ddf dvc: comply with both flake8 and black
flake8 provides some additional features like unused vars check, etc, so
it makes total sense to use it as well.

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-02-04 00:10:36 +02:00
Ruslan Kuprieiev 4069e34130 dvc: use black instead of flake8
Fixes #1546

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2019-01-30 03:48:36 +02:00
Mr. Outis 0d46921ec7 py2: add unicode literals
Fix #1512
2019-01-27 20:40:21 -06:00
Mr. Outis 4c420d5f4d logger [rfc]: Use a module instead of a class
- Remove deprecation warnings
- Format error output and remove duplicated info

Close #1086
2018-12-23 21:06:39 -06:00
Ruslan Kuprieiev ee16d2f1d7 system: fix typo
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-04 12:00:01 +02:00
Ruslan Kuprieiev 2da24b38a7 windows: don't follow symlink
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-04 11:49:58 +02:00
Ruslan Kuprieiev 2d9ec1005b unprotect: only process symlinks and hardlinks
Fixes #1373

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-12-04 10:57:53 +02:00
Ruslan Kuprieiev 532c9df053 dagascii: don't use unreleased wait_for_input method
Fixes #1359

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-11-23 04:38:26 +02:00
Ruslan Kuprieiev 7e1b584479 system: add a better message if symlink failed 2018-11-07 21:21:45 +02:00
Ruslan Kuprieiev b415ae9a21 system: catch link exceptions
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-22 22:44:33 +03:00
Ruslan Kuprieiev 03eac5b0e5 link: properly catch error on windows
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-21 03:43:19 +03:00
Ruslan Kuprieiev e7c920d783 system: add workaround for buggy os.lstat().st_ino
Fixes #1241

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-19 22:56:53 +03:00
Ruslan Kuprieiev 53652399b3 system: ensure inode value is correct
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-16 05:12:16 +03:00
Amjad Saadeh ecb4f0b998 hdfs output files: fix import error when using hdfs output files on windows
When output files are located on hdfs an import error has been raised.

Fixes #1215
2018-10-12 14:43:08 +02:00
Ruslan Kuprieiev 4b525fe1fc system: fix typo in _reflink_darwin 2018-10-11 16:05:27 +03:00
Ruslan Kuprieiev a6ea2f3db9 dvc: stop using ntfsutils module
Fixes #1162

Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-10-07 17:49:29 +03:00
Ruslan Kuprieiev 76c2f812f0 osx: reflink: check if clib has clonefile()
Signed-off-by: Ruslan Kuprieiev <[email protected]>
2018-09-23 14:39:09 +03:00