Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
4,534 Commits 7 Branches 579 Tags
b6b07dde8b43c18ecb353e1e187c92e2b4ff0060
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexander Schepanovski b6b07dde8b dvc: make graph consist of stages not their paths 2019-12-30 12:07:52 +07:00
Alexander Schepanovski 322b354dd5 test: fix and migrate version tests 2019-12-27 22:07:46 +07:00
Alexander Schepanovski 5f3e416e60 test: fix test_ignore_external in Python 3.5 2019-12-27 15:51:07 +07:00
Alexander Schepanovski 094482249a ui: fix extra newline in dvc version 
Additionally don't try to detect cache fs type unless cache dir exists.
 2019-12-27 15:45:41 +07:00
Alexander Schepanovski e8c321e9a3 test: test ignore nuances 
- nested files are ignored
- external dirs are not affected
 2019-12-24 12:46:32 +07:00
Alexander Schepanovski 6c0cda8c7b ui: reorder checkout options to go from common to rare 
This is the order we use in docs. It also makes sense.
 2019-12-17 18:40:26 +07:00
Alexander Schepanovski 5d1f01d58f checkout: iron out --relink related messages 2019-12-13 19:00:06 +07:00
Alexander Schepanovski 1143a8c57f test: add checkout parse-cli-api test 2019-12-12 20:16:17 +07:00
Alexander Schepanovski b8cf906edb test: stop erepo_dir from undoing parent chdir 2019-12-11 20:19:38 +07:00
Alexander Schepanovski a1643d576d test: fix run_copy fixture when used with scm 2019-12-11 20:19:02 +07:00
Alexander Schepanovski 9607e5f873 get: refactor, clean up and fix dvc get implementation 
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway

Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
 2019-12-09 21:02:49 +03:00
Alexander Schepanovski 3610961144 dvc: warn to run dvc checkout --relink in cache.type change 2019-12-09 13:46:10 +03:00
Alexander Schepanovski 2f79e362f4 dvc: add checkout --relink 2019-12-09 13:25:32 +03:00
Alexander Schepanovski 3b2666e46b remote: remove unused exception class 2019-12-09 12:45:56 +03:00
Alexander Schepanovski 88370e1b00 dvc: reorganize imports in dvc.remote.base 2019-12-09 12:40:56 +03:00
Alexander SchepanovskiandMr. Outis 1da133b9ed test: update dir_helpers usage 
Co-Authored-By: Mr. Outis <[email protected]>
 2019-12-09 11:34:10 +03:00
Alexander SchepanovskiandMr. Outis a59041aeab test: update dir_helpers usage 
Co-Authored-By: Mr. Outis <[email protected]>
 2019-12-09 11:33:35 +03:00
Alexander SchepanovskiandMr. Outis 0cd6106189 Update tests/dir_helpers.py 
Co-Authored-By: Mr. Outis <[email protected]>
 2019-12-09 11:24:41 +03:00
Alexander Schepanovski 6c12a2e555 test: fix unicode test_ignore issue under Windows Py2 2019-12-07 17:17:18 +01:00
Alexander Schepanovski 79902988fc test: fix files set comparisons in dvcignore tests under windows 2019-12-07 13:17:08 +01:00
Alexander Schepanovski 77fcd75b7d test: use utf-8 in tmp_dir.gen() 2019-12-07 13:16:42 +01:00
Alexander Schepanovski 42096c5bfc test: migrate test_ignore to new helpers 2019-12-07 11:09:52 +01:00
Alexander Schepanovski c6c67ace20 test: create empty dirs via tmp_dir.gen() 2019-12-07 10:32:37 +01:00
Alexander Schepanovski 5bec700761 test: accept Paths in tmp_dir.gen() 2019-12-06 18:36:17 +01:00
Alexander Schepanovski 659eb04705 test: write a usage text for dir_helpers 2019-12-06 18:34:05 +01:00
Alexander SchepanovskiRuslan KuprieievPaweł Redzyński
 		c27988a042 test: minor fixture refactor fixes from @pared and @efiop 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-Authored-By: Paweł Redzyński <[email protected]>
 2019-12-02 23:33:44 +01:00
Alexander Schepanovski 0385810662 test: fix python 2, 3.5 and mac issues with new fixtures 2019-12-02 23:32:11 +01:00
Alexander Schepanovski fcebad821e test: refactor tmp dir helper fixtures 
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
 2019-12-02 23:31:54 +01:00
Alexander Schepanovski d1cf770a3c dvc: don't make an unneeded copy on commit 
Fix #2016 without all the complications of #2358.
 2019-12-02 23:08:45 +01:00
Alexander Schepanovski 30d58e14a0 Revert "remote: base: don't checkout existing file (#2358)" 
This reverts commit 5ec1d274ae.

Save tests to try make them work, and `_get_cache_type()` to be used
later.
 2019-12-02 19:06:35 +01:00
Alexander Schepanovski 4ab4e43606 dvc: return to using .changed() shortcut in .checkout() 2019-12-02 14:38:39 +01:00
Alexander Schepanovski 865ed6f8ea test: skip non supported remotes fast in api tests 2019-11-29 23:42:26 +01:00
Alexander Schepanovski 8af1aeefb4 remote: refactor ssh ask password code 2019-11-28 21:10:35 +01:00
Alexander Schepanovski 73eac3b6cc remote: protect all remote client/session creation code with locks 
With @cached_property last write wins, so all threads will use the same
session. However, several might be created concurrently wasting
resources and even potentially asking something from the user.

This change guarantees only one client instance will be created for a
remote.
 2019-11-28 21:05:20 +01:00
Alexander Schepanovski 8681cc3cd1 perf: optimize cache listing for local, ssh and hdfs 
- make all three lazy
- simplify local one
- use deque instead of list for hdfs
 2019-11-22 14:43:01 +01:00
Alexander Schepanovski 8da0390c57 remote: small .save_info()/.get_checksum() cleanup 2019-11-22 12:58:08 +01:00
Alexander Schepanovski 0c2a475e93 dvc: remove unused config paths 
These are remnants from old style remotes configuration
 2019-11-20 20:33:46 +01:00
Alexander Schepanovski 3bffe9dbff dvc: drop schema from requirements 2019-11-18 17:24:40 +01:00
Alexander Schepanovski 2aa750cb48 perf: switch schema validation library for config 2019-11-18 17:22:05 +01:00
Alexander Schepanovski 3943b0eb57 perf: a small stage validation optimization 2019-11-15 16:31:46 +01:00
Alexander Schepanovski 78fad73f2b perf: switch schema validation library 
voluptuous is compiling schema to python function, which makes it
faster.
 2019-11-14 22:33:28 +01:00
Alexander Schepanovski 329d072c22 perf: explain why two yaml parsers are used 2019-11-13 16:35:05 +01:00
Alexander Schepanovski 58c9e5557c perf: optimize yaml parsing for stages 2019-11-11 13:41:50 +01:00
Alexander Schepanovski b83564d80e perf: refactor URLInfo to not use ParseResult inside 
This permites URLInfo construction from parts without building/reparsing
string, which makes it around 20% faster. Code is also simpler and a bit
shorter now.

Other improvements:
- added URLInfo.replace(path=...)
- URLInfo._path now stringifies properly
- protected against passing path not starting with /
- do not allow query, params, fragments and passwords in urls (they were
silently ignored previously)
 2019-11-01 17:33:00 +01:00
Alexander Schepanovski 77fcd49695 dvc: fix tqdm dep (#2694) 
tqdm==4.35.0 was throwing KeyError: 'percentage'.
 2019-10-30 16:43:18 +02:00
Alexander Schepanovski 86811aac20 imp: dedup and parallelize fetching imported deps (#2654) 
* imp: dedup and parallelize external fetches

This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.

* imp: test that fetch works on non workspace deps
 2019-10-23 18:02:50 +03:00
Alexander Schepanovski 30341438e0 gc: added --all-commits (#2643) 
* dvc: fix branch suffixes in status/push/pull

* gc: add --all-commits flag

* gc: fix gc warning

Say that we always leave things used in the working tree
 2019-10-23 18:00:31 +03:00
Alexander Schepanovski 33b0eefec0 dvc: dedup used cache (#2630) 
* dvc: dedup used cache

This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts

* dvc: delay checksum names concatenation
 2019-10-21 12:12:56 +03:00
Alexander Schepanovski 1f58fc6f42 dvc: dedup brancher (#2639) 2019-10-20 17:32:42 +03:00
Alexander Schepanovski c26233acbb dvc: reset dvcignore 
This depends on filesystem and affects stages collection so must be
reset.
 2019-10-13 19:18:53 +07:00
Alexander Schepanovski 44264857c3 ui/perf: do not clone same repo multiple times (#2583) 
* ui/perf: do not clone same repo multiple times

Closes #2490

External repos are still created in /tmp. They might be reused until
main() end. If no perfect match is found then a repo with the same url
is copied (not cloned again) and adjusted to fit a request: rev and
cache_dir.

NOTE: test_download_error_pulling_imported_stage() was modified to not
rely on clone error.

* test: fix import update test

* test: fix external_repo test under Windows

* dvc: fix non-master default branch for external repos

* dvc: remove unused .unset_dir() method
 2019-10-11 13:56:37 +03:00
Alexander Schepanovski 0203fd4017 perf: import dvc.repo dynamically 2019-09-27 15:08:42 +07:00
Alexander Schepanovski e2eb4ae0c3 perf: delay cache creation 
The older code instantiates non-local cache remotes on Repo creation if
any of those are configured. And remote instantiation loads their API
libs.
 2019-09-24 09:03:03 +07:00
Alexander Schepanovski 4006c6434e dvc: remove dead code in DataCloud 2019-09-23 20:00:19 +07:00
Alexander Schepanovski cb5ff6e085 perf: import some things later 
Saves on startup time
 2019-09-20 16:45:02 +07:00
Alexander Schepanovski ffb52b256b perf: do not import http.server each run 
It is only used in tests.
 2019-09-17 18:41:05 +07:00
Alexander Schepanovski eac0aa13b2 perf: use packaging.version instead of pkg_resources 
This is the same functionality without a cost of loading a huge package.
 2019-09-17 14:32:31 +07:00
Alexander Schepanovski 293a50d75c perf: use fast entry points on non-wheel install 
This avoids importing `pkg_resources` on startup, which saves around 200
ms.

NOTE: wheel installer uses its own entry points template, which is the same
fastentrypoints.py script provides, so no gain there.
 2019-09-12 22:33:59 +07:00
Alexander Schepanovski 3c3378bdca dvc: make lock lifetime long by default 2019-09-11 14:21:12 +07:00
Alexander Schepanovski d9d85f3cb0 dvc: fix locks in Python2.7/Windows 
Return to zc.lockfile lock in Python 2, NFS is not supported there
anyway since sqlite3 doesn't support URIs.
 2019-09-11 14:17:21 +07:00
Alexander Schepanovski f29f07ca88 dvc: do not lock sqlite db 2019-09-11 14:17:16 +07:00
Alexander Schepanovski 6aeccfcc07 dvc: fix long lock filenames under Windows 2019-09-06 16:29:23 +07:00
Alexander Schepanovski 1bdaca0d6c dvc: switch to flufl lock 
This locks employs hardlink trick, which works on NFS and alike unlike
fcntl/open locks.

Closes #1918.
 2019-09-06 16:29:09 +07:00
Alexander Schepanovski b52265a1d2 ui: remove extraneous stage.save() 
stage.update() already does that via .run(). That was causing printing a
confusing message to user.
 2019-08-27 15:04:37 +07:00
Alexander Schepanovski c68b76f843 ui: remove "Reproducing ..." message 
It as always followed by a more specific one, so it's superfluous.
 2019-08-27 15:02:21 +07:00
Alexander Schepanovski edcd1724b4 cache: skip non-checksum files around cache 
Fixes #2394
 2019-08-24 15:04:25 +07:00
Alexander Schepanovski 8984e9fed4 dvc: use wdir == "." on add 
Fixes #2404
 2019-08-24 10:23:41 +07:00
Alexander Schepanovski 85289ad00b api: wrap not found errors 
api.open()/read() will now raise OutputFileMissingError and
Remote*.open() will raise FileNotFoundError if target file is absent.
 2019-08-15 23:20:48 +07:00
Alexander Schepanovski 60a983b63f api: stream api.open() for local, SSH and HDFS 2019-08-13 18:19:52 +07:00
Alexander Schepanovski 641eb57b27 api: stream data in api.open() for S3, GS and OSS 
Employ the same sign url and use resumable download approach tested for
Azure.
 2019-08-13 14:09:51 +07:00
Alexander Schepanovski e5e29065c8 api: make .open() stream data for Azure 
This employs generating temporary signed public url and then streaming
from it. The bulk of the code might be reused for other remotes.

Url streaming does resume on connection error using "Range" header and
regenerates public url too.

If remote class doesn't provide .open() then this falls back to pulling
a file and opening it locally.
 2019-08-09 21:51:43 +07:00
Alexander Schepanovski 50b5b87304 dvc: batch push/pull/fetch/status 
This will speed up cache collection and then run single cloud op for all
collected cache.

Fixes #2308.
 2019-07-30 13:33:01 +03:00
Alexander Schepanovski 59eb929516 ssh: do not ask for same password repeatedly 2019-07-23 16:25:40 +03:00
Alexander Schepanovski 35a8d69c4f dvc: ignore Stage.meta in md5 calculations 
Meta change surely should not trigger a repro run. Closes #2209.
 2019-07-23 16:00:22 +03:00
Alexander Schepanovski f00a325114 dvc: refactor dict_md5() 
- stop using mutable default value
- use list and dict comprehensions to speed it up and make nicer
- employ exhaustive if..elif..else
 2019-07-23 16:00:12 +03:00
Alexander Schepanovski 20f1976e03 remote: fixed Repo.push/pull/fetch/status() to default to 1 job 2019-07-23 12:23:43 +03:00
Alexander Schepanovski f37ab9431b perf: stop on first changed thing in Stage.changed() 2019-07-23 15:29:49 +07:00
Alexander Schepanovski 1f90b426fb remote: prettify get_connection/pool() signatures 2019-07-21 13:41:04 +07:00
Alexander Schepanovski 6afc2a38ba test: close pools on tests cleanup 
This avoids weird errors as pools are collected by gc in broken env.
 2019-07-21 13:40:29 +07:00
Alexander Schepanovski 31f1ded315 remote: don't close sftp connections until parent ssh conn closed 
Since this is multiplexing over the same TCP connection no system
resources are withheld, so this should be a no-brainer optimization.

As a side-effect this should fix "open max - close all - try open -
Administratively prohobited" error @darabi experiencing in #2280.
 2019-07-21 13:38:44 +07:00
Alexander Schepanovski 624ccfc744 remote: honor --jobs for status collection 2019-07-21 12:50:35 +07:00
Alexander Schepanovski 67e50cd71e remote: speed up ssh dir checksum 2019-07-18 22:13:42 +07:00
Alexander Schepanovski ed387d824d remote: do not swallow exceptions by ssh pool 2019-07-17 22:20:43 +07:00
Alexander Schepanovski 6010ce54cd remote: manually close all ssh pools at the end 
Python 2 fails to do it automatically occasionally and shows weird error
messages.

This was harmless but confusing.
 2019-07-17 22:20:43 +07:00
Alexander Schepanovski db19957b8a Download refactor (#2275) 
* remote: drop not working http resume

Closes #2136

* remote: drop @transfer_context()

It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.

* remote: drop upload/download chunking
 2019-07-16 16:27:44 +03:00
Alexander Schepanovski 9766d34cec remote: support older linuxes as ssh remotes (#2276) 
* remote: support older linuxes as ssh remotes

Work around md5sum not having --tag in Red Hat 6.3 and friends.
Fixes #2242.

* remote: optimize consequent ssh md5 calls
 2019-07-16 14:15:28 +03:00
Alexander Schepanovski b5a26e5853 remote: fix collect dir progress (#2270) 
We were resolving futures after ThreadPoolExecutor exited and finished
everything.
 2019-07-15 18:31:19 +03:00
Alexander Schepanovski b9a3c7c038 remote: use ssh connections pool (#2264) 
* remote: use ssh connections pool

* dvc: never use paramiko DEBUG mode

It's ridiculously verbose, makes logs unreadable.
 2019-07-14 23:30:24 +03:00
Alexander Schepanovski 732fd01cc5 test: drop a reimport (#2247) 2019-07-10 03:24:59 +03:00
Alexander Schepanovski 90b1a80240 test: test ssh dir output (#2246) 2019-07-10 03:24:35 +03:00
Alexander Schepanovski 16c16d334b dvc: skip unused checks in repro -f (#2245) 2019-07-10 03:22:07 +03:00
Alexander Schepanovski 14ce9d6d4a dvc: stop trying to fetch from remote when looking for git object (#2241) 
Closes #2213
 2019-07-08 17:57:25 +03:00
Alexander Schepanovski c4836b4578 dvc: optmize PathInfo.isin() 
Should help with #2203
 2019-07-05 13:18:12 +07:00
Alexander Schepanovski 7180f49df3 remote: fix UrlInfo.relative_to() 2019-07-04 22:43:24 +07:00
Alexander Schepanovski 89428db0f4 remote: optimize UrlInfo.isin() 
This will use cached `_path`, which will use cached `_cparts`. Saves
some time skipping repetitive parsing.
 2019-07-04 22:39:05 +07:00
Alexander Schepanovski 4934a3b2fe dvc: fix #2219 2019-07-04 19:51:27 +07:00
Alexander Schepanovski be1d696188 dvc: fix prompt message while collecting used cache 2019-07-03 14:43:31 +07:00
Alexander Schepanovski 835a04a186 remote: fix ssh dir outputs 2019-07-02 18:49:19 +07:00
Alexander Schepanovski 30304c3454 remote: always ignore ENOENT in ssh.rmdir() 2019-07-02 18:49:10 +07:00
Alexander Schepanovski 766b7d8818 dvc: uninstall external repo properly 2019-07-01 19:08:50 +07:00