Stone_Red/dvc
1
0
Fork 0
mirror of https://github.com/Stone-Red-Code/dvc.git synced 2026-11-02 07:18:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,010 Commits 7 Branches 579 Tags
426972e55f155d817e1db975afa2f25dbf860445
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexander Schepanovski 7168a9c50a erepo: pull each time 
Fixes #3062.
 2020-01-23 23:09:57 +07:00
Alexander Schepanovski 5af7a9d850 test: add git_dir 2020-01-21 20:56:04 +07:00
Alexander Schepanovski 91e024c8fa test: do not commit copy.py to git 2020-01-20 20:35:10 +07:00
Alexander Schepanovski 1d891a6e77 test: provide make_tmp_dir fixture 2020-01-20 11:57:54 +07:00
Alexander Schepanovski 4117206c67 test: drop repo_template fixture 2020-01-20 11:49:13 +07:00
Alexander Schepanovski 72812b519e api: fix get_url() non-dvc repo error message 2020-01-19 23:01:53 +07:00
Alexander Schepanovski eb97e0b393 external: do not fail with no remote if cache is available 2020-01-19 21:39:32 +07:00
Alexander Schepanovski ee022eada0 api: adjust get_url requiring dvc repo to the new reality 2020-01-19 21:38:37 +07:00
Alexander Schepanovski 9de0643d97 api: allow rev on local repos 2020-01-19 21:38:30 +07:00
Alexander Schepanovski 033ef88a4e external: refactor external_repo() and its users 
Some things are fixed along the way:
- no cache shared between incompatible funcs
- some incosistent exceptions are now consistent
- import updates for git files now works both for dvc and git repos
- "auto-generated-upstream already exists" fixed
- dvc.api.open()/read() works with git repos now

Fixes #3124, #2976. Closes #3172.
 2020-01-19 21:38:06 +07:00
Alexander Schepanovski be90c38182 summon: nice up summon code 
- proper doc-string for prepare_summon()
- summon -> summon_dict in summon() to not overwrite a glo bal name.
This was not a bug, but a bad practice, which might have caused some
issues in the future.
- better name for _get_object_spec()
 2020-01-13 15:22:16 +07:00
Alexander Schepanovski d0de9d0ed0 summon: revert dvc.api._invoke_mehtod() considerations 
It's now thread-safe, but the rest is still relevant.
 2020-01-13 15:19:54 +07:00
Alexander Schepanovski 4a3dd63720 summon: separate prepare_summon() 2020-01-10 22:02:58 +07:00
Alexander Schepanovski c548b0792d summon: fix open() shadowing 2020-01-10 21:59:58 +07:00
Alexander Schepanovski afd29e5dad summon: protect chdir/import/call with a thread lock 2020-01-10 20:55:28 +07:00
Alexander Schepanovski bafac6dec3 ui: fix grammar in the error message 2020-01-07 17:26:01 +07:00
Alexander Schepanovski 1b1f8f9d22 ui: error out for meaningless options for non-cloud dvc status 2020-01-07 17:25:55 +07:00
Alexander Schepanovski a19deb2076 dvc: optimize Git.is_tracked() 
Closes #3000
 2020-01-04 11:32:16 +07:00
Alexander Schepanovski 0352adc639 py3: use yield from where appropriate 2020-01-04 11:03:30 +07:00
Alexander Schepanovski 5dd2d7b192 test: do not use repo template in erepo_dir 2020-01-04 10:47:13 +07:00
Alexander Schepanovski b194bfa1ff test: do not bootstrap versions in erepo_dir 2020-01-04 10:47:13 +07:00
Alexander Schepanovski b2d083fd88 dvc: optimize and clean up stages collection 2020-01-03 22:06:52 +07:00
Alexander Schepanovski 5e92a079b8 dvc: autowrap repo tree into CleanTree 2020-01-03 21:46:05 +07:00
Alexander Schepanovski ecd511fc69 dvc: safely return to saved tree in brancher 2020-01-03 19:09:34 +07:00
Alexander Schepanovski 77b937b001 dvc: drop unused params in Repo.brancher() 2020-01-03 19:05:54 +07:00
Alexander Schepanovski 1e5fb102a2 test: add @TmpDir.chdir() 2020-01-03 16:58:32 +07:00
Alexander Schepanovski b8cb6d72a9 metrics: don't show working tree if it is clean 2020-01-01 10:59:13 +07:00
Alexander Schepanovski bd0b28d7c2 ui: add checkout --relink to autocomplete scripts 2019-12-31 14:04:51 +07:00
Alexander Schepanovski b6b07dde8b dvc: make graph consist of stages not their paths 2019-12-30 12:07:52 +07:00
Alexander Schepanovski 322b354dd5 test: fix and migrate version tests 2019-12-27 22:07:46 +07:00
Alexander Schepanovski 5f3e416e60 test: fix test_ignore_external in Python 3.5 2019-12-27 15:51:07 +07:00
Alexander Schepanovski 094482249a ui: fix extra newline in dvc version 
Additionally don't try to detect cache fs type unless cache dir exists.
 2019-12-27 15:45:41 +07:00
Alexander Schepanovski e8c321e9a3 test: test ignore nuances 
- nested files are ignored
- external dirs are not affected
 2019-12-24 12:46:32 +07:00
Alexander Schepanovski 6c0cda8c7b ui: reorder checkout options to go from common to rare 
This is the order we use in docs. It also makes sense.
 2019-12-17 18:40:26 +07:00
Alexander Schepanovski 5d1f01d58f checkout: iron out --relink related messages 2019-12-13 19:00:06 +07:00
Alexander Schepanovski 1143a8c57f test: add checkout parse-cli-api test 2019-12-12 20:16:17 +07:00
Alexander Schepanovski b8cf906edb test: stop erepo_dir from undoing parent chdir 2019-12-11 20:19:38 +07:00
Alexander Schepanovski a1643d576d test: fix run_copy fixture when used with scm 2019-12-11 20:19:02 +07:00
Alexander Schepanovski 9607e5f873 get: refactor, clean up and fix dvc get implementation 
Things changed:
- both no output and no git file result into PathMissingError now
- fail on missing cache or cache download error loudly
- removed sqlite bypass, since we use nolock now anyway

Refactoring:
- streamlined logic and made it similar to Repo.open() one
- moved exclusive exception classes to dvc.repo.get
 2019-12-09 21:02:49 +03:00
Alexander Schepanovski 3610961144 dvc: warn to run dvc checkout --relink in cache.type change 2019-12-09 13:46:10 +03:00
Alexander Schepanovski 2f79e362f4 dvc: add checkout --relink 2019-12-09 13:25:32 +03:00
Alexander Schepanovski 3b2666e46b remote: remove unused exception class 2019-12-09 12:45:56 +03:00
Alexander Schepanovski 88370e1b00 dvc: reorganize imports in dvc.remote.base 2019-12-09 12:40:56 +03:00
Alexander SchepanovskiandMr. Outis 1da133b9ed test: update dir_helpers usage 
Co-Authored-By: Mr. Outis <[email protected]>
 2019-12-09 11:34:10 +03:00
Alexander SchepanovskiandMr. Outis a59041aeab test: update dir_helpers usage 
Co-Authored-By: Mr. Outis <[email protected]>
 2019-12-09 11:33:35 +03:00
Alexander SchepanovskiandMr. Outis 0cd6106189 Update tests/dir_helpers.py 
Co-Authored-By: Mr. Outis <[email protected]>
 2019-12-09 11:24:41 +03:00
Alexander Schepanovski 6c12a2e555 test: fix unicode test_ignore issue under Windows Py2 2019-12-07 17:17:18 +01:00
Alexander Schepanovski 79902988fc test: fix files set comparisons in dvcignore tests under windows 2019-12-07 13:17:08 +01:00
Alexander Schepanovski 77fcd75b7d test: use utf-8 in tmp_dir.gen() 2019-12-07 13:16:42 +01:00
Alexander Schepanovski 42096c5bfc test: migrate test_ignore to new helpers 2019-12-07 11:09:52 +01:00
Alexander Schepanovski c6c67ace20 test: create empty dirs via tmp_dir.gen() 2019-12-07 10:32:37 +01:00
Alexander Schepanovski 5bec700761 test: accept Paths in tmp_dir.gen() 2019-12-06 18:36:17 +01:00
Alexander Schepanovski 659eb04705 test: write a usage text for dir_helpers 2019-12-06 18:34:05 +01:00
Alexander SchepanovskiRuslan KuprieievPaweł Redzyński
 		c27988a042 test: minor fixture refactor fixes from @pared and @efiop 
Co-Authored-By: Ruslan Kuprieiev <[email protected]>
Co-Authored-By: Paweł Redzyński <[email protected]>
 2019-12-02 23:33:44 +01:00
Alexander Schepanovski 0385810662 test: fix python 2, 3.5 and mac issues with new fixtures 2019-12-02 23:32:11 +01:00
Alexander Schepanovski fcebad821e test: refactor tmp dir helper fixtures 
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
 2019-12-02 23:31:54 +01:00
Alexander Schepanovski d1cf770a3c dvc: don't make an unneeded copy on commit 
Fix #2016 without all the complications of #2358.
 2019-12-02 23:08:45 +01:00
Alexander Schepanovski 30d58e14a0 Revert "remote: base: don't checkout existing file (#2358)" 
This reverts commit 5ec1d274ae.

Save tests to try make them work, and `_get_cache_type()` to be used
later.
 2019-12-02 19:06:35 +01:00
Alexander Schepanovski 4ab4e43606 dvc: return to using .changed() shortcut in .checkout() 2019-12-02 14:38:39 +01:00
Alexander Schepanovski 865ed6f8ea test: skip non supported remotes fast in api tests 2019-11-29 23:42:26 +01:00
Alexander Schepanovski 8af1aeefb4 remote: refactor ssh ask password code 2019-11-28 21:10:35 +01:00
Alexander Schepanovski 73eac3b6cc remote: protect all remote client/session creation code with locks 
With @cached_property last write wins, so all threads will use the same
session. However, several might be created concurrently wasting
resources and even potentially asking something from the user.

This change guarantees only one client instance will be created for a
remote.
 2019-11-28 21:05:20 +01:00
Alexander Schepanovski 8681cc3cd1 perf: optimize cache listing for local, ssh and hdfs 
- make all three lazy
- simplify local one
- use deque instead of list for hdfs
 2019-11-22 14:43:01 +01:00
Alexander Schepanovski 8da0390c57 remote: small .save_info()/.get_checksum() cleanup 2019-11-22 12:58:08 +01:00
Alexander Schepanovski 0c2a475e93 dvc: remove unused config paths 
These are remnants from old style remotes configuration
 2019-11-20 20:33:46 +01:00
Alexander Schepanovski 3bffe9dbff dvc: drop schema from requirements 2019-11-18 17:24:40 +01:00
Alexander Schepanovski 2aa750cb48 perf: switch schema validation library for config 2019-11-18 17:22:05 +01:00
Alexander Schepanovski 3943b0eb57 perf: a small stage validation optimization 2019-11-15 16:31:46 +01:00
Alexander Schepanovski 78fad73f2b perf: switch schema validation library 
voluptuous is compiling schema to python function, which makes it
faster.
 2019-11-14 22:33:28 +01:00
Alexander Schepanovski 329d072c22 perf: explain why two yaml parsers are used 2019-11-13 16:35:05 +01:00
Alexander Schepanovski 58c9e5557c perf: optimize yaml parsing for stages 2019-11-11 13:41:50 +01:00
Alexander Schepanovski b83564d80e perf: refactor URLInfo to not use ParseResult inside 
This permites URLInfo construction from parts without building/reparsing
string, which makes it around 20% faster. Code is also simpler and a bit
shorter now.

Other improvements:
- added URLInfo.replace(path=...)
- URLInfo._path now stringifies properly
- protected against passing path not starting with /
- do not allow query, params, fragments and passwords in urls (they were
silently ignored previously)
 2019-11-01 17:33:00 +01:00
Alexander Schepanovski 77fcd49695 dvc: fix tqdm dep (#2694) 
tqdm==4.35.0 was throwing KeyError: 'percentage'.
 2019-10-30 16:43:18 +02:00
Alexander Schepanovski 86811aac20 imp: dedup and parallelize fetching imported deps (#2654) 
* imp: dedup and parallelize external fetches

This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.

* imp: test that fetch works on non workspace deps
 2019-10-23 18:02:50 +03:00
Alexander Schepanovski 30341438e0 gc: added --all-commits (#2643) 
* dvc: fix branch suffixes in status/push/pull

* gc: add --all-commits flag

* gc: fix gc warning

Say that we always leave things used in the working tree
 2019-10-23 18:00:31 +03:00
Alexander Schepanovski 33b0eefec0 dvc: dedup used cache (#2630) 
* dvc: dedup used cache

This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts

* dvc: delay checksum names concatenation
 2019-10-21 12:12:56 +03:00
Alexander Schepanovski 1f58fc6f42 dvc: dedup brancher (#2639) 2019-10-20 17:32:42 +03:00
Alexander Schepanovski c26233acbb dvc: reset dvcignore 
This depends on filesystem and affects stages collection so must be
reset.
 2019-10-13 19:18:53 +07:00
Alexander Schepanovski 44264857c3 ui/perf: do not clone same repo multiple times (#2583) 
* ui/perf: do not clone same repo multiple times

Closes #2490

External repos are still created in /tmp. They might be reused until
main() end. If no perfect match is found then a repo with the same url
is copied (not cloned again) and adjusted to fit a request: rev and
cache_dir.

NOTE: test_download_error_pulling_imported_stage() was modified to not
rely on clone error.

* test: fix import update test

* test: fix external_repo test under Windows

* dvc: fix non-master default branch for external repos

* dvc: remove unused .unset_dir() method
 2019-10-11 13:56:37 +03:00
Alexander Schepanovski 0203fd4017 perf: import dvc.repo dynamically 2019-09-27 15:08:42 +07:00
Alexander Schepanovski e2eb4ae0c3 perf: delay cache creation 
The older code instantiates non-local cache remotes on Repo creation if
any of those are configured. And remote instantiation loads their API
libs.
 2019-09-24 09:03:03 +07:00
Alexander Schepanovski 4006c6434e dvc: remove dead code in DataCloud 2019-09-23 20:00:19 +07:00
Alexander Schepanovski cb5ff6e085 perf: import some things later 
Saves on startup time
 2019-09-20 16:45:02 +07:00
Alexander Schepanovski ffb52b256b perf: do not import http.server each run 
It is only used in tests.
 2019-09-17 18:41:05 +07:00
Alexander Schepanovski eac0aa13b2 perf: use packaging.version instead of pkg_resources 
This is the same functionality without a cost of loading a huge package.
 2019-09-17 14:32:31 +07:00
Alexander Schepanovski 293a50d75c perf: use fast entry points on non-wheel install 
This avoids importing `pkg_resources` on startup, which saves around 200
ms.

NOTE: wheel installer uses its own entry points template, which is the same
fastentrypoints.py script provides, so no gain there.
 2019-09-12 22:33:59 +07:00
Alexander Schepanovski 3c3378bdca dvc: make lock lifetime long by default 2019-09-11 14:21:12 +07:00
Alexander Schepanovski d9d85f3cb0 dvc: fix locks in Python2.7/Windows 
Return to zc.lockfile lock in Python 2, NFS is not supported there
anyway since sqlite3 doesn't support URIs.
 2019-09-11 14:17:21 +07:00
Alexander Schepanovski f29f07ca88 dvc: do not lock sqlite db 2019-09-11 14:17:16 +07:00
Alexander Schepanovski 6aeccfcc07 dvc: fix long lock filenames under Windows 2019-09-06 16:29:23 +07:00
Alexander Schepanovski 1bdaca0d6c dvc: switch to flufl lock 
This locks employs hardlink trick, which works on NFS and alike unlike
fcntl/open locks.

Closes #1918.
 2019-09-06 16:29:09 +07:00
Alexander Schepanovski b52265a1d2 ui: remove extraneous stage.save() 
stage.update() already does that via .run(). That was causing printing a
confusing message to user.
 2019-08-27 15:04:37 +07:00
Alexander Schepanovski c68b76f843 ui: remove "Reproducing ..." message 
It as always followed by a more specific one, so it's superfluous.
 2019-08-27 15:02:21 +07:00
Alexander Schepanovski edcd1724b4 cache: skip non-checksum files around cache 
Fixes #2394
 2019-08-24 15:04:25 +07:00
Alexander Schepanovski 8984e9fed4 dvc: use wdir == "." on add 
Fixes #2404
 2019-08-24 10:23:41 +07:00
Alexander Schepanovski 85289ad00b api: wrap not found errors 
api.open()/read() will now raise OutputFileMissingError and
Remote*.open() will raise FileNotFoundError if target file is absent.
 2019-08-15 23:20:48 +07:00
Alexander Schepanovski 60a983b63f api: stream api.open() for local, SSH and HDFS 2019-08-13 18:19:52 +07:00
Alexander Schepanovski 641eb57b27 api: stream data in api.open() for S3, GS and OSS 
Employ the same sign url and use resumable download approach tested for
Azure.
 2019-08-13 14:09:51 +07:00
Alexander Schepanovski e5e29065c8 api: make .open() stream data for Azure 
This employs generating temporary signed public url and then streaming
from it. The bulk of the code might be reused for other remotes.

Url streaming does resume on connection error using "Range" header and
regenerates public url too.

If remote class doesn't provide .open() then this falls back to pulling
a file and opening it locally.
 2019-08-09 21:51:43 +07:00
Alexander Schepanovski 50b5b87304 dvc: batch push/pull/fetch/status 
This will speed up cache collection and then run single cloud op for all
collected cache.

Fixes #2308.
 2019-07-30 13:33:01 +03:00