This permites URLInfo construction from parts without building/reparsing
string, which makes it around 20% faster. Code is also simpler and a bit
shorter now.
Other improvements:
- added URLInfo.replace(path=...)
- URLInfo._path now stringifies properly
- protected against passing path not starting with /
- do not allow query, params, fragments and passwords in urls (they were
silently ignored previously)
* imp: dedup and parallelize external fetches
This prevents same fetches for external artifacts, groups them by source
and fetches in parallel.
* imp: test that fetch works on non workspace deps
* dvc: fix branch suffixes in status/push/pull
* gc: add --all-commits flag
* gc: fix gc warning
Say that we always leave things used in the working tree
* dvc: dedup used cache
This is a preparation step to make garbage collector work with more
commits. It has a value of its own:
- less exists calls to show cloud status or prepare push/pull
- same names are not repeated in status and pbar texts
* dvc: delay checksum names concatenation
* ui/perf: do not clone same repo multiple times
Closes#2490
External repos are still created in /tmp. They might be reused until
main() end. If no perfect match is found then a repo with the same url
is copied (not cloned again) and adjusted to fit a request: rev and
cache_dir.
NOTE: test_download_error_pulling_imported_stage() was modified to not
rely on clone error.
* test: fix import update test
* test: fix external_repo test under Windows
* dvc: fix non-master default branch for external repos
* dvc: remove unused .unset_dir() method
This avoids importing `pkg_resources` on startup, which saves around 200
ms.
NOTE: wheel installer uses its own entry points template, which is the same
fastentrypoints.py script provides, so no gain there.
This employs generating temporary signed public url and then streaming
from it. The bulk of the code might be reused for other remotes.
Url streaming does resume on connection error using "Range" header and
regenerates public url too.
If remote class doesn't provide .open() then this falls back to pulling
a file and opening it locally.
Since this is multiplexing over the same TCP connection no system
resources are withheld, so this should be a no-brainer optimization.
As a side-effect this should fix "open max - close all - try open -
Administratively prohobited" error @darabi experiencing in #2280.
* remote: drop not working http resume
Closes#2136
* remote: drop @transfer_context()
It is not needed for ssh anymore and GS/S3 are switched to client
instances in cached properties.
* remote: drop upload/download chunking
* remote: support older linuxes as ssh remotes
Work around md5sum not having --tag in Red Hat 6.3 and friends.
Fixes#2242.
* remote: optimize consequent ssh md5 calls
Bugs fixed along the way:
- unable to copy between buckets in GS
- unneeded connection init in RemoteGS.copy()
- connection not closed on some errors in RemoteSSH.download()
- inconsistent progress bar handling
Replaced Path* dataclasses with PathInfo/URLInfo/CloudURLInfo, which
know how parse, serialize, stringify and join themselves. They also
provide useful accessors like part of an url or pathlib.Path like
.parent or .name.
Along the way two families of bugs where fixed and should never occur in
the future:
- paths not matched as strings anymore, e.g. using str.startswith()
- Output.path is eliminated as ambiguous
Both families might had caused undefined surprising behavior when local
and remote paths clash.
As a part of this:
- DataCloud._cloud was refactored into `.get_remote()`
- data stopped using internal _cloud
- added a parametrization for remote tests
Now simple `py.test` command won't use DEBUG log level and `py.test ...
--log-level=LEVEL` will do what it's supposed to.
Staging continues to use DEBUG log level.
This is part of #1888 resolution.
Note we call main() to spare process overhead, this leads to main()
inheriting log level and later tests inheriting whatever main() run
left. This makes main to behave.
* test: clean up test dirs
Closes#1770. Partially does #1888.
* dvc: close anything opened
* test: finalize git repo explicitly to remove test dir later
This was previously an issue on Windows.
* test: remove never used copy of load_stage_file()
* dvc: save comments and custom formatting in stage files
* Add a couple of TODOs
* test: test repro doesn't clear comments and formatting in .dvc
As part of this I started writing pytest-like tests and added commonly
used fixtures.
* test: simplify and prettify dvc formatting retained test
Also test that line comments after md5 fields are retained.
* test: fix tests
* dvc: fix stage and metrics opens not closed
* dvc: always use utf-8 in stage files
It used to encode in system default encoding. Not sure if PyYAML was
handling this silently or we were just lucky all the way.
* test: unit test apply_diff()
* test: remove TODOs and ASK
* dvc: polish dvc-formatting PR
* dvc: move stage load/dump utils to a new stage utils file
* dvc: clean up apply_diff()
* test: test stage md5 ignores comments
* test: remove pytests starting marker
