* experiments: move executor into a module
* experiments: separate executor into base/local
* rename LocalExecutor to TempDirExecutor, run in .dvc/tmp by default
* git: restore HEAD using reset not checkout in detached_head()
* experiments: support calling BaseExecutor.reproduce directly for
workspace runs
* experiments: run single experiments in local workspace by default
* update experiments tests for workspace changes
* experiments: update checkpoints for workspace runs
* update checkpoints tests
* update test_show
* add force param for apply
* exp apply: add --force option
* exp run/res: add --temp option to force tempdir execution for single
experiments
* force apply in non-apply specific tests
* experiments: add utils.exp_commits for iterating over experiment commits
* repo: add support for all_experiments in brancher and used_cache
* git: make all_commits return commits in standard refs instead of --all
* gc: add --all-experiments to preserve cache for experiments
* update gc tests
* repo: support pushing and fetching specific revs internally
* exp push: push cache for experiment by default
* exp pull: pull cache for experiment by default
* fix empty git repo bug
* git: remove scm from backends, make backends configurable
* git: implement get/set/remove ref in gitpython
* git: support reflog --rewrite for stash
* tests: support testing individual backends in unit/scm/test_git
* stash: create reflog if it does not exist for gitpython backend
* git: fix Stash.push args typing
* experiments: handle case where args file was inadvertently committed to
git
* add test for args file being committed to git
* experiments: commit to refs/exps/... instead of branch heads
* update exp show
* restore checkpoint_resume kwarg
* refactor new() to use experiments stash
* experiments: use refspecs to populate and collect executors
* experiments: use new stash and merge on exp checkout
* experiments: fix executor chdir
* experiments: update run and show
* skip checkout after reproduce
* checkpoints: use experiments refspecs
* check if workspace was stashed before applying
* exp show: unify checkpoint/standalone experiment collection
* git: move iter_refs to backend
* update for scm changes
* experiments: replace exp checkout in favor of exp apply
* git: handle rev_parse errors in resolve_commit
* update refs containing and baseline check
* fix sigint handling for checkpoints
* update checkpoint exists behavior
* fix checkpoint baseline behavior
* update tests
* experiments: update gc
* update show tests
* fix type checking errors
* update diff
* scm: add diff, describe
* experiments: generate human readable names
* exp run/resume: allow user specified exp name with -n/--name
* update tests
* experiments: add `exp branch` for promoting experiment to full branch
* fix test
* fix missing file
* experiments: explicitly close scm instances before cleaning up tmp dir
* git: add support for dulwich Stash
* git: start adding support for custom refs
* git: add Stash and helper context managers for preserving
workspace state
* scm: cleanup refs/stash and add push_refspec
* tests: add tests for git refs and stash
* fix push_refspec
* tests: add tests for git push_refspec
* scm: add git fetch-refspecs
* add test for fetch refspecs
* scm: add test for git refs containing
* remove extraneous log msgs
* fix typing issues
* stash: only try to push if workspace is dirty
* update dulwich to 0.20.14
* scm: move dulwich into scm.backend
* move stash internals to backends
* scm: move git code into dvc/scm/git/... module
* git: use ABC for BaseGitBackend
* fix rebase issue
* git: fix refspec tests on windows
- git file:// URLs need posix style paths
* git: move get_refs_containing to backend
* scm: add callback for handling diverged refs during fetch
* tree: use file_mode in upload() for trees that support modes
* experiments: preserve file_mode for files in the workspace
* skip permissions tests on windows
* LocalTree: separate chmod from protect
* fix rebase issue
* experiments: Allow branching from checkpoints with modified hyper params
* experiments: initial checkpoints tree functionality for `exp show`
* go back to flat/sortable view
* add test for resume w/modified params
* tests: move experiments fixtures into conftest
* add show checkpoints test
* test checkpoint table rendering
* remove dead code
* test show for checkpoint branches
* experiments: refactor regular/checkpoint experiment separation
* fix scm_checkout pull bug
* experiments: return stages from executor.reproduce
* since executor has to move to threadpool execution anyways, we no
longer have to work around the pickle repro result limitation
* refactor new/resume behavior
* refactor checkpoints tests
* use -f to force checkpoints reset/restart
* checkpoints: use 'exp run' and 'exp res[ume]' commands
* stage: all outs can be missing for checkpoint stages
* regular outputs may not be written at the time a checkpoint is
generated, but subsequent checkouts of that stage should not fail due
to the missing outs
* commit stage on checkpoint generation otherwise non-persist cached outs
are lost
* checkpoints: wait for stage proc to end after killing via terminate()
* stage: fix issues with killing checkpoint stage
* refactor signaling (use threading.Event instead of Condition)
* fix windows bug (kill all child processes via taskkill)
* experiments: refactor executor collection
* tests: move checkpoint tests into separate module
* refactor experiment tests to use stage fixture
* add: warn on cache link failure instead of error
* add test case
* fix linted string
Co-authored-by: Paweł Redzyński <[email protected]>
Co-authored-by: Paweł Redzyński <[email protected]>
* repo: make Repo.scm a cached attribute
* checkpoints: create final checkpoint after repro when needed
* use correct exception rev in _collect_and_commit
* stage: make runs checkpointable
* api: add make_checkpoint API call
* tests: add test for checkpointed stage
* experiments: implement checkpoints callback chain
* initial working state
* use .dvc/tmp for checkpoint signal file location
* force repro for checkpoint experiments
* checkout: add flag to allow missing persistent outputs
(i.e. not yet created)
* use allow_persist_missing on checkpoint experiment repro
* executor: force checkout before checkpoints experiment repro
* move checkpoint commands to `dvc exp run`
* support resuming checkpoint runs with --continue
* fix git bugs
* experiments: include checkpoint commits in `exp show`
* use 1sec sleep timers
* fix apply_workspace test conflict
* tests: add test for `dvc exp run`
* cleanup signal file behavior
* fix bool conversion
* tests: add tests for checkpoint and checkpoint_continue
* fix styling
* fix checkpoint stage test
* fix checkpoint monitor thread condition/notification
* dvctree: use string paths for performance reasons in
_get_granular_checksum()
* dvctree: cache most recently accessed metadata
* dvctree: cache output dir cache during walk() calls
* diff: use get_file_hash directly instead of get_hash when appropriate
* dvctree: cache most recen dir_cache hashes in _get_granular_checksum
* revert metadata cache changes
- mostly unnecessary after granular checksum changes, will be better
addressed by the future exception handling updates
* fix windows path issue
* HashInfo: store HashInfo.dir_info as a dict instead of list
* DvcTree: use hash_info.dir_info for getting granular file hashes
* use str relpath
* use correct dir_info hash type
* fix windows paths
* diff: support missing cache status for workspace diff
* diff: add --hide-missing option to hide cache missing status
* update command tests
* update diff func tests
* only show missing cache status when diffing local workspace
* suppress all output if no changes and --hide-missing
* use hide_missing instead of show_missing
* use is_dvc
* diff: use RepoTree to get granular results for dir outs
* update tests for dir file granularity
* diff: prefer repo_tree.get_hash when possible
* DvcTree: handle extra files in dirty dvc out dirs
* experiments: cleanup show UI
* show git tag/branch names instead of SHAs by default
* make timestamp formatting more like the viewer
* update tests
* place timestamps in their own column
* update command test
* exp: remove unused regex
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* plots: add -e/--experiment option for plots diff
* experiments: add method to get baseline for a given experiment
* plots: default to diff against baseline instead of workspace/head for experiments
* use exact match for experiments RE since name-rev may return relative
commits
* executor: abstract tmpdir population using tree upload
* experiments: unmark flaky test
* fix wrong tree download
* LocalTree: use os.replace instead of os.rename in _upload
* experiments: use RepoTree for executor output collection
* experiments: use tree.download for output collection
* cleanup docstring
* use string name for tqdm
* experiments: add options to include/exclude specific metrics/params columns in
`dvc exp show`
* allow param/metric file to be provided to keep consistency with other
dvc commands
* match nested params/metrics names
* tests: add test to ensure dir hashes are not pushed for incomplete
directories
* push: do not push .dir file for directory with missing files
* update test
* tests: test importing into subdirectory
* import: fix bug when output subdir doesnt already exist
* use urlparse to determine if output path is local, not os.path.exists
* account for windows drive letters
* import: error out if output subdirectory does not exist
* restore Experiments.chdir
* used in experiments.show
* experiments: make checkout work without `git reset --hard`
* experiments: remove --force parameter
* experiments: reject checkout if current workspace commit doesnt match
expected baseline
* experiments: add initial local tmpdir executor
* fix detached head pull
* experiments: include unchanged (unrepro'd) stages in experiment hash
* experiments: repro single experiment using tmpdir executor by default
* stage experiments as stash commits before running
* include repro args/kwargs in stashed experiments
* support running arbitrary experiment commits (including stash commits)
* experiments: use ProcessPoolExecutor
* experiments: add `dvc exp` alias for `dvc experiments`
* experiments: add `dvc repro --experiment --queue`
* `--queue` can be used to stage an experiment for future execution
* experiments: show queued (unexecuted) experiments in `dvc exp show`
* revert ProcessPoolExecutor change
* experiments: add `dvc repro --run-all [--jobs]`
* `--run-all` can be used to run all queued experiments in parallel
* cleanup output
* experiments: use ProcessPoolExecutor
* fix returning unpicklable objects error
* update tests
* fix windows cleanup issue
* on windows tempdir cannot be removed if we are chdir'd into that
directory
* erepo: use shallow clones when rev is branch or tag name
* tests: test erepo shallow clone behavior
* erepo: fix detached head after shallow cloning a tag
* move unshallow logic into its own function
* erepo: default rev should be origin HEAD not local HEAD
* use functools.partial
* review fixes
* comment shallow clone behavior in erepo
* don't duplicate unnecessary fetch/pull call
* experiments: add initial experiments.show
* only includes metrics + params (no code/data information)
* experiments: basic `dvc experiments show` functionality
* experiments: clone and run repro inside clone workspace
* repro: add -e/--experiment option
* experiments: hash experiments to identify duplicates
* experiments: update show
* experiments: checkout experiment after running repro
* experiments: add `dvc experiments checkout`
* experiments: add simple `dvc experiments diff` command
* just shows combined output from `metrics diff` and `plots diff` for
now
* add very simple tests
* add some useful output messages
* experiments: suppress help messages while feature is in development
* experiment related commands will be accessible but hidden from the
default command help messages
* add simple experiments command tests
* experiments: disable feature by default
* experiments only enabled in test environment (DVC_TEST) or when
core.experiments config option is true
* use fetch on checkout missing revs
* s3,gs: don't append "/" to prefixed path when listing objects
* remote: use optional prefix kwarg for walk_files
* if prefix is True, path will be walked as a prefix
* if prefix is False, path will be treated as a directory (trailing
slash will be appended for S3-like remotes)
* tests: add unit test for BaseRemoteTree.list_paths
* trees: BaseTree.walk should handle errors like os.walk
* use onerror callback if exceptions should be logged or raised
* repo.ls: use onerror for tree.walk
* remote: move constants into trees
* remote: replace Remote() Cache() helper functions
* output: update for remote changes
* cache/data_cloud: update for remote changes
* dependency: update for remote changes
* stage: update for remote changes
* remote: move remaining non-index methods into tree
* tests: update remote unit tests
* remote: LocalCache needs to lock remote index, not cache index
* fix bugs from moved PARAM_CHECKSUM/PARAM_RELPATH
* tests: repo tree open needs state context
* remote: make Remote.gc a class method
* operates on obj.tree, so that GC can be done with both remote and
cache instances
* bug fixes
* tests: update test_data_cloud
* bug fixes
* tests: update func tests
* remote: move get_file_checksum() into tree
* makes RemoteTree and RepoTree consistent with regard to checksum
calculation
* remote: save() now takes explicit tree parameter
* when tree is remote.tree, save will be a move + link operation
(same as default existing behavior)
* when saving path from a different tree, save will be a copy operation
* tests: update for moved remote/tree functions
* remote: move get_checksum into tree
* remote: separate cloud remote and cache classes
* tests: update unit tests for remote/cache separation
* remote: cloud cache should extend cloud remote
* remote: move LocalRemote.get
* tests: update func tests
* dependency: update for moved get_checksum
* remote: fix state lookup bug
* dvcignore: ignore paths outside of CleanTree root
* remote: local remote/cache should always use its own working tree
* repo tree (CleanTree) instance should not be relevant to local cache
operations. When the dvcignore CleanTree is relevant, repo.tree should
be used. For local remote/cache operations, the local remote work_tree
should be used.
* state: tie state to local cache working tree
* tests: update tests
* state: use os.path.exists instead of WorkingTree.exists in some cases
* remote: move upload()/download() into tree
* remote: move path_info/path_cls into tree
* remote.azure: finish moving methods into tree
* remote.gs: finish moving methods into tree
* remote.hdfs: finish moving methods to tree
* remote.http: finish moving methods into tree
* remote.oss: finish moving methods into tree
* remote.s3: finish moving methods into tree
* remote.ssh: finish moving methods into tree
* remote.gdrive: finish moving methods into tree
* tests: update remote unit tests
* tests: update func tests
* remote: use walk_files() for all remotes
* list_cache_paths() now uses tree.walk_files() for all remotes except
local/ssh
* fix DS warnings
* bugfixes
* BaseRemote: start moving tree functions into BaseRemoteTree
* remote: add LocalRemoteTree
* remote: add cloud remote trees
* remote: move remove(), makedirs() into tree
* remote: move move(), copy(), symlink(), hardlink(), reflink() into tree
* fix attributes for moved remote methods
* tests: update unit tests for moved remote tree methods
* tests: mv unit/remote/test_remote_dir.py unit/remote/test_remote_tree.py
* fix attributes for moved remote tree functions
* tests: update func tests for moved remote tree methods
* test fixes
* fix DS warnings
* ignore: dvcignore root is distinct from tree_root
* for subrepos, dvcignore root should be the subrepo's .dvcignore, not
host repo's .dvcignore
* tests: add test for subrepo dvcignore behavior
* tests: add test for metrics show -T with pre-existing tags
* brancher: ignore git revisions that do not contain repo
* tests: update test to include cmd
* CleanTree: tree_root for subrepos should be the repo root, not scm root
* only force subrepo root for git trees
* fix windows shell quotes
* tests: test that CleanTree works outside of walk()
* ignore: CleanTree should always check ignores, not just for walk()
* CleanTree: handle broken symlinks
* handle case for linking from shared local cache dirs
* Fix windows relpath/different drive issue
* erepo: load external repos directly from git tree
* erepo: add fetch_external
* remote/RepoTree: add callback for keeping track of download counts when
saving tree to cache
* repo: use erepo.fetch_external() in repo.fetch()
* erepo: return cache save_infos in fetch_external
* DependencyRepo: use erepo.fetch_external and cache.checkout instead of
pull_to
* erepo: remove pull_to
* DvcTree: support granularity for isdir()
* DvcTree: fix indentation bug
* RepoTree: add copytree() method
* LocalRemote: use working tree for local cache walk_files
* repo: state should be noop for GitTree based erepos
* erepo: add get_external()
* test_get: remove unneeded caplog check
* erepo: add get_rev()
* tests: fix erepo test bug
* ls: use RepoTree
* remote: support trees in _collect_dir
* remote: support trees in get_dir_checksum
* RepoTree: fix dir_cache relpaths in windows
* remote: fix duplicated fetch bug
* utils: don't skip md5 dos2unix check for git blobs
* otherwise md5 checksums will not match when comparing git version to
cached filesystem version on windows
* DvcTree: use out.dir_cache instead of out.collect_use_dir_cache
* RepoTree: support filtering out dvcfiles during walk()
* RepoTree: support recursive isdvc()
* remote: support computing checksums for tree objects
* RepoTree: pull dir contents on walk() if in `fetch` mode
* tests: add func test for DVC dir pull on RepoTree.walk()
* DvcTree: support granular open()
* RepoTree: add get_file_checksum()
* remote: revert checksum changes
* remote: add tree support for cache.save()
* tests: test saving directly to cache from RepoTree
* remote: local cache needs its own WorkingTree instance when repo.tree is
GitTree
* RepoTree: state context should be entered by open()/walk() caller
* RepoTree: only pull dir contents when cache has changed
* tree: update tree tests
* fix ds warnings, update comments
* fix windows tests
* GitTree: add stat() function
* BaseTree: make walk_files a base tree method
* WorkingTree: add stat() method (wrapper for os.stat)
* utils.fs: support non-working tree for get_mtime_and_size
* utils.fs: add copy_fobj_to_file
* utils: support using tree.open in file_md5
* DvcTree: support stream or fetch on open()
* tests: test DvcTree dir cache support for walk()
* RepoTree: finish implementation of BaseTree functions
* repo: use RepoTree/DvcTree.open() for repo.open_by_relpath
* repo: only use RepoTree when needed for now
* fix docstring
* RepoTree: only use DvcTree.open for DVC outs
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
