* state: only load when needed
* state: don't use separate load/dump
* checkout: group link removal logic together
* state: use diskcache.Cache
* staging: use pbar and save to state
* state: don't pass the whole repo to __init__
* staging: explicitly pass state object
* fs: get rid of odb property
* state: don't close the connection to cache
* repo: close: close state
* dvcignore: Fix incorrect ignored output computation
* dvcignore: Fixed unignored patterns
Unignored patterns were shown in the dvc status output and caused Error.
Also made a bit of a refactoring in `matches` and `ignore`.
* dvcignore: Fix functional tests
Not sure if I follow, but tests assume that unignored files should trigger `check-ignore`. What was the intention?
* dvcignore: Fix stage.wdir being MagicMock
* Use `is_ignored` instead of `check_ignore`
1. use is_ignored to judge DVC ignore status.
2. use check_ignore to show ignore patterns.
* Solve tests fails
* Add a new tests
* Return to the old check_ignore logic.
* Pass the pylint
* Simplify ignore tests
Co-authored-by: karajan1001 <[email protected]>
* Implement export-to-remote functionality
* Redistrubition of functionality from tree to remote
* Pass chunk size into the open_url
* Move functionality away from remote to cache
* Basic progress bars for both transfer types
* Proper job calculation
* Redesign .open() usage in order to work with different trees
* Align the signatures of other open() functions to accept **kwargs
* Implement chunked uploading to the ssh
* Use local cache to compute directory hash (md5)
* S3: multipart upload test
* Wrap local cache's hash_to_path with PathInfo
* Use .isdir() instead of a try/except statement for calculating the from_infos
* New CLI layout (add/import-url)
* Adjust CLI tests to accept new arguments
* tests: import_url/add func/unit tests
* Introduce cache.transfer()
* Invalid CLI combination tests
* Store dir_info key in the transfer_directory
* LocalTree.upload_fobj
* Tests for utils
* Lazy imports on some places
* --to-remote migration (and repo._transfer)
* Use proper temporary location, more efficient pbar calc
* Directory transfer logic rewrite
* Use shutil.copyfileobj()
* Rebase...
* Better tests
* Inline transfer operation to .add() / .transfer()
* Open in the 'rb' mode for the tests
* More tests, better invalid option handling at repo level
* .read() certain amount of bytes to validate the content
* Better invalid argument handling
* Better progress bars by using .getsize() API
* For fallback upload, use the from_info.name instead of temporary naem
* introduce --jobs for add/import-url
* Allow ignoring subrepos in dvcignore
I implemented this inside dvcignore, and it
should not require "internal" APIs hack as we discussed before.
Right now, `ignore_subrepos` will only work for the paths of subrepo,
not anything inside it. And the API user is more or less only
RepoTree, so should be safe (we switch DvcTree and repo.tree based
on path prefixes, so this is only required during `walk`, so as it
shows the repo that were previously dvcignored and so that we could
switch the trees later during walk.
So, if dir is `dir -> repo`
So, `tree.walk("dir")` will return `[]`,
whereas `tree.walk("dir", ignore_subrepos=False)` will return `["repo"]`
But, successive walk will just return `("repo", [], [])`, and could
just be ignored as a side-effect.
* Fix tests
This PR is just a showcase to see how we like it. I've converted a few
tests to it, but there will be much more to come.
```
assert (tmp_dir / "file") == "contents"
assert (tmp_dir / "dir").read_text() == {"subdir": {"file": "hello"}}
```
This approach is nicer because it allows us to use a declaratory
asserts instead of relying on fs files. The problem becomes much more
apparent once you get into remote workspaces, where we `trees_equal`
doesn't work.
* pylint: enable attribute-outside-init check
* fix tests
* Fix mock for time
This was an interesting one. I was mocking `time` on function definition
and then, using those later on test run. When all the tests were run, there
would be some delay that'd make tests fail. Hmmm ...
* pylint: enable checks for imports
* ci: add github action for pylint
* enable pylint on pre-commit
* Install requirements
* use local pylint installation rather than of pre-commit's
* fix import errors
* more checks
* Silence unused imports
* Make a copy of dict
* drop python 3.5 support
* dvc: remove fspath() use inside os.path calls
* dvc: replace dvc.compat.fspath with os.fspath
* tests: replace dvc.compat.fspath with os.fspath
* utils: relpath() should use fspath/str
* utils: os.readlink doesn't support path-like obj on Windows until 3.8
* dvc: use lockfile as yaml instead of json
* refactor Dvcfile into Pipeline file and Single stage file
* fix tests
* dvc: fix outputs
* add more tests for collection of outputs
* add tests for data cloud/get/import/ls
* tests: test for checkouts
* Allow other checksums other than md5
* tests: use iterdir instead of os.listdir
* cleanup errors reported by cc and ds
* utils: throw DvcException instead of plain Exception
* tests: use yaml.load instead of json.load
* run: split assignments
* Update dvc/stage/exceptions.py
Co-authored-by: Ruslan Kuprieiev <[email protected]>
* dont disable existing loggers when dvc is set up
* corrected code style
* dont disable existing loggers, fixes
Co-authored-by: Guro Bokum <[email protected]>
* dvc: refactor config
Advantages:
- read like a dict
- modify as a dict for testing or manipulation purposes
- wrap modifications into context manager to save them:
with config.edit("local") as conf:
conf["core"]["remote"] = "new-default"
Design decisions:
- dropped string constants
- remote sections are parsed into a simple subdict upon load
- remote config schema is now defined by url scheme
- relative paths are handled transparently upon load/save
- user config and remote manipulations are moved to commands
- same dev manipulations are done by simply modifying a config dict
* test: do not set cache.type for remotes that doesn't support it
* dvc: fix windows absolute paths validation in config
* test: fix windows separate drive test
* dvc: stylistic fixes in config
* made '_ignored' check if the given file is already ignored by the root .gitignore. Added corresponding functional test
* code formatting
* Changed method input, adjusted tests, formatted code
* typo
* tests: use git logger fix in conftest
See https://github.com/iterative/dvc/issues/3167
* scm: git: remove redundant comments
Co-authored-by: Ruslan Kuprieiev <[email protected]>
- `tmp_dir` descends from Path
- convenient template methods to generate files and dirs, add them to
dvc and git and commit to git
- independent of old basic_env structures
- can go in any order
- modular: start with empty `tmp_dir`, initialize git with `scm`, dvc
repo with `dvc` add "repo template" with `repo_template`, ...
Ruslan Kuprieiev
Timofey Yefimovandkarajan1001
Batuhan Taskaya
Paweł Redzyński
Saugat Pachhai
Peter Rowlands (변기호)
Casper da Costa-Luis
ttekampeandGuro Bokum
Alexander Schepanovski
Aljoscha SteffensandRuslan Kuprieiev
Jorge Orpinel
Aman Sharma
Mr. Outis