fix(azure-ai-projects): skip all dot-prefixed directories in evaluator upload#46063
fix(azure-ai-projects): skip all dot-prefixed directories in evaluator upload#46063w-javed wants to merge 2 commits intofeature/azure-ai-projects/2.0.2from
Conversation
…r upload Change skip_dirs filter to exclude any directory starting with '.' instead of only '.git' and '.venv'. This covers .venv, .git, .mypy_cache, .tox, .pytest_cache, and any other hidden/tool directories. Applied to both sync and async upload functions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
w-javed
requested review from
bobogogo1990,
dargilco,
glharper,
howieleung,
kingernupur,
nick863,
trangevi and
trrwilson
as code owners
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
nagkumar91
left a comment
nagkumar91
left a comment
There was a problem hiding this comment.
nit: Now that dot-prefixed directories are excluded, consider applying the same treatment to files — e.g. .env, .DS_Store, .gitignore would still get uploaded. A simple file_name.startswith('.') guard in the existing skip check would close that gap:
if file_name.startswith('.') or any(file_name.endswith(ext) for ext in skip_extensions):
continueExtend the existing skip logic to also exclude dot-prefixed files (e.g. .env, .DS_Store, .gitignore) from evaluator uploads, matching the treatment already applied to dot-prefixed directories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Skips ``__pycache__``, ``.git``, ``.venv``, ``venv``, ``node_modules`` | ||
| Skips directories starting with ``.`` (e.g. ``.git``, ``.venv``), | ||
| ``__pycache__``, ``venv``, ``node_modules`` | ||
| directories and ``.pyc`` / ``.pyo`` files. |
There was a problem hiding this comment.
I don't think it's a good idea to hard code a list of folders, and this is specific to the environment you happen to run the code. A better way is to add an input variable like folder_exclutions_pattern: Optional[re.Pattern] = None, where you provide a reg ex of the folders to exclude. Default is to include all folders.
And per one of my comments earlier, please add file_pattern: Optional[re.Pattern] = None, to match what we have in dataset upload
3 participants
Description
Change the evaluator upload
skip_dirsfilter to exclude any directory starting with.instead of only hardcoding.gitand.venv. This covers:.venv,.git(previously handled).mypy_cache,.tox,.pytest_cache,.ruff_cache,.idea,.vscode, etc.Applied to both sync (
_patch_evaluators.py) and async (_patch_evaluators_async.py) upload functions.Changes
.gitand.venvfrom the explicitskip_dirsset (now covered by thestartswith('.')check)not d.startswith('.')condition to thedirs[:]filter inos.walk