PKI index optimization using memory-mapped hash table#68891
Open
dwoz wants to merge 42 commits intosaltstack:masterfrom
Open
PKI index optimization using memory-mapped hash table#68891dwoz wants to merge 42 commits intosaltstack:masterfrom
dwoz wants to merge 42 commits intosaltstack:masterfrom
Conversation
dwoz
changed the title
…ssions - Use _indices dict in localfs_key to isolate index objects by pki_dir - Ensure PKI index files have 0600 permissions in verify_env - Move index ownership/permission logic to end of verify_env for efficiency
- Remove expensive open/close on every operation - Use salt.utils.files.wait_lock for multi-process safe writes - Use realpath for path comparisons in index caching - Use explicit write() instead of truncate() for robust file allocation - Ensure PKI index files have 0600 permissions
- Remove expensive open/close on every operation for better performance - Use fcntl.flock for multi-process safe writes without deadlock risks - Robustly handle atomic file swaps in open() - Ensure full file allocation during initialization for macOS/Windows compatibility
- Move .pki_index.mmap from /etc to /var/cache/salt/master - Update verify_env signature to accept opts and handle permissions in new location - Update all verify_env call sites to pass opts - Fix NameError in verify_env and PermissionError in unit tests
- Wrap index operations in localfs_key with pki_index_enabled check - Remove all temporary debug logs - Maintain MmapCache robustness and relocated cachedir path
- Use valid_id in list_all to ensure consistent filtering - Move salt.output, mmap_cache, and pki imports inside functions to avoid early load issues - Ensure pki_index_enabled is respected in all cache paths
- Revert signature changes for list_keys, all_keys, and list_status to fix unit test mocks - Implement directory-scan fallback in localfs_key.list_all when index is disabled - Move imports inside functions to avoid circular dependencies and loader issues
- Hash PKI index filenames to prevent collisions in multi-master/test environments - Restore unit test mock compatibility by only using list_all() when index is enabled - Add clean_path() security check to list_all() keys scan - Update verify_env() to handle hashed index filenames - Fix rebuild_index runner test by enabling pki_index_enabled: True
…ility - Revert Key.list_keys(), Key.all_keys(), Key.local_keys() to exact master baseline - Revert localfs_key.list_() to exact master baseline - Keep O(1) index optimization as a strictly isolated separate path - Finalize denied_keys branch in list_all() with clean_path checks
Only add minimal hook for pki_index_enabled in master.py maintenance loop.
- Fix critical indentation error in master.py maintenance loop - Hard revert key.py, master.py, verify.py to master baseline - Keep PKI optimization only as a strictly isolated opt-in path
- Move hashlib and pwd imports to top-level in localfs_key.py - Add pylint: disable=import-outside-toplevel for necessary late imports - Move localfs_key import to top-level in master.py
Move import-outside-toplevel to where it is needed.
- Initialize base variable in store() - Use base_dir for Path object to avoid shadowing
Remove opts from verify_env call to match master signature.
Use localfs_key_cache alias for local import to avoid shadowing global salt namespace.
- Hard revert master.py to master baseline to break circular imports - Move PKI index update hooks into key.py (accept/delete) - Hooks are strictly opt-in and isolated in try/except blocks - Verified 10/10 pylint score locally
- Resolve error handling regression in saltutil.regen_keys - Remove unnecessary logic changes in minions.py - Restore cached_property in cache/__init__.py - Maintain O(1) PKI optimization as strictly isolated opt-in feature
- Document new salt-run pki commands - Add master configuration reference for PKI index settings - Add changelog entry for O(1) PKI optimization
- Add pki_index_shards documentation - Improve detail on pki_index_size and pki_index_slot_size
- Create doc/topics/performance/ guide - Add PKI Index Operations documentation - Link Performance section from main Table of Contents
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a generic memory-mapped cache utility to eliminate$O(N)$ directory scans when verifying minion IDs on the Master.
What does this PR do?
What issues does this PR fix or reference?
Fixes
Previous Behavior
Remove this section if not relevant
New Behavior
Remove this section if not relevant
Merge requirements satisfied?
[NOTICE] Bug fixes or features added to Salt require tests.
Commits signed with GPG?
Yes/No