Fix: Confidence Blacklist Timeout — Eliminate Redundant Protection/FP Checks on 4.5M IPs#7
Closed
Fix: Confidence Blacklist Timeout — Eliminate Redundant Protection/FP Checks on 4.5M IPs#7
Conversation
Agent-Logs-Url: https://github.com/juergen2025sys/NETSHIELD/sessions/d480cef9-1501-496f-bfde-669a5280f513 Co-authored-by: juergen2025sys <220786640+juergen2025sys@users.noreply.github.com>
…/is_in_fp_set calls, add FP pre-filter with binary search Agent-Logs-Url: https://github.com/juergen2025sys/NETSHIELD/sessions/d480cef9-1501-496f-bfde-669a5280f513 Co-authored-by: juergen2025sys <220786640+juergen2025sys@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
juergen2025sys
April 15, 2026 18:02
View session
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
Confidence Blacklistworkflow consistently timed out at 59 minutes and was repeatedly cancelled. The Python step produced no visible output (due to stdout buffering) before being killed.Root Cause
update_confidence_blacklist.ymlcalledis_protected_entry()andis_in_fp_set()redundantly on all 4.5 million IPs in the seen_db/combined blacklist:is_protected_entryoncombined_ipsloadis_protected_entryin main loop conditionis_in_fp_setin main loop conditionis_protected_entryin append conditionsThese checks are entirely redundant because
combined_threat_blacklist_ipv4.txtis already pre-filtered for protected entries and FP IPs at write-time inupdate_combined_blacklist.yml.Fix
is_protected_entryfromcombined_ipsloading — the file is already filteredcombined_ipsagainst FP IPs once before the loop:is_in_fp_set(ip)from inner loop —combined_ipsis now pre-filteredis_protected_entryguards from confidence append conditionsPYTHONUNBUFFERED: "1"so Python output is visible in the logsResult