Modernization Roadmap for main branch

[iAmGiG]

Overview

Full modernization of codebase to modern Python/TensorFlow stack.

Target Stack

• Python: 3.11+
• TensorFlow: 2.17+
• Pandas: 2.2+ (breaking changes from 2.0)
• NumPy: 2.0+ (breaking changes)
• Scikit-learn: 1.5+
• Federated Learning: Flower (flwr) instead of TFF

Breaking Changes to Address

1. Pandas 2.0+ Migration

• DataFrame.append() removed → Use pd.concat()
• Impact: ~10 locations
• Files: train_og.py, test.py, classification scripts

2. Keras 3.0 Migration

• Standalone keras deprecated
• Options: tensorflow.keras or new keras 3.0 package
• Impact: All imports, model loading/saving
• .h5 format may need migration to .keras

3. TensorFlow 2.17+ Changes

• Optimizer API changes
• Model saving format changes
• Callback signatures may differ

4. NumPy 2.0 Changes

• Type system changes
• Some deprecated functions removed
• May impact array operations

5. Federated Learning Rewrite

• Replace TensorFlow Federated with Flower framework
• TFF 0.40.0 has impossible dependencies (jaxlib conflicts)
• Flower is better documented, actively maintained
• Original FL code never used actual botnet data anyway

Migration Strategy

1. Fix deprecations first (pandas, keras imports)
2. Update TensorFlow code for 2.17+ API
3. Test classification and anomaly detection modules
4. Rewrite FL implementation with Flower (new code, not port)
5. Add proper unit tests
6. Add CI/CD pipeline

Dependencies

• Blocked by: CRITICAL: Data leakage in scaler fitting #13 (data leakage - should fix first)
• Related: Fix deprecated pandas.append() calls #14, Mixed keras imports causing version conflicts #15, Test/train data split overlap concern #16

Branch

Work will be done on main branch after archive-2020-fixed is complete.

References

• Flower docs: https://flower.ai/docs/
• TensorFlow 2.17 migration: https://www.tensorflow.org/guide/migrate
• Pandas 2.0 migration: https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html

[iAmGiG]

Updated with comprehensive modernization roadmap. See docs/MODERNIZATION_ROADMAP.md for full details covering all 7 phases of modernization over 14 weeks.

[iAmGiG]

Sprint 1 Complete ✅

Date: 2025-10-26
Commit: d56d93e
Status: Pushed to GitHub

Completed

• ✅ Architecture visualization (Architecture Visualization with Graphviz #25)
• ✅ 11 diagrams generated (current_* vs target_*)
• ✅ Comprehensive planning documentation
• ✅ Environment: Python 3.12.12, TensorFlow 2.19.1, Flower 1.22
• ✅ README updated with Architecture section
• ✅ Repository cleaned and organized

Documentation Created

• docs/MODERNIZATION_ROADMAP.md - Complete 14-week plan
• docs/BLEEDING_EDGE_STACK.md - Technology stack
• docs/TOOLING_EVOLUTION_2020_2025.md - FL research
• docs/PRIORITY_ADJUSTMENTS.md - Decision history
• docs/architecture/ - Full architecture docs + diagrams

Next: Sprint 2

Starting Directory Restructuring (#26)

• Target: Modern src/ package layout
• Timeline: Weeks 3-4