Enhanced Documentation Generator - Complete AST Intelligence

Related Code Files:

code-intelligence-toolkit/doc_generator_enhanced.py - Enhanced documentation generator with full AST tool integration
code-intelligence-toolkit/doc_generator.py - Original documentation generator (data flow analysis only)
code-intelligence-toolkit/data_flow_tracker_v2.py - Core data flow analysis engine
code-intelligence-toolkit/navigate_ast.py - Symbol definition lookup
code-intelligence-toolkit/method_analyzer_ast.py - Method call flow analysis
code-intelligence-toolkit/cross_file_analysis_ast.py - Module dependency analysis
code-intelligence-toolkit/show_structure_ast_v4.py - Code structure visualization
code-intelligence-toolkit/trace_calls_ast.py - Execution path analysis

Overview

The Enhanced Documentation Generator integrates ALL available AST analysis tools to provide comprehensive, intelligent code documentation with multi-dimensional insights.

Key Enhancements over Original

Complete AST Tool Integration: Uses all 5 major AST analysis tools
Interactive Documentation: Rich HTML with tabbed navigation
Multi-dimensional Analysis: Symbol navigation, call flows, dependencies, structure, execution paths
New Documentation Styles: Architecture docs, call graph visualization
Comprehensive Analysis Mode: Maximum depth with all tools combined
Interactive Output Format: Tabbed HTML interface with detailed sections

Architecture

Enhanced AST Analyzer (`EnhancedASTAnalyzer`)

The core wrapper class that orchestrates all AST tools:

class EnhancedASTAnalyzer:
    def navigate_to_symbol(symbol_name) -> Dict[str, Any]
    def analyze_method_calls(method_name, scope) -> Dict[str, Any]
    def analyze_cross_file_dependencies(target_symbol, scope) -> Dict[str, Any]
    def get_code_structure(include_annotations) -> Dict[str, Any]
    def trace_execution_paths(entry_point, max_depth) -> Dict[str, Any]
    def get_data_flow_analysis(variable, direction) -> Dict[str, Any]
    def get_comprehensive_analysis(target_name, target_type) -> Dict[str, Any]

Tool Integration Details

AST Tool	Purpose	Integration Method	Output Used
`navigate_ast.py`	Symbol definition lookup	Direct subprocess call	Line numbers, symbol locations
`method_analyzer_ast.py`	Call flow analysis	Subprocess with `--trace-flow --show-args`	Method calls, parameter tracking
`cross_file_analysis_ast.py`	Module dependencies	Subprocess with `--scope --max-depth`	Cross-file relationships
`show_structure_ast_v4.py`	Code structure	Subprocess with annotation filtering	Hierarchical code view
`trace_calls_ast.py`	Execution paths	Subprocess with entry point tracing	Execution flow analysis
`data_flow_tracker_v2.py`	Variable tracking	Direct Python import	Backward/forward dependencies

New Features

1. Documentation Styles

Original Styles

api-docs: Technical API documentation
user-guide: User-friendly guides
technical: Deep technical analysis
quick-ref: Quick reference format
tutorial: Tutorial-style with examples

New Enhanced Styles

architecture: Module-level architecture documentation
call-graph: Call graph visualization and analysis

2. Depth Levels

Original Levels

surface: Basic signature and purpose
medium: Include dependencies and flow
deep: Complete analysis with all details

New Enhanced Level

comprehensive: All tools, maximum depth, complete analysis

3. Output Formats

Original Formats

markdown: Standard Markdown
html: Basic HTML
docstring: Python docstring format
rst: reStructuredText format

New Enhanced Format

interactive: Rich HTML with tabbed navigation and interactive features

Usage Examples

1. Comprehensive Function Analysis

# Generate comprehensive interactive docs for a function
./run_any_python_tool.sh doc_generator_enhanced.py mymodule.py calculate_total \
    --style api-docs \
    --depth comprehensive \
    --format interactive \
    --output calculate_total_docs.html \
    --verbose

Output Features:

Symbol navigation with exact line numbers
Complete call flow analysis showing all method calls
Data flow tracking for all variables
Cross-file dependency analysis
Code structure visualization
Execution path tracing

2. Architecture Documentation

# Generate module architecture documentation
./run_any_python_tool.sh doc_generator_enhanced.py mymodule.py \
    --style architecture \
    --depth deep \
    --format html \
    --output architecture.html

Generated Sections:

Module structure hierarchy
Cross-file dependencies
Import/export relationships
Module coupling analysis

3. Call Graph Visualization

# Generate call graph documentation
./run_any_python_tool.sh doc_generator_enhanced.py mymodule.py main_function \
    --style call-graph \
    --depth comprehensive \
    --format markdown \
    --output callgraph.md

Generated Content:

Method call relationships
Parameter flow tracking
Execution path analysis
Call stack documentation

4. Interactive Class Documentation

# Generate interactive class documentation
./run_any_python_tool.sh doc_generator_enhanced.py mymodule.py MyClass \
    --style technical \
    --depth comprehensive \
    --format interactive \
    --output MyClass_docs.html

Interactive HTML Features

The interactive output format provides a rich, tabbed interface:

Navigation Tabs

Overview: Basic information and analysis summary
Navigation: Symbol definition lookup and cross-references
Call Flow: Method call analysis and parameter tracking
Data Flow: Variable dependency tracking and impact analysis
Structure: Code hierarchy and organization
Dependencies: Cross-file relationships and module coupling

Interactive Elements

Tabbed Navigation: Switch between analysis dimensions
Collapsible Sections: Detailed information on demand
Code Highlighting: Syntax-highlighted code blocks
Error Handling: Clear indication of analysis successes/failures
Responsive Design: Works on desktop and mobile

Analysis Capabilities by Language

Python (Full Support)

✅ Complete AST analysis
✅ Variable dependency tracking
✅ Method call flow analysis
✅ Cross-file dependency analysis
✅ Execution path tracing
✅ Symbol definition lookup

Java (Enhanced Support)

✅ AST-based parsing via javalang
✅ Method and class structure analysis
✅ Basic cross-file analysis
⚠️ Limited type inference (declared types only)
⚠️ Basic state tracking (field assignments only)

JavaScript/TypeScript (Basic Support)

✅ Structure analysis via esprima
⚠️ Limited semantic analysis
⚠️ Text-based dependency tracking

Error Handling and Robustness

Comprehensive Error Handling

# Each tool integration includes:
- Timeout protection (30s per tool)
- Exception catching and logging
- Graceful degradation
- Cache mechanism for expensive operations
- Verbose error reporting when enabled

Fallback Mechanisms

If one AST tool fails, others continue
Analysis continues with available data
Clear error reporting in output
Fallback to basic analysis if enhanced tools fail

Performance Considerations

Caching Strategy

Results cached by tool + arguments
Expensive operations cached across calls
Memory-efficient for large codebases

Timeout Protection

30-second timeout per AST tool
Prevents hanging on complex analysis
Configurable via environment variables

Parallel Opportunities

Multiple tools can run independently
Future enhancement: parallel tool execution
Current: sequential with caching

Integration with Existing Workflows

Backward Compatibility

Fully compatible with original doc_generator.py API
Same command-line interface
Enhanced features opt-in via new flags

Tool Chain Integration

# Can be integrated into build pipelines
make docs-enhanced:
    ./run_any_python_tool.sh doc_generator_enhanced.py src/main.py \
        --style api-docs --depth comprehensive --format interactive \
        --output docs/api.html

# Batch documentation generation
for file in src/*.py; do
    ./run_any_python_tool.sh doc_generator_enhanced.py "$file" \
        --style architecture --depth deep --format html \
        --output "docs/$(basename "$file" .py)_arch.html"
done

Future Enhancement Opportunities

1. Parallel Tool Execution

Run AST tools in parallel for faster analysis
Requires thread-safe caching mechanism

2. Advanced Visualization

Interactive call graph diagrams
Data flow visualization with D3.js
Module dependency graphs

3. Template System

Customizable documentation templates
Theme support for different organizations
Template inheritance and composition

4. API Integration

RESTful API for documentation generation
Integration with documentation platforms
Webhook support for automated updates

5. Language Support Expansion

Enhanced Java analysis with better type inference
C/C++ support via clang AST
Go language support

Troubleshooting

Common Issues

Tool Not Found

Error: Tool navigate_ast.py failed
Solution: Ensure run_any_python_tool.sh wrapper is available

Timeout Issues

Error: Analysis timeout after 30s
Solution: Use lower depth level or smaller scope

Java Analysis Limited

Note: Java analysis inherits limitations from javalang
Solution: Use for structure analysis, not deep semantic analysis

Debugging

# Enable verbose output for debugging
./run_any_python_tool.sh doc_generator_enhanced.py myfile.py function_name \
    --verbose \
    --depth medium  # Start with medium depth for debugging

Example Output

Interactive HTML Preview

<\!DOCTYPE html>
<html>
<head><title>Interactive Docs: calculate_total</title></head>
<body>
    <div class="container">
        <h1>Function: calculate_total</h1>
        
        <div class="nav-pills">
            <button onclick="showSection('overview')">Overview</button>
            <button onclick="showSection('navigation')">Navigation</button>
            <button onclick="showSection('callflow')">Call Flow</button>
            <button onclick="showSection('dataflow')">Data Flow</button>
            <button onclick="showSection('structure')">Structure</button>
            <button onclick="showSection('crossfile')">Dependencies</button>
        </div>
        
        <\!-- Tabbed content sections -->
    </div>
</body>
</html>

Markdown Output Preview

# Function: calculate_total

## Quick Navigation
- **Definition**: Line 42
- **File**: `src/billing.py`

## Call Flow Analysis

calculate_total() calls: ├── validate_items() [line 45] ├── apply_discounts() [line 48] └── calculate_tax() [line 52]


## Data Flow
### Variable: `total_amount`
**Dependencies**:
- `item_prices`: list
- `discount_rate`: float

**Affects**:
- `final_bill`: dict
- `payment_due`: float

## Cross-File Impact
- Used by: `InvoiceGenerator`, `OrderProcessor`
- Changes affect: 8 files, 23 functions

This enhanced documentation generator provides unprecedented insight into code structure, behavior, and relationships through comprehensive AST analysis integration.

FilesExpand file tree

ENHANCED_DOC_GENERATOR_README.md

Latest commit

History