Skip to content

PROJECT_ANALYSIS.md

Latest commit

 

History

History
425 lines (342 loc) · 13.4 KB

PROJECT_ANALYSIS.md

File metadata and controls

425 lines (342 loc) · 13.4 KB

KAI Project - Comprehensive Analysis

Executive Summary

KAI is a sophisticated C++23 multi-language runtime system featuring:

  • Custom object system with tri-color mark-and-sweep garbage collection
  • Three primary languages: Pi (stack-based), Rho (functional), and Tau (IDL/C++-like)
  • Stack-based executor with continuation support
  • Network layer for peer-to-peer communication
  • Interactive console with REPL capabilities

Current Status: Core system stable, TestPi runs without segfaults, Rho basic expressions work. Advanced Pi recursion tests have segfaults, Rho function implementation incomplete. Active development on language features and network generation.

Architecture Overview

Core Object System

Registry (Include/KAI/Core/Registry.h, Source/Library/Core/Source/Registry.cpp)

  • Central object management hub using handle-based references
  • Tri-color garbage collection (white/grey/black sets)
  • Memory management via pluggable allocators
  • Type registration and class management
  • Handle range: monotonic counter from 0

Object Model

  • Handle-based references (not raw pointers)
  • Storage templates for type-safe object storage
  • Reflection system for runtime type information
  • Smart pointer support (recent migration from raw pointers)

Garbage Collection Strategy

  • Mark-and-sweep with tri-color algorithm
  • White set: condemned objects
  • Grey set: may reference white/grey/black
  • Black set: no references to white objects
  • Known issue: HACK at Registry.cpp:201 to avoid cycles

Executor System

Executor (Include/KAI/Executor/Executor.h, Source/Library/Executor/)

  • Stack-based execution engine with data and context stacks
  • Continuation-based control flow
  • Primarily executes Pi bytecode (lowest level)
  • Operations defined in Operation::Type enum
  • Trace levels for debugging execution

Key Components:

  • Data stack: operand storage
  • Context stack: scope management
  • Continuation: code to execute
  • Compiler: language-specific translation to Pi

Execution Flow:

  1. High-level code (Rho/Tau) → Parser → AST
  2. Translator converts AST → Pi continuation
  3. Executor runs Pi operations on stacks
  4. Results accumulate on data stack

Language Implementations

Pi Language (Stack-Based, Low-Level)

Location: Include/KAI/Language/Pi/, Source/Library/Language/Pi/ Status: 93.3% test pass rate (208/223 passing)

Characteristics:

  • RPN (Reverse Polish Notation) syntax
  • Direct stack manipulation
  • Operations: arithmetic, control flow, stack ops
  • Continuations as first-class objects
  • Direct executor bytecode

Example:

5 3 +        # Push 5, push 3, add → result 8
{ 42 } []    # Define continuation with 42, execute

Current Issues:

  • Backtick shell operations failing (security disabled)
  • Some advanced control flow patterns incomplete

Rho Language (Functional, High-Level)

Location: Include/KAI/Language/Rho/, Source/Library/Language/Rho/ Status: ~55% test pass rate, active development area

Characteristics:

  • Functional syntax similar to Python/F#
  • Translates to Pi continuations
  • Recent fixes: if/else, else-if chains, do-while loops
  • Supports: functions, lambdas, loops, comprehensions

Example:

fun factorial(n) {
    if (n <= 1) 1
    else n * factorial(n - 1)
}

for (i in 0..10) {
    print(i)
}

Recent Work (from git log):

  • Enhanced do-while loop support (commit 88138d89)
  • Loop transpilation to Pi (commit 691f62fe)
  • Break/continue statements (commit ccf43e64)
  • Array/map indexing (commit 26471454)
  • Modulo and not-equals operators (commit 2c691745)

Current Issues:

  • Some for-loop syntax variations problematic
  • Complex nested control structures
  • Function scoping edge cases
  • Backtick operations disabled

Tau Language (IDL, Network Code Generation)

Location: Include/KAI/Language/Tau/, Source/Library/Language/Tau/ Status: 91.7% test pass rate (100/109 passing)

Characteristics:

  • Interface Definition Language (IDL)
  • C++-like syntax for defining interfaces
  • Generates proxy/agent code for networking
  • Supports: classes, namespaces, async methods, templates

Example (from tests):

namespace Network {
    interface IChat {
        async void SendMessage(string msg);
        int GetUserCount();
    }
}

Recent Work:

  • 100 comprehensive IDL tests added (commit 718dc23b)
  • Nested namespace and async method fixes (commit 44827fb1)
  • Interface generation tests (commit b2b66487)
  • ChatP2P test conversion (commit 9f12f566)

Current Issues:

  • Some network-related code generation tests failing
  • Related to ENet stub implementation

Build System

CMake Configuration (CMakeLists.txt)

  • C++23 standard required
  • Defaults to Clang/Ninja (faster builds)
  • GCC support via -DBUILD_GCC=ON
  • Build options:
    • KAI_BUILD_TEST_ALL (ON)
    • KAI_BUILD_CORE_TEST (ON)
    • KAI_BUILD_TEST_LANG (ON)
    • KAI_BUILD_TEST_NETWORK (OFF - networking disabled by default)
    • ENABLE_SHELL_SYNTAX (OFF - security feature)

Output Structure:

  • Binaries: Bin/ or build/Bin/
  • Libraries: Same directory
  • Tests: Bin/Test/

Dependencies:

  • Boost (system, filesystem, program_options, date_time, regex)
  • GoogleTest (downloaded if not found)
  • ENet (optional, stubbed by default)

Test Infrastructure

Test Organization:

Test/
├── Core/              # Core system tests (147 tests, 100% pass)
├── Language/
│   ├── TestPi/       # Pi language tests (runs without segfaults, basic pass, advanced fail due to implementation bugs)
│   ├── TestRho/      # Rho language tests (basic expressions work, functions disabled due to implementation issues)
│   └── TestTau/      # Tau language tests (109 tests, 91.7% pass)
└── Console/          # Console tests (20 tests, 25% pass)

Test Execution:

  • ./Scripts/run_all_tests.sh - All tests
  • ./Bin/Test/TestCore - Core tests only
  • ./Bin/Test/TestRho --gtest_filter="*pattern*" - Filtered tests

Test Status Summary (from TEST_SUMMARY.md):

  • Core: Excellent (100%)
  • Pi: Runs without segfaults (basic pass, advanced fail due to implementation bugs)
  • Tau: Good (91.7%)
  • Rho: Basic expressions work, functions disabled due to implementation issues - active development focus
  • Console: Needs work (25%)

Current Work in Progress

Modified Files (from git status):

  1. ExecutorPerform.cpp - Executor operation implementations
  2. RhoParser.cpp - Rho language parsing
  3. RhoTranslate.cpp / RhoTranslator.cpp - Rho to Pi translation
  4. Test files - Various language tests

New Test Files (untracked):

  • RhoBreakContinueTests.cpp - Break/continue statement tests
  • RhoComprehensiveTests.cpp - Comprehensive Rho feature tests
  • PiSuspendResumeReplaceTests.cpp - Pi continuation management

Recent Commit Themes:

  1. Tau networking (a1057674, 718dc23b, 44827fb1) - IDL code generation
  2. Rho control structures (f2809723, 691f62fe, ccf43e64) - loops, break/continue
  3. Rho operators (2c691745, 26471454) - modulo, indexing, comparisons
  4. Build system (f8429ff4, b23e9816) - C++23 upgrade, log directory fixes

Key Technical Details

Object References

  • Use Handle not raw pointers
  • Object wraps Handle + Registry reference
  • Pointer<T> for type-safe access
  • Storage<T> for actual object storage
  • Value<T> combines Object with type safety

Continuation Model

  • Continuations are sequences of operations/objects
  • Stack-based execution with PC (program counter)
  • Control flow via continuation swapping
  • If/else implemented as conditional continuation execution

Translation Pipeline Example (Rho → Pi):

Rho: if (x > 5) 42 else 99

Parse → AST:
  IfThenElse
    ├─ condition: x > 5
    ├─ then: 42
    └─ else: 99

Translate → Pi:
  x 5 >           # Condition
  { 42 } { 99 }   # Then/else continuations
  if              # If operation

Execute:
  Push x, push 5, compare
  Push two continuations
  Execute 'if' operation
  Result on stack

Memory Management

  • Registry owns all objects via instances_ map
  • Tri-color GC for automatic cleanup
  • Retained objects: manually managed, not GC'd
  • Pin/Unpin for temporary GC protection
  • Known issue: cycle handling requires manual intervention

Known Issues and TODOs

High Priority (from TODO.md)

  1. GC cycles - Registry.cpp:201 HACK
  2. Rho for-loops - Semicolon syntax parsing issues
  3. Shell operations - All backtick tests failing (security disabled)
  4. Console features - Tab completion, history (25% pass rate)

Language-Specific Issues

Pi:

  • Code blocks { } → continuation translation incomplete
  • begin/until operations not implemented
  • Math ops: sin, cos, pow, sqrt, abs missing
  • String ops: at, slice, toint, tofloat missing

Rho:

  • For-loop syntax variations
  • Nested scope resolution with & operation
  • Some recursive patterns
  • Complex control flow combinations

Tau:

  • Network code generation tests (related to ENet stub)
  • Some async pattern tests

Build System

  • TestCore linking errors with Executor library
  • clang-tidy warnings in ImGui
  • Duplicate library inclusions

Development Patterns

Adding a New Operation

  1. Define in Operation::Type enum (Include/KAI/Executor/Operation.h)
  2. Implement in Executor::Perform() (Source/Library/Executor/Source/ExecutorPerform.cpp)
  3. Add translator support if language-specific
  4. Write tests in appropriate Test/Language directory

Adding Language Features

  1. Update lexer (e.g., RhoLexer) for new tokens
  2. Update parser (e.g., RhoParser) for syntax
  3. Update translator (e.g., RhoTranslator) to generate Pi
  4. Add comprehensive tests
  5. Update documentation

Testing Strategy

  • Unit tests for individual operations
  • Integration tests for language features
  • Comprehensive test suites for each language
  • Use --gtest_filter for focused testing
  • Trace levels for debugging (KAI_TRACE)

Project Statistics

Lines of Code (estimate from structure):

  • Core: ~15K lines
  • Languages: ~30K lines (Pi, Rho, Tau, Lisp, Hlsl)
  • Tests: ~50K lines
  • Apps: ~5K lines
  • Total: ~100K+ lines

Test Count:

  • Total: ~1,048 tests
  • Core: 147 tests
  • Languages: ~872 tests
  • Console: 20 tests
  • Other: ~9 tests

File Structure:

  • Include/KAI/: ~200 header files
  • Source/: ~150 implementation files
  • Test/: ~150 test files
  • Scripts/: ~30 automation scripts

Performance Characteristics

  • Stack-based execution (fast for Pi operations)
  • GC overhead (tri-color mark-and-sweep)
  • Handle indirection (lookup in Registry)
  • Continuation creation cost (object allocation)
  • Translation overhead (Rho/Tau → Pi)

Security Features

  • ENABLE_SHELL_SYNTAX - Disabled by default
  • Backtick operations blocked in production
  • Sandboxed execution environment
  • No direct file system access from languages
  • Network features optional (KAI_BUILD_NETWORK)

Future Directions (inferred from codebase)

  1. Complete Rho implementation - Focus on remaining ~45% failing tests
  2. Network layer - Complete Tau code generation for distributed systems
  3. Performance optimization - Reduce continuation allocation overhead
  4. Documentation - Complete language guides
  5. Reflection system - Enhanced type traits and property access
  6. Console improvements - Tab completion, history, better UX

Critical Files Reference

Core System:

  • Registry: Include/KAI/Core/Registry.h, Source/Library/Core/Source/Registry.cpp
  • Object: Include/KAI/Core/Object.h
  • Executor: Include/KAI/Executor/Executor.h
  • Operation: Include/KAI/Executor/Operation.h

Languages:

  • Pi: Include/KAI/Language/Pi/, Source/Library/Language/Pi/
  • Rho: Include/KAI/Language/Rho/, Source/Library/Language/Rho/
  • Tau: Include/KAI/Language/Tau/, Source/Library/Language/Tau/

Key Implementation Files:

  • ExecutorPerform.cpp: Operation implementations
  • RhoTranslator.cpp: Rho to Pi translation
  • RhoParser.cpp: Rho syntax parsing
  • TauParser.cpp: Tau IDL parsing

Test Entry Points:

  • Test/Language/TestPi/TestMain.cpp
  • Test/Language/TestRho/TestMain.cpp
  • Test/Language/TestTau/TestMain.cpp
  • Test/Core/TestMain.cpp

Build and Development Workflow

Initial Setup:

cmake -B build -G Ninja
cmake --build build --config Release

Common Commands:

./Scripts/run_all_tests.sh          # All tests
./Scripts/run_rho_tests.sh          # Rho tests only
./Bin/Test/TestRho --gtest_filter="RhoControlStructuresTestsFixed.*"

Development Cycle:

  1. Modify code in Source/Library or Include/KAI
  2. Build: cmake --build build
  3. Test: ./Bin/Test/TestRho (or specific test)
  4. Debug: Use KAI_TRACE() and trace levels
  5. Iterate

Conclusion

KAI is a mature multi-language runtime with a solid core (100% tests passing) and active development on higher-level language features. The project demonstrates sophisticated language implementation techniques including continuation-based control flow, tri-color garbage collection, and multi-language interoperability. Current focus is on stabilizing the Rho language implementation and expanding Tau's network code generation capabilities.

Primary Development Areas:

  1. Rho language stabilization (55% → 90%+ pass rate)
  2. Tau network code generation completion
  3. Performance optimization
  4. Documentation and examples

Strengths:

  • Solid core object system
  • Working garbage collection
  • Multiple language support
  • Active development and testing

Challenges:

  • Rho language edge cases
  • Network layer completion
  • Console UX improvements
  • Documentation completeness