Add Comprehensive Integration Test Suite with Benchmarks and Snapshot Testing

[unclesp1d3r]

Overview

Add comprehensive integration test suite with performance benchmarks and snapshot testing to ensure the stringy binary analyzer works correctly across different binary formats, platforms, and edge cases.

Context

The stringy project aims to be a smarter alternative to the strings command with format-specific knowledge. To ensure reliability and prevent regressions:

• Integration tests validate end-to-end CLI behavior and binary parsing across formats (ELF, PE, Mach-O, etc.)
• Benchmarks track performance characteristics to prevent performance regressions
• Snapshot tests verify output consistency across versions and ensure changes to output formatting are intentional
• Cross-platform validation ensures the tool works correctly on Linux, macOS, and Windows

Proposed Solution

1. Integration Test Suite Structure

Create tests/integration/ directory with the following structure for organizing integration tests by format and functionality.

2. Benchmark Suite Expansion

Extend benches/ with comprehensive performance tests covering ELF, PE, Mach-O formats, large file handling, and core string extraction algorithms.

3. Key Test Scenarios

CLI Integration Tests:

• Help text and version display
• Output format options (JSON, plain text)
• Error handling for invalid inputs
• File path resolution
• Encoding detection

Format-Specific Tests:

• String extraction from different sections (.rodata, .data, etc.)
• Handling of stripped vs. non-stripped binaries
• Unicode and multi-byte character support
• Malformed binary handling

Snapshot Tests (using insta):

• JSON output format stability
• Human-readable output formatting
• Error message consistency

Performance Benchmarks:

• Small file processing (< 1MB)
• Medium file processing (1-10MB)
• Large file processing (> 10MB)
• Different string density ratios

4. Implementation Steps

1. Phase 1: Basic Integration Tests

   • Set up test fixtures with sample binaries
   • Implement CLI behavior tests
   • Add helper utilities for running CLI and capturing output

2. Phase 2: Format-Specific Tests

   • Add ELF-specific integration tests
   • Add PE-specific integration tests
   • Add Mach-O-specific integration tests

3. Phase 3: Snapshot Testing

   • Integrate insta for output verification
   • Create baseline snapshots for all test cases
   • Add CI validation for snapshot changes

4. Phase 4: Benchmark Suite

   • Expand criterion benchmarks for all formats
   • Add comparative benchmarks (vs. GNU strings)
   • Set up performance regression detection in CI

5. Phase 5: Cross-Platform Validation

   • Run tests on Linux, macOS, and Windows in CI
   • Add platform-specific edge case handling
   • Document platform differences

Acceptance Criteria

• Integration test coverage for CLI functionality
• Format-specific tests for ELF, PE, and Mach-O binaries
• Snapshot tests for all output formats using insta
• Comprehensive benchmark suite using criterion
• Tests pass on Linux, macOS, and Windows
• CI pipeline includes integration tests and benchmarks
• Test fixtures documented and reproducible
• Performance baseline established for regression detection
• Code coverage > 80% for core parsing logic

Dependencies

• Blocked by: Basic Test Infrastructure (must be completed first)
• Testing frameworks: criterion (already added), insta (already added), tempfile (already added)
• Test fixtures: Need sample binaries for different formats

Technical Notes

• Use tempfile for temporary test file creation
• Leverage assert_cmd or similar for CLI testing
• Consider using cargo-nextest for faster test execution
• Document how to update snapshots with cargo insta review
• Set up separate CI job for benchmarks to avoid noisy results

Task-ID

stringy-analyzer/comprehensive-integration-tests

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

[traycerai]

Warning

Rate limit exceeded. Please try after 1182 seconds.