Implement Encoding Confidence Scoring and Integrate Section Weights into Ranking System

[unclesp1d3r]

Summary

Implement encoding confidence scoring and integrate existing section weights into the string ranking system to support the overall string relevance scoring algorithm.

Current State

✅ Already Implemented:

• Section weight calculation based on SectionType (PE, ELF, Mach-O)
• Section classification logic
• Unit tests for section weight calculation
• SectionInfo.weight field populated during binary parsing

❌ Not Yet Implemented:

• Encoding confidence scoring algorithm
• String relevance scoring integration
• Population of FoundString.score field

Requirements

From the architecture design (concept.md), the ranking algorithm is:

Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty

This issue addresses the first two components: SectionWeight (integration) and EncodingConfidence (implementation).

5.1: Encoding Confidence Scoring

Implement confidence scoring for each detected encoding type based on:

1. Character validity: Percentage of valid characters in detected encoding
2. Printability: Ratio of printable to non-printable characters
3. Null termination: Proper null termination for C-style strings
4. Byte alignment: Correct alignment for multi-byte encodings (UTF-16)
5. Control character ratio: Lower scores for high control character density

Proposed Confidence Scale: 0.0 to 10.0

• 9.0-10.0: High confidence (clean ASCII/UTF-8, properly aligned UTF-16)
• 7.0-8.9: Medium confidence (some irregularities but valid)
• 5.0-6.9: Low confidence (marginal validity, needs context)
• < 5.0: Very low confidence (likely noise)

5.5: Section Weight Integration

Integrate existing section weights into the string scoring system:

1. Add section weight to FoundString scoring calculation
2. Apply section weight as a multiplier or additive component
3. Ensure strings from high-weight sections (e.g., .rodata, __cstring) rank higher

Proposed Implementation

1. Encoding Confidence Module (src/scoring/encoding_confidence.rs)

pub fn calculate_encoding_confidence(
    text: &str,
    encoding: Encoding,
    raw_bytes: &[u8],
) -> f32 {
    match encoding {
        Encoding::Ascii => score_ascii_confidence(text, raw_bytes),
        Encoding::Utf8 => score_utf8_confidence(text, raw_bytes),
        Encoding::Utf16Le | Encoding::Utf16Be => score_utf16_confidence(text, raw_bytes, encoding),
    }
}

fn score_ascii_confidence(text: &str, raw_bytes: &[u8]) -> f32 {
    let printable_count = text.chars().filter(|c| c.is_ascii_graphic() || c.is_ascii_whitespace()).count();
    let total_chars = text.chars().count();
    let printable_ratio = printable_count as f32 / total_chars.max(1) as f32;
    
    // Null-terminated strings get a bonus
    let null_terminated = raw_bytes.last() == Some(&0);
    let base_score = printable_ratio * 10.0;
    
    if null_terminated { base_score.min(10.0) } else { (base_score * 0.95).min(9.5) }
}

2. String Scoring Integration (src/scoring/mod.rs)

pub fn calculate_string_score(
    string: &FoundString,
    section_weight: f32,
    encoding_confidence: f32,
) -> i32 {
    let base_score = section_weight + encoding_confidence;
    // Round to integer for storage
    base_score.round() as i32
}

3. Update String Extraction

Modify string extraction logic to:

1. Calculate encoding confidence when creating FoundString
2. Look up section weight from ContainerInfo
3. Compute and assign final score

Testing Requirements

Unit Tests

• test_ascii_confidence_scoring: Various ASCII string qualities
• test_utf8_confidence_scoring: Valid/invalid UTF-8 sequences
• test_utf16_confidence_scoring: Aligned/misaligned UTF-16 data
• test_section_weight_integration: Score calculation with different section weights

Integration Tests

• Extract strings from real binaries and verify scoring
• High-weight sections should produce higher-scored strings
• Clean encodings should score higher than noisy ones

Acceptance Criteria

• Encoding confidence scoring implemented for all encoding types
• Section weight integrated into string scoring
• FoundString.score field populated during extraction
• Unit tests achieve >90% coverage for scoring logic
• Integration tests verify end-to-end scoring behavior
• Documentation updated with scoring algorithm details

Dependencies

• Blocked by: Ranking System Foundation
• Blocks: Semantic Tagging (#TBD), Noise Penalty Calculation (#TBD)

References

• Architecture: concept.md - Ranking Algorithm section
• Related code: src/container/{pe,elf,macho}.rs - Section weight calculation
• Requirements: 5.1 (Encoding Confidence), 5.5 (Section Weight Integration)

---

Task-ID: stringy-analyzer/section-weight-scoring

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# Fetch the current issue details
echo "=== Current Issue Details ==="
gh issue view 22 --json title,body,labels,assignees,milestone,state

echo -e "\n=== Repository Structure ==="
# Get an overview of the repository structure
fd -t f -e rs -e toml -e md --max-depth 2

echo -e "\n=== Search for SectionType ==="
# Search for SectionType definition and usage
rg -n "enum SectionType|struct SectionType|SectionType" --type rust -A 5 -B 2

echo -e "\n=== Search for section weight and scoring ==="
# Search for section weight and scoring related code
rg -n "section.*weight|encoding.*confidence|confidence.*score" --type rust -i -A 3 -B 2

Length of output: 50607

---

🏁 Script executed:

#!/bin/bash

# Search for Encoding definition and usage
echo "=== Encoding Type Definition ==="
rg -n "enum Encoding|struct Encoding|Encoding::" --type rust -A 10 -B 2

echo -e "\n=== FoundString Definition ==="
# Look at FoundString to understand the current structure
rg -n "struct FoundString" --type rust -A 15 -B 2

echo -e "\n=== Search for ranking or scoring systems ==="
# Look for any ranking/scoring infrastructure
rg -n "rank|score|priority" --type rust -i -C 2 | head -80

Length of output: 4211

---

🏁 Script executed:

#!/bin/bash

# Find string extraction/scanning code
echo "=== String Extraction/Scanner Files ==="
fd -e rs | grep -E "scan|extract|string" | head -20

echo -e "\n=== String Extraction Implementation ==="
rg -n "fn.*extract.*string|fn.*scan|fn.*detect.*encoding" --type rust -i -A 5 -B 2 | head -100

echo -e "\n=== Look for how score field is currently used ==="
rg -n "score:" --type rust -C 3 | head -60

Length of output: 1680

---

🏁 Script executed:

#!/bin/bash

# Read the extraction module
echo "=== Extraction Module Contents ==="
cat src/extraction/mod.rs

echo -e "\n=== Check for any confidence-related code ==="
rg -n "confidence" --type rust -C 3

echo -e "\n=== Look at requirements mentioned in docs or concept ==="
cat concept.md | head -200

Length of output: 6249

---

🏁 Script executed:

#!/bin/bash

# Update the issue with enhanced content
gh issue edit 22 \
  --title "Implement Encoding Confidence Scoring and Integrate Section Weights into Ranking System" \
  --body "## Summary

Implement encoding confidence scoring and integrate existing section weights into the string ranking system to support the overall string relevance scoring algorithm.

## Current State

✅ **Already Implemented:**
- Section weight calculation based on \`SectionType\` (PE, ELF, Mach-O)
- Section classification logic
- Unit tests for section weight calculation
- \`SectionInfo.weight\` field populated during binary parsing

❌ **Not Yet Implemented:**
- Encoding confidence scoring algorithm
- String relevance scoring integration
- Population of \`FoundString.score\` field

## Requirements

From the architecture design (concept.md), the ranking algorithm is:

\`\`\`
Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty
\`\`\`

This issue addresses the first two components: **SectionWeight** (integration) and **EncodingConfidence** (implementation).

### 5.1: Encoding Confidence Scoring

Implement confidence scoring for each detected encoding type based on:

1. **Character validity**: Percentage of valid characters in detected encoding
2. **Printability**: Ratio of printable to non-printable characters  
3. **Null termination**: Proper null termination for C-style strings
4. **Byte alignment**: Correct alignment for multi-byte encodings (UTF-16)
5. **Control character ratio**: Lower scores for high control character density

**Proposed Confidence Scale:** 0.0 to 10.0
- **9.0-10.0**: High confidence (clean ASCII/UTF-8, properly aligned UTF-16)
- **7.0-8.9**: Medium confidence (some irregularities but valid)
- **5.0-6.9**: Low confidence (marginal validity, needs context)
- **< 5.0**: Very low confidence (likely noise)

### 5.5: Section Weight Integration

Integrate existing section weights into the string scoring system:

1. Add section weight to \`FoundString\` scoring calculation
2. Apply section weight as a multiplier or additive component
3. Ensure strings from high-weight sections (e.g., \`.rodata\`, \`__cstring\`) rank higher

## Proposed Implementation

### 1. Encoding Confidence Module (\`src/scoring/encoding_confidence.rs\`)

\`\`\`rust
pub fn calculate_encoding_confidence(
    text: &str,
    encoding: Encoding,
    raw_bytes: &[u8],
) -> f32 {
    match encoding {
        Encoding::Ascii => score_ascii_confidence(text, raw_bytes),
        Encoding::Utf8 => score_utf8_confidence(text, raw_bytes),
        Encoding::Utf16Le | Encoding::Utf16Be => score_utf16_confidence(text, raw_bytes, encoding),
    }
}

fn score_ascii_confidence(text: &str, raw_bytes: &[u8]) -> f32 {
    let printable_count = text.chars().filter(|c| c.is_ascii_graphic() || c.is_ascii_whitespace()).count();
    let total_chars = text.chars().count();
    let printable_ratio = printable_count as f32 / total_chars.max(1) as f32;
    
    // Null-terminated strings get a bonus
    let null_terminated = raw_bytes.last() == Some(&0);
    let base_score = printable_ratio * 10.0;
    
    if null_terminated { base_score.min(10.0) } else { (base_score * 0.95).min(9.5) }
}
\`\`\`

### 2. String Scoring Integration (\`src/scoring/mod.rs\`)

\`\`\`rust
pub fn calculate_string_score(
    string: &FoundString,
    section_weight: f32,
    encoding_confidence: f32,
) -> i32 {
    let base_score = section_weight + encoding_confidence;
    // Round to integer for storage
    base_score.round() as i32
}
\`\`\`

### 3. Update String Extraction

Modify string extraction logic to:
1. Calculate encoding confidence when creating \`FoundString\`
2. Look up section weight from \`ContainerInfo\`
3. Compute and assign final score

## Testing Requirements

### Unit Tests

- \`test_ascii_confidence_scoring\`: Various ASCII string qualities
- \`test_utf8_confidence_scoring\`: Valid/invalid UTF-8 sequences
- \`test_utf16_confidence_scoring\`: Aligned/misaligned UTF-16 data
- \`test_section_weight_integration\`: Score calculation with different section weights

### Integration Tests

- Extract strings from real binaries and verify scoring
- High-weight sections should produce higher-scored strings
- Clean encodings should score higher than noisy ones

## Acceptance Criteria

- [ ] Encoding confidence scoring implemented for all encoding types
- [ ] Section weight integrated into string scoring
- [ ] \`FoundString.score\` field populated during extraction
- [ ] Unit tests achieve >90% coverage for scoring logic
- [ ] Integration tests verify end-to-end scoring behavior
- [ ] Documentation updated with scoring algorithm details

## Dependencies

- **Blocked by:** Ranking System Foundation
- **Blocks:** Semantic Tagging (#TBD), Noise Penalty Calculation (#TBD)

## References

- Architecture: \`concept.md\` - Ranking Algorithm section
- Related code: \`src/container/{pe,elf,macho}.rs\` - Section weight calculation
- Requirements: 5.1 (Encoding Confidence), 5.5 (Section Weight Integration)

---

**Task-ID:** stringy-analyzer/section-weight-scoring"

echo "Issue updated successfully!"

Length of output: 89

---

🏁 Script executed:

#!/bin/bash

# List available labels in the repository
echo "=== Available Repository Labels ==="
gh label list --limit 100

echo -e "\n=== Current Issue State ==="
gh issue view 22 --json title,labels,assignees,milestone | jq '.'

Length of output: 2745

---

🏁 Script executed:

#!/bin/bash

# Add MVP and priority:medium labels
gh issue edit 22 --add-label "MVP,priority:medium"

echo -e "\n=== Updated Issue State ==="
gh issue view 22 --json title,labels,url | jq '.'

echo -e "\n✅ Issue enhancement complete!"

Length of output: 1401

---

@unclesp1d3r I've successfully enhanced issue #22 with comprehensive context and a detailed proposed solution!

What I Changed

📝 Title Updated

• Before: "Section Weight Scoring"
• After: "Implement Encoding Confidence Scoring and Integrate Section Weights into Ranking System"

The new title accurately reflects that section weights are already implemented and the real work is encoding confidence scoring plus integration.

🔍 Key Findings

After analyzing the codebase, I discovered:

✅ Already Implemented:

• Section weight calculation for PE, ELF, and Mach-O formats
• Complete unit tests for weight calculation
• Section classification based on SectionType

❌ Still Needed:

• Encoding confidence scoring algorithm
• String relevance scoring integration
• Population of FoundString.score field

📋 Enhanced Issue Body

Added:

1. Current State section clearly showing what's done vs. what's needed
2. Detailed requirements for encoding confidence (5.1) and section weight integration (5.5)
3. Proposed implementation with code examples for:
   • Encoding confidence module (src/scoring/encoding_confidence.rs)
   • String scoring integration (src/scoring/mod.rs)
   • Confidence scale: 0.0-10.0 based on character validity, printability, null termination, etc.
4. Testing requirements with specific test cases
5. Acceptance criteria checklist
6. Dependencies and references to related code and documentation

🏷️ Labels Added

• MVP - Since this is in the v0.1 milestone
• priority:medium - Core ranking feature, but blocked by foundation work

🔗 View the Enhanced Issue

#22

The issue now provides a clear roadmap for implementing the ranking system's scoring components with concrete implementation guidance and acceptance criteria.

[traycerai]

Plan

Observations

The codebase has a solid foundation with section weight calculation already implemented across all three container parsers (ELF, PE, Mach-O). The SectionInfo.weight field is populated with values ranging from 1.0 to 10.0 based on section type and name. The FoundString structure has a score: i32 field that is currently unpopulated. The extraction module (src/extraction/mod.rs) is essentially empty, providing a clean slate for implementing the scoring integration. The architecture document (concept.md) defines the ranking formula as Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty, and this issue focuses on the first two components.

Approach

Implement encoding confidence scoring based on character validity, printability, null termination, byte alignment, and control character ratios. Integrate existing section weights from SectionInfo into the string scoring calculation. Create a new scoring module with functions for calculating encoding confidence (0.0-10.0 scale) and combining it with section weights to populate FoundString.score during string extraction.

Reasoning

I explored the repository structure, examined the core data structures in src/types.rs, reviewed all three container parsers to understand the existing section weight implementation, and studied the architecture documentation to understand the overall ranking system design. The section weight calculation is fully implemented with comprehensive unit tests, while the scoring and extraction modules are ready for implementation.

Proposed File Changes

📄 src/scoring/mod.rs ^(NEW) 🔗

References

• 📄 src/types.rs
• 📄 src/container/mod.rs
• 📄 src/container/elf.rs
• 📄 src/scoring/encoding_confidence.rs ^(NEW)

Create the main scoring module that exports the public API for string scoring. This module should:

• Define and export the calculate_encoding_confidence function that takes text, encoding type, and raw bytes as parameters and returns a confidence score (f32) between 0.0 and 10.0
• Define and export the calculate_string_score function that combines section weight and encoding confidence to produce the final score (i32)
• Re-export the encoding-specific scoring functions from the submodules for easier access
• Include module documentation explaining the scoring algorithm and scale

The module should serve as the main entry point for all scoring operations, similar to how src/container/mod.rs works for container parsing.
Create unit tests for the encoding confidence scoring module:

ASCII confidence tests:

• test_ascii_confidence_high_quality - Test clean ASCII string ("Hello World") should score 9.0-10.0
• test_ascii_confidence_with_null_termination - Test null-terminated ASCII string should get bonus
• test_ascii_confidence_with_control_chars - Test ASCII with control characters should score lower (7.0-8.0)
• test_ascii_confidence_low_printable_ratio - Test ASCII with many non-printable chars should score low (< 5.0)

UTF-8 confidence tests:

• test_utf8_confidence_valid_unicode - Test valid UTF-8 with Unicode characters should score high
• test_utf8_confidence_mixed_quality - Test UTF-8 with some invalid sequences should score medium

UTF-16 confidence tests:

• test_utf16le_confidence_aligned - Test properly aligned UTF-16LE string should score high
• test_utf16le_confidence_misaligned - Test misaligned UTF-16LE should score lower
• test_utf16le_confidence_null_terminated - Test null-terminated UTF-16LE (0x00 0x00) should get bonus
• test_utf16be_confidence - Test UTF-16BE encoding confidence

Integration tests:

• test_section_weight_integration - Test that section weight is properly added to encoding confidence
• test_score_calculation_range - Test that final scores are in expected range (0-20)
• test_score_rounding - Test that scores are properly rounded to integers

Edge cases:

• test_empty_string_confidence - Test handling of empty strings
• test_single_char_confidence - Test very short strings
• test_very_long_string_confidence - Test long strings

Tests should use the #[cfg(test)] module pattern and follow the existing test structure seen in src/container/elf.rs and other parser tests.

📄 src/scoring/encoding_confidence.rs ^(NEW) 🔗

References

• 📄 src/types.rs

Implement the encoding confidence scoring algorithm with separate functions for each encoding type:

Main function:

• calculate_encoding_confidence(text: &str, encoding: Encoding, raw_bytes: &[u8]) -> f32 - Dispatcher that routes to encoding-specific functions based on the encoding type

ASCII confidence scoring:

• score_ascii_confidence(text: &str, raw_bytes: &[u8]) -> f32 - Calculate confidence for ASCII strings
  • Calculate printable character ratio (using is_ascii_graphic() and is_ascii_whitespace())
  • Check for null termination (bonus if raw_bytes.last() == Some(&0))
  • Calculate control character ratio and apply penalty
  • Base score: printable_ratio * 10.0
  • Apply null termination bonus (0.5 points) if present
  • Apply control character penalty if ratio > 0.1
  • Clamp result to 0.0-10.0 range

UTF-8 confidence scoring:

• score_utf8_confidence(text: &str, raw_bytes: &[u8]) -> f32 - Calculate confidence for UTF-8 strings
  • Similar to ASCII but with UTF-8 specific validation
  • Check for valid UTF-8 sequences
  • Calculate printable character ratio (using Unicode character properties)
  • Check null termination
  • Apply similar scoring logic as ASCII

UTF-16 confidence scoring:

• score_utf16_confidence(text: &str, raw_bytes: &[u8], encoding: Encoding) -> f32 - Calculate confidence for UTF-16 strings
  • Verify byte alignment (even length for UTF-16)
  • Check for proper null termination (0x00 0x00 pattern)
  • Calculate valid character ratio by examining byte pairs
  • For UTF-16LE: check that high bytes are mostly zero for ASCII-range characters
  • For UTF-16BE: check that low bytes are mostly zero for ASCII-range characters
  • Apply alignment penalty if misaligned
  • Base score: valid_char_ratio * 10.0
  • Apply null termination bonus
  • Clamp to 0.0-10.0 range

Helper functions:

• calculate_printable_ratio(text: &str) -> f32 - Calculate ratio of printable to total characters
• calculate_control_char_ratio(text: &str) -> f32 - Calculate ratio of control characters
• is_null_terminated(raw_bytes: &[u8], encoding: Encoding) -> bool - Check for proper null termination based on encoding
• check_utf16_alignment(raw_bytes: &[u8]) -> bool - Verify UTF-16 byte alignment

All functions should include comprehensive documentation with examples of score ranges for different string qualities.

📄 src/scoring/string_score.rs ^(NEW) 🔗

References

• 📄 src/types.rs

Implement the string score calculation that integrates section weight with encoding confidence:

Main function:

• calculate_string_score(section_weight: f32, encoding_confidence: f32) -> i32 - Combine section weight and encoding confidence into final score
  • Add section_weight + encoding_confidence
  • Round to nearest integer using .round() as i32
  • Clamp to reasonable range (0-20 based on max section weight of 10.0 + max confidence of 10.0)
  • Return the final score

Helper function:

• calculate_score_with_context(string: &FoundString, section_weight: f32, encoding_confidence: f32) -> i32 - Alternative function that takes a FoundString and calculates score with additional context
  • This function can be used when more context is available
  • Currently just calls calculate_string_score but provides extension point for future enhancements (semantic boost, noise penalty)

The module should be simple and focused, as it's primarily combining the two components. Future enhancements (semantic boost, noise penalty) will be added in separate issues as mentioned in the GitHub issue dependencies.

📄 src/lib.rs ^(MODIFY) 🔗

Add the new scoring module to the library's module tree:

• Add pub mod scoring; after the existing module declarations (around line 48, after pub mod types;)
• Add re-exports for commonly used scoring functions in the re-export section (around line 51-54):
  • Re-export scoring::calculate_encoding_confidence
  • Re-export scoring::calculate_string_score

Update the module documentation comment (around lines 38-42) to reflect that scoring functionality is now implemented:

• Change the scoring module status from "🚧 Framework ready" to "✅ Complete" or similar
• Add a brief note about the scoring capabilities

This follows the same pattern used for other modules in the library.

📄 src/extraction/mod.rs ^(MODIFY) 🔗

References

• 📄 src/types.rs
• 📄 src/scoring/mod.rs ^(NEW)
• 📄 docs/src/string-extraction.md

Implement the string extraction logic that integrates scoring. Since this file is currently empty (just a comment), this will be a substantial addition:

Core extraction function:

• extract_strings_from_section(section: &SectionInfo, data: &[u8], config: &ExtractionConfig) -> Vec<FoundString> - Extract strings from a section with scoring
  • Scan the section data for strings based on encoding types
  • For each found string, calculate encoding confidence using scoring::calculate_encoding_confidence
  • Calculate final score using scoring::calculate_string_score with section.weight and encoding confidence
  • Populate all fields of FoundString including the score field
  • Return vector of FoundString instances

Helper functions:

• extract_ascii_strings(data: &[u8], offset: u64, min_len: usize) -> Vec<RawString> - Extract ASCII/UTF-8 strings from raw bytes
• extract_utf16_strings(data: &[u8], offset: u64, min_len: usize, endianness: Endianness) -> Vec<RawString> - Extract UTF-16 strings
• create_found_string(raw: RawString, section: &SectionInfo, encoding_confidence: f32) -> FoundString - Convert RawString to FoundString with scoring

Configuration structure:

• ExtractionConfig struct with fields for min_ascii_len, min_utf16_len, enabled_encodings, etc.

Internal types:

• RawString struct to hold intermediate extraction results before scoring (text, offset, encoding, raw_bytes)

The extraction logic should integrate with the scoring module by:

1. Extracting raw strings from section data
2. Calculating encoding confidence for each string using scoring::calculate_encoding_confidence
3. Using the section's weight field from SectionInfo
4. Calling scoring::calculate_string_score to get the final score
5. Populating the FoundString.score field with the calculated value

Reference the extraction algorithms described in docs/src/string-extraction.md for implementation details.

📄 tests/integration_scoring.rs ^(NEW) 🔗

References

• 📄 tests/integration_elf.rs
• 📄 tests/fixtures/README.md

Create integration tests that verify end-to-end scoring behavior with real binary data:

Test structure:

• Use the existing test fixtures in tests/fixtures/ (test_binary_elf, test_binary_macho, test_binary.c)
• Follow the pattern established in tests/integration_elf.rs, tests/integration_pe.rs, and tests/integration_macho.rs

Test cases:

• test_elf_string_scoring - Extract strings from ELF binary and verify scores are populated

  • Parse the test_binary_elf fixture
  • Extract strings from .rodata section (should have high section weight of 10.0)
  • Verify that clean ASCII strings in .rodata have high scores (18-20)
  • Verify that strings in .data section have lower scores due to lower section weight (5.0)

• test_macho_string_scoring - Extract strings from Mach-O binary and verify scores

  • Parse the test_binary_macho fixture
  • Extract strings from __TEXT,__cstring section (weight 10.0)
  • Verify high-quality strings score appropriately

• test_section_weight_affects_score - Verify that section weight directly impacts final score

  • Compare identical strings from different sections
  • String from .rodata (weight 10.0) should score ~10 points higher than same string from .data (weight 5.0)

• test_encoding_confidence_affects_score - Verify that encoding quality impacts score

  • Compare clean vs noisy strings from same section
  • Clean string should score higher than noisy string even in same section

• test_score_field_populated - Verify that FoundString.score field is properly populated

  • Extract strings and verify score field is not zero/default
  • Verify scores are in reasonable range (0-20)

Assertions:

• Verify score field is populated (not 0 or default value)
• Verify scores are within expected range based on section weight + encoding confidence
• Verify relative ordering (high-weight sections produce higher scores)
• Verify score calculation is deterministic (same input produces same score)

Use snapshot testing with insta crate if available (as seen in tests/snapshots/ directory) to capture and verify scoring results.

📄 docs/src/ranking.md ^(MODIFY) 🔗

References

• 📄 src/container/elf.rs
• 📄 src/container/pe.rs
• 📄 src/container/macho.rs
• 📄 src/scoring/encoding_confidence.rs ^(NEW)

Update the ranking documentation to reflect the actual implementation:

Section Weight section (lines 13-57):

• Update the weight values table to match the actual implementation (float values 1.0-10.0 instead of integers 40, 35, etc.)
• Clarify that section weights are calculated per-format in the container parsers
• Update the code example to reflect the actual implementation in src/container/elf.rs, src/container/pe.rs, and src/container/macho.rs
• Note that format-specific bonuses are already built into the section weight calculation

Encoding Confidence section (lines 59-102):

• Update to reflect the 0.0-10.0 scale as implemented
• Document the actual confidence factors:
  • Character validity percentage
  • Printability ratio
  • Null termination bonus
  • Byte alignment for UTF-16
  • Control character ratio penalty
• Update the code example to match the actual function signatures in src/scoring/encoding_confidence.rs
• Provide specific score ranges:
  • 9.0-10.0: High confidence (clean ASCII/UTF-8, properly aligned UTF-16)
  • 7.0-8.9: Medium confidence (some irregularities but valid)
  • 5.0-6.9: Low confidence (marginal validity)
  • < 5.0: Very low confidence (likely noise)

Complete Scoring Implementation section (lines 257-286):

• Update to reflect the current implementation status
• Note that this issue implements SectionWeight + EncodingConfidence
• Document that SemanticBoost and NoisePenalty are planned for future issues
• Update the score range table to reflect the current 0-20 range (section weight 0-10 + encoding confidence 0-10)

Add Implementation Status section:

• Add a new section documenting what's implemented vs planned
• ✅ Section Weight Calculation (implemented in container parsers)
• ✅ Encoding Confidence Scoring (implemented in this issue)
• 🚧 Semantic Boost (planned)
• 🚧 Noise Penalty (planned)

Reference the actual implementation files for developers who want to understand the details.

📄 README.md ^(MODIFY) 🔗

References

• 📄 src/types.rs

Update the README to reflect the new scoring capabilities:

Features section:

• Add a bullet point about intelligent string ranking based on section weight and encoding confidence
• Mention that strings are scored on a 0-20 scale combining section importance and encoding quality

Status/Roadmap section (if present):

• Update to show that encoding confidence scoring and section weight integration are now implemented
• Mark these features as complete in any roadmap or status indicators

Usage examples (if present):

• If there are code examples, update them to show that FoundString now includes populated score fields
• Add a note that strings can be sorted by score for relevance-based analysis

Keep changes minimal and focused on documenting the new scoring functionality without rewriting the entire README.

Diagram

sequenceDiagram
    participant Binary as Binary File
    participant Parser as Container Parser
    participant Extractor as String Extractor
    participant Scorer as Scoring Module
    participant Result as FoundString
    
    Binary->>Parser: Parse binary
    Parser->>Parser: Calculate section weights
    Note over Parser: ELF: .rodata=10.0<br/>PE: .rdata=10.0<br/>Mach-O: __cstring=10.0
    Parser-->>Extractor: SectionInfo with weight
    
    Extractor->>Extractor: Scan section for strings
    Extractor->>Extractor: Extract raw strings
    
    loop For each raw string
        Extractor->>Scorer: calculate_encoding_confidence(text, encoding, bytes)
        Scorer->>Scorer: Check printability ratio
        Scorer->>Scorer: Check null termination
        Scorer->>Scorer: Check byte alignment (UTF-16)
        Scorer->>Scorer: Check control char ratio
        Scorer-->>Extractor: confidence (0.0-10.0)
        
        Extractor->>Scorer: calculate_string_score(section_weight, confidence)
        Scorer->>Scorer: Add weights: section + confidence
        Scorer->>Scorer: Round to integer
        Scorer-->>Extractor: final_score (0-20)
        
        Extractor->>Result: Create FoundString with score
    end
    
    Result-->>Binary: Scored strings ready for analysis

Loading

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a solid foundation with section weight calculation already implemented across all three container parsers (ELF, PE, Mach-O). The `SectionInfo.weight` field is populated with values ranging from 1.0 to 10.0 based on section type and name. The `FoundString` structure has a `score: i32` field that is currently unpopulated. The extraction module (`src/extraction/mod.rs`) is essentially empty, providing a clean slate for implementing the scoring integration. The architecture document (`concept.md`) defines the ranking formula as `Score = SectionWeight + EncodingConfidence + SemanticBoost - NoisePenalty`, and this issue focuses on the first two components.

### Approach

Implement encoding confidence scoring based on character validity, printability, null termination, byte alignment, and control character ratios. Integrate existing section weights from `SectionInfo` into the string scoring calculation. Create a new `scoring` module with functions for calculating encoding confidence (0.0-10.0 scale) and combining it with section weights to populate `FoundString.score` during string extraction.

### Reasoning

I explored the repository structure, examined the core data structures in `src/types.rs`, reviewed all three container parsers to understand the existing section weight implementation, and studied the architecture documentation to understand the overall ranking system design. The section weight calculation is fully implemented with comprehensive unit tests, while the scoring and extraction modules are ready for implementation.

## Mermaid Diagram

sequenceDiagram
    participant Binary as Binary File
    participant Parser as Container Parser
    participant Extractor as String Extractor
    participant Scorer as Scoring Module
    participant Result as FoundString
    
    Binary->>Parser: Parse binary
    Parser->>Parser: Calculate section weights
    Note over Parser: ELF: .rodata=10.0<br/>PE: .rdata=10.0<br/>Mach-O: __cstring=10.0
    Parser-->>Extractor: SectionInfo with weight
    
    Extractor->>Extractor: Scan section for strings
    Extractor->>Extractor: Extract raw strings
    
    loop For each raw string
        Extractor->>Scorer: calculate_encoding_confidence(text, encoding, bytes)
        Scorer->>Scorer: Check printability ratio
        Scorer->>Scorer: Check null termination
        Scorer->>Scorer: Check byte alignment (UTF-16)
        Scorer->>Scorer: Check control char ratio
        Scorer-->>Extractor: confidence (0.0-10.0)
        
        Extractor->>Scorer: calculate_string_score(section_weight, confidence)
        Scorer->>Scorer: Add weights: section + confidence
        Scorer->>Scorer: Round to integer
        Scorer-->>Extractor: final_score (0-20)
        
        Extractor->>Result: Create FoundString with score
    end
    
    Result-->>Binary: Scored strings ready for analysis

## Proposed File Changes

### src/scoring/mod.rs(NEW)

References: 

- src/types.rs
- src/container/mod.rs
- src/container/elf.rs
- src/scoring/encoding_confidence.rs(NEW)

Create the main scoring module that exports the public API for string scoring. This module should:

- Define and export the `calculate_encoding_confidence` function that takes text, encoding type, and raw bytes as parameters and returns a confidence score (f32) between 0.0 and 10.0
- Define and export the `calculate_string_score` function that combines section weight and encoding confidence to produce the final score (i32)
- Re-export the encoding-specific scoring functions from the submodules for easier access
- Include module documentation explaining the scoring algorithm and scale

The module should serve as the main entry point for all scoring operations, similar to how `src/container/mod.rs` works for container parsing.
Create unit tests for the encoding confidence scoring module:

**ASCII confidence tests:**
- `test_ascii_confidence_high_quality` - Test clean ASCII string ("Hello World") should score 9.0-10.0
- `test_ascii_confidence_with_null_termination` - Test null-terminated ASCII string should get bonus
- `test_ascii_confidence_with_control_chars` - Test ASCII with control characters should score lower (7.0-8.0)
- `test_ascii_confidence_low_printable_ratio` - Test ASCII with many non-printable chars should score low (< 5.0)

**UTF-8 confidence tests:**
- `test_utf8_confidence_valid_unicode` - Test valid UTF-8 with Unicode characters should score high
- `test_utf8_confidence_mixed_quality` - Test UTF-8 with some invalid sequences should score medium

**UTF-16 confidence tests:**
- `test_utf16le_confidence_aligned` - Test properly aligned UTF-16LE string should score high
- `test_utf16le_confidence_misaligned` - Test misaligned UTF-16LE should score lower
- `test_utf16le_confidence_null_terminated` - Test null-terminated UTF-16LE (0x00 0x00) should get bonus
- `test_utf16be_confidence` - Test UTF-16BE encoding confidence

**Integration tests:**
- `test_section_weight_integration` - Test that section weight is properly added to encoding confidence
- `test_score_calculation_range` - Test that final scores are in expected range (0-20)
- `test_score_rounding` - Test that scores are properly rounded to integers

**Edge cases:**
- `test_empty_string_confidence` - Test handling of empty strings
- `test_single_char_confidence` - Test very short strings
- `test_very_long_string_confidence` - Test long strings

Tests should use the `#[cfg(test)]` module pattern and follow the existing test structure seen in `src/container/elf.rs` and other parser tests.

### src/scoring/encoding_confidence.rs(NEW)

References: 

- src/types.rs

Implement the encoding confidence scoring algorithm with separate functions for each encoding type:

**Main function:**
- `calculate_encoding_confidence(text: &str, encoding: Encoding, raw_bytes: &[u8]) -> f32` - Dispatcher that routes to encoding-specific functions based on the encoding type

**ASCII confidence scoring:**
- `score_ascii_confidence(text: &str, raw_bytes: &[u8]) -> f32` - Calculate confidence for ASCII strings
  - Calculate printable character ratio (using `is_ascii_graphic()` and `is_ascii_whitespace()`)
  - Check for null termination (bonus if `raw_bytes.last() == Some(&0)`)
  - Calculate control character ratio and apply penalty
  - Base score: printable_ratio * 10.0
  - Apply null termination bonus (0.5 points) if present
  - Apply control character penalty if ratio > 0.1
  - Clamp result to 0.0-10.0 range

**UTF-8 confidence scoring:**
- `score_utf8_confidence(text: &str, raw_bytes: &[u8]) -> f32` - Calculate confidence for UTF-8 strings
  - Similar to ASCII but with UTF-8 specific validation
  - Check for valid UTF-8 sequences
  - Calculate printable character ratio (using Unicode character properties)
  - Check null termination
  - Apply similar scoring logic as ASCII

**UTF-16 confidence scoring:**
- `score_utf16_confidence(text: &str, raw_bytes: &[u8], encoding: Encoding) -> f32` - Calculate confidence for UTF-16 strings
  - Verify byte alignment (even length for UTF-16)
  - Check for proper null termination (0x00 0x00 pattern)
  - Calculate valid character ratio by examining byte pairs
  - For UTF-16LE: check that high bytes are mostly zero for ASCII-range characters
  - For UTF-16BE: check that low bytes are mostly zero for ASCII-range characters
  - Apply alignment penalty if misaligned
  - Base score: valid_char_ratio * 10.0
  - Apply null termination bonus
  - Clamp to 0.0-10.0 range

**Helper functions:**
- `calculate_printable_ratio(text: &str) -> f32` - Calculate ratio of printable to total characters
- `calculate_control_char_ratio(text: &str) -> f32` - Calculate ratio of control characters
- `is_null_terminated(raw_bytes: &[u8], encoding: Encoding) -> bool` - Check for proper null termination based on encoding
- `check_utf16_alignment(raw_bytes: &[u8]) -> bool` - Verify UTF-16 byte alignment

All functions should include comprehensive documentation with examples of score ranges for different string qualities.

### src/scoring/string_score.rs(NEW)

References: 

- src/types.rs

Implement the string score calculation that integrates section weight with encoding confidence:

**Main function:**
- `calculate_string_score(section_weight: f32, encoding_confidence: f32) -> i32` - Combine section weight and encoding confidence into final score
  - Add section_weight + encoding_confidence
  - Round to nearest integer using `.round() as i32`
  - Clamp to reasonable range (0-20 based on max section weight of 10.0 + max confidence of 10.0)
  - Return the final score

**Helper function:**
- `calculate_score_with_context(string: &FoundString, section_weight: f32, encoding_confidence: f32) -> i32` - Alternative function that takes a FoundString and calculates score with additional context
  - This function can be used when more context is available
  - Currently just calls `calculate_string_score` but provides extension point for future enhancements (semantic boost, noise penalty)

The module should be simple and focused, as it's primarily combining the two components. Future enhancements (semantic boost, noise penalty) will be added in separate issues as mentioned in the GitHub issue dependencies.

### src/lib.rs(MODIFY)

Add the new `scoring` module to the library's module tree:

- Add `pub mod scoring;` after the existing module declarations (around line 48, after `pub mod types;`)
- Add re-exports for commonly used scoring functions in the re-export section (around line 51-54):
  - Re-export `scoring::calculate_encoding_confidence`
  - Re-export `scoring::calculate_string_score`

Update the module documentation comment (around lines 38-42) to reflect that scoring functionality is now implemented:
- Change the scoring module status from "🚧 Framework ready" to "✅ Complete" or similar
- Add a brief note about the scoring capabilities

This follows the same pattern used for other modules in the library.

### src/extraction/mod.rs(MODIFY)

References: 

- src/types.rs
- src/scoring/mod.rs(NEW)
- docs/src/string-extraction.md

Implement the string extraction logic that integrates scoring. Since this file is currently empty (just a comment), this will be a substantial addition:

**Core extraction function:**
- `extract_strings_from_section(section: &SectionInfo, data: &[u8], config: &ExtractionConfig) -> Vec<FoundString>` - Extract strings from a section with scoring
  - Scan the section data for strings based on encoding types
  - For each found string, calculate encoding confidence using `scoring::calculate_encoding_confidence`
  - Calculate final score using `scoring::calculate_string_score` with section.weight and encoding confidence
  - Populate all fields of FoundString including the score field
  - Return vector of FoundString instances

**Helper functions:**
- `extract_ascii_strings(data: &[u8], offset: u64, min_len: usize) -> Vec<RawString>` - Extract ASCII/UTF-8 strings from raw bytes
- `extract_utf16_strings(data: &[u8], offset: u64, min_len: usize, endianness: Endianness) -> Vec<RawString>` - Extract UTF-16 strings
- `create_found_string(raw: RawString, section: &SectionInfo, encoding_confidence: f32) -> FoundString` - Convert RawString to FoundString with scoring

**Configuration structure:**
- `ExtractionConfig` struct with fields for min_ascii_len, min_utf16_len, enabled_encodings, etc.

**Internal types:**
- `RawString` struct to hold intermediate extraction results before scoring (text, offset, encoding, raw_bytes)

The extraction logic should integrate with the scoring module by:
1. Extracting raw strings from section data
2. Calculating encoding confidence for each string using `scoring::calculate_encoding_confidence`
3. Using the section's weight field from `SectionInfo`
4. Calling `scoring::calculate_string_score` to get the final score
5. Populating the `FoundString.score` field with the calculated value

Reference the extraction algorithms described in `docs/src/string-extraction.md` for implementation details.

### tests/integration_scoring.rs(NEW)

References: 

- tests/integration_elf.rs
- tests/fixtures/README.md

Create integration tests that verify end-to-end scoring behavior with real binary data:

**Test structure:**
- Use the existing test fixtures in `tests/fixtures/` (test_binary_elf, test_binary_macho, test_binary.c)
- Follow the pattern established in `tests/integration_elf.rs`, `tests/integration_pe.rs`, and `tests/integration_macho.rs`

**Test cases:**
- `test_elf_string_scoring` - Extract strings from ELF binary and verify scores are populated
  - Parse the test_binary_elf fixture
  - Extract strings from .rodata section (should have high section weight of 10.0)
  - Verify that clean ASCII strings in .rodata have high scores (18-20)
  - Verify that strings in .data section have lower scores due to lower section weight (5.0)

- `test_macho_string_scoring` - Extract strings from Mach-O binary and verify scores
  - Parse the test_binary_macho fixture
  - Extract strings from __TEXT,__cstring section (weight 10.0)
  - Verify high-quality strings score appropriately

- `test_section_weight_affects_score` - Verify that section weight directly impacts final score
  - Compare identical strings from different sections
  - String from .rodata (weight 10.0) should score ~10 points higher than same string from .data (weight 5.0)

- `test_encoding_confidence_affects_score` - Verify that encoding quality impacts score
  - Compare clean vs noisy strings from same section
  - Clean string should score higher than noisy string even in same section

- `test_score_field_populated` - Verify that FoundString.score field is properly populated
  - Extract strings and verify score field is not zero/default
  - Verify scores are in reasonable range (0-20)

**Assertions:**
- Verify score field is populated (not 0 or default value)
- Verify scores are within expected range based on section weight + encoding confidence
- Verify relative ordering (high-weight sections produce higher scores)
- Verify score calculation is deterministic (same input produces same score)

Use snapshot testing with `insta` crate if available (as seen in `tests/snapshots/` directory) to capture and verify scoring results.

### docs/src/ranking.md(MODIFY)

References: 

- src/container/elf.rs
- src/container/pe.rs
- src/container/macho.rs
- src/scoring/encoding_confidence.rs(NEW)

Update the ranking documentation to reflect the actual implementation:

**Section Weight section (lines 13-57):**
- Update the weight values table to match the actual implementation (float values 1.0-10.0 instead of integers 40, 35, etc.)
- Clarify that section weights are calculated per-format in the container parsers
- Update the code example to reflect the actual implementation in `src/container/elf.rs`, `src/container/pe.rs`, and `src/container/macho.rs`
- Note that format-specific bonuses are already built into the section weight calculation

**Encoding Confidence section (lines 59-102):**
- Update to reflect the 0.0-10.0 scale as implemented
- Document the actual confidence factors:
  - Character validity percentage
  - Printability ratio
  - Null termination bonus
  - Byte alignment for UTF-16
  - Control character ratio penalty
- Update the code example to match the actual function signatures in `src/scoring/encoding_confidence.rs`
- Provide specific score ranges:
  - 9.0-10.0: High confidence (clean ASCII/UTF-8, properly aligned UTF-16)
  - 7.0-8.9: Medium confidence (some irregularities but valid)
  - 5.0-6.9: Low confidence (marginal validity)
  - < 5.0: Very low confidence (likely noise)

**Complete Scoring Implementation section (lines 257-286):**
- Update to reflect the current implementation status
- Note that this issue implements SectionWeight + EncodingConfidence
- Document that SemanticBoost and NoisePenalty are planned for future issues
- Update the score range table to reflect the current 0-20 range (section weight 0-10 + encoding confidence 0-10)

**Add Implementation Status section:**
- Add a new section documenting what's implemented vs planned
- ✅ Section Weight Calculation (implemented in container parsers)
- ✅ Encoding Confidence Scoring (implemented in this issue)
- 🚧 Semantic Boost (planned)
- 🚧 Noise Penalty (planned)

Reference the actual implementation files for developers who want to understand the details.

### README.md(MODIFY)

References: 

- src/types.rs

Update the README to reflect the new scoring capabilities:

**Features section:**
- Add a bullet point about intelligent string ranking based on section weight and encoding confidence
- Mention that strings are scored on a 0-20 scale combining section importance and encoding quality

**Status/Roadmap section (if present):**
- Update to show that encoding confidence scoring and section weight integration are now implemented
- Mark these features as complete in any roadmap or status indicators

**Usage examples (if present):**
- If there are code examples, update them to show that FoundString now includes populated score fields
- Add a note that strings can be sorted by score for relevance-based analysis

Keep changes minimal and focused on documenting the new scoring functionality without rewriting the entire README.

---

Developer Humor

🎯 Scoring strings like a pro!\n> \n> We weigh the sections, check the bytes,\n> Count the printables, spot the slights.\n> UTF-16 aligned? That's a win!\n> Null-terminated? Bonus points within!\n> \n> From .rodata's gold to .data's clay,\n> Each string now knows its ranking way! ⭐

---

Execution Information

Branch: main
Commit: bdd6206

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.