⚡️ Speed up function pytest_cmd_tokens by 125% in PR #1887 (codeflash_python)

[codeflash-ai]

⚡️ This pull request contains optimizations for PR #1887

If you approve this dependent PR, these changes will be merged into the original PR branch codeflash_python.

This PR will be automatically closed if the original PR is merged.

---

📄 125% (1.25x) speedup for pytest_cmd_tokens in codeflash/verification/test_runner.py

⏱️ Runtime : 4.67 milliseconds → 2.08 milliseconds (best of 126 runs)

📝 Explanation and details

The optimization adds a fast-path that scans the input string once for shell-special characters (whitespace, quotes, backslashes, etc.) and returns [cmd] directly if none are found, bypassing shlex.split entirely. Line profiler shows the original spent 99.2% of time in shlex.split (~50 µs per call); the optimized version skips that call in 504 of 510 cases (when PYTEST_CMD is the simple token "pytest"), reducing per-call cost to ~180 ns for the fast-path. For the 6 invocations that hit the fallback (complex commands with quotes/spaces), the scan overhead (~1 µs) is negligible compared to the 2.08 ms overall runtime, yielding a 124% speedup with no regressions.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 510 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage	100.0%

🌀 Click to see Generated Regression Tests

from __future__ import annotations

# imports
from codeflash.verification.test_runner import pytest_cmd_tokens, setup_pytest_cmd


def test_returns_single_token_with_posix_true():
    # Basic behavior: with default constant "pytest" we always get a single token list
    result = pytest_cmd_tokens(True)  # 11.9μs -> 1.89μs (528% faster)
    # Must be a list of length 1
    assert isinstance(result, list), "Expected a list result"
    assert len(result) == 1, "Expected exactly one token for simple command"
    # The single token must equal the module constant
    assert result[0] == "pytest", "Token content must match PYTEST_CMD"


def test_returns_single_token_with_posix_false():
    # Same as above but with posix=False; behavior should not change for simple command
    result = pytest_cmd_tokens(False)  # 11.1μs -> 1.83μs (504% faster)
    assert result == ["pytest"], "Expected identical output for posix=False with a simple token"


def test_returned_list_is_independent_between_calls():
    # Ensure each call returns a fresh list (no aliasing)
    a = pytest_cmd_tokens(True)  # 11.6μs -> 1.75μs (560% faster)
    b = pytest_cmd_tokens(True)
    # They should be distinct objects (mutating one must not affect the other)
    assert a is not b, "Each call should return a new list object"  # 7.21μs -> 651ns (1008% faster)
    a[0] = "modified"
    # Changing a must not change b
    assert b == ["pytest"], "Mutating returned list from one call must not affect subsequent calls"


def test_quote_handling_differs_between_posix_modes():
    setup_pytest_cmd('pytest "arg with space" plain')
    result_posix_true = pytest_cmd_tokens(True)  # 21.2μs -> 23.5μs (9.88% slower)
    result_posix_false = pytest_cmd_tokens(False)

    assert isinstance(result_posix_true, list), "posix=True should return a list"  # 15.5μs -> 16.1μs (3.43% slower)
    assert isinstance(result_posix_false, list), "posix=False should return a list"

    assert len(result_posix_true) >= 2, "posix=True should parse quoted argument as separate token"
    assert "arg with space" in result_posix_true or any("arg" in t for t in result_posix_true), (
        "Should contain parsed argument"
    )

    assert len(result_posix_false) >= 2, "posix=False should also parse the command"
    assert result_posix_true != result_posix_false or len(result_posix_true) == len(result_posix_false), (
        "posix modes may differ in token structure"
    )

    setup_pytest_cmd("pytest")


def test_integer_bool_equivalence():
    # Passing integer 1/0 should behave like True/False because bool is a subclass of int
    assert pytest_cmd_tokens(1) == pytest_cmd_tokens(True), (
        "Integer 1 should behave like True for posix argument"
    )  # 11.6μs -> 1.69μs (587% faster)
    assert pytest_cmd_tokens(0) == pytest_cmd_tokens(False), (
        "Integer 0 should behave like False for posix argument"
    )  # 6.88μs -> 691ns (896% faster)


def test_large_number_of_tokens():
    test_cases = [
        " ".join([f"token{i}" for i in range(100)]),
        " ".join([f"arg{i}" for i in range(100, 150)]),
        " ".join([f"opt{i}" for i in range(150, 250)]),
        " ".join([f"param{i}" for i in range(250, 500)]),
    ]

    for idx, cmd in enumerate(test_cases):
        setup_pytest_cmd(cmd)
        result = pytest_cmd_tokens(True)  # 1.70ms -> 1.70ms (0.177% faster)
        expected_len = len(cmd.split())
        assert len(result) == expected_len, f"Test case {idx}: Expected {expected_len} tokens, got {len(result)}"
        assert result[0] == cmd.split()[0], f"Test case {idx}: First token mismatch"
        assert result[-1] == cmd.split()[-1], f"Test case {idx}: Last token mismatch"

    setup_pytest_cmd("pytest")

from __future__ import annotations

import shlex

# imports
from codeflash.verification.test_runner import pytest_cmd_tokens


def test_posix_true_returns_list():
    """Test that calling with is_posix=True returns a list."""
    result = pytest_cmd_tokens(is_posix=True)  # 12.1μs -> 2.19μs (453% faster)
    assert isinstance(result, list), "Result should be a list"


def test_posix_false_returns_list():
    """Test that calling with is_posix=False returns a list."""
    result = pytest_cmd_tokens(is_posix=False)  # 11.1μs -> 1.97μs (463% faster)
    assert isinstance(result, list), "Result should be a list"


def test_posix_true_contains_pytest():
    """Test that result with is_posix=True contains 'pytest' token."""
    result = pytest_cmd_tokens(is_posix=True)  # 11.6μs -> 1.95μs (492% faster)
    assert len(result) > 0, "Result should not be empty"
    assert "pytest" in result, "Result should contain 'pytest'"


def test_posix_false_contains_pytest():
    """Test that result with is_posix=False contains 'pytest' token."""
    result = pytest_cmd_tokens(is_posix=False)  # 11.0μs -> 1.83μs (500% faster)
    assert len(result) > 0, "Result should not be empty"
    assert "pytest" in result, "Result should contain 'pytest'"


def test_posix_true_exact_value():
    """Test that is_posix=True produces expected shlex.split result."""
    result = pytest_cmd_tokens(is_posix=True)  # 11.7μs -> 1.93μs (504% faster)
    expected = shlex.split("pytest", posix=True)
    assert result == expected, "Result should match shlex.split with posix=True"


def test_posix_false_exact_value():
    """Test that is_posix=False produces expected shlex.split result."""
    result = pytest_cmd_tokens(is_posix=False)  # 10.9μs -> 1.92μs (468% faster)
    expected = shlex.split("pytest", posix=False)
    assert result == expected, "Result should match shlex.split with posix=False"


def test_return_type_is_list_of_strings_posix_true():
    """Test that all elements in result are strings when is_posix=True."""
    result = pytest_cmd_tokens(is_posix=True)  # 11.7μs -> 1.89μs (515% faster)
    for item in result:
        assert isinstance(item, str), f"All items should be strings, got {type(item)}"


def test_return_type_is_list_of_strings_posix_false():
    """Test that all elements in result are strings when is_posix=False."""
    result = pytest_cmd_tokens(is_posix=False)  # 10.7μs -> 1.91μs (460% faster)
    for item in result:
        assert isinstance(item, str), f"All items should be strings, got {type(item)}"


def test_posix_true_non_empty():
    """Test that result with is_posix=True is non-empty."""
    result = pytest_cmd_tokens(is_posix=True)  # 11.6μs -> 1.91μs (504% faster)
    assert len(result) > 0, "Result should not be empty for is_posix=True"


def test_posix_false_non_empty():
    """Test that result with is_posix=False is non-empty."""
    result = pytest_cmd_tokens(is_posix=False)  # 10.7μs -> 1.95μs (449% faster)
    assert len(result) > 0, "Result should not be empty for is_posix=False"


def test_posix_true_single_element():
    """Test that is_posix=True with simple pytest command returns single element."""
    result = pytest_cmd_tokens(is_posix=True)  # 11.6μs -> 2.06μs (465% faster)
    # "pytest" without any quotes or spaces should tokenize to a single element
    assert len(result) == 1, "Simple 'pytest' should tokenize to one element"


def test_posix_false_single_element():
    """Test that is_posix=False with simple pytest command returns single element."""
    result = pytest_cmd_tokens(is_posix=False)  # 10.8μs -> 1.94μs (453% faster)
    # "pytest" without any quotes or spaces should tokenize to a single element
    assert len(result) == 1, "Simple 'pytest' should tokenize to one element"


def test_posix_true_first_element_is_pytest():
    """Test that first element is exactly 'pytest' when is_posix=True."""
    result = pytest_cmd_tokens(is_posix=True)  # 11.5μs -> 1.98μs (480% faster)
    assert result[0] == "pytest", "First element should be 'pytest'"


def test_posix_false_first_element_is_pytest():
    """Test that first element is exactly 'pytest' when is_posix=False."""
    result = pytest_cmd_tokens(is_posix=False)  # 10.9μs -> 1.94μs (463% faster)
    assert result[0] == "pytest", "First element should be 'pytest'"


def test_posix_true_returns_new_list_each_call():
    """Test that multiple calls with is_posix=True return equal but distinct lists."""
    result1 = pytest_cmd_tokens(is_posix=True)  # 11.6μs -> 1.97μs (489% faster)
    result2 = pytest_cmd_tokens(is_posix=True)
    assert result1 == result2, "Results should be equal"  # 7.43μs -> 901ns (725% faster)
    assert result1 is not result2, "Results should be different list objects"


def test_posix_false_returns_new_list_each_call():
    """Test that multiple calls with is_posix=False return equal but distinct lists."""
    result1 = pytest_cmd_tokens(is_posix=False)  # 10.8μs -> 1.96μs (453% faster)
    result2 = pytest_cmd_tokens(is_posix=False)
    assert result1 == result2, "Results should be equal"  # 6.79μs -> 822ns (726% faster)
    assert result1 is not result2, "Results should be different list objects"


def test_posix_true_vs_posix_false_consistency():
    """Test that both posix modes produce the same result for simple 'pytest'."""
    result_posix_true = pytest_cmd_tokens(is_posix=True)  # 11.4μs -> 1.97μs (479% faster)
    result_posix_false = pytest_cmd_tokens(is_posix=False)
    # For a simple command without quotes/escapes, both should tokenize the same
    assert result_posix_true == result_posix_false, (
        "Simple 'pytest' should tokenize identically"
    )  # 6.99μs -> 792ns (783% faster)


def test_posix_true_immutability_of_result():
    """Test that modifying returned list doesn't affect subsequent calls."""
    result1 = pytest_cmd_tokens(is_posix=True)  # 11.5μs -> 1.92μs (498% faster)
    result1.append("test")  # Mutate the returned list
    result2 = pytest_cmd_tokens(is_posix=True)
    assert len(result2) == 1, (
        "Function should return fresh list, not affected by mutations"
    )  # 6.99μs -> 751ns (831% faster)


def test_posix_false_immutability_of_result():
    """Test that modifying returned list doesn't affect subsequent calls."""
    result1 = pytest_cmd_tokens(is_posix=False)  # 10.9μs -> 1.98μs (449% faster)
    result1.append("test")  # Mutate the returned list
    result2 = pytest_cmd_tokens(is_posix=False)
    assert len(result2) == 1, (
        "Function should return fresh list, not affected by mutations"
    )  # 6.65μs -> 711ns (836% faster)


def test_posix_true_boolean_true():
    """Test with explicit boolean True value."""
    result = pytest_cmd_tokens(True)  # 11.3μs -> 1.75μs (546% faster)
    assert isinstance(result, list), "Should return list for boolean True"
    assert "pytest" in result, "Should contain 'pytest' for boolean True"


def test_posix_false_boolean_false():
    """Test with explicit boolean False value."""
    result = pytest_cmd_tokens(False)  # 10.6μs -> 1.71μs (516% faster)
    assert isinstance(result, list), "Should return list for boolean False"
    assert "pytest" in result, "Should contain 'pytest' for boolean False"


def test_posix_true_all_elements_non_empty_strings():
    """Test that all string elements are non-empty when is_posix=True."""
    result = pytest_cmd_tokens(is_posix=True)  # 11.6μs -> 1.98μs (487% faster)
    for item in result:
        assert len(item) > 0, "All string elements should be non-empty"


def test_posix_false_all_elements_non_empty_strings():
    """Test that all string elements are non-empty when is_posix=False."""
    result = pytest_cmd_tokens(is_posix=False)  # 10.7μs -> 2.05μs (422% faster)
    for item in result:
        assert len(item) > 0, "All string elements should be non-empty"


def test_posix_true_repeated_calls_consistency():
    """Test that calls with is_posix=True with varying sequence produce consistent results."""
    modes = [True, False, True, True, False, True, False, False, True, True]
    results_by_mode = {True: [], False: []}

    for mode in modes:
        result = pytest_cmd_tokens(is_posix=mode)  # 64.6μs -> 7.48μs (764% faster)
        results_by_mode[mode].append(result)

    true_baseline = results_by_mode[True][0]
    for i, result in enumerate(results_by_mode[True][1:], 1):
        assert result == true_baseline, f"True call {i} produced different result"

    false_baseline = results_by_mode[False][0]
    for i, result in enumerate(results_by_mode[False][1:], 1):
        assert result == false_baseline, f"False call {i} produced different result"


def test_posix_false_repeated_calls_consistency():
    """Test that calls with is_posix=False with varying sequence produce consistent results."""
    modes = [False, True, False, False, True, False, True, True, False, False]
    results_by_mode = {True: [], False: []}

    for mode in modes:
        result = pytest_cmd_tokens(is_posix=mode)  # 63.6μs -> 7.29μs (772% faster)
        results_by_mode[mode].append(result)

    false_baseline = results_by_mode[False][0]
    for i, result in enumerate(results_by_mode[False][1:], 1):
        assert result == false_baseline, f"False call {i} produced different result"

    true_baseline = results_by_mode[True][0]
    for i, result in enumerate(results_by_mode[True][1:], 1):
        assert result == true_baseline, f"True call {i} produced different result"


def test_posix_true_alternating_calls_consistency():
    """Test alternating calls between is_posix=True and False with varying indices."""
    results_by_mode = {True: [], False: []}

    mode_sequence = [
        True,
        False,
        True,
        False,
        False,
        True,
        False,
        True,
        True,
        False,
        True,
        True,
        False,
        False,
        True,
        False,
        True,
        False,
        False,
        True,
        False,
        True,
        True,
        False,
        False,
        False,
        True,
        True,
        False,
        False,
        True,
        False,
        True,
        False,
        False,
        True,
        False,
        True,
        True,
        False,
        True,
        True,
        False,
        False,
        True,
        False,
        True,
        False,
        False,
        True,
    ]

    for mode in mode_sequence:
        result = pytest_cmd_tokens(is_posix=mode)  # 275μs -> 30.2μs (812% faster)
        results_by_mode[mode].append(result)

    true_baseline = results_by_mode[True][0]
    for i, result in enumerate(results_by_mode[True][1:], 1):
        assert result == true_baseline, f"True call {i} produced inconsistent result"

    false_baseline = results_by_mode[False][0]
    for i, result in enumerate(results_by_mode[False][1:], 1):
        assert result == false_baseline, f"False call {i} produced inconsistent result"


def test_posix_true_result_length_consistency_1000_times():
    """Test that result length is consistent with varied call patterns."""
    mode_sequence = [True if i % 3 != 0 else False for i in range(50)]
    lengths = []

    for mode in mode_sequence:
        result = pytest_cmd_tokens(is_posix=mode)  # 275μs -> 29.6μs (831% faster)
        lengths.append((mode, len(result)))

    true_lengths = [l for m, l in lengths if m is True]
    false_lengths = [l for m, l in lengths if m is False]

    if true_lengths:
        first_true_length = true_lengths[0]
        for i, length in enumerate(true_lengths[1:], 1):
            assert length == first_true_length, f"True call {i} produced different length"

    if false_lengths:
        first_false_length = false_lengths[0]
        for i, length in enumerate(false_lengths[1:], 1):
            assert length == first_false_length, f"False call {i} produced different length"


def test_posix_false_result_length_consistency_1000_times():
    """Test that result length is consistent with varied call patterns."""
    mode_sequence = [False if i % 3 != 0 else True for i in range(50)]
    lengths = []

    for mode in mode_sequence:
        result = pytest_cmd_tokens(is_posix=mode)  # 270μs -> 29.7μs (810% faster)
        lengths.append((mode, len(result)))

    false_lengths = [l for m, l in lengths if m is False]
    true_lengths = [l for m, l in lengths if m is True]

    if false_lengths:
        first_false_length = false_lengths[0]
        for i, length in enumerate(false_lengths[1:], 1):
            assert length == first_false_length, f"False call {i} produced different length"

    if true_lengths:
        first_true_length = true_lengths[0]
        for i, length in enumerate(true_lengths[1:], 1):
            assert length == first_true_length, f"True call {i} produced different length"


def test_posix_true_first_element_consistency_1000_times():
    """Test that first element is consistently 'pytest' with varied patterns."""
    mode_sequence = [True if i % 2 == 0 else False for i in range(50)]

    for i, mode in enumerate(mode_sequence):
        result = pytest_cmd_tokens(is_posix=mode)  # 274μs -> 29.6μs (827% faster)
        assert result[0] == "pytest", f"Call {i} (mode={mode}) first element not 'pytest'"


def test_posix_false_first_element_consistency_1000_times():
    """Test that first element is consistently 'pytest' with varied patterns."""
    mode_sequence = [False if i % 2 == 0 else True for i in range(50)]

    for i, mode in enumerate(mode_sequence):
        result = pytest_cmd_tokens(is_posix=mode)  # 273μs -> 29.7μs (821% faster)
        assert result[0] == "pytest", f"Call {i} (mode={mode}) first element not 'pytest'"


def test_posix_true_all_elements_strings_1000_times():
    """Test that all returned elements are strings with varied call patterns."""
    mode_sequence = [True if i % 5 != 0 else False for i in range(50)]

    for i, mode in enumerate(mode_sequence):
        result = pytest_cmd_tokens(is_posix=mode)  # 275μs -> 29.7μs (826% faster)
        for j, item in enumerate(result):
            assert isinstance(item, str), f"Call {i} (mode={mode}), element {j} not a string"


def test_posix_false_all_elements_strings_1000_times():
    """Test that all returned elements are strings with varied call patterns."""
    mode_sequence = [False if i % 5 != 0 else True for i in range(50)]

    for i, mode in enumerate(mode_sequence):
        result = pytest_cmd_tokens(is_posix=mode)  # 266μs -> 29.9μs (790% faster)
        for j, item in enumerate(result):
            assert isinstance(item, str), f"Call {i} (mode={mode}), element {j} not a string"


def test_posix_true_pytest_in_result_1000_times():
    """Test that 'pytest' is in result with varied patterns (is_posix=True)."""
    mode_sequence = [True if i % 4 != 0 else False for i in range(50)]

    for i, mode in enumerate(mode_sequence):
        result = pytest_cmd_tokens(is_posix=mode)  # 273μs -> 29.4μs (831% faster)
        assert "pytest" in result, f"Call {i} (mode={mode}) did not contain 'pytest'"


def test_posix_false_pytest_in_result_1000_times():
    """Test that 'pytest' is in result with varied patterns (is_posix=False)."""
    mode_sequence = [False if i % 4 != 0 else True for i in range(50)]

    for i, mode in enumerate(mode_sequence):
        result = pytest_cmd_tokens(is_posix=mode)  # 266μs -> 29.7μs (797% faster)
        assert "pytest" in result, f"Call {i} (mode={mode}) did not contain 'pytest'"

To edit these changes git checkout codeflash/optimize-pr1887-2026-03-24T23.21.53 and push.

[claude]

Claude finished @codeflash-ai[bot]'s task in 1m 42s —— View job

---

PR Review Summary

Prek Checks

✅ Passes — no linting or formatting issues.

Code Review

Classification: SMALL (15 lines added, 1 deleted in a single file)

Verdict: Correct optimization. ✅

The change adds a fast-path to pytest_cmd_tokens (codeflash/verification/test_runner.py:36) that avoids calling shlex.split when PYTEST_CMD contains no shell-special characters (whitespace, quotes, backslashes, #).

The logic is sound:

• shlex.split("pytest", posix=True/False) always returns ["pytest"] for a plain token with no special characters.
• The character scan (for ch in cmd) is linear but very fast for short strings — far cheaper than shlex.split's pure-Python lexer overhead.
• The _SHLEX_SPECIAL set (" \t\n\r\x0b\x0c'\"\\#") is conservative and correct — it covers all characters that would cause shlex to produce output different from [cmd].
• The empty-string fast-path (if not cmd: return []) correctly matches shlex.split("") → [].

No bugs, no security issues, no breaking API changes. The 510 generated regression tests all passed, including tests with complex quoted commands that exercise the shlex.split fallback path.

Duplicate Detection

No duplicates detected.

Test Coverage

No new public functions added; existing tests cover the changed code path.

---

Last updated: 2026-03-24T23:23 UTC

[claude]

CI failures are pre-existing on the base branch (not caused by this PR): unit-tests, end-to-end tests, JS optimization tests, and other integration checks all fail on the codeflash_python base branch. Leaving open for merge once base branch CI is fixed.

[claude]

CI failures are pre-existing on the base branch codeflash_python (not caused by this PR). All unit-tests, js-*, tracer-replay, and other checks are failing identically across PRs #1890, #1891, and #1895. Leaving open for merge once base branch CI is fixed.

[claude]

CI failures are pre-existing on the base branch (not caused by this PR): unit-tests (all Python versions), js-cjs-function-optimization, js-esm-async-optimization, js-ts-class-optimization, java-tracer-e2e, topological-sort-worktree-optimization, tracer-replay. Leaving open for merge once base branch CI is fixed.

[claude]

CI failures are pre-existing on the base branch, not caused by this PR. The same checks fail on base PR 1887 (codeflash_python): unit-tests, js-optimization checks, tracer-replay, topological-sort-worktree-optimization, init-optimization, java-tracer-e2e. Leaving open for merge once base branch CI is fixed.

[claude]

CI failures are pre-existing on the base branch codeflash_python (PR #1887, not caused by this optimization PR). Leaving open for merge once base branch CI is fixed.

[claude]

CI failures are pre-existing on the base branch (not caused by this PR): unit-tests, tracer-replay, js-cjs-function-optimization, js-esm-async-optimization, js-ts-class-optimization, java-tracer-e2e, topological-sort-worktree-optimization. Leaving open for merge once base branch CI is fixed.