Skip to content

fix: use async LLM calls in async task output conversion#5252

Open
shivam2407 wants to merge 1 commit intocrewAIInc:mainfrom
shivam2407:fix/async-export-output-5230
Open

fix: use async LLM calls in async task output conversion#5252
shivam2407 wants to merge 1 commit intocrewAIInc:mainfrom
shivam2407:fix/async-export-output-5230

Conversation

@shivam2407
Copy link
Copy Markdown

Summary

Fixes #5230

When tasks execute via akickoff(), _export_output() calls synchronous llm.call() inside async methods (_aexecute_core, _ainvoke_guardrail_function), blocking the event loop and severely degrading async throughput.

This PR adds async variants throughout the output conversion chain so that async task execution never blocks the event loop with synchronous LLM calls. Sync paths are completely untouched.

Changes

File Change
base_output_converter.py Added ato_pydantic() / ato_json() with asyncio.to_thread defaults for backward compat
internal_instructor.py Added async instructor client (_get_async_client) + ato_pydantic() / ato_json(). Extracted _build_provider_model_string() to deduplicate sync/async client factories
converter.py Added Converter.ato_pydantic() / ato_json() using llm.acall(). Added module-level aconvert_to_model(), ahandle_partial_json(), aconvert_with_instructions()
task.py Added _aexport_output() using aconvert_to_model(). Updated all 3 async call sites
test_converter.py 9 new async tests verifying llm.call() is never invoked from async paths

Design decisions

  • OutputConverter base class: New async methods have asyncio.to_thread defaults so existing custom subclasses work without modification. Converter overrides with native async.
  • ato_pydantic fallback: Uses inline regex extraction instead of calling sync handle_partial_json (which would fall through to llm.call()).
  • InternalInstructor: Lazy-creates async instructor client via _get_async_client(), cached after first use. Uses instructor.from_litellm(acompletion) for litellm or instructor.from_provider(..., async_mode="async") for other providers.

Test plan

  • 9 new async tests all pass (pytest -k "async or ato_")
  • All 51 existing converter tests still pass (no regressions)
  • Each async test asserts llm.call.assert_not_called() — sync path never invoked
  • Key regression test: test_ato_pydantic_validation_failure_never_calls_sync covers the ValidationError fallback path
  • Manual testing with akickoff() + output_pydantic task in a real crew

)

When tasks run via akickoff(), _export_output() called synchronous
llm.call() inside async methods (_aexecute_core, _ainvoke_guardrail_function),
blocking the event loop and degrading async throughput.

Add async variants throughout the output conversion chain:
- OutputConverter: ato_pydantic/ato_json with asyncio.to_thread defaults
- InternalInstructor: async instructor client + ato_pydantic/ato_json
- Converter: ato_pydantic/ato_json using llm.acall()
- Module-level: aconvert_to_model, ahandle_partial_json, aconvert_with_instructions
- Task: _aexport_output() replacing 3 sync call sites in async methods

Sync paths are completely untouched for backward compatibility.
@shivam2407
Copy link
Copy Markdown
Author

Hi @lorenzejay @greysonlalonde — this PR fixes #5230 (sync LLM calls blocking the event loop in async workflows). Would appreciate a review when you get a chance. Happy to address any feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Do not invoke synchronous call() on LLM from asynchronous workflow in _export_output / converter

1 participant