[Feat]: Improve server concurrency architecture #869
Description
Is your feature request related to a problem? Please describe.
The final goal of this effort is to relax or describe limitations on concurrency in AgentExecutor.
In existing version there is multiple hidden assumptions internally between classes as well as in API (e.g. AgentExecutor.cancel() can't wait for AgentExecutor.execute to finish).
Describe the solution you'd like
The plan is to make incremental changes on concurrency to ensure we can track down invalid change if new concurrency issue is introduced.
Describe alternatives you've considered
No response
Additional context
No response
Code of Conduct
- I agree to follow this project's Code of Conduct