Using AWS CRT client for S3 upload resulting in Thread leak during Integ Test case execution.

[pranikum]

Describe the bug

As part of Opensearch s3 plugin. We are migrating from Netty client to crt client to upload files to s3. Our Integration tests have check for ThreadLeaks once the Test Case execution completes. We are seeing ThreadLeak Error on the completion of execution when CRT client is being used.

  2> com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.opensearch.repositories.s3.S3RepositoryThirdPartyTests: 
       1) Thread[id=83, name=AwsEventLoop 3, state=RUNNABLE, group=main]
            at (empty stack)
       2) Thread[id=74, name=AwsEventLoop 2, state=RUNNABLE, group=main]
            at (empty stack)
       3) Thread[id=79, name=AwsEventLoop 8, state=RUNNABLE, group=main]
            at (empty stack)
        at __randomizedtesting.SeedInfo.seed([13DA5FCB891811FE]:0)

on Investigation we are seeing that the native threads used by EventLoopGroup as CRTResource is not getting terminated properly.
Is this known behaviour?
Can something be done to clear these threads?

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

ThreadLeak should not happen

Current Behavior

ThreadLeak is seen.

  2> com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.opensearch.repositories.s3.S3RepositoryThirdPartyTests: 
       1) Thread[id=83, name=AwsEventLoop 3, state=RUNNABLE, group=main]
            at (empty stack)
       2) Thread[id=74, name=AwsEventLoop 2, state=RUNNABLE, group=main]
            at (empty stack)
       3) Thread[id=79, name=AwsEventLoop 8, state=RUNNABLE, group=main]
            at (empty stack)
        at __randomizedtesting.SeedInfo.seed([13DA5FCB891811FE]:0)

Reproduction Steps

Reference Issue for Switching to CRT: opensearch-project/OpenSearch#18800
Reference PR: opensearch-project/OpenSearch#18800
We have currently averted the issue by making "@ThreadLeakScope(ThreadLeakScope.Scope.NONE)"

https://github.com/opensearch-project/OpenSearch/pull/18800/files#diff-0e02bc404b49fa24e689c323473778abf40549bbacd9473ac0c192331e53bf0dR92

Possible Solution

No response

Additional Information/Context

No response

aws-crt-java version used

2.30.31

Java version used

21

Operating System and version

Linux, MAC

[DmitriyMusatkin]

CRT create several static resources that have effectively global lifetime if you dont override defaults. EventLoopGroup, i.e. c side thread pool, is one of them. You can explicitly release those through

aws-crt-java/src/main/java/software/amazon/awssdk/crt/CrtResource.java

Line 427 in f4e09df

public static void waitForNoResources() {