[SU] Extendable test timeouts

[khodya]

Extendable test timeouts

It is hard to predict what test timeout to set in Software Update tests -- it depends on the image size. Implement a feedback mechanism in tests that prompts a user whether the timeout should be extended and extends the timeout.

reschedule method

One of the possible implementations is reschedule method and asynchronous context managers.

Example:

import asyncio

async def my_task():
    try:
        # Start with a 5-second timeout
        async with asyncio.timeout(5) as delay:
            print("Starting long task...")
            await asyncio.sleep(3)
            
            # Something happened! We need more time.
            # Extend it by another 5 seconds from right now
            new_deadline = asyncio.get_running_loop().time() + 5
            delay.reschedule(new_deadline)
            print("Timeout extended!")
            
            await asyncio.sleep(4) 
    except TimeoutError:
        print("Task timed out anyway!")

asyncio.run(my_task())

[jtrejoespinoza-grid]

There are some findings and ideas that will help to discuss in the development of this issue.

Mobly timeouts

• default_timeout = 60 * 2 = 120 seconds ( if not set any test over this time will fail )
• rpc_max_timeout_sec (remote procedure call max timeout) :

Actions that are requested:

--timeout : timeout for the test
--cert-timeout: Framework may need to have a custom timeout for certification ( larger that ussual timeout), when this expires the
framework may ask for more time ( maybe just double the time), if not just fail the test. Also can work as information on how much the test should take to run.

To achieve the action on --cert-timeout (request for more time), a subprocess should launch with this timeout and check if the process need to be cancelled or continue,
this mean that there should be two subprocess one for the timer and other for the main process should be running in a subprocess.

Problems that this might cause or find

It will be required to define the timeouts (Matter and Mobly), because if mobly has a lower timeout than matter implementation it will cause a TimeoutError.
This will be more noticable in test like SU where one step can last up to 5 minutes this will trigger the default timeout from mobly ( 2 minutes )

Current idea:

if --cert-timeout:
    timeout = cert_timeout
    mobly.default_timeout = cert_timeout * n  ( timeout must be bigger as there may be manual actions and any timeout lower or equial on mobly will likely fail the test)
    Launch both subprocess ( timer and test)
    if Wait for test to complete or timer has reached timeout
        if timer complete and test is not complete:
            Ask if want to wait more time
                yes: 
                    keep waiting until test is complete or timeout
                    if timeout: 
                        fail_test (TimeoutError)
                    if test_failed:
                        on_errror
                else:
                    end_the_test as failed ( TimeoutError)
    Send status of passed.
    Complete test

Other findings

• The TimeoutError exception block on run() on runner.py is never reached because mobly catches this and avoid resending the TimeoutError
• The same for General Exception its already catched at mobly , the only exception that is catched is TestAbortSignal.