Test framework

Goal, Overview and Developer's notes

The goal of this project was initially to create an automated, distributed testing framework for SharkNetMessenger. Through trial and error over a long period of time, it was shown that the core scenarios between two peers had to be formulated first, before attempting to program a custom scenario generator for each possible combination.

Alpha testing was the first attempt at testing the core scenario. The test scenarios started with two-peer encounters and became more complex when combined with other atomic tests. The end goal was to ensure that every atomic test was combined with every other test. However, this resulted in many redundancies and was archived in favour of a second, even more simplified version comprising basic, complex and hub (TBC) encounter tests. (see: Field Tests).

This version uses a hierarchy of scripts: a test orchestrator manages the runner scripts and helpers that run a complete test suite comprising a set of defined scenarios.

Future versions of the test framework could include the SharkNetMessenger CLI with its own test commands, replacing the test orchestrator bash script and eliminating the need for external scripting.

Ever since testing began, synchronising the peers has posed a significant challenge due to the occurrence of race conditions or port failures. Implementing the block/release feature greatly reduced wait time juggling, but did not eliminate the necessary wait time between automatically triggered operations entirely. While the test timeouts have largely been reduced, systemic faults or changing behaviour in different environments cannot be ruled out or ignored.

Please note that scripts may need to be modified for different operating systems. The bash scripts were written under macOS 13.7.8 (Ventura). Although portability was considered during development, it is not guaranteed. It should also be noted that test success seems to depend on the terminal itself. For example, the VSCode terminal appears to be more successful than the IntelliJ terminal. The macOS native terminal is not particularly notable in this respect.

Clearly, the testing is far from over. The results are inconsistent. Running the test multiple times results in different and unpredictable outcomes each time. Even the most successful (and elementary) tests, especially SC (see below), often fail due to timeout.

General structure

The updated version of the test framework uses bash scripts to orchestrate the execution, evaluation and accumulation of encounter, complex and hub test scenarios.

Architecture

Scenario generation layer

• A Java .jar file (scriptgenerator.jar) that compiles an executable of the script generator. Its artefacts are the command lists of each peer corresponding to the aforementioned scenarios. As of April 2026, it is not possible to modify the payload or peer number of the scripts without amending the source code. This decision was made to reduce complexity in this phase of testing. The numerical details of the scenario scripts will be mentioned in upcoming chapters.

Scenario Execution layer

Essentials:

• Bash scripts:

1. prepAndExecute.sh: The main orchestrator. It discovers scenario directories. Copies the necessary files, such as SharkNetMessenger and the test runners, into the scenario directories. It executes each scenario with a timeout and orphan process cleanup logic. Exports run artefacts in the 'testRun' and 'testRunFailed' categories. It builds a final zip archive of results.

With the correct argument, the IP address of the test target can be changed. Localhost is used by default for local testing.

2. Runtime scripts: runTCPCoreScenario.sh and runHubCoreScenario.sh. These scripts validate the contents of the scenario directory, generate any necessary files for the test runs, and ensure that processes are fully terminated before moving on to other scenarios, thus ensuring test isolation. They also enforce the timeout mechanism monitored by the parent process.

Extras/command list debugging helpers:

Single scenario runner: runSingleScenario.sh This runs a single scenario as a background process, rather than running all scenarios consecutively.

Run a single scenario with each peer running in a separate terminal window for visualisation purposes, instead of running it as a background process.

Cleanup helper: cleanup_scenarios.sh This cleans up remnant scenarios and messenger data from past runs, ensuring test run integrity by deleting all previous scenario files.

Kill_shark_processes.sh: kills orphan processes resulting from frozen processes that have failed to be manually terminated and are interfering with ongoing tests due to shared ports.

Results layer

Each run is archived in a dated folder and a zipped summary is created. Failed test runs are collected in a designated folder for easy access. Local evaluations are collected into a single file per run. Error messages are collected in a separate file.

Main Entry Points:

The script generator .jar file is created by running coreScenarioGenerator.sh in the project root directory. This exports the .jar file to the basic_testing folder, which is the current phase that the script generator caters for.

The main test orchestrator is prepAndExecute.sh in the basic_testing folder. The alpha_testing folder includes an older version of the script which does not meet the needs of the basic_testing phase. To start a single test run of all scripts, prepAndExecute.sh should be started in the folder into which the scriptgenerator.jar output goes. In this case, this is the basic_testing folder.

Scenario generator layer

The corresponding package is src/asapTestEngine.

The main generator entrypoint:

• CoreScenarioOutput.java class.

Command primitives and constants:

can be found in CommandListToFile.java. Some newer classes use references and constants from the messenger source files. (There is a parameter parsing utility under ScenarioParamAllocation.java that is no longer in use, but which is unfortunately still partially used in the current version and thus cannot be deleted without further refactoring.)

Scenario definition classes:

• Basic Encounter tests:(see wiki) (CoreBasicEncounter.java)

  • CS: Connect, then send, between two peers.
  • SC: Send, then connect, between two peers.

• Complex Encounter tests: (see wiki) (CoreScenarioTCPChain.java)

  • TCP Chain: TCP connection between a line of peers. To reduce complexity, the number of peers is limited to three. (CoreScenarioTCPStar.java)
  • TCP Star: TCP connection between peers, where each peer connects with a single peer, putting the latter into a central position and creating a star formation.

• Hub Mediated Encounter Tests: (see wiki) (CoreScenarioHub.java)

  • Hub Stalling: A hub mediated connection in which one peer sends messages of desired size (3 files of 1kB in the current version)
  • Hub Length: A hub mediated connection where peers (4 peers in the current version) route a message of desired length (default: 1kB) from the first peer to the last.

Usage

1. execute: ./corescenariogenerator.sh
2. cd basic_testing
3. execute: ./prepAndExecute.sh
4. wait for the processes to terminate
5. check eval.txt for results!

Example folder structure after a test run:

basic_testing/
├── .runs/
│   └── macAlone/
├── errorlog.txt
├── eval.txt
├── kill_shark_processes.sh
├── prepAndExecute.sh
├── runHubCoreScenario.sh
├── runSingleScenario.sh
├── runSingleScenarioTerminals.sh
├── runTCPCoreScenario.sh
├── scriptgenerator.jar
├── SharkNetMessengerCLI.jar
├── testRuns/
│   ├── CHAINLTX/
│   ├── CHAINX/
│   ├── CS/
│   ├── HUBTX_LENGTH/
│   ├── HUBT_STALLING/
│   ├── SC/
│   ├── STARXCS/
│   ├── STARXSC/
│   └── scriptgenerator_output_*.zip
├── testRunsFailed/
│   └── (empty)
└── testresults_*.zip

• macAlone: meaning a single Mac local test. This would be modified in the script in a different setting (export $TEST_ENV in prepAndExecute.sh)