Improve concurrency of build processes on AIX machines.

[sxa]

Many of our current AIX machines have significant processing power and RAM (Most are 4-core with SMT8) and 32GiB if RAM. We are currently throttling some of them to -j8 or lower which means the machines are likely to be underutilised relative to their ability:

build/ansible/inventory.yml

Line 122 in 7d8af5b

server_jobs: 8

This issue will cover experimenting to see if we can increase this, as duering some recent testing I have been running directly on the machine with values of -j16 without issues (Noting that it's also possible that running through jenkins has different effects...)

The limitations were initially put in due to the analysis in #2566 and then added to our automation in #2574 - it's not clear which versions of Node were affected but I will also try node.js 22 to cover both ends of what we are supporting. I will also note that the specific machine which was listed as failing with anything greater than 3 is currently offline.

[richardlau]

Just to note, we need to do the testing for the worst case scenario -- no populated ccache (or near to 100% miss rate). Problems usually arose with V8-canary or V8 update PRs.

[sxa]

Just to note, we need to do the testing for the worst case scenario -- no populated ccache (or near to 100% miss rate). Problems usually arose with V8-canary or V8 update PRs.

Yep I'm doing all of the testing with ccache disabled for now in order to stretch the machine resources. but initially only with the node builds.

[sxa]

Unfortunately I lost my earlier table as firefox crashed on me ...

At present osuosl-be-4 and ibm-aix73-3 are giving me linker problems (stopping with return code 8) and it's not clear why. I've got one ccached build through on osuosl-be-4 and still diagnosing why it doesn't always work.

These times are from a clean build without a populated ccache:

System	Processors	default	Node26-j16	Node26-j20	Node20-j20	Node25-j20
ibm-be-2	4*P9,SMT8	99m (-j8)	69m/75m	69m	36m	67m
osuosl-be-4	4*P8,SMT8	151m (-j6)	140m/138m	159m	56m	Stuck? [*]
ibm-aix73-3 (8G)	4*P9,SMT8	174m (-j5)			58m (-j8)	174m (-j4)
ibm-aix73-3 (16G)	4*P8,SMT8	88m (-j12)	87m	OOM	45m

[*] - It got a bit stuck while at the stage where it was creating the libnode.a archive. Probably best not to push it this high. It hadn't ran out of memory but the make process stuck

Note: I also tried -j12 on ibm-be-2 and it took 77 minutes
Also tried another run with -j20 on main and it stalled or otherwise caused a disconnect on my ssh session after 33 minutes, so with that and the apparent lack of improvement over -j16 shown in the table above when it does complete I suspect that's the limit of how far we should push this.

  g++-12 -o /home/iojs/sxa/node/out/Release/obj.target/v8_base_without_compiler/deps/v8/src/handles/global-handles.o ../deps/v8/src/handles/global-handles.cc '-D_GLIBCXX_USE_CXX11_ABI=1' '-D_FILE_OFFSET_BITS=64' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DICU_NO_USER_DATA_OVERRIDE' '-DV8_GYP_BUILD' '-DV8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_PPC64' '-D_LINUX_SOURCE_COMPAT=1' '-D_ALL_SOURCE=1' '-DV8_EMBEDDER_STRING="-node.11"' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT=1' '-DOBJECT_PRINT' '-DV8_INTL_SUPPORT' '-DV8_ATOMIC_OBJECT_FIELD_WRITES' '-DV8_ENABLE_LAZY_SOURCE_POSITIONS' '-DV8_USE_SIPHASH' '-DNDEBUG' '-DV8_WIN64_UNWINDING_INFO' '-DV8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH' '-DV8_USE_ZLIB' '-DV8_ENABLE_LEAPTIERING' '-DV8_ENABLE_SPARKPLUG' '-DV8_ENABLE_TURBOFAN' '-DV8_ENABLE_WEBASSEMBLY' '-DV8_ENABLE_JAVASCRIPT_PROMISE_HOOKS' '-DV8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA' '-DV8_ALLOCATION_FOLDING' '-DV8_ALLOCATION_SITE_TRACKING' '-DV8_ADVANCED_BIGINT_ALGORITHMS' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=1' '-DUCONFIG_NO_BREAK_ITERATION=0' '-DTOOLCHAIN_MISS_ASM_HWCAP_H' '-DHWY_BROKEN_EMU128=0' -I../deps/v8 -I../deps/v8/include -I/home/iojs/sxa/node/out/Release/obj/gen/inspector-generated-output-root -I../deps/v8/third_party/inspector_protocol -I/home/iojs/sxa/node/out/Release/obj/gen -I/home/iojs/sxa/node/out/Release/obj/gen/inspector-generated-output-root/include -I/home/iojs/sxa/node/out/Release/obj/gen/generate-bytecode-output-root -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common -I../deps/v8/third_party/zlib -I../deps/v8/third_party/zlib/google -I../deps/v8/third_party/fp16/src/include -I../deps/v8/third_party/highway/src -I../deps/v8/third_party/simdutf -I../deps/v8/third_party/abseil-cpp  -pthread -Wno-unused-parameter -maix64 -Wno-attributes -Wno-strict-overflow -Wno-return-type -Wno-int-in-bool-context -Wno-deprecated -Wno-stringop-overflow -Wno-stringop-overread -Wno-restrict -Wno-array-bounds -Wno-nonnull -Wno-dangling-pointer -flax-vector-conversions -ffp-contract=off -mcpu=power5+ -mfprnd -mno-popcntb -maix64 -fdollars-in-identifiers -fno-extern-tls-init -O3 -fno-omit-frame-pointer -fdata-sections -ffunction-sections -O3 -fno-rtti -fno-exceptions -fno-strict-aliasing -std=gnu++20 -Wno-invalid-offsetof -MMD -MF /home/iojs/sxa/node/out/Release/.deps//home/iojs/sxa/node/out/Release/obj.target/v8_base_without_compiler/deps/v8/src/handles/global-handles.o.d.raw   -c
Timeout, server 169.54.113.165 not responding.
sxa:nodebuild$

The AIX 7.3 machine with 8GiB of RAM struggled with -j8 and caused processes to be killed (actualyl most of the subjobs disappeared and then the underlying make froze, which was a bit odd...) -j5 seems like it might be safe though so that's what I've used for the times on that machine in the table above.

[sxa]

Just to note, we need to do the testing for the worst case scenario -- no populated ccache (or near to 100% miss rate). Problems usually arose with V8-canary or V8 update PRs.

We don't have any separate V8 build jobs in our CI for AIX do we? Confirmed answer as "No" so that should not affect what we set these values to on the machines.