fix: parallel generation, storage env vars, MPI topology, settle guard, CI thread cap

[russfellows]

Overview

This PR bundles four related improvement areas that together make dlio_benchmark more correct on multi-node clusters, significantly faster at data generation, and usable without a handcrafted YAML file when working with cloud object stores.

All changes are backwards-compatible: default values preserve existing behaviour.

Issues closed: #9, #10, #11, #12, #13 (and clarification #6b)
New tests: 28 tests added in tests/test_remaining_issues.py, all pass.

---

Change Details

1 — MPI Topology Fix for Thread Auto-Sizing (Issue 12)

Files: dlio_benchmark/utils/utility.py, dlio_benchmark/utils/config.py

Problem: read_threads auto-sizing divided os.cpu_count() by comm_size (total world size). On a 4-node × 16-rank job this produced 1 thread per rank instead of the correct 16-CPU ÷ 4-ranks-per-node = 4 threads per rank.

Fix: Added DLIOMPI.ranks_per_node() — a topology-aware method that returns the number of MPI ranks on the current node. It is safe to call in CHILD_INITIALIZED state (spawned workers) where mpi_ppn_list is not yet set; it falls back to comm_size as a conservative estimate. Both read_threads and write_threads auto-sizing now use ranks_per_node().

Before (4-node, 16-rank job, 64-core nodes):
  read_threads = cpu_count // comm_size = 64 // 16 = 4  ← wrong

After:
  read_threads = cpu_count // ranks_per_node = 64 // 4 = 16  ← correct

---

2 — Parallel Data Generation (Issues 10, 11, 6b)

Files: dlio_benchmark/utils/config.py, dlio_benchmark/data_generator/data_generator.py

A new write_threads: int field is added to ConfigArguments (default 1, auto-sized like read_threads). DataGenerator._generate_files() is rewritten with a two-phase design:

• Phase 1 (main thread, sequential): all (index, dims, file_seed, output_path) tuples are pre-computed from the master RNG to ensure determinism regardless of thread count.
• Phase 2 (parallel): each worker gets its pre-computed file_seed and a fresh np.random.default_rng(seed=file_seed) — no shared mutable state. For object stores, N threads = N concurrent uploads (Issue 11).

Determinism guarantee: write_threads=1 and write_threads=8 produce byte-for-byte identical files for the same config and master seed.

A clarifying comment was also added to DataGenerator.__init__() confirming that the early derive_configurations() call is intentional (Issue 6b).

---

3 — Storage Environment-Variable Overrides (Issue 9)

File: dlio_benchmark/utils/config.py — _apply_env_overrides()

Map well-known environment variables into ConfigArguments fields. Existing YAML / CLI values always win; env vars only fill fields still at their default.

Environment variable	Config field
DLIO_STORAGE_TYPE	args.storage_type
DLIO_BUCKET	args.storage_root
DLIO_STORAGE_LIBRARY	args.storage_options['storage_library']
AWS_ACCESS_KEY_ID	args.storage_options['access_key_id']
AWS_SECRET_ACCESS_KEY	args.storage_options['secret_access_key']
AWS_ENDPOINT_URL	args.storage_options['endpoint_url']
AWS_REGION	args.storage_options['region']

---

4 — Post-Generation Settle Guard for Object Stores (Issue 13)

Files: dlio_benchmark/utils/config.py, dlio_benchmark/main.py

New optional configuration parameter:

storage:
  post_generation_settle_seconds: 5.0   # default: 0.0 (disabled)

After generation completes, _apply_settle_guard(args, comm) is called:

• No-op when storage_type == LOCAL_FS or settle == 0.0 — zero behaviour change for existing configs.
• Active otherwise: rank 0 sleeps for the configured duration, then all ranks barrier together so they proceed once the store has settled.

---

5 — CI Thread Cap (DLIO_MAX_AUTO_THREADS)

DLIO_MAX_AUTO_THREADS environment variable caps the auto-sized thread count (default 8 in production, set to 2 in CI via .github/workflows/ci.yml and tests/conftest.py). This prevents resource contention when the full test suite runs on shared CI runners.

---

Tests

New file: tests/test_remaining_issues.py — 28 tests:

Test class	Tests	Covers
TestIssue12_RanksPerNode	4	ranks_per_node() correctness and safety
TestIssue12_AutoSizingDenominator	1	read_threads uses ranks_per_node()
TestIssue10_WriteThreadsField	3	write_threads field, default, auto-size
TestIssue10_ParallelGeneration	4	ThreadPoolExecutor used, files created, determinism, Issue 6b comment
TestIssue9_StorageEnvOverrides	11	All env vars applied; YAML wins; priority ordering
TestIssue13_SettleGuard	5	Field exists, default=0, sleep called for S3, not for LOCAL_FS, not for settle=0

Full suite result: 94 passed, 32 skipped.

[russfellows]

Working on fixing the problematic "pydftracer" code, and also limiting the tests to use only Python 3.12. Fingers crossed.

[russfellows]

It turns out that problematic DFTracer isn't used in the REAL code, but used extensively during the CI tests. So needed to restore it. I tried to disable requiring it, which worked when running real tests, but not CI tests. Fun stuff.

[russfellows]

Still lots of issues with the massive number of tests all testing (now) incorrect assumptions, like "did we manipulate an image correctly?" The answer now is "no, and we don't care." So lots of bad, outdated assumptions baked into these hundreds of tests cases.

[russfellows]

Update the CI tests. Now they are LIGHT years faster.

[russfellows]

Fixed missing pyarrow in requirements.txt