Save Parquet File for Single-cells during Colony Simulation without MongoDB?

[katha815]

Problem Description

I am using ecoli_engine_process.py for colony simulation under baseline conditions. While I can successfully save snapshots of the colony state, I continuously encounter errors when trying to emit Parquet files for single-cell data. I wonder if anyone has successfully done this without an online emitter?

The errors and attempted debugging steps are shown below. I noticed the implemented antibiotic simulation uses an online database as the emitter, so I included observations about that in the last section.

It could be a different underlying reason why my simulation failed, and I would appreciate any insights or suggestions!

Environment

• vEcoli commit: (current main branch)
• Python 3.12
• Config: spatial.json inheritance with "emitter": "parquet"

Goal

Resume a colony simulation from a saved JSON state file (baseline_2gen_seed_0_colony_t6000.json) and output to Parquet files.

My Configuration File

configs/colony_baseline_test2.json:

{
    "inherit_from": ["spatial.json"],
    "description": "Test2: read from trial1, run for another generation, save parquet per cell, save final state",
    "initial_colony_file": "baseline_2gen_seed_0_colony_t6000",

    "seed": 0,
    "sim_data_path": "out/all_media_conditions1/parca/kb/simData.cPickle",
    
    "emitter": "parquet",
    "emitter_arg": {
        "out_dir": "out/colony_runs/baseline_3rd_gen_seed_0"
    },
    "emit_config": false,

    "max_duration": 9000,
    
    "save": true,
    "save_times": [9000],
    "colony_save_prefix": "baseline_3rd_gen",
    
    "parallel": false,
    
    "engine_process_reports": [
        ["boundary"],
        ["bulk"],
        ["listeners"],
        ["environment", "exchange"]
    ]
}

---

Issue 1: NumpyRandomStateSerializer Serialization/Deserialization Mismatch

Command:

python ecoli/experiments/ecoli_engine_process.py --config configs/colony_baseline_test2.json

Error:

Traceback (most recent call last):
  File "ecoli/experiments/ecoli_engine_process.py", line 526, in <module>
    run_simulation(config)
  File "ecoli/experiments/ecoli_engine_process.py", line 389, in run_simulation
    initial_state = get_state_from_file(...)
  File "ecoli/library/json_state.py", line 168, in get_state_from_file
    return json.loads(f.read(), object_hook=custom_decoder)
  File "ecoli/library/serialize.py", line 83, in deserialize
    data = orjson.loads(data)
orjson.JSONDecodeError: unexpected character: line 1 column 1 (char 0)

Possible Root Cause:

• serialize() at line 70-72 appears to output Python tuple format: ('MT19937', [...])
• deserialize() at line 82 uses orjson.loads() which expects JSON array format: ["MT19937", [...]]

File: ecoli/library/serialize.py

Original code (lines 78-85):

def deserialize(self, data):
    matched_regex = self.regex_for_serialized.fullmatch(data)
    if matched_regex:
        data = matched_regex.group(1)
    data = orjson.loads(data)
    rng = np.random.RandomState()
    rng.set_state(data)
    return rng

Attempted Fix: Replace orjson.loads() with ast.literal_eval() to handle Python tuple format:

def deserialize(self, data):
    import ast
    matched_regex = self.regex_for_serialized.fullmatch(data)
    if matched_regex:
        data = matched_regex.group(1)
    if data.startswith("("):
        data = ast.literal_eval(data)
    else:
        data = orjson.loads(data)
    rng = np.random.RandomState()
    rng.set_state(tuple(data))
    return rng

---

Issue 2: Parquet Emitter Cannot Handle pint.Quantity in Config Metadata

Error (after attempting to fix Issue 1):

Traceback (most recent call last):
  File "ecoli/library/parquet_emitter.py", line 963, in emit
    v = np.asarray(v, dtype=np_dtype(v, k))
  File "ecoli/library/parquet_emitter.py", line 706, in np_dtype
    raise ValueError(f"{field_name} has unsupported type {type(val)}.")
ValueError: spatial_environment_config__multibody__bounds has unsupported type <class 'pint.Quantity'>.

During handling of the above exception, another exception occurred:
  File "ecoli/library/parquet_emitter.py", line 967, in emit
    v = pl.Series([v])
TypeError: not yet implemented: Nested object types

Possible Root Cause:

• spatial.json contains pint.Quantity values (e.g., "!units[50 micrometer]")
• np_dtype() doesn't appear to handle pint.Quantity, and falls back to Polars
• pl.Series([v]) also seems unable to handle pint.Quantity objects

File: ecoli/library/parquet_emitter.py

Original code (line 967):

v = pl.Series([v])

Attempted Fix: Convert unsupported types to string:

v = pl.Series([str(v)])

---

Issue 3: emit_config Setting Not Passed to Engine

Note: Even after the str(v) fix, I attempted to disable config emission via JSON config "emit_config": false, but it appeared to have no effect.

Possible Root Cause:
ecoli_engine_process.py does not seem to pass the emit_config parameter to the Engine constructor.

File: ecoli/experiments/ecoli_engine_process.py

Original code (around line 465):

engine = Engine(
    processes=composite.processes,
    topology=composite.topology,
    initial_state=initial_state,
    experiment_id=experiment_id,
    emitter=emitter_config,
    progress_bar=config["progress_bar"],
    metadata=metadata,
    profile=config["profile"],
    initial_global_time=config.get("start_time", 0.0),
)

Attempted Fix: Add emit_config parameter:

engine = Engine(
    ...
    initial_global_time=config.get("start_time", 0.0),
    emit_config=config.get("emit_config", False),
)

---

Issue 4: Parquet Emitter Assumes agents Key in Data Structure

Error (after attempting to fix Issues 1-3):

Traceback (most recent call last):
  File "ecoli/experiments/ecoli_engine_process.py", line 485, in run_simulation
    colony_save_states(engine, config)
  File "ecoli/experiments/ecoli_engine_process.py", line 255, in colony_save_states
    engine.update(time_to_next_save)
  ...
  File "ecoli/processes/engine_process.py", line 505, in next_update
    self.emitter.emit(emit_config)
  File "ecoli/library/parquet_emitter.py", line 1007, in emit
    if len(data["data"]["agents"]) > 1:
KeyError: 'agents'

Possible Root Cause:

• ParquetEmitter.emit() appears to expect a data["data"]["agents"] structure (outer simulation)
• EngineProcess inner emitter seems to send cell data directly without the agents wrapper
• The inner emitter is configured via inner_emitter in the EngineProcess config

File: ecoli/library/parquet_emitter.py (line 1007)

---

Observations on Implemented Antibiotic Simulation

The tet_amp_sim.py uses a different configuration that appears to avoid these issues:

# From configs/cloud.json (inherited by antibiotics.json)
{
    "emitter": "database",  # MongoDB, not Parquet
    "emitter_arg": {"host": "10.138.0.75:27017", "emit_limit": 4100000}
}

• MongoDB can handle arbitrary Python objects including pint.Quantity
• No JSON serialization issues with NumpyRandomState
• No agents key structure assumptions

---

Summary Table

Issue	File	Line	Status
1. RandomState serialize/deserialize mismatch	serialize.py	78-85	Attempted fix with ast.literal_eval()
2. pint.Quantity not supported	parquet_emitter.py	967	Attempted fix with str(v)
3. emit_config not passed to Engine	ecoli_engine_process.py	~470	Attempted fix by adding parameter
4. Missing agents key handling	parquet_emitter.py	1007	UNRESOLVED

[thalassemia]

Thanks for trying out the model and for the issue report. I developed the Parquet emitter and all the workflow infrastructure after the colony publication. Since no one in the lab is running colony simulations right now, I did not test or update the colony code (including ecoli_engine_process.py) to support the new emitter and other infrastructure. That is also why colonies are not mentioned in our public documentation.

If you are interested in running colony simulations, your best bet is to use the release of the model associated with the publication, which used MongoDB as the exclusive emitter. I would not recommend it though as that version lacks a lot of the updates, optimizations, and enhancements of the latest version. Is there a specific reason why you are interested in colony simulations? If not, I would recommend running batches of simulations using workflows instead.

If you want to try modernizing ecoli_engine_process.py, I'll address the issues you've encountered one-by-one.

Issue 1: NumpyRandomStateSerializer Serialization/Deserialization Mismatch

This is a legitimate bug that I was unaware of because none of our current workflows emit NumPy RandomState objects. Feel free to create a PR with the proposed fix.

Issue 2: Parquet Emitter Cannot Handle pint.Quantity in Config Metadata

I became aware of this recently and fixed it in ecoli_master_sim.py on our private clone of the repo. A big batch of these private changes should make their way into the public repo over the next month or so. You can make the same fix in ecoli_engine_process.py by calling serialize_value on the metadata created in this line.

Issue 3: emit_config Setting Not Passed to Engine

This option is confusingly named. It does not control whether the vEcoli config JSON is emitted. That config JSON is included in metadata as described above, which is always emitted. Instead, this is a vivarium-core option that tells the Engine whether to include the serialized initial state in the configuration emit. That being said, the fact that the emit_* options do not currently do anything in ecoli_engine_process.py is a real bug (they are honored in ecoli_master_sim.py, which we actively use). Feel free to make a PR adding those options. I'll just note that these are already set to False by default (here in vEcoli and here in vivarium-core), so setting them False in your config will not do anything even after this fix.

Issue 4: Parquet Emitter Assumes agents Key in Data Structure

This is an interesting issue. Instead of having the Parquet emitter handle both types of simulation states, I think it makes more sense to modify the inner simulation of ecoli_engine_process.py to match the simulation state of ecoli_master_sim.py, which puts everything under ("agents", <agent ID>,). This would be a more involved fix, but I prefer that to adding further complexity to the Parquet emitter. Feel free to give that a shot if you want.

[katha815]

Thank you for your quick response! I am using the colony simulation because I am interested in the E. Coli colony behaviour generally, and think it would be more beneficial to apply any ML accelerator in colony content. I am in an early stage of this research project, so I haven't had a specialised case for the simulation.

Therefore, I am trying to run ecoli_engine_process.py instead of tet_amp_sim.py - because I think the latter includes some customised processes for antibiotic response?

Sure, I can send the pull requests if these fixes are valid.

For your suggestion about Issue 4, can you explain a bit more what you would recommend to do the fix? It sounds better for me if I can avoid setting up MongoDB for the current simulation.

On the other hand, will the referred release of the model include information about MongoDB setup guidance? I would appreciate it if you could give me some guidance about it.

Thanks again!

[thalassemia]

At the moment, the only advantage of the colony model is that you can place all the cells in a shared environment. However, there are currently very few processes that leverage this fact. We have a process that models each cell as a physical entity that experiences Brownian motion and displaces other cells (but does not model cell adhesion, pili, flagella, etc.). We also have processes that model the diffusion and reaction of species in the external environment (e.g. diffusion of antibiotics and hydrolysis by beta-lactamases released during cell lysis).

To be perfectly honest, the value that we got from colony sims in the antibiotics paper was greatly limited by the computational demands of running so many concurrent simulations. For reference, we ran all the simulations in that paper using the largest VMs on Google Cloud at the time (224 vCPUs, 896GB RAM). Even then, it took many hours just to reach 512 cells (at which point, we hit the memory limit). Long story short, we got almost no new insight from simulating so few cells in the same environment instead of running thousands of independent, non-interacting cell lineages (vertical vs horizontal scaling).

Almost all ongoing work in the lab is being done using independent simulations. This paradigm is what the new workflow infrastructure was built to support. Unless you have access to extremely well-equipped machines, you would be able to generate a lot more meaningful data using workflows instead of colony sims.

I'll work on addressing issue number 4 in a separate PR. The EngineProcess class is crazy complicated, and it's taking me a moment to remember how everything works. I have most of the issues ironed out, but one open question is how to handle emits from the outer simulation (external environment, cell positions) vs the inner simulation (internal molecule counts, etc). The best idea I have right now is to emit their data under two different experiment IDs, appending _outer and _inner to the user-configured experiment ID.

Regarding MongoDB, the MongoDB emitter is built into vivarium-core, so you can refer to their documentation for setup instructions. I just want to note that we no longer use the MongoDB emitter in our current work, so the query patterns and analysis scripts that you write for that output will not transfer.

[thalassemia]

Long story short, there are a bunch of changes in our private repo that we are unfortunately not able to bring into the public repo right now, including many important bug fixes and improvements. One of those bug fixes solves the test failure on your PR (see here). Other improvements simplify the process of adding support for arbitrary SLURM clusters, which would make it even easier for you to run very large batches of sims through the workflow API. I also want to point out it is already possible to run workflows on arbitrary SLURM clusters following these slightly more complicated instructions.

With regards to colony sims, I spent some time today fixing issue 4 on this branch of my fork. Feel free to try this code, but be warned that I don't have any spare bandwidth to work on this and cannot provide any further documentation or guarantees. As such, I'd still strongly recommend sticking to workflows instead of colony sims, if possible.

Sorry for the huge delay and for not being able to help you further with this. Thanks again for your patience and interest in the model!