Skip to content

Conversation

EwoutH
Copy link
Member

@EwoutH EwoutH commented Oct 7, 2025

Problem

When using batch_run() with a single seed value and multiple iterations, all iterations use the same seed, producing identical results instead of independent replications.

parameters = {'seed': 42}
batch_run(MyModel, parameters, iterations=10)
# All 10 iterations use seed=42 → identical results

See #2835.

Solution

Modify _model_run_func to automatically increment the seed for each iteration: 42, 43, 44, etc.

if 'seed' in kwargs and kwargs['seed'] is not None and iteration > 0:
    seed_value = kwargs['seed']
    if isinstance(seed_value, (int, float)) and not isinstance(seed_value, bool):
        kwargs = kwargs.copy()
        kwargs['seed'] = int(seed_value) + iteration

Behavior changes

  • seed=42, iterations=3: currently all use 42, now uses 42, 43, 44
  • seed=[42, 43, 44], iterations=1: unchanged
  • No seed specified: unchanged (random)

Code that passes a single seed with multiple iterations will get different results. The current behavior seems like a bug (why run multiple identical iterations?), but this technically breaks existing code.

Review

I'm in doubt about this. What if users change have other random elements in their model? Do we do good obscuring this?

Secondly, is this a bugfix or a breaking change? Should we treat it as a fix and merge, or wait for a major version?

Might close #2835. @dylan-munson curious what you think.

When using batch_run() with a single seed value and multiple iterations, all iterations were using the same seed, producing identical results instead of independent replications. This defeats the purpose of running multiple iterations.

This commit modifies _model_run_func to automatically increment the seed for each iteration (seed, seed+1, seed+2, ...) when a numeric seed is provided. This ensures:

- Each iteration produces different random outcomes
- Results remain reproducible (same base seed → same sequence)
- Backward compatibility with seed arrays (no modification if seed is already an iterable passed via parameters)
- Unchanged behavior when no seed is specified (each iteration gets random seed from OS)

The fix only applies when:
1. A 'seed' parameter exists in kwargs
2. The seed value is not None
3. The iteration number is > 0
4. The seed is a single numeric value (int/float, not bool)
@EwoutH EwoutH added bug Release notes label breaking Release notes label labels Oct 7, 2025
Copy link

github-actions bot commented Oct 7, 2025

Performance benchmarks:

Model Size Init time [95% CI] Run time [95% CI]
BoltzmannWealth small 🔵 +0.5% [-0.2%, +1.2%] 🔵 -0.1% [-0.2%, +0.1%]
BoltzmannWealth large 🔵 -0.3% [-1.4%, +0.8%] 🔵 +0.1% [-3.2%, +3.4%]
Schelling small 🔵 -0.9% [-1.2%, -0.6%] 🔵 -2.1% [-2.5%, -1.6%]
Schelling large 🔵 -1.5% [-2.2%, -0.6%] 🔵 -2.6% [-6.8%, +1.4%]
WolfSheep small 🔵 -0.9% [-1.3%, -0.6%] 🔵 +1.0% [+0.7%, +1.2%]
WolfSheep large 🔵 +2.1% [+1.1%, +3.0%] 🔴 +5.4% [+4.1%, +6.7%]
BoidFlockers small 🔵 -1.5% [-2.3%, -0.5%] 🔵 -0.7% [-1.0%, -0.4%]
BoidFlockers large 🔵 -1.3% [-2.2%, -0.4%] 🔵 -0.9% [-1.6%, -0.1%]

@EwoutH EwoutH requested a review from quaquel October 7, 2025 19:08
@quaquel
Copy link
Member

quaquel commented Oct 7, 2025

I agree that this needs to be fixed. However, using subsequent integers with Mersenne Twister, Python's default RNG, is a bad idea.

From wikipedia: "A consequence of poor diffusion is that two instances of the generator, started with initial states that are almost the same, will usually output nearly the same sequence for many iterations". Also, using a seed with many zeros (like 42) is actually bad as well. One option is to just use time.time() every single time and return this seed value for reproducibility.

As an aside, numpy's rng is much better and I believe we should move all mesa code over to using this while deprecating the use of python's stdlib random library.

@tpike3
Copy link
Member

tpike3 commented Oct 8, 2025

Considering how important this is, maybe we should just go all in and do the switch to numpy and its rng and then have seed options like system time and hierarchical seeding?

Does it have to be a breaking change? Could we keep the old behavior and just add a warning?

@quaquel
Copy link
Member

quaquel commented Oct 8, 2025

Does it have to be a breaking change? Could we keep the old behavior and just add a warning?

Moving the internals over should be possible as a non-breaking change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking Release notes label bug Release notes label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants