Batch Backtesting and Parallel Optimization Support

[xsa-dev]

Is your feature request related to a problem? Please describe.
Running multiple backtests in Jesse (for example, scanning many coin pairs or strategies) is currently very slow and cumbersome when done one by one. After each code change or parameter tweak, I have to backtest strategies sequentially, which wastes a huge amount of time. There’s no easy way to batch or parallelize these backtests in one go. For instance, if a single backtest takes 1 minute and I want to run 100 variations, it ends up taking ~100 minutes of waiting . This is frustrating because it prevents me from fully utilizing my CPU cores and makes large-scale experiments almost impractical. Moreover, each run reloads the same historical data separately, adding redundant data loading overhead for every backtest . In summary, the lack of batch processing (and parallel execution) means slower research iterations, under-utilized hardware, and missed opportunities to discover better strategies or optimize parameters quickly.

Describe the solution you’d like.
I would like to have a batch backtesting feature with parallel execution support. In practice, this means an ability to define a set of backtest jobs (different timeframes, symbols, strategies, or parameter configurations) and run them as a single batch task. The system would handle loading the required data once and reuse it across tests whenever possible, avoiding repetitive disk reads. Ideally, these batch backtests could run in parallel across multiple CPU cores or machines – for example, by leveraging a framework like Ray for distributed processing – so that we can dramatically speed up overall runtime. From a user perspective, this could be a CLI command or function (e.g. a “screener” or bulk backtest command) where we list multiple routes/strategies or multiple parameter sets, and Jesse executes them concurrently and then outputs a comparative result summary. The key is to have a scalable, parallelizable pipeline for backtesting and optimization, so that researchers can efficiently utilize all available computing power and integrate Jesse into machine learning workflows (such as hyperparameter tuning or reinforcement learning loops) without being bottlenecked by sequential execution.

Describe alternatives you’ve considered.
• Sequential Scripts: Writing custom scripts to loop through strategies one by one. This works for small numbers of tests but becomes impractically slow for large batches, and it still reloads data each time.
• Multiple Processes Manually: Manually running multiple Jesse instances or processes in parallel (or using Python multiprocessing). This is complex to set up and not officially supported, and can run into issues with data downloading or memory use since there’s no shared data context.
• Jesse’s Benchmark Mode: The new benchmark page/feature allows running a set of backtests and comparing results, which is helpful, but as far as I know it still executes them sequentially (and within the UI). It doesn’t fully utilize parallel CPU cores or make it easy to script large-scale experiments or integration with other tools.
• Reduced Testing: In some cases, I’m forced to test on smaller data ranges or fewer variations because doing everything sequentially would take too long. This means potentially missing out on finding the best strategies or parameters due to time constraints.

Additional context.
This request was inspired by a suggestion from Saleh in a recent YouTube video (the “Crypto Coin Screener in Python” video on the Algo-trading with Saleh channel). In that video, Saleh acknowledges the need for batch processing of data loading and backtesting, and encouraged users to request this feature. I want to express my strong support and solidarity for that idea: a full-fledged batch backtesting capability (with parallel, scalable execution) would be a game-changer for Jesse. It would allow us to build much more efficient research pipelines, especially when using machine learning libraries or doing exhaustive strategy optimization. Some community effort has already started in this direction – for example, I’ve begun a draft implementation in PR #537 (titled “Screener backtest batches and optimization routes & batches”) to explore the concept. This highlights that there is significant interest in the community for such functionality. I believe implementing batch parallel backtesting (potentially with a powerful library like Ray for scaling) will unlock faster experimentation and help Jesse users get more value out of the framework.