mamba2 mixer different outputs between fast, slow, training, and inference paths.

[tsengalb99]

System Info

- huggingface_hub version: 0.31.4
- Platform: Linux-5.15.0-140-generic-x86_64-with-glibc2.35
- Python version: 3.13.0
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /home/alberttseng/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: at676
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.9.0.dev20250803+cu129
- Jinja2: 3.1.6
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: 11.2.1
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 2.2.6
- pydantic: 2.10.3
- aiohttp: 3.11.18
- hf_xet: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/alberttseng/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/alberttseng/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/alberttseng/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /home/alberttseng/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

Who can help?

@ArthurZucker @Cyrilvallez

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

In

transformers/src/transformers/models/mamba2/modeling_mamba2.py

Line 656 in 0720e20

def forward(

, when given the same input to hidden_states and setting the other arguments to None, I get different results for the fast path (self.cuda_kernels_forward) with self.training = True, the fast path with self.training = False, and the slow path.

Expected behavior

These should all be equivalent.

[vasqu]

Hey @tsengalb99,

what model are you using to check? The bigger the model the easier it is to get discrepancies as different (non) kernels are involved.

The slow path is known to have a larger difference especially in lower precision contexts, any improvements there are appreciated. For the training vs inference path (in fast path), this is likely due to having a completely fused kernel mamba_split_conv1d_scan_combined vs. applying 2 kernels one after the other, i.e. causal_conv1d_fn then mamba_chunk_scan_combined. We strictly follow the original codebase at https://github.com/state-spaces/mamba/tree/main for applying the kernels. So if this issue appears here, then it should also appear there as well, i.e. it's hard to solve on our side.

Tl;dr: Fast Path - different kernel applications can easily introduce fluctuations and would likely appear in the original codebase as well. Slow Path - known to be less precise and is more optimized towards speed than accuracy.

[tsengalb99]

I am trying to train my own model that uses a single Mamba2 decoder layer inside of it. I'm getting a very large discrepancy (not numerical error, but >>100% error), so it seems like there is a significant implementation difference between the different paths.

[vasqu]

Can you share your config for the model / a script for reproducibility? Imo, it is still not easy to say whether this is an issue in the orginal kernels rather than the usage of them.

The backward pass in the mamba kernels is non-deterministic just fyi. Possible relevant issues

• The problem of Mamba block input consistency but output fluctuation in model validation environment state-spaces/mamba#548
• Some questions about different implementations of the SSD algorithm state-spaces/mamba#545
• The Accuracy Problem of Mamba Operator state-spaces/mamba#75