1 Release published by 1 person

• v2.3.0 NVIDIA Neural Modules 2.3.0

  published May 8, 2025

88 Pull requests merged by 37 people

• Remove adapter_path from base AutoResume and refactor PEFT checkpoint handling

  #12565 merged May 11, 2025

• Add fp8_param argument back to mixed precision plugin for backward compatibility

  #13522 merged May 10, 2025

• add memory profile interface to perf scripts

  #13413 merged May 10, 2025

• Refactor Distillation PP support to use new MCore API

  #13065 merged May 10, 2025

• Add Llama4 GHA

  #13442 merged May 9, 2025

• Add Resume_path to llama_nemotron models

  #13515 merged May 9, 2025

• Improve error message when HF checkpoint cannot be loaded

  #13513 merged May 9, 2025

• chore(🤖): Bump NVIDIA/Megatron-LM to bcbede5... (2025-05-09)

  #13510 merged May 9, 2025

• ci: Add more files to filter

  #13517 merged May 9, 2025

• add extra hyena tests

  #13097 merged May 9, 2025

• Autodetect model_type and dtype for deployment using TRT-LLM backend

  #13209 merged May 9, 2025

• Cherry pick Adding more export tests (13410) into r2.3.0

  #13450 merged May 9, 2025

• Cherry pick Adding additional unit tests for the deploy module (13411) into r2.3.0

  #13449 merged May 9, 2025

• Cherry pick new perf configs (13110) into r2.3.0

  #13431 merged May 9, 2025

• Cherry pick Handle boolean args for performance scripts and log received config (13291) into r2.3.0

  #13416 merged May 9, 2025

• Cherry pick Allow fp8 param gather when using FSDP (13267) into r2.3.0

  #13383 merged May 9, 2025

• Cherry pick Fix skipme handling (13244) into r2.3.0

  #13376 merged May 9, 2025

• Cherry-pick Add recipe and ci scripts for qwen2vl to r2.3.0

  #13336 merged May 9, 2025

• Cherry pick [automodel] bump liger-kernel to 0.5.8 + fallback (13260) into r2.3.0

  #13308 merged May 9, 2025

• Cherry pick build: MAMBA_TAG=2e16fc3062cdcd4ebef27a9aa4442676e1c7edf4 (13173) into r2.3.0

  #13263 merged May 9, 2025

• Cherry pick Use explicitly cached canary-1b-flash in CI tests (13237) into r2.3.0

  #13508 merged May 9, 2025

• Bump to 2.3.1

  #13507 merged May 9, 2025

• Updating the long context performance number for B200

  #13468 merged May 8, 2025

• Publish 2.3.0

  #13506 merged May 8, 2025

• Cherry pick Update changelog for r2.3.0 (13501) into r2.3.0

  #13502 merged May 8, 2025

• Fix changelog formatting

  #13505 merged May 8, 2025

• Enabling flash decode for float16 precision only

  #13471 merged May 8, 2025

• Update 2.3.0 changelog

  #13504 merged May 8, 2025

• ci: Remove trt-llm breakpoint

  #13499 merged May 8, 2025

• Update 2.3.0 changelog

  #13503 merged May 8, 2025

• Update changelog for r2.3.0

  #13501 merged May 8, 2025

• [automodel] fallback FP8 + LCE -> FP8 + CE

  #13349 merged May 8, 2025

• ci: Run selective triggering on dockerfiles and dependencies

  #13493 merged May 8, 2025

• added use-fast tokenizer argument

  #12986 merged May 8, 2025

• Move libsox-fmt-all from Dockerfile.ci.export_deploy to Dockerfile.ci

  #13452 merged May 8, 2025

• Support customization of a few parameters in scripts/vlm/llava_next_pretrain

  #12218 merged May 7, 2025

• ci: Upload on schedule

  #13491 merged May 7, 2025

• ci: No runs on main

  #13490 merged May 7, 2025

• Fix typo in the performance script

  #13487 merged May 7, 2025

• Ko3n1g/ci/selective triggering 4

  #13489 merged May 7, 2025

• ci: Do not run any tests if no match is found

  #13479 merged May 7, 2025

• set ignore_virtual to is_pipeline_first/last_stage given a swap of ig…

  #13160 merged May 7, 2025

• Mingyuanm/GitHub ci flux

  #12761 merged May 7, 2025

• Add Warning to Export when output_path exists

  #13465 merged May 7, 2025

• Add profiling changes

  #13484 merged May 7, 2025

• Distillation with NMH-4B

  #13485 merged May 7, 2025

• ci: Bump dependencies

  #12819 merged May 7, 2025

• Fix BNR 2 unit test + input, case where input length was not specified

  #13467 merged May 7, 2025

• chore(🤖): Bump NVIDIA/Megatron-LM to d580efc... (2025-05-07)

  #13475 merged May 7, 2025

• Remove cuda method from ModelPT

  #13394 merged May 7, 2025

• Migrate Hyena to Megatron inference_context.

  #13436 merged May 7, 2025

• [Docs] Fix incorrectly formatted reference tags

  #13445 merged May 7, 2025

• Fix llava tokenizer caused nan issue

  #13466 merged May 6, 2025

• Use expandable cuda memory segmentation

  #13418 merged May 6, 2025

• Add CI test for local checkpointing

  #13012 merged May 6, 2025

• Streaming Sortformer Diarizer: model and module code [PR 2]

  #13201 merged May 6, 2025

• Fix full te layer forward

  #13243 merged May 6, 2025

• [Audio] tests for score-based and flow matching enhancement models

  #13406 merged May 6, 2025

• add more detailed description

  #13464 merged May 6, 2025

• Adding tests for Schroedinger Bridge model

  #13401 merged May 6, 2025

• ci: Enter queue only with passing linting

  #13462 merged May 6, 2025

• Add slice_with_offset and dry_run Support for Tar Dataset Creation; New Script for Partial Conversion

  #10511 merged May 6, 2025

• fix speechlm data module

  #13362 merged May 6, 2025

• ci: Disable broken neva tests

  #13461 merged May 6, 2025

• Ko3n1g/ci/selective triggering 3

  #13460 merged May 6, 2025

• [automodel] ignore tail padding in TPS calculation

  #13329 merged May 6, 2025

• [Audio] fix a flaky test (and also make some tests run faster)

  #13439 merged May 6, 2025

• Adding more export tests

  #13410 merged May 6, 2025

• Adding additional unit tests for the deploy module

  #13411 merged May 5, 2025

• Ko3n1g/ci/fix dependency tree

  #13448 merged May 5, 2025

• Ko3n1g/ci/fix dependency tree

  #13447 merged May 5, 2025

• Ko3n1g/ci/fix dependency tree

  #13444 merged May 5, 2025

• ci: Fix deps tree for tests

  #13443 merged May 5, 2025

• ci: Remove jq

  #13440 merged May 5, 2025

• Improve Nemo2Exporter for Models Using Custom Modelling Files on HF

  #13400 merged May 5, 2025

• Update flagged docs links

  #13391 merged May 5, 2025

• [automodel] rename returned keys from tokens to input_ids

  #13280 merged May 5, 2025

• ci: Add tests to selective triggering

  #13404 merged May 5, 2025

• ci: Success only if Run CICD label attached

  #13430 merged May 5, 2025

• Add NCCL cfg interface to perf scripts

  #13407 merged May 5, 2025

• ci: Skip link check on github links

  #13425 merged May 5, 2025

• Fix loss compute and reduction

  #13295 merged May 5, 2025

• ci: Disable flaky audio test

  #13435 merged May 5, 2025

• tests: Disable flaky audio test

  #13429 merged May 5, 2025

• HF-T5 exporter fixes + HF-AutoTokenizer fix

  #12899 merged May 5, 2025

• new perf configs

  #13110 merged May 5, 2025

• Add use sharp argument

  #13207 merged May 5, 2025

• [automodel] add FirstRankPerNode

  #13373 merged May 4, 2025

36 Pull requests opened by 24 people

• Fix masking of <pad> tokens in AED inference

  #13428 opened May 5, 2025

• [automodel] move examples

  #13434 opened May 5, 2025

• Add CallbackGroup & Metadata factory function

  #13437 opened May 5, 2025

• UCB r2.3.0

  #13441 opened May 5, 2025

• Move loop under LLM collection

  #13446 opened May 5, 2025

• Remove SDXL quantization tutorial

  #13453 opened May 6, 2025

• perf scripts updates

  #13456 opened May 6, 2025

• flux 12b fixes

  #13459 opened May 6, 2025

• initial peft implementation

  #13463 opened May 6, 2025

• ci: Remove optional marker

  #13469 opened May 6, 2025

• [Audio] TransformerUNet: predictive model support added

  #13470 opened May 6, 2025

• Cherry pick `Add CI test for local checkpointing (#13012)` into `r2.3.0`

  #13472 opened May 6, 2025

• [Automodel] Fix CP device_mesh issue, use PTL distsampler

  #13473 opened May 6, 2025

• use null tokenizer

  #13480 opened May 7, 2025

• Alit/nmh4b

  #13481 opened May 7, 2025

• ci: Bump dependencies (#12819)

  #13482 opened May 7, 2025

• deepseek finetuning callback error change

  #13483 opened May 7, 2025

• Incorporate 25.04 NeMo Patches

  #13488 opened May 7, 2025

• [NeMo 2.0] Nemotron 49B Super HF export Bugfix

  #13495 opened May 8, 2025

• ci: Enable codecov checks

  #13497 opened May 8, 2025

• Update vLLMExporter to use vLLM V1

  #13498 opened May 8, 2025

• Tdt buffered inference fix

  #13500 opened May 8, 2025

• remove blocks unused to increase coverage

  #13511 opened May 9, 2025

• remove the restriction of load_model_state_dict to cfsdp

  #13512 opened May 9, 2025

• Fixing #13509

  #13514 opened May 9, 2025

• HF export in nemo.export

  #13516 opened May 9, 2025

• Hyena SE B2B Kernel integration

  #13518 opened May 9, 2025

• Enable changing quant_cfg via command line in the ptq.py script

  #13519 opened May 9, 2025

• Add use_sharp and use user buffer registration args in perf scripts

  #13521 opened May 9, 2025

• Reconfigure 'limit_<train|val>_batches'

  #13523 opened May 9, 2025

• chore(🤖): Bump `NVIDIA/Megatron-LM` to `fa5eec6...` (2025-05-10)

  #13524 opened May 10, 2025

• chore(🤖): Bump `NVIDIA/Megatron-LM` to `f25dceb...` (2025-05-10)

  #13525 opened May 10, 2025

• Multi token bleu

  #13526 opened May 10, 2025

• Add NemotronH Performance Script

  #13528 opened May 10, 2025

• chore(🤖): Bump `NVIDIA/Megatron-LM` to `fa5eec6...` (2025-05-11)

  #13530 opened May 11, 2025

• chore(🤖): Bump `NVIDIA/Megatron-LM` to `16aeade...` (2025-05-11)

  #13531 opened May 11, 2025

2 Issues closed by 2 people

• Do you know that GeForce RTX 5090 Graphics Card was not supported by NeMO?

  #13423 closed May 6, 2025

• torch.distributed.DistNetworkError

  #12805 closed May 6, 2025

12 Issues opened by 12 people

• Distributed support for preprocess_data_for_megatron.py

  #13529 opened May 10, 2025

• How do you train with speech_to_text_finetune.py and specify Cer instead of Wer

  #13527 opened May 10, 2025

• cannot import name 'L2Norm' from 'megatron.core.transformer.torch_norm'

  #13520 opened May 9, 2025

• UsUsing MegatronCommOverlapCallback(tp_comm_overlap=True) causes segfault.

  #13509 opened May 9, 2025

• How to perform knowledge distillation for Hyena architecture models using NeMo?

  #13496 opened May 8, 2025

• how to ssl train a predictor network for rnnt asr models

  #13486 opened May 7, 2025

• Llama4Omni Export LoRA Weights to Hugging Face

  #13477 opened May 7, 2025

• Possible to convert EncDecRNNTBPEModel into TensorRT and run them through Triton inference server?

  #13457 opened May 6, 2025

• export_ckpt failed due to AssertionError: dtype mismatch between source and target state dicts

  #13455 opened May 6, 2025

• Tips for Gemma 3 1B pretraining

  #13438 opened May 5, 2025

• Missing Data files that are .JSONL FIles for Evaluation and Generating Preds

  #13432 opened May 5, 2025

• on GB200 python 3.12.3, reinstall.sh failed with latest tag v2.3.0.rc4

  #13426 opened May 4, 2025

62 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Add Parakeet Hybrid RNNT CTC BPE Model with target language support

  #13360 commented on May 8, 2025 • 36 new comments

• SpeechLM2 collection

  #12617 commented on May 9, 2025 • 33 new comments

• first commit

  #12477 commented on May 9, 2025 • 23 new comments

• SFT Script with HuggingFace chat template support

  #13273 commented on May 9, 2025 • 21 new comments

• Duplex s2s new speech decoder

  #13203 commented on May 7, 2025 • 10 new comments

• add CTC beam search

  #13337 commented on May 6, 2025 • 9 new comments

• Punctuation Marks in Timestamps

  #13353 commented on May 8, 2025 • 8 new comments

• Hugging face model deployment with Ray Serve

  #13395 commented on May 8, 2025 • 6 new comments

• Tests for evaluation with NVIDIA Evals Factory

  #12985 commented on May 9, 2025 • 5 new comments

• Batched streaming RNN-T and TDT: new buffered inference + support cache-aware

  #9106 commented on May 9, 2025 • 5 new comments

• feat - GPTSFTChatDataset alignment with OpenAI Messages, compatibility with packed sequences

  #13367 commented on May 10, 2025 • 4 new comments

• Implement Speculative transform script for GPT models

  #12863 commented on May 6, 2025 • 3 new comments

• NeMo2 -> vLLM export via HF

  #13304 commented on May 9, 2025 • 3 new comments

• Adding fix to absolute embedding

  #13414 commented on May 5, 2025 • 2 new comments

• Test Hyena mixer CP equivalency

  #13330 commented on May 9, 2025 • 2 new comments

• [automodel][draft] Integrate Megatron Custom FSDP2 into NeMo Automodel.

  #13250 commented on May 9, 2025 • 0 new comments

• Draft: Debug runner

  #13255 commented on May 10, 2025 • 0 new comments

• fix adapter in/out_features size for share_expert when use overlap.

  #13228 commented on May 10, 2025 • 0 new comments

• feat: add max_num_unfinalized_calls to AsyncFinalizerCallback

  #13221 commented on May 9, 2025 • 0 new comments

• ci: Add lint checks for function docstring parameters using pylilnt

  #13200 commented on May 8, 2025 • 0 new comments

• Added multi distopt knobs

  #13196 commented on May 9, 2025 • 0 new comments

• Update Deepseek config for performance + add guard for using HF model directly

  #13188 commented on May 9, 2025 • 0 new comments

• Add vlm api for nemo run

  #13180 commented on May 8, 2025 • 0 new comments

• Incompatible with numpy>2.0

  #12378 commented on May 4, 2025 • 0 new comments

• [DRAFT] LLama4 + nemo.tron

  #13265 commented on May 8, 2025 • 0 new comments

• Remove fp8 model init context because it is handled by MCORE

  #13290 commented on May 7, 2025 • 0 new comments

• [audio] Improve test coverage for audio losses

  #13309 commented on May 6, 2025 • 0 new comments

• Add MLlama export_ckpt

  #13346 commented on May 9, 2025 • 0 new comments

• Update extra_requires and requirements

  #13359 commented on May 9, 2025 • 0 new comments

• [automodel] add find_unused_parameters=True for DDP

  #13366 commented on May 10, 2025 • 0 new comments

• fix eval_beamsearch_ngram_ctc

  #13388 commented on May 9, 2025 • 0 new comments

• Add Qwen2VL export_ckpt

  #13398 commented on May 8, 2025 • 0 new comments

• adding back in PlaceholderFilter

  #13402 commented on May 10, 2025 • 0 new comments

• Automodel mvp 0.2

  #13420 commented on May 5, 2025 • 0 new comments

• nemo logging breaks hydra submitit launcher

  #12776 commented on May 4, 2025 • 0 new comments

• ERROR: Failed building wheel for fasttext, ERROR: Could not build wheels for fasttext, which is required to install pyproject.toml-based projects

  #13419 commented on May 5, 2025 • 0 new comments

• pruning-distillation guidance or docs for Multimodal model

  #12975 commented on May 5, 2025 • 0 new comments

• SLU Model classifier input not named

  #12882 commented on May 7, 2025 • 0 new comments

• Cache Aware Streaming script yields different results for different batch_sizes

  #12840 commented on May 7, 2025 • 0 new comments

• why CTCDecodingConfig dont work?

  #13155 commented on May 8, 2025 • 0 new comments

• nemo2riva "TypeError: Can't instantiate abstract class ModelPT with abstract methods list_available_models, setup_training_data, setup_validation_data"

  #12751 commented on May 9, 2025 • 0 new comments

• speech_llm/modular_audio_gpt_train.py is not running while freeze_audio_encoder: False

  #12627 commented on May 9, 2025 • 0 new comments

• SDE with GPU acceleration

  #8657 commented on May 6, 2025 • 0 new comments

• [NeMo2.0] Add MCore FSDP2 support

  #11216 commented on May 5, 2025 • 0 new comments

• Add safetensor option when saving and restoring models

  #11549 commented on May 5, 2025 • 0 new comments

• Add nemo1 to nemo2 conversion for neva

  #11860 commented on May 8, 2025 • 0 new comments

• Fixed normalization of feature vector and weight vector

  #12246 commented on May 6, 2025 • 0 new comments

• [resiliency] Add in process integration for Nemo2

  #12589 commented on May 10, 2025 • 0 new comments

• Variable global and micro batch sizes for different GPUs

  #12640 commented on May 6, 2025 • 0 new comments

• feat: support exp manager checkpointing with object store via Multi-Storage Client

  #12747 commented on May 6, 2025 • 0 new comments

• Adding more doc-strings to megatron_parallel.py

  #12767 commented on May 10, 2025 • 0 new comments

• sft nemo2.0 checkin

  #12794 commented on May 10, 2025 • 0 new comments

• Lookahead attention finetuning

  #12896 commented on May 5, 2025 • 0 new comments

• PhysicalAI Collection

  #12910 commented on May 6, 2025 • 0 new comments

• Transducer with Transformer-Decoder (GPT-like)

  #13030 commented on May 7, 2025 • 0 new comments

• Fix test failures and structural issues

  #13051 commented on May 6, 2025 • 0 new comments

• Intermediate-tensor distillation support

  #13069 commented on May 8, 2025 • 0 new comments

• Update lazy trt compile

  #13075 commented on May 8, 2025 • 0 new comments

• Add option distributed_size to MegatronDistributedFusedAdam

  #13102 commented on May 9, 2025 • 0 new comments

• IPL Mixin for pseudo-labeling

  #13129 commented on May 5, 2025 • 0 new comments

• allow matching by type

  #13157 commented on May 7, 2025 • 0 new comments

• Add optional --profiler flag to test PytorchProfilerCallback with trace checks

  #13176 commented on May 7, 2025 • 0 new comments