Support for NemotronH Nano VLM

[danielafrimi]

Adds a new multimodal model implementation: vllm/model_executor/models/nano_nemotron_vl.py

for online serving do the following:

1. vllm serve <hf-card> --runner generate --max-model-len 8192 --trust_remote_code

2. text prompt for video/image

        "role":
        "user",
        "content": [
            {
                "type": "text",
                # "text": "What's in this image?" #  <image> placeholder is in the prompt
                 "text": "What's in this video and whats on the tv?" # <video> placeholder is in the prompt
            },
            {
                "type": "video_url",
                "video_url": {
                    "url": video_url
                },

                # "type": "image_url",
                # "image_url": {
                #     "url": image_url
                # },
            },
        ],
    }],

[gemini-code-assist]

Code Review

This pull request introduces support for the NemotronH Nano VLM model, with the core logic implemented in the new vllm/model_executor/models/nano_nemotron_vl.py file. The implementation is comprehensive, but I've identified a critical issue in the model registry that would prevent the model from loading, along with a few high-severity issues including a debug print statement, a questionable hardcoded limit, and a stateful implementation detail that could be refactored for better robustness.

[DarkLight1337]

Thanks for implementing this in vLLM! Some initial comments

[danielafrimi]

@DarkLight1337 Thanks for reviewing!

ping you back

[DarkLight1337]

Can you add this model to tests/models/multimodal/processing/test_common.py to validate the processing logic?

[DarkLight1337]

Also need to add it to the test registry tests/models/registry.py

[danielafrimi]

Thanks!

@DarkLight1337 There isn't a HF card for this model yet - we will release it in the upcoming month.
so for adding the model into tests/models/registry.py i dont know exactly the model repo name ( i can add it after the model release).
Same for tests/models/multimodal/processing/test_common.py it looks like i need to add the HF model_id.

what do you think?

[DarkLight1337]

Inside tests/models/registry.py, you can add an entry with a dummy model name and set is_available_online=False, that should skip the test for the model

[danielafrimi]

@DarkLight1337 Thanks for the review.
added the model to registry

[DarkLight1337]

Let's inherit from InternVL processor to avoid duplicate code

[danielafrimi]

@DarkLight1337 BaseNanoNemotronVLProcessor and BaseInternVLProcessor don’t follow the exact same logic.
For example:

In NanoNemotron we don’t use min/max/max_dynamic_patch, but we do persist normalization attributes.
Image/video processing is largely similar between the two, but NanoNemotron assumes that image/video placeholders are in the prompt, which leads to slightly different handling (also the processing itself).

[DarkLight1337]

Ok sure, thanks for the explanation

[DarkLight1337]

Is the vision model part not implemented in vLLM yet?

[danielafrimi]

yes, i have a draft PR which implemented the RADIO model with vLLM native.
after the current PR will merge ill rebase the below one

PR

[danielafrimi]

@DarkLight1337 added both class function which is required to calc the mamba page size. however those impl are in the llm model and in the vlm one (vllm/model_executor/models/config.py).

what do you think about this WAR? wondering if there any nice impl for this.

[DarkLight1337]

@tdoublep probably has a better understanding of this

[tdoublep]

The implementation here looks good to me. We need these class methods get_mamba_state_shape_from_config because in vllm/model_executor/models/config.py we do some manipulation of the cache config (e.g., modifying the attention block size) and we need to be able to do this without actually instantiating the model.

[danielafrimi]

@DarkLight1337 fix the above + add some comments. let me know what you think.

[danielafrimi]

@DarkLight1337 Can we approve it? or is any other changes needed

[DarkLight1337]

LGTM then, thanks