Direct integration with mlx-vlm

[hermeschen1116]

Hello, first, thank you for the great work for releasing this model.
But I have some question about the integration with mlx-vlm.
I see you provide a patch for the previous commit in mlx-vlm repo. Is it possible to have a direct integration with mlx-vlm or provide a patch for latest version?
It's more easier for package management and can take advantage from newer version.

[yanyaoer]

for example:

uv run --with mlx-vlm python -m mlx_vlm.generate --model mlx-community/SmolVLM-Instruct-8bit \  # maybe use mlx-commuity/ml-fastvlm
  --max-tokens 100 --temp 0.0 --prompt 'Describe image in detail, include all text' \
  --image https://github.com/apple/ml-fastvlm/raw/main/docs/acc_vs_latency_qwen-2.png

Files: ['https://github.com/apple/ml-fastvlm/raw/main/docs/acc_vs_latency_qwen-2.png']
Prompt: <|im_start|>User:<image>Describe image in detail, include all text<end_of_utterance>
Assistant:
 Here a graph is labeled Avg 5 VLM Eval. The x-axis measures Time To First Token (ms) while the y-axis measures Avg 5 VLM Eval. There are 5 different colored dots on the graph. FastViTHD (ours) is blue, SigLIP-SO400M is brown, ConvNeXt-XXL is purple, and ConvNeXt-L is green. There is a linear scale of 0
==========
Prompt: 1199 tokens, 106.301 tokens-per-sec
Generation: 100 tokens, 17.429 tokens-per-sec
Peak memory: 3.483 GB

[hermeschen1116]

Update: I think migrate the architecture of the model to make it compatible with llava_next or other llava architectures which already supported by mlx_vlm would be another method. I found there are some discussions about this under the issue of integrating fastvlm into transformers so maybe you already work on that?