[Bug]: NVIDIA Thor with CUDA capability sm_110 is not compatible

[sc-hua]

Your current environment

The output of python collect_env.py

Collecting environment information...
/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/torch/cuda/__init__.py:327: UserWarning: 
NVIDIA Thor with CUDA capability sm_110 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_80 sm_90 sm_100 sm_120.
If you want to use the NVIDIA Thor GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.3 LTS (aarch64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0.dev20251012+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.11 | packaged by Anaconda, Inc. | (main, Jun  5 2025, 12:59:05) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-6.8.12-tegra-aarch64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.48
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA Thor
Nvidia driver version        : 580.00
cuDNN version                : Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_adv.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_cnn.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_heuristic.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_ops.so.9.12.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               14
On-line CPU(s) list:                  0-13
Vendor ID:                            ARM
Model name:                           -
Model:                                0
Thread(s) per core:                   1
Core(s) per cluster:                  14
Socket(s):                            -
Cluster(s):                           1
Stepping:                             r0p0
CPU(s) scaling MHz:                   61%
CPU max MHz:                          2601.0000
CPU min MHz:                          54.0000
BogoMIPS:                             2000.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
L1d cache:                            896 KiB (14 instances)
L1i cache:                            896 KiB (14 instances)
L2 cache:                             14 MiB (14 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-13
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.4.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.15.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.2.1
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.24
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pytorch-triton==3.5.0+git7416ffcb
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0.dev20251012+cu128
[pip3] torchvision==0.25.0.dev20251013
[pip3] transformers==4.57.0
[conda] flashinfer-python                    0.4.0                     pypi_0           pypi
[conda] numpy                                2.2.6                     pypi_0           pypi
[conda] nvidia-cublas-cu12                   12.8.4.1                  pypi_0           pypi
[conda] nvidia-cuda-cupti-cu12               12.8.90                   pypi_0           pypi
[conda] nvidia-cuda-nvrtc-cu12               12.8.93                   pypi_0           pypi
[conda] nvidia-cuda-runtime-cu12             12.8.90                   pypi_0           pypi
[conda] nvidia-cudnn-cu12                    9.10.2.21                 pypi_0           pypi
[conda] nvidia-cudnn-frontend                1.15.0                    pypi_0           pypi
[conda] nvidia-cufft-cu12                    11.3.3.83                 pypi_0           pypi
[conda] nvidia-cufile-cu12                   1.13.1.3                  pypi_0           pypi
[conda] nvidia-curand-cu12                   10.3.9.90                 pypi_0           pypi
[conda] nvidia-cusolver-cu12                 11.7.3.90                 pypi_0           pypi
[conda] nvidia-cusparse-cu12                 12.5.8.93                 pypi_0           pypi
[conda] nvidia-cusparselt-cu12               0.7.1                     pypi_0           pypi
[conda] nvidia-cutlass-dsl                   4.2.1                     pypi_0           pypi
[conda] nvidia-ml-py                         13.580.82                 pypi_0           pypi
[conda] nvidia-nccl-cu12                     2.27.5                    pypi_0           pypi
[conda] nvidia-nvjitlink-cu12                12.8.93                   pypi_0           pypi
[conda] nvidia-nvshmem-cu12                  3.3.24                    pypi_0           pypi
[conda] nvidia-nvtx-cu12                     12.8.90                   pypi_0           pypi
[conda] pytorch-triton                       3.5.0+git7416ffcb         pypi_0           pypi
[conda] pyzmq                                27.1.0                    pypi_0           pypi
[conda] torch                                2.10.0.dev20251012+cu128  pypi_0           pypi
[conda] torchvision                          0.25.0.dev20251013        pypi_0           pypi
[conda] transformers                         4.57.0                    pypi_0           pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.0rc2.dev389+ge51928192.d20251013 (git sha: e51928192, date: 20251013)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-13    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib64:
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_xxx

🐛 Describe the bug

The raw output log

(llm) xxx@xxx-thor:~/proj$ vllm serve --config /home/xxx/proj/infer/vllm_config/holo-1.5-7b.yaml
INFO 10-14 17:06:40 [__init__.py:224] Automatically detected platform cuda.
(APIServer pid=107364) INFO 10-14 17:06:44 [api_server.py:1870] vLLM API server version 0.11.0rc2.dev389+ge51928192.d20251013
(APIServer pid=107364) INFO 10-14 17:06:44 [utils.py:239] non-default args: {'port': 7878, 'model': '/home/xxx/proj/ckpts/holo1_5-7b', 'trust_remote_code': True, 'max_model_len': 16000, 'served_model_name': ['holo1.5-7b'], 'gpu_memory_utilization': 0.3, 'enable_prefix_caching': True}
(APIServer pid=107364) INFO 10-14 17:06:51 [model.py:653] Resolved architecture: Qwen2_5_VLForConditionalGeneration
(APIServer pid=107364) `torch_dtype` is deprecated! Use `dtype` instead!
(APIServer pid=107364) INFO 10-14 17:06:51 [model.py:1714] Using max model len 16000
(APIServer pid=107364) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=107364) INFO 10-14 17:06:51 [scheduler.py:225] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 10-14 17:06:56 [__init__.py:224] Automatically detected platform cuda.
(EngineCore_DP0 pid=107532) INFO 10-14 17:06:59 [core.py:727] Waiting for init message from front-end.
(EngineCore_DP0 pid=107532) INFO 10-14 17:06:59 [core.py:94] Initializing a V1 LLM engine (v0.11.0rc2.dev389+ge51928192.d20251013) with config: model='/home/xxx/proj/ckpts/holo1_5-7b', speculative_config=None, tokenizer='/home/xxx/proj/ckpts/holo1_5-7b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=holo1.5-7b, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.unified_mla_attention', 'vllm.unified_mla_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None}
(EngineCore_DP0 pid=107532) /home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/torch/cuda/__init__.py:327: UserWarning: 
(EngineCore_DP0 pid=107532) NVIDIA Thor with CUDA capability sm_110 is not compatible with the current PyTorch installation.
(EngineCore_DP0 pid=107532) The current PyTorch install supports CUDA capabilities sm_80 sm_90 sm_100 sm_120.
(EngineCore_DP0 pid=107532) If you want to use the NVIDIA Thor GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
(EngineCore_DP0 pid=107532) 
(EngineCore_DP0 pid=107532)   warnings.warn(
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] EngineCore failed to start.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] Traceback (most recent call last):
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 781, in run_engine_core
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 553, in __init__
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     super().__init__(
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 103, in __init__
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self._init_executor()
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self.collective_rpc("init_device")
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 73, in collective_rpc
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2975, in run_method
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     return func(*args, **kwargs)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 330, in init_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 171, in init_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     current_platform.set_device(self.device)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 85, in set_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     _ = torch.zeros(1, device=device)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] 
(EngineCore_DP0 pid=107532) Process EngineCore_DP0:
(EngineCore_DP0 pid=107532) Traceback (most recent call last):
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=107532)     self.run()
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=107532)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 794, in run_engine_core
(EngineCore_DP0 pid=107532)     raise e
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 781, in run_engine_core
(EngineCore_DP0 pid=107532)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=107532)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 553, in __init__
(EngineCore_DP0 pid=107532)     super().__init__(
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 103, in __init__
(EngineCore_DP0 pid=107532)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=107532)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=107532)     self._init_executor()
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=107532)     self.collective_rpc("init_device")
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 73, in collective_rpc
(EngineCore_DP0 pid=107532)     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=107532)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2975, in run_method
(EngineCore_DP0 pid=107532)     return func(*args, **kwargs)
(EngineCore_DP0 pid=107532)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 330, in init_device
(EngineCore_DP0 pid=107532)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=107532)     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 171, in init_device
(EngineCore_DP0 pid=107532)     current_platform.set_device(self.device)
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 85, in set_device
(EngineCore_DP0 pid=107532)     _ = torch.zeros(1, device=device)
(EngineCore_DP0 pid=107532)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
(EngineCore_DP0 pid=107532) Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=107532) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=107532) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=107532) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=107532) 
(APIServer pid=107364) Traceback (most recent call last):
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/bin/vllm", line 7, in <module>
(APIServer pid=107364)     sys.exit(main())
(APIServer pid=107364)              ^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=107364)     args.dispatch_function(args)
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 63, in cmd
(APIServer pid=107364)     uvloop.run(run_server(args))
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
(APIServer pid=107364)     return __asyncio.run(
(APIServer pid=107364)            ^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=107364)     return runner.run(main)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=107364)     return self._loop.run_until_complete(task)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=107364)     return await main
(APIServer pid=107364)            ^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1914, in run_server
(APIServer pid=107364)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1930, in run_server_worker
(APIServer pid=107364)     async with build_async_engine_client(
(APIServer pid=107364)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=107364)     return await anext(self.gen)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 191, in build_async_engine_client
(APIServer pid=107364)     async with build_async_engine_client_from_engine_args(
(APIServer pid=107364)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=107364)     return await anext(self.gen)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 238, in build_async_engine_client_from_engine_args
(APIServer pid=107364)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=107364)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/utils/__init__.py", line 1335, in inner
(APIServer pid=107364)     return fn(*args, **kwargs)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 205, in from_vllm_config
(APIServer pid=107364)     return cls(
(APIServer pid=107364)            ^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 129, in __init__
(APIServer pid=107364)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=107364)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
(APIServer pid=107364)     return AsyncMPClient(*client_args)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 809, in __init__
(APIServer pid=107364)     super().__init__(
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 470, in __init__
(APIServer pid=107364)     with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=107364)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=107364)     next(self.gen)
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 815, in launch_core_engines
(APIServer pid=107364)     wait_for_engine_startup(
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 872, in wait_for_engine_startup
(APIServer pid=107364)     raise RuntimeError(
(APIServer pid=107364) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

# run script
vllm serve --config /home/xxx/proj/infer/vllm_config/holo-1.5-7b.yaml

# core log
...
(EngineCore_DP0 pid=107532) /home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/torch/cuda/__init__.py:327: UserWarning: 
(EngineCore_DP0 pid=107532) NVIDIA Thor with CUDA capability sm_110 is not compatible with the current PyTorch installation.
(EngineCore_DP0 pid=107532) The current PyTorch install supports CUDA capabilities sm_80 sm_90 sm_100 sm_120.
...
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 85, in set_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     _ = torch.zeros(1, device=device)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
...

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[sc-hua]

Another question: When will vllm support cu130 (on thor)?

[DarkLight1337]

cc @johnnynunez

[johnnynunez]

Your current environment

The output of python collect_env.py

Collecting environment information...
/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/torch/cuda/__init__.py:327: UserWarning: 
NVIDIA Thor with CUDA capability sm_110 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_80 sm_90 sm_100 sm_120.
If you want to use the NVIDIA Thor GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.3 LTS (aarch64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0.dev20251012+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.11 | packaged by Anaconda, Inc. | (main, Jun  5 2025, 12:59:05) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-6.8.12-tegra-aarch64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.48
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA Thor
Nvidia driver version        : 580.00
cuDNN version                : Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_adv.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_cnn.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_heuristic.so.9.12.0
/usr/lib/aarch64-linux-gnu/libcudnn_ops.so.9.12.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         aarch64
CPU op-mode(s):                       64-bit
Byte Order:                           Little Endian
CPU(s):                               14
On-line CPU(s) list:                  0-13
Vendor ID:                            ARM
Model name:                           -
Model:                                0
Thread(s) per core:                   1
Core(s) per cluster:                  14
Socket(s):                            -
Cluster(s):                           1
Stepping:                             r0p0
CPU(s) scaling MHz:                   61%
CPU max MHz:                          2601.0000
CPU min MHz:                          54.0000
BogoMIPS:                             2000.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
L1d cache:                            896 KiB (14 instances)
L1i cache:                            896 KiB (14 instances)
L2 cache:                             14 MiB (14 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-13
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.4.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.15.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.2.1
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.24
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pytorch-triton==3.5.0+git7416ffcb
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0.dev20251012+cu128
[pip3] torchvision==0.25.0.dev20251013
[pip3] transformers==4.57.0
[conda] flashinfer-python                    0.4.0                     pypi_0           pypi
[conda] numpy                                2.2.6                     pypi_0           pypi
[conda] nvidia-cublas-cu12                   12.8.4.1                  pypi_0           pypi
[conda] nvidia-cuda-cupti-cu12               12.8.90                   pypi_0           pypi
[conda] nvidia-cuda-nvrtc-cu12               12.8.93                   pypi_0           pypi
[conda] nvidia-cuda-runtime-cu12             12.8.90                   pypi_0           pypi
[conda] nvidia-cudnn-cu12                    9.10.2.21                 pypi_0           pypi
[conda] nvidia-cudnn-frontend                1.15.0                    pypi_0           pypi
[conda] nvidia-cufft-cu12                    11.3.3.83                 pypi_0           pypi
[conda] nvidia-cufile-cu12                   1.13.1.3                  pypi_0           pypi
[conda] nvidia-curand-cu12                   10.3.9.90                 pypi_0           pypi
[conda] nvidia-cusolver-cu12                 11.7.3.90                 pypi_0           pypi
[conda] nvidia-cusparse-cu12                 12.5.8.93                 pypi_0           pypi
[conda] nvidia-cusparselt-cu12               0.7.1                     pypi_0           pypi
[conda] nvidia-cutlass-dsl                   4.2.1                     pypi_0           pypi
[conda] nvidia-ml-py                         13.580.82                 pypi_0           pypi
[conda] nvidia-nccl-cu12                     2.27.5                    pypi_0           pypi
[conda] nvidia-nvjitlink-cu12                12.8.93                   pypi_0           pypi
[conda] nvidia-nvshmem-cu12                  3.3.24                    pypi_0           pypi
[conda] nvidia-nvtx-cu12                     12.8.90                   pypi_0           pypi
[conda] pytorch-triton                       3.5.0+git7416ffcb         pypi_0           pypi
[conda] pyzmq                                27.1.0                    pypi_0           pypi
[conda] torch                                2.10.0.dev20251012+cu128  pypi_0           pypi
[conda] torchvision                          0.25.0.dev20251013        pypi_0           pypi
[conda] transformers                         4.57.0                    pypi_0           pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.0rc2.dev389+ge51928192.d20251013 (git sha: e51928192, date: 20251013)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-13    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib64:
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_xxx

🐛 Describe the bug

The raw output log

(llm) xxx@xxx-thor:~/proj$ vllm serve --config /home/xxx/proj/infer/vllm_config/holo-1.5-7b.yaml
INFO 10-14 17:06:40 [__init__.py:224] Automatically detected platform cuda.
(APIServer pid=107364) INFO 10-14 17:06:44 [api_server.py:1870] vLLM API server version 0.11.0rc2.dev389+ge51928192.d20251013
(APIServer pid=107364) INFO 10-14 17:06:44 [utils.py:239] non-default args: {'port': 7878, 'model': '/home/xxx/proj/ckpts/holo1_5-7b', 'trust_remote_code': True, 'max_model_len': 16000, 'served_model_name': ['holo1.5-7b'], 'gpu_memory_utilization': 0.3, 'enable_prefix_caching': True}
(APIServer pid=107364) INFO 10-14 17:06:51 [model.py:653] Resolved architecture: Qwen2_5_VLForConditionalGeneration
(APIServer pid=107364) `torch_dtype` is deprecated! Use `dtype` instead!
(APIServer pid=107364) INFO 10-14 17:06:51 [model.py:1714] Using max model len 16000
(APIServer pid=107364) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=107364) INFO 10-14 17:06:51 [scheduler.py:225] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 10-14 17:06:56 [__init__.py:224] Automatically detected platform cuda.
(EngineCore_DP0 pid=107532) INFO 10-14 17:06:59 [core.py:727] Waiting for init message from front-end.
(EngineCore_DP0 pid=107532) INFO 10-14 17:06:59 [core.py:94] Initializing a V1 LLM engine (v0.11.0rc2.dev389+ge51928192.d20251013) with config: model='/home/xxx/proj/ckpts/holo1_5-7b', speculative_config=None, tokenizer='/home/xxx/proj/ckpts/holo1_5-7b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=holo1.5-7b, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm.unified_attention', 'vllm.unified_attention_with_output', 'vllm.unified_mla_attention', 'vllm.unified_mla_attention_with_output', 'vllm.mamba_mixer2', 'vllm.mamba_mixer', 'vllm.short_conv', 'vllm.linear_attention', 'vllm.plamo2_mamba_mixer', 'vllm.gdn_attention', 'vllm.sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None}
(EngineCore_DP0 pid=107532) /home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/torch/cuda/__init__.py:327: UserWarning: 
(EngineCore_DP0 pid=107532) NVIDIA Thor with CUDA capability sm_110 is not compatible with the current PyTorch installation.
(EngineCore_DP0 pid=107532) The current PyTorch install supports CUDA capabilities sm_80 sm_90 sm_100 sm_120.
(EngineCore_DP0 pid=107532) If you want to use the NVIDIA Thor GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
(EngineCore_DP0 pid=107532) 
(EngineCore_DP0 pid=107532)   warnings.warn(
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] EngineCore failed to start.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] Traceback (most recent call last):
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 781, in run_engine_core
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 553, in __init__
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     super().__init__(
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 103, in __init__
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self._init_executor()
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self.collective_rpc("init_device")
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 73, in collective_rpc
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2975, in run_method
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     return func(*args, **kwargs)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 330, in init_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 171, in init_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     current_platform.set_device(self.device)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 85, in set_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]     _ = torch.zeros(1, device=device)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] 
(EngineCore_DP0 pid=107532) Process EngineCore_DP0:
(EngineCore_DP0 pid=107532) Traceback (most recent call last):
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=107532)     self.run()
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=107532)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 794, in run_engine_core
(EngineCore_DP0 pid=107532)     raise e
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 781, in run_engine_core
(EngineCore_DP0 pid=107532)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=107532)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 553, in __init__
(EngineCore_DP0 pid=107532)     super().__init__(
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 103, in __init__
(EngineCore_DP0 pid=107532)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=107532)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=107532)     self._init_executor()
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=107532)     self.collective_rpc("init_device")
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 73, in collective_rpc
(EngineCore_DP0 pid=107532)     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=107532)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2975, in run_method
(EngineCore_DP0 pid=107532)     return func(*args, **kwargs)
(EngineCore_DP0 pid=107532)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 330, in init_device
(EngineCore_DP0 pid=107532)     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=107532)     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 171, in init_device
(EngineCore_DP0 pid=107532)     current_platform.set_device(self.device)
(EngineCore_DP0 pid=107532)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 85, in set_device
(EngineCore_DP0 pid=107532)     _ = torch.zeros(1, device=device)
(EngineCore_DP0 pid=107532)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
(EngineCore_DP0 pid=107532) Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=107532) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=107532) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=107532) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=107532) 
(APIServer pid=107364) Traceback (most recent call last):
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/bin/vllm", line 7, in <module>
(APIServer pid=107364)     sys.exit(main())
(APIServer pid=107364)              ^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=107364)     args.dispatch_function(args)
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 63, in cmd
(APIServer pid=107364)     uvloop.run(run_server(args))
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
(APIServer pid=107364)     return __asyncio.run(
(APIServer pid=107364)            ^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=107364)     return runner.run(main)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=107364)     return self._loop.run_until_complete(task)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=107364)     return await main
(APIServer pid=107364)            ^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1914, in run_server
(APIServer pid=107364)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1930, in run_server_worker
(APIServer pid=107364)     async with build_async_engine_client(
(APIServer pid=107364)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=107364)     return await anext(self.gen)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 191, in build_async_engine_client
(APIServer pid=107364)     async with build_async_engine_client_from_engine_args(
(APIServer pid=107364)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=107364)     return await anext(self.gen)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 238, in build_async_engine_client_from_engine_args
(APIServer pid=107364)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=107364)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/utils/__init__.py", line 1335, in inner
(APIServer pid=107364)     return fn(*args, **kwargs)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 205, in from_vllm_config
(APIServer pid=107364)     return cls(
(APIServer pid=107364)            ^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 129, in __init__
(APIServer pid=107364)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=107364)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
(APIServer pid=107364)     return AsyncMPClient(*client_args)
(APIServer pid=107364)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 809, in __init__
(APIServer pid=107364)     super().__init__(
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 470, in __init__
(APIServer pid=107364)     with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=107364)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=107364)     next(self.gen)
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 815, in launch_core_engines
(APIServer pid=107364)     wait_for_engine_startup(
(APIServer pid=107364)   File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 872, in wait_for_engine_startup
(APIServer pid=107364)     raise RuntimeError(
(APIServer pid=107364) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

run script

vllm serve --config /home/xxx/proj/infer/vllm_config/holo-1.5-7b.yaml

core log

...
(EngineCore_DP0 pid=107532) /home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/torch/cuda/init.py:327: UserWarning:
(EngineCore_DP0 pid=107532) NVIDIA Thor with CUDA capability sm_110 is not compatible with the current PyTorch installation.
(EngineCore_DP0 pid=107532) The current PyTorch install supports CUDA capabilities sm_80 sm_90 sm_100 sm_120.
...
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] File "/home/xxx/miniconda3/envs/llm/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 85, in set_device
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] _ = torch.zeros(1, device=device)
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107532) ERROR 10-14 17:07:18 [core.py:790] torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
...

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

it is compatible. You have to compile vllm with cuda 13

[johnnynunez]

you are using wrong pytorch for example: PyTorch version : 2.10.0.dev20251012+cu128

you have to use:
pip3 install torch==2.9.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/test/cu130

[sc-hua]

thanks @johnnynunez , i tried the qwen2.5-3b, and it worked well with your installation script.

But there's a bit of a problem with qwen2.5vl-3b.

• env setup script:

conda activate vllm-cu130
pip3 install torch==2.9.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/test/cu130
git clone https://github.com/vllm-project/vllm.git
cd vllm
python use_existing_torch.py
pip install -r requirements/build.txt
pip install --no-build-isolation -e .

• core error log:

...
(EngineCore_DP0 pid=320603) Internal Triton PTX codegen error
(EngineCore_DP0 pid=320603) `ptxas` stderr:
(EngineCore_DP0 pid=320603) ptxas fatal   : Value 'sm_110a' is not defined for option 'gpu-name'
...

full error log here

(vllm-cu130) xxx@xxx-thor:~/proj$ vllm serve --config '/home/xxx/proj/infer/vllm_config/qwen2.5vl-3b.yaml'
INFO 10-15 01:07:53 [__init__.py:224] Automatically detected platform cuda.
(APIServer pid=320511) INFO 10-15 01:07:57 [api_server.py:1870] vLLM API server version 0.1.dev1+g98f30b8cb.d20251014
(APIServer pid=320511) INFO 10-15 01:07:57 [utils.py:239] non-default args: {'port': 7878, 'model': '/home/xxx/proj/ckpts/qwen2_5vl-3b', 'trust_remote_code': True, 'max_model_len': 16000, 'served_model_name': ['qwen2.5vl-3b'], 'gpu_memory_utilization': 0.3, 'enable_prefix_caching': True}
(APIServer pid=320511) INFO 10-15 01:07:57 [model.py:645] Resolved architecture: Qwen2_5_VLForConditionalGeneration
(APIServer pid=320511) `torch_dtype` is deprecated! Use `dtype` instead!
(APIServer pid=320511) INFO 10-15 01:07:57 [model.py:1706] Using max model len 16000
(APIServer pid=320511) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=320511) INFO 10-15 01:07:57 [scheduler.py:225] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 10-15 01:08:00 [__init__.py:224] Automatically detected platform cuda.
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:03 [core.py:727] Waiting for init message from front-end.
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:03 [core.py:94] Initializing a V1 LLM engine (v0.1.dev1+g98f30b8cb.d20251014) with config: model='/home/xxx/proj/ckpts/qwen2_5vl-3b', speculative_config=None, tokenizer='/home/xxx/proj/ckpts/qwen2_5vl-3b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qwen2.5vl-3b, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': '', 'custom_ops': [], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention', 'vllm::sparse_attn_indexer'], 'use_inductor': True, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None}
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:05 [parallel_state.py:1325] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:05 [topk_topp_sampler.py:58] Using FlashInfer for top-p & top-k sampling.
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:07 [gpu_model_runner.py:2849] Starting to load model /home/xxx/proj/ckpts/qwen2_5vl-3b...
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:08 [gpu_model_runner.py:2879] Loading model from scratch...
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:08 [cuda.py:405] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:02<00:02,  2.87s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:05<00:00,  2.98s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:05<00:00,  2.96s/it]
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:14 [default_loader.py:314] Loading weights took 6.17 seconds
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:15 [gpu_model_runner.py:2911] Model loading took 7.1550 GiB and 6.435936 seconds
(EngineCore_DP0 pid=320603) INFO 10-15 01:08:15 [gpu_model_runner.py:3676] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) ================================================================
(EngineCore_DP0 pid=320603) Internal Triton PTX codegen error
(EngineCore_DP0 pid=320603) `ptxas` stderr:
(EngineCore_DP0 pid=320603) ptxas fatal   : Value 'sm_110a' is not defined for option 'gpu-name'
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) Repro command: /home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_110a /tmp/tmpyrn4vdgn.ptx -o /tmp/tmpyrn4vdgn.ptx.o
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) //
(EngineCore_DP0 pid=320603) // Generated by LLVM NVPTX Back-End
(EngineCore_DP0 pid=320603) //
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) .version 8.7
(EngineCore_DP0 pid=320603) .target sm_110a
(EngineCore_DP0 pid=320603) .address_size 64
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603)     // .globl       rotary_kernel           // -- Begin function rotary_kernel
(EngineCore_DP0 pid=320603)                                         // @rotary_kernel
(EngineCore_DP0 pid=320603) .visible .entry rotary_kernel(
(EngineCore_DP0 pid=320603)     .param .u64 .ptr .global .align 1 rotary_kernel_param_0,
(EngineCore_DP0 pid=320603)     .param .u64 .ptr .global .align 1 rotary_kernel_param_1,
(EngineCore_DP0 pid=320603)     .param .u64 .ptr .global .align 1 rotary_kernel_param_2,
(EngineCore_DP0 pid=320603)     .param .u64 .ptr .global .align 1 rotary_kernel_param_3,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_4,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_5,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_6,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_7,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_8,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_9,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_10,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_11,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_12,
(EngineCore_DP0 pid=320603)     .param .u32 rotary_kernel_param_13,
(EngineCore_DP0 pid=320603)     .param .u64 .ptr .global .align 1 rotary_kernel_param_14,
(EngineCore_DP0 pid=320603)     .param .u64 .ptr .global .align 1 rotary_kernel_param_15
(EngineCore_DP0 pid=320603) )
(EngineCore_DP0 pid=320603) .reqntid 128
(EngineCore_DP0 pid=320603) {
(EngineCore_DP0 pid=320603)     .reg .pred      %p<11>;
(EngineCore_DP0 pid=320603)     .reg .b32       %r<83>;
(EngineCore_DP0 pid=320603)     .reg .b64       %rd<17>;
(EngineCore_DP0 pid=320603)     .loc    1 13 0                          // rotary.py:13:0
(EngineCore_DP0 pid=320603) $L__func_begin0:
(EngineCore_DP0 pid=320603)     .loc    1 13 0                          // rotary.py:13:0
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) // %bb.0:
(EngineCore_DP0 pid=320603)     ld.param.b32    %r3, [rotary_kernel_param_5];
(EngineCore_DP0 pid=320603) $L__tmp0:
(EngineCore_DP0 pid=320603)     .loc    1 41 26                         // rotary.py:41:26
(EngineCore_DP0 pid=320603)     mov.u32         %r12, %ctaid.x;
(EngineCore_DP0 pid=320603)     .loc    1 55 15                         // rotary.py:55:15
(EngineCore_DP0 pid=320603)     shl.b32         %r1, %r12, 3;
(EngineCore_DP0 pid=320603)     .loc    1 55 26                         // rotary.py:55:26
(EngineCore_DP0 pid=320603)     setp.lt.s32     %p1, %r1, %r3;
(EngineCore_DP0 pid=320603)     @%p1 bra        $L__BB0_2;
(EngineCore_DP0 pid=320603)     bra.uni         $L__BB0_1;
(EngineCore_DP0 pid=320603) $L__BB0_2:
(EngineCore_DP0 pid=320603)     .loc    1 0 26                          // rotary.py:0:26
(EngineCore_DP0 pid=320603)     ld.param.b32    %r11, [rotary_kernel_param_13];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r10, [rotary_kernel_param_12];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r9, [rotary_kernel_param_11];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r8, [rotary_kernel_param_10];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r7, [rotary_kernel_param_9];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r6, [rotary_kernel_param_8];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r5, [rotary_kernel_param_7];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r4, [rotary_kernel_param_6];
(EngineCore_DP0 pid=320603)     ld.param.b32    %r2, [rotary_kernel_param_4];
(EngineCore_DP0 pid=320603)     ld.param.b64    %rd4, [rotary_kernel_param_3];
(EngineCore_DP0 pid=320603)     ld.param.b64    %rd3, [rotary_kernel_param_2];
(EngineCore_DP0 pid=320603)     ld.param.b64    %rd2, [rotary_kernel_param_1];
(EngineCore_DP0 pid=320603)     ld.param.b64    %rd1, [rotary_kernel_param_0];
(EngineCore_DP0 pid=320603)     .loc    1 43 30                         // rotary.py:43:30
(EngineCore_DP0 pid=320603)     mov.u32         %r53, %ctaid.z;
(EngineCore_DP0 pid=320603)     .loc    1 48 32                         // rotary.py:48:32
(EngineCore_DP0 pid=320603)     mul.lo.s32      %r54, %r6, %r53;
(EngineCore_DP0 pid=320603)     .loc    1 48 20                         // rotary.py:48:20
(EngineCore_DP0 pid=320603)     mad.wide.s32    %rd11, %r54, 4, %rd1;
(EngineCore_DP0 pid=320603)     .loc    1 42 29                         // rotary.py:42:29
(EngineCore_DP0 pid=320603)     mov.u32         %r55, %ctaid.y;
(EngineCore_DP0 pid=320603)     .loc    1 48 62                         // rotary.py:48:62
(EngineCore_DP0 pid=320603)     mul.lo.s32      %r56, %r8, %r55;
(EngineCore_DP0 pid=320603)     .loc    1 48 51                         // rotary.py:48:51
(EngineCore_DP0 pid=320603)     mad.wide.s32    %rd12, %r56, 4, %rd11;
(EngineCore_DP0 pid=320603)     .loc    1 47 28                         // rotary.py:47:28
(EngineCore_DP0 pid=320603)     mul.lo.s32      %r57, %r9, %r53;
(EngineCore_DP0 pid=320603)     .loc    1 47 16                         // rotary.py:47:16
(EngineCore_DP0 pid=320603)     mad.wide.s32    %rd13, %r57, 4, %rd2;
(EngineCore_DP0 pid=320603)     .loc    1 47 56                         // rotary.py:47:56
(EngineCore_DP0 pid=320603)     mul.lo.s32      %r58, %r11, %r55;
(EngineCore_DP0 pid=320603)     .loc    1 47 45                         // rotary.py:47:45
(EngineCore_DP0 pid=320603)     mad.wide.s32    %rd14, %r58, 4, %rd13;
(EngineCore_DP0 pid=320603)     .loc    1 44 36                         // rotary.py:44:36
(EngineCore_DP0 pid=320603)     shr.u32         %r59, %r4, 31;
(EngineCore_DP0 pid=320603)     add.s32         %r60, %r4, %r59;
(EngineCore_DP0 pid=320603)     shr.s32         %r61, %r60, 1;
(EngineCore_DP0 pid=320603)     .loc    1 57 40                         // rotary.py:57:40
(EngineCore_DP0 pid=320603)     mov.u32         %r62, %tid.x;
(EngineCore_DP0 pid=320603)     bfe.u32         %r63, %r62, 4, 3;
(EngineCore_DP0 pid=320603)     .loc    1 57 27                         // rotary.py:57:27
(EngineCore_DP0 pid=320603)     or.b32  %r64, %r63, %r1;
(EngineCore_DP0 pid=320603)     .loc    1 59 21                         // rotary.py:59:21
(EngineCore_DP0 pid=320603)     add.s32         %r65, %r2, %r64;
(EngineCore_DP0 pid=320603)     .loc    1 67 57                         // rotary.py:67:57
(EngineCore_DP0 pid=320603)     shl.b32         %r66, %r62, 2;
(EngineCore_DP0 pid=320603)     and.b32         %r67, %r66, 60;
(EngineCore_DP0 pid=320603)     .loc    1 67 49                         // rotary.py:67:49
(EngineCore_DP0 pid=320603)     mad.lo.s32      %r68, %r10, %r64, %r67;
(EngineCore_DP0 pid=320603)     .loc    1 67 17                         // rotary.py:67:17
(EngineCore_DP0 pid=320603)     mad.wide.s32    %rd7, %r68, 4, %rd14;
(EngineCore_DP0 pid=320603)     .loc    1 68 56                         // rotary.py:68:56
(EngineCore_DP0 pid=320603)     mad.lo.s32      %r69, %r61, %r65, %r67;
(EngineCore_DP0 pid=320603)     .loc    1 68 21                         // rotary.py:68:21
(EngineCore_DP0 pid=320603)     mul.wide.s32    %rd15, %r69, 4;
(EngineCore_DP0 pid=320603)     add.s64         %rd5, %rd3, %rd15;
(EngineCore_DP0 pid=320603)     .loc    1 69 21                         // rotary.py:69:21
(EngineCore_DP0 pid=320603)     add.s64         %rd6, %rd4, %rd15;
(EngineCore_DP0 pid=320603)     .loc    1 71 40                         // rotary.py:71:40
(EngineCore_DP0 pid=320603)     setp.lt.s32     %p8, %r65, %r5;
(EngineCore_DP0 pid=320603)     .loc    1 71 73                         // rotary.py:71:73
(EngineCore_DP0 pid=320603)     setp.lt.s32     %p9, %r67, %r61;
(EngineCore_DP0 pid=320603)     .loc    1 71 54                         // rotary.py:71:54
(EngineCore_DP0 pid=320603)     and.pred        %p2, %p8, %p9;
(EngineCore_DP0 pid=320603)     mov.b32         %r17, 1065353216;
(EngineCore_DP0 pid=320603)     .loc    1 71 12                         // rotary.py:71:12
(EngineCore_DP0 pid=320603)     // begin inline asm
(EngineCore_DP0 pid=320603)     mov.u32 %r13, %r17;
(EngineCore_DP0 pid=320603)     mov.u32 %r14, %r17;
(EngineCore_DP0 pid=320603)     mov.u32 %r15, %r17;
(EngineCore_DP0 pid=320603)     mov.u32 %r16, %r17;
(EngineCore_DP0 pid=320603)     @%p2 ld.global.v4.b32 { %r13, %r14, %r15, %r16 }, [ %rd5 + 0 ];
(EngineCore_DP0 pid=320603)     // end inline asm
(EngineCore_DP0 pid=320603)     mov.b32         %r25, 0;
(EngineCore_DP0 pid=320603)     .loc    1 74 12                         // rotary.py:74:12
(EngineCore_DP0 pid=320603)     // begin inline asm
(EngineCore_DP0 pid=320603)     mov.u32 %r21, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r22, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r23, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r24, %r25;
(EngineCore_DP0 pid=320603)     @%p2 ld.global.v4.b32 { %r21, %r22, %r23, %r24 }, [ %rd6 + 0 ];
(EngineCore_DP0 pid=320603)     // end inline asm
(EngineCore_DP0 pid=320603)     .loc    1 77 35                         // rotary.py:77:35
(EngineCore_DP0 pid=320603)     setp.lt.s32     %p10, %r64, %r3;
(EngineCore_DP0 pid=320603)     .loc    1 77 46                         // rotary.py:77:46
(EngineCore_DP0 pid=320603)     and.pred        %p4, %p10, %p9;
(EngineCore_DP0 pid=320603)     .loc    1 77 12                         // rotary.py:77:12
(EngineCore_DP0 pid=320603)     // begin inline asm
(EngineCore_DP0 pid=320603)     mov.u32 %r29, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r30, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r31, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r32, %r25;
(EngineCore_DP0 pid=320603)     @%p4 ld.global.v4.b32 { %r29, %r30, %r31, %r32 }, [ %rd7 + 0 ];
(EngineCore_DP0 pid=320603)     // end inline asm
(EngineCore_DP0 pid=320603)     .loc    1 80 16                         // rotary.py:80:16
(EngineCore_DP0 pid=320603)     mul.wide.s32    %rd16, %r61, 4;
(EngineCore_DP0 pid=320603)     add.s64         %rd8, %rd7, %rd16;
(EngineCore_DP0 pid=320603)     .loc    1 80 12                         // rotary.py:80:12
(EngineCore_DP0 pid=320603)     // begin inline asm
(EngineCore_DP0 pid=320603)     mov.u32 %r37, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r38, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r39, %r25;
(EngineCore_DP0 pid=320603)     mov.u32 %r40, %r25;
(EngineCore_DP0 pid=320603)     @%p4 ld.global.v4.b32 { %r37, %r38, %r39, %r40 }, [ %rd8 + 0 ];
(EngineCore_DP0 pid=320603)     // end inline asm
(EngineCore_DP0 pid=320603)     .loc    1 86 29                         // rotary.py:86:29
(EngineCore_DP0 pid=320603)     mul.f32         %r70, %r21, %r37;
(EngineCore_DP0 pid=320603)     mul.f32         %r71, %r22, %r38;
(EngineCore_DP0 pid=320603)     mul.f32         %r72, %r23, %r39;
(EngineCore_DP0 pid=320603)     mul.f32         %r73, %r24, %r40;
(EngineCore_DP0 pid=320603)     .loc    1 86 24                         // rotary.py:86:24
(EngineCore_DP0 pid=320603)     neg.f32         %r74, %r70;
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r45, %r13, %r29, %r74;
(EngineCore_DP0 pid=320603)     neg.f32         %r75, %r71;
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r46, %r14, %r30, %r75;
(EngineCore_DP0 pid=320603)     neg.f32         %r76, %r72;
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r47, %r15, %r31, %r76;
(EngineCore_DP0 pid=320603)     neg.f32         %r77, %r73;
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r48, %r16, %r32, %r77;
(EngineCore_DP0 pid=320603)     .loc    1 87 29                         // rotary.py:87:29
(EngineCore_DP0 pid=320603)     mul.f32         %r78, %r13, %r37;
(EngineCore_DP0 pid=320603)     mul.f32         %r79, %r14, %r38;
(EngineCore_DP0 pid=320603)     mul.f32         %r80, %r15, %r39;
(EngineCore_DP0 pid=320603)     mul.f32         %r81, %r16, %r40;
(EngineCore_DP0 pid=320603)     .loc    1 87 24                         // rotary.py:87:24
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r49, %r21, %r29, %r78;
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r50, %r22, %r30, %r79;
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r51, %r23, %r31, %r80;
(EngineCore_DP0 pid=320603)     fma.rn.f32      %r52, %r24, %r32, %r81;
(EngineCore_DP0 pid=320603)     .loc    1 89 55                         // rotary.py:89:55
(EngineCore_DP0 pid=320603)     mad.lo.s32      %r82, %r7, %r64, %r67;
(EngineCore_DP0 pid=320603)     .loc    1 89 21                         // rotary.py:89:21
(EngineCore_DP0 pid=320603)     mad.wide.s32    %rd9, %r82, 4, %rd12;
(EngineCore_DP0 pid=320603)     .loc    1 90 22                         // rotary.py:90:22
(EngineCore_DP0 pid=320603)     // begin inline asm
(EngineCore_DP0 pid=320603)     @%p4 st.global.v4.b32 [ %rd9 + 0 ], { %r45, %r46, %r47, %r48 };
(EngineCore_DP0 pid=320603)     // end inline asm
(EngineCore_DP0 pid=320603)     .loc    1 92 18                         // rotary.py:92:18
(EngineCore_DP0 pid=320603)     add.s64         %rd10, %rd9, %rd16;
(EngineCore_DP0 pid=320603)     .loc    1 93 12                         // rotary.py:93:12
(EngineCore_DP0 pid=320603)     // begin inline asm
(EngineCore_DP0 pid=320603)     @%p4 st.global.v4.b32 [ %rd10 + 0 ], { %r49, %r50, %r51, %r52 };
(EngineCore_DP0 pid=320603)     // end inline asm
(EngineCore_DP0 pid=320603) $L__BB0_1:                              // %common.ret
(EngineCore_DP0 pid=320603)     .loc    1 0 0                           // rotary.py:0
(EngineCore_DP0 pid=320603)     ret;
(EngineCore_DP0 pid=320603) $L__tmp1:
(EngineCore_DP0 pid=320603) $L__func_end0:
(EngineCore_DP0 pid=320603)                                         // -- End function
(EngineCore_DP0 pid=320603) }
(EngineCore_DP0 pid=320603)     .file   1 "/home/xxx/proj/vllm/vllm/vllm_flash_attn/ops/triton/rotary.py"
(EngineCore_DP0 pid=320603)     .section        .debug_abbrev
(EngineCore_DP0 pid=320603)     {
(EngineCore_DP0 pid=320603) .b8 1                                   // Abbreviation Code
(EngineCore_DP0 pid=320603) .b8 17                                  // DW_TAG_compile_unit
(EngineCore_DP0 pid=320603) .b8 0                                   // DW_CHILDREN_no
(EngineCore_DP0 pid=320603) .b8 37                                  // DW_AT_producer
(EngineCore_DP0 pid=320603) .b8 8                                   // DW_FORM_string
(EngineCore_DP0 pid=320603) .b8 19                                  // DW_AT_language
(EngineCore_DP0 pid=320603) .b8 5                                   // DW_FORM_data2
(EngineCore_DP0 pid=320603) .b8 3                                   // DW_AT_name
(EngineCore_DP0 pid=320603) .b8 8                                   // DW_FORM_string
(EngineCore_DP0 pid=320603) .b8 16                                  // DW_AT_stmt_list
(EngineCore_DP0 pid=320603) .b8 6                                   // DW_FORM_data4
(EngineCore_DP0 pid=320603) .b8 27                                  // DW_AT_comp_dir
(EngineCore_DP0 pid=320603) .b8 8                                   // DW_FORM_string
(EngineCore_DP0 pid=320603) .b8 0                                   // EOM(1)
(EngineCore_DP0 pid=320603) .b8 0                                   // EOM(2)
(EngineCore_DP0 pid=320603) .b8 0                                   // EOM(3)
(EngineCore_DP0 pid=320603)     }
(EngineCore_DP0 pid=320603)     .section        .debug_info
(EngineCore_DP0 pid=320603)     {
(EngineCore_DP0 pid=320603) .b32 91                                 // Length of Unit
(EngineCore_DP0 pid=320603) .b8 2                                   // DWARF version number
(EngineCore_DP0 pid=320603) .b8 0
(EngineCore_DP0 pid=320603) .b32 .debug_abbrev                      // Offset Into Abbrev. Section
(EngineCore_DP0 pid=320603) .b8 8                                   // Address Size (in bytes)
(EngineCore_DP0 pid=320603) .b8 1                                   // Abbrev [1] 0xb:0x54 DW_TAG_compile_unit
(EngineCore_DP0 pid=320603) .b8 116                                 // DW_AT_producer
(EngineCore_DP0 pid=320603) .b8 114
(EngineCore_DP0 pid=320603) .b8 105
(EngineCore_DP0 pid=320603) .b8 116
(EngineCore_DP0 pid=320603) .b8 111
(EngineCore_DP0 pid=320603) .b8 110
(EngineCore_DP0 pid=320603) .b8 0
(EngineCore_DP0 pid=320603) .b8 2                                   // DW_AT_language
(EngineCore_DP0 pid=320603) .b8 0
(EngineCore_DP0 pid=320603) .b8 114                                 // DW_AT_name
(EngineCore_DP0 pid=320603) .b8 111
(EngineCore_DP0 pid=320603) .b8 116
(EngineCore_DP0 pid=320603) .b8 97
(EngineCore_DP0 pid=320603) .b8 114
(EngineCore_DP0 pid=320603) .b8 121
(EngineCore_DP0 pid=320603) .b8 46
(EngineCore_DP0 pid=320603) .b8 112
(EngineCore_DP0 pid=320603) .b8 121
(EngineCore_DP0 pid=320603) .b8 0
(EngineCore_DP0 pid=320603) .b32 .debug_line                        // DW_AT_stmt_list
(EngineCore_DP0 pid=320603) .b8 47                                  // DW_AT_comp_dir
(EngineCore_DP0 pid=320603) .b8 104
(EngineCore_DP0 pid=320603) .b8 111
(EngineCore_DP0 pid=320603) .b8 109
(EngineCore_DP0 pid=320603) .b8 101
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 116
(EngineCore_DP0 pid=320603) .b8 97
(EngineCore_DP0 pid=320603) .b8 110
(EngineCore_DP0 pid=320603) .b8 103
(EngineCore_DP0 pid=320603) .b8 121
(EngineCore_DP0 pid=320603) .b8 97
(EngineCore_DP0 pid=320603) .b8 110
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 112
(EngineCore_DP0 pid=320603) .b8 114
(EngineCore_DP0 pid=320603) .b8 111
(EngineCore_DP0 pid=320603) .b8 106
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 104
(EngineCore_DP0 pid=320603) .b8 115
(EngineCore_DP0 pid=320603) .b8 99
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 118
(EngineCore_DP0 pid=320603) .b8 108
(EngineCore_DP0 pid=320603) .b8 108
(EngineCore_DP0 pid=320603) .b8 109
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 118
(EngineCore_DP0 pid=320603) .b8 108
(EngineCore_DP0 pid=320603) .b8 108
(EngineCore_DP0 pid=320603) .b8 109
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 118
(EngineCore_DP0 pid=320603) .b8 108
(EngineCore_DP0 pid=320603) .b8 108
(EngineCore_DP0 pid=320603) .b8 109
(EngineCore_DP0 pid=320603) .b8 95
(EngineCore_DP0 pid=320603) .b8 102
(EngineCore_DP0 pid=320603) .b8 108
(EngineCore_DP0 pid=320603) .b8 97
(EngineCore_DP0 pid=320603) .b8 115
(EngineCore_DP0 pid=320603) .b8 104
(EngineCore_DP0 pid=320603) .b8 95
(EngineCore_DP0 pid=320603) .b8 97
(EngineCore_DP0 pid=320603) .b8 116
(EngineCore_DP0 pid=320603) .b8 116
(EngineCore_DP0 pid=320603) .b8 110
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 111
(EngineCore_DP0 pid=320603) .b8 112
(EngineCore_DP0 pid=320603) .b8 115
(EngineCore_DP0 pid=320603) .b8 47
(EngineCore_DP0 pid=320603) .b8 116
(EngineCore_DP0 pid=320603) .b8 114
(EngineCore_DP0 pid=320603) .b8 105
(EngineCore_DP0 pid=320603) .b8 116
(EngineCore_DP0 pid=320603) .b8 111
(EngineCore_DP0 pid=320603) .b8 110
(EngineCore_DP0 pid=320603) .b8 0
(EngineCore_DP0 pid=320603)     }
(EngineCore_DP0 pid=320603)     .section        .debug_macinfo  {       }
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) ================================================================
(EngineCore_DP0 pid=320603) please share the reproducer above with Triton project.
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] EngineCore failed to start.
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] Traceback (most recent call last):
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/compiler.py", line 468, in make_cubin
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     subprocess.run(ptxas_cmd, check=True, close_fds=False, stderr=flog)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/subprocess.py", line 571, in run
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     raise CalledProcessError(retcode, process.args,
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] subprocess.CalledProcessError: Command '['/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas', '-lineinfo', '-v', '--gpu-name=sm_110a', '/tmp/tmpyrn4vdgn.ptx', '-o', '/tmp/tmpyrn4vdgn.ptx.o']' returned non-zero exit status 255.
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] 
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] During handling of the above exception, another exception occurred:
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] 
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] Traceback (most recent call last):
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 781, in run_engine_core
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 553, in __init__
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     super().__init__(
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 110, in __init__
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 221, in _initialize_kv_caches
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/v1/executor/abstract.py", line 88, in determine_available_memory
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/executor/uniproc_executor.py", line 74, in collective_rpc
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/utils/__init__.py", line 2977, in run_method
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return func(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return func(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/v1/worker/gpu_worker.py", line 280, in determine_available_memory
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     self.model_runner.profile_run()
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/v1/worker/gpu_model_runner.py", line 3692, in profile_run
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     dummy_encoder_outputs = self.model.get_multimodal_embeddings(
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 1589, in get_multimodal_embeddings
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     vision_embeddings = self._process_image_input(multimodal_input)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 1370, in _process_image_input
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     image_embeds = self.visual(pixel_values, grid_thw=grid_thw_list)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 919, in forward
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     hidden_states = blk(
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                     ^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 509, in forward
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     x_attn = self.attn(
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]              ^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 403, in forward
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     qk_rotated = apply_rotary_pos_emb_vision(qk_concat, rotary_pos_emb)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_vl.py", line 310, in apply_rotary_pos_emb_vision
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     output = rotary_emb_function(t_, cos, sin).type_as(t)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/vllm_flash_attn/layers/rotary.py", line 124, in apply_rotary_emb
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return ApplyRotaryEmb.apply(
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/autograd/function.py", line 581, in apply
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return super().apply(*args, **kwargs)  # type: ignore[misc]
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/vllm_flash_attn/layers/rotary.py", line 50, in forward
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     out = apply_rotary(
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]           ^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/proj/vllm/vllm/vllm_flash_attn/ops/triton/rotary.py", line 203, in apply_rotary
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     rotary_kernel[grid](
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/runtime/jit.py", line 733, in run
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     kernel = self.compile(src, target=target, options=options.__dict__)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/compiler/compiler.py", line 320, in compile
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     next_module = compile_ir(module, metadata)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/compiler.py", line 520, in <lambda>
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     stages["cubin"] = lambda src, metadata: self.make_cubin(src, metadata, options, self.target.arch)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/compiler.py", line 503, in make_cubin
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790]     raise PTXASError(error)
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] triton.runtime.errors.PTXASError: PTXAS error: Internal Triton PTX codegen error
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] `ptxas` stderr:
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] ptxas fatal   : Value 'sm_110a' is not defined for option 'gpu-name'
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] 
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] Repro command: /home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_110a /tmp/tmpyrn4vdgn.ptx -o /tmp/tmpyrn4vdgn.ptx.o
(EngineCore_DP0 pid=320603) ERROR 10-15 01:08:20 [core.py:790] 
(EngineCore_DP0 pid=320603) Process EngineCore_DP0:
(EngineCore_DP0 pid=320603) Traceback (most recent call last):
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/compiler.py", line 468, in make_cubin
(EngineCore_DP0 pid=320603)     subprocess.run(ptxas_cmd, check=True, close_fds=False, stderr=flog)
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/subprocess.py", line 571, in run
(EngineCore_DP0 pid=320603)     raise CalledProcessError(retcode, process.args,
(EngineCore_DP0 pid=320603) subprocess.CalledProcessError: Command '['/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas', '-lineinfo', '-v', '--gpu-name=sm_110a', '/tmp/tmpyrn4vdgn.ptx', '-o', '/tmp/tmpyrn4vdgn.ptx.o']' returned non-zero exit status 255.
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) During handling of the above exception, another exception occurred:
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) Traceback (most recent call last):
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=320603)     self.run()
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=320603)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 794, in run_engine_core
(EngineCore_DP0 pid=320603)     raise e
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 781, in run_engine_core
(EngineCore_DP0 pid=320603)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=320603)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 553, in __init__
(EngineCore_DP0 pid=320603)     super().__init__(
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 110, in __init__
(EngineCore_DP0 pid=320603)     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=320603)                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/engine/core.py", line 221, in _initialize_kv_caches
(EngineCore_DP0 pid=320603)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=320603)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/executor/abstract.py", line 88, in determine_available_memory
(EngineCore_DP0 pid=320603)     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/executor/uniproc_executor.py", line 74, in collective_rpc
(EngineCore_DP0 pid=320603)     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=320603)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/utils/__init__.py", line 2977, in run_method
(EngineCore_DP0 pid=320603)     return func(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=320603)     return func(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/worker/gpu_worker.py", line 280, in determine_available_memory
(EngineCore_DP0 pid=320603)     self.model_runner.profile_run()
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/v1/worker/gpu_model_runner.py", line 3692, in profile_run
(EngineCore_DP0 pid=320603)     dummy_encoder_outputs = self.model.get_multimodal_embeddings(
(EngineCore_DP0 pid=320603)                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 1589, in get_multimodal_embeddings
(EngineCore_DP0 pid=320603)     vision_embeddings = self._process_image_input(multimodal_input)
(EngineCore_DP0 pid=320603)                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 1370, in _process_image_input
(EngineCore_DP0 pid=320603)     image_embeds = self.visual(pixel_values, grid_thw=grid_thw_list)
(EngineCore_DP0 pid=320603)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=320603)     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=320603)     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 919, in forward
(EngineCore_DP0 pid=320603)     hidden_states = blk(
(EngineCore_DP0 pid=320603)                     ^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=320603)     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=320603)     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 509, in forward
(EngineCore_DP0 pid=320603)     x_attn = self.attn(
(EngineCore_DP0 pid=320603)              ^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=320603)     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=320603)     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 403, in forward
(EngineCore_DP0 pid=320603)     qk_rotated = apply_rotary_pos_emb_vision(qk_concat, rotary_pos_emb)
(EngineCore_DP0 pid=320603)                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/model_executor/models/qwen2_vl.py", line 310, in apply_rotary_pos_emb_vision
(EngineCore_DP0 pid=320603)     output = rotary_emb_function(t_, cos, sin).type_as(t)
(EngineCore_DP0 pid=320603)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/vllm_flash_attn/layers/rotary.py", line 124, in apply_rotary_emb
(EngineCore_DP0 pid=320603)     return ApplyRotaryEmb.apply(
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/torch/autograd/function.py", line 581, in apply
(EngineCore_DP0 pid=320603)     return super().apply(*args, **kwargs)  # type: ignore[misc]
(EngineCore_DP0 pid=320603)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/vllm_flash_attn/layers/rotary.py", line 50, in forward
(EngineCore_DP0 pid=320603)     out = apply_rotary(
(EngineCore_DP0 pid=320603)           ^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/proj/vllm/vllm/vllm_flash_attn/ops/triton/rotary.py", line 203, in apply_rotary
(EngineCore_DP0 pid=320603)     rotary_kernel[grid](
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/runtime/jit.py", line 419, in <lambda>
(EngineCore_DP0 pid=320603)     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(EngineCore_DP0 pid=320603)                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/runtime/jit.py", line 733, in run
(EngineCore_DP0 pid=320603)     kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup)
(EngineCore_DP0 pid=320603)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/runtime/jit.py", line 861, in _do_compile
(EngineCore_DP0 pid=320603)     kernel = self.compile(src, target=target, options=options.__dict__)
(EngineCore_DP0 pid=320603)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/compiler/compiler.py", line 320, in compile
(EngineCore_DP0 pid=320603)     next_module = compile_ir(module, metadata)
(EngineCore_DP0 pid=320603)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/compiler.py", line 520, in <lambda>
(EngineCore_DP0 pid=320603)     stages["cubin"] = lambda src, metadata: self.make_cubin(src, metadata, options, self.target.arch)
(EngineCore_DP0 pid=320603)                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=320603)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/compiler.py", line 503, in make_cubin
(EngineCore_DP0 pid=320603)     raise PTXASError(error)
(EngineCore_DP0 pid=320603) triton.runtime.errors.PTXASError: PTXAS error: Internal Triton PTX codegen error
(EngineCore_DP0 pid=320603) `ptxas` stderr:
(EngineCore_DP0 pid=320603) ptxas fatal   : Value 'sm_110a' is not defined for option 'gpu-name'
(EngineCore_DP0 pid=320603) 
(EngineCore_DP0 pid=320603) Repro command: /home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_110a /tmp/tmpyrn4vdgn.ptx -o /tmp/tmpyrn4vdgn.ptx.o
(EngineCore_DP0 pid=320603) 
[rank0]:[W1015 01:08:20.377829112 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=320511) Traceback (most recent call last):
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/bin/vllm", line 7, in <module>
(APIServer pid=320511)     sys.exit(main())
(APIServer pid=320511)              ^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=320511)     args.dispatch_function(args)
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/entrypoints/cli/serve.py", line 62, in cmd
(APIServer pid=320511)     uvloop.run(run_server(args))
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
(APIServer pid=320511)     return __asyncio.run(
(APIServer pid=320511)            ^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=320511)     return runner.run(main)
(APIServer pid=320511)            ^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=320511)     return self._loop.run_until_complete(task)
(APIServer pid=320511)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=320511)     return await main
(APIServer pid=320511)            ^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/entrypoints/openai/api_server.py", line 1914, in run_server
(APIServer pid=320511)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/entrypoints/openai/api_server.py", line 1930, in run_server_worker
(APIServer pid=320511)     async with build_async_engine_client(
(APIServer pid=320511)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=320511)     return await anext(self.gen)
(APIServer pid=320511)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/entrypoints/openai/api_server.py", line 191, in build_async_engine_client
(APIServer pid=320511)     async with build_async_engine_client_from_engine_args(
(APIServer pid=320511)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=320511)     return await anext(self.gen)
(APIServer pid=320511)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/entrypoints/openai/api_server.py", line 238, in build_async_engine_client_from_engine_args
(APIServer pid=320511)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=320511)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/utils/__init__.py", line 1336, in inner
(APIServer pid=320511)     return fn(*args, **kwargs)
(APIServer pid=320511)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/v1/engine/async_llm.py", line 205, in from_vllm_config
(APIServer pid=320511)     return cls(
(APIServer pid=320511)            ^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/v1/engine/async_llm.py", line 129, in __init__
(APIServer pid=320511)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=320511)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
(APIServer pid=320511)     return AsyncMPClient(*client_args)
(APIServer pid=320511)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/v1/engine/core_client.py", line 807, in __init__
(APIServer pid=320511)     super().__init__(
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/v1/engine/core_client.py", line 468, in __init__
(APIServer pid=320511)     with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=320511)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=320511)   File "/home/xxx/miniconda3/envs/vllm-cu130/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=320511)     next(self.gen)
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/v1/engine/utils.py", line 815, in launch_core_engines
(APIServer pid=320511)     wait_for_engine_startup(
(APIServer pid=320511)   File "/home/xxx/proj/vllm/vllm/v1/engine/utils.py", line 872, in wait_for_engine_startup
(APIServer pid=320511)     raise RuntimeError(
(APIServer pid=320511) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

[johnnynunez]

thanks @johnnynunez , i tried the qwen2.5-3b, and it worked well with your installation script.

But there's a bit of a problem with qwen2.5vl-3b.

• env setup script:

conda activate vllm-cu130
pip3 install torch==2.9.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/test/cu130
git clone https://github.com/vllm-project/vllm.git
cd vllm
python use_existing_torch.py
pip install -r requirements/build.txt
pip install --no-build-isolation -e .

• core error log:

...
(EngineCore_DP0 pid=320603) Internal Triton PTX codegen error
(EngineCore_DP0 pid=320603) `ptxas` stderr:
(EngineCore_DP0 pid=320603) ptxas fatal   : Value 'sm_110a' is not defined for option 'gpu-name'
...

full error log here

it is easy to fix

export TORCH_CUDA_ARCH_LIST=11.0a # Thor, for Spark 12.1a
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

[sc-hua]

it works, thanks a lot :)

[johnnynunez]

it works, thanks a lot :)

i am curious, did you install from vllm nightly?