problem about the vision tower

[Champerrin]

**When I trying to predict with llava-fastvithd_0.5b_stage3, it's totally fine.

But then I tried llava-fastvithd_7b_stage3 or llava-fastvithd_7b_stage2, here goes a error that:**

Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00, 1.20s/it]
Some parameters are on the meta device because they were offloaded to the cpu.
Traceback (most recent call last):
File "/mnt/linuxdata/ml-fastvlm-main/predict.py", line 87, in
predict(args)
File "/mnt/linuxdata/ml-fastvlm-main/predict.py", line 55, in predict
output_ids = model.generate(
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/mnt/linuxdata/ml-fastvlm-main/llava/model/language_model/llava_qwen.py", line 126, in generate
) = self.prepare_inputs_labels_for_multimodal(
File "/mnt/linuxdata/ml-fastvlm-main/llava/model/llava_arch.py", line 210, in prepare_inputs_labels_for_multimodal
image_features = self.encode_images(images)
File "/mnt/linuxdata/ml-fastvlm-main/llava/model/llava_arch.py", line 142, in encode_images
image_features = self.get_model().get_vision_tower()(images)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/accelerate/hooks.py", line 176, in new_forward
output = module._old_forward(*args, **kwargs)
File "/mnt/linuxdata/ml-fastvlm-main/llava/model/multimodal_encoder/mobileclip_encoder.py", line 72, in forward
return self.forward_images(images)
File "/mnt/linuxdata/ml-fastvlm-main/llava/model/multimodal_encoder/mobileclip_encoder.py", line 85, in forward_images
image_forward_outs = self.vision_tower(images.to(device=self.device, dtype=self.dtype), return_image_embeddings=True)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/accelerate/hooks.py", line 171, in new_forward
args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/accelerate/hooks.py", line 370, in pre_forward
return send_to_device(args, self.execution_device), send_to_device(
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 169, in send_to_device
return honor_type(
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 81, in honor_type
return type(obj)(generator)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 170, in
tensor, (send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys) for t in tensor)
File "/mnt/linuxdata/anaconda3/envs/fastvlm/lib/python3.
NotImplementedError: Cannot copy out of meta tensor; no data!