Runtime error on tensor dimensionnality in automatic mask generator

[vchalmel]

Hello, I tried to replicate some experiment (from RapidBenthos on github) using segment-anything and I get a runtime error "RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 2 is not equal to len(dims) = 3"

I have no clue about the root cause of that error and I would like some explanations about what is going on in that stack trace to know in which direction to look.

the code is using segment-anything-py==1.0 through segment-geospatial==0.9.0

File [~/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/automatic_mask_generator.py:163](http://127.0.0.1:8888/lab/tree/Git/RapidBenthos/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/automatic_mask_generator.py#line=162), in SamAutomaticMaskGenerator.generate(self, image)
    138 """
    139 Generates masks for the given image.
    140 
   (...)
    159          the mask, given in XYWH format.
    160 """
    162 # Generate masks
--> 163 mask_data = self._generate_masks(image)
    165 # Filter small disconnected regions and holes in masks
    166 if self.min_mask_region_area > 0:

File [~/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/automatic_mask_generator.py:206](http://127.0.0.1:8888/lab/tree/Git/RapidBenthos/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/automatic_mask_generator.py#line=205), in SamAutomaticMaskGenerator._generate_masks(self, image)
    204 data = MaskData()
    205 for crop_box, layer_idx in zip(crop_boxes, layer_idxs):
--> 206     crop_data = self._process_crop(image, crop_box, layer_idx, orig_size)
    207     data.cat(crop_data)
    209 # Remove duplicate masks between crops

File [~/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/automatic_mask_generator.py:236](http://127.0.0.1:8888/lab/tree/Git/RapidBenthos/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/automatic_mask_generator.py#line=235), in SamAutomaticMaskGenerator._process_crop(self, image, crop_box, crop_layer_idx, orig_size)
    234 cropped_im = image[y0:y1, x0:x1, :]
    235 cropped_im_size = cropped_im.shape[:2]
--> 236 self.predictor.set_image(cropped_im)
    238 # Get points for this crop
    239 points_scale = np.array(cropped_im_size)[None, ::-1]

File [~/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/predictor.py:58](http://127.0.0.1:8888/lab/tree/Git/RapidBenthos/Git/RapidBenthos/venv/lib/python3.12/site-packages/segment_anything/predictor.py#line=57), in SamPredictor.set_image(self, image, image_format)
     56 input_image = self.transform.apply_image(image)
     57 input_image_torch = torch.as_tensor(input_image, device=self.device)
---> 58 input_image_torch = input_image_torch.permute(2, 0, 1).contiguous()[None, :, :, :]
     60 self.set_torch_image(input_image_torch, image.shape[:2])

RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 2 is not equal to len(dims) = 3

[heyoeyo]

It seems to be saying the input image only has 2 dimensions (e.g. height & width) while the code is trying to re-arrange something with 3 dimensions (e.g. height, width & channels). So I'd guess the problem is the input is a grayscale image, while the model expects RGB.
If you're loading the image with opencv, you can make it rgb before giving it to the model with something like:

rgb_img = cv2.cvtColor(gray_img, cv2.COLOR_GRAY2RGB)

[vchalmel]

Thanks for the reply, I will try to find how to add something like this