Handling the labels with 0 samples

[Fatima-Haouari]

Hi,
I am using the class balanced focal loss to compute the loss for each batch of my training data. However for some batches the number of samples for some labels can be 0 which causes the weights to be equal to nan and consequently the loss will be nan.

1. Could you please advise on how to mitigate this issue?
2. Is the number of samples per class related to the full training data or to the batch we want to compute the loss for?
3. If I am fine-tuning my models and computing the loss for the validation set should I consider the samples per class according to the validation set?
   Your help is really appreciated.

[fcakyon]

Hello @Fatima-Haouari, thanks for your interest!

1-2: The number of samples per class is related to the full training data, not the batch data.
3: Validation loss should be also calculated by the training stats. Using different stats in training and val loss would create a misalignment between the losses.

[Fatima-Haouari]

Thanks for your quick response. I appreciate it. When you say full training data this means including the train and valid sets samples right? or I just use my train data to count occurrences of each class?

[fcakyon]

Actually, we use only the training data (not including the val samples) when calculating the stats since the model update will be calculated using the loss of training data.

[Fatima-Haouari]

Great thanks.

[ardiantovn]

Hi, I noticed that this line of code
weights = (1.0 - self.beta) / np.array(effective_num)
may cause a division by zero if any label with 0 sample.
What if this code were changed as follows:
weights = np.where(effective_num != 0., (1.0 - self.beta) / effective_num, 0.).
Therefore, the label with 0 sample will have weight_value=0 instead of np.infinity.