the network output will become nan

[15066952332]

I am very grateful to the author for the pytorch version code. During my training, I found that the network output will become nan. Has the author encountered such a problem?

[RonyAbecidan]

Hello,

Normally, for each image, the expected output should be a binary map with just pixels equal to 0 or 1 but I guess that you have some masks that are not satisfying this constraint. This may explain your error ;)

[15066952332]

Thank you very much for your answer, but I encountered this problem during the training process. During the training, the output of the first few iterations is 0~1, and then the iterations will appear nan. Have you ever encountered this kind of problem? Or is the size of my data set image related to this? My picture input is 512*512，I found out that the model is output from IMTFE part becomes nan

[RonyAbecidan]

I am not sure why it is becoming nan suddenly. Can you share your training code ? I will check the IMTFE meanwhile. Maybe you have to force the output pixels to be in [0,1] in your loss using a clipping. Can you also check that all your target masks contain only 0 or 1 pixels ?

[15066952332]

Thanks a lot，This is the code of my train

The model is defined as Mantranet
“Can you also check that all your target masks contain only 0 or 1 pixels ?”Some of my target masks are tampered images (the mask has 0 and 1), and some are untampered images (the mask is all 0). Will this data set be affected?

[RonyAbecidan]

I have some comments about your code that may explain your error :

1 - You need to provide probabilities to BCE_loss in "seg_out". To do so, you have to use "nn.Sigmoid()" and apply it to seg_out before passing it in the BCE_loss (Cf https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html)
2 - I don't understand why do you need to use "nn.functional.interpolate". Mantranet is supposed to give you back an output with the same size as the input.

[RonyAbecidan]

Hello,

Did you fix your issues finally ?

[tmdrn9]

hello, i have same problem. i use nn.BCEWithLogitsLoss and my target masks contain only 0 or 1 pixels... Could you help me please?

[RonyAbecidan]

hello, i have same problem. i use nn.BCEWithLogitsLoss and my target masks contain only 0 or 1 pixels... Could you help me please?

I think this could happen because some probabilities returned are too close to zero making the loss diverging. People usually add a small epsilon value to the prediction to prevent this divergence. Could you try that and tell me if it's fixed ? :)