Coordinates, projection, geo-referencing

[Atmoboran]

Hello,

I was wondering if the models require specific CRS settings in order to work correctly? I have not found any information about this. The demos on huggingface seem to work with different CRS and the example files also seem to have different CRSs, but just want to make sure if this is correct or if the models will work better if the correct CRS is used.

Best regards,
atmoboran

[Joao-L-S-Almeida]

Hi, @Atmoboran. Could you provide more information ? For what models are you specifically interested in ?

[daniszw]

Hi @Atmoboran!
Are you referring to specific models? If so, which examples are you talking about?
In general, if you are using a pre-trained model to finetune it to a downstream task, the closer the data is to the pre-trained data the better (by closer I mean similar data characteristics like resolution, bands/variables, CRS, and so on). However, this doesn’t mean the model cannot be adapted to different data characteristics. Depending on how different is your data compared to the data the model was pre-trained with, it might take several epochs to learn this adaptation, or the pre-training knowledge might not help much. In any case, this is a general idea, and depending on the model it might require some specific type of data/set of variables.

[Atmoboran]

Hi @Joao-L-S-Almeida and @daniszw ,
thanks for the quick answers and sorry for not specifying my problem. I'm looking at the Prithvi model (1.0 and 2.0) demos on huggingface (https://huggingface.co/spaces/ibm-nasa-geospatial/Prithvi-EO-2.0-Demo).
"The demos on huggingface seem to work with different CRS" - I meant that I have downloaded and created my own HLS data and tested the inference and it worked even though the CRS was different compared to the example files provided on huggingface. I can't really tell the difference in performance, as the demo doesn't provide a MSE score (would be a nice feature to add to the demo btw).
"However, this doesn’t mean the model cannot be adapted to different data characteristics." - Ok, good to know. For me the question of how I should pre-process my own data in order to have the best possible inference/ fine-tuning is not really well communicated in the docs. I think it would be nice to have an example workflow for applying the models from scratch, as I guess most users will want to use it on their own data.

Thanks and best regards,
Boran

[daniszw]

Thanks for the clarification and suggestions :) these are good points.
For inference with the pre-trained model without any fine-tuning, I would recommend using HLS data similar to the examples in the demo (similar CRS and image sizes, but the model should be able to handle some differences there) and keep the 6 bands we used in pre-training (RGB, NIR, SWIR1, SWIR2).
For fine-tuning, if you are not freezing the encoder, you have more freedom to change the data characteristics more significantly as you are adapting the model to the new data. However, as I mentioned, if the data is very different indeed, it might take several epochs for the model to learn this adaptation. We fine-tuned Prithvi-EO in several data scenarios (RGB inputs only, higher resolutions, different image sizes...). You can checkout our technical report for more details and also some examples of fine-tuning here.

[Atmoboran]

Just to add up to this conversation: Here is my investigation of all the example files of the various Prithvi EO-2.0 demos (https://huggingface.co/ibm-nasa-geospatial).

So we can see the differences in the used CRSs. So the question is: if model_X is pre-trained with CRS_1, will the performance be worse if it is fine-tuned with CRS_2 or is the model able to understand geolocation independent of the CRS choice/ is it able to do the geo-transformation itself?

This is what the technical report says about geolocation:

"For the 3D positional encodings, we first generate 1D sin/cos encodings individually for time, height, and width dimensions
and then combine them together into a single, 3D positional encoding.
Second, we considered metadata on geolocation (center latitude and longitude) and date
(year and day-of-year ranging 1-365) in pretraining. The model receives time and location
information for each sample and encodes them independently using 2D sin/cos encoding. ... Since this metadata is often not
available, we decided to include it not as additional inputs in patch embeddings, but as a simple bias, similar to the idea of positional encodings. For the same reason, we also added a drop mechanism during pretraining that randomly drops the geolocation and/or the temporal data to help the model learn how to handle the absence of this information."