different structure files?

[borondics]

Hi @joshduran,

Awesome work! However, I ran into a problem. I have some files that are generated during running the "Check Signal" in OPUS which I would like to process on the fly. The structure might be different because there is a "list index out of range" error.

I attached two examples here from different spectrometers.

It would be an enormous help if your package could handle them. I can send you more examples if you need!

Cheers and thanks in advance!

examples.zip

[joshduran]

Thank you for reaching out and supplying some example files. I only had time for a quick look, but so far this is what I know:

One of the files you provided is in the standard format but was missing a file block reference that has been there for every other file I've tested. That reference is not critical for reading the file contents so I fixed the code to ignore that reference if it is missing. That file only had an interferogram and parameters present.

The other files were not formatted the same as a standard opus file. At first glance it looks like it will be a substantial effort since the format did not look similar.

I'm curious what version of opus you're using that allows you to snap a spectrum from the check signal screen. My version doesn't have that feature.

For now, I just released a beta version that has preliminary support for reading reports. Despite its beta status, the rest of the code is stable and includes the fix I mentioned for the one file that is compatible. You can try it to see if it will at least help you for a subset of what you need.

[borondics]

@joshduran thanks for looking into this! I tested the add_mev branch and I can now read the a5333.0 but as you said not the a354.0 file. This is actually the one I really need. :/

Recording this file doesn't involve OPUS, I did it through the instrument interface and downloaded it from the browser directly so it is the instrument (for sure because I did it on two different spectrometers) or the firmware that is different.

If you want I could send you more examples of this type with some variation. I think it would be more practical to discuss via email especially for sharing larger files. My email is ferenc.borondics--at--synchrotron-soleil.fr

Thanks for all the efforts!!!

[borondics]

OK, after some more debugging it seems that a354.0 is corrupted, even my OPUS can't open it. However, here is an equivalent file, which I could open but didn't fully debug yet.

When I open this in OPUS I get the following report:

Notice that there is no data block under the file name on the left but there is an IgSm/Compact field that seems to contain the interferogram I need.

I attach the file here:

a.0.zip

[borondics]

Doing more diagnostics today I see that I can read the file but there are some weird values in the interferogram. If I plot the whole data.y I get this:

and if I go it from an arbitrarily selected point (in this case data point 100) on the IFG looks just fine.

The question is how those values appear in the dataset. In this particular case it is point 74 that has a large negative value. I will record more files and check if this is a systematic thing.

[joshduran]

@borondics, sorry I just noticed you replied with more info. Thanks for the new file and it does seem like I should be able to get these files to work once I figure out the precise change needed.

What I'm noticing is that this interferogram is classified somewhat differently than a normal data set (which is where the "Compact" name likely comes from). What I will have to check is if this classification implies that the data should be read with different logic, or if it is just a way of declaring to the user what type of data it is. This is what I see when plotting the igsm data extracted with the current version of brukeropus:

If I zoom into the left hand of that plot I see this:

This to me looks like all of the data up to point 103 or 104 is garbage or corrupt in some way. Looking at your table, the very first point should have a value of -0.001096, but brukeropus is reading it as 3.0204e-41. I'm wondering if there is potentially a data type issue here, but it will take me a little while to debug. It is strange that the rest of the spectrum appears to be normal...

[joshduran]

Upon further investigation, I am realizing that this classification of data block has its own unique storage format. It contains some sort of header information that precedes the floating point data. I was able to manually extract the true interferogram by offsetting the start of the data parsing:

the first value of this interferogram is: -0.001095525, which is consistent with OPUS.

A workaround I could publish is to simply ignore the header information in the block; I imagine most users would be fine with this, but it would be more elegant to figure out how to parse this new header.

Would you mind generating a few more files that are very similar, but different from a.0? For example, you could change the spectral resolution, wanted high frequency, or how/when the file is captured to attempt to get some variety in that header with varying interferogram points.

Thanks!

[joshduran]

@borondics , go ahead and try the latest version (1.3) I released earlier today. It has a quick fix for these compact spectra. The fix worked for the small sample set I tried, but let me know if you run into any files that do not work properly. Since these are "compact" spectra according to opus, I have appended them with a "_c" to denote that. For example, your compact sample interferogram would be accessed through the"igsm_c" attribute. They otherwise behave as normal data spectra.

Thanks for bringing this to my attention!

[borondics]

Hi @joshduran, thanks a lot for the improvements! Sorry for the silence, I am swamped in the past weeks. I'll try them when I get to my spectrometer and it is not used by colleagues and let you know about the result.