ENH: Add a script to query fMRI compliant data from OpenNeuro

[jhlegarreta]

Add a script to query fMRI compliant data from OpenNeuro.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.35%. Comparing base (fb8f3f3) to head (726a6ea).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #89   +/-   ##
=======================================
  Coverage   71.35%   71.35%           
=======================================
  Files          23       23           
  Lines        1138     1138           
  Branches      139      139           
=======================================
  Hits          812      812           
  Misses        283      283           
  Partials       43       43           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jhlegarreta]

I revisited this.

In 736e276 I split the OpenNeuro querying and BOLD feature extraction/analysis/selection. Querying OpenNeuro dataset information and querying the files of each dataset is also split into two files: scripts/query_datasets_openneuro.py and scripts/query_mri_dataset_files_openneuro.py in order to separate querying OpenNeuro (datasets and files). The rationale is to minimize the number of times we need to query OpenNeuro. For now, 736e276 only does the querying job.

I've only tested scripts/query_mri_dataset_files_openneuro.py with the first ten datasets that are listed by the first script's output.

Next is to develop/refactor the necessary parts for extracting the relevant features and selecting those datasets/files complying with the criteria into different scripts.

[jhlegarreta]

Re #89 (comment)

Next is to develop/refactor the necessary parts for extracting the relevant features and selecting those datasets/files complying with the criteria into different scripts.

Done in subsequent commits.

I think I am done with this. Thoughts:

• Some datasets are no longer BIDS-compliant (visible as errors/warnings when browsing OpenNeuro) or seem to have some issues. Do we want to identify those? How can we identify these programmatically?
• I am only keeping those datasets that have human in their species field. Most datasets have the species field empty in the response received from GraphQl. Not sure if this field is the same as in the JSON sidecar. Should we fetch those where the tag is empty? Should we double-check with the JSON sidecar? Looking for e.g. "mouse", "mice", "rat", macaque", "non-human primate" in the description sounds brittle. If we decide to include any of these a visual QA (if an automatic one does not exist) should be done.
• As opposed to the initial approach in ea8ee4c#diff-c8a4bb6425ad95a7dd63f31feb2f1ef0a676b7fd2659e6cb16d4d91891cd7e40R184 and ea8ee4c#diff-c8a4bb6425ad95a7dd63f31feb2f1ef0a676b7fd2659e6cb16d4d91891cd7e40R245, I am exclusively relying on the file list and shape of the NIfTI header instead of the JSON sidecar to get the number of runs/timepoints.
• I am not checking whether there is some meaningful fMRI task, e.g. ea8ee4c#diff-c8a4bb6425ad95a7dd63f31feb2f1ef0a676b7fd2659e6cb16d4d91891cd7e40R270

Once these have been addressed and the scripts do not need updates, I'll squash all commits.

Additionally, as I've split all this into 4 different scripts, I think it has become apparent that most of the scripts in the scripts folder do not belong here, and should be put into a separate repository where all scripts to produce context results should dwell, keeping here only those scripts that are necessary for the package to run, if any. We may want to do this now to avoid polluting the folder.

[jhlegarreta]

Re: the commit message 4f9b1e9, re-run earlier commits again; 9f66a50 still incurs a deadlock; 2567252 brings query_mri_dataset_files_openneuro.py to completion (without handling errors), so I will try to bring changes one by one to query_mri_dataset_files_openneuro.py and see where the deadlock starts to happen. Thanks for the patience.

[jhlegarreta]

OK, this is working. Takes like 11 hours, but it is working, as opposed to less than two hours with a headlong rush that only does

return response.json()["data"]["snapshot"]["files"]

or

return response.json().get("data", {}).get("snapshot").get("files", []) or []

(the latter only solving part of the NoneType object is not iterable exceptions)

There is probably things to improve, but I think it serves the purpose and I've spent enough time on this. The retries are nice as they demonstrate that we've made efforts to get the widest possible set of cases: e.g. in the last run, all datasets were queried successfully (then getting the number of bold runs in the selection script fails for reasons related to malformed NIfTI files or other reasons, i.e. unrelated to this).

I will go ahead and squash all commits into one and then fix the style issues. At that point this will be ready to be merged. Apologies the noise generated by all the commits here.

The decision to transfer all these scripts to another repository (x-ref #89 (comment)) can be made once this is merged and be applied to all scripts.

Edit: I see that there is some issue when returning the failed queries in scripts/query_mri_dataset_files_openneuro.py depending on the exception: no failed queries may be reported, but the written dataset_id.tsv files may not account for all datasets. Anyways, the offset is only a few datasets: 4 in two different runs I did. So I am inclined to go ahead with this as is, and open an issue; I've already spent a lot of time with this. Unless we come up with a way to simulate responses from a server in a consistent manner, debugging this will be hard.

[jhlegarreta]

Edit: I see that there is some issue when returning the failed queries in scripts/query_mri_dataset_files_openneuro.py depending on the exception: no failed queries may be reported, but the written dataset_id.tsv files may not account for all datasets. Anyways, the offset is only a few datasets: 4 in two different runs I did. So I am inclined to go ahead with this as is, and open an issue; I've already spent a lot of time with this. Unless we come up with a way to simulate responses from a server in a consistent manner, debugging this will be hard.

Last force push (eeeed67) solves this.

[oesteban]

Looks good overall. I missed having examples in the initial docstring of every file.

Considering that all these scripts are heavily related, and the shared static variables, would it make sense to aggregate all of them into a single script, and select the behavior with click commands? Something like:

$ openneuro_query.py profile
$ openneuro_query.py datasets
$ openneuro_query.py files
$ openneuro_query.py select

[jhlegarreta]

Considering that all these scripts are heavily related, and the shared static variables, would it make sense to aggregate all of them into a single script, and select the behavior with click commands?

I like the idea, but I'd propose to do it in a separate PR in order to have a working version at least. I know merging this clutters the repository, and in other circumstances, I'd be for adopting the approach here, but in this case I believe that I've spent too much time on this PR, so I'd rather have something merged and work from there. Also, as said, I think these scripts, together with the rest of the scripts that are not essential for the package, should dwell in a separate repository. So I'd vote for merging this as is to have a record of something that works, then transferring to another repository and converting to a click-based script in that repository.

[jhlegarreta]

I missed having examples in the initial docstring of every file.

Examples added.

[jhlegarreta]

So, feel free to open a new repository to specifically host only these scripts. I will push the scripts to there when that is done, then work on the click re-implementation, then (mid- to long-term) to generalizing to any modality.