Reproducibility about stagin HOT 4 CLOSED

Hi @GMLB1997,

Could you try with the HCP dataset directly downloaded from https://db.humanconnectome.org?
The downloaded data should be organized as indicated in the dataset section of the README.md.
Data downloaded from other source cannot be guaranteed to reproduce exact result because the preprocessing step may differ.

from stagin.

Hi. The HCP dataset used in ST-GCN is also downloaded from https://db.humanconnectome.org, but just different brain parcellation, and I think model comparison on the same dataset with the same parcellation is fair.

from stagin.

Thank you for pointing out an important point. I agree with you that it is very difficult to appropriately compare performance of different GNN-fMRI models, given that not only the model, but also how the 4D fMRI data is processed can largely affect the results. I am not sure about the HCP dataset that you are using so I cannot give you a definite answer, but I will assume that the only difference is the parcellation, as you have mentioned.

Different parcellation can matter, and affect the results. If number of nodes get smaller, applying STAGIN should be cautious since our method employs binarization of the FC matrix to obtain unweighted simple graph. If the number of node is small (let's say 20), the graph structure may not carry enough information after the binarization. As far as I know, ST-GCN does not binarize the FC matrix, keeping the weights of the FC matrix. Maybe binarizing the input FC in ST-GCN or using the weighted input FC in STAGIN may reflect a slightly more 'fair' comparison. (I would say 'slightly' because there are still many other factors that causes performance difference between the two models, leading to 'unfair' benchmarking)

Another point that should be considered is that additional hyperparameter tuning might be required to fit the ROI-Timeseries data from another parcellation. If you directly apply any other type of data to the model, it might overfit and show suboptimal results because the hyperparameters are set to the best ones regarding the dataset/parcellation in our experimental settings. This means that if you try to fit ROI-Timeseries data extracted from our pipeline to another model (say ST-GCN) directly, this may also give suboptimal results on that model too.

I believe that there is a large need for benchmarking the best data processing pipelines for the fMRI-GNN (and other neural network applications) experiments. I hope that I will be able to benchmark various ways to process the fMRI data for neural network application, but would also deeply appreciate any researcher who would do the important job!

from stagin.

Thanks for your reply!

from stagin.