help
help > RE: Question reg second level results
Dec 11, 2015 01:12 AM | Alfonso Nieto-Castanon - Boston University
RE: Question reg second level results
Hi Macia and Johnson,
Just to add a related comment, the issue of the choice of voxel-level height threshold has recently re-gained some attention in Eklund et al. "can parametric statistical methods be trusted for fMRI based group studies" manuscript (this is still a pre-print so take with a grain of salt). The point of this manuscript is that cluster-level statistics may be biased/invalid when used in the context of "relatively high" voxel-level thresholds (uncorrected height thresholds above p=.001). In general some of the random field theory assumptions and approximations are asymptotic and only exact when the chosen threshold are very low. Some original investigations about this issue (e.g. Hayasaka S1, Nichols TE. (2003) Validating cluster size inference: random field and permutation methods. Neuroimage 20(4):2343-56) seemed to suggest that using relatively liberal height thresholds (e.g. uncorrected p=.01 height thresholds) would results in conservative cluster-level statistics (not liberal/invalid), while the new manuscript (and Nichols is also an author in this manuscript) would suggest the opposite (too liberal/invalid cluster-level stats when used in combination with p=.01 height thresholds). The bottom line of all this may be that we should probably use caution when using height thresholds above p=.001 uncorrected, and if you do wish to use higher thresholds (e.g. lower thresholds are most sensitive to strong activations, while higher thresholds should be more sensitive to weaker activation clusters that may extend over larger areas) it is probably a good recommendation to use instead (or at least validate your results using) non-parametric statistics.
In the next release of CONN, which should be out just in a few days, we are addressing this concern by adding the ability to use non-parametric statistics for all of your second-level analyses (this was already available in CONN for surface-based and ROI-to-ROI analyses, we have now also added non-parametric statistics for voxel-based analyses as well), so you will be able to choose between parametric (i.e. random field theory) and non-parametric (residual permutation/randomization tests) statistics to further investigate this issue on your own to see how this applies to connectivity analyses, and/or use your preferred method in your specific second-level analyses. We will keep an eye on this issue as well just to see if a community consensus is reached and a change in the default settings in CONN is also warranted (the current default settings for voxel-based analyses uses a p=.001 uncorrected height threshold combined with FDR-corrected cluster-level p<.05 threshold (parametric statistics), and currently all indications seem to suggest that this is a safe/valid approach)
Hope this helps
Alfonso
Originally posted by Macià Buades-Rotger:
Just to add a related comment, the issue of the choice of voxel-level height threshold has recently re-gained some attention in Eklund et al. "can parametric statistical methods be trusted for fMRI based group studies" manuscript (this is still a pre-print so take with a grain of salt). The point of this manuscript is that cluster-level statistics may be biased/invalid when used in the context of "relatively high" voxel-level thresholds (uncorrected height thresholds above p=.001). In general some of the random field theory assumptions and approximations are asymptotic and only exact when the chosen threshold are very low. Some original investigations about this issue (e.g. Hayasaka S1, Nichols TE. (2003) Validating cluster size inference: random field and permutation methods. Neuroimage 20(4):2343-56) seemed to suggest that using relatively liberal height thresholds (e.g. uncorrected p=.01 height thresholds) would results in conservative cluster-level statistics (not liberal/invalid), while the new manuscript (and Nichols is also an author in this manuscript) would suggest the opposite (too liberal/invalid cluster-level stats when used in combination with p=.01 height thresholds). The bottom line of all this may be that we should probably use caution when using height thresholds above p=.001 uncorrected, and if you do wish to use higher thresholds (e.g. lower thresholds are most sensitive to strong activations, while higher thresholds should be more sensitive to weaker activation clusters that may extend over larger areas) it is probably a good recommendation to use instead (or at least validate your results using) non-parametric statistics.
In the next release of CONN, which should be out just in a few days, we are addressing this concern by adding the ability to use non-parametric statistics for all of your second-level analyses (this was already available in CONN for surface-based and ROI-to-ROI analyses, we have now also added non-parametric statistics for voxel-based analyses as well), so you will be able to choose between parametric (i.e. random field theory) and non-parametric (residual permutation/randomization tests) statistics to further investigate this issue on your own to see how this applies to connectivity analyses, and/or use your preferred method in your specific second-level analyses. We will keep an eye on this issue as well just to see if a community consensus is reached and a change in the default settings in CONN is also warranted (the current default settings for voxel-based analyses uses a p=.001 uncorrected height threshold combined with FDR-corrected cluster-level p<.05 threshold (parametric statistics), and currently all indications seem to suggest that this is a safe/valid approach)
Hope this helps
Alfonso
Originally posted by Macià Buades-Rotger:
Hi Johnson,
I'm only a CONN user but I think I can answer your question. Usual whole-brain acceptable thresholds are p<0.005 or p<0.001 uncorrected at the peak level, with a cluster-wise FDR or FWE correction of p<0.05. If you use a peak-level p<0.05 threshold, as you have seen, you get huge clusters which, of course, survive cluster-level correction because it is statistically unlikely that such big clusters appear by chance. However, these massive clusters actually arise because you used an excessively liberal peak-level threshold, and they are furthermore not informative because they are very widespread and don't actually take advantage of the spatial resolution of fMRI.
Bottomline: you want to obtain small but reliable clusters, so use more astringent thresholds.
I hope that helped!
All the best,
Macià
I'm only a CONN user but I think I can answer your question. Usual whole-brain acceptable thresholds are p<0.005 or p<0.001 uncorrected at the peak level, with a cluster-wise FDR or FWE correction of p<0.05. If you use a peak-level p<0.05 threshold, as you have seen, you get huge clusters which, of course, survive cluster-level correction because it is statistically unlikely that such big clusters appear by chance. However, these massive clusters actually arise because you used an excessively liberal peak-level threshold, and they are furthermore not informative because they are very widespread and don't actually take advantage of the spatial resolution of fMRI.
Bottomline: you want to obtain small but reliable clusters, so use more astringent thresholds.
I hope that helped!
All the best,
Macià
Threaded View
| Title | Author | Date |
|---|---|---|
| Johnson H | Oct 17, 2011 | |
| Jake Flinthoff | Aug 15, 2018 | |
| Alfonso Nieto-Castanon | Oct 17, 2011 | |
| Johnson H | Oct 17, 2011 | |
| Alfonso Nieto-Castanon | Oct 17, 2011 | |
| Johnson H | Oct 17, 2011 | |
| Macià Buades-Rotger | Dec 9, 2015 | |
| Alfonso Nieto-Castanon | Dec 11, 2015 | |