help > Leave one out
Showing 1-7 of 7 posts
Display:
Results per page:
May 18, 2016  07:05 AM | David White
Leave one out
Has anyone figured out how to run leave one out analysis in conn?
May 18, 2016  07:05 AM | Jeff Browndyke
RE: Leave one out
I would love to know about this too.  It would be a great addition to the voxel-wise approaches in CONN.
 
Jeff
May 18, 2016  08:05 AM | Thomas DeRamus
RE: Leave one out
As would I. I could see this maybe working by setting up a batch that performs the analyses separately for each of the subject sets-1 and then exporting the values that you're interested in, but I imagine doing it that way would eat up space really fast.

-Thomas
Jun 23, 2016  03:06 PM | Alfonso Nieto-Castanon - Boston University
RE: Leave one out
Hi David/Jeff/Thomas

Thanks for the feedback/suggestion, this is indeed part of the planned capabilities to be added to CONN but I do not have quite a target date for this yet. In the meantime, you may want to give the attached spm_crossvalidation utility a try. This basically works on any open SPM results window, so if you want to use this with CONN you would need to first use the 'Display SPM' button in the results explorer window. After that, you simply type spm_crossvalidation in Matlab's command window and that will run N analyses (each analysis removing one subject), and compute the suprathreshold mask for each of those analyses. If you specify where exactly you would want to extract contrast data from (e.g. averaging from all suprathreshold voxels) that will also extract from these masks, and from the corresponding out-of-sample subject in each case, contrast values within the suprathreshold masks for all defined contrasts.

Hope this helps
Alfonso

 
Originally posted by Thomas DeRamus:
As would I. I could see this maybe working by setting up a batch that performs the analyses separately for each of the subject sets-1 and then exporting the values that you're interested in, but I imagine doing it that way would eat up space really fast.

-Thomas
Aug 22, 2016  01:08 PM | Jennifer Barredo
RE: Leave one out
Hi Alfonso,

Thank you for posting the CONN version of your SPM cross-validation scripts.  The scripts work very nicely, but I have a few questions and could use some guidance on reporting the LOOCV output.

1) Are the values stored in the CV_DATA variable the GLM prediction errors after leaving out a given subject, or are they correlations between PE estimated using the full dataset vs. the LOOCV data set?  

2) Would running a t-test against zero on the CV_DATA be the correct way to validate a cluster extracted from a seed-to-voxel analysis?  

3) If a t-test were used and the cluster of interest was extracted from a between-groups contrast, a statistically significant t-test would be interpreted as positive evidence of validity.  If the cluster was extracted from a single group or condition however, a null rather than significant finding would be considered positive evidence.  Is this correct?


Thanks,
Jennifer
Dec 28, 2018  01:12 PM | Jeff Browndyke
RE: Leave one out
Kicking this thread to the top to get clarification on the spm_crossvalidation script output.

What does the CV_CONTRASTS variable field indicate?  I see that the STAT = T, but does the "contrast 'connectivity result' cross-validated value(s)" represent?

Likewise, what do the CV_DATA and CV_VOXELS denote?  

Per Jennifer's question, is a t-test against zero on the CV_DATA what is output as the CV_CONTRASTS?

Any guidance on interpretation of this script's output would be most appreciated.

Thanks and warm regards,
Jeff
Jan 10, 2019  02:01 PM | Jeff Browndyke
RE: Leave one out
Kicking to the top.

J