Hi there,

I performed seed to voxel fc with bivariate correlation and produced second level maps per condition per seed using a within-subjects design.

when I open the saved .nii masks in mricron, or when I plot beta values in SPM, the atan(r) values are SUPER low, they range from 0.05 to 0.3. Most papers with fc threshold their z-maps above say z=3, but this is impossible for me. 

I wanted to know if I should be worried, since the t-values of significant clusters have no bearing on the strength of the correlations themselves.

Any help appreciated. Thanks!

crystal

Hi Crystal,

I would not be worried. The beta values in these analyses (Fisher transformed correlation coefficient values, often called Z-values) are not the same as the normal distribution standard scores values (often called z-values), the former are measures of effect size that directly relate to the correlation coefficient units, while the latter are measures of statistical strength that directly relate to the T- values of your results. If you want to transform z- values to equivalent T-statistics, simply use the transformation:

   T=tinv(normcdf(z),dof);

(for example, for a simple t-test with 20 subjects, where dof=19, a z-value of 3 corresponds approximately to a T-value of 3.45).

If you want to transform your Fisher transformed Z-values to correlation coefficient values simply use the transformation:

  r=tanh(Z);

(for example, a Z-value of 3 would correspond to a very unrealistic correlation coefficient of 0.995; Z-values in the .05 to .30 range look perfectly fine to me -they roughly correspond to r-values in the .05 to .29 range)

Hope this clarifies

Best
Alfonso


Originally posted by Crystal Goh:

Hi, 

I'd like to follow up on this post after searching on a similar head-scratcher. I see the difference between the two Z/z values above, thanks. However, I'm unsure how r-values of 0.05-0.29 are considered correlated; these values seems low.

In our groups work, we're comparing two conditions. There is a strong effect size and near significant p (p=0.0527) when I compare connectivity values (i.e. fisher transformed Z-values) between conditions but the individual conditions don't seem very correlated with the task (they have mean fisher values of -0.05411 and 0.116). 

Other information:
I performed a PPI (regression (bivariate)) analysis with 14 subjects. 

Michael

Hi Michael,

This is a very interesting and complex question. First, in your particular example you are using PPI analyses using regression (bivariate) measures, so the actual values that are entered into your second-level analyses would be regression coefficients (instead of Fisher-transformed correlation coefficients) associated with the interaction between the "psychological effect" (your task) and the "physiological effect" (the source ROI), and the interpretation of these effect-size measures is not straightforward. In particular effect sizes in PPI analyses do not represent absolute measures of connectivity during your task conditions but rather relative measures of connectivity-changes associated with the presence of your task. If using a block design, and PSC analysis units, for example, one possible way to characterize the units of these regression coefficients would be something along the lines of how much the ratio "percent-signal-change in target ROI/voxel for each unit percent-signal-change in source ROI/seed" changes in the presence of your task (compared to a common baseline condition that includes the entire acquisition), so the ~.17 difference between the -.054 and .116 effect sizes represents a difference of .17 between conditions in the ratio above (not a difference between conditions in Fisher-transformed correlation values).

Last, regarding what constitutes "high" or "low" correlation values, in practice for ROI-to-ROI analyses average resting-state correlation values between two ROIs have a distribution like the attached figure, where most (~90%) of the significant ROI-to-ROI connections show absolute correlations below .30, and many (~50%) show correlations below .10 (again only considering the significant ROI-to-ROI connections; e.g. P-fdr<.05). Of course, this touches on the question of the difference between statistical significance and practical significance, but that is probably the topic for a longer discussion (see for example Friston 2014 "Sample size and the fallacies of classical inference").

Best
Alfonso 
Originally posted by Michael King:

Hi Alfonso, 

Thank you for the clear answer(s), as usual. And for the direction to Friston's paper. 

Best,
Michael

Hi Alfonso, 

I'm interested in learning more about the ROI-ROI curve you provided in the last message. I have searched online but only found related topics. Can you suggest any literature?

Best,
Michael

Hello,

Is it statistically appropriate to extract mean first eigenvariates across a significant between-groups cluster, for each subject, from a 2nd level conn factorial model, and then calculate cohen's d based on the group mean of those values?

Best,
Patrick

Hi Patrick,

It is relatively standard to perform these form of post-hoc analyses in order to better interpret, for example, the actual connectivity strengths in each group after a between-group comparison. Beyond this (aiding in the interpretation of your results) there is not much that you should be doing with those values since, due to "double-dipping", the actual effect sizes extracted will typically be inflated, and any statistics derived from these values will be invalid (liberal p-values). This inflation will always occur as long as the contrast used in your original analyses (the one defining the clusters of interest) and the contrast that you want to look at in your post-hoc analyses are not orthogonal. The amount of inflation depends on multiple factors, but most importantly on the power of your original analyses (underpowered designs tend to produce significant inflation of effect sizes, while the amount of inflation in well-powered designs is considerably smaller). If your intention is to obtain Cohen's d values from a pilot study to be used for power analyses (to help you design future studies) I would recommend using a cross-validation approach in order to better estimate the expected strength of the between-group differences in the population. 

Hope this helps
Alfonso
Originally posted by Patrick McConnell:

Thanks, Alfonso!

The approach I took was initially a hypothesis driven bivariate correlation (between-groups), seed-voxel analysis with a a R & L seed.  I thresholded results in SPM at p<.001 and p<.05 cluster FWE and made functional ROIs from significant results and fed those seeds back into the conn model to explore potential connectivity paths.  I ended up with a path from roi1 to roi2 and roi3, and from roi2 to roi4.  I extracted single subject eigenvariates across each cluster to determine the pattern of correlation in each group (e.g., +/-, +/+, -/-) and to get an indication of effect size.  

To explore potential effective connectivity between these functionally defined regions, I ran a bivariate regression (within-group), ROI-ROI analysis using those paths shown to be significant, finding all of them to be bidirectionally significant in one group but not the other (p <.001, p< .05 FDR), although the magnitude of t-stat varied by direction.  To explore this further, I went in and extracted regression coefficients for each subject (only where significant bidirectional effects were observed) and ran paired-samples t-tests to see if the magnitude of t-stat was significantly larger for one direction (e.g., roi1-->roi2 vs. roi2-roi1) than the other.  

So,

1) Is my approach statistically valid, or "double-dipping"?
2) Is it appropriate to infer directionality of effective connectivity based on the above approach?

Thanks!!!

-Patrick

Hi Patrick,

I am not sure I am understanding correctly, could you please clarify the following?

a) when you say "fed those seeds back into the conn model to explore potential connectivity paths", I am assuming you entered the mask of significant clusters from your original seed-to-voxel analyses (looking at between-group differences in connectivity with a R&L seed) as new ROIs into CONN, and then performed ROI-to-ROI first-level analyses using bivariate correlations and only these new ROIs as sources. In your second-level analyses you then looked at average (across both groups) ROI-to-ROI effects to determine significant connections between these ROIs (roi1-roi2, roi1-roi3, and roi2-roi4). Am I interpreting correctly?

b) not sure what you mean by "extracted single subject eigenvariates across each cluster to determine pattern of correlation in each group", are you referring to the step above (but now using single-group contrasts in your second-level analyses) or are you talking about a different set of analyses?

c) the second paragraph is clearer I believe, just rephrasing to make sure I am interpreting correctly: you then run a new set of first-level ROI-to-ROI analyses using your roi1 to roi4 ROIs as sources, and now using bivariate regression measures, and then looked at second-level within-group effects, limiting your results to only those ROI-to-ROI connections of interest (those where you found in the step (a) above significant across-group connectivity effects; i.e. roi1-roi2, roi1-roi3, etc.). In addition, for the subset of connections where you found bi-directional significant effects in this newer (bivariate-regression) set of analyses you then extracted the connectivity strengths for each subject and performed paired t-tests to explore directionality effects (e.g. higher roi1-roi2 vs. roi2-roi1 effects). Let me know if I am misinterpreting/misrepresenting anything here.

Thanks!
Alfonso
 
Originally posted by Patrick McConnell:

Alfonso,

Thank you very much for the thoughtful and detailed reply.  I'm sorry that I wasn't clearer in my original post.  From c) your understanding/interpretation is spot-on.  Just to rephrase again:

1) seed --> voxel bivariate regression with 5-mm spheres and a priori non-directional hypotheses. 
2) marsbar --> write functional clusters from SPM.mat in SPM; extract eigenvariates from these clusters to explore directionality of group mean correlation (positive v inverse / strength).
3) enter new rois back into seed--> voxel bivariate correlation analysis; repeat step 2) for any significant target rois.  
4) repeat step 3) one more time.
5) calculate cohen's d for each group using group mean eigenvariate for each significant roi.
6) bivariate regression ROI-ROI analysis within-groups using previously determined ROIs.
7) where both directions (e.g., roi1 --> roi2, roi2 --> roi1) were significant, extract single-subject regression coefficients and run paired-sample t-tests on roi1 --> roi2 vs. roi --> roi2 to determine if there is any indication of effective connectivity.  
8) we also extracted denoised time series for each roi to explore visually.

Thanks again for the help!!!

-Patrick

Hi Patrick,

Thanks for the clarification. The only thing I am not sure yet I am interpreting correctly is whether in step (3) you looked at the main effect across both subject groups (i.e. select both groups and enter a [.5 .5] contrast) or at the between-group differences (i.e. select both groups and enter a [-1 1] contrast) when defining the significant target ROIs. I am assuming you probably meant the latter.

In that case the Cohen's d values computed in step (5) will show some amount of bias due to the original selection of ROIs based on a non-orthogonal contrast (the ROIs created in each iteration of step (3) are based on the same between-subject contrast [-1 1] as that effectively looked at when computing the Cohen's d between-group difference measures). The bias will be smaller but still there if only the first iteration of step (3) was based on the same [-1 1] contrast while the second iteration of step (3) used the orthogonal contrast [.5 .5] instead, and there would be no bias if both of the iterations of step (3) were based on the orthogonal contrast [.5 .5] instead. Note also that these biases will translate to invalid statistics for your analyses in step (5) and (6), but not in the directionality tests in step (7) since the amount of inflation is not expected to have any preferred directionality. So overall I would say that (up to the interpretation of which second-level between-subjects contrast you used to define the different ROIs) you are likely incurring in some amount of "double-dipping". Of course this does not mean that your analyses are incorrect, I do believe the analyses are perfectly fine as a way to further explore your found effects, you should just phrase them as post-hoc exploratory analyses instead of confirmatory hypothesis testing and let the readers/reviewers know of the associated limitations (e.g. require independent replication with a-priori/independent ROI selection procedure)

Let me know if you would like me to further clarify any of this
Best
Alfonso

Originally posted by Patrick McConnell:

Alfonso,

Thanks again for the thoughtful reply.  

For the initial seed-voxel bivariate correlations (step 1) and follow-up seed-voxel bivariate correlations (step 3/4), I generated the SPM.mat file using an F-test [1 0; 0 1] but then used t-contrasts to find the significant correlations (1, -1 and -1,1).  

In step 6, results (i.e., single subject regression coefficients) were used only to perform a within-group dependent samples t-test to determine whether there were statistically significant differences in magnitude of regression coefficient for each path (e.g., roi1 --> roi2 vs. roi2 --> roi1).  Denoised time-series were only used for visualization.   I understand why the Cohen's D will be biased for the follow-up seed-voxel bivariate correlations and the ROI-ROI bivariate regressions, but is there any issue of bias with the Cohen's D values calculated from the initial seed-voxel analyses (e.g., roi1a --> roi2a and roi1b --> roi2b)?  

We are basically seeing two separate paths, one of which seems to be strengthened in Tx group (roi1a path), and another that seems to be weakened in Tx group (roi1b path). Would it be valid to determine power/sample size for future studies based on these Cohen's D values?

Thanks again for sticking with me through this and being so generous with your replies.

All the best,
Patrick

Sorry for the confusion, but I just noticed an error:

should read: 

1) seed --> voxel bivariate correlation with 5-mm spheres and a priori non-directional hypotheses.

Hi Patrick,

Some thoughts on your questions below

Best
Alfonso
Originally posted by Patrick McConnell:
I am assuming here that roi1a and roi1b represent your a priori seeds and roi2a and roi2b represent the clusters that show significant between-group differences in connectivity with each of these seeds respectively (step 1). If this is correct then, yes, unfortunately the average connectivity between roi1a and roi2a will be biased towards showing higher between-group connectivity differences and higher Cohen's D values for this particular sample of subjects than what you would expect to obtain using the same masks in the population (because the roi2a mask has been defined from this same group of subjects in a non-orthogonal way). 

For the reasons above those sample size estimates are likely going to be conservative (too few subjects). For these cases I typically use leave-one-out cross-validation to obtain unbiased measures of your between-group differences (basically you estimate roi2a using the same between-group test but applied only to N-1 subjects, and then extract the average connectivity from the resulting roi2a mask from the left-out subject alone, and repeat this process for each subject). If you want to give it a try the attached function (spm_crossvalidation) will take the current second-level analysis being displayed in SPM and apply this cross-validation procedure to obtain unbiased measures of your effect sizes.

Hope this helps
Alfonso

Fantastic!  Very clear explanation. Thanks so much!

Best,
-Patrick