Hello Alfonso and others,
I've run a few second-level ROI-to-ROI analyses (mostly 2-sample t-tests) involving comparisons between different combinations of four participant groups with different sizes (n = 9, 10, 17, and 17). A reviewer has asked us to address the possibility that some results might be due to differences in sample size, rather than (or in addition to) true differences in connectivity.

For example, when we compare groups 1 and 4 (n=9 vs. n=17), there are about 15 significantly different connections, but when we compare groups 3 and 4  (n=17 vs. n=17), there are about 35 significant differences. We would like to gain some insight into whether there are fewer differences between groups 1 and 4 than 3 and 4 because there is less power in the first analysis due to group 1 (n=9) being smaller than group 3 (n=17). I was thinking it might be helpful to compare the variance in the correlation coefficients for each group (since those are the data in each of the t-tests in my analyses), as this would at least let me see if we're in violation of the assumption of homogeneity of variance, but my network has more than 700 connections so it's not really feasible to try to compute those values manually. Is there some way to efficiently obtain the variances (maybe a distribution of variances across each group's network) from the CONN results just to get an idea of how things compare between groups? Or do you have any suggestions for a better/alternative way to account for sample size differences? 

Any suggestions or guidance would be very much appreciated.

Thank you,
Jeff

Hi Jeffrey,

This is an interesting question, and the reviewer is correct to point out that evaluating the difference in the number of significant/supra-threshold connections between two different analyses does not allow you to properly infer whether the differences between the two pairs of groups involved are similar or not. In addition, the expectation of decreased power in this sort of unbalanced designs is somewhat independent of the heteroscedasticity assumption of the GLM model or the homogeneity of variances assumption in a two-sample t-test, It is simply due to the smaller-group's average connectivity estimate having larger unknowns / standard error due to its smaller sample-size (this difference in standard errors is expected even if the within-group standard deviations are exactly the same across the two groups). Because of this I believe that a more direct way to evaluate whether in your results the difference in the number of supra-threshold connections between the two analyses is due to differences in power/sensitivity vs. differences in effect-size would be to actually compare the effect-sizes of the between-group differences/comparisons in both analyses. 

There are, of course, a lot of different ways to go about exploring these effect-sizes. Since the two between-group comparisons in your case involve three groups (Group1vsGroup4, and Group3vs.Group4) one relatively straightforward approach would be to perform a conjunction of a [-1 1 0] contrast and a [-1 -1 2] contrast (when selecting Group1, Group3, and Group4 in this order). The first between-subjects contrast identifies those connections where Group1 and Group3 have different connectivity, and the second contrast identifies those connections where the difference between Group1 and Group4 is different from the difference between Group3 and Group4). Those connections that appear as significant (using two-sided tests) in both results will be the ones where you can confidently say that the strength of the Group1vsGroup4 difference in connectivity differs from the strength of the Group3vsGroup4 difference in connectivity. Then simply display those between-group differences in connectivity across these same connections in order to evaluate whether in fact the Group3vsGroup4 differences appear to be larger/stronger than the Group1vsGroup4 differences, which would support your original observation regarding the number of suprathreshold connections in each individual analysis but without the potential biases due to difference in power/sensitivity between those individual analyses. 

Hope this helps
Alfonso


Originally posted by Jeffrey Johnson:

Hi Alfonso,
Thank you very much for your response. I have a few follow-up questions as I want to be sure I understand and execute everything correctly if I move forward with this method.

1) To run the conjunction analysis in CONN, would I simply select Groups 1, 3 and 4 (in that order) and enter [-1 1 0]; [-1 -1 2] in the between-subjects contrast? 

2) As you said, the first contrast of the conjunction analysis will identify connections that differ between Groups 1 and 3 (i.e., Group1 - Group3). I'm a little less clear on the second part ([-1 -1 2]) -- will it identify connections that differ between Group4 and the combination of Group1 & Group3 (i.e., Group4 - (Group1+Group3))?

3) If I'm on track with part 2, above, then the conjunction analysis should return those connections that are different between Groups1 and 3 AND that are also different between Group 4 and (Group1+Group3) -- is that correct? If so, as you said in your initial response, we would interpret those results as connections where "the strength of the Group1vsGroup4 difference in connectivity differs from the strength of the Group3vsGroup4 difference in connectivity."

4) In your response, you said to "display those between-group differences in connectivity across these same connections in order to evaluate whether in fact the Group3vsGroup4 differences appear to be larger/stronger than the Group1vsGroup4 differences" -- would I just do that by calling up the results in the Results Explorer, selecting all of my ROIs, and using an analysis-level FDR correction?

5) Conceptually, I'm not clear how the conjunction analysis overcomes the potential bias due to different sample sizes. Can you say clarify that at all?

6) Would it be reasonable to add an additional covariate to this analysis, such as average framewise displacement? And if so, I assume I would simply select it in the subject effects menu and then add it to both parts of the conjunction analysis, as in [-1 1 0 0]; [-1 -1 2 0]. Does that seem correct?

Sorry about all of the questions. I really appreciate your help!

Best,
Jeff