help
help > RE: Differences in group size--compare within-group variance?
Mar 22, 2019 12:03 AM | Alfonso Nieto-Castanon - Boston University
RE: Differences in group size--compare within-group variance?
Hi Jeffrey,
This is an interesting question, and the reviewer is correct to point out that evaluating the difference in the number of significant/supra-threshold connections between two different analyses does not allow you to properly infer whether the differences between the two pairs of groups involved are similar or not. In addition, the expectation of decreased power in this sort of unbalanced designs is somewhat independent of the heteroscedasticity assumption of the GLM model or the homogeneity of variances assumption in a two-sample t-test, It is simply due to the smaller-group's average connectivity estimate having larger unknowns / standard error due to its smaller sample-size (this difference in standard errors is expected even if the within-group standard deviations are exactly the same across the two groups). Because of this I believe that a more direct way to evaluate whether in your results the difference in the number of supra-threshold connections between the two analyses is due to differences in power/sensitivity vs. differences in effect-size would be to actually compare the effect-sizes of the between-group differences/comparisons in both analyses.
There are, of course, a lot of different ways to go about exploring these effect-sizes. Since the two between-group comparisons in your case involve three groups (Group1vsGroup4, and Group3vs.Group4) one relatively straightforward approach would be to perform a conjunction of a [-1 1 0] contrast and a [-1 -1 2] contrast (when selecting Group1, Group3, and Group4 in this order). The first between-subjects contrast identifies those connections where Group1 and Group3 have different connectivity, and the second contrast identifies those connections where the difference between Group1 and Group4 is different from the difference between Group3 and Group4). Those connections that appear as significant (using two-sided tests) in both results will be the ones where you can confidently say that the strength of the Group1vsGroup4 difference in connectivity differs from the strength of the Group3vsGroup4 difference in connectivity. Then simply display those between-group differences in connectivity across these same connections in order to evaluate whether in fact the Group3vsGroup4 differences appear to be larger/stronger than the Group1vsGroup4 differences, which would support your original observation regarding the number of suprathreshold connections in each individual analysis but without the potential biases due to difference in power/sensitivity between those individual analyses.
Hope this helps
Alfonso
Originally posted by Jeffrey Johnson:
This is an interesting question, and the reviewer is correct to point out that evaluating the difference in the number of significant/supra-threshold connections between two different analyses does not allow you to properly infer whether the differences between the two pairs of groups involved are similar or not. In addition, the expectation of decreased power in this sort of unbalanced designs is somewhat independent of the heteroscedasticity assumption of the GLM model or the homogeneity of variances assumption in a two-sample t-test, It is simply due to the smaller-group's average connectivity estimate having larger unknowns / standard error due to its smaller sample-size (this difference in standard errors is expected even if the within-group standard deviations are exactly the same across the two groups). Because of this I believe that a more direct way to evaluate whether in your results the difference in the number of supra-threshold connections between the two analyses is due to differences in power/sensitivity vs. differences in effect-size would be to actually compare the effect-sizes of the between-group differences/comparisons in both analyses.
There are, of course, a lot of different ways to go about exploring these effect-sizes. Since the two between-group comparisons in your case involve three groups (Group1vsGroup4, and Group3vs.Group4) one relatively straightforward approach would be to perform a conjunction of a [-1 1 0] contrast and a [-1 -1 2] contrast (when selecting Group1, Group3, and Group4 in this order). The first between-subjects contrast identifies those connections where Group1 and Group3 have different connectivity, and the second contrast identifies those connections where the difference between Group1 and Group4 is different from the difference between Group3 and Group4). Those connections that appear as significant (using two-sided tests) in both results will be the ones where you can confidently say that the strength of the Group1vsGroup4 difference in connectivity differs from the strength of the Group3vsGroup4 difference in connectivity. Then simply display those between-group differences in connectivity across these same connections in order to evaluate whether in fact the Group3vsGroup4 differences appear to be larger/stronger than the Group1vsGroup4 differences, which would support your original observation regarding the number of suprathreshold connections in each individual analysis but without the potential biases due to difference in power/sensitivity between those individual analyses.
Hope this helps
Alfonso
Originally posted by Jeffrey Johnson:
Hello Alfonso and others,
I've run a few second-level ROI-to-ROI analyses (mostly 2-sample t-tests) involving comparisons between different combinations of four participant groups with different sizes (n = 9, 10, 17, and 17). A reviewer has asked us to address the possibility that some results might be due to differences in sample size, rather than (or in addition to) true differences in connectivity.
For example, when we compare groups 1 and 4 (n=9 vs. n=17), there are about 15 significantly different connections, but when we compare groups 3 and 4 (n=17 vs. n=17), there are about 35 significant differences. We would like to gain some insight into whether there are fewer differences between groups 1 and 4 than 3 and 4 because there is less power in the first analysis due to group 1 (n=9) being smaller than group 3 (n=17). I was thinking it might be helpful to compare the variance in the correlation coefficients for each group (since those are the data in each of the t-tests in my analyses), as this would at least let me see if we're in violation of the assumption of homogeneity of variance, but my network has more than 700 connections so it's not really feasible to try to compute those values manually. Is there some way to efficiently obtain the variances (maybe a distribution of variances across each group's network) from the CONN results just to get an idea of how things compare between groups? Or do you have any suggestions for a better/alternative way to account for sample size differences?
Any suggestions or guidance would be very much appreciated.
Thank you,
Jeff
I've run a few second-level ROI-to-ROI analyses (mostly 2-sample t-tests) involving comparisons between different combinations of four participant groups with different sizes (n = 9, 10, 17, and 17). A reviewer has asked us to address the possibility that some results might be due to differences in sample size, rather than (or in addition to) true differences in connectivity.
For example, when we compare groups 1 and 4 (n=9 vs. n=17), there are about 15 significantly different connections, but when we compare groups 3 and 4 (n=17 vs. n=17), there are about 35 significant differences. We would like to gain some insight into whether there are fewer differences between groups 1 and 4 than 3 and 4 because there is less power in the first analysis due to group 1 (n=9) being smaller than group 3 (n=17). I was thinking it might be helpful to compare the variance in the correlation coefficients for each group (since those are the data in each of the t-tests in my analyses), as this would at least let me see if we're in violation of the assumption of homogeneity of variance, but my network has more than 700 connections so it's not really feasible to try to compute those values manually. Is there some way to efficiently obtain the variances (maybe a distribution of variances across each group's network) from the CONN results just to get an idea of how things compare between groups? Or do you have any suggestions for a better/alternative way to account for sample size differences?
Any suggestions or guidance would be very much appreciated.
Thank you,
Jeff
Threaded View
| Title | Author | Date |
|---|---|---|
| Jeffrey Johnson | Mar 20, 2019 | |
| Alfonso Nieto-Castanon | Mar 22, 2019 | |
| Jeffrey Johnson | Mar 22, 2019 | |