help
help > RE: Regarding group-level matrix thresholding
Aug 9, 2015 02:08 AM | Andrew Zalesky
RE: Regarding group-level matrix thresholding
Hi Alfonso,
Thanks for your feedback! Consider a connection with the following edge weights:
patients=[-0.9,-0.7,-0.8];
controls=[0.6,0.5,0.8];
Performing an initial one-sample t-test on the *pooled* set of patients and controls will mean that this connection is excluded from statistical inference. But a between-group difference is clearly evident in this case and hence we increase the false negative rate.
In terms of orthogonality, I don't really see this as two nested tests (although I definitely understand your point). For example, the first test (one-sample) can be performed on an independent set of controls to which the second test (two-sample) is performed.
Andrew
Originally posted by Alfonso Nieto-Castanon:
Thanks for your feedback! Consider a connection with the following edge weights:
patients=[-0.9,-0.7,-0.8];
controls=[0.6,0.5,0.8];
Performing an initial one-sample t-test on the *pooled* set of patients and controls will mean that this connection is excluded from statistical inference. But a between-group difference is clearly evident in this case and hence we increase the false negative rate.
In terms of orthogonality, I don't really see this as two nested tests (although I definitely understand your point). For example, the first test (one-sample) can be performed on an independent set of controls to which the second test (two-sample) is performed.
Andrew
Originally posted by Alfonso Nieto-Castanon:
Hi
Andrew,
I am possibly missing something about the general context of this question but I believe that in general one requires the two contrasts to be orthogonal in order to have valid results. In your example, the a priori contrast (Y>0, or [0 1]) is not orthogonal to the main contrast (Y-X>0, or [-1 1]) so that is going to result in inflated false positives in your main contrast (i.e. the null hypothesis distribution of "Y-X" values is going to be biased towards positive values when selecting only those connections that survive any "Y > threshold" value). So in this example I imagine that strictly speaking only an a priori contrast [1 1] (looking at average effect across the two groups) would be valid when then testing for between-group differences. That said, in practice this "double-dipping" false positive inflation effect may be really small, particularly given that typically you want to choose a liberal a priori threshold in order not to incurr in excessive false negatives, and in some other cases the false positives may even be reduced instead -e.g. if your a priori contrast selects X>0 instead of Y>0 connections in the example above-, so this might be what you are referring to. On the other hand perhaps all of this discussion is framed in the context of independent samples for the a priori and main contrast tests (and in that case all of the above is irrelevant), or perhaps the NBS measures in general or the permutation tests in particular are somehow more robust to this sort of departures from the a priori assumptions and I have not really thought this through, so please feel free to correct me if I am missing something obvious.
Best
Alfonso
Originally posted by Andrew Zalesky:
I am possibly missing something about the general context of this question but I believe that in general one requires the two contrasts to be orthogonal in order to have valid results. In your example, the a priori contrast (Y>0, or [0 1]) is not orthogonal to the main contrast (Y-X>0, or [-1 1]) so that is going to result in inflated false positives in your main contrast (i.e. the null hypothesis distribution of "Y-X" values is going to be biased towards positive values when selecting only those connections that survive any "Y > threshold" value). So in this example I imagine that strictly speaking only an a priori contrast [1 1] (looking at average effect across the two groups) would be valid when then testing for between-group differences. That said, in practice this "double-dipping" false positive inflation effect may be really small, particularly given that typically you want to choose a liberal a priori threshold in order not to incurr in excessive false negatives, and in some other cases the false positives may even be reduced instead -e.g. if your a priori contrast selects X>0 instead of Y>0 connections in the example above-, so this might be what you are referring to. On the other hand perhaps all of this discussion is framed in the context of independent samples for the a priori and main contrast tests (and in that case all of the above is irrelevant), or perhaps the NBS measures in general or the permutation tests in particular are somehow more robust to this sort of departures from the a priori assumptions and I have not really thought this through, so please feel free to correct me if I am missing something obvious.
Best
Alfonso
Originally posted by Andrew Zalesky:
Hi,
I certainly don't think that the "one group" option is necessarily incorrect.
Which option is most appropriate depends on the hypothesis you are testing.
For example, if the alternative hypothesis is a reduction in connectivity in group X relative to group Y, then thresholding based on a one-sample t-test in group Y alone (the controls) might be justifiable. Thresholding based on groups X and Y combined can potentially exclude too many connections from subsequent testing, resulting in false negatives.
Andrew
Originally posted by :
I certainly don't think that the "one group" option is necessarily incorrect.
Which option is most appropriate depends on the hypothesis you are testing.
For example, if the alternative hypothesis is a reduction in connectivity in group X relative to group Y, then thresholding based on a one-sample t-test in group Y alone (the controls) might be justifiable. Thresholding based on groups X and Y combined can potentially exclude too many connections from subsequent testing, resulting in false negatives.
Andrew
Originally posted by :
Hi Andrew,
I appreciate your very quick response.
I believe I follow, but I want to be certain that I do:
So, say we are interested in identifying group-level abnormalities associated with a diseased group as compared to a healthy group. It would be incorrect to include only connections that are different from zero in both groups *or* one group, and correct to include only connections that are different from zero across all individuals (diseased and healthy), right?
I've seen published papers in my subfield that have included connections based on only *one* group's values differing from zero, which it sounds like is incorrect, right?
Do you have a citation for a paper that properly does this in an NBS study of effects associated with a particular group? It would be helpful to look over a paper that has done it correctly.
Thank you!
I appreciate your very quick response.
I believe I follow, but I want to be certain that I do:
So, say we are interested in identifying group-level abnormalities associated with a diseased group as compared to a healthy group. It would be incorrect to include only connections that are different from zero in both groups *or* one group, and correct to include only connections that are different from zero across all individuals (diseased and healthy), right?
I've seen published papers in my subfield that have included connections based on only *one* group's values differing from zero, which it sounds like is incorrect, right?
Do you have a citation for a paper that properly does this in an NBS study of effects associated with a particular group? It would be helpful to look over a paper that has done it correctly.
Thank you!
Threaded View
| Title | Author | Date |
|---|---|---|
| Aug 6, 2015 | ||
| Andrew Zalesky | Aug 6, 2015 | |
| Aug 6, 2015 | ||
| Andrew Zalesky | Aug 6, 2015 | |
| Alfonso Nieto-Castanon | Aug 6, 2015 | |
| Andrew Zalesky | Aug 9, 2015 | |
| Aug 10, 2015 | ||
| Andrew Zalesky | Aug 10, 2015 | |
| Corinna Bauer | Mar 23, 2016 | |
| Andrew Zalesky | Mar 24, 2016 | |
| Aug 6, 2015 | ||
