Dear Andrew,

Thank you for a wonderful tool!

I'm interested in the following text from the NBS reference manual:

"A common heuristic is to threshold the set of connectivity matrices at the group level and exclude connections that do not exceed the given threshold. For example, a preliminary one-sample t-test can be applied to each connection to assess whether the mean value of connectivity strength differs from zero. Connections that do not differ from zero are then excluded."

Do you recommend such an approach? Should we not be concerned that selecting connections based on group 'extremes' will lead to an inevitable conclusion that the other group will be less extreme, and that therefore we are biasing our statistical analysis when we use NBS. I am thinking of it as something like regression toward the mean, where we inevitably are generating group differences by selecting connections that are defined by extremes in any one group. Somehow it seems like between group differences are not independent from single group differences from 0.

Can you explain how I might be thinking about this incorrectly?

Do you have citations for papers that you think have implemented such a thresholding approach in a sound manner?

Thanks!

Hi,

We are not suggesting to limit hypothesis testing to "group extremes".

We are rather suggesting that it is of little value to perform hypothesis testing between a pair of regions for which our measure of connectivity is identically zero for all individuals in the sample.

For example, structural connectivity matrices are typically sparse, and thus the number of multiple comparisons can be reduced by omitting connections that are identically zero across all individuals (or not significantly different from zero). We know a priori that the null cannot be rejected for such connections.

I see no bias in doing this.

Andrew

Originally posted by :

Hi Andrew,

I appreciate your very quick response.

I believe I follow, but I want to be certain that I do:

So, say we are interested in identifying group-level abnormalities associated with a diseased group as compared to a healthy group. It would be incorrect to include only connections that are different from zero in both groups *or* one group, and correct to include only connections that are different from zero across all individuals (diseased and healthy), right?

I've seen published papers in my subfield that have included connections based on only *one* group's values differing from zero, which it sounds like is incorrect, right?

Do you have a citation for a paper that properly does this in an NBS study of effects associated with a particular group? It would be helpful to look over a paper that has done it correctly.

Thank you!

Hi,

I certainly don't think that the "one group" option is necessarily incorrect.

Which option is most appropriate depends on the hypothesis you are testing.

For example, if the alternative hypothesis is a reduction in connectivity in group X relative to group Y, then thresholding based on a one-sample t-test in group Y alone (the controls) might be justifiable. Thresholding based on groups X and Y combined can potentially exclude too many connections from subsequent testing, resulting in false negatives.

Andrew

Originally posted by :

Hi Andrew,

I appreciate help.

So it seems that it is sound to include only connections based on connectivity differences from zero using individuals from only one group, both groups combined, or connections that are different from zero in either group (i.e., one connection may be different from zero in one group, another in the other group, but both would be included in the analysis using NBS). These choices will have impact on potential false negatives, but they are theoretically sound.

Thank you!

Hi Andrew,

I am possibly missing something about the general context of this question but I believe that in general one requires the two contrasts to be orthogonal in order to have valid results. In your example, the a priori contrast (Y>0, or [0 1]) is not orthogonal to the main contrast (Y-X>0, or [-1 1]) so that is going to result in inflated false positives in your main contrast (i.e. the null hypothesis distribution of "Y-X" values is going to be biased towards positive values when selecting only those connections that survive any "Y > threshold" value). So in this example I imagine that strictly speaking only an a priori contrast [1 1] (looking at average effect across the two groups) would be valid when then testing for between-group differences. That said, in practice this "double-dipping" false positive inflation effect may be really small, particularly given that typically you want to choose a liberal a priori threshold in order not to incurr in excessive false negatives, and in some other cases the false positives may even be reduced instead -e.g. if your a priori contrast selects X>0 instead of Y>0 connections in the example above-, so this might be what you are referring to. On the other hand perhaps all of this discussion is framed in the context of independent samples for the a priori and main contrast tests (and in that case all of the above is irrelevant), or perhaps the NBS measures in general or the permutation tests in particular are somehow more robust to this sort of departures from the a priori assumptions and I have not really thought this through, so please feel free to correct me if I am missing something obvious. 

Best
Alfonso 
Originally posted by Andrew Zalesky:

Hi Alfonso,

Thanks for your feedback! Consider a connection with the following edge weights:

patients=[-0.9,-0.7,-0.8];

controls=[0.6,0.5,0.8];

Performing an initial one-sample t-test on the *pooled* set of patients and controls will mean that this connection is excluded from statistical inference. But a between-group difference is clearly evident in this case and hence we increase the false negative rate. 

In terms of orthogonality, I don't really see this as two nested tests (although I definitely understand your point). For example, the first test (one-sample) can be performed on an independent set of controls to which the second test (two-sample) is performed.

Andrew

Originally posted by Alfonso Nieto-Castanon:

Thank you Andrew and Alfonso.

Andrew, in your most recent comment you've stated: "In terms of orthogonality, I don't really see this as two nested tests (although I definitely understand your point). For example, the first test (one-sample) can be performed on an independent set of controls to which the second test (two-sample) is performed."

I appreciate that if you have conducted a one-sample test on an independent set of controls that the issue of nested tests is not a problem for the group differences test. But what about if you do as I suggested in my previous comments, where we would include connections that differed in either the diseased group and/or the control group, and then conduct the two-sample test on these connections? This seems to be the test sets that Alfonso is concerned about ("Y>0, or [0 1]) is not orthogonal to the main contrast (Y-X>0, or [-1 1]") If I'm understanding correctly. Is it sound to threshold the network based on group differences in either group (i.e., connection ~= 0 in diseased or controls), and then conduct the group-level test on only these connections?

Thank you!

Hi,

I think that taking the conjunction of two independent a priori tests might bring us closer to violating the independence issue that Alfonso pointed out.

If you are losing sleep over this, you might want to consider simply avoiding any prior thresholding. Prior thresholding can be useful but it is definitely not essential. Everything works fine without it, but the number of comparisons might be unnecessarily large. The idea of the prior test is to exclude from the inference procedure any edges for which you believe a priori cannot show an effect.

Andrew

Originally posted by :

>
> Hi Andrew,
> I have a technical question about how to actually set the matrix up after
> thresholding. I have thresholded my data based on the connections that are
> significantly >0 in the control group and am uncertain how to remove the
> non-significant data points. Currently they remain in the matrix as a 0, so
> I would think that the number of comparisons still remains the same.
>
> Any advice would be greatly appreciated.
> Thanks!
>
> Corinna
>

Hi Corinna

If a connection is 0 for all subjects, then the NBS will automatically ignore that connection during statistical testing.

In other words, if the connectivity value for a given connection is zero in all connectivity matrices, that specific connection is ignored by default and the number of multiple comparisons reduced accordingly.

Andrew

Originally posted by Corinna Bauer: