Hi Andrew,

I am using NBS toolbox to examine between group connection differences. First, thank you for developing such great toolbox!

I have a couple of questions here:

1. If I understand correctly, the main purpose of NBS is to find a significant "component" or sub-network between groups, but what if i also want to look at whether there is increased or discreased connections in the two groups but maybe those connections are not in one sub-network?

2. Just followed the first question, if I conducted unpaired ttest on every connection and applied with multiple correction (Benjamini–Hochberg or more strict like Bonferroni), I was wondering are there any "statistically power" issues when I do something like that?

Should I also do permutation test like NBS? In my view, I just want to see whether there is differences in each connection, rather than extracting a significant component, so probally I don't need to do permutation test?; I am not sure my understanding is right or not.

3.  I read papers that mentioned we should also do thresholding analyses on raw network before we conduct the network analyses because of possible spurious connections. I was wondering what is your opinion about that? 

Thank you very much!

Have a great one!
Ting

Hi Ting, 

thanks for your interest. 

1. The NBS is not well suited to cases where the effect of interest is constrained to a single isolated connection. FDR may be a better option in such a case. NBS can detect single connections (component size = 1), but it is not really designed for this case. 

2. Yes. If your sample size is small and you are considering many connections, power issues will inevitably be a concern. If no connections are found, it may simply be that your sample size is too small to detect the effect of interest. 

3. Yes - thresholding can be helpful in many cases and many pros and cons need to be considered.

Andrew

Originally posted by Ting Qiu:

Hi Andrew,

Thank you so much for your e-mail!

Followed the second question above, I know when we perform two-sample ttest on every connection of the network, we believe that the data is normally distributed for each stat; if we not sure the data is not normally distributed, we could do non parametric test, like two sample permutation test.

Now I am just thinking: if the data is normally distributed, I performed ttest on every connection of the network for the two groups; aftter FDR correction, I got some significant results. But now I also want to see if these  results are really "significant" rather than randomly significant,so what I want to do is:

step 1: I first use the real group data to do ttest for each single connection;let's say p < 0.05 for this test.

step 2: for the permutation part, I first disorganize the group and reorder the group, then perform ttest and get p-value;

step 3: iterate step 2 5000 times;

step 4: get the probability of p < 0.05 for the 5000 permutation, P = num(p<0.05)/5000; if P < 0.05, then I would say the result (for the real data) is significant.

I was wondering whether it is necessary to do "permutation test" like that or whether it is rational to do like that?

Again, thank you very much for your useful suggestions!

Have a nice one!
Ting

Hi Ting, 

For the FDR option in NBS, it is important that that number of permutations is very high (i.e. 500,000 or more permutations). 

The steps that you have described appear to be consistent with a permutation test, although the null distribution would usually be computed for the t-statistic rather than the p-value. This can be helpful if the data is not normally distributed. 

I am not sure what you mean by "randomly significant" vs "really significant". 

Andrew

Originally posted by Ting Qiu:

Hi again!

I really appreciate for your kind help Andrew!

I am sorry to have you confused. 

If the data is not normally distributed, then we had better to use two sample permutation test, my confusion is:  should we also conduct ttest when we do the multiple test each time? but if so, we will think the data we compare is normally distributed when perform ttest?

Have a nice one!
Ting

Hi Ting, 

In a permutation test, the t-test is only used as measure of variation/difference. We don't really rely on the parametric assumptions of the t-test when using it as a measure of variation in a permutation test, since we never compute a p-value from the t-distribution.

Permutations tests are better at handling non normally distributed data than parametric alternatives like the t-test. 

Andrew

Originally posted by Ting Qiu:

Hi Andrew,

Thank you very much for your reply!

That make sense! I really appreciate for your help!

All the best,
Ting