Hi,

First off thanks for the excellent toolbox. I am using the NBS to perform a paired ttest on a small number of subjects (n=4) in an animal experiment. If I am not mistaken, the small sample size should preclude any meaningful nonparametric testing for the FWE-corrected p-value of the largest component, since there are only 2^4=16 possible permutations.

First off, I am curious why I can exceed 16 total permutations when using the NBS - is resampling done with replacement? Maybe I missed it, but I don't see that specified in the original publication or in the manual.

More importantly, I am interested in the possibility of creating a parametric alternative to computing an FWE-corrected p-value. I am working with functional connectivity data, so all my edges are r-to-z transformed correlation values. How appropriate (or inappropriate) would it be to simply build a null distribution of connectivity matrices by simulating for each edge a normal distribution of correlation values based on the pooled sample mean and variance for that edge? Do I need to take additional steps to account for the covariance structure?

Thanks in advance!

-David

Hi David,

Apologies for the delayed response. You are right, 2^4=16 is the total number of distinct permutations in the case of a one-sample t-test.

The NBS samples with replacement, so it will always generate exactly as many permutations as requested. Of course, when n=4, it is likely that many permutations are identical. Most permutation-based software uses this approach, since it becomes prohibitive to generate all possible permutations for typical sample sizes (n=20).

The hypothesis you are seeking to test is not clear to me. In any case, testing for between-group difference with 4 subjects will probably be challenging. If you are simply interested in determining whether the correlation at an edge is significantly different from zero, then thresholding based on a Z-distribution sounds reasonable to me. To combine the p-values at a given edge across the 4 subjects, you might then want to consider Fisher's Method: http://en.wikipedia.org/wiki/Fisher%27s_...

I hope that helps,

Andrew


Originally posted by David Grayson:

Hi again,

Thanks for your response - I am finally following up on this one after a long time.

Just to refresh, the question is how can one control for multiple comparisons across edges when sample size isn't large enough to use the vanilla version of the NBS? I have 4 subjects with functional connectivity data, each has undergone an experimental manipulation which provides a difference measure for the correlation at each edge (before vs after). I can assess the significance of the change per subject using a Z-distribution and can combine p-values across subjects using Fisher's method as you pointed out.

However, this still leaves me with the tricky question in this dataset of how I might control for familywise error rate across edges? My analysis included 80 nodes, which means FDR is just not powerful enough. While I could winnow down the nodes through some more strict hypothesis-testing, it would be nice if possible to run some other kind of FWE correction on the whole 80-node set, using topological theory, like the NBS does, to compare the observed cluster size against some theoretical null distribution of cluster sizes. Is there any reasonable way to do this that you could envision, or is this case just too unconventional?

Thanks again,

David

David, conducting any kind of statistical analysis on 4 observation is very challenging. This is not something specific to the NBS.

You can generate a z-stat as you describe, but estimating the std is challenging with observations.

Andrew

Originally posted by David Grayson: