help > Change of nr of edges with population size
Showing 1-7 of 7 posts
Oct 15, 2018 12:10 PM | francesca sibilia
Change of nr of edges with population size
Hi NBS expert,
I am comparing the effect of stress on my population, dividing it into LowStress and HighStress groups. I ran a t-test ([1,-1] contrast, FDR analysis) to check for decreased connectivity in the HighStress group. My initial population size was N=1020, and I got 6 significant edges that are decreased in the HS group. Later on, I had to decrease the number of my participants, and now the sample size is N=976. I ran the same analysis with the same parameters, and I got only 3 significant different edges between the two groups ( and they are not the same!)
Is it possible that changing the number of subjects in my data set influences the results as well? What does it change in the edges calculation when I remove some participants?
Thanks in advance for your reply.
Best regards,
Fran
I am comparing the effect of stress on my population, dividing it into LowStress and HighStress groups. I ran a t-test ([1,-1] contrast, FDR analysis) to check for decreased connectivity in the HighStress group. My initial population size was N=1020, and I got 6 significant edges that are decreased in the HS group. Later on, I had to decrease the number of my participants, and now the sample size is N=976. I ran the same analysis with the same parameters, and I got only 3 significant different edges between the two groups ( and they are not the same!)
Is it possible that changing the number of subjects in my data set influences the results as well? What does it change in the edges calculation when I remove some participants?
Thanks in advance for your reply.
Best regards,
Fran
Oct 16, 2018 12:10 AM | Andrew Zalesky
RE: Change of nr of edges with population size
Hi Fran,
removing or adding subjects can affect statistical significance. This is true for any kind of statistical comparison.
Most importantly, when using FDR, it is crucial to increase the number of permutations to 100,000 or more. Note that 5000 permutations is NOT sufficient with FDR. Using 5000 permutations with FDR will yield unstable results.
Andrew
Originally posted by francesca sibilia:
removing or adding subjects can affect statistical significance. This is true for any kind of statistical comparison.
Most importantly, when using FDR, it is crucial to increase the number of permutations to 100,000 or more. Note that 5000 permutations is NOT sufficient with FDR. Using 5000 permutations with FDR will yield unstable results.
Andrew
Originally posted by francesca sibilia:
Hi NBS expert,
I am comparing the effect of stress on my population, dividing it into LowStress and HighStress groups. I ran a t-test ([1,-1] contrast, FDR analysis) to check for decreased connectivity in the HighStress group. My initial population size was N=1020, and I got 6 significant edges that are decreased in the HS group. Later on, I had to decrease the number of my participants, and now the sample size is N=976. I ran the same analysis with the same parameters, and I got only 3 significant different edges between the two groups ( and they are not the same!)
Is it possible that changing the number of subjects in my data set influences the results as well? What does it change in the edges calculation when I remove some participants?
Thanks in advance for your reply.
Best regards,
Fran
I am comparing the effect of stress on my population, dividing it into LowStress and HighStress groups. I ran a t-test ([1,-1] contrast, FDR analysis) to check for decreased connectivity in the HighStress group. My initial population size was N=1020, and I got 6 significant edges that are decreased in the HS group. Later on, I had to decrease the number of my participants, and now the sample size is N=976. I ran the same analysis with the same parameters, and I got only 3 significant different edges between the two groups ( and they are not the same!)
Is it possible that changing the number of subjects in my data set influences the results as well? What does it change in the edges calculation when I remove some participants?
Thanks in advance for your reply.
Best regards,
Fran
Oct 19, 2018 01:10 PM | francesca sibilia
RE: Change of nr of edges with population size
Hi Andrew,
Thanks for your reply.
I used 10,000 permutations, because I read that is an acceptable minimum number of permutations. Also, can the number of permutations be influenced by the population size (less/more permutations if N is bigger/smaller)?
This is more for my own knowledge, since I am quite new using NBS and would like to understand more of it.
Thanks and regards,
Fran
Thanks for your reply.
I used 10,000 permutations, because I read that is an acceptable minimum number of permutations. Also, can the number of permutations be influenced by the population size (less/more permutations if N is bigger/smaller)?
This is more for my own knowledge, since I am quite new using NBS and would like to understand more of it.
Thanks and regards,
Fran
Oct 19, 2018 11:10 PM | Andrew Zalesky
RE: Change of nr of edges with population size
Hi Fran,
For the NBS, 10,000 permutations is more than sufficient. Even 5,000 should be enough.
However, when using FDR, very substantially more permutations should be generated, otherwise the results will not be reliable. For this reason, we will use a parameteric version of FDR is future versions of the software.
Sample size should not be a major factor determining the number of permutations; unless your sample size is very small, in which case all possible permutations can be enumerated.
Andrew
Originally posted by francesca sibilia:
For the NBS, 10,000 permutations is more than sufficient. Even 5,000 should be enough.
However, when using FDR, very substantially more permutations should be generated, otherwise the results will not be reliable. For this reason, we will use a parameteric version of FDR is future versions of the software.
Sample size should not be a major factor determining the number of permutations; unless your sample size is very small, in which case all possible permutations can be enumerated.
Andrew
Originally posted by francesca sibilia:
Hi Andrew,
Thanks for your reply.
I used 10,000 permutations, because I read that is an acceptable minimum number of permutations. Also, can the number of permutations be influenced by the population size (less/more permutations if N is bigger/smaller)?
This is more for my own knowledge, since I am quite new using NBS and would like to understand more of it.
Thanks and regards,
Fran
Thanks for your reply.
I used 10,000 permutations, because I read that is an acceptable minimum number of permutations. Also, can the number of permutations be influenced by the population size (less/more permutations if N is bigger/smaller)?
This is more for my own knowledge, since I am quite new using NBS and would like to understand more of it.
Thanks and regards,
Fran
Oct 23, 2018 04:10 PM | francesca sibilia
RE: Change of nr of edges with population size
Hi Andrew,
I changed the number to permutations from 10,000 to 100,000 and I got not significant results. Is this an expected result (increasing the number of permutations) or maybe I did something wrong ?
Also would you suggest to use weighted or binary matrices as input for FDR?
Thanks and regards,
Fran
I changed the number to permutations from 10,000 to 100,000 and I got not significant results. Is this an expected result (increasing the number of permutations) or maybe I did something wrong ?
Also would you suggest to use weighted or binary matrices as input for FDR?
Thanks and regards,
Fran
Oct 25, 2018 12:10 AM | Andrew Zalesky
RE: Change of nr of edges with population size
The most likely explanation is that the results for 10,000 were not
reliable. It is possible that you don't have any significant
findings.
You should trust the results with 100,000 permutations for FDR.
10,000 permutations is NOT SUFFICIENT for FDR and cannot be used for publication.
Originally posted by francesca sibilia:
You should trust the results with 100,000 permutations for FDR.
10,000 permutations is NOT SUFFICIENT for FDR and cannot be used for publication.
Originally posted by francesca sibilia:
Hi Andrew,
I changed the number to permutations from 10,000 to 100,000 and I got not significant results. Is this an expected result (increasing the number of permutations) or maybe I did something wrong ?
Also would you suggest to use weighted or binary matrices as input for FDR?
Thanks and regards,
Fran
I changed the number to permutations from 10,000 to 100,000 and I got not significant results. Is this an expected result (increasing the number of permutations) or maybe I did something wrong ?
Also would you suggest to use weighted or binary matrices as input for FDR?
Thanks and regards,
Fran
Oct 25, 2018 01:10 PM | francesca sibilia
RE: Change of nr of edges with population size
Thanks for your reply.
Best,
Fran
Best,
Fran