help > The abnormal random and actual sizes during permutation
Showing 1-3 of 3 posts
Feb 16, 2022 04:02 PM | Chanyuan Gu
The abnormal random and actual sizes during permutation
Hi Andrew,
I compared two groups using NBS and T-test, but the random and actual sizes are hard to explain. For one direction (e.g., 1 -1), the random and actual sizes are both 0, and the p-value is 1. However, it still showed a subnetwork even though the P-value is 1 after the permutation. For another direction (e.g., -1 1), the random and actual sizes are both large, almost over 2000. The subnetwork that NBS reported consisted of all nodes (90) and 3438 edges. I also adjusted the threshold from small (e.g., 1) to large values (e.g., 15), but the results were the same for both directions.
I have already used NBS for a while, but I have never met this issue before. Hence, I am lost here, could you please give me some suggestions to solve this issue? Thank you so much!
Best,
Gucy
I compared two groups using NBS and T-test, but the random and actual sizes are hard to explain. For one direction (e.g., 1 -1), the random and actual sizes are both 0, and the p-value is 1. However, it still showed a subnetwork even though the P-value is 1 after the permutation. For another direction (e.g., -1 1), the random and actual sizes are both large, almost over 2000. The subnetwork that NBS reported consisted of all nodes (90) and 3438 edges. I also adjusted the threshold from small (e.g., 1) to large values (e.g., 15), but the results were the same for both directions.
I have already used NBS for a while, but I have never met this issue before. Hence, I am lost here, could you please give me some suggestions to solve this issue? Thank you so much!
Best,
Gucy
Feb 16, 2022 11:02 PM | Andrew Zalesky
RE: The abnormal random and actual sizes during permutation
Hi Gucy,
- If the observed (actual) size and the size under the null hypothesis are 0, then it makes sense that the p-value is 1. This means that you cannot reject the null hypothesis. This is consistent behaviour. I am not sure what you mean by "it still showed a subnetwork even though the P-value is 1". If you put a p-value threshold of 0.05, no network should be shown in this case.
- If you are findings that all edges from a massive network that remains significant, even for very high thresholds, it is likely that your design matrix is not specified correctly. For example, you might be detecting a mean effect rather than a between-group difference. Consider checking the design matrix.
Andrew
Originally posted by Chanyuan Gu:
- If the observed (actual) size and the size under the null hypothesis are 0, then it makes sense that the p-value is 1. This means that you cannot reject the null hypothesis. This is consistent behaviour. I am not sure what you mean by "it still showed a subnetwork even though the P-value is 1". If you put a p-value threshold of 0.05, no network should be shown in this case.
- If you are findings that all edges from a massive network that remains significant, even for very high thresholds, it is likely that your design matrix is not specified correctly. For example, you might be detecting a mean effect rather than a between-group difference. Consider checking the design matrix.
Andrew
Originally posted by Chanyuan Gu:
Hi Andrew,
I compared two groups using NBS and T-test, but the random and actual sizes are hard to explain. For one direction (e.g., 1 -1), the random and actual sizes are both 0, and the p-value is 1. However, it still showed a subnetwork even though the P-value is 1 after the permutation. For another direction (e.g., -1 1), the random and actual sizes are both large, almost over 2000. The subnetwork that NBS reported consisted of all nodes (90) and 3438 edges. I also adjusted the threshold from small (e.g., 1) to large values (e.g., 15), but the results were the same for both directions.
I have already used NBS for a while, but I have never met this issue before. Hence, I am lost here, could you please give me some suggestions to solve this issue? Thank you so much!
Best,
Gucy
I compared two groups using NBS and T-test, but the random and actual sizes are hard to explain. For one direction (e.g., 1 -1), the random and actual sizes are both 0, and the p-value is 1. However, it still showed a subnetwork even though the P-value is 1 after the permutation. For another direction (e.g., -1 1), the random and actual sizes are both large, almost over 2000. The subnetwork that NBS reported consisted of all nodes (90) and 3438 edges. I also adjusted the threshold from small (e.g., 1) to large values (e.g., 15), but the results were the same for both directions.
I have already used NBS for a while, but I have never met this issue before. Hence, I am lost here, could you please give me some suggestions to solve this issue? Thank you so much!
Best,
Gucy
Feb 17, 2022 12:02 AM | Chanyuan Gu
RE: The abnormal random and actual sizes during permutation
Hi Andrew,
Thank you so much for your reply!
Right, there should be no network when the p-value is 1 and the threshold is 0.05. However, one network was still reported by the NBS (please find attached the "P-value_1.png"). Also, I checked my design matrix, and it seems that the matrix is fine (please find attached the "design.txt"). In total, eight columns were included in the matrix. The second column represents group 1, and the third column represents group 2. Others indicate intercept or the covariates that I would like to control. Hence, the contrasts that I used are [0 1 -1 0 0 0 0 0] and [0 -1 1 0 0 0 0 0].
Could you please help me check the design matrix because I can not find out errors about the design matrix? Thank you so much!
Best,
Gucy
Originally posted by Andrew Zalesky:
Thank you so much for your reply!
Right, there should be no network when the p-value is 1 and the threshold is 0.05. However, one network was still reported by the NBS (please find attached the "P-value_1.png"). Also, I checked my design matrix, and it seems that the matrix is fine (please find attached the "design.txt"). In total, eight columns were included in the matrix. The second column represents group 1, and the third column represents group 2. Others indicate intercept or the covariates that I would like to control. Hence, the contrasts that I used are [0 1 -1 0 0 0 0 0] and [0 -1 1 0 0 0 0 0].
Could you please help me check the design matrix because I can not find out errors about the design matrix? Thank you so much!
Best,
Gucy
Originally posted by Andrew Zalesky:
Hi Gucy,
- If the observed (actual) size and the size under the null hypothesis are 0, then it makes sense that the p-value is 1. This means that you cannot reject the null hypothesis. This is consistent behaviour. I am not sure what you mean by "it still showed a subnetwork even though the P-value is 1". If you put a p-value threshold of 0.05, no network should be shown in this case.
- If you are findings that all edges from a massive network that remains significant, even for very high thresholds, it is likely that your design matrix is not specified correctly. For example, you might be detecting a mean effect rather than a between-group difference. Consider checking the design matrix.
Andrew
Originally posted by Chanyuan Gu:
- If the observed (actual) size and the size under the null hypothesis are 0, then it makes sense that the p-value is 1. This means that you cannot reject the null hypothesis. This is consistent behaviour. I am not sure what you mean by "it still showed a subnetwork even though the P-value is 1". If you put a p-value threshold of 0.05, no network should be shown in this case.
- If you are findings that all edges from a massive network that remains significant, even for very high thresholds, it is likely that your design matrix is not specified correctly. For example, you might be detecting a mean effect rather than a between-group difference. Consider checking the design matrix.
Andrew
Originally posted by Chanyuan Gu:
Hi Andrew,
I compared two groups using NBS and T-test, but the random and actual sizes are hard to explain. For one direction (e.g., 1 -1), the random and actual sizes are both 0, and the p-value is 1. However, it still showed a subnetwork even though the P-value is 1 after the permutation. For another direction (e.g., -1 1), the random and actual sizes are both large, almost over 2000. The subnetwork that NBS reported consisted of all nodes (90) and 3438 edges. I also adjusted the threshold from small (e.g., 1) to large values (e.g., 15), but the results were the same for both directions.
I have already used NBS for a while, but I have never met this issue before. Hence, I am lost here, could you please give me some suggestions to solve this issue? Thank you so much!
Best,
Gucy
I compared two groups using NBS and T-test, but the random and actual sizes are hard to explain. For one direction (e.g., 1 -1), the random and actual sizes are both 0, and the p-value is 1. However, it still showed a subnetwork even though the P-value is 1 after the permutation. For another direction (e.g., -1 1), the random and actual sizes are both large, almost over 2000. The subnetwork that NBS reported consisted of all nodes (90) and 3438 edges. I also adjusted the threshold from small (e.g., 1) to large values (e.g., 15), but the results were the same for both directions.
I have already used NBS for a while, but I have never met this issue before. Hence, I am lost here, could you please give me some suggestions to solve this issue? Thank you so much!
Best,
Gucy