help
81 Subscribers 
help > Multiple primary thresholds correction
Showing 1-4 of 4 posts
Nov 17, 2021 01:11 PM | Peder Lillebostad
Multiple primary thresholds correction
Hello NBS experts,
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
Nov 17, 2021 10:11 PM | Andrew Zalesky
RE: Multiple primary thresholds correction
Hi Peder,
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
Hello NBS experts,
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
Nov 18, 2021 03:11 PM | Peder Lillebostad
RE: Multiple primary thresholds correction
Thank you for the quick and detailed reply!
So if I understand you correctly, it is not methodologically flawed to dynamically "optimize" a primary threshold (i.e. not deciding on a priori values)? Of course provided that I am transparent about the methods.
Thank you very much.
Peder
Originally posted by Andrew Zalesky:
So if I understand you correctly, it is not methodologically flawed to dynamically "optimize" a primary threshold (i.e. not deciding on a priori values)? Of course provided that I am transparent about the methods.
Thank you very much.
Peder
Originally posted by Andrew Zalesky:
Hi
Peder,
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
Hello NBS experts,
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
Nov 18, 2021 09:11 PM | Andrew Zalesky
RE: Multiple primary thresholds correction
Thanks Peder,
to clarify, systematically evaluating hundreds of thresholds and then cherry-picking and reporting a threshold for which the p-value is smallest is not really appropriate in my opinion. I'm not sure if that's what you mean by dynamic optimization.
I was referring to consideration of a couple of thresholds.
You may want to consider the TFCE version of NBS, which does not require specification of a threshold.
best,
Andrew
Originally posted by Peder Lillebostad:
to clarify, systematically evaluating hundreds of thresholds and then cherry-picking and reporting a threshold for which the p-value is smallest is not really appropriate in my opinion. I'm not sure if that's what you mean by dynamic optimization.
I was referring to consideration of a couple of thresholds.
You may want to consider the TFCE version of NBS, which does not require specification of a threshold.
best,
Andrew
Originally posted by Peder Lillebostad:
Thank you for the quick and detailed reply!
So if I understand you correctly, it is not methodologically flawed to dynamically "optimize" a primary threshold (i.e. not deciding on a priori values)? Of course provided that I am transparent about the methods.
Thank you very much.
Peder
Originally posted by Andrew Zalesky:
So if I understand you correctly, it is not methodologically flawed to dynamically "optimize" a primary threshold (i.e. not deciding on a priori values)? Of course provided that I am transparent about the methods.
Thank you very much.
Peder
Originally posted by Andrew Zalesky:
Hi
Peder,
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
Hello NBS experts,
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder