help
help > RE: Multiple primary thresholds correction
Nov 18, 2021 03:11 PM | Peder Lillebostad
RE: Multiple primary thresholds correction
Thank you for the quick and detailed reply!
So if I understand you correctly, it is not methodologically flawed to dynamically "optimize" a primary threshold (i.e. not deciding on a priori values)? Of course provided that I am transparent about the methods.
Thank you very much.
Peder
Originally posted by Andrew Zalesky:
So if I understand you correctly, it is not methodologically flawed to dynamically "optimize" a primary threshold (i.e. not deciding on a priori values)? Of course provided that I am transparent about the methods.
Thank you very much.
Peder
Originally posted by Andrew Zalesky:
Hi
Peder,
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
yes - testing a range of thresholds technically creates a multiple comparisons problem. However, if only a few different thresholds are tested and these are reported in a paper, I would argue that multiple comparisons correction is potentially overkill.
You may want to select a single threshold that corresponds to the minimum meaningful effect size in your experiment.
If you are determined to correct for multiple comparisons across thresholds, FDR is a good option. A more principled approach would be to use NBS-TFCE, which does not require a primary threshold choice, and thus overcomes this problem.
best,
Andrew
Originally posted by Peder Lillebostad:
Hello NBS experts,
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
In the manual you recommend to test a range of primary thresholds. If I'm not mistaken, this brings about another problem of multiple testing, and ought to be corrected for.
But here is the problem: doing an FDR correction would be too conservative, because the values are not independent for obvious reasons. Does there exist a clever solution to this?
Thanks a lot,
Peder
Threaded View
| Title | Author | Date |
|---|---|---|
| Peder Lillebostad | Nov 17, 2021 | |
| Andrew Zalesky | Nov 17, 2021 | |
| Peder Lillebostad | Nov 18, 2021 | |
| Andrew Zalesky | Nov 18, 2021 | |