help > Non-parametric cluster correction single subj
Showing 1-11 of 11 posts
Nov 23, 2017 03:11 AM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
Non-parametric cluster correction single subj
Dear Alfonso,
I think I ran into a bug, or at least an undocumented limitation.
I would like to use a non-parametric cluster correction for a single subject against a control group.
The calculation of the permutation test runs fine, but after the calculation, I get a threshold equivalent to k > 1 (thus cluster uncorrected, equivalent to cluster-size = 0) for any non-parametric threshold type I choose, and any threshold value!
Clearly this means that the permutation test could not work correctly in this instance. However I can make it work with all my other studies involving group A vs group B, or session 1 vs session 2, so I guess the limitation here is because one group contains only a single subject.
Is it really impossible to do a permutation test with only a single subject? If that's the case, maybe adding a warning window would be useful? (to avoid experimenters using the non-parametric test when in fact it does not correct anything).
Thank you very much for clarifying this issue and helping!
Warm regards,
Stephen Larroque
I think I ran into a bug, or at least an undocumented limitation.
I would like to use a non-parametric cluster correction for a single subject against a control group.
The calculation of the permutation test runs fine, but after the calculation, I get a threshold equivalent to k > 1 (thus cluster uncorrected, equivalent to cluster-size = 0) for any non-parametric threshold type I choose, and any threshold value!
Clearly this means that the permutation test could not work correctly in this instance. However I can make it work with all my other studies involving group A vs group B, or session 1 vs session 2, so I guess the limitation here is because one group contains only a single subject.
Is it really impossible to do a permutation test with only a single subject? If that's the case, maybe adding a warning window would be useful? (to avoid experimenters using the non-parametric test when in fact it does not correct anything).
Thank you very much for clarifying this issue and helping!
Warm regards,
Stephen Larroque
Nov 27, 2017 03:11 PM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
In fact I'm not sure if this is a bug or if it is perfectly normal
to have this kind of results, are cluster-wise non-parametric
corrections expected to work in CONN for a single subject (as group
A) vs a controls group (group B)?
BTW, you probably know about it, but apparently a team solved the issue of non-parametric testing in fMRI and in particular the temporal autocorrelation issue by devising a "blockwise" permutation scheme: Adolf, D., Weston, S., Baecke, S., Luchtmann, M., Bernarding, J., & Kropf, S. (2014). Increasing the reliability of data analysis of functional magnetic resonance imaging by applying a new blockwise permutation method. Frontiers in neuroinformatics, 8 ; (implemented in SPM toolbox StabMultip, link provided in the article).
BTW, you probably know about it, but apparently a team solved the issue of non-parametric testing in fMRI and in particular the temporal autocorrelation issue by devising a "blockwise" permutation scheme: Adolf, D., Weston, S., Baecke, S., Luchtmann, M., Bernarding, J., & Kropf, S. (2014). Increasing the reliability of data analysis of functional magnetic resonance imaging by applying a new blockwise permutation method. Frontiers in neuroinformatics, 8 ; (implemented in SPM toolbox StabMultip, link provided in the article).
Nov 29, 2017 11:11 PM | Alfonso Nieto-Castanon - Boston University
Non-parametric cluster correction single subj
Dear Stephen,
Thank you very much for reporting this. I am not exactly sure but even though the group-with-a-single-subject test is a little bit of a corner case I would still expect randomization/permutation tests to work in this scenario. I will take a closer look and let you know what I find.
Thanks again
Alfonso
Originally posted by Stephen L.:
Thank you very much for reporting this. I am not exactly sure but even though the group-with-a-single-subject test is a little bit of a corner case I would still expect randomization/permutation tests to work in this scenario. I will take a closer look and let you know what I find.
Thanks again
Alfonso
Originally posted by Stephen L.:
In fact I'm not sure if this is a bug or if it
is perfectly normal to have this kind of results, are cluster-wise
non-parametric corrections expected to work in CONN for a single
subject (as group A) vs a controls group (group B)?
BTW, you probably know about it, but apparently a team solved the issue of non-parametric testing in fMRI and in particular the temporal autocorrelation issue by devising a "blockwise" permutation scheme: Adolf, D., Weston, S., Baecke, S., Luchtmann, M., Bernarding, J., & Kropf, S. (2014). Increasing the reliability of data analysis of functional magnetic resonance imaging by applying a new blockwise permutation method. Frontiers in neuroinformatics, 8 ; (implemented in SPM toolbox StabMultip, link provided in the article).
BTW, you probably know about it, but apparently a team solved the issue of non-parametric testing in fMRI and in particular the temporal autocorrelation issue by devising a "blockwise" permutation scheme: Adolf, D., Weston, S., Baecke, S., Luchtmann, M., Bernarding, J., & Kropf, S. (2014). Increasing the reliability of data analysis of functional magnetic resonance imaging by applying a new blockwise permutation method. Frontiers in neuroinformatics, 8 ; (implemented in SPM toolbox StabMultip, link provided in the article).
Nov 30, 2017 01:11 AM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
Thank you very much Alfonso!
Just to be safe, I think I will stick to parametric cluster-size FDR for the study where I encounter this case, as it corrects more than non-parametric tests, including cluster-mass FWE!
If it can make your work easier, I can also send you the whole study with the CONN project? I can setup a server and you can just download the file from a web browser :-)
Just to be safe, I think I will stick to parametric cluster-size FDR for the study where I encounter this case, as it corrects more than non-parametric tests, including cluster-mass FWE!
If it can make your work easier, I can also send you the whole study with the CONN project? I can setup a server and you can just download the file from a web browser :-)
Nov 30, 2017 01:11 AM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
But just a thought I had, but I am no expert at all in permutation
tests as I am still discovering them: I think it might not
necessarily be a bug but an inherent feature of the permutation
test at the group level with very skewed groups: since we only have
one subject in GROUP_A, any other permutation is just (GROUP_B - 1
GROUP_B's subject + 1 GROUP_A's patient), so I guess the
variability will be in fact close to none and the result of the
permutations always be very close to GROUP_B. So when we do a
contrast, the correct labelling will be super significant compared
to all other permutations of the labels, since all permutations
will basically be the GROUP_B average connectivity.
If this reasoning is correct, this means that it is useless to do a permutation test with only a single subject in one group, because anyway the controls group will always ensure a stable null hypothesis based on only the controls, thus it will always make any voxel significant in the single subject when we do a contrast... I don't know if this makes sense :-/
If this reasoning is correct, this means that it is useless to do a permutation test with only a single subject in one group, because anyway the controls group will always ensure a stable null hypothesis based on only the controls, thus it will always make any voxel significant in the single subject when we do a contrast... I don't know if this makes sense :-/
Dec 10, 2017 06:12 PM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
I found
[url=https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=FSL;18112e43.0903]a
discussion with Thomas Nicols on this issue,[/url] it seems my
reasoning was quite close to what he says there, but apparently he
affirms this is not an issue (except that the test might not be
very sensitive).
In other words, applying a permutation test on a single-subject against group study is only risking being too conservative. Which is very fine!
In other words, applying a permutation test on a single-subject against group study is only risking being too conservative. Which is very fine!
Dec 10, 2017 07:12 PM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
But still I don't believe the results I got were correct in my
specific case, as the non-parametric cluster-wise correction was
equivalent to no correction :-/
Dec 11, 2017 07:12 PM | Alfonso Nieto-Castanon - Boston University
Non-parametric cluster correction single subj
Dear Stephen,
Thank you very much for this thread, I have been looking at this for the past few days and I believe you are absolutely right and there is in fact an issue with the standard randomization procedure in these group-vs-one-sample comparison scenarios (e.g. a two-sample t-test where one group has a single subject). I am attaching a potential patch, this is still an ongoing work since I am still working on running simulations to more fully validate the new procedure but if you would like to give it a try and let me know if you run into any issues that would be fantastic (I implemented this patch on the development version, which also includes things like the ability to parallelize the permutation/randomization procedure; I believe I have sent you all necessary files to have this patch working with version 17f but please let me know if you run into any installation issues).
To elaborate a bit, the procedure CONN uses (same as FSL randomise) for permutation/randomization analyses is not a simple permutation approach, but rather a "randomization of residuals" approach. The procedure consists of, first, computing the second-level model residuals for each subject, and then, many times, building a new dataset by randomly flipping the signs of these residuals and running the second-level analyses on these new data, using the results of these randomized datasets to build a distribution of your statistic of interest (e.g. cluster sizes) under the null hypothesis. The main advantage of this "randomization of residuals" approach (e.g. CONN, FSL) compared to the "permutations of residuals" or purely "permutation" approaches (e.g. SnPM, BROCCOLI) is that it works exactly in the same way for all GLM analyses, while a permutation approach requires different permutation schemes for different sorts of analyses and it does not apply to certain scenarios (e.g. one-sample t-test). Having said that, you are absolutely right that in this group-vs-one-sample comparison scenarios the standard randomization of residuals approach does not offer the correct statistics. The reason for this seems to be related to the inability of the "sign-flipping" procedure to build the proper null distribution for the group that contains a single sample/subject. The "fix" that I introduced in the attached file is to use, instead of flipping the signs (or permuting) the residuals, a full multiplication by a random orthogonal matrix (since both permutation and sign-flip operations can be considered special-cases of a orthogonal transformation, and orthogonal transformations are the most general class of transformations guaranteeing that the randomized data has exactly the same spatial covariance structure as your original data).
Let me know your thoughts/comments
Alfonso
Originally posted by Stephen L.:
Thank you very much for this thread, I have been looking at this for the past few days and I believe you are absolutely right and there is in fact an issue with the standard randomization procedure in these group-vs-one-sample comparison scenarios (e.g. a two-sample t-test where one group has a single subject). I am attaching a potential patch, this is still an ongoing work since I am still working on running simulations to more fully validate the new procedure but if you would like to give it a try and let me know if you run into any issues that would be fantastic (I implemented this patch on the development version, which also includes things like the ability to parallelize the permutation/randomization procedure; I believe I have sent you all necessary files to have this patch working with version 17f but please let me know if you run into any installation issues).
To elaborate a bit, the procedure CONN uses (same as FSL randomise) for permutation/randomization analyses is not a simple permutation approach, but rather a "randomization of residuals" approach. The procedure consists of, first, computing the second-level model residuals for each subject, and then, many times, building a new dataset by randomly flipping the signs of these residuals and running the second-level analyses on these new data, using the results of these randomized datasets to build a distribution of your statistic of interest (e.g. cluster sizes) under the null hypothesis. The main advantage of this "randomization of residuals" approach (e.g. CONN, FSL) compared to the "permutations of residuals" or purely "permutation" approaches (e.g. SnPM, BROCCOLI) is that it works exactly in the same way for all GLM analyses, while a permutation approach requires different permutation schemes for different sorts of analyses and it does not apply to certain scenarios (e.g. one-sample t-test). Having said that, you are absolutely right that in this group-vs-one-sample comparison scenarios the standard randomization of residuals approach does not offer the correct statistics. The reason for this seems to be related to the inability of the "sign-flipping" procedure to build the proper null distribution for the group that contains a single sample/subject. The "fix" that I introduced in the attached file is to use, instead of flipping the signs (or permuting) the residuals, a full multiplication by a random orthogonal matrix (since both permutation and sign-flip operations can be considered special-cases of a orthogonal transformation, and orthogonal transformations are the most general class of transformations guaranteeing that the randomized data has exactly the same spatial covariance structure as your original data).
Let me know your thoughts/comments
Alfonso
Originally posted by Stephen L.:
But still I don't believe the results I got were
correct in my specific case, as the non-parametric cluster-wise
correction was equivalent to no correction :-/
Dec 11, 2017 08:12 PM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
Hey Alfonso!
That's AWESOME!!! Thank you so much for the patch and the detailed explanations!
I just tested the patch, it also needed a few missing resource files (conn_vproject_icon06.jpg and conn_vproject_icon07.jpg), so I just placed dummy images instead and it works :-)
First thing I notice is that it seems about 2x slower when doing 1000 iterations, but it WORKS!
My knowledge of randomization procedures are too limited to give any feedback on the technical side, but from my results I could compare non-parametric results with parametric results, and also I tried to change the cluster-wise threshold, all the non-parametric results now seem quite correct! As expected, for the exact same threshold type and amount, the non-parametric correction is a bit more conservative than the parametric one, and changing the amount now change the effect size as expected.
So from my practical observations on my dataset, it seems the patch is indeed working very fine! :D
Thank you VERY much Alfonso, that was a very quick and well done patch!
(and BTW I'm very curious what are the 2 new visualization options ;p)
Best regards!
Stephen
That's AWESOME!!! Thank you so much for the patch and the detailed explanations!
I just tested the patch, it also needed a few missing resource files (conn_vproject_icon06.jpg and conn_vproject_icon07.jpg), so I just placed dummy images instead and it works :-)
First thing I notice is that it seems about 2x slower when doing 1000 iterations, but it WORKS!
My knowledge of randomization procedures are too limited to give any feedback on the technical side, but from my results I could compare non-parametric results with parametric results, and also I tried to change the cluster-wise threshold, all the non-parametric results now seem quite correct! As expected, for the exact same threshold type and amount, the non-parametric correction is a bit more conservative than the parametric one, and changing the amount now change the effect size as expected.
So from my practical observations on my dataset, it seems the patch is indeed working very fine! :D
Thank you VERY much Alfonso, that was a very quick and well done patch!
(and BTW I'm very curious what are the 2 new visualization options ;p)
Best regards!
Stephen
Feb 8, 2018 09:02 PM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
Originally posted by Alfonso Nieto-Castanon:
After reading a bit more about non-parametric issues for functional mri, I think I understand better the above sentence, and that's a great thing you took this approach. When you mention FSL, are referring specifically to FSL PALM? If that's the case, I read that the approaches implemented there are the state-of-the-art and more generic, so that's great if CONN implements a similar approach! :D
Best!
Stephen Larroque
... The main
advantage of this "randomization of residuals" approach (e.g. CONN,
FSL) compared to the "permutations of residuals" or purely
"permutation" approaches (e.g. SnPM, BROCCOLI) is that it works
exactly in the same way for all GLM analyses, while a permutation
approach requires different permutation schemes for different sorts
of analyses and it does not apply to certain scenarios (e.g.
one-sample t-test)...
After reading a bit more about non-parametric issues for functional mri, I think I understand better the above sentence, and that's a great thing you took this approach. When you mention FSL, are referring specifically to FSL PALM? If that's the case, I read that the approaches implemented there are the state-of-the-art and more generic, so that's great if CONN implements a similar approach! :D
Best!
Stephen Larroque
Feb 8, 2018 09:02 PM | Stephen L. - Coma Science Group, GIGA-Consciousness, Hospital & University of Liege
RE: Non-parametric cluster correction single subj
BTW one last remark: if the approach you implemented here is new,
you should definitely publish it, as I am sure it will find many
interesting uses for various studies (since single-case studies are
very common in clinical settings), your "fix" can be salvaging!