help > Design matrix and contrast for 2x2 design
Showing 1-18 of 18 posts
Aug 3, 2017 03:08 PM | David de Wide
Design matrix and contrast for 2x2 design
Dear Dr. Zalesky,
Thank you to you and your collegues for the wonderful tools and toolboxes you have given the neuroscientific research community!
I've been working with the NBS toolbox for most of the day, and while I have made great progress in understanding what I am doing, I'm still unsure about which design matrix to use (I've made quite a few).
The help section for statistical model was very useful, but after creating several different matrices and analyses, I'm still unsure which is correct.
Simply put, I have 48 matrices belonging to 12 participants (within subjects design). They were tested on 2 seperate days (Placebo vs Drug) and had two resting state scans on each day (No music vs Music). The order of the matrices (90x90x48) is as follows:
S01 Placebo No-music (0 0)
till
S12 Placebo No-music (0 0)
S01 Placebo Music (0 1)
till
S12 Placebo Music (0 1)
S01 Drug No-music (1 0)
till
S12 Drug No-music (1 0)
S01 Drug Music (1 1)
till
S12 Drug Music (1 1)
I am interested in the main effects of drug and music, but more specifically in the interaction effect between drug and music (and subsequently visualizing this in BrainNet). Using FDR there are barely any suriving edges, and the NBS seems perfect for situations like these.
My main issue has been the design matrix. As pictured above, I initially had a simple 48x2 with 1s and 0s (and later 1 and -1), but was consistently getting 1000s of significant edges for certain thresholds (so clearly not looking at the specific interaction). Could you tell me the dimensions of the required/recommended design matrix? I've attempted a 48x3 for A, B and AxB, 48x5 for A1, A2, B1, B2, AB, but am confused about the requirement of a column for each participant?
If someone here could provide me with some tips/pointers about the required dimensions for the matrix, the interaction contrast, and how the first line in the matrxi would look, I'm sure I can figure out the rest through deductive reasoning. Any help would be greatly appericiated.
Sincerely,
David
ps. Would you recommend using raw R values or Fisher Z transformed R values?
Thank you to you and your collegues for the wonderful tools and toolboxes you have given the neuroscientific research community!
I've been working with the NBS toolbox for most of the day, and while I have made great progress in understanding what I am doing, I'm still unsure about which design matrix to use (I've made quite a few).
The help section for statistical model was very useful, but after creating several different matrices and analyses, I'm still unsure which is correct.
Simply put, I have 48 matrices belonging to 12 participants (within subjects design). They were tested on 2 seperate days (Placebo vs Drug) and had two resting state scans on each day (No music vs Music). The order of the matrices (90x90x48) is as follows:
S01 Placebo No-music (0 0)
till
S12 Placebo No-music (0 0)
S01 Placebo Music (0 1)
till
S12 Placebo Music (0 1)
S01 Drug No-music (1 0)
till
S12 Drug No-music (1 0)
S01 Drug Music (1 1)
till
S12 Drug Music (1 1)
I am interested in the main effects of drug and music, but more specifically in the interaction effect between drug and music (and subsequently visualizing this in BrainNet). Using FDR there are barely any suriving edges, and the NBS seems perfect for situations like these.
My main issue has been the design matrix. As pictured above, I initially had a simple 48x2 with 1s and 0s (and later 1 and -1), but was consistently getting 1000s of significant edges for certain thresholds (so clearly not looking at the specific interaction). Could you tell me the dimensions of the required/recommended design matrix? I've attempted a 48x3 for A, B and AxB, 48x5 for A1, A2, B1, B2, AB, but am confused about the requirement of a column for each participant?
If someone here could provide me with some tips/pointers about the required dimensions for the matrix, the interaction contrast, and how the first line in the matrxi would look, I'm sure I can figure out the rest through deductive reasoning. Any help would be greatly appericiated.
Sincerely,
David
ps. Would you recommend using raw R values or Fisher Z transformed R values?
Aug 5, 2017 12:08 AM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
Hi David,what you have is a 2 x 2 ANOVA with repeated measure on
both factors. So you will need to use exchange blocks in your
design to account for the repeated measures.
Let's assume you only have two subjects (you can easily generalize my below suggestions to 12 subjects). Let's use P and D to denote placebo and drug. Let's use M and N to denote music and no music. So "PN1" means the no music placebo condition of Subject 1. Your design matrix should look like this:
PN1 0 0 0 1 0
PN2 0 0 0 0 1
PM1 0 1 0 1 0
PM2 0 1 0 0 1
DN1 1 0 0 1 0
DN2 1 0 0 0 1
DM1 1 1 1 1 0
DM2 1 1 1 0 1
The 1st column is the main effect of drug. 2nd column is main effect of music. 3rd column is the interaction between music and drug. Column 4 and 5 is the mean of subject 1 and 2, respectively.
You should be able to extend this design matrix for 12 subjects. Note that if you have 12 subjects, you should have 15 columns in total and 12 rows. 1 column for each subject + 2 main effects + 1 interaction = 15 columns.
To test for main effect of drug, contrast is: [1 0 0 0 0] or [-1 0 0 0 0]
Main effect of music: [0 1 0 0 0] or [0 -1 0 0 0]
Interaction: [0 0 1 0 0] or [0 0 -1 0 0]
Choose: "t-test"
The exchange block should be:
[1 2 1 2 1 2 1 2]
So I have given you all the information for the case of 2 subjects. Now you should be able to generalize this to the case of 12 subjects.
Let me now if you need help to generalize from 2 to 12 subjects.
Using r or r-to-z usually makes no difference. Reviewers might bring it up - so you might as well use r-to-z.
Andrew
Originally posted by David de Wide:
Let's assume you only have two subjects (you can easily generalize my below suggestions to 12 subjects). Let's use P and D to denote placebo and drug. Let's use M and N to denote music and no music. So "PN1" means the no music placebo condition of Subject 1. Your design matrix should look like this:
PN1 0 0 0 1 0
PN2 0 0 0 0 1
PM1 0 1 0 1 0
PM2 0 1 0 0 1
DN1 1 0 0 1 0
DN2 1 0 0 0 1
DM1 1 1 1 1 0
DM2 1 1 1 0 1
The 1st column is the main effect of drug. 2nd column is main effect of music. 3rd column is the interaction between music and drug. Column 4 and 5 is the mean of subject 1 and 2, respectively.
You should be able to extend this design matrix for 12 subjects. Note that if you have 12 subjects, you should have 15 columns in total and 12 rows. 1 column for each subject + 2 main effects + 1 interaction = 15 columns.
To test for main effect of drug, contrast is: [1 0 0 0 0] or [-1 0 0 0 0]
Main effect of music: [0 1 0 0 0] or [0 -1 0 0 0]
Interaction: [0 0 1 0 0] or [0 0 -1 0 0]
Choose: "t-test"
The exchange block should be:
[1 2 1 2 1 2 1 2]
So I have given you all the information for the case of 2 subjects. Now you should be able to generalize this to the case of 12 subjects.
Let me now if you need help to generalize from 2 to 12 subjects.
Using r or r-to-z usually makes no difference. Reviewers might bring it up - so you might as well use r-to-z.
Andrew
Originally posted by David de Wide:
Dear Dr. Zalesky,
Thank you to you and your collegues for the wonderful tools and toolboxes you have given the neuroscientific research community!
I've been working with the NBS toolbox for most of the day, and while I have made great progress in understanding what I am doing, I'm still unsure about which design matrix to use (I've made quite a few).
The help section for statistical model was very useful, but after creating several different matrices and analyses, I'm still unsure which is correct.
Simply put, I have 48 matrices belonging to 12 participants (within subjects design). They were tested on 2 seperate days (Placebo vs Drug) and had two resting state scans on each day (No music vs Music). The order of the matrices (90x90x48) is as follows:
S01 Placebo No-music (0 0)
till
S12 Placebo No-music (0 0)
S01 Placebo Music (0 1)
till
S12 Placebo Music (0 1)
S01 Drug No-music (1 0)
till
S12 Drug No-music (1 0)
S01 Drug Music (1 1)
till
S12 Drug Music (1 1)
I am interested in the main effects of drug and music, but more specifically in the interaction effect between drug and music (and subsequently visualizing this in BrainNet). Using FDR there are barely any suriving edges, and the NBS seems perfect for situations like these.
My main issue has been the design matrix. As pictured above, I initially had a simple 48x2 with 1s and 0s (and later 1 and -1), but was consistently getting 1000s of significant edges for certain thresholds (so clearly not looking at the specific interaction). Could you tell me the dimensions of the required/recommended design matrix? I've attempted a 48x3 for A, B and AxB, 48x5 for A1, A2, B1, B2, AB, but am confused about the requirement of a column for each participant?
If someone here could provide me with some tips/pointers about the required dimensions for the matrix, the interaction contrast, and how the first line in the matrxi would look, I'm sure I can figure out the rest through deductive reasoning. Any help would be greatly appericiated.
Sincerely,
David
ps. Would you recommend using raw R values or Fisher Z transformed R values?
Thank you to you and your collegues for the wonderful tools and toolboxes you have given the neuroscientific research community!
I've been working with the NBS toolbox for most of the day, and while I have made great progress in understanding what I am doing, I'm still unsure about which design matrix to use (I've made quite a few).
The help section for statistical model was very useful, but after creating several different matrices and analyses, I'm still unsure which is correct.
Simply put, I have 48 matrices belonging to 12 participants (within subjects design). They were tested on 2 seperate days (Placebo vs Drug) and had two resting state scans on each day (No music vs Music). The order of the matrices (90x90x48) is as follows:
S01 Placebo No-music (0 0)
till
S12 Placebo No-music (0 0)
S01 Placebo Music (0 1)
till
S12 Placebo Music (0 1)
S01 Drug No-music (1 0)
till
S12 Drug No-music (1 0)
S01 Drug Music (1 1)
till
S12 Drug Music (1 1)
I am interested in the main effects of drug and music, but more specifically in the interaction effect between drug and music (and subsequently visualizing this in BrainNet). Using FDR there are barely any suriving edges, and the NBS seems perfect for situations like these.
My main issue has been the design matrix. As pictured above, I initially had a simple 48x2 with 1s and 0s (and later 1 and -1), but was consistently getting 1000s of significant edges for certain thresholds (so clearly not looking at the specific interaction). Could you tell me the dimensions of the required/recommended design matrix? I've attempted a 48x3 for A, B and AxB, 48x5 for A1, A2, B1, B2, AB, but am confused about the requirement of a column for each participant?
If someone here could provide me with some tips/pointers about the required dimensions for the matrix, the interaction contrast, and how the first line in the matrxi would look, I'm sure I can figure out the rest through deductive reasoning. Any help would be greatly appericiated.
Sincerely,
David
ps. Would you recommend using raw R values or Fisher Z transformed R values?
Aug 6, 2017 10:08 PM | David de Wide
RE: Design matrix and contrast for 2x2 design
Hey Andrew,
Thank you for your quick and detailed reply.
Based on your suggestions I have contructed the 48x15 design matrix (included as attachement), contrasts and exchange blocks. As a sanity check, could you confirm that this is correct?
The contrasts for the interaction effect would be [0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 ] or [0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 ]. What is the exact difference between these two versions of the contrast?
The exhange blocks would be [1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12]
At a range of thresholds there appear to be no changes for the interaction effect with a lower p value than 0.2.
I had an additional question regarding contrasts and network analysis, given your expertise. Is there a way to look at (significant) changes in modularity, rich club coefficient or participation coefficient for the interaction of music and drug specifically?
Thank you so much for all your help.
Sincerely,
David
Thank you for your quick and detailed reply.
Based on your suggestions I have contructed the 48x15 design matrix (included as attachement), contrasts and exchange blocks. As a sanity check, could you confirm that this is correct?
The contrasts for the interaction effect would be [0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 ] or [0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 ]. What is the exact difference between these two versions of the contrast?
The exhange blocks would be [1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12]
At a range of thresholds there appear to be no changes for the interaction effect with a lower p value than 0.2.
I had an additional question regarding contrasts and network analysis, given your expertise. Is there a way to look at (significant) changes in modularity, rich club coefficient or participation coefficient for the interaction of music and drug specifically?
Thank you so much for all your help.
Sincerely,
David
Aug 7, 2017 01:08 AM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
Hi David, everything looks correct.
The differences between the two contrasts is that one will test whether connectivity in increased in the music/drug condition, while the other contrast will test for a decrease in connectivity in this condition. In other words, each contrast corresponds to a one-sided alternative hypothesis.
Alternatively, you can select F-test, in which case a two sided alternative hypothesis will be considered (increase or decrease). In this case, you only need to consider the contrast with a "1".
If you want to test for differences in modularity, rich club etc, you might want to consider the Brain Connectivity Toolbox (BCT). This requires some matlab expertise (BCT does not have a gui).
Andrew
Originally posted by David de Wide:
The differences between the two contrasts is that one will test whether connectivity in increased in the music/drug condition, while the other contrast will test for a decrease in connectivity in this condition. In other words, each contrast corresponds to a one-sided alternative hypothesis.
Alternatively, you can select F-test, in which case a two sided alternative hypothesis will be considered (increase or decrease). In this case, you only need to consider the contrast with a "1".
If you want to test for differences in modularity, rich club etc, you might want to consider the Brain Connectivity Toolbox (BCT). This requires some matlab expertise (BCT does not have a gui).
Andrew
Originally posted by David de Wide:
Hey Andrew,
Thank you for your quick and detailed reply.
Based on your suggestions I have contructed the 48x15 design matrix (included as attachement), contrasts and exchange blocks. As a sanity check, could you confirm that this is correct?
The contrasts for the interaction effect would be [0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 ] or [0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 ]. What is the exact difference between these two versions of the contrast?
The exhange blocks would be [1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12]
At a range of thresholds there appear to be no changes for the interaction effect with a lower p value than 0.2.
I had an additional question regarding contrasts and network analysis, given your expertise. Is there a way to look at (significant) changes in modularity, rich club coefficient or participation coefficient for the interaction of music and drug specifically?
Thank you so much for all your help.
Sincerely,
David
Thank you for your quick and detailed reply.
Based on your suggestions I have contructed the 48x15 design matrix (included as attachement), contrasts and exchange blocks. As a sanity check, could you confirm that this is correct?
The contrasts for the interaction effect would be [0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 ] or [0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 ]. What is the exact difference between these two versions of the contrast?
The exhange blocks would be [1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12]
At a range of thresholds there appear to be no changes for the interaction effect with a lower p value than 0.2.
I had an additional question regarding contrasts and network analysis, given your expertise. Is there a way to look at (significant) changes in modularity, rich club coefficient or participation coefficient for the interaction of music and drug specifically?
Thank you so much for all your help.
Sincerely,
David
Aug 7, 2017 10:08 AM | David de Wide
RE: Design matrix and contrast for 2x2 design
Dear Andrew,
Great, I suspected that was the difference based on the areas, but the -1 contrast of Music was very similar to the 1 contrast of the interaction effect (same ROIs, similar t-scores), so I wanted to make sure.
Regarding the BCT, I've already been working with it and made several loop based scripts (for different thresholds) to generate binary matrices and run several of weighted unidirect/binary undirected metrics.
However, with regards to statistical testing, I've only found a way to compare two conditions (i.e. Drug v Placebo or Music v No music), but have been unable to determine how/if to test for an interaction effect using all four conditions. Unlike regular mean scores, I can't subtract the means of the music conditions and then compare those results between drug and placebo.
Sincerely,
David
Great, I suspected that was the difference based on the areas, but the -1 contrast of Music was very similar to the 1 contrast of the interaction effect (same ROIs, similar t-scores), so I wanted to make sure.
Regarding the BCT, I've already been working with it and made several loop based scripts (for different thresholds) to generate binary matrices and run several of weighted unidirect/binary undirected metrics.
However, with regards to statistical testing, I've only found a way to compare two conditions (i.e. Drug v Placebo or Music v No music), but have been unable to determine how/if to test for an interaction effect using all four conditions. Unlike regular mean scores, I can't subtract the means of the music conditions and then compare those results between drug and placebo.
Sincerely,
David
Aug 7, 2017 11:08 AM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
Hi David,
it is possible for the ROIs to be the same for the 1 and -1 contrasts, but the actual edges (connections) should generally be different. You might want to try different thresholds.
Unfortunately, the BCT does not include extensive functionality for statistical testing. You might need to import the network measures computed by BCT into a statistical package that you feel comfortable with and perform hypothesis testing there.
Andrew
Originally posted by David de Wide:
it is possible for the ROIs to be the same for the 1 and -1 contrasts, but the actual edges (connections) should generally be different. You might want to try different thresholds.
Unfortunately, the BCT does not include extensive functionality for statistical testing. You might need to import the network measures computed by BCT into a statistical package that you feel comfortable with and perform hypothesis testing there.
Andrew
Originally posted by David de Wide:
Dear Andrew,
Great, I suspected that was the difference based on the areas, but the -1 contrast of Music was very similar to the 1 contrast of the interaction effect (same ROIs, similar t-scores), so I wanted to make sure.
Regarding the BCT, I've already been working with it and made several loop based scripts (for different thresholds) to generate binary matrices and run several of weighted unidirect/binary undirected metrics.
However, with regards to statistical testing, I've only found a way to compare two conditions (i.e. Drug v Placebo or Music v No music), but have been unable to determine how/if to test for an interaction effect using all four conditions. Unlike regular mean scores, I can't subtract the means of the music conditions and then compare those results between drug and placebo.
Sincerely,
David
Great, I suspected that was the difference based on the areas, but the -1 contrast of Music was very similar to the 1 contrast of the interaction effect (same ROIs, similar t-scores), so I wanted to make sure.
Regarding the BCT, I've already been working with it and made several loop based scripts (for different thresholds) to generate binary matrices and run several of weighted unidirect/binary undirected metrics.
However, with regards to statistical testing, I've only found a way to compare two conditions (i.e. Drug v Placebo or Music v No music), but have been unable to determine how/if to test for an interaction effect using all four conditions. Unlike regular mean scores, I can't subtract the means of the music conditions and then compare those results between drug and placebo.
Sincerely,
David
Aug 11, 2017 03:08 PM | David de Wide
RE: Design matrix and contrast for 2x2 design
Hey Andrew,
Thank you again for all your help. The NBS is a great tool and has been very useful so far. I've been working hard at applying the NBS and the BCT over the past week and have made quite a bit of progress. However, I do have some remaining quesstion that I haven't been able to find an answer to.
1) After reading several of your articles using the NBS and BCT, I decided to apply partial correlation to my data (similar to the ADHD study) instead of full correlation (similar to the schizoprenia study). However, this changed many of the implicated ROIs dramatically. I am considering presenting both, but am unsure of the merit of partial correlation in this case, as the results are now less intuitive. Which form of correlation do you think is most appropriate? (I also looked into Sparse Partial Correlation, but could not find a matlab plugin).
2) Using the NBS with all 48 scans, and looking at the contrast of the drug effect, I am effectively using both the music and non-music scan together (which themselves are different). Would it be "cleaner" to load a seperate design and dataset of 24 scans and look at the effect of drug seperately for both music and no-music? Or is there an additional contrast I can put in to control for the effect of music on drug (and vice versa)?
3) Before the NBS, I was only aware of regular methods of FDR correction. Using the NBS I initially found significant edges using FDR, despite not finding any when using traditional FDR methods on my 90x90 ANOVA table. I later read about the high number of permutations required, and although this removed all FDR finding, 50.000 is taking a very long time. Is there a lower number that would still be considered "safe"? How do I refer to this method of FDR in my article (i.e. permutation based FDR)?
4) Is there an overview somewhere of which BCT metrics require values to be postive/unsigned?
5) Using Graphvar/GTG, I've been able to do some statistical testing on the network metrics. However, these only allow for a single factor repeated measures, so I still can't look at the contrast of a 2x2 design. Would it be a good to compare each pair seperately and plot all 4 in the same graph?
Thank you for your willingness to answer questions. It's been a massive help, as neither of my supervisors (nor anyone on my floor) has any experience using graph theory based metrics.
Sincerely,
David
Thank you again for all your help. The NBS is a great tool and has been very useful so far. I've been working hard at applying the NBS and the BCT over the past week and have made quite a bit of progress. However, I do have some remaining quesstion that I haven't been able to find an answer to.
1) After reading several of your articles using the NBS and BCT, I decided to apply partial correlation to my data (similar to the ADHD study) instead of full correlation (similar to the schizoprenia study). However, this changed many of the implicated ROIs dramatically. I am considering presenting both, but am unsure of the merit of partial correlation in this case, as the results are now less intuitive. Which form of correlation do you think is most appropriate? (I also looked into Sparse Partial Correlation, but could not find a matlab plugin).
2) Using the NBS with all 48 scans, and looking at the contrast of the drug effect, I am effectively using both the music and non-music scan together (which themselves are different). Would it be "cleaner" to load a seperate design and dataset of 24 scans and look at the effect of drug seperately for both music and no-music? Or is there an additional contrast I can put in to control for the effect of music on drug (and vice versa)?
3) Before the NBS, I was only aware of regular methods of FDR correction. Using the NBS I initially found significant edges using FDR, despite not finding any when using traditional FDR methods on my 90x90 ANOVA table. I later read about the high number of permutations required, and although this removed all FDR finding, 50.000 is taking a very long time. Is there a lower number that would still be considered "safe"? How do I refer to this method of FDR in my article (i.e. permutation based FDR)?
4) Is there an overview somewhere of which BCT metrics require values to be postive/unsigned?
5) Using Graphvar/GTG, I've been able to do some statistical testing on the network metrics. However, these only allow for a single factor repeated measures, so I still can't look at the contrast of a 2x2 design. Would it be a good to compare each pair seperately and plot all 4 in the same graph?
Thank you for your willingness to answer questions. It's been a massive help, as neither of my supervisors (nor anyone on my floor) has any experience using graph theory based metrics.
Sincerely,
David
Aug 12, 2017 01:08 AM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
Hi David,
1. The choice between full, partial and sparse correlation is difficult. Pros and cons associated with each method. I recommend full correlation in the first instance, but some people in the field would disagree.
2. If you are simply interested in the effect of drug versus no drug, you should delete the music column and the interaction column and rerun the model. You will still need a column for each subject to model each subject's mean. I don't think you should throw away the music scans for this analysis.
3. No. You MUST run at least 50000 - 100000 permutations with FDR. The results will be highly unreliable otherwise. Refer to this as FDR with p-values computed using permutation testing.
4. Generally, the name of the function provides clues about whether the measure is suitable for signed networks. For example, "bu" at the end of the function name means "binary undirected". See BCT webpage for more details.
5. Your design is repeated in both factors. Therefore a 2 x 2 with only one repeated measure is not appropriate. You could import your data into SPSS and perform testing there.
Andrew
Originally posted by David de Wide:
1. The choice between full, partial and sparse correlation is difficult. Pros and cons associated with each method. I recommend full correlation in the first instance, but some people in the field would disagree.
2. If you are simply interested in the effect of drug versus no drug, you should delete the music column and the interaction column and rerun the model. You will still need a column for each subject to model each subject's mean. I don't think you should throw away the music scans for this analysis.
3. No. You MUST run at least 50000 - 100000 permutations with FDR. The results will be highly unreliable otherwise. Refer to this as FDR with p-values computed using permutation testing.
4. Generally, the name of the function provides clues about whether the measure is suitable for signed networks. For example, "bu" at the end of the function name means "binary undirected". See BCT webpage for more details.
5. Your design is repeated in both factors. Therefore a 2 x 2 with only one repeated measure is not appropriate. You could import your data into SPSS and perform testing there.
Andrew
Originally posted by David de Wide:
Hey Andrew,
Thank you again for all your help. The NBS is a great tool and has been very useful so far. I've been working hard at applying the NBS and the BCT over the past week and have made quite a bit of progress. However, I do have some remaining quesstion that I haven't been able to find an answer to.
1) After reading several of your articles using the NBS and BCT, I decided to apply partial correlation to my data (similar to the ADHD study) instead of full correlation (similar to the schizoprenia study). However, this changed many of the implicated ROIs dramatically. I am considering presenting both, but am unsure of the merit of partial correlation in this case, as the results are now less intuitive. Which form of correlation do you think is most appropriate? (I also looked into Sparse Partial Correlation, but could not find a matlab plugin).
2) Using the NBS with all 48 scans, and looking at the contrast of the drug effect, I am effectively using both the music and non-music scan together (which themselves are different). Would it be "cleaner" to load a seperate design and dataset of 24 scans and look at the effect of drug seperately for both music and no-music? Or is there an additional contrast I can put in to control for the effect of music on drug (and vice versa)?
3) Before the NBS, I was only aware of regular methods of FDR correction. Using the NBS I initially found significant edges using FDR, despite not finding any when using traditional FDR methods on my 90x90 ANOVA table. I later read about the high number of permutations required, and although this removed all FDR finding, 50.000 is taking a very long time. Is there a lower number that would still be considered "safe"? How do I refer to this method of FDR in my article (i.e. permutation based FDR)?
4) Is there an overview somewhere of which BCT metrics require values to be postive/unsigned?
5) Using Graphvar/GTG, I've been able to do some statistical testing on the network metrics. However, these only allow for a single factor repeated measures, so I still can't look at the contrast of a 2x2 design. Would it be a good to compare each pair seperately and plot all 4 in the same graph?
Thank you for your willingness to answer questions. It's been a massive help, as neither of my supervisors (nor anyone on my floor) has any experience using graph theory based metrics.
Sincerely,
David
Thank you again for all your help. The NBS is a great tool and has been very useful so far. I've been working hard at applying the NBS and the BCT over the past week and have made quite a bit of progress. However, I do have some remaining quesstion that I haven't been able to find an answer to.
1) After reading several of your articles using the NBS and BCT, I decided to apply partial correlation to my data (similar to the ADHD study) instead of full correlation (similar to the schizoprenia study). However, this changed many of the implicated ROIs dramatically. I am considering presenting both, but am unsure of the merit of partial correlation in this case, as the results are now less intuitive. Which form of correlation do you think is most appropriate? (I also looked into Sparse Partial Correlation, but could not find a matlab plugin).
2) Using the NBS with all 48 scans, and looking at the contrast of the drug effect, I am effectively using both the music and non-music scan together (which themselves are different). Would it be "cleaner" to load a seperate design and dataset of 24 scans and look at the effect of drug seperately for both music and no-music? Or is there an additional contrast I can put in to control for the effect of music on drug (and vice versa)?
3) Before the NBS, I was only aware of regular methods of FDR correction. Using the NBS I initially found significant edges using FDR, despite not finding any when using traditional FDR methods on my 90x90 ANOVA table. I later read about the high number of permutations required, and although this removed all FDR finding, 50.000 is taking a very long time. Is there a lower number that would still be considered "safe"? How do I refer to this method of FDR in my article (i.e. permutation based FDR)?
4) Is there an overview somewhere of which BCT metrics require values to be postive/unsigned?
5) Using Graphvar/GTG, I've been able to do some statistical testing on the network metrics. However, these only allow for a single factor repeated measures, so I still can't look at the contrast of a 2x2 design. Would it be a good to compare each pair seperately and plot all 4 in the same graph?
Thank you for your willingness to answer questions. It's been a massive help, as neither of my supervisors (nor anyone on my floor) has any experience using graph theory based metrics.
Sincerely,
David
Aug 18, 2017 03:08 PM | David de Wide
RE: Design matrix and contrast for 2x2 design
Hey Andrew,
I've been working hard over the past week on both the NBS and the BCT. I have a few final questions that I hope you would be able to help me with. You've been extremely gratious and forhtcoming with your expertise, and I honestly could not appericiate it more. I tried getting most of these answers from the available pages of your recent book, but as an unpaid intern living abroad I can't afford it at the moment.
1) So if I understand you correctly, only the functions that include "signed networks" as an option can deal with negative values? (i.e. local_assortativity_wu_sign.m or clustering_coef_wu_sign.m). For the sake of simplicitiy, I feel it is probably wise to stick to binary undirected values. I've given up on all but the original BCT and am just writing scripts instead of a UI. In this case, should I apply some form of normalization on the thresholded matrices before computing/plotting the metrics?
2) I've seen different approaches to this in different articles, but should I pick a single or small range of thresholds (i.e. 30% or 20-30%, as these values are more often significant than the 30-50% range), or plot changes from density 1 to 50 (relative)?
3) Follow-up question 2. Should I enter the individual values (1 per subjects so 12 per theshold) into a repeated measures anova with 2x2(x50 in case of 50 density levels), or should I run paired t-tests on each of the pairs (resulting in 50 t-values, one for each density level) and create contrasts by subtracting the measures for music from the no-music scan for each drug, and then a paired t-test between the 2 contrasts? Then conduct FDR on the pvalues?
4) Do I need to control for multiple comparisons for the NBS? Since the effects of music on functional connectivity seem to be diametrically opposed for (Drug reduces FCD in certain ROIs, which is increased by music, while placebo has higher FCD but is decreased by music) I've looked at both main effects seperately, as well as in the full model. This leaves me with 6 outcomes for the full model (Drug connectivity increase (+) and decrease (-), Music + and -, Interaction + and -), and 8 additional outcomes (Drug effect no-music + and -, Drug effect music + and -, Music effect placebo + and -, and Music effect drug + and -). For each of these outcomes (8 out of 14 have significant edges) I also ran several thesholds from ~3 to ~4.5 to funnel down the number of edges to the most influential.
5) How should I interpret negative interaction effects? Based on my design matrix(PCB NM, PCB M, Drug NM, Drug M), is the contrast DrugvPCB or PCBvDrug and similar for Music V No music? When calculating t-tests based on mean functional connectivity (mean for each colmun in the 90x90 matrix), I subtracted Music from No-Music and Drug from Placebo. I'm unsure if the NBS uses the same logic in the contrasts.
6) Most articles seem to only mention the benefit of partial correlation (models direct connection by reducing suprious/unrelated connectivity between the remaining pairs), but what exactly is the downside of this? I've decided to follow your advice and stick to full correlation, but not entirely sure how to defend that decision.
7) Are the p-value for the network and t the t-values for the significant edges saved anywhere in the NBS.out output file? I've saved the NBS file and binary matrix for each analysis but I can only find the alpha level I selected in the UI, but not the lowest p-value that was found for the network). I'm afraid I'll have to run the analyses again and write down each p-value as well as the t-values between the signfiicant edges.
Hopefully these will be my final questions. If you're interested in the results, I can probably send you my internship report once it is finished. Thanks again for your help.
Kind regards,
David
I've been working hard over the past week on both the NBS and the BCT. I have a few final questions that I hope you would be able to help me with. You've been extremely gratious and forhtcoming with your expertise, and I honestly could not appericiate it more. I tried getting most of these answers from the available pages of your recent book, but as an unpaid intern living abroad I can't afford it at the moment.
1) So if I understand you correctly, only the functions that include "signed networks" as an option can deal with negative values? (i.e. local_assortativity_wu_sign.m or clustering_coef_wu_sign.m). For the sake of simplicitiy, I feel it is probably wise to stick to binary undirected values. I've given up on all but the original BCT and am just writing scripts instead of a UI. In this case, should I apply some form of normalization on the thresholded matrices before computing/plotting the metrics?
2) I've seen different approaches to this in different articles, but should I pick a single or small range of thresholds (i.e. 30% or 20-30%, as these values are more often significant than the 30-50% range), or plot changes from density 1 to 50 (relative)?
3) Follow-up question 2. Should I enter the individual values (1 per subjects so 12 per theshold) into a repeated measures anova with 2x2(x50 in case of 50 density levels), or should I run paired t-tests on each of the pairs (resulting in 50 t-values, one for each density level) and create contrasts by subtracting the measures for music from the no-music scan for each drug, and then a paired t-test between the 2 contrasts? Then conduct FDR on the pvalues?
4) Do I need to control for multiple comparisons for the NBS? Since the effects of music on functional connectivity seem to be diametrically opposed for (Drug reduces FCD in certain ROIs, which is increased by music, while placebo has higher FCD but is decreased by music) I've looked at both main effects seperately, as well as in the full model. This leaves me with 6 outcomes for the full model (Drug connectivity increase (+) and decrease (-), Music + and -, Interaction + and -), and 8 additional outcomes (Drug effect no-music + and -, Drug effect music + and -, Music effect placebo + and -, and Music effect drug + and -). For each of these outcomes (8 out of 14 have significant edges) I also ran several thesholds from ~3 to ~4.5 to funnel down the number of edges to the most influential.
5) How should I interpret negative interaction effects? Based on my design matrix(PCB NM, PCB M, Drug NM, Drug M), is the contrast DrugvPCB or PCBvDrug and similar for Music V No music? When calculating t-tests based on mean functional connectivity (mean for each colmun in the 90x90 matrix), I subtracted Music from No-Music and Drug from Placebo. I'm unsure if the NBS uses the same logic in the contrasts.
6) Most articles seem to only mention the benefit of partial correlation (models direct connection by reducing suprious/unrelated connectivity between the remaining pairs), but what exactly is the downside of this? I've decided to follow your advice and stick to full correlation, but not entirely sure how to defend that decision.
7) Are the p-value for the network and t the t-values for the significant edges saved anywhere in the NBS.out output file? I've saved the NBS file and binary matrix for each analysis but I can only find the alpha level I selected in the UI, but not the lowest p-value that was found for the network). I'm afraid I'll have to run the analyses again and write down each p-value as well as the t-values between the signfiicant edges.
Hopefully these will be my final questions. If you're interested in the results, I can probably send you my internship report once it is finished. Thanks again for your help.
Kind regards,
David
Aug 19, 2017 12:08 AM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
Hi David,
many questions here so I will keep my response brief. You might want to consider visiting a lab/collaborator with expertise in graph analysis.
1. Yes - "signed" means "can deal with negative edge weights". Many different methods to normalize network metrics (e.g. with respect to a degree-matched random graph).
2. Perhaps 5% - 40%. You need to justify what is a reasonable range for your data. You need to decide and carefully justify based on data.
3. Best not to run a test at every threshold. Consider computing the area under the curve (AUC) and perform only on 2 x 2 anova on the AUC which is essentially a summary measure across all thresholds. Take a look at previous graph papers using AUC.
4. I assume you mean multiple comparisons across the different contrasts. Technically you should control for multiple comparisons across independent contrasts but many don't bother.
5. Interactions in NBS are interpreted EXACTLY the same as a typical ANOVA interaction.
6. Partial correlation can underestimate true clustering of network.
7. Yes - all these variables are all stored in the output matrix. Take a look at the reference manual for details.
Andrew
Originally posted by David de Wide:
many questions here so I will keep my response brief. You might want to consider visiting a lab/collaborator with expertise in graph analysis.
1. Yes - "signed" means "can deal with negative edge weights". Many different methods to normalize network metrics (e.g. with respect to a degree-matched random graph).
2. Perhaps 5% - 40%. You need to justify what is a reasonable range for your data. You need to decide and carefully justify based on data.
3. Best not to run a test at every threshold. Consider computing the area under the curve (AUC) and perform only on 2 x 2 anova on the AUC which is essentially a summary measure across all thresholds. Take a look at previous graph papers using AUC.
4. I assume you mean multiple comparisons across the different contrasts. Technically you should control for multiple comparisons across independent contrasts but many don't bother.
5. Interactions in NBS are interpreted EXACTLY the same as a typical ANOVA interaction.
6. Partial correlation can underestimate true clustering of network.
7. Yes - all these variables are all stored in the output matrix. Take a look at the reference manual for details.
Andrew
Originally posted by David de Wide:
Hey Andrew,
I've been working hard over the past week on both the NBS and the BCT. I have a few final questions that I hope you would be able to help me with. You've been extremely gratious and forhtcoming with your expertise, and I honestly could not appericiate it more. I tried getting most of these answers from the available pages of your recent book, but as an unpaid intern living abroad I can't afford it at the moment.
1) So if I understand you correctly, only the functions that include "signed networks" as an option can deal with negative values? (i.e. local_assortativity_wu_sign.m or clustering_coef_wu_sign.m). For the sake of simplicitiy, I feel it is probably wise to stick to binary undirected values. I've given up on all but the original BCT and am just writing scripts instead of a UI. In this case, should I apply some form of normalization on the thresholded matrices before computing/plotting the metrics?
2) I've seen different approaches to this in different articles, but should I pick a single or small range of thresholds (i.e. 30% or 20-30%, as these values are more often significant than the 30-50% range), or plot changes from density 1 to 50 (relative)?
3) Follow-up question 2. Should I enter the individual values (1 per subjects so 12 per theshold) into a repeated measures anova with 2x2(x50 in case of 50 density levels), or should I run paired t-tests on each of the pairs (resulting in 50 t-values, one for each density level) and create contrasts by subtracting the measures for music from the no-music scan for each drug, and then a paired t-test between the 2 contrasts? Then conduct FDR on the pvalues?
4) Do I need to control for multiple comparisons for the NBS? Since the effects of music on functional connectivity seem to be diametrically opposed for (Drug reduces FCD in certain ROIs, which is increased by music, while placebo has higher FCD but is decreased by music) I've looked at both main effects seperately, as well as in the full model. This leaves me with 6 outcomes for the full model (Drug connectivity increase (+) and decrease (-), Music + and -, Interaction + and -), and 8 additional outcomes (Drug effect no-music + and -, Drug effect music + and -, Music effect placebo + and -, and Music effect drug + and -). For each of these outcomes (8 out of 14 have significant edges) I also ran several thesholds from ~3 to ~4.5 to funnel down the number of edges to the most influential.
5) How should I interpret negative interaction effects? Based on my design matrix(PCB NM, PCB M, Drug NM, Drug M), is the contrast DrugvPCB or PCBvDrug and similar for Music V No music? When calculating t-tests based on mean functional connectivity (mean for each colmun in the 90x90 matrix), I subtracted Music from No-Music and Drug from Placebo. I'm unsure if the NBS uses the same logic in the contrasts.
6) Most articles seem to only mention the benefit of partial correlation (models direct connection by reducing suprious/unrelated connectivity between the remaining pairs), but what exactly is the downside of this? I've decided to follow your advice and stick to full correlation, but not entirely sure how to defend that decision.
7) Are the p-value for the network and t the t-values for the significant edges saved anywhere in the NBS.out output file? I've saved the NBS file and binary matrix for each analysis but I can only find the alpha level I selected in the UI, but not the lowest p-value that was found for the network). I'm afraid I'll have to run the analyses again and write down each p-value as well as the t-values between the signfiicant edges.
Hopefully these will be my final questions. If you're interested in the results, I can probably send you my internship report once it is finished. Thanks again for your help.
Kind regards,
David
I've been working hard over the past week on both the NBS and the BCT. I have a few final questions that I hope you would be able to help me with. You've been extremely gratious and forhtcoming with your expertise, and I honestly could not appericiate it more. I tried getting most of these answers from the available pages of your recent book, but as an unpaid intern living abroad I can't afford it at the moment.
1) So if I understand you correctly, only the functions that include "signed networks" as an option can deal with negative values? (i.e. local_assortativity_wu_sign.m or clustering_coef_wu_sign.m). For the sake of simplicitiy, I feel it is probably wise to stick to binary undirected values. I've given up on all but the original BCT and am just writing scripts instead of a UI. In this case, should I apply some form of normalization on the thresholded matrices before computing/plotting the metrics?
2) I've seen different approaches to this in different articles, but should I pick a single or small range of thresholds (i.e. 30% or 20-30%, as these values are more often significant than the 30-50% range), or plot changes from density 1 to 50 (relative)?
3) Follow-up question 2. Should I enter the individual values (1 per subjects so 12 per theshold) into a repeated measures anova with 2x2(x50 in case of 50 density levels), or should I run paired t-tests on each of the pairs (resulting in 50 t-values, one for each density level) and create contrasts by subtracting the measures for music from the no-music scan for each drug, and then a paired t-test between the 2 contrasts? Then conduct FDR on the pvalues?
4) Do I need to control for multiple comparisons for the NBS? Since the effects of music on functional connectivity seem to be diametrically opposed for (Drug reduces FCD in certain ROIs, which is increased by music, while placebo has higher FCD but is decreased by music) I've looked at both main effects seperately, as well as in the full model. This leaves me with 6 outcomes for the full model (Drug connectivity increase (+) and decrease (-), Music + and -, Interaction + and -), and 8 additional outcomes (Drug effect no-music + and -, Drug effect music + and -, Music effect placebo + and -, and Music effect drug + and -). For each of these outcomes (8 out of 14 have significant edges) I also ran several thesholds from ~3 to ~4.5 to funnel down the number of edges to the most influential.
5) How should I interpret negative interaction effects? Based on my design matrix(PCB NM, PCB M, Drug NM, Drug M), is the contrast DrugvPCB or PCBvDrug and similar for Music V No music? When calculating t-tests based on mean functional connectivity (mean for each colmun in the 90x90 matrix), I subtracted Music from No-Music and Drug from Placebo. I'm unsure if the NBS uses the same logic in the contrasts.
6) Most articles seem to only mention the benefit of partial correlation (models direct connection by reducing suprious/unrelated connectivity between the remaining pairs), but what exactly is the downside of this? I've decided to follow your advice and stick to full correlation, but not entirely sure how to defend that decision.
7) Are the p-value for the network and t the t-values for the significant edges saved anywhere in the NBS.out output file? I've saved the NBS file and binary matrix for each analysis but I can only find the alpha level I selected in the UI, but not the lowest p-value that was found for the network). I'm afraid I'll have to run the analyses again and write down each p-value as well as the t-values between the signfiicant edges.
Hopefully these will be my final questions. If you're interested in the results, I can probably send you my internship report once it is finished. Thanks again for your help.
Kind regards,
David
Sep 27, 2019 10:09 AM | Liam Nestor - Imperial College London
RE: Design matrix and contrast for 2x2 design
Dear Andrew
I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups (controls -n=36; patients - n=57) who had two randomised drug visits (placebo; drug). Should I simply follow, and generalise, the information you provided to David? With thanks, Liam.
I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups (controls -n=36; patients - n=57) who had two randomised drug visits (placebo; drug). Should I simply follow, and generalise, the information you provided to David? With thanks, Liam.
Sep 28, 2019 01:09 AM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
Yes - the advice still holds.
Originally posted by Liam Nestor:
Originally posted by Liam Nestor:
Dear Andrew
I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups (controls -n=36; patients - n=57) who had two randomised drug visits (placebo; drug). Should I simply follow, and generalise, the information you provided to David? With thanks, Liam.
I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups (controls -n=36; patients - n=57) who had two randomised drug visits (placebo; drug). Should I simply follow, and generalise, the information you provided to David? With thanks, Liam.
Sep 28, 2019 05:09 AM | Liam Nestor - Imperial College London
RE: Design matrix and contrast for 2x2 design
Just to clarify - can I have both groups in the same analysis even
though one has 36 subjects and the other has 57? It's not going to
matter with the exchange blocks? Best wishes, Liam.
Sep 28, 2019 11:09 PM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
A mismatch in the sample size should
not be a problem.
Originally posted by Liam Nestor:
Originally posted by Liam Nestor:
Just to clarify - can I have both groups in the
same analysis even though one has 36 subjects and the other has 57?
It's not going to matter with the exchange blocks? Best wishes,
Liam.
Sep 30, 2019 02:09 PM | Liam Nestor - Imperial College London
RE: Design matrix and contrast for 2x2 design
Thanks Andrew
I am following the advice previously given. It's getting everything right, except it's telling me I don't have any exchange blocks, even though I loading an exchange blocks text file (attached) - am I populating the text file properly? Best, Liam.
I am following the advice previously given. It's getting everything right, except it's telling me I don't have any exchange blocks, even though I loading an exchange blocks text file (attached) - am I populating the text file properly? Best, Liam.
Oct 1, 2019 03:10 AM | Andrew Zalesky
RE: Design matrix and contrast for 2x2 design
Hi Liam,
the exchange block file seems to look ok. Perhaps try directly pasting the numbers into the GUI. Note that the software should run without a exchange blocks in any case.
Andrew
Originally posted by Liam Nestor:
the exchange block file seems to look ok. Perhaps try directly pasting the numbers into the GUI. Note that the software should run without a exchange blocks in any case.
Andrew
Originally posted by Liam Nestor:
Thanks Andrew
I am following the advice previously given. It's getting everything right, except it's telling me I don't have any exchange blocks, even though I loading an exchange blocks text file (attached) - am I populating the text file properly? Best, Liam.
I am following the advice previously given. It's getting everything right, except it's telling me I don't have any exchange blocks, even though I loading an exchange blocks text file (attached) - am I populating the text file properly? Best, Liam.
Oct 1, 2019 01:10 PM | Liam Nestor - Imperial College London
RE: Design matrix and contrast for 2x2 design
Thanks Andrew – pasting them in with a semicolon between each
numbers works for the exchange blocks. Best wishes, Liam.
Jan 11, 2020 09:01 AM | yue zhang
RE: Design matrix and contrast for 2x2 design
Dear Andrew
I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups in patients with migraine(Group A:n=29; Group B:n=25)who received two therapy method,separately.(Group A: received real vagus nerve stimulation; Group B received sham Vagus nerve stimulation). so I had four groups(Group A1:before real stimulation,Group A2: after real stimulation, Group B1: before sham stimulation,Group B2: after sham stimulation ).
I am interested in the main effects of stimulation and time,but more specifically in the interaction effect between stimulation and time. My main issue has been the design matrix and the contrast. Can you help me? Any help would be greatly appericiated.
Sincerely,
zhangyue
I am also hoping to perform a similar 2x2 repeated measures anova in NBS. I have two groups in patients with migraine(Group A:n=29; Group B:n=25)who received two therapy method,separately.(Group A: received real vagus nerve stimulation; Group B received sham Vagus nerve stimulation). so I had four groups(Group A1:before real stimulation,Group A2: after real stimulation, Group B1: before sham stimulation,Group B2: after sham stimulation ).
I am interested in the main effects of stimulation and time,but more specifically in the interaction effect between stimulation and time. My main issue has been the design matrix and the contrast. Can you help me? Any help would be greatly appericiated.
Sincerely,
zhangyue