sdm-help-list > Questions about pre-processing report and replication concern
Showing 1-4 of 4 posts
Display:
Results per page:
May 31, 2020  12:05 PM | Yen-Wen Chen
Questions about pre-processing report and replication concern
Dear experts, 

I'm new in meta-analysis. I downloaded the latest version SDM, version 6.21.

I first practiced the execution of the software using tutorial data and everything went smoothly.

Then, I tried to apply the same parameters used from the tutorial practice to my own data (with VBM structural data).

One difference I noticed was the summary report after the pre-processing. 

At the top of the report webpage, it showed the "Peak height meta-analysis for '(all)' thresholded studies". However, with my own data, there were multiple columns with different number of thresholded studies but no column with '(all)' thresholded studies (please kindly find the attached screenshot for reference). I didn't get any warning messages from the execution log.

The "Pre-processing summary" with Maximum and Minimum peaks seemed to be roughly matched with the original studies.

I'm wondering if the reported "Peak height meta-analysis for '(xx)' thresholded studies" indicated anything about the data? (i.e., if there's something wrong caused that I didn't have the result from "meta-analysis for '(all)' thresholded studies")

Also a follow-up question, 

Since the execution was completed without errors, I went ahead with the Mean analysis (which also completed without errors) and Thresholding with p < .005 with cluster size > 10 voxels. I got NULL results for both positive and negative peaks. 

However, this null result didn't replicate a recently published meta-analysis study included the same studies using the same threshold (I intended to included the same studies with the attempt to validate my methodology). (this published study found two clusters for positive and four clusters for negative correlation)

I carefully read the methodology provided by this published article, but it was only with some general descriptions. The only difference I could find was the SDM software version, which this published study used AES-SDM (version 5.15). 

I also carefully double checked my coded peaks for each of the included studies.

I was very confused of the discrepancy of the results... I'm wondering if you can kindly give me some opinion for the possible explanations of the discrepancy. 

Any inputs will be very helpful!

Thank you so much!

Best, 
Samantha
Jun 16, 2020  12:06 PM | Yen-Wen Chen
RE: Questions about pre-processing report and replication concern
Dear experts, 

Here is a brief update about the replication issue.

After contacting the authors of the meta-analysis study, the coding files and parameters were identical between two sets of data (with the same studies included). 

However, the mean analysis results still didn't replicate the published results... (the published study had both positive and negative peaks at uncorrected p < .005, but I didn't have any peaks at the same threshold (even with more liberal p < .05).

The major difference, as I can think of, between two analyses is the SDM versions. I used the SDM-PSI (version 6.21) and the published study used AES-SDM (version 5.15).

Does it make sense that different SDM versions having inconsistent results (the difference in my case is non-negligible...)?

Any opinion or suggestion will be helpful!

Thank you so much.

Best, 
Samantha


Originally posted by Yen-Wen Chen:
Dear experts, 

I'm new in meta-analysis. I downloaded the latest version SDM, version 6.21.

I first practiced the execution of the software using tutorial data and everything went smoothly.

Then, I tried to apply the same parameters used from the tutorial practice to my own data (with VBM structural data).

One difference I noticed was the summary report after the pre-processing. 

At the top of the report webpage, it showed the "Peak height meta-analysis for '(all)' thresholded studies". However, with my own data, there were multiple columns with different number of thresholded studies but no column with '(all)' thresholded studies (please kindly find the attached screenshot for reference). I didn't get any warning messages from the execution log.

The "Pre-processing summary" with Maximum and Minimum peaks seemed to be roughly matched with the original studies.

I'm wondering if the reported "Peak height meta-analysis for '(xx)' thresholded studies" indicated anything about the data? (i.e., if there's something wrong caused that I didn't have the result from "meta-analysis for '(all)' thresholded studies")

Also a follow-up question, 

Since the execution was completed without errors, I went ahead with the Mean analysis (which also completed without errors) and Thresholding with p < .005 with cluster size > 10 voxels. I got NULL results for both positive and negative peaks. 

However, this null result didn't replicate a recently published meta-analysis study included the same studies using the same threshold (I intended to included the same studies with the attempt to validate my methodology). (this published study found two clusters for positive and four clusters for negative correlation)

I carefully read the methodology provided by this published article, but it was only with some general descriptions. The only difference I could find was the SDM software version, which this published study used AES-SDM (version 5.15). 

I also carefully double checked my coded peaks for each of the included studies.

I was very confused of the discrepancy of the results... I'm wondering if you can kindly give me some opinion for the possible explanations of the discrepancy. 

Any inputs will be very helpful!

Thank you so much!

Best, 
Samantha
Jun 18, 2020  09:06 AM | Lydia Fortea - Instituto de Investigaciones Biom├ędicas August Pi i Sunyer (IDIBAPS)
RE: Questions about pre-processing report and replication concern
Dear Samantha, 

The softwares AES-SDM and SDM-PSI could be considered as two different methods, so it is possibel they produce different results. SDM-PSI is much more robust than AES-SDM

Kind regards,
Lydia
Jun 18, 2020  09:06 AM | Yen-Wen Chen
RE: Questions about pre-processing report and replication concern
Hi Lydia, 

Thanks for the clarification! And thanks for your amazing works to update and improve the software!

Best wishes, 
Sam.