help
help > RE: Replication problem across different machines
May 29, 2018 05:05 PM | Alfonso Nieto-Castanon - Boston University
RE: Replication problem across different machines
Hi Audreyana,
Regarding the .mat files for the first two computers, I have not looked in enough detail but there seem to be differences in (at least) how preprocessing, denoising, and first-level analyses were run on the two computers. Regarding preprocessing, on computer1 you run two preprocessing sequences, in two consecutive days, the first being a standard MNI-space preprocessing pipeline, and the second being a similar sequence but using only session #2 data (on computer2 you seem to have run just a single standard MNI-space preprocessing pipeline). It is possible that the computer1 second preprocessing step was run only on subject #3, given the observable differences in motion parameters / etc for that subject. but unfortunately 17f did not log that info so I cannot give you more details. Then there are also differences in the Denoising step, where in computer1 you included only the 'effect of rest' regressor in the list of confounding terms while in computer2 in in addition to this you also included the two session effects ('effect of time1' and 'effect of time2'). Last there were also differences in the first-level analyses, where in computer1 you entered 164 ROIs, while in computer2 you entered 168 ROIs (you have four dmn ROIs in there that do not appear in computer1)
Regarding the document detailing the specs of each system, the differences in SPM12 releases between your machines (6685, 6906, and 7219) can potentially cause small but noticeable differences in the normalization results, so that would be something to look for. Other than that, differences in operating system, Matlab version, processor, or memory should have no impact at all in your results. I would recommend to try to characterize the earliest place where you observe differences in the results (are there any differences, for each subject, in their first-level QA measures? if not, are there differences in their extracted ROI-level BOLD timeseries? etc.) as well as whether those differences appear for individual subjects or across all subjects, and that should give you some hints as to what might have caused each of those differences.
Hope this helps
Alfonso
Originally posted by Audreyana Jagger:
Regarding the .mat files for the first two computers, I have not looked in enough detail but there seem to be differences in (at least) how preprocessing, denoising, and first-level analyses were run on the two computers. Regarding preprocessing, on computer1 you run two preprocessing sequences, in two consecutive days, the first being a standard MNI-space preprocessing pipeline, and the second being a similar sequence but using only session #2 data (on computer2 you seem to have run just a single standard MNI-space preprocessing pipeline). It is possible that the computer1 second preprocessing step was run only on subject #3, given the observable differences in motion parameters / etc for that subject. but unfortunately 17f did not log that info so I cannot give you more details. Then there are also differences in the Denoising step, where in computer1 you included only the 'effect of rest' regressor in the list of confounding terms while in computer2 in in addition to this you also included the two session effects ('effect of time1' and 'effect of time2'). Last there were also differences in the first-level analyses, where in computer1 you entered 164 ROIs, while in computer2 you entered 168 ROIs (you have four dmn ROIs in there that do not appear in computer1)
Regarding the document detailing the specs of each system, the differences in SPM12 releases between your machines (6685, 6906, and 7219) can potentially cause small but noticeable differences in the normalization results, so that would be something to look for. Other than that, differences in operating system, Matlab version, processor, or memory should have no impact at all in your results. I would recommend to try to characterize the earliest place where you observe differences in the results (are there any differences, for each subject, in their first-level QA measures? if not, are there differences in their extracted ROI-level BOLD timeseries? etc.) as well as whether those differences appear for individual subjects or across all subjects, and that should give you some hints as to what might have caused each of those differences.
Hope this helps
Alfonso
Originally posted by Audreyana Jagger:
Hello Conn Team,
Recently we computed a repeated measures ROI to ROI analysis on 4 subjects but noticed that two different computers calculated very different results. After noticing this, we checked that we used the same processing steps (default SPM preprocessing with conservatives ART settings) and the same versions of CONN. However, we still received very different results on the two machines. After discovering this issue, we calculated the same project on 3 more computers and found different results from these computers as well.
I am uncertain if this is a computer science based problem, a script problem, human error problem, or something else, but we wanted to bring this issue to your attention. I have included attachments that include the conn.mat files for the first two computers, and a document reporting the specs of each of the 5 computers.
Please let us know what we may need to do and thank you for your time.
Audreyana
Recently we computed a repeated measures ROI to ROI analysis on 4 subjects but noticed that two different computers calculated very different results. After noticing this, we checked that we used the same processing steps (default SPM preprocessing with conservatives ART settings) and the same versions of CONN. However, we still received very different results on the two machines. After discovering this issue, we calculated the same project on 3 more computers and found different results from these computers as well.
I am uncertain if this is a computer science based problem, a script problem, human error problem, or something else, but we wanted to bring this issue to your attention. I have included attachments that include the conn.mat files for the first two computers, and a document reporting the specs of each of the 5 computers.
Please let us know what we may need to do and thank you for your time.
Audreyana
Threaded View
| Title | Author | Date |
|---|---|---|
| Audreyana Jagger | May 29, 2018 | |
| Alfonso Nieto-Castanon | May 29, 2018 | |
| Audreyana Jagger | May 31, 2018 | |
| Merve Kaptan | May 29, 2018 | |
| Audreyana Jagger | May 29, 2018 | |