help
help > RE: Setup taking a very long time
Aug 27, 2016 01:08 PM | Caitlin Duncan
RE: Setup taking a very long time
Hello,
I have a follow-up question to this. I am only processing 25 subjects, and I began running the setup ~48 hours ago and it's only 35% done with ROI extraction. I knew the processing would take time, but this seems excessive. Everything is stored locally. Is this normal?
Thanks!
Best,
Caitlin
Originally posted by Alfonso Nieto-Castanon:
I have a follow-up question to this. I am only processing 25 subjects, and I began running the setup ~48 hours ago and it's only 35% done with ROI extraction. I knew the processing would take time, but this seems excessive. Everything is stored locally. Is this normal?
Thanks!
Best,
Caitlin
Originally posted by Alfonso Nieto-Castanon:
Hi Ami,
Yes, the ROI extraction step can be particularly slow when the data is stored remotely. For relatively large datasets (e.g. hundreds of subjects) stored on a remote server I would strongly recommend using the parallelization options in CONN to speed-up your analyses (see if you institution offers cluster computing resources, if so you probably only need to know the scheduler used -e.g. Grid Engine, PBS, etc.- to use CONN's parallelization options). That can naturally speed up your analyses by several orders of magnitude. If that is not possible, then yes, moving your raw data to a local drive will still speed up all of your analyses considerably, so that is your next best choice. If your raw data is stored remotely, but your conn project is been saved to a local storage, then the Preprocessing and Setup steps will still take a relatively long time (since those steps use the raw data -remotely stored), but the subsequent Denoising/first-level/second-level analyses will be faster (since those steps no longer require the original raw data).
Hope this helps
Alfonso
Originally posted by Ami Tsuchida:
Yes, the ROI extraction step can be particularly slow when the data is stored remotely. For relatively large datasets (e.g. hundreds of subjects) stored on a remote server I would strongly recommend using the parallelization options in CONN to speed-up your analyses (see if you institution offers cluster computing resources, if so you probably only need to know the scheduler used -e.g. Grid Engine, PBS, etc.- to use CONN's parallelization options). That can naturally speed up your analyses by several orders of magnitude. If that is not possible, then yes, moving your raw data to a local drive will still speed up all of your analyses considerably, so that is your next best choice. If your raw data is stored remotely, but your conn project is been saved to a local storage, then the Preprocessing and Setup steps will still take a relatively long time (since those steps use the raw data -remotely stored), but the subsequent Denoising/first-level/second-level analyses will be faster (since those steps no longer require the original raw data).
Hope this helps
Alfonso
Originally posted by Ami Tsuchida:
Hello,
I finished setting up functional/structural/ROI files for 200+ subjects, and pressed done in the Setup page of the GUI. It seems to be running without an error, but it's taking a very long time. In the data folder I see that it has created DATA_Subject...mat for each subject, as well as COV_Subject... and COND_Subject...mat files. It seems to be in the process of generating ROI_Subject...mat, but this is taking roughly 1 hour per subject. Is this normal?
I'm guessing that this is the step where it is extracting the time series from each ROI, but I selected average ts rather than PCA, so I'm not sure what's taking so long. I provided subject-specific gm/wm/csf masks previously processed with spm, as well as custom ROIs containing 384 labels.
Elsewhere in the forum I saw that it's faster to write files locally, so I have project folder setup locally, and all the mat files are saved there. However, the raw functional/structural/ROI files are on a different server (or links to the files on a mounted server). If that's causing it to slow things down, do you recommend copying all the raw data? I wanted to avoid it if it's not necessary.
Please let me know if there is a quicker way to process them.
Thank you,
Ami
I finished setting up functional/structural/ROI files for 200+ subjects, and pressed done in the Setup page of the GUI. It seems to be running without an error, but it's taking a very long time. In the data folder I see that it has created DATA_Subject...mat for each subject, as well as COV_Subject... and COND_Subject...mat files. It seems to be in the process of generating ROI_Subject...mat, but this is taking roughly 1 hour per subject. Is this normal?
I'm guessing that this is the step where it is extracting the time series from each ROI, but I selected average ts rather than PCA, so I'm not sure what's taking so long. I provided subject-specific gm/wm/csf masks previously processed with spm, as well as custom ROIs containing 384 labels.
Elsewhere in the forum I saw that it's faster to write files locally, so I have project folder setup locally, and all the mat files are saved there. However, the raw functional/structural/ROI files are on a different server (or links to the files on a mounted server). If that's causing it to slow things down, do you recommend copying all the raw data? I wanted to avoid it if it's not necessary.
Please let me know if there is a quicker way to process them.
Thank you,
Ami
Threaded View
| Title | Author | Date |
|---|---|---|
| Ami Tsuchida | Jul 5, 2016 | |
| Alfonso Nieto-Castanon | Jul 6, 2016 | |
| Caitlin Duncan | Aug 27, 2016 | |
| Alfonso Nieto-Castanon | Aug 30, 2016 | |
| Ami Tsuchida | Jul 12, 2016 | |
| Ami Tsuchida | Jul 7, 2016 | |