help > 122 voxelwise (8k) CMs - memory problem
Showing 1-5 of 5 posts
Display:
Results per page:
Jul 22, 2015  10:07 AM | Loreen Mamerow
122 voxelwise (8k) CMs - memory problem
Dear NBS users,

I have started using the NBS toolbox and am finding it incredibly useful for analysing structural connectivity. So far, I used the AAL parcellation on 112 subjects, and everything ran smoothly. I have now however tried to run similar analyses using voxelwise parcellation that yielded 8057x8057 CMs for each of the 112 subjects. Regardless of whether I load a 3D .mat file (subjects=3rd dimension, ~330MB) or whether I load 112 .txt files (folder = 116GB), I get the error message "Your system has run out of application memory".

I tried various options, including running the analysis from an otherwise empty 1TB external hard drive (i.e., what I presumed is more than enough room to accommodate whatever space is required during processing and write out), but the error remains and the analysis does not run. From what I can gather with respect to my memory specifications, there should be enough free space to carry out the computations. 

Would you have any idea how much memory I need to run this analysis and whether there are settings in matlab (I am running version R2014b on OSX 10.9.5) that can aid the process? Or whether this might be something else entirely?

Any advice and comments would be greatly appreciated!

Thank you & regards, Loreen
Jul 23, 2015  06:07 AM | Andrew Zalesky
RE: 122 voxelwise (8k) CMs - memory problem
Hi Loreen,

The memory error is most likely owing to a lack of RAM on your computer system, rather than a lack of memory on your hard drive. Storing an manipulating 116 GB of connectivity matrices is going to be very computationally burdensome. Performing voxel-resolution analysis may require custom scripts and parallel processing.

I suggest that you consider using a computer system with more RAM, or alternatively, using a parcellation that does not comprise as many nodes. There are many parcellations available that are of a higher resolution than the AAL but are not as extreme as a voxel level parcellation.

I hope that helps,

Andrew

Originally posted by Loreen Mamerow:
Dear NBS users,

I have started using the NBS toolbox and am finding it incredibly useful for analysing structural connectivity. So far, I used the AAL parcellation on 112 subjects, and everything ran smoothly. I have now however tried to run similar analyses using voxelwise parcellation that yielded 8057x8057 CMs for each of the 112 subjects. Regardless of whether I load a 3D .mat file (subjects=3rd dimension, ~330MB) or whether I load 112 .txt files (folder = 116GB), I get the error message "Your system has run out of application memory".

I tried various options, including running the analysis from an otherwise empty 1TB external hard drive (i.e., what I presumed is more than enough room to accommodate whatever space is required during processing and write out), but the error remains and the analysis does not run. From what I can gather with respect to my memory specifications, there should be enough free space to carry out the computations. 

Would you have any idea how much memory I need to run this analysis and whether there are settings in matlab (I am running version R2014b on OSX 10.9.5) that can aid the process? Or whether this might be something else entirely?

Any advice and comments would be greatly appreciated!

Thank you & regards, Loreen
Jul 23, 2015  07:07 AM | Loreen Mamerow
RE: 122 voxelwise (8k) CMs - memory problem
Dear Andrew,

Many thanks for your response, much appreciated!

So, I take it that, in principle, the scripts implemented in NBS do not struggle with the size of the file (the 330MB 3D .mat file also causes the error), but rather that I need to see whether I can adapt my system if I want to proceed with a higher resolution parcellation than AAL?

Thanks again!

Regards, Loreen


Originally posted by Andrew Zalesky:
Hi Loreen,

The memory error is most likely owing to a lack of RAM on your computer system, rather than a lack of memory on your hard drive. Storing an manipulating 116 GB of connectivity matrices is going to be very computationally burdensome. Performing voxel-resolution analysis may require custom scripts and parallel processing.

I suggest that you consider using a computer system with more RAM, or alternatively, using a parcellation that does not comprise as many nodes. There are many parcellations available that are of a higher resolution than the AAL but are not as extreme as a voxel level parcellation.

I hope that helps,

Andrew

Originally posted by Loreen Mamerow:
Dear NBS users,

I have started using the NBS toolbox and am finding it incredibly useful for analysing structural connectivity. So far, I used the AAL parcellation on 112 subjects, and everything ran smoothly. I have now however tried to run similar analyses using voxelwise parcellation that yielded 8057x8057 CMs for each of the 112 subjects. Regardless of whether I load a 3D .mat file (subjects=3rd dimension, ~330MB) or whether I load 112 .txt files (folder = 116GB), I get the error message "Your system has run out of application memory".

I tried various options, including running the analysis from an otherwise empty 1TB external hard drive (i.e., what I presumed is more than enough room to accommodate whatever space is required during processing and write out), but the error remains and the analysis does not run. From what I can gather with respect to my memory specifications, there should be enough free space to carry out the computations. 

Would you have any idea how much memory I need to run this analysis and whether there are settings in matlab (I am running version R2014b on OSX 10.9.5) that can aid the process? Or whether this might be something else entirely?

Any advice and comments would be greatly appreciated!

Thank you & regards, Loreen
Jul 23, 2015  08:07 AM | Andrew Zalesky
RE: 122 voxelwise (8k) CMs - memory problem
Hi Loreen,

The Matlab implementation of the NBS has been extensively tested on networks up to a few thousand nodes.

Dealing with networks comprising 5000+ nodes may require parallelization of the processing pipeline.

Andrew

Originally posted by Loreen Mamerow:
Dear Andrew,

Many thanks for your response, much appreciated!

So, I take it that, in principle, the scripts implemented in NBS do not struggle with the size of the file (the 330MB 3D .mat file also causes the error), but rather that I need to see whether I can adapt my system if I want to proceed with a higher resolution parcellation than AAL?

Thanks again!

Regards, Loreen


Originally posted by Andrew Zalesky:
Hi Loreen,

The memory error is most likely owing to a lack of RAM on your computer system, rather than a lack of memory on your hard drive. Storing an manipulating 116 GB of connectivity matrices is going to be very computationally burdensome. Performing voxel-resolution analysis may require custom scripts and parallel processing.

I suggest that you consider using a computer system with more RAM, or alternatively, using a parcellation that does not comprise as many nodes. There are many parcellations available that are of a higher resolution than the AAL but are not as extreme as a voxel level parcellation.

I hope that helps,

Andrew

Originally posted by Loreen Mamerow:
Dear NBS users,

I have started using the NBS toolbox and am finding it incredibly useful for analysing structural connectivity. So far, I used the AAL parcellation on 112 subjects, and everything ran smoothly. I have now however tried to run similar analyses using voxelwise parcellation that yielded 8057x8057 CMs for each of the 112 subjects. Regardless of whether I load a 3D .mat file (subjects=3rd dimension, ~330MB) or whether I load 112 .txt files (folder = 116GB), I get the error message "Your system has run out of application memory".

I tried various options, including running the analysis from an otherwise empty 1TB external hard drive (i.e., what I presumed is more than enough room to accommodate whatever space is required during processing and write out), but the error remains and the analysis does not run. From what I can gather with respect to my memory specifications, there should be enough free space to carry out the computations. 

Would you have any idea how much memory I need to run this analysis and whether there are settings in matlab (I am running version R2014b on OSX 10.9.5) that can aid the process? Or whether this might be something else entirely?

Any advice and comments would be greatly appreciated!

Thank you & regards, Loreen
Jul 23, 2015  08:07 AM | Loreen Mamerow
RE: 122 voxelwise (8k) CMs - memory problem
Dear Andrew,

Sounds good, I'll try that! 

Thanks again & best wishes, Loreen