help
help > RE: Reducing Disk Storage Use
Aug 8, 2016 05:08 PM | Alfonso Nieto-Castanon - Boston University
RE: Reducing Disk Storage Use
Hi Andrew
Some general space-managing tips are the following:
1) original data folders. Any Preprocessing results (e.g. realignment, coregistration, normalization, etc.) will be stored in the same folders where the original data was stored. You may remove there any intermediate step that will no longer be needed (e.g. you may for example keep the final functional data swardata.nii, the unsmoothed functional data wardata.nii, and the original data data.nii and remove the other itermediate steps rdata.nii and ardata.nii)
2) conn_*/data folder. This contains the data imported from (1) above into CONN during the Setup step, which will be used/needed to run the Denoising step. If you are no longer planning/needing ro re-run the Denoising step you may remove the data in this folder entirely.
3) conn_*/results/preprocessing folder. This contains the data resulting from CONN's Denoising step (2) above, which will be used/needed to run any First-level analysis step. If you are no longer planning/needing to re-run any First-level analysis step you may remove the data in this folder entirely.
4) conn_*/results/firstlevel/ANALYSISNAME. This contains the data resulting from CONN's first-level analysis step (3) above, which will be used/needed to run Second-level results analyses. If you are no longer planning/needing to run any new second-leve analysis (or if you just want to remove a particular first-level analysis) you may remove this folder entirely (note: also remove first the corresponding analysis in CONN's gui).
5) conn_*/results/secondlevel/ANALYSISNAME. This contains the data resulting from CONN's second-level analysis step (4) above, which will be used for displaying your second-level results (e.g. using "results explorer" guis). If you have analyses that are no longer needed/used you may simply remove the corresponding folders.
Another general recommendation to minimize used storage would be, for seed-to-voxel first-level analyses, to only enter there the seeds that you are planning to use (by default all ROIs will be entered there, which creates a lot of seed-to-voxel maps -one per subject/condition/seed combination-). A "space-consciuos" user would create two first-level analyses: one where only "ROI-to-ROI" analyses are run, and where all of the ROIs are entered there for good measure; and a second first-level analysis where only "seed-to-voxel" analyses are run, and where only a few selected ROIs/seeds of interest are entered there. The same recommendation applies to multiple conditions. By default all first-level analyses will typically be run separately for each condition defined in your project (e.g. to later be able to compare FC across conditions). If you are only planning to explore the results of a particular analysis for one or a few conditions of interest then simply select those conditions when running your first-level analyses (so that condition-specific estimates are not obtained for the remaining conditions, which will save space and time when running these analyses)
Hope this helps
Alfonso
Originally posted by Andrew OShea:
Some general space-managing tips are the following:
1) original data folders. Any Preprocessing results (e.g. realignment, coregistration, normalization, etc.) will be stored in the same folders where the original data was stored. You may remove there any intermediate step that will no longer be needed (e.g. you may for example keep the final functional data swardata.nii, the unsmoothed functional data wardata.nii, and the original data data.nii and remove the other itermediate steps rdata.nii and ardata.nii)
2) conn_*/data folder. This contains the data imported from (1) above into CONN during the Setup step, which will be used/needed to run the Denoising step. If you are no longer planning/needing ro re-run the Denoising step you may remove the data in this folder entirely.
3) conn_*/results/preprocessing folder. This contains the data resulting from CONN's Denoising step (2) above, which will be used/needed to run any First-level analysis step. If you are no longer planning/needing to re-run any First-level analysis step you may remove the data in this folder entirely.
4) conn_*/results/firstlevel/ANALYSISNAME. This contains the data resulting from CONN's first-level analysis step (3) above, which will be used/needed to run Second-level results analyses. If you are no longer planning/needing to run any new second-leve analysis (or if you just want to remove a particular first-level analysis) you may remove this folder entirely (note: also remove first the corresponding analysis in CONN's gui).
5) conn_*/results/secondlevel/ANALYSISNAME. This contains the data resulting from CONN's second-level analysis step (4) above, which will be used for displaying your second-level results (e.g. using "results explorer" guis). If you have analyses that are no longer needed/used you may simply remove the corresponding folders.
Another general recommendation to minimize used storage would be, for seed-to-voxel first-level analyses, to only enter there the seeds that you are planning to use (by default all ROIs will be entered there, which creates a lot of seed-to-voxel maps -one per subject/condition/seed combination-). A "space-consciuos" user would create two first-level analyses: one where only "ROI-to-ROI" analyses are run, and where all of the ROIs are entered there for good measure; and a second first-level analysis where only "seed-to-voxel" analyses are run, and where only a few selected ROIs/seeds of interest are entered there. The same recommendation applies to multiple conditions. By default all first-level analyses will typically be run separately for each condition defined in your project (e.g. to later be able to compare FC across conditions). If you are only planning to explore the results of a particular analysis for one or a few conditions of interest then simply select those conditions when running your first-level analyses (so that condition-specific estimates are not obtained for the remaining conditions, which will save space and time when running these analyses)
Hope this helps
Alfonso
Originally posted by Andrew OShea:
Hello,
In our lab we have multiple users who use CONN on a computing cluster. I was wondering if anyone knew of any strategies in reducing data usage when running many analyses? We are running into issues with users generating multiple TB worth of data. Although storage is relatively cheap now, we don't want to pay for more than we need. Is there anyway to purge intermediate files, or files not needed for certain analyses? Are there any particular space heavy options?
In our lab we have multiple users who use CONN on a computing cluster. I was wondering if anyone knew of any strategies in reducing data usage when running many analyses? We are running into issues with users generating multiple TB worth of data. Although storage is relatively cheap now, we don't want to pay for more than we need. Is there anyway to purge intermediate files, or files not needed for certain analyses? Are there any particular space heavy options?
Threaded View
| Title | Author | Date |
|---|---|---|
| Andrew OShea | Aug 5, 2016 | |
| Alfonso Nieto-Castanon | Aug 8, 2016 | |
| Marianna La Rocca | Mar 8, 2020 | |
| Amr Farahat | Jul 24, 2018 | |
| Alfonso Nieto-Castanon | Jul 24, 2018 | |
| Amr Farahat | Jul 27, 2018 | |
| Andrew OShea | Aug 16, 2016 | |
