help
help > RE: HPC - SLURM - Parralellisation with conn
Dec 21, 2020 02:12 PM | sophieb
RE: HPC - SLURM - Parralellisation with conn
Dear Alfonso,
It seems that it worked very well, I submitted my job (32Gb), and the toolbox divided the initial job into 4 jobs (8Gb for each).
As soon as I changed in my .sh job file the memory from 20 to 32GB (4*8), it seemed to have run.
When I look at the node.xxxx.stdout files, it seems that all steps have been completed.
However, I have a question. On our HCP, we are requested to submit jobs to the scheduler (requesting needed memory, cores, nodes etc.), using a .sh file.
I am planning on doing such analyses on 31 lesions*1018 subjects. The test analysis I ran was on 4 subjects and requested 8Gb of memory/job.
I am worried that if I submit my .sh job with my 1018 subjects on 1 node, 24 cores, using the conn parallelization option, I may reach the max of allowed ram ~256Gb/node and that my job will be killed.
1) Can I specify several nodes to do my analyses when using the conn parallelization option? Will the parallelization process take it into account?
2) If I split my initial big .sh job within the .sh file into different smaller jobs with different arguments such as output directories , start/last subjects ID, (jobs itself containing my matlab script with the conn parallelization option ON), this will produce different 1st level output directories (with maps having similar names Subject01_Condition01_Source01..).
Is it problematic then to use the generated maps from different output directories into a second second level -even though each first subject of each loop will be named subject01? Of course I will carefully preserve the order of the Subjects/conditions/sources, in my second level.
What do you think will be the easiest way to parallelize such analysis?
Sorry if this is not super clear, happy to reformulate if necessary
Thanks a lot,
Sophie
It seems that it worked very well, I submitted my job (32Gb), and the toolbox divided the initial job into 4 jobs (8Gb for each).
As soon as I changed in my .sh job file the memory from 20 to 32GB (4*8), it seemed to have run.
When I look at the node.xxxx.stdout files, it seems that all steps have been completed.
However, I have a question. On our HCP, we are requested to submit jobs to the scheduler (requesting needed memory, cores, nodes etc.), using a .sh file.
I am planning on doing such analyses on 31 lesions*1018 subjects. The test analysis I ran was on 4 subjects and requested 8Gb of memory/job.
I am worried that if I submit my .sh job with my 1018 subjects on 1 node, 24 cores, using the conn parallelization option, I may reach the max of allowed ram ~256Gb/node and that my job will be killed.
1) Can I specify several nodes to do my analyses when using the conn parallelization option? Will the parallelization process take it into account?
2) If I split my initial big .sh job within the .sh file into different smaller jobs with different arguments such as output directories , start/last subjects ID, (jobs itself containing my matlab script with the conn parallelization option ON), this will produce different 1st level output directories (with maps having similar names Subject01_Condition01_Source01..).
Is it problematic then to use the generated maps from different output directories into a second second level -even though each first subject of each loop will be named subject01? Of course I will carefully preserve the order of the Subjects/conditions/sources, in my second level.
What do you think will be the easiest way to parallelize such analysis?
Sorry if this is not super clear, happy to reformulate if necessary
Thanks a lot,
Sophie
Threaded View
Title | Author | Date |
---|---|---|
sophieb | Dec 18, 2020 | |
sophieb | Jan 15, 2021 | |
sophieb | Jan 14, 2021 | |
sophieb | Jan 13, 2021 | |
sophieb | Jan 11, 2021 | |
Alfonso Nieto-Castanon | Jan 11, 2021 | |
Alfonso Nieto-Castanon | Jan 11, 2021 | |
sophieb | Jan 12, 2021 | |
Alfonso Nieto-Castanon | Jan 12, 2021 | |
sophieb | Jan 12, 2021 | |
Alfonso Nieto-Castanon | Jan 26, 2021 | |
sophieb | Jan 12, 2021 | |
sophieb | Jan 11, 2021 | |
sat2020 | Dec 18, 2020 | |
Alfonso Nieto-Castanon | Dec 18, 2020 | |
sophieb | Dec 18, 2020 | |
Alfonso Nieto-Castanon | Dec 18, 2020 | |
sophieb | Dec 18, 2020 | |
Alfonso Nieto-Castanon | Dec 18, 2020 | |
sophieb | Dec 19, 2020 | |
Alfonso Nieto-Castanon | Dec 19, 2020 | |
sophieb | Dec 21, 2020 | |
Alfonso Nieto-Castanon | Dec 21, 2020 | |
sophieb | Jan 2, 2021 | |
sophieb | Jan 8, 2021 | |
sophieb | Dec 22, 2020 | |