help
help > RE: CONN batch scripting: How to skip already-done things?
May 12, 2021 11:05 PM | Alfonso Nieto-Castanon - Boston University
RE: CONN batch scripting: How to skip already-done things?
Hi Karl,
Sorry about that, the batch.Setup.overwrite field only applies to the Setup step, similarly for the batch.Denoising.overwrite and batch.Analysis.overwrite fields. Unfortunately there is no equivalent 'overwrite' option for preprocessing, but you could either use the batch.Setup.add field to explicitly add/preprocess new subjects to an existing project, or manually select which subjects you want to preprocess/analyze using the general batch.subjects field.
In your case I believe the simplest way would have been to modify your script to only include the information about your new 4 subjects (e.g. change the contents of Setup.functionals, Setup.structurals, etc.), and then use Setup.isnew=0 and Setup.add=1 so that CONN adds this new information to your project and preprocesses only those new subjects (and then for the Setup/Denoising/First-level steps, the overwrite option would suffice so there is no need to change anything else in the scripts there)
Hope this helps
Alfonso
Originally posted by Karl Lerud:
Sorry about that, the batch.Setup.overwrite field only applies to the Setup step, similarly for the batch.Denoising.overwrite and batch.Analysis.overwrite fields. Unfortunately there is no equivalent 'overwrite' option for preprocessing, but you could either use the batch.Setup.add field to explicitly add/preprocess new subjects to an existing project, or manually select which subjects you want to preprocess/analyze using the general batch.subjects field.
In your case I believe the simplest way would have been to modify your script to only include the information about your new 4 subjects (e.g. change the contents of Setup.functionals, Setup.structurals, etc.), and then use Setup.isnew=0 and Setup.add=1 so that CONN adds this new information to your project and preprocesses only those new subjects (and then for the Setup/Denoising/First-level steps, the overwrite option would suffice so there is no need to change anything else in the scripts there)
Hope this helps
Alfonso
Originally posted by Karl Lerud:
Thanks for the reply Alfonso. I have a project
with ~95 subjects, and the only preprocessing I am using CONN for
is functional_art. All other preprocessing is done before import to
CONN. So I executed my script, and everything seems to run
correctly to completion. Then I added 4 more subjects to the
project, and simply changed Setup.isnew, Setup.overwrite,
Denoising.overwrite, and Analysis.overwrite all to 0 (which were
all 1 the first time around), and ran the script. This seems like
the correct way to go according to what you said and the
documentation.
The first thing that happened is that it ran the art preprocessing on all 99 subjects, overwriting and redoing this for the first 95. It also correctly did the art preprocessing for the 4 new subjects, but of course I didn't need to spend all that time redoing the first 95.
Then it did the rest fairly quickly, so I think telling it not to overwrite the rest of the analysis worked. Is there a way I can tell if it only processed the 4 recent subjects, or if it re-processed everything? And at the very least, it seems like it shouldn't have to redo the art preprocessing for everyone. Is there anything I am doing wrong, and is there a way to make this work as expected? Thanks,
Karl
Originally posted by Alfonso Nieto-Castanon:
The first thing that happened is that it ran the art preprocessing on all 99 subjects, overwriting and redoing this for the first 95. It also correctly did the art preprocessing for the 4 new subjects, but of course I didn't need to spend all that time redoing the first 95.
Then it did the rest fairly quickly, so I think telling it not to overwrite the rest of the analysis worked. Is there a way I can tell if it only processed the 4 recent subjects, or if it re-processed everything? And at the very least, it seems like it shouldn't have to redo the art preprocessing for everyone. Is there anything I am doing wrong, and is there a way to make this work as expected? Thanks,
Karl
Originally posted by Alfonso Nieto-Castanon:
Hi Karl,
The options are batch.Setup.overwrite, batch.Denoising.overwrite, and batch.Analysis.overwrite (set those fields to 0 when running the Setup, Denoising, or first-level analysis steps, respectively, in order to skip already-processed subject/ROIs; see all conn_batch field details in https://web.conn-toolbox.org/resources/c...)
Best
Alfonso
Originally posted by Karl Lerud:
The options are batch.Setup.overwrite, batch.Denoising.overwrite, and batch.Analysis.overwrite (set those fields to 0 when running the Setup, Denoising, or first-level analysis steps, respectively, in order to skip already-processed subject/ROIs; see all conn_batch field details in https://web.conn-toolbox.org/resources/c...)
Best
Alfonso
Originally posted by Karl Lerud:
Could I bump this? My question is basically just
how to skip already-processed subjects/ROIs in a CONN batch script.
Thanks,
Karl
Karl
Threaded View
| Title | Author | Date |
|---|---|---|
| Karl Lerud | Apr 15, 2021 | |
| Karl Lerud | Apr 27, 2021 | |
| Alfonso Nieto-Castanon | May 3, 2021 | |
| Karl Lerud | May 12, 2021 | |
| Alfonso Nieto-Castanon | May 12, 2021 | |
| Karl Lerud | Nov 10, 2021 | |
| Alfonso Nieto-Castanon | Nov 11, 2021 | |
| vaishali | Apr 22, 2025 | |
| Karl Lerud | Nov 11, 2021 | |