help
help > RE: CONN batch scripting: How to skip already-done things?
Nov 11, 2021 12:11 AM | Alfonso Nieto-Castanon - Boston University
RE: CONN batch scripting: How to skip already-done things?
Hi Karl,
Sorry about that, and thanks for reporting this issue, that was a limitation/bug in the code where the batch fields Setup.subjects.groups and Setup.subjects.effects were not compatible with the Setup.add option.
I have fixed that in the attached patch (this patch is for release 20b, to install it copy the attached file to your conn distribution folder overwriting the file with the same name there). Please let me know if you still run into any issues
Best
Alfonso
Originally posted by Karl Lerud:
Sorry about that, and thanks for reporting this issue, that was a limitation/bug in the code where the batch fields Setup.subjects.groups and Setup.subjects.effects were not compatible with the Setup.add option.
I have fixed that in the attached patch (this patch is for release 20b, to install it copy the attached file to your conn distribution folder overwriting the file with the same name there). Please let me know if you still run into any issues
Best
Alfonso
Originally posted by Karl Lerud:
Hi Alfonso, I wanted to follow up on this. I am
trying to follow your advice about adding more subjects, and only
preprocessing the new subjects. I have done what you suggest, also
trying to follow the instructions for Setup.add in the batch
documentation. In addition to what it says there, I have also set
Setup.RT to only have the values for the new subjects, and also
changed Setup.nsubjects to be the number of new subjects (as
opposed to all in the project), hopefully that is all correct so
far.
It is giving me the following error:
Index exceeds the number of array elements (6).
Error in conn_batch (line 1196)
CONN_x.Setup.l2covariates.values{nsub}{nl2covariates}=(batch.Setup.subjects.groups(nsub)==ngroup);
Error in multielectrodeCONNsetup_paper (line 331)
conn_batch(conn_x)
In this case, I am adding 6 new subjects, and that brings the total number in this project to 109. I checked, and internally to conn_batch, CONN_x.Setup.nsubjects is 109, not 6. I am not sure if that is right, or what to expect. I can provide more information or lines of code if that is necessary, hopefully this is enough information to diagnose what is going on. Thanks,
Karl
Originally posted by Alfonso Nieto-Castanon:
It is giving me the following error:
Index exceeds the number of array elements (6).
Error in conn_batch (line 1196)
CONN_x.Setup.l2covariates.values{nsub}{nl2covariates}=(batch.Setup.subjects.groups(nsub)==ngroup);
Error in multielectrodeCONNsetup_paper (line 331)
conn_batch(conn_x)
In this case, I am adding 6 new subjects, and that brings the total number in this project to 109. I checked, and internally to conn_batch, CONN_x.Setup.nsubjects is 109, not 6. I am not sure if that is right, or what to expect. I can provide more information or lines of code if that is necessary, hopefully this is enough information to diagnose what is going on. Thanks,
Karl
Originally posted by Alfonso Nieto-Castanon:
Hi Karl,
Sorry about that, the batch.Setup.overwrite field only applies to the Setup step, similarly for the batch.Denoising.overwrite and batch.Analysis.overwrite fields. Unfortunately there is no equivalent 'overwrite' option for preprocessing, but you could either use the batch.Setup.add field to explicitly add/preprocess new subjects to an existing project, or manually select which subjects you want to preprocess/analyze using the general batch.subjects field.
In your case I believe the simplest way would have been to modify your script to only include the information about your new 4 subjects (e.g. change the contents of Setup.functionals, Setup.structurals, etc.), and then use Setup.isnew=0 and Setup.add=1 so that CONN adds this new information to your project and preprocesses only those new subjects (and then for the Setup/Denoising/First-level steps, the overwrite option would suffice so there is no need to change anything else in the scripts there)
Hope this helps
Alfonso
Originally posted by Karl Lerud:
Sorry about that, the batch.Setup.overwrite field only applies to the Setup step, similarly for the batch.Denoising.overwrite and batch.Analysis.overwrite fields. Unfortunately there is no equivalent 'overwrite' option for preprocessing, but you could either use the batch.Setup.add field to explicitly add/preprocess new subjects to an existing project, or manually select which subjects you want to preprocess/analyze using the general batch.subjects field.
In your case I believe the simplest way would have been to modify your script to only include the information about your new 4 subjects (e.g. change the contents of Setup.functionals, Setup.structurals, etc.), and then use Setup.isnew=0 and Setup.add=1 so that CONN adds this new information to your project and preprocesses only those new subjects (and then for the Setup/Denoising/First-level steps, the overwrite option would suffice so there is no need to change anything else in the scripts there)
Hope this helps
Alfonso
Originally posted by Karl Lerud:
Thanks for the reply Alfonso. I have a project
with ~95 subjects, and the only preprocessing I am using CONN for
is functional_art. All other preprocessing is done before import to
CONN. So I executed my script, and everything seems to run
correctly to completion. Then I added 4 more subjects to the
project, and simply changed Setup.isnew, Setup.overwrite,
Denoising.overwrite, and Analysis.overwrite all to 0 (which were
all 1 the first time around), and ran the script. This seems like
the correct way to go according to what you said and the
documentation.
The first thing that happened is that it ran the art preprocessing on all 99 subjects, overwriting and redoing this for the first 95. It also correctly did the art preprocessing for the 4 new subjects, but of course I didn't need to spend all that time redoing the first 95.
Then it did the rest fairly quickly, so I think telling it not to overwrite the rest of the analysis worked. Is there a way I can tell if it only processed the 4 recent subjects, or if it re-processed everything? And at the very least, it seems like it shouldn't have to redo the art preprocessing for everyone. Is there anything I am doing wrong, and is there a way to make this work as expected? Thanks,
Karl
Originally posted by Alfonso Nieto-Castanon:
The first thing that happened is that it ran the art preprocessing on all 99 subjects, overwriting and redoing this for the first 95. It also correctly did the art preprocessing for the 4 new subjects, but of course I didn't need to spend all that time redoing the first 95.
Then it did the rest fairly quickly, so I think telling it not to overwrite the rest of the analysis worked. Is there a way I can tell if it only processed the 4 recent subjects, or if it re-processed everything? And at the very least, it seems like it shouldn't have to redo the art preprocessing for everyone. Is there anything I am doing wrong, and is there a way to make this work as expected? Thanks,
Karl
Originally posted by Alfonso Nieto-Castanon:
Hi Karl,
The options are batch.Setup.overwrite, batch.Denoising.overwrite, and batch.Analysis.overwrite (set those fields to 0 when running the Setup, Denoising, or first-level analysis steps, respectively, in order to skip already-processed subject/ROIs; see all conn_batch field details in https://web.conn-toolbox.org/resources/c...)
Best
Alfonso
Originally posted by Karl Lerud:
The options are batch.Setup.overwrite, batch.Denoising.overwrite, and batch.Analysis.overwrite (set those fields to 0 when running the Setup, Denoising, or first-level analysis steps, respectively, in order to skip already-processed subject/ROIs; see all conn_batch field details in https://web.conn-toolbox.org/resources/c...)
Best
Alfonso
Originally posted by Karl Lerud:
Could I bump this? My question is basically just
how to skip already-processed subjects/ROIs in a CONN batch script.
Thanks,
Karl
Karl
Threaded View
| Title | Author | Date |
|---|---|---|
| Karl Lerud | Apr 15, 2021 | |
| Karl Lerud | Apr 27, 2021 | |
| Alfonso Nieto-Castanon | May 3, 2021 | |
| Karl Lerud | May 12, 2021 | |
| Alfonso Nieto-Castanon | May 12, 2021 | |
| Karl Lerud | Nov 10, 2021 | |
| Alfonso Nieto-Castanon | Nov 11, 2021 | |
| vaishali | Apr 22, 2025 | |
| Karl Lerud | Nov 11, 2021 | |