Dear all,

My preprocessing pipeline (1st, standard) stopped in the middle of functional segmentation & normalisation because of lack of space. Segments finished correctly. Since this is quite a big amount of data, how to continue now correctly and avoid double files?

I would appreciate your advise
Greetings,
Lucas

Dear Lucas,

CONN uses a kind of "atomic" approach to its changes of state, so normally if the preprocessing stopped halfway, CONN did not change the input files in the project (you can check that manually by looking in the Structural and Functional tabs in Setup). If the input files are the original files, then CONN should normally overwrite the preprocessed files when you redo the preprocessing.

Hope this helps,
Stephen

Dear Stephen,

thank you. The functional files in the Setup are already referenced to "au". And the data files have various size.
I wonder what to do now.

Greetings,
Lucas

The process takes a lot of time, so I would like to save the previous steps...

For example, if I start the preprocessing halfway (removing previous steps until functional segmentation and normalisation), if some files are inside some folders and not in others, will it overwrite them? If not, I think the best would be somehow to clean up the folders (?)

Greetings,
Lucas

Dear Lucas,

Following up on Stephen's response, when running preprocessing locally each individual preprocessing step in CONN is treated as an atomic unit (each step is run sequentially and your CONN project is only updated after a step finishes correctly for all subjects) so generally you may simply re-start preprocessing, first removing from your pipeline all of the individual steps that have already finished correctly, continuing from the one where it stopped last time. In your case I would first double-check that all of your subjects functional files are referenced as "au*.nii" in your CONN project, and then re-start preprocessing and remove from the existing pipeline the already-finished steps (i.e. remove the first four steps, from functional realignment to outlier detection) in order to continue the standard pipeline from the segmentation&normalization step.

note: this will repeat the segmentation&normalization step for some of the subjects (since that step did not finished for all subjects). That is a little waste of time but generally acceptable. If you want to skip segmentation&normalization on the subjects where that step has already finished, that can be done as well but it is a bit more involved (e.g. you need to manually import the normalized functional and ROI gray/white/csf masks for those subjects that have already been processed, then run segmentation&normalization on the remaining subjects, and then continue from the following step now applied to all subjects again) so I generally recommend simply re-starting from the partially-completed step instead, which is far simpler and less vulnerable to human error. 

Hope this helps
Alfonso
Originally posted by Lucas Moro:

Thank you Alfonso!
Thank you Stephen!
 
This is very very helpful and well appreciated!

Greetings
Lucas