Hi all

I've performed the Setup step using a SGE computing environment to separate into several jobs (n=20, total of 180 subjects). This runs smoothly, jobs are (apparently) merged, and I can move on to the Denoising step. However, once I want to start Denoising, Conn tells me there are still pending jobs, and starts merging again. This then takes ages, and Conn then saves, but jobs remain pending. I'm at loss as to why this happens. Has anyone run into similar issues?

Thanks for any pointers!

Eugenio

Nevermind, I solved it. This was something completely unrelated to CONN, but rather how our local SGE interacted with MATLAB.

Cheers

Eugenio

Hi Eugenio,
I am curious about this, it seems like perhaps some issue with file permissions? (e.g. if CONN is not able to delete .dmat files generated by the SGE scheduler in your projects' directory). I know you already posted that this was solved, but I was wondering if you can tell me more details about why this was failing (just to see if I can try to avoid other users from running into a similar scenario in the the future)
Thanks!
Alfonso
Originally posted by Eugenio Abela:

Hi Alfonso

thank you very much for following up, and sorry for the late reply! 

This was some weeks back, so I hope I remember correctly. Also, my understanding of SGE is rather pedestrian (I'm a medical doctor, so computational stuff tends to befuddle me).

Essentially, there was a collision between an SGE job script I wrote and my /.cshrc. The script was a wrapper around a MATLAB script that ran SPM12 / CAT12 ahead of a CONN batch (for some bespoke preprocessing). The script and the .cshrc both loaded MATLAB, but different versions, and that let the whole thing stall.

Another problem I had is that this was a very large connectome (n=1000, Genome Superstruct Project data), and even if it ran, the .dmat files took ages to load. So I split the whole thing up, processed each subject in its own CONN project, and later merged with conn_merge.m

So, here is how the final SGE job script looked like (comments for this email underlined)

Basic settings:
#!/bin/tcsh#$ -N matlab_job
#$ -o /data/group/epilepsy/Abela/logs/
#$ -e /data/group/epilepsy/Abela/logs/
#$ -q global
#$ -l h_vmem=20G
#$ -l h_rt=20:00:00

Key step: purge any modules loaded by .cshrc
setenv SGE_CLEAN 1

# First unload any modules loaded by ~/.cshrc, then load the defaults
module purge
module load nan/default
module load sge
# Load script dependent modules here
module load matlab/9.5.0
module load spm/12-7487

Actual wrapper: this takes a text file with patient indexes (file paths) and executes the matlab script netloc_conn for each of them

# set the working variables
set working_code=/data/group/epilepsy/Abela/code/conn
set sge_index=${working_code}/gsp_conn_index.txt
set sge_id=${working_code}/gsp_conn_subID.txt

# Don't included the .m extention
set matlabscript=netloc_conn_sub

# Extend the Matlab search to include your own .m files
if ( $?MATLABPATH ) then
setenv MATLABPATH ${MATLABPATH}:${working_code}
else
setenv MATLABPATH ${working_code}
endif

# Search the file for the SGE_TASK_ID number as a line number
setenv subj_dir 'sed -n ${SGE_TASK_ID}p ${sge_index}|awk '{print $1}''
setenv subj_id 'sed -n ${SGE_TASK_ID}p ${sge_id}|awk '{print $1}''

# Print out files and the Matlab running config
# Good for debugging
echo "-----------------------------------------------"
echo "Working file information"
echo "Selected subject :"${subj_id}
echo "Selected script :"${matlabscript}
echo "-----------------------------------------------"
#echo "Matlab running defaults"
#The defaults used before running your script below.
#matlab -n

Hope that makes sense! If you know of any trick to speed up the .dmat loading, I would be all ears!

Kind regards, 

Eugenio