Hi Chris,

You are right there are a few ways to make this work (e.g. as you suggest we could create a new profile that submits jobs from a compute-node through a ssh remote execution to a login node), but perhaps the way that would be most generally-useful to others in a similar situation as yours would be if I simply add an option to CONN to submit jobs using Matlab's own Parallel Server functionality, adding that option to the current list of compatible job schedulers, as this is becoming an increasingly common scenario. If you are up for it perhaps I could use your help to test/debug this? The only potential difficulty that I am thinking of is whether the compute-nodes in your system have access to the same shared storage as your users' computers (where a user's CONN project and data is stored) and/or whether that access may be through a different search path in the remote filesystem. Can you give me some details about how is the storage organized in your cluster, where exactly you would expect a user saves their data and CONN project (fMRI often requires large datasets, and many analyses require reading and writing considerable amounts of large files), and how that is accessed from both a user's local computer and a compute-node?

Thanks!

Alfonso

Originally posted by cramos:

My name is Chris and I work for TACC (Texas Advanced Computing Center) where I maintain our installations of matlab through the Lmod module system on our HPC clusters. I am trying to help some of our users get set up with using CONN on our compute machines. The problem is that CONN assumes it's running on a head node and submits the job with slurm sbatch commands. We don't allow users to run matlab on the head nodes (login nodes) on any TACC machines since those are shared nodes and matlab is notoriously a bit of a memory hog, so instead we have the options to either run directly on the compute nodes (which are not shared on our systems) or submit slurm jobs remotely from their local computer's matlab shell/GUI via matlab's Paralllel Server.

Using CONN while running directly on a compute node is technically doable, but job submissions are disabled from the computes, so the workaround is to ssh to a login node from within the job to submit the subsequent CONN job. I believe that can be done by modifying the sbatch command CONN uses, but they are still left with having to wait in the queue until their initial job launches on the compute node to even start with CONN. Instead, we'd like for them to be able to stage their jobs locally so they don't have to wait in the queue, and then they can send off the CONN computation portion to our machines with a saved cluster configuation that we'll get them set up with ourselves. 

My issue is that I can't figure out a way to make CONN recognize and use the saved cluser configuration. Is there a way to make this work? 

Thanks in advance! 

-Chris