My name is Chris and I work for TACC (Texas Advanced Computing Center) where I maintain our installations of matlab through the Lmod module system on our HPC clusters. I am trying to help some of our users get set up with using CONN on our compute machines. The problem is that CONN assumes it's running on a head node and submits the job with slurm sbatch commands. We don't allow users to run matlab on the head nodes (login nodes) on any TACC machines since those are shared nodes and matlab is notoriously a bit of a memory hog, so instead we have the options to either run directly on the compute nodes (which are not shared on our systems) or submit slurm jobs remotely from their local computer's matlab shell/GUI via matlab's Paralllel Server.
Using CONN while running directly on a compute node is technically doable, but job submissions are disabled from the computes, so the workaround is to ssh to a login node from within the job to submit the subsequent CONN job. I believe that can be done by modifying the sbatch command CONN uses, but they are still left with having to wait in the queue until their initial job launches on the compute node to even start with CONN. Instead, we'd like for them to be able to stage their jobs locally so they don't have to wait in the queue, and then they can send off the CONN computation portion to our machines with a saved cluster configuation that we'll get them set up with ourselves.
My issue is that I can't figure out a way to make CONN recognize and use the saved cluser configuration. Is there a way to make this work?
Thanks in advance!
-Chris
