My name is Chris and I work for TACC (Texas Advanced Computing Center) where I maintain our installations of matlab through the Lmod module system on our HPC clusters. I am trying to help some of our users get set up with using CONN on our compute machines. The problem is that CONN assumes it's running on a head node and submits the job with slurm sbatch commands. We don't allow users to run matlab on the head nodes (login nodes) on any TACC machines since those are shared nodes and matlab is notoriously a bit of a memory hog, so instead we have the options to either run directly on the compute nodes (which are not shared on our systems) or submit slurm jobs remotely from their local computer's matlab shell/GUI via matlab's Paralllel Server.
Using CONN while running directly on a compute node is technically doable, but job submissions are disabled from the computes, so the workaround is to ssh to a login node from within the job to submit the subsequent CONN job. I believe that can be done by modifying the sbatch command CONN uses, but they are still left with having to wait in the queue until their initial job launches on the compute node to even start with CONN. Instead, we'd like for them to be able to stage their jobs locally so they don't have to wait in the queue, and then they can send off the CONN computation portion to our machines with a saved cluster configuation that we'll get them set up with ourselves.
My issue is that I can't figure out a way to make CONN recognize and use the saved cluser configuration. Is there a way to make this work?
Thanks in advance!
-Chris
Hi Chris,
You are right there are a few ways to make this work (e.g. as you suggest we could create a new profile that submits jobs from a compute-node through a ssh remote execution to a login node), but perhaps the way that would be most generally-useful to others in a similar situation as yours would be if I simply add an option to CONN to submit jobs using Matlab's own Parallel Server functionality, adding that option to the current list of compatible job schedulers, as this is becoming an increasingly common scenario. If you are up for it perhaps I could use your help to test/debug this? The only potential difficulty that I am thinking of is whether the compute-nodes in your system have access to the same shared storage as your users' computers (where a user's CONN project and data is stored) and/or whether that access may be through a different search path in the remote filesystem. Can you give me some details about how is the storage organized in your cluster, where exactly you would expect a user saves their data and CONN project (fMRI often requires large datasets, and many analyses require reading and writing considerable amounts of large files), and how that is accessed from both a user's local computer and a compute-node?
Thanks!
Alfonso
Originally posted by cramos:
My name is Chris and I work for TACC (Texas Advanced Computing Center) where I maintain our installations of matlab through the Lmod module system on our HPC clusters. I am trying to help some of our users get set up with using CONN on our compute machines. The problem is that CONN assumes it's running on a head node and submits the job with slurm sbatch commands. We don't allow users to run matlab on the head nodes (login nodes) on any TACC machines since those are shared nodes and matlab is notoriously a bit of a memory hog, so instead we have the options to either run directly on the compute nodes (which are not shared on our systems) or submit slurm jobs remotely from their local computer's matlab shell/GUI via matlab's Paralllel Server.
Using CONN while running directly on a compute node is technically doable, but job submissions are disabled from the computes, so the workaround is to ssh to a login node from within the job to submit the subsequent CONN job. I believe that can be done by modifying the sbatch command CONN uses, but they are still left with having to wait in the queue until their initial job launches on the compute node to even start with CONN. Instead, we'd like for them to be able to stage their jobs locally so they don't have to wait in the queue, and then they can send off the CONN computation portion to our machines with a saved cluster configuation that we'll get them set up with ourselves.
My issue is that I can't figure out a way to make CONN recognize and use the saved cluser configuration. Is there a way to make this work?
Thanks in advance!
-Chris
Hi Alfonso!
That would be great, and I'd be happy to help test and debug! As you mentioned, I do think it would be the most generally useful to work with parallel server since that eliminates users being stuck waiting for a compute node to become available. Depending on what they are requesting (e.g. quantity of nodes, duration of the runs, etc.) wait times can range anywhere between a couple of hours to well over a day, all the while they would have to be sitting on the initial compute node hoping the nested job launches and completes before getting kicked off. That is to say, the ssh method is risky and not ideal.
That's a great point about the file systems and one I hadn't considered. The compute machines each have their own file systems (with one of them, "/work", being connected across all compute machines for easy sharing and backup). The rest are unique to each individual compute cluster, but the naming conventions (i.e. "/home1", "/work", and "/scratch") are uniform across all the systems for convenience. Each of those is accessible from the login nodes or compute nodes of their respective machine. Data transfers are typically done with command-line tools like rsync and scp, but managing data from a local machine can also be done either using an sftp client or the browser-based data management tool, Globus. Each cluster's “/scratch" file system is where we direct all users to stage their computation and IO because those spaces have no user quota limits on how much data they can store and the hardware of each “/scratch" is best equipped to handle the load. To balance this, purges are run at semi-regular intervals to clear out old, untouched data, so it is strongly recommended that copies of inputs and final results be kept and offloaded to other storage locations to avoid potential loss.
The vast majority of user workflows are to first transfer necessary data to the machine, then connect to a login node via ssh to build and stage jobs, then finally submit jobs with sbatch to whichever partition they choose. We also have a few browser-based portals for job submissions and those are primarily for a convenient method to launch interactive jobs that require a GUI or some other visual element, which are done through VNC and DCV. That’s how the CONN users here have been connecting thus far and they’ve just run the computation on the same node in the interactive job, rather than offloading the computation to another or multiple other nodes. All jobs are strictly submitted through login nodes though, where even the portals are also doing so under the hood. Our cluster configurations for submitting remotely with Parallel Server do the same.
Let me know if that answers your question or what other info you might need.
Thank you!
-Chris