Hi Alfonso!

That would be great, and I'd be happy to help test and debug! As you mentioned, I do think it would be the most generally useful to work with parallel server since that eliminates users being stuck waiting for a compute node to become available. Depending on what they are requesting (e.g. quantity of nodes, duration of the runs, etc.) wait times can range anywhere between a couple of hours to well over a day, all the while they would have to be sitting on the initial compute node hoping the nested job launches and completes before getting kicked off. That is to say, the ssh method is risky and not ideal. 

That's a great point about the file systems and one I hadn't considered. The compute machines each have their own file systems (with one of them, "/work", being connected across all compute machines for easy sharing and backup). The rest are unique to each individual compute cluster, but the naming conventions (i.e. "/home1", "/work", and "/scratch") are uniform across all the systems for convenience. Each of those is accessible from the login nodes or compute nodes of their respective machine. Data transfers are typically done with command-line tools like rsync and scp, but managing data from a local machine can also be done either using an sftp client or the browser-based data management tool, Globus. Each cluster's “/scratch" file system is where we direct all users to stage their computation and IO because those spaces have no user quota limits on how much data they can store and the hardware of each “/scratch" is best equipped to handle the load. To balance this, purges are run at semi-regular intervals to clear out old, untouched data, so it is strongly recommended that copies of inputs and final results be kept and offloaded to other storage locations to avoid potential loss. 

The vast majority of user workflows are to first transfer necessary data to the machine, then connect to a login node via ssh to build and stage jobs, then finally submit jobs with sbatch to whichever partition they choose. We also have a few browser-based portals for job submissions and those are primarily for a convenient method to launch interactive jobs that require a GUI or some other visual element, which are done through VNC and DCV. That’s how the CONN users here have been connecting thus far and they’ve just run the computation on the same node in the interactive job, rather than offloading the computation to another or multiple other nodes. All jobs are strictly submitted through login nodes though, where even the portals are also doing so under the hood. Our cluster configurations for submitting remotely with Parallel Server do the same. 

Let me know if that answers your question or what other info you might need. 

Thank you!

-Chris