Hi Dr Landman and everyone,

I am looking into running the JIST layouts via the command line, as encourageingly detailed in this webpage:
http://www.nitrc.org/plugins/mwiki/index...

However, I can't seem to find the java run-file as needed for this:
mipavjava edu.jhu.ece.iacl.jist.cli.runLayout CATNAP-PARv3.layout --help
I see that the webpage I am using as a reference is quite old (2009). I suspect that there are some changes... For example my Layout files have the extension LayoutXML now. Could you help me with finding a new updated command example?

Thank you!

--- update: I have solved this by adding '-Xms32g -Xmx128g' java options :) ----

Hi Andrew and Bennett,

Just a quick question regarding JIST-run: what is the uni of xDefaultMem? I am currently assuming it is MB like it is in the GUI. SO to request 32G I am entereing 32000. Is this correct?
On a side note, are there any settings you know of in java that could prevent it from using this amount memory even when it is requested by JIST-run?

Thanks!!!

Emma

Got it! \"Normalized mutual information\" did the trick. Sorry - I think I am creating the longest thread in history!

Hi Andrew,

Thank you agauin for yesterday's help! I really think this is going to work now! I have decided to just write a script using the modules instead of the layouts. It will also give me more flexibility. 

I have one more very specific question. I would like to run the registration module, with the options "-inCost Normalized mutual information" and "-inDegrees Rigid - 6". But it looks like the JIST-ruin command does not like spaces in the arguments. I've tried "" '' [] {} to no avail... Do you know how to run something like that?

Cheers,

Emma

p.s. I attached the files to this thread, see below, if you still want to try solving the layout problem. But, as mentioned, I don't mind doing it without layouts.

Hi Andrew,

I just checked 2 things, which may help to know:
- changing the filetype from .nii.gz to .nii does not make a difference
- the "external file" option gives the same problem as the "external volume" option (e.g. adding an atlas.txt file to a segmentation module)

Emma

Ok, I will try to attach them here.
It really is just about the external volume function, I believe. For the rest everything seems to be working just fine.

Are you able to send an example scan and the layout you are trying to run? I think we have hit the point where I am not able to debug any further without actually being able to work with the files you are using.

However, if running layouts proves too difficult I could write a script with all the modules separately. But, this would be quite a lot of work compared to just running layouts.... So if we could get the JIST_run layout option to work with flexible input I'd be much happier!

No, I am just testing the command line functions on one module. Once it works on that I will use the method to run big layouts. The layouts I eventually intend to run contain dozens of modules and take 10 hours to run. Those would definitely not be convenient to debug!

Emma

Are you just running the one module? If so, why are you using the layout? Just run it as a single module. The layouts are more challenging to debug.

Hi,

The output.txt file it creates is empty.

Running the module separately works fine, and creates the expected output. 
JIST-run module -c edu.jhu.ece.iacl.plugins.utilities.volume.MedicAlgorithmThreshold -- -inLower 1.5 -inVolume ~/Documents/8348498_UNI_cortex.nii.gz

So does running the layout if I use a fixed input volume.

Best wishes,
Emma

Is there anything in the output file? Also can you try running the individual modules in your layout serially through JIST-run? Both of those things may help identify the issue.

Sure - thank you! Here it is:

emmasprooten$ JIST-run layout -l ~/Dropbox/MYELIN_DINSE/layouts/testcommandline.LayoutXML -- -inExternal ~/Documents/8348498_UNI_cortex.nii.gz -help
cli usage: layoutfile [-h] [-inExternal ] [-outExecution ] [-xClean]
[-xDebugLvl ] [-xDefaultMem ] [-xDir ] [-xFile
] [-xJreLoc ] [-xMaxProcess ] [-xPrefExt ]
[-xRunOutOfProcess] [-xUseGridEng ]

JIST Adapter 1.8 beta
Algorithm Version: 1.8
-h,--help Print this message.
-inExternal External Volume [file] (required)
-outExecution Execution Time [string]
-xClean Remove all previously generated results in the
output directory
-xDebugLvl Set Debug output level[0-9]: 0 = Error messages
only, 1 = Major events and warnings, ect. Lvl 5+
maybe cause performance decrease.
-xDefaultMem Set Set default maximum heap size
-xDir Request Output: Processing Directory (default
current) [directory] (optional)
-xFile Request Output: Results File (default output.txt)
[file] (optional)
-xJreLoc Set location of JRE to use
-xMaxProcess Set default maximum number of processes.
-xPrefExt Set Prefered Output Image Extension
-xRunOutOfProcess Use the layout file preferences for the number of
simultaneous jobs and use a separate process for
each job.
-xUseGridEng Sets the use of the Grid Engine[true/false]
Provided by: JIST (Java Image Science Toolkit) Command Line Interface
version 1.11
http://www.nitrc.org/projects/jist/
cli ####################################################################
cli Initializing MIPAV/JIST Framework
cli ####################################################################
cli Interpretting command line arguments
cli ####################################################################
testcommandline
outExecution = =
inExternal = /Users/emmasprooten/Documents/8348498_UNI_cortex.nii.gz = /Users/emmasprooten/Documents/8348498_UNI_cortex.nii.gz
outProc = /Users/emmasprooten/Downloads
outResult = output.txt
cli ####################################################################
edu.jhu.ece.iacl.jist.pipeline.PipeScheduler: finished naturally.
Processing destinations...
Validating contexts...
edu.jhu.ece.iacl.jist.pipeline.gui.ProcessManager: stopping via interrupt.
cli ####################################################################
cli Done: /Users/emmasprooten/Dropbox/MYELIN_DINSE/layouts/testcommandline.LayoutXML
cli FAILED
cli ####################################################################

Can you copy the whole help for your layout? i.e just run the same thing but with -help at the end?

Thanks! I just had to pick "external volume" instead of "external file". Now I have the -inExternal flag option for my layout.

It is however not running:

emmasprooten$ JIST-run layout -l ~/layouts/testcommandline.LayoutXML -- -inExternal ~/Documents/8348498_UNI_cortex.nii.gz
cli ####################################################################
cli Initializing MIPAV/JIST Framework
cli ####################################################################
cli Interpretting command line arguments
cli ####################################################################
testcommandline
outExecution = =
inExternal = ~/Documents/8348498_UNI_cortex.nii.gz = ~/Documents/8348498_UNI_cortex.nii.gz
outProc = /Users/emmasprooten/Documents/mipav_plugins/PyJIST-0.1
outResult = output.txt
cli ####################################################################
edu.jhu.ece.iacl.jist.pipeline.PipeScheduler: finished naturally.
Processing destinations...
Validating contexts...
edu.jhu.ece.iacl.jist.pipeline.gui.ProcessManager: stopping via interrupt.
cli ####################################################################
cli Done: /Users/emmasprooten/Dropbox/MYELIN_DINSE/layouts/testcommandline.LayoutXML
cli FAILED
cli ####################################################################

It runs fine if I replace "external volume" with "volume collection" and define a fixed input.

It strikes me a little strange that it reports the -inExternal value twice. Could this indicate the problem?

Emma

It sounds like we are getting a little closer. I am glad to hear that. I am not as familiar with externalizing layouts as I am with other portions of JIST. I believe you can do something like

JIST-run layout -l -- -help

and it will give you a list of the command line parameters for the particular layout. That should include the externalized parameters which you defined in your layout file.

Hi Andrew,

That is working! Creates a nice random image!

Also I was now able to run a test layout via:
JIST-run layout -l ~/Dropbox/MYELIN_DINSE/layouts/testcommandline.LayoutXML

Now, I would like to run big layouts for many subjects automatically on the command line. Ideally: would I be able to add options/flags to this to specify the inputs and outputs? I tried to add the "external file" module to a simple test layout in the GUI, but I am not sure how this works and these "external file" modules do not want to connect to the test mdule (in this case "Image calculator".

Thanks again!

Emma

Are you certain the mipav path you are providing is correct? I'm looking at our configuration for mac and the path we use is /Applications/mipav/. The search functionality in PyJIST is a little hard to work with. Can you run the following command and let me know what happens?

JIST-run module -c edu.jhu.bme.smile.demo.RandomVol -- -inSize 20 -inSize2 20 -inSize3 20 -outRand1 $(pwd)/test.nii.gz

Andrew

Thanks, Andrew. I had to change the plugins dir, and also add the java path. So making progress! But now I am getting this:

> emmasprooten$ JIST-run search --plugindir /Users/emmasprooten/mipav/plugins --mipav /Applications/mipav/gov/nih/mipav/ -a
Exception: java.lang.NullPointerException thrown from the UncaughtExceptionHandler in thread "main"

Apologies for posting so many questions but having little experience with java code, this is difficult for me to troubleshoot.

Best wishes,
Emma

Hi Emma,
Are you sure the plugins path you are giving is correct? I just loaded JIST on a mac and I found the proper plugin directory was ~/mipav/plugins/.

Hi Andrew and Bennett,

Thanks for the pyjist suggestion. Meanwhile I have tried this on my mac, and I am getting the following error after installing and checking the first basic command:

emmasprooten$ JIST-run search --plugindir /Applications/mipav/modules --mipav /Applications/mipav/ -a
/bin/sh: None: command not found

It seems like I might have missed something elementary and fundamental here. /bin/sh definitely works! JIST-run search -h also works fine. I also added pyjist it to the $PATH. I also tried with and without --plugindir and --mipav flags.

Do I need to add a path to Java? Or is it an OS problem? I also have a linux machine but it's old so I am not sure how well it would cope with the high resolution data I am processing.

Many thanks,
Emma

Hi Emma,

Just to clarify what you are looking to do, you are trying to run a jist layout file but you are running in to some issues.

Getting jist to run from the command line is definitely not the easiest thing, especially on windows. Just out of curiosity, have you looked into our new package PyJIST? You can find a basic overview of it here https://www.nitrc.org/plugins/mwiki/inde...

The purpose of PyJIST, and its executable JIST-run, is to alleviate a lot of the challenges you are facing right now. I had to modify the code a little to make it work on my windows box, but it overall was not a problem. In general though, the batch processing works significantly better on unix based machines than windows ones.

If PyJIST and JIST-run do not work for you let me know and we can continue helping you debug this issue.
Andrew

Okay, that is very helpful!!!

I found the runLayout.class file.

And I tried:
$ alias mipavjava="/cygdrive/c/ProgramData/Oracle/Java/javapath/java -classpath /cygdrive/c/Program Files/mipav/plugins/:/cygdrive/c/Program Files/mipav/:find /cygdrive/c/Program\ Files/mipav/ -name \*.jar | sed 's#/cygdrive/c/Program Files/mipav/#:/cygdrive/c/Program\ Files/mipav/#' | tr -d '\n' | sed 's/^://' | sed 's/Program Files/Program\\ Files/g''"

I think this worked because now I get a new prompt when typing
$ mipavjava
>

But nothing happens when I try different ways to run the LayoutXML is this because of the XML filetype part? For example:
$ mipavjava edu.jhu.ece.iacl.jist.cli.runLayout testcommandline.LayoutXML
>
$ mipavjava /cygdrive/c/Users/sprooe01/mipav/plugins/edu/jhu/ece/iacl/jist/cli/runLayout testcommandline.LayoutXML
> 


Thanks again!

Emma

Okay, that is very helpful!!!

I found the runLayout.class file.

And I tried:
$ alias mipavjava="/cygdrive/c/ProgramData/Oracle/Java/javapath/java -classpath /cygdrive/c/Program Files/mipav/plugins/:/cygdrive/c/Program Files/mipav/:find /cygdrive/c/Program\ Files/mipav/ -name \*.jar | sed 's#/cygdrive/c/Program Files/mipav/#:/cygdrive/c/Program\ Files/mipav/#' | tr -d '\n' | sed 's/^://' | sed 's/Program Files/Program\\ Files/g''"

I think this worked because now I get a new prompt when typing
$ mipavjava
>

But nothing happens when I try different ways to run the LayoutXML is this because of the XML filetype part? For example:
$ mipavjava edu.jhu.ece.iacl.jist.cli.runLayout testcommandline.LayoutXML
>
$ mipavjava /cygdrive/c/Users/sprooe01/mipav/plugins/edu/jhu/ece/iacl/jist/cli/runLayout testcommandline.LayoutXML
> 


Thanks again!

Emma

They're different. The mipav java command uses MIPAVs JRE and sets up the classpath to find all of the jar files inside of mipav (lots of which are necessary for jist). 

With respect to your find command, runLayout is compiled, so it will show up like "runLayout.class". Try throwing another wildcard on the end and you should be able to find its location.

Best,
Steve

Thanks again!

Thus I assume that your mipavjava is identical to me running my normal java. Is this correct?
In that case I would do somethung like:
/cygdrive/c/ProgramData/Oracle/Java/javapath/java .runLayout  .layoutXML

But I still can't find the runLayout file:
$ find /cygdrive/d -name "*runLayout"
$ find /cygdrive/c -name "*runLayout"
$ find /cygdrive/c/ -name jist
/cygdrive/c/Users/sprooe01/mipav/plugins/de/mpg/cbs/jist
/cygdrive/c/Users/sprooe01/mipav/plugins/edu/jhu/cs/cisst/jist
/cygdrive/c/Users/sprooe01/mipav/plugins/edu/jhu/ece/iacl/jist
/cygdrive/c/Users/sprooe01/mipav/plugins/jist

Emma

p.s. don't worry about the "cygdrive in the paths, that's just because I am using cygwin. "/cygdrive/c/" is identical to "C://".

Hi Emma,

mipava java is an alias. See here:

alias mipavjava="/home/bennett/mipav-4.3.0/jre/bin/java -classpath /home/bennett/mipav/plugins/:/home/bennett/mipav-4.3.0/:'find /home/bennett/mipav-4.3.0/ -name \*.jar | sed 's#/home/bennett/mipav-4.3.0/#:/home/bennett/mipav-4.3.0/#' | tr -d '\n' | sed 's/^://''"

With respect to *.runLayout. It is in your mipav install directory here:
/plugins/edu/jhu/ece/iacl/jist/cli/runLayout

Cheers,
Steve

Hi Drs. Landman and Plassard,

Thank you for your quick responses. To answer your questions:

I am able to run my layouts within the JIST GIU. I want to be able to run any kind of layout I build in the GUI (and that works from the GUI) from command line, so that I can make the processing more efficient for a larger sample. 

I can't find any "mipavjava*" file on my c-drive (I installed mipav on c). Also I can't find any file ending on "*runLayout" either. 

Put simply, I am looking for something like the example command above that would work in my environment. I am running Mipav 7.2.0 and JIST 2.0-2013Nov6. I am using a variety of CBS modules, no DTI involved, just T1 anatomy.

Thanks!

Emma

A few other questions

- are you looking to run the DTI layout in particular or are you wanting to run any layouts from the command line? 

- have you been able to run layouts from inside the program?

Hi Emma,
Can you clarify what you are looking for? The mipavjava executable, the runLayout class file, or the .layout file?
I think I can help you sort this out.
Thanks!
Andrew