Yep, that was clear, not sure where the gui ideas came from.  MIPAV only needs the xvfb buffer, no gui interface is needed after you record the script.  The script is then run from the command line.