Dear Pierre:


   It's not a good news. I have tested more jobs, and the problem is already
exists. Maybe it is easy to understand. Because of Copying, the
stauts_backup.mat is replaced all the time. And there will be a moment that the
file doesn't exist. I think it is the problem.

   But using try...catch..., it doesn't matter. The only
problem is that it will print 'unexpected end-of-file while reading compressed
data" in the command window.

   I think import the message which is printed in the command window to a txt file will address this problem.


Best Regards

Zaixu

Dear Zaixu,

It is an interesting problem you're posing: how to montitor the changes in job status going on in PSOM for a third-party software. The status are stored in a .mat file accessed by a single process (the so-called pipeline manager). As you noted, if another application tries to access it during the execution, this will cause trouble because the file is constantly being re-written. 

The simplest fix I can think of now is to generate a text file with a summary of status changes. Some sort of a news feed I guess. This file will just keep growing, and because the only changes are new lines being added, it will be possible to access its content without messing up with the updates (that's what a "tail -f" command does in bash, and it's possible to emulate this behavior from Matlab). PIPE_history.txt is almost what you need, but its content is a bit complicated to parse for two reasons:
1. The history accumulates from one pipeline execution to another. You are probably interested only in monitoring the current pipeline execution, not previous ones. So you would need to implement a mechanism to detect when the new pipeline execution started. Doable but not completely straightforward.
2. The history does not provides status statements for jobs that are already finished when a pipeline is started multiple times. It will not be possible for you to decide if a job is finished or scheduled for execution, unless you parse the output of the pipeline initialization stage as well. This is messy.

I am going to add a new file to the logs folder, called PIPE_news_feed.txt. This file will be reset at each pipeline execution. It will have one line per status change:

date , job name , status

If a pipeline is executed multiple times and some jobs are already finished (and kept as is), each finished jobs will be listed as a news. Using that file, you should be able to update a list of the status of all jobs in real time. It's not as simple as loading a .mat file, but not too complicated either. I'll implement that next week and send you a development version of PSOM so you can test it.

Best,

Pierre

Dear Pierre:

Thanks very much .
 
Look forward to your update.
 
Best Regards
 
Zaixu

Dear Zaixu,

I just commited a new version of PSOM (v690) which features a new file PIPE_news_feed.csv in the logs folder. It is a comma-separated values (csv) file, with one line per job submission/completion/failure. This file is reset everytime the pipeline is started. Jobs that were already completed/failed before anything is processed are listed as such. This file is useful to monitor the activity of the pipeline manager for third-party software.

Please do not hesitate if you have questions. 

Best,

Pierre

Dear Pierre:
 
    Thanks for your help very much .
 
Best Regards,
 
Zaixu