NITRC Pipeline system for Octave and Matlab Forum: open-discussion

Deactivate PSOM for debuging

Arnaud Delorme — Sat, 27 Oct 2018 18:08:52 GMT

Is it possible to temporarily deactivate the error catching method of PSOM to debug a function?

Arno

I have a problem about psom_run_script.m

Jianing Zhang — Fri, 21 Jul 2017 13:24:48 GMT

Dear Pierre:
There are some mistakes when I run the psom program:

Error: File: psom_run_script.m Line: 413 Column: 29
Unexpected MATLAB expression.

The Line 413 Column 29 is
psom_ppn = getenv("PSOM_WORKER_PPN")

1）Double quotes are invalid characters in MATLAB, and I found out the Syntax os getenv is single quotes(http://cn.mathworks.com/help/matlab/ref/getenv.html), so I I turned double quotes into single quotes;
2) But MATLAB still display the mistake "Error: File: psom_run_script.m Line: 413 Column: 29
Unexpected MATLAB expression."

Thanks so much.

RE: Restart job if files_out are missing

Joel Parker — Wed, 17 May 2017 20:03:50 GMT

Thanks for the reply!

Yes, I was thinking of a use case like make: a simple `make` or `make all` will regenerate everything necessary to make the `all` target, including to recreate any dependency files.

It sounds like a similar process for PSOM would be to create an 'all' job that depends on the others, or something similar. I'll play with it for my case.

RE: Restart job if files_out are missing

Pierre Bellec — Mon, 27 Feb 2017 4:13:04 GMT

I somehow missed this post, sorry for the very late answer. PSOM does not automatically reprocess jobs with missing outputs. The main reason is that PSOM allows for "clean up" jobs, i.e. jobs that remove files. If you re-run a pipeline to change a job at the end, you will not expect to re-run all jobs that generate intermediate outputs, unless those are necessary to run the jobs that have changed.

So, when re-running pipelines, any change in options or input/output file names will trigger a re-start of a job. If this job requires inputs that are missing, and that can be generated by other jobs, those will also be restarted (and that behavior is recursive).

You can force to restart the job called 'job' with the following option for `psom_run_pipeline`:

opt.restart = { 'job' };

There is no flag to restart jobs with missing outputs. Would not be hard to implement though, if you think this is useful.

Best,

Pierre

Restart job if files_out are missing

Joel Parker — Tue, 24 Jan 2017 16:40:32 GMT

Maybe I had the wrong expectation, but I assumed that if I created a job like this:
[code]<span style="white-space: pre;"> </span>pipeline.job.command = 'mycommand';
<span style="white-space: pre;"> </span>pipeline.job.files_out = 'myfile.txt';[/code]
Then if 'myfile.txt' was missing, 'job' would be run. But what I actually see is that if I run psom_run_pipeline once, then delete 'myfile.txt', then run psom_run_pipeline again, 'job' is not rerun.

Was my expectation wrong, or is there some other way to trigger this behavior?

RE: File '1.txt' when 'background' mode is used

Zaixu Cui — Tue, 14 Jul 2015 6:15:52 GMT

Dear Pierre,

I'm sorry.
Maybe it's my modification of the original code.

The new version work well.
Thanks very much.

Best wishes

Zaixu

RE: File '1.txt' when 'background' mode is used

Pierre Bellec — Fri, 10 Jul 2015 3:37:35 GMT

Dear Zaixu,

I was unable to find the line of code you are referring to. In the current version of the code there is no file '1.txt'. This does not ring a bell, and I would be surprised if any component of PSOM wrote files without specifying the full path. I suspect this may be a modification to the original code. If you do find the problematic code in the head revision of PSOM, please let me know the line number:
https://github.com/SIMEXP/psom/blob/master/psom_run_script.m

Best,

Pierre

RE: File '1.txt' when 'background' mode is used

Zaixu Cui — Mon, 06 Jul 2015 8:50:52 GMT

Dear Pierre,

I think I have found the solution of this problem.

I have modified 'fopen('1.txt', 'w')' in the script 'psom_run_script.m' to 'fopen(['~' filesep '1.txt'], 'w')'.
As we know, in the linux '~' is a symbol of the personal directory, so this command will create file 1.txt in the personal directory of the user and there is no permission problem.

Best wishes

Zaixu

File '1.txt' when 'background' mode is used

Zaixu Cui — Sat, 04 Jul 2015 1:20:33 GMT

Dear Pierre,

I have noticed that PSOM will create a file named '1.txt' in the current path of matlab when 'background' mode is used (the code is in the psom_run_script.m).
[size= 10px]However, if we don't write permission in the current path, then the error occurred.[/size]
[size= 10px]In fact, this is very common when PANDA used.[/size]

[size= 10px]I recommend that this file can be stored in the personal fold (/home/username) of users in Linux.[/size]
[size= 10px]Because we can ensure that the user have the write permission for the fold /home/username.[/size]

[size= 10px]Best wishes[/size]

[size= 10px]Zaixu[/size]
[size= 10px]
[/size]

[size= 10px] [/size]

RE: Jobs keep on running

Pierre Bellec — Mon, 22 Jun 2015 21:36:06 GMT

Dear Chaoyi,

I realize this update probably comes too late to be relevant anymore, but I am posting this as a future reference. I just released PSOM 1.2.0 which should be stable on *nix systems.
https://github.com/SIMEXP/psom/releases/tag/v1.2.0

One of the new features is the detection of inactive jobs, which become marked as failed. You should therefore not have jobs "running" forever. You can use opt.nb_resub to automatically resubmit jobs that fail a number of times before giving up. So if some of your jobs randomly crash with an out of memory, setting opt.nb_resub to 1 or 2 may be enough to complete the pipeline fully automatically despite the failures.

I hope this helps. Best,

Pierre