code_aster 17.4.0 docker image cannot be interrupted cleanly

vishwas

Hello,
I am using code_aster docker image from simvia docker page(17.4.0)
I launch the run normally with mpi_nbcpu = 4 To cleanly terminate job, I used this command:
pids=$(pgrep -fl python3 | tail -n 4 | awk '{print $1}') && kill -SIGUSR1 $pids
I get the following error:
[1,0]<stdout>: ╔════════════════════════════════════════════════════════════════════════════════════════════════╗
[1,0]<stdout>: ║ <I> <SUPERVIS_96> ║
[1,0]<stdout>: ║ ║
[1,0]<stdout>: ║ Réception du signal USR1. Interruption du calcul demandée... ║
[1,0]<stdout>: ║ ║
[1,0]<stdout>: ╚════════════════════════════════════════════════════════════════════════════════════════════════╝
[1,0]<stdout>:
[1,2]<stdout>:
[1,2]<stdout>: ╔════════════════════════════════════════════════════════════════════════════════════════════════╗
[1,2]<stdout>: ║ <I> <SUPERVIS_96> ║
[1,2]<stdout>: ║ ║
[1,2]<stdout>: ║ Réception du signal USR1. Interruption du calcul demandée... ║
[1,2]<stdout>: ║ ║
[1,2]<stdout>: ╚════════════════════════════════════════════════════════════════════════════════════════════════╝
[1,2]<stdout>:
[1,2]<stdout>:
[1,2]<stdout>: EXECUTION_CODE_ASTER_EXIT_124=-10
[1,2]<stdout>:
[1,2]<stdout>:
[1,2]<stdout>: restoring result databases from 'BASE_PREC'...
[1,2]<stdout>: WARNING: execution failed (command file #1): <F>_ABNORMAL_ABORT
[1,2]<stdout>:
[1,2]<stdout>: # ------------------------------------------------------------------------------
[1,2]<stdout>: Content of /tmp/run_aster_dz75b5w8/proc.2 after execution:
[1,2]<stdout>: .:
[1,2]<stdout>: total 270780
[1,2]<stdout>: -rw-r--r-- 1 user user 407 Apr 14 01:16 124.export
[1,2]<stdout>: drwxr-xr-x 2 user user 4096 Apr 14 01:16 REPE_IN
[1,2]<stdout>: drwxr-xr-x 2 user user 4096 Apr 14 01:16 REPE_OUT
[1,2]<stdout>: -rwxrwxrwx 1 user user 4152220 Apr 14 00:59 fort.20
[1,2]<stdout>: -rw-r--r-- 1 user user 269220 Apr 14 04:47 fort.6
[1,2]<stdout>: -rw-r--r-- 1 user user 0 Apr 14 01:16 fort.8
[1,2]<stdout>: -rw-r--r-- 1 user user 0 Apr 14 01:16 fort.9
[1,2]<stdout>: -rw-r--r-- 1 user user 94208008 Apr 14 04:45 glob.1
[1,2]<stdout>: -rw-r--r-- 1 user user 28585 Apr 14 01:16 stage1_extruded.comm.changed.py
[1,2]<stdout>: -rw-r--r-- 1 user user 180224008 Apr 14 04:47 vola.1
[1,2]<stdout>:
[1,2]<stdout>: REPE_OUT:
[1,2]<stdout>: total 0
[1,2]<stdout>:
[1,2]<stdout>:
[1,2]<stdout>: # ------------------------------------------------------------------------------
[1,2]<stdout>: Execution summary
[1,2]<stdout>: cpu system cpu+sys elapsed
[1,2]<stdout>: --------------------------------------------------------------------------------
[1,2]<stdout>: Preparation of environment 0.00 0.00 0.00 0.00
[1,2]<stdout>: Execution of code_aster 0.01 0.01 0.02 12638.58
[1,2]<stdout>: Copying results 0.00 0.00 0.00 0.01
[1,2]<stdout>: --------------------------------------------------------------------------------
[1,2]<stdout>: Total 0.01 0.01 0.02 12638.59

[1,2]<stdout>: --------------------------------------------------------------------------------

prterun detected that one or more processes exited with non-zero status,
thus causing the job to be terminated. The first process to do so was:
Process name: [prterun-1c6f5133f2d2-103@1,2]
Exit code: 255

What could be the reason for abnormal abort?

Thanks
Anirudh

basile-simvia

Hello vishwas

Just to make sure I understand the use case correctly:

if the goal is simply to obtain the final result, then indeed the safest approach is to let the computation finish normally. In that case, there should be no need to send SIGUSR1.

So I think the first point to clarify is: why do you need to interrupt the run before its natural end?

vishwas

Hello,
This is a big issue. Sometimes results are saved and sometimes not. I have lost many good results that could have provided better insight into physics of the problem.
From many trials I had, it happens more often when the solver is in middle of intense calculations like at 0.2s/ 1s.

Regards

jacob

Hi, for nproc = 2 worked

singularity shell ~/code_aster_17.4.0.sif << END
kill -SIGUSR1 135727 135734
END

the awk command seems to get not all pids correctly.

and I am launching it like

singularity shell ~/code_aster_17.4.0.sif << END
run_aster case.export
END

jacob

pids=$(top -b -n 1 | awk '/python3/ && $9>0 {print $1}') # 9th is the CPU column
echo "$pids"

singularity shell ~/code_aster_17.4.0.sif << END
kill -SIGUSR1 $pids
END

jacob

For example when running limit load analysis, you don't want to wait until it fails on its own when it is near the limit load.
Or for debugging purposes - you get results at least for some % of the load.

vishwas

Hi,
There could be many reasons to interrupt.
-If the simulation is taking a lot of time, it might be a good idea to stop it to view partial results. These results are very helpful to make sure one is going in the right direction.
-Simulation starts out well and then becomes slow or taking too much time per iteration, then also one would want to stop without loosing results.
As an example, Calculix allows user to view updated results after every converged time step, which is really handy.

Regards
Anirudh

jacob

vishwas
How do you run it?
like this?

singularity shell ~/code_aster_17.4.0.sif << END
run_aster case.export
END

I have found out that
run_aster -wtemporary case.export

creates folder "temporary" in my current directory and all run files there. Exactly what I wanted because
rep_trav in .export does nothing. Is there any documentation for it?

vishwas

Hello,
I use this for singularity:
cd ~/containers && singularity run ./salome_meca-lgpl-2024.1.0-1-20240327-scibian-11.sif shell
as_run path/to/export/file.export

and this for docker:
docker run --rm -it --mount type=bind,source=$(pwd),target=/home/user/data simvia/code_aster:17.4.0

A question regarding your command for clean interrupt.
If I paste this in terminal

pids=$(top -b -n 1 | awk '/python3/ && $9>0 {print $1}') # 9th is the CPU column
echo "$pids"
kill -SIGUSR1 $pids

IT works but If I paste this in terminal
pids=$(top -b -n 1 | awk '/python3/ && $9>0 {print $1}') && echo "$pids" && kill -SIGUSR1 $pids

It does not and only return pids.
Whats wrong using &&?

Anirudh

mf

vishwas

Hello,
just guessing but in the Linux command line 'command1 && command2' only executes command2 if command1 finished without problems. If you want a one-liner of several commands in succession like above then use ; instead of &&. Then in 'command1 ; command2;' command2 will always be executed, regardless.

Mario.