Remarks on the new salome_meca-lgpl-2024.1.0-1-20240327 released today

mf

Hello all,

I just downloaded the new container salome_meca-lgpl-2024.1.0-1-20240327.sif. I could not get the previous version to work on my system, somewhere there's a thread about that.

However, I also encountered an error in this version, but I was able to resolve it right away. Therefore I thought it'd be nice to share that right away. Some info about my system:
Ubuntu 24.04LTS,
Nvidia GTX3090 with Driver Version: 535.183.01,
singularity-ce version 4.1.4-noble (not the latest!).

I launched the container the usual way with:

singularity run --app install salome_meca-lgpl-2024.1.0-1-20240327-scibian-11.sif

Then I launch Salome-Meca 2024 with:

./salome_meca-lgpl-2024.1.0-1-20240327-scibian-11

The window pops up, I can do some stuff, change to other modules.... BUT: when I changed to Asterstudy, SM2024 crashes, giving me this error message (rather long, sorry):

Open MPI's OFI driver detected multiple equidistant NICs from the current process,
but had insufficient information to ensure MPI processes fairly pick a NIC for use.
This may negatively impact performance. A more modern PMIx server is necessary to
resolve this issue.

AsterStudy: jeu.-09-18:12:41.638 AsterStudy is activating...
AsterStudy: jeu.-09-18:12:41.638 Creating workspace...
AsterStudy: jeu.-09-18:12:41.661 Refreshing configuration on localhost...
A process has executed an operation involving a call
to the fork() system call to create a child process.

As a result, the libfabric EFA provider is operating in
a condition that could result in memory corruption or
other system errors.

For the libfabric EFA provider to work safely when fork()
is called, you will need to set the following environment
variable:
RDMAV_FORK_SAFE

However, setting this environment variable can result in
signficant performance impact to your application due to
increased cost of memory registration.

You may want to check with your application vendor to see
if an application-level alternative (of not using fork)
exists.

Your job will now abort.
[HP-Z8-G4-Workstation:3765134] *** Process received signal ***
[HP-Z8-G4-Workstation:3765134] Signal: Abandon (6)
[HP-Z8-G4-Workstation:3765134] Signal code: (-6)
[HP-Z8-G4-Workstation:3765134] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x73b3fd2cf140]
[HP-Z8-G4-Workstation:3765134] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x73b3fa6cad51]
[HP-Z8-G4-Workstation:3765134] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x73b3fa6b4537]
[HP-Z8-G4-Workstation:3765134] [ 3] /usr/lib/x86_64-linux-gnu/libfabric.so.1(+0x7c14e)[0x73b3e0a2914e]
[HP-Z8-G4-Workstation:3765134] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x80f18)[0x73b3fa712f18]
[HP-Z8-G4-Workstation:3765134] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_fork+0x20)[0x73b3fa759a70]
[HP-Z8-G4-Workstation:3765134] [ 6] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x245697)[0x73b404a35697]
[HP-Z8-G4-Workstation:3765134] [ 7] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x10de74)[0x73b4048fde74]
[HP-Z8-G4-Workstation:3765134] [ 8] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyObject_MakeTpCall+0x90)[0x73b4048b7b00]
[HP-Z8-G4-Workstation:3765134] [ 9] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyEval_EvalFrameDefault+0x8632)[0x73b4048692f2]
[HP-Z8-G4-Workstation:3765134] [10] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x1a5fdc)[0x73b404995fdc]
[HP-Z8-G4-Workstation:3765134] [11] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyFunction_Vectorcall+0x9d)[0x73b4048b2d2d]
[HP-Z8-G4-Workstation:3765134] [12] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyEval_EvalFrameDefault+0x77f7)[0x73b4048684b7]
[HP-Z8-G4-Workstation:3765134] [13] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x1a5fdc)[0x73b404995fdc]
[HP-Z8-G4-Workstation:3765134] [14] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyFunction_Vectorcall+0x9d)[0x73b4048b2d2d]
[HP-Z8-G4-Workstation:3765134] [15] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyObject_FastCallDictTstate+0xce)[0x73b4048ba04e]
[HP-Z8-G4-Workstation:3765134] [16] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyObject_Call_Prepend+0xe0)[0x73b4048ba1f0]
[HP-Z8-G4-Workstation:3765134] [17] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x1308f2)[0x73b4049208f2]
[HP-Z8-G4-Workstation:3765134] [18] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x125b3a)[0x73b404915b3a]
[HP-Z8-G4-Workstation:3765134] [19] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyObject_MakeTpCall+0x90)[0x73b4048b7b00]
[HP-Z8-G4-Workstation:3765134] [20] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyEval_EvalFrameDefault+0x76a1)[0x73b404868361]
[HP-Z8-G4-Workstation:3765134] [21] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x1a5fdc)[0x73b404995fdc]
[HP-Z8-G4-Workstation:3765134] [22] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyFunction_Vectorcall+0x9d)[0x73b4048b2d2d]
[HP-Z8-G4-Workstation:3765134] [23] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyEval_EvalFrameDefault+0x5f8d)[0x73b404866c4d]
[HP-Z8-G4-Workstation:3765134] [24] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x70033)[0x73b404860033]
[HP-Z8-G4-Workstation:3765134] [25] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyEval_EvalFrameDefault+0x77f7)[0x73b4048684b7]
[HP-Z8-G4-Workstation:3765134] [26] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x1a5fdc)[0x73b404995fdc]
[HP-Z8-G4-Workstation:3765134] [27] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyFunction_Vectorcall+0x9d)[0x73b4048b2d2d]
[HP-Z8-G4-Workstation:3765134] [28] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(PyEval_EvalFrameDefault+0x77f7)[0x73b4048684b7]
[HP-Z8-G4-Workstation:3765134] [29] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x1a5fdc)[0x73b404995fdc]
[HP-Z8-G4-Workstation:3765134] *** End of error message ***

The message gives the solution away, very smart. The solution is to set the environment variable:

export RDMAV_FORK_SAFE=1

before doing the ./salome_meca-lgpl-2024.1.0-1-20240327-scibian-11.

So, in essence, a workaround is to execute the following commands in succesion:

singularity run --app install salome_meca-lgpl-2024.1.0-1-20240327-scibian-11.sif
export RDMAV_FORK_SAFE=1
./salome_meca-lgpl-2024.1.0-1-20240327-scibian-11

That's it! Hope it helps, BEFORE someone else runs into the same problem,

Mario.

AsterO'dactyle

Thank you Mario !

k_zurawski

what might be the problem? No Code_Aster version available...

mf

I forgot to add, that the changes made in the nvliblist.conf file are still in place. See this thread:

https://forum.code-aster.org/public/d/27539-new-version-sm-salome-meca-lgpl-202310-4-20240327-questions/2

Otherwise, no changes have been made.

Mario.

k_zurawski

This is what I'm getting in terminal:
`w2@W2:~$ singularity run salome_meca-lgpl-2024.1.0-1-20240327-scibian-11.sif

running salome_meca in software rendering mode
++ using container salome_meca-lgpl-2024.1.0-1-20240327-scibian-11.sif
++ using software rendering mode (should use mesa_salome starter)
AsterStudy: jeu.-09-20:39:41.679 AsterStudy is activating...
AsterStudy: jeu.-09-20:39:41.680 Creating workspace...
AsterStudy: jeu.-09-20:39:41.702 Refreshing configuration on localhost...
AsterStudy: jeu.-09-20:40:59.732 workspace is ready.
AsterStudy: jeu.-09-20:41:01.624 AsterStudy activated
AsterStudy: jeu.-09-20:41:26.662 Refreshing configuration on localhost...
AsterStudy: jeu.-09-20:41:38.440 Refreshing configuration on localhost...
AsterStudy: jeu.-09-20:41:57.849 Refreshing configuration on cronos.hpc.edf.fr...
AsterStudy: jeu.-09-20:41:57.995 exit code: 255
AsterStudy: jeu.-09-20:42:01.650 Refreshing configuration on gaia.hpc.edf.fr...
AsterStudy: jeu.-09-20:42:01.719 exit code: 255
AsterStudy: jeu.-09-20:42:04.671 Refreshing configuration on localhost...
double free or corruption (out)
[W2:00063] *** Process received signal ***
[W2:00063] Signal: Abandon (6)
[W2:00063] Signal code: (-6)
[W2:00063] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f05dc058140]
[W2:00063] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f05d9453d51]
[W2:00063] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f05d943d537]
[W2:00063] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x7a458)[0x7f05d9495458]
[W2:00063] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x8174a)[0x7f05d949c74a]
[W2:00063] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x82d78)[0x7f05d949dd78]
[W2:00063] [ 6] /usr/lib/python3/dist-packages/PyQt5/QtCore.abi3.so(+0x2137ad)[0x7f0577f007ad]
[W2:00063] [ 7] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x1131a7)[0x7f05e368c1a7]
[W2:00063] [ 8] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(+0x2686c7)[0x7f05e37e16c7]
[W2:00063] [ 9] /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0(Py_FinalizeEx+0x74)[0x7f05e3760a84]
[W2:00063] [10] SALOME_Session_Server_No_Server(Z18AbstractGUIAppMainI14GUIAppNewStyleEiiPPc+0x2651)[0x5647f6779ef1]
[W2:00063] [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f05d943ed7a]
[W2:00063] [12] SALOME_Session_Server_No_Server(start+0x2a)[0x5647f677517a]
[W2:00063] *** End of error message ***
ERROR:salomeContext:Unexpected error:
ERROR : 20:47:21.418 : Unexpected error: ( callsite is _startSalome of file "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/SALOME/bin/salome/salomeContext.py" at line 309 )
Traceback (most recent call last):
File "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/SALOME/bin/salome/salomeContext.py", line 299, in _startSalome
res = getattr(self, command)(options) # run appropriate method
File "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/SALOME/bin/salome/salomeContext.py", line 390, in sessionless
runSalome.runSalome()
File "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/KERNEL/bin/salome/runSalome.py", line 338, in runSalome
foreGround(args, ior_fakens_filename)
File "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/KERNEL/bin/salome/runSalome.py", line 308, in foreGround
killProcessSSL(port,[session_pid])
File "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/KERNEL/bin/salome/killSalomeWithPort.py", line 313, in killProcessSSL
__killPids(pids_list)
File "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/KERNEL/bin/salome/killSalomeWithPort.py", line 256, in __killPids
__killProcesses(processes)
File "/opt/salome_meca/V2024.1.0_scibian_univ/prerequisites/Cea_archive-9120/BINARIES-DB11/KERNEL/bin/salome/killSalomeWithPort.py", line 237, in killProcesses
process.terminate()
File "/usr/lib/python3/dist-packages/psutil/init__.py", line 271, in wrapper
raise NoSuchProcess(self.pid, self.name)
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=63)`

AsterO'dactyle

No problem for me on my Linux ubuntu 22.04 machine.
Do you use access to Amazon Web Services?

mf

AsterO'dactyle
Hello all,

I have some good news and some bad news.

I managed to compile a stable MPI-version with CA 16.7 in this container salome_meca-lgpl-2024.1.0-1-20240327.

The bad news is: I only managed to do this WITHOUT Metis and without Petsc. I can live without Petsc, but without Metis, the MPI-version does NOT make much sense, now does it?

However, I noticed the following: even in the stable 16.7 in this container there is no Metis. Why is that? Is this intentional, I suppose so? The basic output of this stable 16.7 looks like this (why does it show 'testing' in the first line?):

             -- CODE_ASTER -- VERSION : DÉVELOPPEMENT STABILISÉE (testing) --                  
                           Version 16.7.0 modifiée le 10/12/2024                                
                           révision 5f2fdcddabc5 - branche 'HEAD'                               
                               Copyright EDF R&D 1991 - 2025                                    
                          Exécution du : Fri Jan 10 10:43:02 2025                               
                          Nom de la machine : HP-Z8-G4-Workstation                              
                                    Architecture : 64bit                                        
                                Type de processeur : x86_64                                     
           Système d'exploitation : Linux-6.8.0-51-generic-x86_64-with-glibc2.31                
                              Langue des messages : fr (UTF-8)                                  
                                 Version de Python : 3.9.2                                      
                                 Version de NumPy : 1.19.5                                      
                                 Parallélisme MPI : inactif                                     
                                Parallélisme OpenMP : actif                                     
                          Nombre de processus OpenMP utilisés : 1                               
                           Version de la librairie HDF5 : 1.10.3                                
                            Version de la librairie MED : 4.1.1                                 
                           Version de la librairie MFront : 4.1.0                               
                           Version de la librairie MUMPS : 5.5.1c                               
                              Librairie PETSc : non disponible                                  
                           Version de la librairie SCOTCH : 7.0.1

My questions are, for the moment, because I did not do a test run yet:

Does this MPI-version even make sense without Metis, will Scotch work (it never worked in my CA-versions in the past)?
I downloaded the prerequisites codeaster-prerequisites-20240327-oss.tar.gz, there is a Metis metis-5.1.0.tar.gz inside, but I cannot untar it, I get an errror something like 'not an archive'. So I cannot include Metis here.

Maybe AsterO'dactyle, can you please comment? Thanks a lot,

Mario.

I will do some testing with this MPI version today. The basic setup is this:

             -- CODE_ASTER -- VERSION : DÉVELOPPEMENT STABILISÉE (testing) --                  
                           Version 16.7.0 modifiée le 10/01/2024                                
                                révision n/a - branche 'n/a'                                    
                               Copyright EDF R&D 1991 - 2025                                    
                          Exécution du : Fri Jan 10 10:33:42 2025                               
                          Nom de la machine : HP-Z8-G4-Workstation                              
                                    Architecture : 64bit                                        
                                Type de processeur : x86_64                                     
           Système d'exploitation : Linux-6.8.0-51-generic-x86_64-with-glibc2.31                
                              Langue des messages : fr (UTF-8)                                  
                                 Version de Python : 3.9.2                                      
                                 Version de NumPy : 1.19.5                                      
                                  Parallélisme MPI : actif                                      
                               Rang du processeur courant : 0                                   
                           Nombre de processeurs MPI utilisés : 1                               
                                Parallélisme OpenMP : actif                                     
                          Nombre de processus OpenMP utilisés : 1                               
                           Version de la librairie HDF5 : 1.10.9                                
                            Version de la librairie MED : 4.1.1                                 
                           Version de la librairie MFront : 4.2.0                               
                          Version de la librairie MUMPS : 5.6.2.2c                              
                              Librairie PETSc : non disponible                                  
                           Version de la librairie SCOTCH : 7.0.4

That's disappointing: I run into an error even before files are copied:

WARNING: If MPI_Abort is called during execution, result files could not be copied.
Running: mpiexec -n 2 --tag-output /opt/salome_meca/V2024.1.0_scibian_univ/tools/Code_aster_MPI-1670/bin/run_aster --wrkdir /tmp/run_aster_jeby4z5j --status-file /home/mario/.tmp_run_aster/run_aster_w8modc0o/status /home/mario/.tmp_run_aster/run_aster_w8modc0o/export.0
++ using container salome_meca-lgpl-2024.1.0-1-20240327-scibian-11.sif
++ prepend system 'libGL*' libs

What can I do? It cannot write to the /tmp folder it seems. Maybe the used CA version is not correct? Perhaps someone with more knowledge is able to help?

k_zurawski

I'm not using Amazon Web Services, but I have additional level of complexity. I'm running S_M on Windows 11 on Debian 12 through WSL2. Which version of Singularity are you using? I'll try tomorrow with Ubuntu through WSL2 unfortunately.

k_zurawski

Is there anything specific that HOST system, where Singularity is installed should have installed/activated?

nicolas_p

If you use the precompiled prerequisities 20240327, there are only for unstable version 17.*

That is main reason why petsc and metis do not work. Normaly, Scotch works. In MPI, I recommand more Pt-schotch ant Parmetis.

For version 16.*, you should use older prerequisities version. On gitlab project, it is writen

From 16.3.2 : salome_meca-edf-2023.1.0-1-20221225-scibian-10 -> so use 20221225 prequisities

mf

nicolas_p

Hello,

I think I have a deja-vu :-). Do I understand correctly, I cannot use this container for a stable MPI version at all (yet)?
For CA 16.7 I would have to use salome_meca-edf-2023.1.0-1-20221225-scibian-10. The problem is, this container never worked on my system (SIGSEGV errors). See https://forum.code-aster.org/public/d/27539-new-version-sm-salome-meca-lgpl-202310-4-20240327-questions

So for CA 16.7 MPI I can use the container salome_meca-lgpl-2022.1.0-1-20221225-scibian-9 (the latest for 16.*) also?

The other choice would be to go for a 17.x version with this newest container, right away. Maybe I will try this. Or maybe I do both....

Thanks a lot,

Mario.

mf

Hello,

I tried SM2022 and CA16.7, with this one I succeeded to build the MPI version.
With SM2024 and CA17.2 I still get the above error, maybe something wrong with user rights, IDK (see image below). The 'stable_mpi' was my mistake, should say 'testing_mpi'. This simulation runs fine in all other versions of this SM2024.

Mario.

eedzo

mf Hi Mario, I also cannot get the mpi version to run whenever I input a number greater than 1 into the 'Number of MPI CPU' field. I was having the same issues with SM2023. I have also tried running the compiled stable release from SM2022 in SM2024 but still doesn't run with a number greater than 1 into the 'Number of MPI CPU' field.

mf

Hello,

I finally ran into some trouble with this container, not on the system mentioned above, but on a different one:
Ubuntu 22.04LTS,
no Nvidia GPU, rather onboard Intel HDA
singularity-ce version 4.0.1-jammy
X11 (no Wayland) also applies to system in first message.

When in Asterstudy, I press a save file dialogue like in IMPR_RESU or IMPR_TABLE, the whole SM2024 crashes. It is the button, with the three dots in between unite and localhost:

As soon as the button is pressed, the window crashes and this is printed to the terminal:

AsterStudy: sam.-11-21:05:50.519 AsterStudy activated
(SALOME_Session_Server_No_Server:3246): Gtk-WARNING **: 21:06:07.094: Failed to measure available space: Erreur d’obtention des informations du système de fichiers pour /media/drive2 : Aucun fichier ou dossier de ce type
(SALOME_Session_Server_No_Server:3246): Gtk-WARNING **: 21:06:35.457: Failed to measure available space: Erreur d’obtention des informations du système de fichiers pour /media/drive2 : Aucun fichier ou dossier de ce type

It seems, the save file dialogue itself crashes and takes the whole SM with it. It tries to measure the size of a mounted drive?
Interestingly, all the open file dialogoues in Asterstudy (liken LIRE_MAILLAGE) are not affected:

Notice, these dialogues are in french, which I do not mind, but they are clearly different from all the others in SM2024. All these work on the other system mentioned above. Strange!

This is what a standard open file window looks like in SM2024:

Mario.

hberro

Hi Mario,

Try binding your /media folder for singularity using

-B /media

When calling the SM launcher.

mf

eedzo
Hello,

yes, container, prerequisites and code_aster_mpi don't match yet. The best I could do was SM2022 and CA16.7mpi.

Mario.

AsterO'dactyle

Hello,

I've managed to compile a 17.2 MPI version on container 2024 without any problems.

1/ Pulling src and devtools repositories from Gitlab
2/ Launch the container in shell mode (add --shell)
3/ Go to ~/dev/codeaster/src
4/ ./waf configure
5/ ./waf install test -n zzzz506c

The random test (with two CPUs) runs and is OK.

mf

AsterO'dactyle
Hello,

apart from the test in the command line, have you tried launching something in SM2024? I got the error in message #11,

Mario.

AsterO'dactyle

k_zurawski

AsterO'dactyle So if it's working on your system, could you please share what version of system, singularity, system language, and hardware are you using?

AsterO'dactyle

Integrated post-treatment in AsterStudy is ok too

eedzo

AsterO'dactyle
When 'Number of MPI CPU' is set to greater than 1 this is the error.