결론만 말하자면, virtualbox에서 가상 cpu의 갯수는 아래의 곱으로 설정을 해주어야 한다.
물리 서버에서 한다면 하이퍼 쓰레드랑 고려해서 적절하게 해주면 될 듯.
아래의 값이 기본값인데 openhpc 에서는 좀 높게 설정하네?
Sockets=1 CoresPerSocket=1 ThreadsPerCore=1
/etc/slurm/slurm.conf.ohpc
Sockets=2 CoresPerSocket=8 ThreadsPerCore=2
Socket은 TCP랑은 1도 상관없는 물리적인 CPU 소켓 갯수를 의미한다.
요즘 추세야 1cpu 멀티코어니까 1로 해도 무방할듯하고
CoresPerSocket은 1개 물리 CPU에 들어있는 physical CPU의 갯수
ThreadsPerCore는 intel 기준 HT 사용시 2로 1개 코어에서 사용하는 쓰레드 갯수를 의미한다.
Sockets Number of physical processor sockets/chips on the node (e.g. "2"). If Sockets is omitted, it will be inferred from CPUs, CoresPerSocket, and ThreadsPerCore. NOTE: If you have multi-core processors, you will likely need to specify these parameters. Sockets and SocketsPerBoard are mutually exclusive. If Sockets is specified when Boards is also used, Sockets is interpreted as SocketsPerBoard rather than total sockets. The default value is 1.
CoresPerSocket Number of cores in a single physical processor socket (e.g. "2"). The CoresPerSocket value describes physical cores, not the logical number of processors per socket. NOTE: If you have multi-core processors, you will likely need to specify this parameter in order to optimize scheduling. The default value is 1.
ThreadsPerCore Number of logical threads in a single physical core (e.g. "2"). Note that the Slurm can allocate resources to jobs down to the resolution of a core. If your system is configured with more than one thread per core, execution of a different job on each thread is not supported unless you configure SelectTypeParameters=CR_CPU plus CPUs; do not configure Sockets, CoresPerSocket or ThreadsPerCore. A job can execute a one task per thread from within one job step or execute a distinct job step on each of the threads. Note also if you are running with more than 1 thread per core and running the select/cons_res or select/cons_tres plugin then you will want to set the SelectTypeParameters variable to something other than CR_CPU to avoid unexpected results. The default value is 1.
CPUs: Count of processors on each compute node. If CPUs is omitted, it will be inferred from: Sockets, CoresPerSocket, and ThreadsPerCore.
Sockets: Number of physical processor sockets/chips on the node. If Sockets is omitted, it will be inferred from: CPUs, CoresPerSocket, and ThreadsPerCore.
CoresPerSocket: Number of cores in a single physical processor socket. The CoresPerSocket value describes physical cores, not the logical number of processors per socket.
ThreadsPerCore: Number of logical threads in a single physical core.
# srun mpi_hello_world srun: error: openhpc-1: task 0: Exited with exit code 2 slurmstepd: error: couldn't chdir to `/root/src/mpitutorial/tutorials/mpi-hello-world/code': No such file or directory: going to /tmp instead slurmstepd: error: execve(): mpi_hello_world: No such file or directory
헐...?!?!?
vnfs를 다시 만들어야 하게 생겼네 ㅠㅠ
# srun mpi_hello_world
srun: error: openhpc-1: task 0: Exited with exit code 127
mpi_hello_world: error while loading shared libraries: libmpi.so.40: cannot open shared object file: No such file or directory
# ldconfig -v | grep mpi
ldconfig: Can't stat /libx32: No such file or directory
ldconfig: Path `/usr/lib' given more than once
ldconfig: Path `/usr/lib64' given more than once
ldconfig: Can't stat /usr/libx32: No such file or directory
/opt/ohpc/pub/mpi/mpich-ucx-gnu9-ohpc/3.3.2/lib:
libmpicxx.so.12 -> libmpicxx.so.12.1.8
libmpi.so.12 -> libopa.so
libmpifort.so.12 -> libmpifort.so.12.1.8
이제는 libmpi.so.12는 사라지고 libmpi.so.40이 없다고 나오는군.. 흐음..
$ srun mpi_hello_world
mpi_hello_world: error while loading shared libraries: libmpi.so.40: cannot open shared object file: No such file or directory
srun: error: openhpc-1: task 0: Exited with exit code 127
$ srun mpi_hello_world
mpi_hello_world: error while loading shared libraries: libgfortran.so.5: cannot open shared object file: No such file or directory
srun: error: openhpc-1: task 0: Exited with exit code 127
libgfortran.i686 : Fortran runtime libquadmath-devel.x86_64 : GCC __float128 support ucx.x86_64 : UCX is a communication library implementing high-performance messaging
아무튼 2개 노드에 1개의 태스크를 하라니 1개는 할당을 못하고 openhpc-1에서만 구동한 것 같고
$ srun -N 2 -n 1 mpi_hello_world srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1 Hello world from processor openhpc-1, rank 0 out of 1 processors
2개 노드에 2개 태스크 하라니 openhpc-1,openhpc-2 노드에서 각각 하나씩 실행한 듯.
$ srun -N 2 -n 2 mpi_hello_world Hello world from processor openhpc-1, rank 0 out of 1 processors Hello world from processor openhpc-2, rank 0 out of 1 processors
다만 2개 노드에 3개 하라니까 그 숫자 이상부터는 task로 할당되는데 실행이 안되는걸 보면 또 설정 문제인가.. ㅠㅠ
$ srun -N 2 -n 3 mpi_hello_world srun: Requested partition configuration not available now srun: job 84 queued and waiting for resources ^Csrun: Job allocation 84 has been revoked srun: Force Terminated job 84
slurmd랑 munged가 좀 튀는데..
너무 순식간에 끝나는 애라 그런가 top에 잡히지도 않는다.
+
서버측의 /var/log/slurmctld.log 를 확인하니
시스템에서 구성된 리소스 보다 크게 잡을순 없다고..
[2020-12-23T02:40:11.527] error: Node openhpc-1 has low socket*core*thread count (1 < 4) [2020-12-23T02:40:11.527] error: Node openhpc-1 has low cpu count (1 < 4) [2020-12-23T02:40:11.527] error: _slurm_rpc_node_registration node=openhpc-1: Invalid argument
+
virtualbox에서 cpu를 4개로 올려주고 설정 바꾸어서 정상작동 확인
$ srun -N 2 -n 4 mpi_hello_world Hello world from processor openhpc-1, rank 0 out of 1 processors Hello world from processor openhpc-1, rank 0 out of 1 processors Hello world from processor openhpc-2, rank 0 out of 1 processors Hello world from processor openhpc-2, rank 0 out of 1 processors
저 rank는 멀까..
$ srun -N 2 -n 8 mpi_hello_world Hello world from processor openhpc-2, rank 0 out of 1 processors Hello world from processor openhpc-2, rank 0 out of 1 processors Hello world from processor openhpc-1, rank 0 out of 1 processors Hello world from processor openhpc-2, rank 0 out of 1 processors Hello world from processor openhpc-1, rank 0 out of 1 processors Hello world from processor openhpc-1, rank 0 out of 1 processors Hello world from processor openhpc-1, rank 0 out of 1 processors Hello world from processor openhpc-2, rank 0 out of 1 processors
아무튼 사용 가능한 코어 갯수를 넘어가면 아래와 같이 무기한 대기가 걸려
사실상 실행을 못하게 되는 것 같기도 하다?
$ srun -N 2 -n 12 mpi_hello_world srun: Requested partition configuration not available now srun: job 92 queued and waiting for resources ^Csrun: Job allocation 92 has been revoked srun: Force Terminated job 92
# srun -n 2 -N 2 --pty /bin/bash srun: Required node not available (down, drained or reserved) srun: job 5 queued and waiting for resources ^Csrun: Job allocation 5 has been revoked srun: Force Terminated job 5
# sinfo -all Tue Dec 22 02:54:55 2020 PARTITION AVAIL TIMELIMIT JOB_SIZE ROOT OVERSUBS GROUPS NODES STATE NODELIST normal* up 1-00:00:00 1-infinite no EXCLUSIV all 2 drained openhpc-[1-2]
# sacct -a JobID JobName Partition Account AllocCPUS State ExitCode ------------ ---------- ---------- ---------- ---------- ---------- -------- 2 bash normal (null) 0 CANCELLED 0:0 3 bash normal (null) 0 CANCELLED 0:0 4 bash normal (null) 0 CANCELLED 0:0 5 bash normal (null) 0 CANCELLED 0:0 6 mpi_hello+ normal (null) 0 CANCELLED 0:0
WHAT ARE THE DIFFERENCES? The difference between the two kinds consist of how the importing tool makes the FMU step forward in time.
Models that represent physical quantities such as temperature or velocity are generally described by differential equations. The way that the quantities change with time depend on the value of the quantities themselves (the state), and what external influence they are subjected to (the inputs).
구글 번역 만세!
그나저나 functional mock-up에서 functional이 기능이 아니라 함수적 이라는 느낌이 점점드는건 왜일까...
차이점은 무엇입니까? 두 종류의 차이점은 가져 오기 도구를 사용하여 FMU가 시간을 앞당기는 방식으로 구성됩니다.
온도 또는 속도와 같은 물리량을 나타내는 모델은 일반적으로 미분 방정식으로 설명됩니다. 수량이 시간에 따라 변하는 방식은 수량 자체 (상태)의 값과 이들이받는 외부 영향 (입력)에 따라 다릅니다.
Model Exchange (ME) – ME FMUs represent the dynamical systems by differential equations. To simulate the system, the importing tool needs to connect the FMU to a numerical solver. The solver sets the FMU internal state, asks for the state derivatives, and determines the step size and how to compute the state at the next time step. Co-Simulation (CS) – CS FMUs contain their own numerical solver. The importing tool sets the FMU inputs, asks the FMU to step forward a given time, and reads the FMU output after the step is completed.
구글 번역 만세2!
model exchange와 Co-Simulation 두가지가 존재하는데, solver의 위치가 어디에 있냐에 따라서 다른걸로 보이기도 한다.
결국은 솔버와 모델 두가지를 작성을 해야 FMU가 작동된다는 것 같은데...?
모델 교환 (ME ) – ME FMU는 미분 방정식으로 동적 시스템을 나타냅니다. 시스템을 시뮬레이션하려면 가져 오기 도구가 FMU를 수치 솔버에 연결해야합니다. 솔버는 FMU 내부 상태를 설정하고 상태 미분을 요청하고 단계 크기와 다음 시간 단계에서 상태를 계산하는 방법을 결정합니다. 공동 시뮬레이션 (CS) – CS FMU에는 자체 수치 솔버가 포함되어 있습니다. 가져 오기 도구는 FMU 입력을 설정하고 FMU에 지정된 시간만큼 앞으로 이동하도록 요청하며 단계가 완료된 후 FMU 출력을 읽습니다.