ORA-27302 skgpspawn3
One of RAC instances crashed in the night and we saw ORA-27300, ORA-27301 and ORA-27302 in the alert log before crashing.
2022-11-21T22:28:01.339829-08:00
Process startup failed, error stack:
2022-11-21T22:28:01.339829-08:00
Errors in file /u01/app/oracle/diag/rdbms/orclcdb/ORCLCDB2/trace/ORCLCDB2_psp0_11075874.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
fork failed with status: 12 means that Process Spawner Process (PSP) tries to spawn some processes, but no memory left for new processes, so it turns to SWAP, unfortunately, it failed due to "Not enough space" on paging space (SWAP).
Solution
Insufficient SWAP is the major problem, but it might not be the root cause. The formal way to solve the insufficient SWAP is to raise SWAP space, then start the database again.
If you decide to increase SWAP size, you may refer to the following posts:
The root cause should also be considered if SWAP is already sufficiently large. Sometimes, there's some process was taking too much memory and SWAP. If the process belongs to Oracle product, you should apply useful patches to the database or clusterware.