Thursday, August 20, 2015

ORA-27300: OS system dependent operation:semget failed with status: 28

linux01.test.com(oracle:RP43DB31)/opt/oracle: srvctl start database -d ORCL
PRCR-1079 : Failed to start resource ora.orcl.db
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0.4.GRD/log/linux01/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'linux01' failed
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0.4.GRD/log/linux02/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'linux02' failed
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy

Solution:
This may be caused by Kernel parameter setting at the server level. Semaphore values may be one of the reasons. Increase the values.

linux01.test.com(oracle:)/opt/oracle: cat /etc/sysctl.conf |grep kernel.sem
kernel.sem = 250 32000 100 128

As root user, edit the /etc/sysctl.conf file and edit the kernel parameters 
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 50000 100 200

and then run the below command to make the values persistent:

/sbin/sysctl -p

Wednesday, August 19, 2015

Discover host targets manually in OEM 12c - emctl config agent addinternaltargets

In case of silent installation of OEM 12c Agent sometimes the host target monitoring is not picked up and it is not visible in OEM console as well.

When checked in the /u01/app/oracle/agent_inst/sysman/emd/targets.xml file we couldn't find information related to the host.

To fix this we need to run "emctl config agent addInternalTargets". This forces the agent to rediscover the host.

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.4.0
OMS Version            : 12.1.0.4.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent_inst
Agent Log Directory    : /u01/app/oracle/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/core/12.1.0.4.0
Agent Process ID       : 3770
Parent Process ID      : 3494
Agent URL              : https://linux01:1830/emd/main/
Local Agent URL in NAT : https://linux01:1830/emd/main/
Repository URL         : https://oem12c.test.com:1159/empbs/upload
Started at             : 2015-08-18 13:46:01
Started by user        : oracle
Operating System       : Linux version 2.6.32-573.1.1.el6.x86_64 (amd64)
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 94.03%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-08-19 00:38:02
Last successful heartbeat to OMS             : 2015-08-19 00:38:02
Next scheduled heartbeat to OMS              : 2015-08-19 00:39:02

---------------------------------------------------------------
Agent is Running and Ready

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
[linux01:1830, oracle_emd]

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl config agent addInternalTargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
[linux01, host]
[linux01:1830, oracle_emd]