OPATCHAUTO-72059: Supplied patch is invalid for current platform.

Issue: While applying GRID patch, opatchauto was throwing below error, environment linux x86_64


# $GRID_HOME/OPatch/opatchauto apply /cme/oradm/software/30070242/26983807 -analyze -oh /opt/grid/product/12.1.0/grid_1

OPatchauto session is initiated at Thu Nov  7 08:35:40 2019
OPATCHAUTO-72059: Invalid patch.
OPATCHAUTO-72059: Supplied patch is invalid for current platform.
OPATCHAUTO-72059: Please provide a correct patch location.

OPatchauto session completed at Thu Nov  7 08:35:42 2019
Time taken to complete the session 0 minute, 2 seconds

opatchauto bootstrapping failed with error code 255.


Solution: Noticed that the "oraclehomeproperties.xml" file in Below location has missing highlighted entries. I corrected those by adding it and the opatchauto worked fine.

cd $GRID_HOME/inventory/ContentsXML

# vi oraclehomeproperties.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<ORACLEHOME_INFO>
   <GUID>638047284.418901274</GUID>
   <HOME CRS="T"/>
   <ARU_PLATFORM_INFO>
      <ARU_ID>226</ARU_ID>
      <ARU_ID_DESCRIPTION>Linux x86-64</ARU_ID_DESCRIPTION>
   </ARU_PLATFORM_INFO>
   <CLUSTER_INFO>
      <LOCAL_NODE NAME="linux01"/>
      <NODE_LIST>
         <NODE NAME="linux01"/>
         <NODE NAME="linux02"/>
      </NODE_LIST>
   </CLUSTER_INFO>
   <PROPERTY_LIST>
      <PROPERTY NAME="ORACLE_BASE" VAL="/opt/oracle"/>
   </PROPERTY_LIST>
</ORACLEHOME_INFO>






# $GRID_HOME/OPatch/opatchauto apply /cme/oradm/software/30070242 -analyze -oh $GRID_HOME

OPatchauto session is initiated at Thu Nov  7 09:00:37 2019

System initialization log file is /opt/grid/product/12.1.0/grid_1/cfgtoollogs/opatchautodb/systemconfig2019-11-07_09-00-40AM.log.

Session log file is /opt/grid/product/12.1.0/grid_1/cfgtoollogs/opatchauto/opatchauto2019-11-07_09-00-53AM.log
The id for this session is K4YH

Executing OPatch prereq operations to verify patch applicability on home /opt/grid/product/12.1.0/grid_1
Patch applicability verified successfully on home /opt/grid/product/12.1.0/grid_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:linux01
CRS Home:/opt/grid/product/12.1.0/grid_1
Version:12.1.0.2.0


==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /cme/oradm/software/30070242/26983807
Log: /opt/grid/product/12.1.0/grid_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-11-07_09-01-01AM_1.log

Patch: /cme/oradm/software/30070242/29938464
Log: /opt/grid/product/12.1.0/grid_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-11-07_09-01-01AM_1.log

Patch: /cme/oradm/software/30070242/29938481
Log: /opt/grid/product/12.1.0/grid_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-11-07_09-01-01AM_1.log

Patch: /cme/oradm/software/30070242/29972716
Log: /opt/grid/product/12.1.0/grid_1/cfgtoollogs/opatchauto/core/opatch/opatch2019-11-07_09-01-01AM_1.log



OPatchauto session completed at Thu Nov  7 09:01:19 2019

ORA-29913: error in executing ODCIEXTTABLEOPEN callout ORA-29400: data cartridge error

I was doing import of a table and started noticing the below error. Even though the database was same.

Tried different methods but it didn't work. In Oracle support it was mentioned that I may be hitting a bug  and i may have to set NLS_LENGTH_SEMANTICS to char at DB level which was set to byte.

I noticed the table carefully and saw it has a dependency on a Oracle TYPE. 

When i saw its script saw that it was varchar2(20)

CREATE OR REPLACE TYPE ORAUSER1.OFFSET_HDR IS VARRAY(50) of VARCHAR2(20); 

SO I created it manually with an option of "char" in datatype.


SQL> CREATE OR REPLACE TYPE ORAUSER2.OFFSET_HDR IS VARRAY(50) of VARCHAR2(20 char)
  2  /

Type created.

Post that the table was imported successfully.

oracle@linux01:/home/oracle(ORADB)$ impdp / directory=DB_EXP_MOUNT dumpfile=exp_365137_2_%U.dmp logfile=imp_365137_2.log remap_schema=ORAUSER1:ORAUSER2 transform=oid:n parallel=4 cluster=no

Import: Release 12.1.0.2.0 - Production on Thu Jul 12 23:14:07 2018

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Advanced Analytics, Oracle Database Vault and Real Application Testin
Master table "OPS$oracle"."SYS_IMPORT_FULL_05" successfully loaded/unloaded
Starting "OPS$oracle"."SYS_IMPORT_FULL_05":  /******** directory=DB_EXP_MOUNT dumpfile=exp_365137_2_%U.dmp logfile=imp_365137_2.log remap_schema=ORAUSER1:ORAUSER2 transform=oid:n parallel=4 cluster=no parfile=exclude_procobj.par table_exists_action=truncate
Processing object type TABLE_EXPORT/TABLE/PROCACT_INSTANCE
Processing object type TABLE_EXPORT/TABLE/TABLE
Table "ORAUSER2"."RBX_CURVES" exists and has been truncated. Data will be loaded but all dependent metadata will be skipped due to table_exists_action of truncate
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
ORA-31693: Table data object "ORAUSER2"."RBX_CURVES" failed to load/unload and is being skipped due to error:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
ORA-39779: type "ORAUSER2"."OFFSET_HDR" not found or conversion to latest version is not possible
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Job "OPS$oracle"."SYS_IMPORT_FULL_05" completed with 1 error(s) at Thu Jul 12 23:14:10 2018 elapsed 0 00:00:02

EM12c : Authentication Failure while login to GUI console with the Correct Password

I was trying to login to OEM GUI console and was getting the below error, I was pretty sure that my password was correct and even tried with different user account but every time it was failing with “Authentication failed. If problem persists, contact your system administrator”

I checked the emoms.trc logs which had the below errors

omsserver1:/opt/oracle/gc_inst12cR5/em/EMGC_OMS1/sysman/log:(oracle)$ view emoms.trc

Error:
2017-11-07 21:52:31,166 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR audit.AuditLogHandler auditLog.378 - Could not log the audit record java.sql.SQLException: ORA-14400: inserted partition key does not map to any partition
ORA-06512: at "SYSMAN.MGMT_AUDIT", line 919
ORA-06512: at "SYSMAN.MGMT_AUDIT", line 417
ORA-06512: at line 1

2017-11-07 21:52:31,168 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR authz.EMSecurityContext doEMLogout.522 - SQLException caught while destroying auditSessionGuid 5D6C463204EBB157E053B71ED00A7F0F
2017-11-07 22:01:37,537 [[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN  auth.EMRepLoginFilter doFilter.458 - InvalidEMUserException caught in EMRepLoginFilter: Failed to login using repository authentication for user: JOBADM

Solution:
Verified that the job_queue_processes are not set to zero and then executed the below command to add the audit partitions. Verified that I am able to login to console post that.

SQL> alter session  set current_schema=sysman;

Session altered.

SQL> exec mgmt_audit_admin.add_audit_partition;

PL/SQL procedure successfully completed.

ORA-08104: this index object is being online built or rebuilt

Got this error after session was killed (or if we have done a ctrl + C) while doing an index rebuild and then trying to rebuild the index again, its seems the cause is – A session failure during an online index rebuild can leave the data dictionary in a state reflecting a rebuild is ongoing when in fact it is not

SQL> ALTER INDEX ORAUSER.TEST_INDX
REBUILD
TABLESPACE EPS_DATA
STORAGE (
         INITIAL     64M
         NEXT        64M
        )
ONLINE;
ALTER INDEX ORAUSER.TEST_INDX
*
ERROR at line 1:
ORA-08104: this index object 18902057 is being online built or rebuilt

SQL> select obj# ,name from obj$ where OBJ#=18902057;

      OBJ# NAME
---------- ------------------------------
  18902057 TEST_INDX

SQL> select obj#,flags from ind$ where obj#=18902057;

      OBJ#      FLAGS
---------- ----------
  18902057        517


SQL> declare
   lv_ret BOOLEAN;
    begin
    lv_ret :=dbms_repair.online_index_clean(18902057);
    end;
    /

PL/SQL procedure successfully completed.

SQL> select obj#,flags from ind$ where obj#=18902057;

      OBJ#      FLAGS
---------- ----------
  18902057          5

SQL> select obj# ,name from obj$ where OBJ#=18902057;

      OBJ# NAME
---------- ------------------------------
  18902057 TEST_INDX

SQL> ALTER INDEX ORAUSER.TEST_INDX
REBUILD
TABLESPACE EPS_DATA
STORAGE (
         INITIAL     64M
         NEXT        64M
        )
ONLINE;

Index altered.

ORA-20000: ** FAILED TO GET COLUMN STATS FOR THE TABLE

SQL> BEGIN
STAT_MAINTENANCE.GATHER_SCHEMA_STATS_JOB('TESTUSER');
END;
/

BEGIN
*
ERROR at line 1:
ORA-20000: ** FAILED TO SET STAT_TIME FOR THE TABLE **: TESTUSER.TEST_TABLE
ERROR: ORA-20000: ** FAILED TO GET COLUMN STATS FOR THE TABLE **:
TESTUSER.TEST_TABLE Make sure Table has Standard Stats. ERROR: ORA-20000:
Unable to get values for column AVAILABLE_DATE
ORA-06512: at "SYS.DBMS_STATS", line 12211
ORA-06512: at "SYS.DBMS_STATS", line 12224
ORA-06512: at "STAT_MAINTENANCE", line 1072
ORA-06512: at "STAT_MAINTENANCE", line 1086
ORA-06512: at "STAT_MAINTENANCE", line 1171
ORA-06512: at "STAT_MAINTENANCE", line 1186
ORA-06512: at "STAT_MAINTENANCE", line 1645
ORA-06512: at "STAT_MAINTENANCE", line 1779
ORA-06512: at line 2



SQL> select * from user_tab_stat_prefs where table_name='TEST_TABLE';

TABLE_NAME   PREFERENCE_NAME      PREFERENCE_VALUE
------------ -------------------- ----------------------------
TEST_TABLE   ESTIMATE_PERCENT     DBMS_STATS.AUTO_SAMPLE_SIZE
TEST_TABLE   METHOD_OPT           FOR ALL COLUMNS SIZE AUTO,FOR COLUMNS "C1" SIZE 1,"C2" SIZE 1,"C3" SIZE 1,"C4" SIZE 1
  
SQL> select DBMS_STATS.GET_PREFS('METHOD_OPT','TESTUSER','TEST_TABLE') from dual;

DBMS_STATS.GET_PREFS('METHOD_OPT','TESTUSER','TEST_TABLE')
------------------------------------------------------------------------------------
FOR ALL COLUMNS SIZE AUTO,FOR COLUMNS "C1" SIZE 1,"C2" SIZE 1,"C3" SIZE 1,"C4" SIZE 1


Describe the table and check if any of the columns were removed.

SQL> DESC TESTUSER.TEST_TABLE

 Name     Null?    Type
 -------- -------- ----------------------------
 P1       NOT NULL NUMBER(38)
 C1                NUMBER(38)
 T1                VARCHAR2(3 CHAR)
 T2                TIMESTAMP(6)
 C2                VARCHAR2(32 CHAR)
 C3                FLOAT(126)
 C5                VARCHAR2(255 CHAR)
 C7       NOT NULL NUMBER(38)

  --- from PREFERENCE_VALUE, remove column dropped and use it below
SQL> EXEC DBMS_STATS.DELETE_TABLE_PREFS ('TESTUSER', 'TEST_TABLE','METHOD_OPT');


SQL> BEGIN
dbms_stats.Set_table_prefs('TESTUSER', 'TEST_TABLE', 'METHOD_OPT' , 'FOR ALL COLUMNS SIZE AUTO,FOR COLUMNS C1 SIZE 1,C2 SIZE 1,C3 SIZE 1');  
END;
/


SQL> select * from user_tab_stat_prefs where table_name='TEST_TABLE';

TABLE_NAME   PREFERENCE_NAME      PREFERENCE_VALUE
------------ -------------------- --------------------------------------
TEST_TABLE   ESTIMATE_PERCENT     DBMS_STATS.AUTO_SAMPLE_SIZE
TEST_TABLE   METHOD_OPT           FOR ALL COLUMNS SIZE AUTO,FOR COLUMNS "C1" SIZE 1,"C2" SIZE 1,"C3" SIZE 1

SQL> select DBMS_STATS.GET_PREFS('METHOD_OPT','TESTUSER','TEST_TABLE') from dual;

DBMS_STATS.GET_PREFS('METHOD_OPT','TESTUSER','TEST_TABLE')
--------------------------------------------------------------------
FOR ALL COLUMNS SIZE AUTO,FOR COLUMNS C1 SIZE 1,C2 SIZE 1,C3 SIZE 1



SQL> BEGIN
STAT_MAINTENANCE.GATHER_SCHEMA_STATS_JOB('TESTUSER');
END;

/

ORA-15135: The ASM cluster is not in rolling patch state

Issue: In one of our cluster nodes when we were doing a Oracle Bundle patching in rolling fashion(involves GI patching as well), We started hitting the below error. We realized that the CRS Stack was down on the other nodes prior to patching due to which nodes cannot communicate about its patching/rolling state to other nodes (because CRS was down on other nodes) and it ends up failing. 

Solution: To overcome this we had to stop the CRS on first node and then start the patching on the next node.

2017-02-23 23:29:47.453 [ORAAGENT(39105)]CRS-5017: The resource action "ora.asm start" encountered the following error: 
2017-02-23 23:29:47.453+ORA-15135: The ASM cluster is not in rolling patch state. The patch level [3351897854] of instance +ASM2 is not the same as the current cluster patch level [2552957799].
. For details refer to "(:CLSN00107:)" in "/opt/oracle/diag/crs/linux02/crs/trace/ohasd_oraagent_orprod.trc".

When doing a opatch lsinventory on GI home it result in the below, we can verify that the patching levels are different. 

Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux01
3351897854 linux02

In case the CRS stack is kept up on all nodes, the information about the patching state is made aware to the other node and it passes through successfully. Below is the result of opatch lsinventory when the CRS stack was kept up.

Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux02,linux01

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.STATS_LOAD [MARKER]

Issue: I was doing an import using datapump and encountered the below error.

Database version: 12.1.0.2.0

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.STATS_LOAD [MARKER]

ORA-30926: unable to get a stable set of rows in the source tables

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95

ORA-06512: at "SYS.KUPW$WORKER", line 11252

----- PL/SQL Call Stack -----

  object      line  object

  handle    number  name

0xe6850a88     27109  package body SYS.KUPW$WORKER

0xe6850a88     11279  package body SYS.KUPW$WORKER

0xe6850a88     24279  package body SYS.KUPW$WORKER

0xe6850a88     20683  package body SYS.KUPW$WORKER

0xe6850a88      4545  package body SYS.KUPW$WORKER

0xe6850a88     12056  package body SYS.KUPW$WORKER

0xe6850a88      2081  package body SYS.KUPW$WORKER

0x3584ffed0         2  anonymous block

Solution:

We have a work around to this issue. Reimport the data with impdp and use an additional parameter EXCLUDE=STATISTICS.

I was able to import the data successfully and post the import I have gathered the stats using the DBMS_STATS package.

How to use OEM to run a script against many databases

We had a requirement to create a particular user in all our 500+ databases. Instead of creating the user individually we thought of leveraging the power of OEM to do so and we succeeded in doing so by just a few clicks.

The steps followed are as followed:
1. Develop a script you want to get it executed.(sample provided below)
2. Go to Enterprise --> Job --> Library
3. In the "create library job" section select "SQL Script" and click "Go"
4. In the "General" section provide the "Name" of Job
5. Provide Target Type as "Database Instance"
6. Click "Parameters" and specify the SQL Script over there (sample provided below)
7. Click "Credentials" and specify the "Database" and "Host" credentials 
8. Click "Schedule" and specify how often you want to run the job, for me it was "One Time (Immediately)"
9. Finally "Save the job" to library

10. Once the job is saved to the Library, Select the job and click the "Submit" button
11. On the new page click "Add" to add the targets(in our case we have selected database instances) and then "Submit" the Job.



Sample SQL:
WHENEVER SQLERROR EXIT FAILURE;
DECLARE
cursor c1 is select DATABASE_ROLE from v$database;
db_role VARCHAR2(40);
BEGIN
open c1;
fetch c1 INTO db_role;
IF db_role='PRIMARY' THEN
execute immediate 'create user test identified by TesT_1_23';
execute immediate 'grant connect, resource to test';
dbms_output.put_line('created TEST user');
ELSE
dbms_output.put_line('STANDBY DATABSE');
END IF;
END;
/

EMD pingOMS error: unable to connect to http server [handshake has no peer]

While upgrading the OEM Agent from 12c to 13c, I was getting the below error. 

I tried all sort of scenarios(such as securing agent, resync agent etc) but all the in vain.

Finally got to know that its due to BUG 22568679 : After upgrade of agents from 12c (12.1.0.5) to 13c (13.1.0.0.0) Agents get stuck 

This issue is encountered if we are using Custom Certificate. This issue is caused due to the wallet checks being more strict between JDK 1.6 and JDK 1.7

Solution: 
To resolve the issue please apply Patch 22568679: After upgrade of agents from 12c (12.1.0.5) to 13c (13.1.0.0.0) Agents get stuck 


Error encountered:

linux01.test.com(oracle:NONE)/u03/app/oracle/agent_inst/bin: ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /u03/app/oracle/agent_inst
Agent Log Directory    : /u03/app/oracle/agent_inst/sysman/log
Agent Binaries         : /u03/app/oracle/agent_13.1.0.0.0
Core JAR Location      : /u03/app/oracle/agent_13.1.0.0.0/jlib
Agent Process ID       : 1769
Parent Process ID      : 1668
Agent URL              : https://linux01:1830/emd/main/
Local Agent URL in NAT : https://linux01:1830/emd/main/
Repository URL         : https://omshost.test.com:4903/empbs/upload
Started at             : 2016-05-10 23:07:34
Started by user        : oracle
Operating System       : Linux version 2.6.32-504.16.2.el6.x86_64 (amd64)
Number of Targets      : 16
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 9
Size of XML files pending upload(MB)         : 0.01
Available disk space on upload filesystem    : 93.07%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable
Last attempted heartbeat to OMS              : 2016-05-10 23:11:24
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2016-05-10 23:11:54

---------------------------------------------------------------
Agent is Running and Ready
linux01.test.com(oracle:NONE)/u03/app/oracle/agent_inst/bin: ./emctl pingOMS
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD pingOMS error: unable to connect to http server at https://omshost.test.com:4903/empbs/upload. [handshake has no peer]


linux01.test.com(oracle:NONE)/u03/app/oracle/agent_inst/bin: ./emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)

TNS-01155: Incorrectly specified SID_LIST_LISTENER parameter in LISTENER.ORA

Issue: while starting the listener we got the below error.

linux01(oracle:orcl1)/home/oracle: lsnrctl start

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 15-SEP-2015 03:43:02

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Starting /opt/oracle/product/11.2.0.4.RAC/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.4.0 - Production
System parameter file is /opt/oracle/product/11.2.0.4.RAC/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/linux01/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
TNS-01155: Incorrectly specified SID_LIST_LISTENER parameter in LISTENER.ORA
 NL-00303: syntax error in NV string

Listener failed to start. See the error message(s) above...

Solution: In the listener.ora file there were blank lines under the section SID_LIST_LISTENER. We removed the blank line entries and brought up the listener. (please note that this may be one of the scenarios)

Erroneous listener.ora file excerpts below:

SID_LIST_LISTENER =
  (SID_LIST =
  (SID_DESC =
      (GLOBAL_DBNAME = ORCL1)
      (ORACLE_HOME = /opt/oracle/product/11.2.0.4.RAC)
      (SID_NAME = ORCL1)
   )
<<<<< blank line
(SID_DESC =
      (GLOBAL_DBNAME = ORCL2)
      (ORACLE_HOME = /opt/oracle/product/11.2.0.4.RAC)
      (SID_NAME = ORCL2)
)
<<<<< blank line

)

ORA-27300: OS system dependent operation:semget failed with status: 28

linux01.test.com(oracle:RP43DB31)/opt/oracle: srvctl start database -d ORCL
PRCR-1079 : Failed to start resource ora.orcl.db
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0.4.GRD/log/linux01/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'linux01' failed
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0.4.GRD/log/linux02/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'linux02' failed
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy

Solution:
This may be caused by Kernel parameter setting at the server level. Semaphore values may be one of the reasons. Increase the values.

linux01.test.com(oracle:)/opt/oracle: cat /etc/sysctl.conf |grep kernel.sem
kernel.sem = 250 32000 100 128

As root user, edit the /etc/sysctl.conf file and edit the kernel parameters 
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 50000 100 200

and then run the below command to make the values persistent:

/sbin/sysctl -p

Discover host targets manually in OEM 12c - emctl config agent addinternaltargets

In case of silent installation of OEM 12c Agent sometimes the host target monitoring is not picked up and it is not visible in OEM console as well.

When checked in the /u01/app/oracle/agent_inst/sysman/emd/targets.xml file we couldn't find information related to the host.

To fix this we need to run "emctl config agent addInternalTargets". This forces the agent to rediscover the host.

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 12.1.0.4.0
OMS Version            : 12.1.0.4.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/app/oracle/agent_inst
Agent Log Directory    : /u01/app/oracle/agent_inst/sysman/log
Agent Binaries         : /u01/app/oracle/core/12.1.0.4.0
Agent Process ID       : 3770
Parent Process ID      : 3494
Agent URL              : https://linux01:1830/emd/main/
Local Agent URL in NAT : https://linux01:1830/emd/main/
Repository URL         : https://oem12c.test.com:1159/empbs/upload
Started at             : 2015-08-18 13:46:01
Started by user        : oracle
Operating System       : Linux version 2.6.32-573.1.1.el6.x86_64 (amd64)
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 94.03%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2015-08-19 00:38:02
Last successful heartbeat to OMS             : 2015-08-19 00:38:02
Next scheduled heartbeat to OMS              : 2015-08-19 00:39:02

---------------------------------------------------------------
Agent is Running and Ready

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
[linux01:1830, oracle_emd]

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl config agent addInternalTargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

linux01(oracle:NONE)/u01/app/oracle/agent_inst/bin: ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
[linux01, host]
[linux01:1830, oracle_emd]

How to change SYSMAN Password in OEM 12c

1. Stop all OMS instances running emctl stop oms.

$OMS_Home/bin/emctl stop oms

Execute the same command on all the OMS machines including the primary OMS machine. Do not include '-all' as the Admin Server needs to be up during this operation.

2. Modify the SYSMAN password:

$OMS_Home/bin/emctl config oms -change_repos_pwd

Provide Repository User's Current Password and Repository User's New Password in command prompt

3. Stop the Admin server on the primary OMS machine and re-start all the OMS:

$OMS_Home/bin/emctl stop oms –all

4. Restart all the Management Services:

$OMS_Home/bin/emctl start oms


oem12c.us.com(oracle)/u01/oracle/product/Middleware/oms/bin: emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

oem12c.us.com(oracle)/u01/oracle/product/Middleware/oms/bin: emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem12c.us.com
HTTP Console Port          : 7789
HTTPS Console Port         : 7801
HTTP Upload Port           : 4890
HTTPS Upload Port          : 4901
EM Instance Home           : /u01/oracle/product/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/oracle/product/gc_inst/em/EMGC_OMS1/sysman/log
SLB or virtual hostname: oemtest.us.com
HTTPS SLB Upload Port : 4901
HTTPS SLB Console Port : 7801
Agent Upload is unlocked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oemtest.us.com:7801/em
Upload URL: https://oemtest.us.com:4901/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : oem12c.us.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: oem12c.us.com
WebTier is Down
Oracle Management Server is Down

BI Publisher is not configured to run on this host.
oem12c.us.com(oracle)/u01/oracle/product/Middleware/oms/bin: emctl config oms -change_repos_pwd
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Repository User's Current Password :
Enter Repository User's New Password :

Changing passwords in backend ...
Passwords changed in backend successfully.
Updating repository password in Credential Store...
Successfully updated Repository password in Credential Store.
Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully changed repository password.


oem12c.us.com(oracle)/u01/oracle/product/Middleware/oms/bin: emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down


oem12c.us.com(oracle)/u01/oracle/product/Middleware/oms/bin: emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

oem12c.us.com(oracle)/u01/oracle/product/Middleware/oms/bin: emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem12c.us.com
HTTP Console Port          : 7789
HTTPS Console Port         : 7801
HTTP Upload Port           : 4890
HTTPS Upload Port          : 4901
EM Instance Home           : /u01/oracle/product/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/oracle/product/gc_inst/em/EMGC_OMS1/sysman/log
SLB or virtual hostname: oemtest.us.com
HTTPS SLB Upload Port : 4901
HTTPS SLB Console Port : 7801
Agent Upload is unlocked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oemtest.us.com:7801/em
Upload URL: https://oemtest.us.com:4901/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : oem12c.us.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: oem12c.us.com
WebTier is Up
Oracle Management Server is Up

BI Publisher is not configured to run on this host.

How to reset spfile parameter value

How to reset parameter value in oracle spfile

The following command can be used reset any parameter specified in spfile. Please note that it will only delete the entry from the spfile but the original value set at instance level will remain the same.

ALTER SYSTEM RESET <parameter_name>;
ALTER SYSTEM RESET <parameter_name> SCOPE=SPFILE SID='INSTANCE_NAME'; (incase of RAC)

SQL> select * from v$spparameter where name like 'processes';

INST_ID SID   NAME       TYPE     VALUE  DISPLAY_VALUE  ISSPEC ORDINAL UPDATE_COMMENT
------- ----- ---------- -------- ------ -------------- ------ ------- --------------
      1 *     processes  integer  300    300            TRUE         1
      1 ORCL1 processes  integer  300    300            TRUE         1
      1 ORCL2 processes  integer  300    300            TRUE         1
 
SQL> alter system reset processes scope=spfile sid='ORCL2';

System altered.

SQL> select * from v$spparameter where name like 'processes';

INST_ID SID   NAME       TYPE     VALUE  DISPLAY_VALUE  ISSPEC ORDINAL UPDATE_COMMENT
------- ----- ---------- -------- ------ -------------- ------ ------- --------------
      1 *     processes  integer  300    300            TRUE         1
      1 ORCL1 processes  integer  300    300            TRUE         1
 
SQL> alter system reset processes scope=spfile sid='ORCL2';

System altered.

SQL> select * from v$spparameter where name like 'processes';

INST_ID SID   NAME       TYPE     VALUE  DISPLAY_VALUE  ISSPEC ORDINAL UPDATE_COMMENT
------- ----- ---------- -------- ------ -------------- ------ ------- --------------
      1 *     processes  integer  300    300            TRUE         1