ORA-39126: Worker unexpected fatal error in KUPW$WORKER.STATS_LOAD [MARKER]

Issue: I was doing an import using datapump and encountered the below error.

Database version: 12.1.0.2.0

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.STATS_LOAD [MARKER]

ORA-30926: unable to get a stable set of rows in the source tables

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95

ORA-06512: at "SYS.KUPW$WORKER", line 11252

----- PL/SQL Call Stack -----

  object      line  object

  handle    number  name

0xe6850a88     27109  package body SYS.KUPW$WORKER

0xe6850a88     11279  package body SYS.KUPW$WORKER

0xe6850a88     24279  package body SYS.KUPW$WORKER

0xe6850a88     20683  package body SYS.KUPW$WORKER

0xe6850a88      4545  package body SYS.KUPW$WORKER

0xe6850a88     12056  package body SYS.KUPW$WORKER

0xe6850a88      2081  package body SYS.KUPW$WORKER

0x3584ffed0         2  anonymous block

Solution:

We have a work around to this issue. Reimport the data with impdp and use an additional parameter EXCLUDE=STATISTICS.

I was able to import the data successfully and post the import I have gathered the stats using the DBMS_STATS package.

How to use OEM to run a script against many databases

We had a requirement to create a particular user in all our 500+ databases. Instead of creating the user individually we thought of leveraging the power of OEM to do so and we succeeded in doing so by just a few clicks.

The steps followed are as followed:
1. Develop a script you want to get it executed.(sample provided below)
2. Go to Enterprise --> Job --> Library
3. In the "create library job" section select "SQL Script" and click "Go"
4. In the "General" section provide the "Name" of Job
5. Provide Target Type as "Database Instance"
6. Click "Parameters" and specify the SQL Script over there (sample provided below)
7. Click "Credentials" and specify the "Database" and "Host" credentials 
8. Click "Schedule" and specify how often you want to run the job, for me it was "One Time (Immediately)"
9. Finally "Save the job" to library

10. Once the job is saved to the Library, Select the job and click the "Submit" button
11. On the new page click "Add" to add the targets(in our case we have selected database instances) and then "Submit" the Job.



Sample SQL:
WHENEVER SQLERROR EXIT FAILURE;
DECLARE
cursor c1 is select DATABASE_ROLE from v$database;
db_role VARCHAR2(40);
BEGIN
open c1;
fetch c1 INTO db_role;
IF db_role='PRIMARY' THEN
execute immediate 'create user test identified by TesT_1_23';
execute immediate 'grant connect, resource to test';
dbms_output.put_line('created TEST user');
ELSE
dbms_output.put_line('STANDBY DATABSE');
END IF;
END;
/

EMD pingOMS error: unable to connect to http server [handshake has no peer]

While upgrading the OEM Agent from 12c to 13c, I was getting the below error. 

I tried all sort of scenarios(such as securing agent, resync agent etc) but all the in vain.

Finally got to know that its due to BUG 22568679 : After upgrade of agents from 12c (12.1.0.5) to 13c (13.1.0.0.0) Agents get stuck 

This issue is encountered if we are using Custom Certificate. This issue is caused due to the wallet checks being more strict between JDK 1.6 and JDK 1.7

Solution: 
To resolve the issue please apply Patch 22568679: After upgrade of agents from 12c (12.1.0.5) to 13c (13.1.0.0.0) Agents get stuck 


Error encountered:

linux01.test.com(oracle:NONE)/u03/app/oracle/agent_inst/bin: ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /u03/app/oracle/agent_inst
Agent Log Directory    : /u03/app/oracle/agent_inst/sysman/log
Agent Binaries         : /u03/app/oracle/agent_13.1.0.0.0
Core JAR Location      : /u03/app/oracle/agent_13.1.0.0.0/jlib
Agent Process ID       : 1769
Parent Process ID      : 1668
Agent URL              : https://linux01:1830/emd/main/
Local Agent URL in NAT : https://linux01:1830/emd/main/
Repository URL         : https://omshost.test.com:4903/empbs/upload
Started at             : 2016-05-10 23:07:34
Started by user        : oracle
Operating System       : Linux version 2.6.32-504.16.2.el6.x86_64 (amd64)
Number of Targets      : 16
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 9
Size of XML files pending upload(MB)         : 0.01
Available disk space on upload filesystem    : 93.07%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable
Last attempted heartbeat to OMS              : 2016-05-10 23:11:24
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2016-05-10 23:11:54

---------------------------------------------------------------
Agent is Running and Ready
linux01.test.com(oracle:NONE)/u03/app/oracle/agent_inst/bin: ./emctl pingOMS
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD pingOMS error: unable to connect to http server at https://omshost.test.com:4903/empbs/upload. [handshake has no peer]


linux01.test.com(oracle:NONE)/u03/app/oracle/agent_inst/bin: ./emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)