Node Manager associated with machine is not reachable | Weblogic

Environment Details:
OS: Linux Server 5.8 64bit
Version: Weblogic Server 12.1.2
Installation location: /u01/app/oracle/product/fmw
DOMAIN_HOME: /u01/app/oracle/product/fmw/user_projects/domains/base_domain

Issue: After starting up the Oracle WebLogic 12c AdminServer server, when trying to start any of our managed servers from console page, we were getting the below error. Verified that the node manager was running but still we were encountering this issue.

<Oct 16, 2013 7:26:10 AM GMT> <WARNING> <Configuration error while reading domain directory>
weblogic.nodemanager.common.ConfigException: The domain 'base_domain' at 'null' was not registered in the nodemanager.domains file and
ic domain registration is not supported. Please register the domain in the nodemanager.domains file.
        at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:285)
        at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:277)
        at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:131)
        at weblogic.nodemanager.server.Handler.run(Handler.java:81)
        at java.lang.Thread.run(Thread.java:724)

Solution: Manually edit the nodemanager.domains and add a domain in that.

Weblogic 12.1.2 | Node Manager Startup Issues

Environment Details:
OS: Linux Server 5.8 64bit
Version: Weblogic Server 12.1.2
Installation location: /u01/app/oracle/product/fmw

Issue: Node Manager was throwing error when we invoked the script startNodeManager.sh script from $WLS_HOME/server/bin

[oraem@linux01 bin]$ pwd
[oraem@linux01 bin]$ /u01/app/oracle/product/fmw/wlserver/server/bin
[oraem@linux01 bin]$ ./startNodeManager.sh 
+ CLASSPATH=/u01/app/oracle/product/jdk1.7.0_40/lib/tools.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic_sp.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/webservices.jar:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/product/fmw/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar::/u01/app/oracle/product/fmw/wlserver/..
+ '[' '!' -z '' ']'
+ '[' '!' -z '' ']'
+ export CLASSPATH
+ export PATH
+ cd /u01/app/oracle/product/fmw/wlserver/../oracle_common/common/nodemanager
+ set -x
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ /u01/app/oracle/product/jdk1.7.0_40/bin/java -client -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/u01/app/oracle/product/fmw/wlserver/../coherence -Dbea.home=/u01/app/oracle/product/fmw/wlserver/.. -Xverify:none -Djava.endorsed.dirs=/u01/app/oracle/product/jdk1.7.0_40/jre/lib/endorsed:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/endorsed -Djava.security.policy=/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/u01/app/oracle/product/jdk1.7.0_40 weblogic.NodeManager -v
<Oct 16, 2013 4:29:07 AM GMT> <INFO> <Loading domains file: /u01/app/oracle/product/fmw/oracle_common/common/nodemanager/nodemanager.domains>
<Oct 16, 2013 4:29:07 AM GMT> <WARNING> <Domains file not found: /u01/app/oracle/product/fmw/oracle_common/common/nodemanager/nodemanager.domains>
<Oct 16, 2013 4:29:08 AM GMT> <INFO> <Loading identity key store: FileName=/u01/app/oracle/product/fmw/oracle_common/common/nodemanager/security/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Oct 16, 2013 4:29:08 AM GMT> <SEVERE> <Fatal error in NodeManager server>
weblogic.nodemanager.common.ConfigException: Identity key store file not found: /u01/app/oracle/product/fmw/oracle_common/common/nodemanager/security/DemoIdentity.jks
        at weblogic.nodemanager.server.SSLConfig.loadKeyStoreConfig(SSLConfig.java:170)
        at weblogic.nodemanager.server.SSLConfig.access$000(SSLConfig.java:32)
        at weblogic.nodemanager.server.SSLConfig$1.run(SSLConfig.java:111)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLConfig.<init>(SSLConfig.java:108)
        at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:121)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:383)
        at weblogic.NodeManager.main(NodeManager.java:31)

+ set +x


Solution: In Weblogic 12.1.2, we have to start the node manager from DOMAIN_HOME/bin using the startNodeManager.sh script

[oraem@linux01 bin]$ pwd
[oraem@linux01 bin]$ /u01/app/oracle/product/fmw/user_projects/domains/base_domain/bin
[oraem@linux01 bin]$ ./startNodeManager.sh CLASSPATH=/u01/app/oracle/product/jdk1.7.0_40/lib/tools.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic_sp.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/webservices.jar:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/product/fmw/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar::/u01/app/oracle/product/fmw/wlserver/..
+ '[' '!' -z '' ']'
+ '[' '!' -z /u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/oracle.jps_12.1.2/jps-manifest.jar ']'
+ CLASSPATH=/u01/app/oracle/product/jdk1.7.0_40/lib/tools.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic_sp.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic.jar:/u01/app/oracle/product/fmw/wlserver/server/lib/webservices.jar:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/product/fmw/wlserver/modules/features/oracle.wls.common.nodemanager_1.0.0.0.jar::/u01/app/oracle/product/fmw/wlserver/..:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/oracle.jps_12.1.2/jps-manifest.jar
+ export CLASSPATH
+ export PATH
+ cd /u01/app/oracle/product/fmw/user_projects/domains/base_domain/nodemanager
+ set -x
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ /u01/app/oracle/product/jdk1.7.0_40/bin/java -client -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/u01/app/oracle/product/fmw/wlserver/../coherence -Dbea.home=/u01/app/oracle/product/fmw/wlserver/.. -Doracle.security.jps.config=/u01/app/oracle/product/fmw/user_projects/domains/base_domain/config/fmwconfig/jps-config-jse.xml -Djava.security.policy=/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic.policy -Dcommon.components.home=/u01/app/oracle/product/fmw/oracle_common -Dopss.version=12.1.2 -Dweblogic.RootDirectory=/u01/app/oracle/product/fmw/user_projects/domains/base_domain -Xverify:none -Djava.endorsed.dirs=/u01/app/oracle/product/jdk1.7.0_40/jre/lib/endorsed:/u01/app/oracle/product/fmw/wlserver/../oracle_common/modules/endorsed -Djava.security.policy=/u01/app/oracle/product/fmw/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/u01/app/oracle/product/jdk1.7.0_40 weblogic.NodeManager -v
<Oct 16, 2013 7:48:02 AM UTC> <INFO> <Loading domains file: /u01/app/oracle/product/fmw/user_projects/domains/base_domain/nodemanager/nodemanager.domains>
<Oct 16, 2013 7:48:03 AM UTC> <INFO> <Loading identity key store: FileName=kss://system/demoidentity, Type=kss, PassPhraseUsed=true>
Oct 16, 2013 7:48:05 AM org.hibernate.validator.util.Version <clinit>
INFO: Hibernate Validator 12.1.2.0.0
Oct 16, 2013 7:48:05 AM org.hibernate.validator.engine.resolver.DefaultTraversableResolver detectJPA
INFO: Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.
Oct 16, 2013 7:48:10 AM oracle.security.jps.internal.idstore.util.LibOvdUtil pushLdapNamesToLibOvd
INFO: Pushed ldap name and types info to libOvd. Ldaps : u01apporacleproductfmwuser_projectsdomainsbase_domainconfigfmwconfigsystem-jazn-data.xmlidstore.xml.provideridstore.xml.
<Oct 16, 2013 7:48:16 AM UTC> <INFO> <Loaded NodeManager configuration properties from '/u01/app/oracle/product/fmw/user_projects/domains/base_domain/nodemanager/nodemanager.properties'>
Node manager v12.1.2

Configuration settings:

NodeManagerHome=/u01/app/oracle/product/fmw/user_projects/domains/base_domain/nodemanager
ListenAddress=linux01.us.oracle.com
ListenPort=5556
ListenBacklog=50
SecureListener=true
AuthenticationEnabled=true
NativeVersionEnabled=true
CrashRecoveryEnabled=false
JavaHome=/u01/app/oracle/product/jdk1.7.0_40
StartScriptEnabled=true
StopScriptEnabled=false
StartScriptName=startWebLogic.sh
StopScriptName=
CoherenceStartScriptEnabled=false
CoherenceStartScriptName=null
LogFile=/u01/app/oracle/product/fmw/user_projects/domains/base_domain/nodemanager/nodemanager.log
LogLevel=INFO
LogLimit=0
LogCount=1
LogAppend=true
LogToStderr=true
LogFormatter=weblogic.nodemanager.server.LogFormatter
DomainsFile=/u01/app/oracle/product/fmw/user_projects/domains/base_domain/nodemanager/nodemanager.domains
DomainsFileEnabled=true
StateCheckInterval=500
UseMACBroadcast=false
DomainRegistrationEnabled=false
DomainsDirRemoteSharingEnabled=false
RotatedFileCount=7
FileSizeKB=500
NumberOfFilesLimited=false
RotationTimeStart=00:00
RotationType=bySize
FileTimeSpan=24
FileTimeSpanFactor=3600000
ProcessDestroyTimeout=20000
Domain name mappings:

base_domain -> /u01/app/oracle/product/fmw/user_projects/domains/base_domain

<Oct 16, 2013 7:48:16 AM UTC> <INFO> <WebLogic Server 12.1.2.0.0  Fri Jun 7 15:16:15 PDT 2013 1530982 WLS_12.1.2.0.0_GENERIC_130607.1100>
<Oct 16, 2013 7:48:16 AM UTC> <INFO> <base_domain> <ManagedServer_2> <Startup configuration properties loaded from "/u01/app/oracle/product/fmw/user_projects/domains/base_domain/servers/ManagedServer_2/data/nodemanager/startup.properties">

<Oct 16, 2013 7:48:16 AM UTC> <INFO> <Secure socket listener started on port 5556, host linux01.us.oracle.com/10.11.12.21>

Oracle RAC 11gR2 Deinstallation Steps

Environment Details:
OS: Linux 5.8
User: oracle
Two Node RAC: linux01, linux02
Database Version: 11.2.0.3.0
Oracle Home: /u01/app/oracle/product/11.2.0/dbhome_1
Grid Home: /u01/app/11.2.0/grid

Steps to be followed to deinstall Oracle RAC
Step 1: Drop the database using the dbca
Step 2: Deinstall the oracle Home using deinstall from $ORACLE_HOME/deinstall
Step 3: Deinstall the Clusterware Software using the deinstall from $GRID_HOME/deinstall
Step 3.a: Run the command which are given at the end of the above Step 3. on all the node and then on the last node from the root


The log output of Step 2:
[oracle@linux01 /u01/app/oracle/product/11.2.0/dbhome_1/deinstall]$ ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /u01/app/oracle/product/11.2.0/dbhome_1
Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0/grid
The following nodes are part of this cluster: linux01,linux02
Checking for sufficient temp space availability on node(s) : 'linux01,linux02'

## [END] Install check configuration ##


Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2013-10-10_12-08-34-AM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2013-10-10_12-08-37-AM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home []: 
Database Check Configuration END

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_check2013-10-10_12-08-41-AM.log 

Enterprise Manager Configuration Assistant END
Oracle Configuration Manager check START
OCM check log file location : /u01/app/oraInventory/logs//ocm_check8922.log
Oracle Configuration Manager check END

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:linux01,linux02
Oracle Home selected for deinstall is: /u01/app/oracle/product/11.2.0/dbhome_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
No Enterprise Manager configuration to be updated for any database(s)
No Enterprise Manager ASM targets to update
No Enterprise Manager listener targets to migrate
Checking the config status for CCR
linux01 : Oracle Home exists with CCR directory, but CCR is not configured
linux02 : Oracle Home exists with CCR directory, but CCR is not configured
CCR check is finished
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2013-10-10_12-08-25-AM.out'
Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2013-10-10_12-08-25-AM.err'

######################## CLEAN OPERATION START ########################

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /u01/app/oraInventory/logs/emcadc_clean2013-10-10_12-08-41-AM.log 

Updating Enterprise Manager ASM targets (if any)
Updating Enterprise Manager listener targets (if any)
Enterprise Manager Configuration Assistant END
Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2013-10-10_12-09-21-AM.log

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2013-10-10_12-09-21-AM.log

De-configuring Listener configuration file on all nodes...
Listener configuration file de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START
OCM clean log file location : /u01/app/oraInventory/logs//ocm_clean8922.log
Oracle Configuration Manager clean END
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/u01/app/oracle/product/11.2.0/dbhome_1' from the central inventory on the local node : Done

Delete directory '/u01/app/oracle/product/11.2.0/dbhome_1' on the local node : Done

The Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is not empty.

Detach Oracle home '/u01/app/oracle/product/11.2.0/dbhome_1' from the central inventory on the remote nodes 'linux02' : Done

Delete directory '/u01/app/oracle/product/11.2.0/dbhome_1' on the remote nodes 'linux02' : Done

The Oracle Base directory '/u01/app/oracle' will not be removed on node 'linux02'. The directory is not empty.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2013-10-10_00-08-03AM' on node 'linux01'
Clean install operation removing temporary directory '/tmp/deinstall2013-10-10_00-08-03AM' on node 'linux02'

## [END] Oracle install clean ##


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
Cleaning the config for CCR
As CCR is not configured, so skipping the cleaning of CCR configuration
CCR clean is finished
Successfully detached Oracle home '/u01/app/oracle/product/11.2.0/dbhome_1' from the central inventory on the local node.
Successfully deleted directory '/u01/app/oracle/product/11.2.0/dbhome_1' on the local node.
Successfully detached Oracle home '/u01/app/oracle/product/11.2.0/dbhome_1' from the central inventory on the remote nodes 'linux02'.
Successfully deleted directory '/u01/app/oracle/product/11.2.0/dbhome_1' on the remote nodes 'linux02'.
Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############


The log output of Step 3:
[oracle@linux01 /u01/app/11.2.0/grid/deinstall]$ ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2013-10-10_02-32-31AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /u01/app/11.2.0/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is: /u01/app/grid
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0/grid
The following nodes are part of this cluster: linux01,linux02
Checking for sufficient temp space availability on node(s) : 'linux01,linux02'

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2013-10-10_02-32-31AM/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2013-10-10_02-32-31AM/logs/netdc_check2013-10-10_02-33-02-AM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2013-10-10_02-32-31AM/logs/asmcadc_check2013-10-10_02-33-06-AM.log

Automatic Storage Management (ASM) instance is detected in this Oracle home /u01/app/11.2.0/grid.
ASM Diagnostic Destination : /u01/app/grid
ASM Diskgroups : +DATA
ASM diskstring : <Default>
Diskgroups will be dropped
De-configuring ASM will drop all the diskgroups and it's contents at cleanup time. This will affect all of the databases and ACFS that use this ASM instance(s).
 If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'. Do you  want to modify above information (y|n) [n]: y
Specify the ASM Diagnostic Destination [/u01/app/grid]: 
Specify the diskstring []: 
Specify the diskgroups that are managed by this ASM instance [+DATA]: 

De-configuring ASM will drop the diskgroups at cleanup time. Do you want deconfig tool to drop the diskgroups y|n [y]: y


######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2.0/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:linux01,linux02
Oracle Home selected for deinstall is: /u01/app/11.2.0/grid
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Following RAC listener(s) will be de-configured: LISTENER
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2013-10-10_02-32-31AM/logs/deinstall_deconfig2013-10-10_02-32-50-AM.out'
Any error messages from this session will be written to: '/tmp/deinstall2013-10-10_02-32-31AM/logs/deinstall_deconfig2013-10-10_02-32-50-AM.err'

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2013-10-10_02-32-31AM/logs/asmcadc_clean2013-10-10_02-34-13-AM.log
ASM Clean Configuration START
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2013-10-10_02-32-31AM/logs/netdc_clean2013-10-10_02-35-55-AM.log

De-configuring RAC listener(s): LISTENER

De-configuring listener: LISTENER
    Stopping listener: LISTENER
    Listener stopped successfully.
    Unregistering listener: LISTENER
    Listener unregistered successfully.
Listener de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END


---------------------------------------->

The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on  the local node after the execution completes on all the remote nodes.

Run the following command as the root user or the administrator on node "linux02".

/tmp/deinstall2013-10-10_02-32-31AM/perl/bin/perl -I/tmp/deinstall2013-10-10_02-32-31AM/perl/lib -I/tmp/deinstall2013-10-10_02-32-31AM/crs/install /tmp/deinstall2013-10-10_02-32-31AM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2013-10-10_02-32-31AM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Run the following command as the root user or the administrator on node "linux01".

/tmp/deinstall2013-10-10_02-32-31AM/perl/bin/perl -I/tmp/deinstall2013-10-10_02-32-31AM/perl/lib -I/tmp/deinstall2013-10-10_02-32-31AM/crs/install /tmp/deinstall2013-10-10_02-32-31AM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2013-10-10_02-32-31AM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode

Press Enter after you finish running the above commands

<----------------------------------------

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/u01/app/11.2.0/grid' from the central inventory on the local node : Done

Delete directory '/u01/app/11.2.0/grid' on the local node : Done

Delete directory '/u01/app/oraInventory' on the local node : Done

Delete directory '/u01/app/grid' on the local node : Done

Detach Oracle home '/u01/app/11.2.0/grid' from the central inventory on the remote nodes 'linux02' : Done

Delete directory '/u01/app/11.2.0/grid' on the remote nodes 'linux02' : Done

Delete directory '/u01/app/oraInventory' on the remote nodes 'linux02' : Done

Delete directory '/u01/app/grid' on the remote nodes 'linux02' : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2013-10-10_02-32-31AM' on node 'linux01'
Clean install operation removing temporary directory '/tmp/deinstall2013-10-10_02-32-31AM' on node 'linux02'

## [END] Oracle install clean ##


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
ASM instance was de-configured successfully from the Oracle home
Following RAC listener(s) were de-configured successfully: LISTENER
Oracle Clusterware is stopped and successfully de-configured on node "linux01"
Oracle Clusterware is stopped and successfully de-configured on node "linux02"
Oracle Clusterware is stopped and de-configured successfully.
Successfully detached Oracle home '/u01/app/11.2.0/grid' from the central inventory on the local node.
Successfully deleted directory '/u01/app/11.2.0/grid' on the local node.
Successfully deleted directory '/u01/app/oraInventory' on the local node.
Successfully deleted directory '/u01/app/grid' on the local node.
Successfully detached Oracle home '/u01/app/11.2.0/grid' from the central inventory on the remote nodes 'linux02'.
Successfully deleted directory '/u01/app/11.2.0/grid' on the remote nodes 'linux02'.
Successfully deleted directory '/u01/app/oraInventory' on the remote nodes 'linux02'.
Successfully deleted directory '/u01/app/grid' on the remote nodes 'linux02'.
Oracle Universal Installer cleanup was successful.


Run 'rm -rf /etc/oraInst.loc' as root on node(s) 'linux01,linux02' at the end of the session.

Run 'rm -rf /opt/ORCLfmap' as root on node(s) 'linux01,linux02' at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

The log output of Step 3.a for the last node:

[root@linux01 ~]# /tmp/deinstall2013-10-10_02-32-31AM/perl/bin/perl -I/tmp/deinstall2013-10-10_02-32-31AM/perl/lib -I/tmp/deinstall2013-10-10_02-32-31AM/crs/install /tmp/deinstall2013-10-10_02-32-31AM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2013-10-10_02-32-31AM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode
Using configuration parameter file: /tmp/deinstall2013-10-10_02-32-31AM/response /deinstall_Ora11g_gridinfrahome1.rsp
Network exists: 1/10.11.32.0/255.255.248.0/eth0, type static
VIP exists: /linux02-v/10.11.37.64/10.11.32.0/255.255.248.0/eth0, hosting node linux02
VIP exists: /linux01-v/10.11.37.66/10.11.32.0/255.255.248.0/eth0, hosting node linux01
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2673: Attempting to stop 'ora.crsd' on 'linux01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'linux01'
CRS-2673: Attempting to stop 'ora.oc4j' on 'linux01'
CRS-2673: Attempting to stop 'ora.asm' on 'linux01'
CRS-2677: Stop of 'ora.asm' on 'linux01' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'linux01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'linux01' has completed
CRS-2677: Stop of 'ora.crsd' on 'linux01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'linux01'
CRS-2673: Attempting to stop 'ora.evmd' on 'linux01'
CRS-2673: Attempting to stop 'ora.asm' on 'linux01'
CRS-2677: Stop of 'ora.evmd' on 'linux01' succeeded
CRS-2677: Stop of 'ora.asm' on 'linux01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'linux01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'linux01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'linux01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'linux01'
CRS-2677: Stop of 'ora.cssd' on 'linux01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'linux01'
CRS-2676: Start of 'ora.cssdmonitor' on 'linux01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'linux01'
CRS-2672: Attempting to start 'ora.diskmon' on 'linux01'
CRS-2676: Start of 'ora.diskmon' on 'linux01' succeeded
CRS-2676: Start of 'ora.cssd' on 'linux01' succeeded
CRS-4611: Successful deletion of voting disk +DATA.
ASM de-configuration trace file location: /tmp/deinstall2013-10-10_02-32-31AM/logs/asmcadc_clean2013-10-10_02-41-58-AM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/deinstall2013-10-10_02-32-31AM/logs/asmcadc_clean2013-10-10_02-41-58-AM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'linux01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'linux01'
CRS-2673: Attempting to stop 'ora.crf' on 'linux01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'linux01'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'linux01'
CRS-2677: Stop of 'ora.mdnsd' on 'linux01' succeeded
CRS-2677: Stop of 'ora.crf' on 'linux01' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'linux01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'linux01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'linux01'
CRS-2677: Stop of 'ora.cssd' on 'linux01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'linux01'
CRS-2677: Stop of 'ora.gipcd' on 'linux01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'linux01'
CRS-2677: Stop of 'ora.gpnpd' on 'linux01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'linux01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

dbua/dbca | Could not create logging folder

Environment Details:
OS: Linux 5.8
Database Version: 11.2.0.3.0
Oracle Home: /u01/app/oracle/product/11.2.0.3.0


Issue: While running dbca/dbua the installer was not able to create the log file and we were getting the below error.

Resolution:  I verified that the ORACLE_HOME & PATH variables were set properly. Then I checked for the other environment variables and came to conclusion that there were other environment variables from where the installer was picking a wrong ORACLE_HOME, hence unset the variable of the incorrect location and again set the ORACLE_HOME variable. It started working.

[oracle@linux01 ~]$ export
declare -x BASH_ENV="/home/oracle/.bashrc"
declare -x CVS_RSH="ssh"
declare -x DISPLAY="localhost:13.0"
declare -x EDITOR="vi"
declare -x G_BROKEN_FILENAMES="1"
declare -x HISTSIZE="1000"
declare -x HOME="/home/oracle"
declare -x HOSTNAME="linux01.us.oracle.com"
declare -x INPUTRC="/etc/inputrc"
declare -x JAVA_HOME="/usr/java/jdk1.7.0_40"
declare -x LD_LIBRARY_PATH="/scratch/oracle/product/12.1.0/dbhome_1/lib"
declare -x LOGNAME="oracle"
declare -x MAIL="/var/spool/mail/oracle"
declare -x OH_PATH="/scratch/oracle/product/12.1.0/dbhome_1/OPatch:/scratch/oracle/product/12.1.0/dbhome_1/bin"
declare -x ORACLE_BASE="/scratch/oracle"
declare -x ORACLE_HOME="/u01/app/oracle/product/11.2.0.3.0"
declare -x ORACLE_INSTANCE="/scratch/oracle/product/instances/WT11gR1_UCM"
declare -x PATH="/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin"
declare -x PWD="/home/oracle"
declare -x QTDIR="/usr/lib64/qt-3.3"
declare -x QTINC="/usr/lib64/qt-3.3/include"
declare -x QTLIB="/usr/lib64/qt-3.3/lib"
declare -x SHELL="/bin/bash"
declare -x SSH_TTY="/dev/pts/5"
declare -x TERM="xterm"
declare -x USER="oracle"

[oracle@linux01 ~]$ export |grep scratch
declare -x LD_LIBRARY_PATH="/scratch/oracle/product/12.1.0/dbhome_1/lib"
declare -x OH_PATH="/scratch/oracle/product/12.1.0/dbhome_1/OPatch:/scratch/oracle/product/12.1.0/dbhome_1/bin"
declare -x ORACLE_BASE="/scratch/oracle"
declare -x ORACLE_INSTANCE="/scratch/oracle/product/instances/WT11gR1_UCM"

[oracle@linux01 ~]$ unset  LD_LIBRARY_PATH OH_PATH ORACLE_BASE ORACLE_INSTANCE

[oracle@linux01 ~]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0.3.0
[oracle@linux01 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@linux01 ~]$ dbua

Oracle Home inventory is corrupted

Environment Details:
OS: Linux 5.8
Database Version: 11.2.0.3.0
Oracle Home: /u01/app/oracle/product/11.2.0.3.0
Inventory Location: $ORACLE_HOME/oraInventory
Issue: The Oracle Home was cloned from a different server, when we try to run the opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc command, we were getting the error mentioned below.

-bash-3.2$ opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc 
Invoking OPatch 11.2.0.1.7

Oracle Interim Patch Installer version 11.2.0.1.7
Copyright (c) 2011, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0.3.0
Central Inventory : /u01/app/oracle/product/11.2.0.3.0/oraInventory
   from           : /u01/app/oracle/product/11.2.0.3.0/oraInst.loc
OPatch version    : 11.2.0.1.7
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11.2.0.3.0/cfgtoollogs/opatch/opatch2013-10-01_01-07-20AM.log

List of Homes on this system:

  Home name= linux_DB__slot_ems12345_oracle_db_tech_st_11_2_0, Location= "/u01/app/oracle/product/11.2.0"
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73

Solution: Did a through investigation and from the error noticed that it was taking a wrong oracle home. One of the suggestions was to recreate the inventory, but I checked the file /u01/app/oracle/product/11.2.0.3.0/oraInventory/ContentsXML/inventory.xml, the contents given below. I have modified the location LOC to the correct Home .i.e. /u01/app/oracle/product/11.2.0.3.0 and saved the file. When I issued the command opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc again, everything was fine and I got the output.

-bash-3.2$ opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc 
Invoking OPatch 11.2.0.1.7

Oracle Interim Patch Installer version 11.2.0.1.7
Copyright (c) 2011, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0.3.0
Central Inventory : /u01/app/oracle/product/11.2.0.3.0/oraInventory
   from           : /u01/app/oracle/product/11.2.0.3.0/oraInst.loc
OPatch version    : 11.2.0.1.7
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11.2.0.3.0/cfgtoollogs/opatch/opatch2013-10-01_01-12-44AM.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0.3.0/cfgtoollogs/opatch/lsinv/lsinventory2013-10-01_01-12-44AM.txt

-----------------------------------------------------------------
Installed Top-level Products (2): 

Oracle Database 11g                                                  11.2.0.3.0
Oracle Database 11g Examples                                         11.2.0.3.0
There are 2 products installed in this Oracle Home.

Interim patches (1) :

Patch  13366268     : applied on Mon Jun 10 21:58:09 PDT 2013
Unique Patch ID:  14394336
   Created on 21 Dec 2011, 08:14:06 hrs PST8PDT
   Bugs fixed:
     13366268
----------------------------------------------------------------
OPatch succeeded.

-bash-3.2$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2011, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>11.2.0.3.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="linux_DB__slot_ems12345_oracle_db_tech_st_11_2_0" LOC="/u01/app/oracle/product/11.2.0" TYPE="O" IDX="1"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>