Issue: In one of our cluster nodes when we were doing a Oracle Bundle patching in rolling fashion(involves GI patching as well), We started hitting the below error. We realized that the CRS Stack was down on the other nodes prior to patching due to which nodes cannot communicate about its patching/rolling state to other nodes (because CRS was down on other nodes) and it ends up failing.
Solution: To overcome this we had to stop the CRS on first node and then start the patching on the next node.
2017-02-23 23:29:47.453 [ORAAGENT(39105)]CRS-5017: The resource action "ora.asm start" encountered the following error:
2017-02-23 23:29:47.453+ORA-15135: The ASM cluster is not in rolling patch state. The patch level [3351897854] of instance +ASM2 is not the same as the current cluster patch level [2552957799].
. For details refer to "(:CLSN00107:)" in "/opt/oracle/diag/crs/linux02/crs/trace/ohasd_oraagent_orprod.trc".
When doing a opatch lsinventory on GI home it result in the below, we can verify that the patching levels are different.
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux01
3351897854 linux02
In case the CRS stack is kept up on all nodes, the information about the patching state is made aware to the other node and it passes through successfully. Below is the result of opatch lsinventory when the CRS stack was kept up.
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux02,linux01
Solution: To overcome this we had to stop the CRS on first node and then start the patching on the next node.
2017-02-23 23:29:47.453 [ORAAGENT(39105)]CRS-5017: The resource action "ora.asm start" encountered the following error:
2017-02-23 23:29:47.453+ORA-15135: The ASM cluster is not in rolling patch state. The patch level [3351897854] of instance +ASM2 is not the same as the current cluster patch level [2552957799].
. For details refer to "(:CLSN00107:)" in "/opt/oracle/diag/crs/linux02/crs/trace/ohasd_oraagent_orprod.trc".
When doing a opatch lsinventory on GI home it result in the below, we can verify that the patching levels are different.
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux01
3351897854 linux02
In case the CRS stack is kept up on all nodes, the information about the patching state is made aware to the other node and it passes through successfully. Below is the result of opatch lsinventory when the CRS stack was kept up.
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux02,linux01
I found this and used it to attempt to finish an ODA Server patch on Node2 of an X4 ODA. It did the trick in letting me manually start up +ASM2 on Node2. However, later the patch failed because it found CRS down on Node1. ODAs are tricky beasts and finally Oracle Support provided a solution that worked for me which you and others might use.
ReplyDeleteAs the grid user one Node1 do sqlplus / as sysasm. When in execute this
Alter System Start Rolling Patch ;
Then go to Node2 and you'll find you can startup +ASM2.
I still have some investigating to do, but wanted to contribute to your useful blog.
Jack
Thank You Jack
DeleteI found similar issue and the workaround helped to bring the ASM up. But when I restart the CRS on the other and ASM failed to come up with same error "ORA-15135: cluster is not in the rolling patch" . Any idea how to fix this permanently ?
ReplyDeleteThanks Samrat Sir!!
ReplyDeleteEven this didn't work for me, So i started the rolling patch from working node and started the crs on issue node.
ReplyDeletecrsctl start rollingpatch