ORA-15135: The ASM cluster is not in rolling patch state

Issue: In one of our cluster nodes when we were doing a Oracle Bundle patching in rolling fashion(involves GI patching as well), We started hitting the below error. We realized that the CRS Stack was down on the other nodes prior to patching due to which nodes cannot communicate about its patching/rolling state to other nodes (because CRS was down on other nodes) and it ends up failing. 

Solution: To overcome this we had to stop the CRS on first node and then start the patching on the next node.

2017-02-23 23:29:47.453 [ORAAGENT(39105)]CRS-5017: The resource action "ora.asm start" encountered the following error: 
2017-02-23 23:29:47.453+ORA-15135: The ASM cluster is not in rolling patch state. The patch level [3351897854] of instance +ASM2 is not the same as the current cluster patch level [2552957799].
. For details refer to "(:CLSN00107:)" in "/opt/oracle/diag/crs/linux02/crs/trace/ohasd_oraagent_orprod.trc".

When doing a opatch lsinventory on GI home it result in the below, we can verify that the patching levels are different. 

Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux01
3351897854 linux02

In case the CRS stack is kept up on all nodes, the information about the patching state is made aware to the other node and it passes through successfully. Below is the result of opatch lsinventory when the CRS stack was kept up.

Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2552957799 linux02,linux01

Oracle Opatch failed with "CheckActiveFilesAndExecutables"

Environment Details:
OS: Linux Server 5.8 64bit
GRID Home Version: 11.2.0.4.0
GRID Home: /opt/oracle/product/11.2.0.4.GRD

Issue: While applying patch, the opatch failed with the below error

Following executables are active : 
/opt/oracle/product/11.2.0.4.GRD/lib/libclntsh.so.11.1 
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.  

Solution:  The above mentioned file is still active or in use.

Check which process/tool is using the file and stop the process cleanly else kill the process.

you can use lsof(list open files) or fuser (identify processes using files or sockets) to identify the process using the file.

[oracle@linux01 ~]$ lsof /opt/oracle/product/11.2.0.4.GRD/lib/libclntsh.so.11.1
COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF     NODE NAME
ons       2342 oracle mem    REG 253,53 53817421 21213079 /apps/oracle/product/11.2.0.4.GRD/lib/libclntsh.so.11.1
ons       2344 oracle mem    REG 253,53 53817421 21213079 /apps/oracle/product/11.2.0.4.GRD/lib/libclntsh.so.11.1


[oracle@linux01 ~]$ /sbin/fuser -cv /opt/oracle/product/11.2.0.4.GRD/lib/libclntsh.so.11.1
                     USER        PID ACCESS COMMAND
/opt/oracle/product/11.2.0.4.GRD/lib/libclntsh.so.11.1:
                     oracle     2342 F.ce. ons
                     oracle     2344 F.ce. ons

Once the processes are identified, stop/kill them:
[oracle@linux01 ~]$ kill -9 2342
[oracle@linux01 ~]$ kill -9 2344