issue after applying OCT 2017 PSU(26635815) on 12cclusterware 12.1.0.2.0 through auto patch utility/Databases were not getting started through srvctl command .

Issue after applying OCT 2017 PSU(26635815) on clusterware through auto patch utility

Issue description:

After applying PSU on 12 clusterware  (12.1.0.2.0),Databases were not getting started through srvctl command .it showing below error.So finally we started database through sqlplus and Therefor getting error by running analyze command for ORACLE RAC HOME


[node1:root:/u02/oracle/12.1.0/grid:] cd OPatch
[node1:root:/u02/oracle/12.1.0/grid/OPatch:] ./opatchauto apply /staging/clusterware_12cpatch/26635815 -analyze -oh /u01/oracle/NFT/product/12.1.0.2

opatchauto must run from one of the homes specified
opatchauto returns with error code = 2
[node1:root:/u02/oracle/12.1.0/grid/OPatch:]  ./opatchauto apply /staging/clusterware_12cpatch/26635815 -analyze

OPatchauto session is initiated at Sun Jan 21 13:38:10 2018

System initialization log file is /u02/oracle/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2018-01-21_01-38-20PM.log.

Failed:
ERROR:
Oracle Grid Infrastructure not configured.
You cannot run '/u02/oracle/12.1.0/grid/bin/cluvfy' without the Oracle Grid Infrastructure.

The result of cluvfy command does not contain OVERALL_STATUS String.
oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException: Unable to determine if "/u01/oracle/NFT/product/12.1.0.2" is a shared oracle home.
Failed:
ERROR:
Oracle Grid Infrastructure not configured.
You cannot run '/u02/oracle/12.1.0/grid/bin/cluvfy' without the Oracle Grid Infrastructure.

The result of cluvfy command does not contain OVERALL_STATUS String.


Solution

After applying OCT 2017 on Grid Home through opatchauto .I believe that
somehow permission have changed of below files and restricted registered
database with grid home to start database through SRVCT

Take backup of oracle RAC home

cd $ORACLE_HOME
pwd
tar -cvf /oswatcher/CH21417/fsnft_home_database.tar .
Ls -ltr /oswatcher/CH21417/fsnft_home_database.tar

Note details

 Doc ID 1989922.1. 

 Check and give permission to 755 to below files on grid home for all nodes

 File Name
 libclntsh.so, libclntshcore.so, libocci.so and libttsh12.so

 node1[/u02/oracle/12.1.0/grid]$ cd lib
node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr libclntsh.so
-rwx------    1 oracrs   dba        61823455 Dec 30 07:42 libclntsh.so
node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr  libclntshcore.so
-rwx------    1 oracrs   dba         4189584 Dec 30 07:42 libclntshcore.so
node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr  libocci.so
-rwx------    1 oracrs   dba         4996222 Dec 30 07:42 libocci.so
node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr  libttsh12.so
-rwx------    1 oracrs   dba        83317612 Dec 30 07:42 libttsh12.so
node1[/u02/oracle/12.1.0/grid/lib]$
node1[/u02/oracle/12.1.0/grid/lib]$ chmod 755 libclntsh.so
node1[/u02/oracle/12.1.0/grid/lib]$ chmod 755  libclntshcore.so
node1[/u02/oracle/12.1.0/grid/lib]$ chmod 755  libocci.so
node1[/u02/oracle/12.1.0/grid/lib]$ chmod 755   libttsh12.so
node1[/u02/oracle/12.1.0/grid/lib]$
node1[/u02/oracle/12.1.0/grid/lib]$chmod 775 libclntshcore.so
node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr libclntsh.so
-rwxr-xr-x    1 oracrs   dba        61823455 Dec 30 07:42 libclntsh.so
node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr  libclntshcore.so
-rwxr-xr-x    1 oracrs   dba         4189584 Dec 30 07:42 libclntshcore.so

node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr  libocci.so
-rwxr-xr-x    1 oracrs   dba         4996222 Dec 30 07:42 libocci.so
node1[/u02/oracle/12.1.0/grid/lib]$ ls -ltr  libttsh12.so
-rwxr-xr-x    1 oracrs   dba        83317612 Dec 30 07:42 libttsh12.so

stop and start the datbase through srvctl command

srvctl status instance -d <database> -i <instnace1>

srvctl start instance -d <database> -i <instnace1>
srvctl status instance -d <database> -i <instnace1>
srvctl stop instance -d <database> -i <instnace1>

Please repeat same step on other node