Issue during 12c patching
-------------------------------
Error whil applying patching on cluserware
----------------------------------
OPatchauto session is initiated at Sat Apr 18 03:17:56 2020
System initialization log file is /u02/oracle/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2020-04-18_03-18-27AM.log.
Session log file is /u02/oracle/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2020-04-18_03-19-42AM.log
The id for this session is BXND
Executing OPatch prereq operations to verify patch applicability on home /u02/oracle/12.2.0.1/grid
Patch applicability verified successfully on home /u02/oracle/12.2.0.1/grid
Bringing down CRS service on home /u02/oracle/12.2.0.1/grid
Prepatch operation log file location: /u02/oracle/PRD/crsdata/fleetwood/crsconfig/crspatch_fleetwood_2020-04-18_03-21-07AM.log
CRS service brought down successfully on home /u02/oracle/12.2.0.1/grid
Start applying binary patch on home /u02/oracle/12.2.0.1/grid
Successfully executed command: /usr/sbin/slibclean
Failed while applying binary patches on home /u02/oracle/12.2.0.1/grid
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : fleetwood->/u02/oracle/12.2.0.1/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u02/oracle/12.2.0.1/grid, host: fleetwood.
Command failed: /u02/oracle/12.2.0.1/grid/OPatch/opatchauto apply /oswatcher/2020_patch/30116802 -oh /u02/oracle/12.2.0.1/grid -target_type cluster -binary -invPtrLoc /u02/oracle/12.2.0.1/grid/oraInst.loc -jre /u02/oracle/12.2.0.1/grid/OPatch/jre -persistresult /u02/oracle/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_fleetwood_crs.ser -analyzedresult /u02/oracle/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_analy
IT look like permission issue .in first node it is onwed by root . it should be grid user
fleetwood:root:/home/root:] ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
-rw-r----- 1 root system 140210504 Aug 08 2018 /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
[fleetwood:root:/home/root:]
in second node ,permission look fine.
pending to apply patch
[fairfield:root:/home/root:] ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
-rw-r----- 1 oracrs oinstall 140210504 Oct 17 2018 /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
[fairfield:root:/home/root:]
As per log , there is bug
As per oracle support comments
Hi,
++ As below bug patches are in place on node#1 now.
BUG 30122814 - OCW OCT 2019
BUG 30122828 - ACFS OCT 2019
BUG 30093408 - TOMCAT
++ You cannot continue with "opatchauto apply" for remaining patches on GI home. The only option is to apply the patch manually using "opatch apply" on node#1.
++ Can you provide the below details?
# ls -lrt /oswatcher/2020_patch/30116802
# <GI_HOME>/bin/kfod op=patches
# <GI_HOME>/bin/kfod op=patchlvl
++ Run the below command on node#1 and share again the below command output
# $GI_HOME/crs/install/rootcrs.sh -prepatch
# $GI_HOME/crs/install/rootcrs.sh -postpatch
# ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
output of above
--------------------
[fleetwood:root:/home/root:]
[fleetwood:root:/home/root:] ls -lrt /oswatcher/2020_patch/30116802
total 272
drwxrwxr-x 4 oracrs oinstall 256 Sep 27 2019 26839277
drwxrwxr-x 5 oracrs oinstall 256 Sep 27 2019 30122828
drwxrwxr-x 4 oracrs oinstall 256 Sep 27 2019 30093408
drwxrwxr-x 4 oracrs oinstall 256 Sep 27 2019 30138470
drwxrwxr-x 2 oracrs oinstall 4096 Sep 27 2019 automation
drwxrwxr-x 5 oracrs oinstall 256 Sep 27 2019 30122814
-rwxrwxr-x 1 oracrs oinstall 0 Sep 27 2019 README.txt
-rwxrwxr-x 1 oracrs oinstall 124674 Sep 27 2019 README.html
-rwxrwxr-x 1 oracrs oinstall 5828 Sep 27 2019 bundle.xml
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid/bin/kfod op=patches
/u02/oracle/12.2.0.1/grid/bin/kfod[22]: %ORACLE_HOME%/bin/kfod.bin: not found.
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid/bin/kfod op=patchlvl
/u02/oracle/12.2.0.1/grid/bin/kfod[22]: %ORACLE_HOME%/bin/kfod.bin: not found.
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid/crs/install/rootcrs.sh -prepatch
Using configuration parameter file: /u02/oracle/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u02/oracle/PRD/crsdata/fleetwood/crsconfig/crspatch_fleetwood_2020-04-18_04-58-22AM.log
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid//crs/install/rootcrs.sh -postpatch
Using configuration parameter file: /u02/oracle/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u02/oracle/PRD/crsdata/fleetwood/crsconfig/crspatch_fleetwood_2020-04-18_04-58-46AM.log
2020/04/18 05:00:06 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/04/18 05:00:07 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2020/04/18 05:00:55 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'fleetwood'
CRS-2672: Attempting to start 'ora.evmd' on 'fleetwood'
CRS-2676: Start of 'ora.mdnsd' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.evmd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'fleetwood'
CRS-2676: Start of 'ora.gpnpd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'fleetwood'
CRS-2676: Start of 'ora.gipcd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fleetwood'
CRS-2676: Start of 'ora.cssdmonitor' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'fleetwood'
CRS-2672: Attempting to start 'ora.diskmon' on 'fleetwood'
CRS-2676: Start of 'ora.diskmon' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.cssd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'fleetwood'
CRS-2672: Attempting to start 'ora.ctssd' on 'fleetwood'
CRS-2676: Start of 'ora.ctssd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'fleetwood'
CRS-2676: Start of 'ora.crf' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'fleetwood'
CRS-2676: Start of 'ora.crsd' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'fleetwood'
CRS-2676: Start of 'ora.asm' on 'fleetwood' succeeded
CRS-6017: Processing resource auto-start for servers: fleetwood
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'fleetwood'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'fairfield'
CRS-2673: Attempting to stop 'ora.fleetwood.vip' on 'fairfield'
CRS-2672: Attempting to start 'ora.ons' on 'fleetwood'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'fairfield' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'fairfield'
CRS-2677: Stop of 'ora.fleetwood.vip' on 'fairfield' succeeded
CRS-2672: Attempting to start 'ora.fleetwood.vip' on 'fleetwood'
CRS-2677: Stop of 'ora.scan1.vip' on 'fairfield' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'fleetwood'
CRS-2676: Start of 'ora.fleetwood.vip' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'fleetwood'
CRS-2676: Start of 'ora.scan1.vip' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'fleetwood'
CRS-2676: Start of 'ora.ons' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'fleetwood'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.asm' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_DATA1.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_FRA1.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_REDO1.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_REDO2.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.MGMT.dg' on 'fleetwood'
CRS-2676: Start of 'ora.E2B01PRD_FRA1.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.E2B01PRD_REDO2.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.MGMT.dg' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.E2B01PRD_REDO1.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.E2B01PRD_DATA1.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.CRS.dg' on 'fleetwood' succeeded
CRS-6016: Resource auto-start has completed for server fleetwood
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1221707709].
SQL Patching tool version 12.2.0.1.0 Production on Sat Apr 18 05:11:01 2020
Copyright (c) 2012, 2018, Oracle. All rights reserved.
Connecting to database...OK
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED GIMR_DSCREP_10
Nothing to roll back
Nothing to apply
SQL Patching tool complete on Sat Apr 18 05:12:20 2020
[fleetwood:root:/home/root:]
[fleetwood:root:/home/root:]
[fleetwood:root:/home/root:] ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
-rw-r----- 1 root system 140210504 Aug 08 2018 /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
[fleetwood:root:/home/root:]
./u02/oracle/12.2.0.1/grid/crs/install/rootcrs.sh -postpatch
./rootcrs.sh -postpatch
============
As per oracle support recommendation ,we need to apply missing patch manually
Steps for Applying the Patch
======================
Execute the following on each node of the cluster in non-shared CRS and DB home environment to apply the patch.
1. Stop the CRS managed resources running from DB homes.
If this is a GI Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>
Note:
You need to make sure that the Oracle ACFS file systems are unmounted (see Section 2.8) and all other Oracle processes are shutdown before you proceed.
2. Run the pre root script.
If this is a GI Home, as the root user execute:
# <GI_HOME>/crs/install/rootcrs.sh -prepatch
3. Patch GI home.
As the GI home owner execute:
$ <GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/%BUGNO%/%DB WLM TRACKING BUG%
$ <GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/%BUGNO%/%DB RU TRACKING BUG%
5. Run the post script.
As the root user execute:
# <GI_HOME>/rdbms/install/rootadd_rdbms.sh
If this is a GI Home, as the root user execute:
# <GI_HOME>/crs/install/rootcrs.sh -postpatch
6. If the message, "A system reboot is recommended before using ACFS is shown, then a reboot must be issued before continuing.
Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver.
7. Start the CRS managed resources that were earlier running from DB homes.
If this is a GI Home environment, as the database home owner execute:
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>
8. For each database instance running on the Oracle home being patched, run the datapatch utility as described in next table.
eg
from root user
earlier used command
./opatchauto apply /oswatcher/2020_patch/30116802 -oh /u02/oracle/12.2.0.1/grid
[Apr 18, 2020 3:24:48 AM] [INFO] Patch 30093408 successfully applied.
[Apr 18, 2020 3:25:56 AM] [INFO] Patch 30122814 successfully applied.
[Apr 18, 2020 3:29:30 AM] [INFO] Patch 30122828 successfully applied.
drwxrwxr-x 4 oracrs oinstall 256 Sep 27 2019 26839277
drwxrwxr-x 5 oracrs oinstall 256 Sep 27 2019 30122828 -done
drwxrwxr-x 4 oracrs oinstall 256 Sep 27 2019 30093408 -done
drwxrwxr-x 4 oracrs oinstall 256 Sep 27 2019 30138470
drwxrwxr-x 5 oracrs oinstall 256 Sep 27 2019 30122814 -done
done means already applied OPatch autom
<UNZIPPED_PATCH_LOCATION>/30116802/30138470
<UNZIPPED_PATCH_LOCATION>/30116802/30122814-done
<UNZIPPED_PATCH_LOCATION>/30116802/30122828-done
<UNZIPPED_PATCH_LOCATION>/30116802/26839277
<UNZIPPED_PATCH_LOCATION>/30116802/30093408- done
we need to apply on missing patching as below
/oswatcher/2020_patch/30116802/26839277
/oswatcher/2020_patch/30116802/30138470
for 11 g
from root user
cd /u02/oracle/12.2.0.1/grid/crs/install
./rootcrs.pl -unlock
from grid user
cd /u02/oracle/12.2.0.1/grid/OPatch
export PATH=$PATH:/u02/oracle/12.2.0.1/grid/OPatch
opatch napply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/26839277
opatch napply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/30138470
Root user
/u02/oracle/12.2.0.1/grid/crs/install/rootcrs.pl -patch
start crs on node1
start database on node1
from 12c syntax has changed
from root user
cd /u02/oracle/12.2.0.1/grid/crs/install/
./rootcrs.sh -prepatch
from grid user
cd /u02/oracle/12.2.0.1/grid/OPatch
export PATH=$PATH:/u02/oracle/12.2.0.1/grid/OPatch
opatch napply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/26839277
./opatch apply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/30138470
from root user
cd /u02/oracle/12.2.0.1/grid/crs/install/
.rootcrs.sh -postpatch
./u02/oracle/12.2.0.1/grid/crs/install/rdbms/install/rootadd_rdbms.sh
No comments:
Post a Comment