Patch conflict and rollback and solution in RAC/Grid Environment
Oracle RAC – Applying and Removing PSUs in Oracle RAC Home with Conflict Detection (12.1.0.2.0)
While running conflict command to check conflict of Patch 30805421 We observed that sub patch -20243268
is conflicting ,So We need to deinstall and start patching through opatchauto
April 2020 patch Details
--------------------
Patch 30805421 - Grid Infrastructure Patch Set Update 12.1.0.2.200414
There are 3 main steps to applying the PSU:
• Detect and rollback conflicts patch.
• Apply Patch on Oracle Home.
• conflict-free version of Patch: 20243268 to proceed with the patching.
Issue
--------
There is a conflict between the DBPSU ( 30700212 -- Database Patch Set Update 12.1.0.2.200414 )
and Patch 20243268: EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G
and we might have to find a conflict-free version of Patch: 20243268 to proceed with the patching
The easiest and quickest way to determine if you have a patch already applied that will conflict with another that will still be applied is through the One-off Patch ConflictDetectionandResolution method.
If no conflict is found you can apply the patch, otherwise you need to resolve the conflicts.
Execute:
#
-MGMTDB:/u02/oracle/12.1.0.2/grid:N # line added by Agent
+ASM2:/u02/oracle/12.1.0.2/grid:N # line added by Agent
TestRAC:/u01/oracle/TestRAC/product/12.1.0.2:N # line added by Agent
[root@server1 ~]# cd /u01/oracle/TestRAC/product/12.1.0.2/OPatch
[root@server1 OPatch]#
[root@server1 OPatch]# export PATH=$PATH:/u01/oracle/TestRAC/product/12.1.0.2/OPatch
[root@server1 OPatch]# ./opatchauto apply /staging/april202_patch/30805421 -analyze -oh /u01/oracle/TestRAC/product/12.1.0.2
OPatchauto session is initiated at Mon Aug 17 14:21:27 2020
System initialization log file is /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchautodb/systemconfig2020-08-17_02-21-37PM.log.
Session log file is /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/opatchauto2020-08-17_02-21-59PM.log
The id for this session is 1XC1
Executing OPatch prereq operations to verify patch applicability on home /u01/oracle/TestRAC/product/12.1.0.2
Patch applicability verified successfully on home /u01/oracle/TestRAC/product/12.1.0.2
Verifying SQL patch applicability on home /u01/oracle/TestRAC/product/12.1.0.2
SQL patch applicability verified successfully on home /u01/oracle/TestRAC/product/12.1.0.2
OPatchAuto successful.
--------------------------------Summary--------------------------------
Analysis for applying patches has failed:
Host:server1
RAC Home:/u01/oracle/TestRAC/product/12.1.0.2
Version:12.1.0.2.0
Analysis for patches has failed.
==Following patches FAILED in analysis for apply:
Patch: /staging/april202_patch/30805421/30700212
Log: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/core/opatch/opatch2020-08-17_14-22-30PM_1.log
Reason: Failed during Analysis: CheckConflictAgainstOracleHome Failed, [ Prerequisite Status: FAILED, Prerequisite output:
Summary of Conflict Analysis:
-----------------------
Patches that can be applied now without any conflicts are :
30703508
Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
20243268, 30700212
Whole composite patch Conflicts/Supersets are:
Composite Patch : 30700212
Conflict with 20243268
Detail Conflicts/Supersets for each patch are:
Sub-Pa ...
==Following patches were SKIPPED:
Patch: /staging/april202_patch/30805421/26983807
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /staging/april202_patch/30805421/30703511
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /staging/april202_patch/30805421/30703508
Log: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/core/opatch/opatch2020-08-17_14-22-30PM_1.log
OPATCHAUTO-72053: Analysis for the patches failed.
OPATCHAUTO-72053: Command execution failed.
OPATCHAUTO-72053: Please check the summary for more details.
OPatchauto session completed at Mon Aug 17 14:23:51 2020
Time taken to complete the session 2 minutes, 24 seconds
[root@server1 OPatch]#
Action Plan
1) Deinstall the currently installed patch :
Patch 20243268: EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G
2) Install the Database Patch Set Update 12.1.0.2.200414 - 30700212
3) Once the DBPSU is installed , download and install the conflict free version of Patch 20243268
Please note you need to download the "12.1.0.2.180417" version of this patch as this was the conflict PSU.
Patch 20243268: EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G
Last Updated 01-May-2018 14:23 (2+ years ago) >>>>>>>>>>>>>>>>>>>>>>>>>
Product Oracle Database - Enterprise Edition
Release Oracle 12.1.0.2.180417 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Platform Linux x86-64
Size 802.3 KB
Download Access Software
Detail Plans
----------
1) Deinstall the currently installed patch :
Patch 20243268: EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G
As the database home owner execute:
[TestRAC@server1OPatch]$
[TestRAC@server1OPatch]$ pwd
/u01/oracle/TestRAC/product/12.1.0.2/OPatch
[TestRAC@server1OPatch]$ id
uid=500(TestRAC) gid=200(dba) groups=200(dba),212(asmadmin),213(asmdba)
[TestRAC@server1OPatch]$
To deinstall the patch, follow these steps:
1. Stop database instance
srvctl stop instance -d <database instance> -i <database instance> -o immediate
2. Deinstall the patch by running the following command:
$ opatch rollback -id 20243268
2) Install the Database Patch Set Update 12.1.0.2.200414 - 30700212
From root user analyze conflict patch and apply patch
Start database instance
export PATH=$PATH:/u01/oracle/TestRAC/product/12.1.0.2/OPatch
./opatchauto apply /staging/april202_patch/30805421 -analyze -oh /u01/oracle/TestRAC/product/12.1.0.2
./opatchauto apply /staging/april202_patch/30805421 -oh /u01/oracle/TestRAC/product/12.1.0.2
3) Once the DBPSU is installed , download and install the conflict free version of Patch 20243268
Please note you need to download the "12.1.0.2.180417" version of this patch as this was the conflict PSU.
database instance should be down before applying patch
srvctl stop instance -d TestRAC -i TestRAC2 -o immediate
export PATH=$PATH:/u01/oracle/TestRAC/product/12.1.0.2/OPatch
1. Set your current directory to the directory where the patch is located and then run the OPatch utility by entering the following commands:
$ cd <PATCH_TOP_DIR>/20243268
$ opatch apply
2. Verify whether the patch has been successfully installed by running the following command:
$ opatch lsinventory
oracrs@server1OPatch]$
[oracrs@server1OPatch]$ ./opatch lspatches
30703511;ACFS PATCH SET UPDATE 12.1.0.2.200414 (30703511)
30703508;OCW PATCH SET UPDATE 12.1.0.2.200414 (30703508)
30700212;Database Patch Set Update : 12.1.0.2.200414 (30700212)
26983807;WLM Patch Set Update: 12.1.0.2.180116 (26983807)
OPatch succeeded.
[oracrs@server1OPatch]$
[testDB@server1OPatch]$
[testDB@server1OPatch]$ ./opatch lspatches
20243268;
30703508;OCW PATCH SET UPDATE 12.1.0.2.200414 (30703508)
30700212;Database Patch Set Update : 12.1.0.2.200414 (30700212)
OPatch succeeded.
[testDB@server1OPatch]$
3. Start the services from the Oracle home.
Other issue during patching
---------------------
Applied patch successfully on clusterware as well as database on first node on Friday ,
but on weekend both node rebooted abruptly, clusterware and database are started successfully on first node whereas on other node , clusterware is not coming up automatically and even manually is not getting started on port1002ovl
It look like due to mismatch of patch level , it is not getting started ,
Solution
1) We took down Node1 and started Node2
2) followed by patching Node2 with the same PSU as Node1
Apr2020 RU has been applied on node2 and all is now running normally
DB instance RACDB2 came up automatically once patch had been applied
3) both the Nodes are back up and operational.
Eg
[TestRAC@server1 OPatch]$ srvctl stop instance -d TestRAC -i TestRAC2 -o immediate
[TestRAC@server1 OPatch]$
[TestRAC@server1 OPatch]$
[TestRAC@server1 OPatch]$ id
uid=500(TestRAC) gid=200(dba) groups=200(dba),212(asmadmin),213(asmdba)
[TestRAC@server1 OPatch]$ ps -ef|grep pmon
oracrs 5342 1 0 13:48 ? 00:00:00 asm_pmon_+ASM2
TestRAC 27413 25732 0 14:29 pts/2 00:00:00 grep pmon
[TestRAC@server1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded
[TestRAC@server1 OPatch]$ id
uid=500(TestRAC) gid=200(dba) groups=200(dba),212(asmadmin),213(asmdba)
[TestRAC@server1 OPatch]$ ./opatch rollback -id 20243268
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/TestRAC/product/12.1.0.2
Central Inventory : /u02/oracle/oraInventory
from : /u01/oracle/TestRAC/product/12.1.0.2/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatch/opatch2020-08-17_14-30-20PM_1.log
Patches will be rolled back in the following order:
20243268
The following patch(es) will be rolled back: 20243268
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/TestRAC/product/12.1.0.2')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Rolling back patch 20243268...
RollbackSession rolling back interim patch '20243268' from OH '/u01/oracle/TestRAC/product/12.1.0.2'
Patching component oracle.rdbms, 12.1.0.2.0...
RollbackSession removing interim patch '20243268' from inventory
Log file location: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatch/opatch2020-08-17_14-30-20PM_1.log
OPatch succeeded.
[TestRAC@server1 OPatch]$
[root@server1 OPatch]# export PATH=$PATH:/u01/oracle/TestRAC/product/12.1.0.2/OPatch
[root@server1 OPatch]# ./opatchauto apply /staging/april202_patch/30805421 -analyze -oh /u01/oracle/TestRAC/product/12.1.0.2
OPatchauto session is initiated at Mon Aug 17 14:32:23 2020
System initialization log file is /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchautodb/systemconfig2020-08-17_02-32-31PM.log.
Session log file is /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/opatchauto2020-08-17_02-32-51PM.log
The id for this session is 3KBP
Executing OPatch prereq operations to verify patch applicability on home /u01/oracle/TestRAC/product/12.1.0.2
Patch applicability verified successfully on home /u01/oracle/TestRAC/product/12.1.0.2
Verifying SQL patch applicability on home /u01/oracle/TestRAC/product/12.1.0.2
SQL patch applicability verified successfully on home /u01/oracle/TestRAC/product/12.1.0.2
OPatchAuto successful.
--------------------------------Summary--------------------------------
Analysis for applying patches has completed successfully:
Host:server1
RAC Home:/u01/oracle/TestRAC/product/12.1.0.2
Version:12.1.0.2.0
==Following patches were SKIPPED:
Patch: /staging/april202_patch/30805421/26983807
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /staging/april202_patch/30805421/30703511
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /staging/april202_patch/30805421/30700212
Log: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/core/opatch/opatch2020-08-17_14-33-16PM_1.log
Patch: /staging/april202_patch/30805421/30703508
Log: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/core/opatch/opatch2020-08-17_14-33-16PM_1.log
OPatchauto session completed at Mon Aug 17 14:33:59 2020
Time taken to complete the session 1 minute, 37 seconds
[root@server1 OPatch]#
srvctl start instance -d TestRAC -i TestRAC2
[root@server1 OPatch]# ./opatchauto apply /staging/april202_patch/30805421 -oh /u01/oracle/TestRAC/product/12.1.0.2
OPatchauto session is initiated at Mon Aug 17 14:53:18 2020
System initialization log file is /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchautodb/systemconfig2020-08-17_02-53-26PM.log.
Session log file is /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/opatchauto2020-08-17_02-53-47PM.log
The id for this session is 4TCY
Executing OPatch prereq operations to verify patch applicability on home /u01/oracle/TestRAC/product/12.1.0.2
Patch applicability verified successfully on home /u01/oracle/TestRAC/product/12.1.0.2
Verifying SQL patch applicability on home /u01/oracle/TestRAC/product/12.1.0.2
SQL patch applicability verified successfully on home /u01/oracle/TestRAC/product/12.1.0.2
Preparing to bring down database service on home /u01/oracle/TestRAC/product/12.1.0.2
Successfully prepared home /u01/oracle/TestRAC/product/12.1.0.2 to bring down database service
Bringing down database service on home /u01/oracle/TestRAC/product/12.1.0.2
Following database(s) and/or service(s) are stopped and will be restarted later during the session: TestRAC
Database service successfully brought down on home /u01/oracle/TestRAC/product/12.1.0.2
Performing prepatch operation on home /u01/oracle/TestRAC/product/12.1.0.2
Perpatch operation completed successfully on home /u01/oracle/TestRAC/product/12.1.0.2
Start applying binary patch on home /u01/oracle/TestRAC/product/12.1.0.2
Binary patch applied successfully on home /u01/oracle/TestRAC/product/12.1.0.2
Performing postpatch operation on home /u01/oracle/TestRAC/product/12.1.0.2
Postpatch operation completed successfully on home /u01/oracle/TestRAC/product/12.1.0.2
Starting database service on home /u01/oracle/TestRAC/product/12.1.0.2
Database service successfully started on home /u01/oracle/TestRAC/product/12.1.0.2
Preparing home /u01/oracle/TestRAC/product/12.1.0.2 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/oracle/TestRAC/product/12.1.0.2
SQL patch applied successfully on home /u01/oracle/TestRAC/product/12.1.0.2
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:server1
RAC Home:/u01/oracle/TestRAC/product/12.1.0.2
Version:12.1.0.2.0
Summary:
==Following patches were SKIPPED:
Patch: /staging/april202_patch/30805421/26983807
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /staging/april202_patch/30805421/30703511
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:
Patch: /staging/april202_patch/30805421/30700212
Log: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/core/opatch/opatch2020-08-17_15-07-02PM_1.log
Patch: /staging/april202_patch/30805421/30703508
Log: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatchauto/core/opatch/opatch2020-08-17_15-07-02PM_1.log
OPatchauto session completed at Mon Aug 17 15:22:30 2020
Time taken to complete the session 29 minutes, 13 seconds
[root@server1 OPatch]#
[root@server1 OPatch]#
[TestRAC@server1 april202_patch]$ id
uid=500(TestRAC) gid=200(dba) groups=200(dba),212(asmadmin),213(asmdba)
[TestRAC@server1 april202_patch]$ pwd
/staging/april202_patch
[TestRAC@server1 april202_patch]$ . oraenv
ORACLE_SID = [TestRAC2] ? TestRAC
The Oracle base remains unchanged with value /u01/oracle/TestRAC
[TestRAC@server1 april202_patch]$ export ORACLE_SID=TestRAC2
[TestRAC@server1 april202_patch]$
[TestRAC@server1 april202_patch]$ export PATH=$PATH:/u01/oracle/TestRAC/product/12.1.0.2/OPatch
[TestRAC@server1 april202_patch]$ id
uid=500(TestRAC) gid=200(dba) groups=200(dba),212(asmadmin),213(asmdba)
[TestRAC@server1 april202_patch]$ pwd
/staging/april202_patch
[TestRAC@server1 april202_patch]$ ls -ltr
total 3481548
drwxrwxr-x 4 TestRAC dba 4096 May 1 2018 20243268
drwxr-xr-x 7 oracrs dba 4096 Mar 31 11:29 30805421
-rwxr-xr-x 1 oracrs dba 3445855633 Aug 13 10:38 p30805421_121020_Linux-x86-64.zip
-rwxr-xr-x 1 oracrs dba 118408624 Aug 13 10:38 p6880880_122010_Linux-x86-64.zip
-rwxr-xr-x 1 TestRAC dba 821509 Aug 17 15:32 p20243268_12102180417_Linux-x86-64.zip
[TestRAC@server1 april202_patch]$ cd 20243268
[TestRAC@server1 20243268]$ ./opatch version
-bash: ./opatch: No such file or directory
[TestRAC@server1 20243268]$ opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.
[TestRAC@server1 20243268]$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /u01/oracle/TestRAC/product/12.1.0.2
Central Inventory : /u02/oracle/oraInventory
from : /u01/oracle/TestRAC/product/12.1.0.2/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatch/opatch2020-08-17_15-40-52PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 20243268
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/TestRAC/product/12.1.0.2')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '20243268' to OH '/u01/oracle/TestRAC/product/12.1.0.2'
Patching component oracle.rdbms, 12.1.0.2.0...
Patch 20243268 successfully applied.
Log file location: /u01/oracle/TestRAC/product/12.1.0.2/cfgtoollogs/opatch/opatch2020-08-17_15-40-52PM_1.log
OPatch succeeded.
[TestRAC@server1 20243268]$
verify applied patch
oracrs@server1OPatch]$
[oracrs@server1OPatch]$ ./opatch lspatches
30703511;ACFS PATCH SET UPDATE 12.1.0.2.200414 (30703511)
30703508;OCW PATCH SET UPDATE 12.1.0.2.200414 (30703508)
30700212;Database Patch Set Update : 12.1.0.2.200414 (30700212)
26983807;WLM Patch Set Update: 12.1.0.2.180116 (26983807)
OPatch succeeded.
[oracrs@server1OPatch]$
[testDB@server1OPatch]$
[testDB@server1OPatch]$ ./opatch lspatches
20243268;
30703508;OCW PATCH SET UPDATE 12.1.0.2.200414 (30703508)
30700212;Database Patch Set Update : 12.1.0.2.200414 (30700212)
OPatch succeeded.
[testDB@server1OPatch]$
No comments:
Post a Comment