EasyReliableDBA

Software Training and dumps for Exam are available on www.easyreliable.com To download dumps. Please visit Links and Just pay $5/Rs300 http://easyreliable.com/easydumps/ This blog is Oracle DBA blog which cover Oracle RAC(10g,11g ,12c and 19c), Dataguard, RMAN,Backup&Recovery,Troubleshooting and Performance Tuning and other topics like Oracle Cloud,MongoDB,Golden Gate and Exadata.Please subscribe below to get update on my blog.We also provide Online SoftwareTraining and DBA Consulting

Friday, 24 July 2020

Oracle Trace File Analyzer (TFA) concept,Installation and Troubleshooting

Oracle Trace File Analyzer (TFA)

Trace File Analyzer (TFA) is Primary diagnostic collection tool, with Database Support Tools Bundle
It has been shipped with the Oracle database since 11.2.0.4, if it is already installed and running but still Oracle recommend downloading the latest version on a regular basis to get the best results. The following location gives an overview of the TFA product, as well as download links and basic instructions.

Download TFA

download and install the latest TFA Support Tools Bundle from Note:1594347.1

If TFA Software is older than 180 days. We need to download from oracle site and install again .Oracle support always want log from the latest version only.

Trace File Analyzer (TFA) & ORAchk/EXAchk

Single combined installer

To install TFA

To install as root:

Download the appropriate installer from above, copy to a target machine and unzip
Run the TFA & ORAchk/EXAchk install command ./ahf_setup [-ahf_loc install_dir] [-data_dir data_dir]

a) To install TFA in default location from root user

./ahf_setup

b) To install instal TFA in specific location from root user

./ahf_setup -ahf_loc /u02/oracle/12.1.0.2/grid/tfa

+ASM1:/u02/oracle/12.1.0.2/grid:N # line added by Agent

-MGMTDB:/u02/oracle/12.1.0.2/grid:N # line added by Agent

RACDB_TEST:/u01/oracle/RACDB_TEST/product/12.1.0.2:N

RACDB_DEV:/u01/oracle/RACDB_DEV/product/12.1.0.2:N

[root@Node1tfa]# ls -ltr /u02/oracle/12.1.0.2/grid/tfa

total 8

drwxr-xr-x 2 root root 4096 Oct 13 2017 bin

drwxr-x--x 3 root root 4096 Jul 1 01:07 Node1

[root@Node1tfa]# ./ahf_setup -ahf_loc /u02/oracle/12.1.0.2/grid/tfa

AHF Installer for Platform Linux Architecture x86_64

AHF Installation Log : /tmp/ahf_install_202000_21323_2020_07_14-15_20_38.log

Starting Autonomous Health Framework (AHF) Installation

AHF Version: 20.2.0 Build Date: 202006260723

TFA is already installed at : /u02/oracle/12.1.0.2/grid/tfa/Node1/tfa_home

Installed TFA Version : 192000 Build ID : 20190426044818

AHF Location : /u02/oracle/12.1.0.2/grid/tfa

AHF Data Directory stores diagnostic collections and metadata.

AHF Data Directory requires at least 5GB (Recommended 10GB) of free space.

Choose Data Directory from below options :

1. /u02/oracle/prod [Free Space : 29071 MB]

2. /u02/oracle/12.1.0.2/grid/tfa [Free Space : 29071 MB]

3. Enter a different Location

Choose Option [1 - 3] : 2

AHF Data Directory : /u02/oracle/12.1.0.2/grid/tfa/oracle.ahf/data

Shutting down TFA : /u02/oracle/12.1.0.2/grid/tfa/Node1/tfa_home

Copying TFA Data Files from /u02/oracle/12.1.0.2/grid/tfa/Node1/tfa_home

Uninstalling TFA : /u02/oracle/12.1.0.2/grid/tfa/Node1/tfa_home

Do you want to add AHF Notification Email IDs ? [Y]|N : N

AHF will also be installed/upgraded on these Cluster Nodes :

1. Node2

The AHF Location and AHF Data Directory must exist on the above nodes

AHF Location : /u02/oracle/12.1.0.2/grid/tfa

AHF Data Directory : /u02/oracle/12.1.0.2/grid/tfa/oracle.ahf/data

Do you want to install/upgrade AHF on Cluster Nodes ? [Y]|N : Y

Extracting AHF to /u02/oracle/12.1.0.2/grid/tfa

Configuring TFA Services

Copying TFA Data Files to AHF

Discovering Nodes and Oracle Resources

Starting TFA Services

.--------------------------------------------------------------------------------.

+-------------+---------------+-------+------+------------+----------------------+

| Node1| RUNNING | 24041 | 5000 | 20.2.0.0.0 | 20200020200626072308 |

'-------------+---------------+-------+------+------------+----------------------'

Running TFA Inventory...

Adding default users to TFA Access list...

.----------------------------------------------------------------------------------.

| Summary of AHF Configuration |

+-----------------+----------------------------------------------------------------+

| Parameter | Value |

+-----------------+----------------------------------------------------------------+

| AHF Location | /u02/oracle/12.1.0.2/grid/tfa |

| TFA Location | /u02/oracle/12.1.0.2/grid/tfa/tfa |

| Orachk Location | /u02/oracle/12.1.0.2/grid/tfa/orachk |

| Data Directory | /u02/oracle/12.1.0.2/grid/tfa/oracle.ahf/data |

| Repository | /u02/oracle/12.1.0.2/grid/tfa/oracle.ahf/data/repository |

| Diag Directory | /u02/oracle/12.1.0.2/grid/tfa/oracle.ahf/data/Node1/diag |

'-----------------+----------------------------------------------------------------'

Starting orachk scheduler from AHF ...

AHF install completed on Node1

Installing AHF on Remote Nodes :

AHF will be installed on Node2, Please wait.

AHF will prompt twice to install/upgrade per Remote Node. So total 2 prompts

Do you want to continue Y|[N] : N

Please install AHF using -local option on all remote nodes manually

AHF binaries are available in /u02/oracle/12.1.0.2/grid/tfa/bin

AHF is successfully installed

Moving /tmp/ahf_install_202000_21323_2020_07_14-15_20_38.log to /u02/oracle/12.1.0.2/grid/tfa/oracle.ahf/data/Node1/diag/ahf/

[root@Node1tfa]#

To check version and status of TFA

tfactl print status

tfactl print version

tfactl print summary

[node1:root:/u02/oracle/12.1.0/grid/tfa/node1/tfa_home/bin:] ./tfactl print status

WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.

.--------------------------------------------------------------------------------------------------.

+---------+---------------+----------+------+------------+----------------------+------------------+

| node1 | RUNNING | 11468966 | 5000 | 18.1.2.0.0 | 18120020180510233906 | COMPLETE |

'---------+---------------+----------+------+------------+----------------------+------------------'

[node1:root:/u02/oracle/12.1.0/grid/tfa/node1/tfa_home/bin:]

To start and stop TFA

$ /u01/app/12.2.0/grid/tfa/bin/tfactl start

Starting TFA..

start: Job is already running: oracle-tfa

Waiting up to 100 seconds for TFA to be started..

. . . . .

Successfully started TFA Process..

. . . . .

TFA Started and listening for commands

Stop Oracle Trace File Analyzer as follows:

Grid_home/tfa/bin/tfactl stop

For example:

$ /u01/app/12.2.0/grid/tfa/bin/tfactl stop

Stopping TFA from the Command Line

Stopped OSWatcher

TFA is running - Will wait 5 seconds (up to 3 times)

TFA-00518 Oracle Trace File Analyzer (TFA) is not running (stopped)

TFA Stopped Successfully

. . .

Successfully stopped TFA..

To collect TFA

. collect the TFA logs from problematic node

$GI_HOME/tfa/bin/tfactl diagcollect -from "MMM/dd/yyyy hh:mm:ss" -to "MMM/dd/yyyy hh:mm:ss"

Specify the "from time" to be 4 hours before and the "to time" to be 4 hours after the time of error (means after performed above steps==>(action plan)).

./tfactl diagcollect -from "JUL/01/2020 00:01:01" -to "JUL/01/2020 12:01:01"

To uninstall TFA and troubleshoot

Can you attempt to reinstall AHF once again ( as root ) using the steps listed below :

1. Uninstall TFA on all nodes:

$TFA_HOME/bin/uninstalltfa

rm -rf $AHF_HOME

rm -rf $AHF_DATA_DIR

rm -rf /etc/oracle.ahf.loc

2. Download the latest AHF installer from MOS Note: 2550798.1 , stage and extract on node 1.

3. Install AHF from node 1:

./ahf_setup -debug -level 6

4. Wait for 5-10 minutes and the certificates should automatically generate and propagate. If this does not happen or you don't want to wait you can execute the following to regenerate the certs:

# tfactl syncnodes –regenerate

If AHF gets installed successfully , you can then attempt to gather a TFA DiagCollection using :

tfactl diagcollect -from "Apr/18/2020 20:00:00" -to "Apr/19/2020 10:00:00"

Locate Oracle Clusterware log files

crs log/grid log/clusterwar log location/cluster alert log location in 11g,12c and 18c

Clusterware Log Location Change in 12.1.0.2

cd $ORACLE_BASE/diag/crs/<node_name>/crs/trace

cd /u02/oracle/NFT/diag/crs/<node_name>/crs/trace

same location in Version 18.10.0.0.0

cd $ORACLE_BASE/diag/crs/<node_name>/crs/trace

cd /u02/oracle/NFT/diag/crs/<node_name>/crs/trace

log location for grid /clusterware

[oracrs@Node1 trace]$

[oracrs@Node1 trace]$ pwd

/u02/oracle/prod/diag/crs/Node1/crs/trace

[oracrs@Node1 trace]$

Clusterware/Grid log location for version 11.2.0.4.0

cd $ORACLE_HOME/log/Node1

/u02/oracle/11.2.0.4/grid/log/<Node Name>

[oracrs@Node1 Node1]$

[oracrs@Node1 Node1]$ pwd

/u02/oracle/11.2.0.4/grid/log/Node1

[oracrs@Node1 Node1]$ echo $ORACLE_HOME

/u02/oracle/11.2.0.4/grid

[oracrs@Node1 Node1]$

oracrs@Node1 Node1]$ ls -ltr

total 3276

drwxr-x--- 2 oracrs dba 4096 Jan 31 2017 srvm

drwxr-x--- 2 root dba 4096 Jan 31 2017 gnsd

drwxr-x--- 2 oracrs dba 4096 Jan 31 2017 diskmon

drwxr-x--- 4 oracrs dba 4096 Jan 31 2017 cvu

drwxr-xr-x 2 root dba 4096 Jan 31 2017 acfssec

drwxr-x--- 2 oracrs dba 4096 Jan 31 2017 acfsrepl

drwxr-x--- 2 oracrs dba 4096 Jan 31 2017 acfslog

drwxrwxr-t 4 root dba 4096 Jan 31 2017 agent

drwxr-x--- 2 oracrs dba 4096 Jan 31 2017 admin

drwxr-x--- 2 root dba 4096 Jan 31 2017 acfsreplroot

drwxr-xr-x 3 root root 4096 Jan 31 2017 acfs

drwxr-x--- 2 oracrs dba 4096 Jan 31 2017 mdnsd

drwxr-x--- 2 oracrs dba 4096 Jan 31 2017 evmd

drwxrwxr-t 5 oracrs dba 4096 Feb 10 2017 racg

drwxr-x--- 2 root dba 4096 Mar 20 2017 crflogd

drwxr-x--- 2 root dba 4096 Jul 20 21:31 crsd

drwxr-x--- 2 root dba 4096 Jul 21 11:26 ohasd

drwxr-x--- 2 oracrs dba 4096 Jul 21 11:26 gpnpd

drwxr-x--- 2 oracrs dba 4096 Jul 22 19:26 cssd

drwxrwxrwt 2 oracrs dba 12288 Jul 23 18:15 client

drwxr-x--- 2 root dba 4096 Jul 23 22:24 ctssd

drwxr-x--- 2 root dba 4096 Jul 24 03:50 crfmond

drwxr-x--- 2 oracrs dba 4096 Jul 24 07:12 gipcd

-rw-rw-r-- 1 oracrs dba 3244917 Jul 24 10:58 alertNode1.log

[oracrs@Node1port1018ovl]$ pwd

/u02/oracle/11.2.0.4/grid/log/Node1

1) Cluster logs ---> $ORACLE_BASE/diag/crs/<HOSTNAME>/crs/trace/

Can you collect ALL logs manually from the below location?. Kindly zip and upload it.

++ /u02/oracle/PRD/diag/crs/Node1/crs/trace/
++ OS log

Collect CRS logs from other nodes as well.

OS logs provided was from 1-May where as issue datea was on 19-April around 2:20

Please upload the OS logs for 19-April , AS discussed you have issue in TFA install/usage timely you can collect the below ZIP and upload from EACH node

1) Cluster logs ---> $ORACLE_BASE/diag/crs/<HOSTNAME>/crs/trace/

lsinventory from GRID and RDBMS instances

2 ) RDBMS traces
--------------
${ORACLE_BASE}/diag/rdbms/$DBNAME/${ORACLE_SID}/trace and ${ORACLE_BASE}/diag/rdbms/$DBNAME/${ORACLE_SID}/trace

3) ASM traces
------------
${ORACLE_BASE}/diag/asm/+asm/+ASMn/trace and ${ORACLE_BASE}/diag/asm/+asm/+ASMn/trace

4) OS logs of 19 April

AIX /bin/errpt -a > /tmp/error_$hostname.log Command needs to be run as root upload logfile generated in /tmp/

Tuesday, 21 July 2020

Manually apply patch on Grid as well as Oracle home step by steps(11.2.0.4.)

Manually apply patch on Grid as well as Oracle home step by steps(11.2.0.4.)

1) Download Oracle April Patch 2020 and opatch utility patch

a) Patch 30805461 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.200414

b) p6880880_112000_Linux-x86-64.zip

2) Check opatch version and recommended to have latest opatch and download it from oracle support (6880880) and unzip it in Grid Home and all available home folder after renaming Old Opatch folder

You must use the OPatch utility version 11.2.0.3.23 or later to apply this patch.

We need to apply patch on Grid first then each database running on that node

3) Stop Database and take backup of Inventory,Grid Home and Home binary

[oracrs@Node1 ~]$

[oracrs@Node1 ~]$ ps -ef|grep pmon

TES1TRAC 1105 1 0 Jun26 ? 00:17:44 ora_pmon_TES1TRAC1

oracrs 4926 1 0 Jun26 ? 00:11:33 asm_pmon_+ASM1

oracrs 6852 6707 0 08:27 pts/0 00:00:00 grep pmon

TES2TRAC 32029 1 0 Jun26 ? 00:20:14 ora_pmon_TES2TRAC1

fuo11nft 32732 1 0 Jun26 ? 00:20:33 ora_pmon_TES3TRAC1

[oracrs@Node1 ~]$

To Stop database instance

srvctl stop instance -d TES1TRAC -i TES1TRAC1 -o immediate

srvctl stop instance -d TES2TRAC -i TES2TRAC1 -o immediate

srvctl stop instance -d TES3TRAC -i TES3TRAC1 -o immediate

To check opatch version and validate inventory on Grid Home

[oracrs@Node1 OPatch]$ ./opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u02/oracle/11.2.0.4/grid

Central Inventory : /u01/oracle/oraInventory

from : /u02/oracle/11.2.0.4/grid/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_08-33-56AM_1.log

Lsinventory Output file location : /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2020-07-21_08-33-56AM.txt

--------------------------------------------------------------------------------

Local Machine Information::

Hostname: Node1

ARU platform id: 226

ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g 11.2.0.4.0

There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------

OPatch succeeded.

[oracrs@Node1 OPatch]$ ./opatch version

OPatch Version: 11.2.0.3.25

OPatch succeeded.

[oracrs@Node1 OPatch]$

stop crs from root

+ASM1:/u02/oracle/11.2.0.4/grid:N # line added by Agent

TES2TRAC:/u01/oracle/TES2TRAC/product/11.2.0.4:N # line added by Agent

fuo11nft:/u01/oracle/fuo11nft/product/11.2.0.4:N # line added by Agent

TES1TRAC:/u01/oracle/TES1TRAC/product/11.2.0.4:N # line added by Agent

[root@Node1 ~]# cd /u02/oracle/11.2.0.4/grid/bin

[root@Node1 bin]# ./crsctl stop crs

- After stopping CRS and take backup of Grid inventory

cd /u01/oracle/oraInventory

tar -cvf /staging/grid_home_inventory.tar .

du -hP /staging/grid_home_inventory.tar

unzip downloaded patch from Grid user and change permission

chmod -R 775 30805461

1) start clusterware and stop database instance on one node

ps -ef|grep pmon

ps -ef|grep tns

Apply patch on Grid home

------------------------

Patch details

-----------

[oracrs@port1019ovl 30805461]$ ls -ltr

total 228

drwxrwxr-x 5 oracrs dba 4096 Jun 26 2019 29509309 - only grid

drwxrwxr-x 5 oracrs dba 4096 Aug 23 2019 29938455 -Both DB homes and Grid home

drwxrwxr-x 28 oracrs dba 4096 Mar 20 07:09 30670774 - Both DB homes and Grid home

-rwxrwxr-x 1 oracrs dba 128986 Mar 23 06:07 PatchSearch.xml

-rwxrwxr-x 1 oracrs dba 549 Mar 23 07:27 bundle.xml

-rwxrwxr-x 1 oracrs dba 0 Mar 23 07:28 README.txt

-rwxrwxr-x 1 oracrs dba 0 Mar 23 07:28 atp_lfp

-rwxrwxr-x 1 oracrs dba 85893 Apr 13 06:59 README.html

[oracrs@port1019ovl 30805461]$

30670774-Both DB homes and Grid home

29938455-OCW PATCH SET UPDATE 11.2.0.4.191015-Both DB homes and Grid home

29509309-ACFS PATCH SET UPDATE 11.2.0.4.190716Footnote 1-Only Grid home

Validate and update Opatch version and Grid Inventory

-----------------------------------------------------------------

update opatch version taking backup of old opatch folder

and unzipping newly download opatch as below

from root user

mv OPatch OPatch_old

cp /staging/patch_sf_2020/p6880880_112000_Linux-x86-64.zip /u02/oracle/11.2.0.4/grid

unzip p6880880_112000_Linux-x86-64.zip

chown -R oracrs:dba OPatch

chmod 775 OPatch

[root@port1021ovl grid]# chown -R oracrs:dba OPatch

[root@port1021ovl grid]# chmod 775 OPatch

[root@port1021ovl grid]# su - oracrs

and set environment for GRID using Grid owner

[oracrs@TES1TRAC l ~]$ . oraenv

ORACLE_SID = [+ASM2] ?

[oracrs@ TES1TRAC ~]$ cd $ORACLE_HOME/OPatch

[oracrs@ TES1TRAC OPatch]$ ./opatch version

OPatch Version: 11.2.0.3.25

OPatch succeeded.

[oracrs@TES1TRAC OPatch]$

[TES1TRAC@Node1 OPatch]$ ./opatch version

OPatch Version: 11.2.0.3.25

OPatch succeeded.

[TES1TRAC@Node1 OPatch]$

From root user

unlock grid home ,it will restart Grid automatically

cd <$ORACLE_HOME>/crs/install

cd /u02/oracle/11.2.0.4/grid/crs/install

./rootcrs.pl -unlock

From Grid user apply patch manually on grid home

set GRID HOME

/u02/oracle/11.2.0.4/grid/OPatch/opatch lsinventory -detail -oh /u02/oracle/11.2.0.4/grid

cd /u02/oracle/11.2.0.4/grid/OPatch

export PATH=$PATH:/u02/oracle/11.2.0.4/grid/Opatch

opatch napply -oh <Grid Home> -local /<Patch location>/30805461/30670774

opatch napply -oh /u02/oracle/11.2.0.4/grid -local /staging/patch_sf_2020/30805461/30670774

opatch apply -oh /u02/oracle/11.2.0.4/grid -local /staging/patch_sf_2020/30805461/29938455

opatch apply -oh /u02/oracle/11.2.0.4/grid -local /staging/patch_sf_2020/30805461/29509309

Root user lock grid home and start grid if not started automatically

cd <Grid Home>/crs/install/rootcrs.pl -patch

cd /u02/oracle/11.2.0.4/grid/crs/install/rootcrs.pl -patch

start crs on node1

start database on node1

==========Grid Patch completed====================

Apply patch on Each Oracle home and only two patch will applied on Oracle Home

------------------------

Start database one by one which database you want to be patched

srvctl start instance -d TES1TRAC -i TES1TRAC1

srvctl start instance -d TES2TRAC -i TES2TRAC1

srvctl start instance -d TES3TRAC -i TES3TRAC1

1) Validate and update Opatch version and inventory

mv OPatch OPatch_old

cp /staging/patch_sf_2020/p6880880_112000_Linux-x86-64.zip /u01/oracle/TES2TRAC/product/11.2.0.4

unzip p6880880_112000_Linux-x86-64.zip

2) verify patch version

[TES1TRAC@Node1 OPatch]$ ./opatch version

OPatch Version: 11.2.0.3.25

OPatch succeeded.

[TES1TRAC@Node1 OPatch]$

Validate inventory for each oracle RAC home

./opatch lsinventory -detail -oh <Oracle Home>

./opatch lsinventory -detail -oh /u01/oracle/TES1TRAC/product/11.2.0.4

start database instance

srvctl start instance -d TES1TRAC -i TES1TRAC1

set oracle home and run from instance owner

Apply patch on oracle home as below

1) run prepatch script

-------------

<patch_location>/29938455/custom/server/29938455/custom/scripts/prepatch.sh -dbhome <Oracle Home>

/staging/patch_sf_2020/30805461/29938455/custom/server/29938455/custom/scripts/prepatch.sh -dbhome /u01/oracle/TES1TRAC/product/11.2.0.4

Apply first patch

--------------

$ORACLE_HOME/OPatch/opatch napply -oh <Oracle Home> -local /staging/patch_sf_2020/30805461/29938455/custom/server/29938455

$ORACLE_HOME/OPatch/opatch napply -oh /u01/oracle/TES1TRAC/product/11.2.0.4 -local /staging/patch_sf_2020/30805461/29938455/custom/server/29938455

Go to directory of patch and Apply Second patch

----------------------------------------------

cd /staging/patch_sf_2020/30805461/30670774

export PATH=$PATH:/u01/oracle/TES1TRAC/product/11.2.0.4/OPatch

opatch apply

run post patch script

----------------

/staging/patch_sf_2020/30805461/29938455/custom/server/29938455/custom/scripts/postpatch.sh -dbhome /u01/oracle/TES1TRAC/product/11.2.0.4

checked applied patches

---------------

[fuo11nft@Node1 30670774]$ opatch lspatches

30670774;Database Patch Set Update : 11.2.0.4.200414 (30670774)

29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)

OPatch succeeded.

[fuo11nft@Node1 30670774]$

Database need to started manually

srvctl start instance -d TES1TRAC -i TES1TRAC1

load modified SQL files

the following steps load modified SQL files into the database. For an Oracle RAC environment, perform these steps on only one node.

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows:

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog

SQL> CONNECT / AS SYSDBA

SQL> STARTUP

SQL> @catbundle.sql psu apply

SQL> QUIT

The catbundle.sql execution is reflected in the dba_registry_history view by a row associated with bundle series PSU.

For information about the catbundle.sql script, see My Oracle Support Document 605795.1 Introduction to Oracle Database catbundle.sql.

If the OJVM PSU was applied for a previous GI PSU patch, you may see invalid Java classes after execution of the catbundle.sql script in the previous step. If this is the case, run utlrp.sql to re-validate these Java classes.

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog

SQL> CONNECT / AS SYSDBA

SQL> @utlrp.sql

EXECUTE dbms_stats.gather_dictionary_stats;

RMCPRD

rman_connect="rmc_hal_${DATABASE}/${DATABASE}_rmc_hal@${rman_sid}"

$ rman catalog username/password@RMCPRD

RMAN> UPGRADE CATALOG;

verification

select count(1) from dba_objects where status like 'INVALID'

set lines 200 pages 200

col action_time format a40

col action format a30

col namespace format a10

col comments format a30

col version format a13

set lin 200 head on feed on pagesize 100

select i.instance_name,r.* from v$instance i, registry$history r

set lines 200 pages 200

column comp_name format a50

SELECT comp_id,comp_name, status, version FROM dba_registry

TVG16_IssueLogROW24_BlankDB

Mon 07/13/2020 5:30 PM - 6:30 PM

repeat same step for other database home

Details

Apply patch on Grid home

oracrs@Node1 patch_sf_2020]$

[oracrs@Node1 patch_sf_2020]$ ls -ltr

total 1365504

drwxr-xr-x 5 oracrs dba 4096 Mar 23 07:28 30805461

-rwxrwxr-x 1 oracrs dba 120388005 Jul 20 11:41 p6880880_112000_Linux-x86-64.zip

-rwxrwxr-x 1 oracrs dba 1277877242 Jul 20 11:42 p30805461_112040_Linux-x86-64.zip

[oracrs@Node1 patch_sf_2020]$ ls -ltr 30805461

total 228

drwxr-xr-x 5 oracrs dba 4096 Jun 26 2019 29509309

drwxr-xr-x 5 oracrs dba 4096 Aug 23 2019 29938455

drwxr-xr-x 28 oracrs dba 4096 Mar 20 07:09 30670774

-rw-rw-r-- 1 oracrs dba 128986 Mar 23 06:07 PatchSearch.xml

-rw-r--r-- 1 oracrs dba 549 Mar 23 07:27 bundle.xml

-rw-r--r-- 1 oracrs dba 0 Mar 23 07:28 README.txt

-rw-r--r-- 1 oracrs dba 0 Mar 23 07:28 atp_lfp

-rw-rw-r-- 1 oracrs dba 85893 Apr 13 06:59 README.html

[oracrs@Node1 patch_sf_2020]$ unzip p30805461_112040_Linux-x86-64.zip

[oracrs@Node1 patch_sf_2020]$ chmod -R 775 30805461

[oracrs@Node1 patch_sf_2020]$ cd $ORACLE_HOME

[oracrs@Node1 grid]$

[oracrs@Node1 grid]$ ps -ef|grep pmon

oracrs 4849 1 0 10:11 ? 00:00:00 asm_pmon_+ASM1

oracrs 22493 20380 0 11:06 pts/0 00:00:00 grep pmon

[oracrs@Node1 grid]$ ps -ef|grep tns

root 29 2 0 10:09 ? 00:00:00 [netns]

oracrs 5934 1 0 10:11 ? 00:00:20 /u02/oracle/11.2.0.4/grid/bin/tnslsnr LISTENER -inherit

oracrs 5958 1 0 10:11 ? 00:00:00 /u02/oracle/11.2.0.4/grid/bin/tnslsnr LISTENER_SCAN1 -inherit

oracrs 22497 20380 0 11:06 pts/0 00:00:00 grep tns

[oracrs@Node1 grid]$ cd OPatch

[oracrs@Node1 OPatch]$ ./opatch version

OPatch Version: 11.2.0.3.25

OPatch succeeded.

[oracrs@Node1 OPatch]$ /u02/oracle/11.2.0.4/grid/OPatch/opatch lsinventory -detail -oh /u02/oracle/11.2.0.4/grid

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u02/oracle/11.2.0.4/grid

Central Inventory : /u01/oracle/oraInventory

from : /u02/oracle/11.2.0.4/grid/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_11-07-34AM_1.log

Lsinventory Output file location : /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2020-07-21_11-07-34AM.txt

--------------------------------------------------------------------------------

Local Machine Information::

Hostname: Node1

ARU platform id: 226

ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Grid Infrastructure 11g 11.2.0.4.0

There are 1 products installed in this Oracle Home.

Installed Products (89):

Agent Required Support Files 10.2.0.4.5

Assistant Common Files 11.2.0.4.0

Automatic Storage Management Assistant 11.2.0.4.0

Bali Share 1.1.18.0.0

Buildtools Common Files 11.2.0.4.0

Character Set Migration Utility 11.2.0.4.0

Cluster Ready Services Files 11.2.0.4.0

Cluster Verification Utility Common Files 11.2.0.4.0

Cluster Verification Utility Files 11.2.0.4.0

Database SQL Scripts 11.2.0.4.0

Deinstallation Tool 11.2.0.4.0

Enterprise Manager Common Core Files 10.2.0.4.5

Enterprise Manager Common Files 10.2.0.4.5

Enterprise Manager plugin Common Files 11.2.0.4.0

Expat libraries 2.0.1.0.1

HAS Common Files 11.2.0.4.0

HAS Files for DB 11.2.0.4.0

Installation Common Files 11.2.0.4.0

Installation Plugin Files 11.2.0.4.0

Installer SDK Component 11.2.0.4.0

Java Development Kit 1.5.0.51.10

LDAP Required Support Files 11.2.0.4.0

OLAP SQL Scripts 11.2.0.4.0

Oracle Advanced Security 11.2.0.4.0

Oracle Clusterware RDBMS Files 11.2.0.4.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Oracle Containers for Java 11.2.0.4.0

Oracle Core Required Support Files 11.2.0.4.0

Oracle Database 11g 11.2.0.4.0

Oracle Database 11g Multimedia Files 11.2.0.4.0

Oracle Database Deconfiguration 11.2.0.4.0

Oracle Database User Interface 2.2.13.0.0

Oracle Database Utilities 11.2.0.4.0

Oracle DBCA Deconfiguration 11.2.0.4.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

Oracle Globalization Support 11.2.0.4.0

Oracle Grid Infrastructure 11g 11.2.0.4.0

Oracle Help For Java 4.2.9.0.0

Oracle Ice Browser 5.2.3.6.0

Oracle Internet Directory Client 11.2.0.4.0

Oracle Java Client 11.2.0.4.0

Oracle JDBC/OCI Instant Client 11.2.0.4.0

Oracle JDBC/THIN Interfaces 11.2.0.4.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

Oracle JVM 11.2.0.4.0

Oracle LDAP administration 11.2.0.4.0

Oracle Locale Builder 11.2.0.4.0

Oracle Multimedia 11.2.0.4.0

Oracle Multimedia Client Option 11.2.0.4.0

Oracle Multimedia Java Advanced Imaging 11.2.0.4.0

Oracle Multimedia Locator 11.2.0.4.0

Oracle Multimedia Locator RDBMS Files 11.2.0.4.0

Oracle Net 11.2.0.4.0

Oracle Net Listener 11.2.0.4.0

Oracle Net Required Support Files 11.2.0.4.0

Oracle Netca Client 11.2.0.4.0

Oracle Notification Service 11.2.0.3.0

Oracle Notification Service (eONS) 11.2.0.4.0

Oracle One-Off Patch Installer 11.2.0.3.4

Oracle Quality of Service Management (Client) 11.2.0.4.0

Oracle Quality of Service Management (Server) 11.2.0.4.0

Oracle RAC Deconfiguration 11.2.0.4.0

Oracle RAC Required Support Files-HAS 11.2.0.4.0

Oracle Recovery Manager 11.2.0.4.0

Oracle Security Developer Tools 11.2.0.4.0

Oracle Text Required Support Files 11.2.0.4.0

Oracle Universal Installer 11.2.0.4.0

Oracle USM Deconfiguration 11.2.0.4.0

Oracle Wallet Manager 11.2.0.4.0

Parser Generator Required Support Files 11.2.0.4.0

Perl Interpreter 5.10.0.0.2

Perl Modules 5.10.0.0.1

PL/SQL 11.2.0.4.0

PL/SQL Embedded Gateway 11.2.0.4.0

Platform Required Support Files 11.2.0.4.0

Precompiler Required Support Files 11.2.0.4.0

RDBMS Required Support Files 11.2.0.4.0

RDBMS Required Support Files for Instant Client 11.2.0.4.0

RDBMS Required Support Files Runtime 11.2.0.4.0

Required Support Files 11.2.0.4.0

Secure Socket Layer 11.2.0.4.0

SQL*Plus 11.2.0.4.0

SQL*Plus Files for Instant Client 11.2.0.4.0

SQL*Plus Required Support Files 11.2.0.4.0

SSL Required Support Files for InstantClient 11.2.0.4.0

Universal Storage Manager Files 11.2.0.4.0

XDK Required Support Files 11.2.0.4.0

XML Parser for Java 11.2.0.4.0

There are 89 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------

OPatch succeeded.

[oracrs@Node1 OPatch]$ cd cd /u02/oracle/11.2.0.4/grid/crs/install

-bash: cd: cd: No such file or directory

[oracrs@Node1 OPatch]$ cd /u02/oracle/11.2.0.4/grid/crs/install

[oracrs@Node1 install]$ ls -ltr /staging/patch_sf_2020/30805461/30670774

total 176

-rwxrwxr-x 1 oracrs dba 21 Mar 20 07:04 README.txt

-rwxrwxr-x 1 oracrs dba 64562 Mar 20 07:04 README.html

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 24006111

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 22502456

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 26925576

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 23054359

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 29913194

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 19769489

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 26392168

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 30298532

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 29141056

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 24732075

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 20760982

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 18031668

drwxrwxr-x 5 oracrs dba 4096 Mar 20 07:04 29497421

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 25869727

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 27734982

drwxrwxr-x 5 oracrs dba 4096 Mar 20 07:04 28729262

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 26609445

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 19121551

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 21352635

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 18522509

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 17478514

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:04 21948347

drwxrwxr-x 5 oracrs dba 4096 Mar 20 07:05 28204707

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:05 20299013

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:05 27338049

drwxrwxr-x 4 oracrs dba 4096 Mar 20 07:05 30670774

-rwxrwxr-x 1 oracrs dba 3073 Mar 20 07:09 patchmd.xml

[oracrs@Node1 install]$ ls -ltr /staging/patch_sf_2020/30805461/29938455

total 32

drwxrwxr-x 4 oracrs dba 4096 Aug 23 2019 etc

drwxrwxr-x 4 oracrs dba 4096 Aug 23 2019 custom

-rwxrwxr-x 1 oracrs dba 18460 Aug 23 2019 README.txt

drwxrwxr-x 15 oracrs dba 4096 Aug 23 2019 files

[oracrs@Node1 install]$ ls -ltr /staging/patch_sf_2020/30805461/29509309

total 20

-rwxrwxr-x 1 oracrs dba 5956 Jun 26 2019 README.txt

drwxrwxr-x 4 oracrs dba 4096 Jun 26 2019 etc

drwxrwxr-x 3 oracrs dba 4096 Jun 26 2019 custom

drwxrwxr-x 6 oracrs dba 4096 Jun 26 2019 files

[oracrs@Node1 install]$ exit

logout

1) unlock Grid home from root user

[root@Node1 grid]# cd cd /u02/oracle/11.2.0.4/grid/crs/install

[root@Node1 grid]# cd /u02/oracle/11.2.0.4/grid/crs/install

[root@Node1 install]# ./rootcrs.pl -unlock

Using configuration parameter file: ./crsconfig_params

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'Node1'

CRS-2673: Attempting to stop 'ora.crsd' on 'Node1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'Node1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'Node1'

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'Node1'

CRS-2673: Attempting to stop 'ora.registry.acfs' on 'Node1'

CRS-2673: Attempting to stop 'ora.FSBDATA.dg' on 'Node1'

CRS-2673: Attempting to stop 'ora.FSBFLASH.dg' on 'Node1'

CRS-2673: Attempting to stop 'ora.FUBDATA.dg' on 'Node1'

CRS-2673: Attempting to stop 'ora.FUBFLASH.dg' on 'Node1'

CRS-2673: Attempting to stop 'ora.FUODATA.dg' on 'Node1'

CRS-2673: Attempting to stop 'ora.FUOFLASH.dg' on 'Node1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'Node1'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.Node1.vip' on 'Node1'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'Node1'

CRS-2677: Stop of 'ora.FSBFLASH.dg' on 'Node1' succeeded

CRS-2677: Stop of 'ora.FUBDATA.dg' on 'Node1' succeeded

CRS-2677: Stop of 'ora.FUODATA.dg' on 'Node1' succeeded

CRS-2677: Stop of 'ora.FSBDATA.dg' on 'Node1' succeeded

CRS-2677: Stop of 'ora.registry.acfs' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.ACFS_CTRLM.dg' on 'Node1'

CRS-2677: Stop of 'ora.FUOFLASH.dg' on 'Node1' succeeded

CRS-2677: Stop of 'ora.Node1.vip' on 'Node1' succeeded

CRS-2672: Attempting to start 'ora.Node1.vip' on 'port1019ovl'

CRS-2677: Stop of 'ora.FUBFLASH.dg' on 'Node1' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'Node1' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'port1019ovl'

CRS-2676: Start of 'ora.Node1.vip' on 'port1019ovl' succeeded

CRS-2677: Stop of 'ora.ACFS_CTRLM.dg' on 'Node1' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'port1019ovl' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'port1019ovl'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'port1019ovl' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'Node1'

CRS-2677: Stop of 'ora.asm' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'Node1'

CRS-2677: Stop of 'ora.ons' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'Node1'

CRS-2677: Stop of 'ora.net1.network' on 'Node1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'Node1' has completed

CRS-2677: Stop of 'ora.crsd' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'Node1'

CRS-2673: Attempting to stop 'ora.evmd' on 'Node1'

CRS-2673: Attempting to stop 'ora.asm' on 'Node1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'Node1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'Node1'

CRS-2677: Stop of 'ora.evmd' on 'Node1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'Node1' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on 'Node1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'Node1' succeeded

CRS-2677: Stop of 'ora.asm' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'Node1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'Node1'

CRS-2677: Stop of 'ora.cssd' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'Node1'

CRS-2677: Stop of 'ora.crf' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'Node1'

CRS-2677: Stop of 'ora.gipcd' on 'Node1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'Node1'

CRS-2677: Stop of 'ora.gpnpd' on 'Node1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'Node1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

Successfully unlock /u02/oracle/11.2.0.4/grid

[root@Node1 install]#

3) Start applying patch now

[oracrs@Node1 ~]$ cd /u02/oracle/11.2.0.4/grid/OPatch

[oracrs@Node1 OPatch]$ export PATH=$PATH:/u02/oracle/11.2.0.4/grid/OPatch

[oracrs@Node1 OPatch]$ ls -ltr /staging/patch_sf_2020/30805461/29509309

total 20

-rwxrwxr-x 1 oracrs dba 5956 Jun 26 2019 README.txt

drwxrwxr-x 4 oracrs dba 4096 Jun 26 2019 etc

drwxrwxr-x 3 oracrs dba 4096 Jun 26 2019 custom

drwxrwxr-x 6 oracrs dba 4096 Jun 26 2019 files

[oracrs@Node1 OPatch]$ export PATH=$PATH:/u02/oracle/11.2.0.4/grid/OPatch

[oracrs@Node1 OPatch]$ opatch napply -oh /u02/oracle/11.2.0.4/grid -local /staging/patch_sf_2020/30805461/30670774

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u02/oracle/11.2.0.4/grid

Central Inventory : /u01/oracle/oraInventory

from : /u02/oracle/11.2.0.4/grid/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_11-17-14AM_1.log

Verifying environment and performing prerequisite checks...

OPatch continues with these patches: 17478514 18031668 18522509 19121551 19769489 20299013 20760982 21352635 21948347 22502456 23054359 24006111 24732075 25869727 26609445 26392168 26925576 27338049 27734982 28204707 28729262 29141056 29497421 29913194 30298532 30670774

Do you want to proceed? [y|n]

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u02/oracle/11.2.0.4/grid')

Is the local system ready for patching? [y|n]

User Responded with: Y

Backing up files...

Applying sub-patch '17478514' to OH '/u02/oracle/11.2.0.4/grid'

ApplySession: Optional component(s) [ oracle.sdo, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...

Patching component oracle.swd.oui, 11.2.0.4.0...

OPatch found the word "error" in the stderr of the make command.

Please look at this stderr. You can re-run this make command.

Stderr output:

chmod: changing permissions of `/u02/oracle/11.2.0.4/grid/bin/extjobO': Operation not permitted

make: [iextjob] Error 1 (ignored)

Composite patch 30670774 successfully applied.

OPatch Session completed with warnings.

Log file location: /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_11-17-14AM_1.log

OPatch completed with warnings.

apply another patch

[oracrs@Node1 OPatch]$

[oracrs@Node1 OPatch]$ opatch apply -oh /u02/oracle/11.2.0.4/grid -local /staging/patch_sf_2020/30805461/29938455

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u02/oracle/11.2.0.4/grid

Central Inventory : /u01/oracle/oraInventory

from : /u02/oracle/11.2.0.4/grid/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_11-21-31AM_1.log

Verifying environment and performing prerequisite checks...

OPatch continues with these patches: 29938455

Do you want to proceed? [y|n]

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u02/oracle/11.2.0.4/grid')

Is the local system ready for patching? [y|n]

User Responded with: Y

Backing up files...

Applying interim patch '29938455' to OH '/u02/oracle/11.2.0.4/grid'

Patching component oracle.crs, 11.2.0.4.0...

Patch 29938455 successfully applied.

Log file location: /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_11-21-31AM_1.log

OPatch succeeded.

apply last patch

[oracrs@Node1 OPatch]$ opatch apply -oh /u02/oracle/11.2.0.4/grid -local /staging/patch_sf_2020/30805461/29509309

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u02/oracle/11.2.0.4/grid

Central Inventory : /u01/oracle/oraInventory

from : /u02/oracle/11.2.0.4/grid/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_11-22-17AM_1.log

Verifying environment and performing prerequisite checks...

OPatch continues with these patches: 29509309

Do you want to proceed? [y|n]

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u02/oracle/11.2.0.4/grid')

Is the local system ready for patching? [y|n]

User Responded with: Y

Backing up files...

Applying interim patch '29509309' to OH '/u02/oracle/11.2.0.4/grid'

Patching component oracle.usm, 11.2.0.4.0...

Patch 29509309 successfully applied.

Log file location: /u02/oracle/11.2.0.4/grid/cfgtoollogs/opatch/opatch2020-07-21_11-22-17AM_1.log

OPatch succeeded.

[oracrs@Node1 OPatch]$

root@Node1 install]# id

uid=0(root) gid=0(root) groups=0(root)

[root@Node1 install]# cd cd /u02/oracle/11.2.0.4/grid//crs/install

[root@Node1 install]# cd /u02/oracle/11.2.0.4/grid//crs/install

[root@Node1 install]# ./rootcrs.pl -patch

Using configuration parameter file: ./crsconfig_params

Installing Trace File Analyzer

CRS-4123: Oracle High Availability Services has been started.

[root@Node1 install]#

root@Node1 install]# ps -ef|grep pmon

oracrs 7072 1 0 11:26 ? 00:00:00 asm_pmon_+ASM1

root 9754 8575 0 11:29 pts/0 00:00:00 grep pmon

[root@Node1 install]#

Now patching grid home completed

=================database patching======================

1) run prepatch after validating opatch version and inventory

drwxr-xr-x 4 RACDB dba 4096 Feb 1 2017 log

drwxr-xr-x 7 RACDB dba 4096 Feb 1 2017 install

drwxr-xr-x 4 RACDB dba 12288 Jan 24 2019 lib

drwxr-x--- 15 RACDB dba 4096 Apr 22 11:49 OPatch

drwxr-xr-x 2 RACDB dba 4096 Jul 21 11:31 dbs

[RACDB@Node1 11.2.0.4]$ ls -ltr /staging/patch_sf_2020/30805461/29938455/custom/server/29938455/custom/scripts/prepatch.sh

-rwxrwxr-x 1 oracrs dba 4566 Aug 23 2019 /staging/patch_sf_2020/30805461/29938455/custom/server/29938455/custom/scripts/prepatch.sh

[RACDB@Node1 11.2.0.4]$ cd OPatch

2)apply patch on Database home

[RACDB@Node1 OPatch]$ ./opatch lsinventory -detail -oh /u01/oracle/RACDB/product/11.2.0.4

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u01/oracle/RACDB/product/11.2.0.4

Central Inventory : /u01/oracle/oraInventory

from : /u01/oracle/RACDB/product/11.2.0.4/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u01/oracle/RACDB/product/11.2.0.4/cfgtoollogs/opatch/opatch2020-07-21_11-34-24AM_1.log

Lsinventory Output file location : /u01/oracle/RACDB/product/11.2.0.4/cfgtoollogs/opatch/lsinv/lsinventory2020-07-21_11-34-24AM.txt

--------------------------------------------------------------------------------

Local Machine Information::

Hostname: Node1

ARU platform id: 226

ARU platform description:: Linux x86-64

Installed Top-level Products (1):

Oracle Database 11g 11.2.0.4.0

There are 1 products installed in this Oracle Home.

Installed Products (136):

Agent Required Support Files 10.2.0.4.5

Assistant Common Files 11.2.0.4.0

Bali Share 1.1.18.0.0

Buildtools Common Files 11.2.0.4.0

Character Set Migration Utility 11.2.0.4.0

Cluster Verification Utility Common Files 11.2.0.4.0

Database Configuration and Upgrade Assistants 11.2.0.4.0

Database SQL Scripts 11.2.0.4.0

Database Workspace Manager 11.2.0.4.0

Deinstallation Tool 11.2.0.4.0

Enterprise Edition Options 11.2.0.4.0

Enterprise Manager Agent 10.2.0.4.5

Enterprise Manager Agent Core Files 10.2.0.4.5

Enterprise Manager Common Core Files 10.2.0.4.5

Enterprise Manager Common Files 10.2.0.4.5

Enterprise Manager Database Plugin -- Agent Support 11.2.0.4.0

Enterprise Manager Database Plugin -- Repository Support 11.2.0.4.0

Enterprise Manager Grid Control Core Files 10.2.0.4.5

Enterprise Manager plugin Common Files 11.2.0.4.0

Enterprise Manager Repository Core Files 10.2.0.4.5

Exadata Storage Server 11.2.0.1.0

Expat libraries 2.0.1.0.1

Generic Connectivity Common Files 11.2.0.4.0

HAS Common Files 11.2.0.4.0

HAS Files for DB 11.2.0.4.0

Installation Common Files 11.2.0.4.0

Installation Plugin Files 11.2.0.4.0

Installer SDK Component 11.2.0.4.0

JAccelerator (COMPANION) 11.2.0.4.0

Java Development Kit 1.5.0.51.10

LDAP Required Support Files 11.2.0.4.0

OLAP SQL Scripts 11.2.0.4.0

Oracle 11g Warehouse Builder Required Files 11.2.0.4.0

Oracle Advanced Security 11.2.0.4.0

Oracle Application Express 11.2.0.4.0

Oracle Call Interface (OCI) 11.2.0.4.0

Oracle Clusterware RDBMS Files 11.2.0.4.0

Oracle Code Editor 1.2.1.0.0I

Oracle Configuration Manager 10.3.8.1.0

Oracle Configuration Manager Client 10.3.2.1.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Oracle Containers for Java 11.2.0.4.0

Oracle Core Required Support Files 11.2.0.4.0

Oracle Data Mining RDBMS Files 11.2.0.4.0

Oracle Database 11g 11.2.0.4.0

Oracle Database 11g Multimedia Files 11.2.0.4.0

Oracle Database Deconfiguration 11.2.0.4.0

Oracle Database Gateway for ODBC 11.2.0.4.0

Oracle Database User Interface 2.2.13.0.0

Oracle Database Utilities 11.2.0.4.0

Oracle Database Vault J2EE Application 11.2.0.4.0

Oracle Database Vault option 11.2.0.4.0

Oracle DBCA Deconfiguration 11.2.0.4.0

Oracle Display Fonts 9.0.2.0.0

Oracle Enterprise Manager Console DB 11.2.0.4.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

Oracle Globalization Support 11.2.0.4.0

Oracle Help For Java 4.2.9.0.0

Oracle Help for the Web 2.0.14.0.0

Oracle Ice Browser 5.2.3.6.0

Oracle Internet Directory Client 11.2.0.4.0

Oracle Java Client 11.2.0.4.0

Oracle JDBC Server Support Package 11.2.0.4.0

Oracle JDBC/OCI Instant Client 11.2.0.4.0

Oracle JDBC/THIN Interfaces 11.2.0.4.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

Oracle JVM 11.2.0.4.0

Oracle Label Security 11.2.0.4.0

Oracle LDAP administration 11.2.0.4.0

Oracle Locale Builder 11.2.0.4.0

Oracle Message Gateway Common Files 11.2.0.4.0

Oracle Multimedia 11.2.0.4.0

Oracle Multimedia Annotator 11.2.0.4.0

Oracle Multimedia Client Option 11.2.0.4.0

Oracle Multimedia Java Advanced Imaging 11.2.0.4.0

Oracle Multimedia Locator 11.2.0.4.0

Oracle Multimedia Locator RDBMS Files 11.2.0.4.0

Oracle Net 11.2.0.4.0

Oracle Net Listener 11.2.0.4.0

Oracle Net Required Support Files 11.2.0.4.0

Oracle Net Services 11.2.0.4.0

Oracle Netca Client 11.2.0.4.0

Oracle Notification Service 11.2.0.3.0

Oracle Notification Service (eONS) 11.2.0.4.0

Oracle ODBC Driver 11.2.0.4.0

Oracle ODBC Driverfor Instant Client 11.2.0.4.0

Oracle OLAP 11.2.0.4.0

Oracle OLAP API 11.2.0.4.0

Oracle OLAP RDBMS Files 11.2.0.4.0

Oracle One-Off Patch Installer 11.2.0.3.4

Oracle Partitioning 11.2.0.4.0

Oracle Programmer 11.2.0.4.0

Oracle Quality of Service Management (Client) 11.2.0.4.0

Oracle RAC Deconfiguration 11.2.0.4.0

Oracle RAC Required Support Files-HAS 11.2.0.4.0

Oracle Real Application Testing 11.2.0.4.0

Oracle Recovery Manager 11.2.0.4.0

Oracle Security Developer Tools 11.2.0.4.0

Oracle Spatial 11.2.0.4.0

Oracle SQL Developer 11.2.0.4.0

Oracle Starter Database 11.2.0.4.0

Oracle Text 11.2.0.4.0

Oracle Text Required Support Files 11.2.0.4.0

Oracle UIX 2.2.24.6.0

Oracle Universal Connection Pool 11.2.0.4.0

Oracle Universal Installer 11.2.0.4.0

Oracle USM Deconfiguration 11.2.0.4.0

Oracle Wallet Manager 11.2.0.4.0

Oracle XML Development Kit 11.2.0.4.0

Oracle XML Query 11.2.0.4.0

Parser Generator Required Support Files 11.2.0.4.0

Perl Interpreter 5.10.0.0.2

Perl Modules 5.10.0.0.1

PL/SQL 11.2.0.4.0

PL/SQL Embedded Gateway 11.2.0.4.0

Platform Required Support Files 11.2.0.4.0

Precompiler Common Files 11.2.0.4.0

Precompiler Required Support Files 11.2.0.4.0

Provisioning Advisor Framework 10.2.0.4.3

RDBMS Required Support Files 11.2.0.4.0

RDBMS Required Support Files for Instant Client 11.2.0.4.0

RDBMS Required Support Files Runtime 11.2.0.4.0

regexp 2.1.9.0.0

Required Support Files 11.2.0.4.0

Sample Schema Data 11.2.0.4.0

Secure Socket Layer 11.2.0.4.0

SQL*Plus 11.2.0.4.0

SQL*Plus Files for Instant Client 11.2.0.4.0

SQL*Plus Required Support Files 11.2.0.4.0

SQLJ Runtime 11.2.0.4.0

SSL Required Support Files for InstantClient 11.2.0.4.0

XDK Required Support Files 11.2.0.4.0

XML Parser for Java 11.2.0.4.0

XML Parser for Oracle JVM 11.2.0.4.0

There are 136 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------

OPatch succeeded.

[RACDB@Node1 OPatch]$ /staging/patch_sf_2020/30805461/29938455/custom/server/29938455/custom/scripts/prepatch.sh -dbhome /u01/oracle/RACDB/product/11.2.0.4

/staging/patch_sf_2020/30805461/29938455/custom/server/29938455/custom/scripts/prepatch.sh completed successfully.

Apply patch on oracle home

[RACDB@Node1 OPatch]$ $ORACLE_HOME/OPatch/opatch napply -oh /u01/oracle/RACDB/product/11.2.0.4 -local /staging/patch_sf_2020/30805461/29938455/custom/server/29938455

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u01/oracle/RACDB/product/11.2.0.4

Central Inventory : /u01/oracle/oraInventory

from : /u01/oracle/RACDB/product/11.2.0.4/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u01/oracle/RACDB/product/11.2.0.4/cfgtoollogs/opatch/opatch2020-07-21_11-34-57AM_1.log

Verifying environment and performing prerequisite checks...

OPatch continues with these patches: 29938455

Do you want to proceed? [y|n]

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/oracle/RACDB/product/11.2.0.4')

Is the local system ready for patching? [y|n]

User Responded with: Y

Backing up files...

Applying interim patch '29938455' to OH '/u01/oracle/RACDB/product/11.2.0.4'

Patching component oracle.rdbms, 11.2.0.4.0...

Patch 29938455 successfully applied.

Log file location: /u01/oracle/RACDB/product/11.2.0.4/cfgtoollogs/opatch/opatch2020-07-21_11-34-57AM_1.log

OPatch succeeded.

apply another patch

[RACDB@Node1 OPatch]$ cd /staging/patch_sf_2020/30805461/30670774

[RACDB@Node1 30670774]$ export PATH=$PATH:/u01/oracle/RACDB/product/11.2.0.4/OPatch

[RACDB@Node1 30670774]$ opatch apply

Oracle Interim Patch Installer version 11.2.0.3.25

Oracle Home : /u01/oracle/RACDB/product/11.2.0.4

Central Inventory : /u01/oracle/oraInventory

from : /u01/oracle/RACDB/product/11.2.0.4/oraInst.loc

OPatch version : 11.2.0.3.25

OUI version : 11.2.0.4.0

Log file location : /u01/oracle/RACDB/product/11.2.0.4/cfgtoollogs/opatch/opatch2020-07-21_11-35-49AM_1.log

Patching component oracle.swd.oui, 11.2.0.4.0...

Patching component oracle.ctx, 11.2.0.4.0...

OPatch found the word "error" in the stderr of the make command.

Please look at this stderr. You can re-run this make command.

Stderr output:

chmod: changing permissions of `/u01/oracle/RACDB/product/11.2.0.4/bin/extjobO': Operation not permitted

make: [iextjob] Error 1 (ignored)

Composite patch 30670774 successfully applied.

OPatch Session completed with warnings.

Log file location: /u01/oracle/RACDB/product/11.2.0.4/cfgtoollogs/opatch/opatch2020-07-21_11-35-49AM_1.log

OPatch completed with warnings.

[RACDB@Node1 30670774]$

verify patch

[RACDB@Node1 30670774]$ opatch lspatches

30670774;Database Patch Set Update : 11.2.0.4.200414 (30670774)

29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)

OPatch succeeded.

[RACDB@Node1 30670774]$ /staging/patch_sf_2020/30805461/29938455/custom/server/29938455/custom/scripts/postpatch.sh -dbhome /u01/oracle/RACDB/product/11.2.0.4

Reading /u01/oracle/RACDB/product/11.2.0.4/install/params.ora..

Parsing file /u01/oracle/RACDB/product/11.2.0.4/bin/racgwrap

Parsing file /u01/oracle/RACDB/product/11.2.0.4/bin/srvctl

Parsing file /u01/oracle/RACDB/product/11.2.0.4/bin/srvconfig

Parsing file /u01/oracle/RACDB/product/11.2.0.4/bin/cluvfy

Verifying file /u01/oracle/RACDB/product/11.2.0.4/bin/racgwrap

Verifying file /u01/oracle/RACDB/product/11.2.0.4/bin/srvctl

Verifying file /u01/oracle/RACDB/product/11.2.0.4/bin/srvconfig

Verifying file /u01/oracle/RACDB/product/11.2.0.4/bin/cluvfy

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/racgwrap

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/srvctl

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/srvconfig

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/cluvfy

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/diskmon.bin

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/lsnodes

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/osdbagrp

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/rawutl

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/srvm/admin/ractrans

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/srvm/admin/getcrshome

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/gnsd

Reapplying file permissions on /u01/oracle/RACDB/product/11.2.0.4/bin/crsdiag.pl

Postpatch completed successfully

[RACDB@Node1 30670774]$