Tuesday 21 April 2020

Configure Goldengate Unidirectional Step by step


Configure Goldengate Unidirectional Step by step

Summary

1) Create New file system for Golden Gate Home
2) Install Oracle GoldenGate software on source and target Machine and create subdirectory from GGSCI .
3)Set up a GoldenGate database user at source database level.
4)Enable supplemental logging at source database level.
5) Enabling Oracle GoldenGate in source the Database
   ENABLE_GOLDENGATE_REPLICATION=true
6) Configure and start change capture to GoldenGate trail files (Extract processes: primary and data pump).
7) Perform initial load to synchronize databases by database-specific or Oracle GoldenGate methods.
8) Configure and start change delivery (Replicat process).

               

Source

1) Set up a GoldenGate database user at source   

          Wallet and master key

                Copy the cwallet.sso created on the target to $OGG_HOME/dirwlt

Credential Store

               
GGSCI (ip-10-190-170-4.eu-west-1.compute.internal) 3> add credentialstore

Credential store created in ./dircrd/.


GGSCI (ip-10-190-170-4.eu-west-1.compute.internal) 6> alter credentialstore add user SD_OGG@APPPROD,  alias sd_ogg
Password:

Credential store in ./dircrd/ altered.

Supplemental Logging


DBLOGIN USERIDALIAS sd_ogg
ADD SCHEMATRANDATA TEST_DBA [ALLOWNONVALIDATEDKEYS]

2017-04-12 11:10:36  INFO    OGG-01788  SCHEMATRANDATA has been added on schema test_dba.

2017-04-12 11:10:36  INFO    OGG-01976  SCHEMATRANDATA for scheduling columns has been added on schema test_dba.


GLOBALS file


GGSCHEMA SD_OGG
ENABLE_HEARTBEAT_TABLE
USE_TRAILDEFS

            Manager process


PORT 7809
PURGEOLDEXTRACTS ./dirdat/cp*, USECHECKPOINTS, MINKEEPHOURS 12
AUTORESTART EXTRACT APP_CAP,  RETRIES 6, WAITMINUTES 10, RESETMINUTES 60
AUTORESTART EXTRACT APP_PUMP, RETRIES 6, WAITMINUTES 10, RESETMINUTES 60
--ALLOWNONVALIDATEDKEYS


Heartbeat Table


DBLOGIN USERIDALIAS sd_ogg
ADD HEARTBEATTABLE, frequency 120, retention_time 10, purge_frequency 2

Extract (Capture process)


                Register the extract process with the database
ADD EXTRACT TEST_CAP, INTEGRATED TRANLOG, BEGIN NOW
ADD EXTTRAIL ./dirdat/cc , EXTRACT TEST_CAP

                TEST_CAP Parameter file
EXTRACT TEST_CAP
USERIDALIAS sd_ogg
ENCRYPTTRAIL AES256
EXTTRAIL ./dirdat/cc
TABLEEXCLUDE TEST_DBA.TC_CREDIT_LIMIT_SNAPSHOT
TABLEEXCLUDE TEST_DBA.ACCOUNT_STATUS_PARAMS
TABLEEXCLUDE TEST_DBA.IDT_ML5910G01
TABLEEXCLUDE TEST_DBA.SL_PROBLEM_PHONES
TABLEEXCLUDE TEST_DBA.IDT_ML5910G03
TABLE TEST_DBA.*


Register the extract process with the database
DBLOGIN USERIDALIAS sd_ogg
register extract TEST_cap database

2017-04-12 11:14:10  INFO    OGG-02003  Extract TEST_CAP successfully registered   with database at SCN 25289823.


            For Initial Load
             capture SCN from Source Database
            1) select Current_SCN from V$database   -- on Source Database
             2) Run EXPDP to a specific SCN.

                  expdp parfile=testdb_table.part
                     
                userid="/ as sysdba"
                DIRECTORY=dumptest
               PARALLEL=4
               COMPRESSION=ALL
                flashback_scn=XXXXXXX
               SCHEMAS=TEST_USER
                METRICS=Y
                DUMPFILE=exp_TEST_USER_23Apr_%U.dmp
               logfile=explog_TEST_USER_23Apr.log

              
                Point EXTRACT to that SCN and start it
ALTER TEST_CAP, SCN XXXXX
INFO TEST_APPP DETAIL
START TEST_CAP




Extract (Pump process)


                Commands to create EXTRACT process:
ADD EXTRACT TEST_PUMP, EXTTRAILSOURCE ./dirdat/oc

                TEST_PUMP Parameter file:
EXTRACT TEST_PUMP
PASSTHRU
RMTHOST ip-10-190-170-4.eu-west-1.compute.internal, MGRPORT 7809
RMTTRAIL ./dirdat/cp
TABLE TEST_DBA.*;

                Start PUMP 
START TEST_PUMP





Target


            Wallet and master key


GGSCI (ip-10-190-170-4.eu-west-1.compute.internal) 2> create wallet

Created wallet at location 'dirwlt'.

Opened wallet at location 'dirwlt'.

GGSCI (ip-10-190-170-4.eu-west-1.compute.internal) 4> add masterkey

Master key ' OGG_DEFAULT_MASTERKEY' added to wallet at location 'dirwlt'.


            Credential Store

               
GGSCI (ip-10-190-170-4.eu-west-1.compute.internal) 3> add credentialstore

Credential store created in ./dircrd/.

alter credentialstore add user SD_OGG@ords1,  alias sd_ogg domain ords1
Password:

Credential store in ./dircrd/ altered.

alter credentialstore add user SD_OGG@ords2,  alias sd_ogg domain ords2
Password:

Credential store in ./dircrd/ altered.

            GLOBALS file


GGSCHEMA SD_OGG
ENABLE_HEARTBEAT_TABLE
USE_TRAILDEFS
--ALLOWNONVALIDATEDKEYS





Manager process


PORT 7809
DYNAMICPORTLIST 7810-7829
PURGEOLDEXTRACTS ./dirdat/cp*, USECHECKPOINTS, MINKEEPHOURS 12
AUTORESTART REPLICAT APP_DEL, RETRIES 6, WAITMINUTES 10, RESETMINUTES 60

Checkpoint Table


DBLOGIN USERIDALIAS sd_ogg domain rds1
add checkpointtable sd_ogg.checkpointtable

Heartbeat Table


DBLOGIN USERIDALIAS sd_ogg domain rds1
ADD HEARTBEATTABLE, frequency 120, retention_time 10, purge_frequency 2

Replicat


                Commands:
ADD REPLICAT TEST_DEL, EXTTRAIL ./dirdat/cp, CHECKPOINTTABLE sd_ogg.checkpointtable

APP_DELIVERY Parameter file
REPLICAT TEST_DEL
--HANDLECOLLISIONS
USERIDALIAS sd_ogg
MAP APP_DBA.*, TARGET APP01PRD_APP_DBA.*;
--MAP pdb2.soe2.logon, target pdb2_soe2.logon keycols(logon_id);
BATCHSQL BATCHESPERQUEUE 100, OPSPERBATCH 2000

                Start REPLICAT process

START TEST_DEL






               



 



Saturday 18 April 2020

Issue during 12c patching



Issue during 12c patching
-------------------------------

Error whil applying patching on cluserware
----------------------------------

OPatchauto session is initiated at Sat Apr 18 03:17:56 2020

System initialization log file is /u02/oracle/12.2.0.1/grid/cfgtoollogs/opatchautodb/systemconfig2020-04-18_03-18-27AM.log.

Session log file is /u02/oracle/12.2.0.1/grid/cfgtoollogs/opatchauto/opatchauto2020-04-18_03-19-42AM.log
The id for this session is BXND

Executing OPatch prereq operations to verify patch applicability on home /u02/oracle/12.2.0.1/grid
Patch applicability verified successfully on home /u02/oracle/12.2.0.1/grid


Bringing down CRS service on home /u02/oracle/12.2.0.1/grid
Prepatch operation log file location: /u02/oracle/PRD/crsdata/fleetwood/crsconfig/crspatch_fleetwood_2020-04-18_03-21-07AM.log
CRS service brought down successfully on home /u02/oracle/12.2.0.1/grid


Start applying binary patch on home /u02/oracle/12.2.0.1/grid
Successfully executed command: /usr/sbin/slibclean

Failed while applying binary patches on home /u02/oracle/12.2.0.1/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : fleetwood->/u02/oracle/12.2.0.1/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u02/oracle/12.2.0.1/grid, host: fleetwood.
Command failed: /u02/oracle/12.2.0.1/grid/OPatch/opatchauto apply /oswatcher/2020_patch/30116802 -oh /u02/oracle/12.2.0.1/grid -target_type cluster -binary -invPtrLoc /u02/oracle/12.2.0.1/grid/oraInst.loc -jre /u02/oracle/12.2.0.1/grid/OPatch/jre -persistresult /u02/oracle/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_fleetwood_crs.ser -analyzedresult /u02/oracle/12.2.0.1/grid/opatchautocfg/db/sessioninfo/sessionresult_analy

IT look like permission issue .in first node it is onwed by root . it should be grid user


fleetwood:root:/home/root:] ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
-rw-r----- 1 root system 140210504 Aug 08 2018 /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
[fleetwood:root:/home/root:]

in second node ,permission look fine.


pending to apply patch

[fairfield:root:/home/root:] ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
-rw-r----- 1 oracrs oinstall 140210504 Oct 17 2018 /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
[fairfield:root:/home/root:]






As per log , there is bug

As per oracle support comments   
Hi,

++ As below bug patches are in place on node#1 now.

BUG 30122814 - OCW OCT 2019
BUG 30122828 - ACFS OCT 2019
BUG 30093408 - TOMCAT

++ You cannot continue with "opatchauto apply" for remaining patches on GI home. The only option is to apply the patch manually using "opatch apply" on node#1.

++ Can you provide the below details?

# ls -lrt /oswatcher/2020_patch/30116802
# <GI_HOME>/bin/kfod op=patches
# <GI_HOME>/bin/kfod op=patchlvl

++ Run the below command on node#1 and share again the below command output

# $GI_HOME/crs/install/rootcrs.sh -prepatch
# $GI_HOME/crs/install/rootcrs.sh -postpatch

# ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip





output of above
--------------------
[fleetwood:root:/home/root:]
[fleetwood:root:/home/root:] ls -lrt /oswatcher/2020_patch/30116802
total 272
drwxrwxr-x    4 oracrs   oinstall        256 Sep 27 2019  26839277
drwxrwxr-x    5 oracrs   oinstall        256 Sep 27 2019  30122828
drwxrwxr-x    4 oracrs   oinstall        256 Sep 27 2019  30093408
drwxrwxr-x    4 oracrs   oinstall        256 Sep 27 2019  30138470
drwxrwxr-x    2 oracrs   oinstall       4096 Sep 27 2019  automation
drwxrwxr-x    5 oracrs   oinstall        256 Sep 27 2019  30122814
-rwxrwxr-x    1 oracrs   oinstall          0 Sep 27 2019  README.txt
-rwxrwxr-x    1 oracrs   oinstall     124674 Sep 27 2019  README.html
-rwxrwxr-x    1 oracrs   oinstall       5828 Sep 27 2019  bundle.xml
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid/bin/kfod op=patches
/u02/oracle/12.2.0.1/grid/bin/kfod[22]: %ORACLE_HOME%/bin/kfod.bin:  not found.
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid/bin/kfod op=patchlvl
/u02/oracle/12.2.0.1/grid/bin/kfod[22]: %ORACLE_HOME%/bin/kfod.bin:  not found.
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid/crs/install/rootcrs.sh -prepatch
Using configuration parameter file: /u02/oracle/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u02/oracle/PRD/crsdata/fleetwood/crsconfig/crspatch_fleetwood_2020-04-18_04-58-22AM.log
[fleetwood:root:/home/root:] /u02/oracle/12.2.0.1/grid//crs/install/rootcrs.sh -postpatch
Using configuration parameter file: /u02/oracle/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u02/oracle/PRD/crsdata/fleetwood/crsconfig/crspatch_fleetwood_2020-04-18_04-58-46AM.log
2020/04/18 05:00:06 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/04/18 05:00:07 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2020/04/18 05:00:55 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'fleetwood'
CRS-2672: Attempting to start 'ora.evmd' on 'fleetwood'
CRS-2676: Start of 'ora.mdnsd' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.evmd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'fleetwood'
CRS-2676: Start of 'ora.gpnpd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'fleetwood'
CRS-2676: Start of 'ora.gipcd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fleetwood'
CRS-2676: Start of 'ora.cssdmonitor' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'fleetwood'
CRS-2672: Attempting to start 'ora.diskmon' on 'fleetwood'
CRS-2676: Start of 'ora.diskmon' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.cssd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'fleetwood'
CRS-2672: Attempting to start 'ora.ctssd' on 'fleetwood'
CRS-2676: Start of 'ora.ctssd' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'fleetwood'
CRS-2676: Start of 'ora.crf' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'fleetwood'
CRS-2676: Start of 'ora.crsd' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'fleetwood'
CRS-2676: Start of 'ora.asm' on 'fleetwood' succeeded
CRS-6017: Processing resource auto-start for servers: fleetwood
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'fleetwood'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'fairfield'
CRS-2673: Attempting to stop 'ora.fleetwood.vip' on 'fairfield'
CRS-2672: Attempting to start 'ora.ons' on 'fleetwood'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'fairfield' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'fairfield'
CRS-2677: Stop of 'ora.fleetwood.vip' on 'fairfield' succeeded
CRS-2672: Attempting to start 'ora.fleetwood.vip' on 'fleetwood'
CRS-2677: Stop of 'ora.scan1.vip' on 'fairfield' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'fleetwood'
CRS-2676: Start of 'ora.fleetwood.vip' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'fleetwood'
CRS-2676: Start of 'ora.scan1.vip' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'fleetwood'
CRS-2676: Start of 'ora.ons' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'fleetwood'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.asm' on 'fleetwood' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_DATA1.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_FRA1.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_REDO1.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.E2B01PRD_REDO2.dg' on 'fleetwood'
CRS-2672: Attempting to start 'ora.MGMT.dg' on 'fleetwood'
CRS-2676: Start of 'ora.E2B01PRD_FRA1.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.E2B01PRD_REDO2.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.MGMT.dg' on 'fleetwood' succeeded
CRS-2676: Start of 'ora.E2B01PRD_REDO1.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.E2B01PRD_DATA1.dg' on 'fleetwood' succeeded
CRS-2664: Resource 'ora.e2b01prd.db' is already running on 'fairfield'
CRS-2676: Start of 'ora.CRS.dg' on 'fleetwood' succeeded
CRS-6016: Resource auto-start has completed for server fleetwood
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1221707709].

SQL Patching tool version 12.2.0.1.0 Production on Sat Apr 18 05:11:01 2020
Copyright (c) 2012, 2018, Oracle.  All rights reserved.

Connecting to database...OK
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED GIMR_DSCREP_10
    Nothing to roll back
    Nothing to apply

SQL Patching tool complete on Sat Apr 18 05:12:20 2020
[fleetwood:root:/home/root:]
[fleetwood:root:/home/root:]
[fleetwood:root:/home/root:] ls -lrt /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
-rw-r-----    1 root     system    140210504 Aug 08 2018  /u02/oracle/12.2.0.1/grid/suptools/orachk.zip
[fleetwood:root:/home/root:]


./u02/oracle/12.2.0.1/grid/crs/install/rootcrs.sh -postpatch

./rootcrs.sh -postpatch




============


As per oracle support recommendation ,we need to apply missing patch manually

Steps for Applying the Patch
======================


Execute the following on each node of the cluster in non-shared CRS and DB home environment to apply the patch.

1. Stop the CRS managed resources running from DB homes.

If this is a GI Home environment, as the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>


Note:

You need to make sure that the Oracle ACFS file systems are unmounted (see Section 2.8) and all other Oracle processes are shutdown before you proceed.



2. Run the pre root script.

If this is a GI Home, as the root user execute:

# <GI_HOME>/crs/install/rootcrs.sh -prepatch

3. Patch GI home.

As the GI home owner execute:


$ <GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/%BUGNO%/%DB WLM TRACKING BUG%

$ <GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/%BUGNO%/%DB RU TRACKING BUG%


5. Run the post script.

As the root user execute:

# <GI_HOME>/rdbms/install/rootadd_rdbms.sh

If this is a GI Home, as the root user execute:

# <GI_HOME>/crs/install/rootcrs.sh -postpatch

6. If the message, "A system reboot is recommended before using ACFS is shown, then a reboot must be issued before continuing.
Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver.

7. Start the CRS managed resources that were earlier running from DB homes.

If this is a GI Home environment, as the database home owner execute:

$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>

8. For each database instance running on the Oracle home being patched, run the datapatch utility as described in next table.


eg



from root user
earlier used command

./opatchauto apply /oswatcher/2020_patch/30116802 -oh /u02/oracle/12.2.0.1/grid


[Apr 18, 2020 3:24:48 AM] [INFO] Patch 30093408 successfully applied.
[Apr 18, 2020 3:25:56 AM] [INFO] Patch 30122814 successfully applied.
[Apr 18, 2020 3:29:30 AM] [INFO] Patch 30122828 successfully applied.
drwxrwxr-x    4 oracrs   oinstall        256 Sep 27 2019  26839277
drwxrwxr-x    5 oracrs   oinstall        256 Sep 27 2019  30122828 -done
drwxrwxr-x    4 oracrs   oinstall        256 Sep 27 2019  30093408 -done
drwxrwxr-x    4 oracrs   oinstall        256 Sep 27 2019  30138470
drwxrwxr-x    5 oracrs   oinstall        256 Sep 27 2019  30122814 -done

done means already applied OPatch autom
<UNZIPPED_PATCH_LOCATION>/30116802/30138470
<UNZIPPED_PATCH_LOCATION>/30116802/30122814-done
<UNZIPPED_PATCH_LOCATION>/30116802/30122828-done
<UNZIPPED_PATCH_LOCATION>/30116802/26839277
<UNZIPPED_PATCH_LOCATION>/30116802/30093408- done

we need to apply on missing patching as below
/oswatcher/2020_patch/30116802/26839277
/oswatcher/2020_patch/30116802/30138470

for 11 g

from root user

cd /u02/oracle/12.2.0.1/grid/crs/install
./rootcrs.pl -unlock

from grid user

cd /u02/oracle/12.2.0.1/grid/OPatch
export PATH=$PATH:/u02/oracle/12.2.0.1/grid/OPatch
opatch napply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/26839277
opatch napply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/30138470



Root user

/u02/oracle/12.2.0.1/grid/crs/install/rootcrs.pl -patch
start crs on node1
start database on node1

from 12c syntax has changed


from root user

cd /u02/oracle/12.2.0.1/grid/crs/install/
./rootcrs.sh -prepatch

from grid user

cd /u02/oracle/12.2.0.1/grid/OPatch
export PATH=$PATH:/u02/oracle/12.2.0.1/grid/OPatch
opatch napply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/26839277
./opatch apply -oh /u02/oracle/12.2.0.1/grid -local /oswatcher/2020_patch/30116802/30138470

from root user
cd /u02/oracle/12.2.0.1/grid/crs/install/
.rootcrs.sh -postpatch


./u02/oracle/12.2.0.1/grid/crs/install/rdbms/install/rootadd_rdbms.sh





Sunday 12 April 2020

Execution Plan Overview and Bad Execution Plan fix in Oracle Database


                              Performance Monitoring


In order to investigate locking session and killing session, DBA need to follow below steps


Login on Database and check blocking session with below commands

1) To check session lock on database

SELECT DECODE(request,0,'Holder: ','Waiter: ') ||
sid sess, id1, id2, lmode, request, type
FROM V$LOCK
WHERE (id1, id2, type) IN (SELECT id1, id2, type FROM V$LOCK WHERE request > 0)
ORDER BY id1, request;


2) To identify session as well as SQL details

select sid,serial#,osuser,username,program,status,SQl_id from v$session where sid=&sid;

or

select s.username, s.sid, s.serial#,t.sql_text "Last SQL"from v$session s, v$sqlarea t where s.sql_address =t.address and
s.sql_hash_value =t.hash_value and s.sid = '&sid';


3) Map OS processes with database process as below

select p.spid,s.sql_id,s.sql_hash_value,s.sql_id,s.sid,s.serial#,s.username,s.event,to_char(s.logon_time,'dd-mon:hh24:mi') from v$session s, v$process p
where p.addr=s.paddr and s.sid  = '&sid';


4) Session can be killed from OS prompt as well as database level

From OS level

$ kill -9 <spid>


From Database level

alter system kill session 'sid,serial#' immediate;
eg

alter system kill session '41,5' immediate;


5) Cross check session at database level.

select sid,serial#,osuser,username,program,status,SQl_id from v$session where sid=&sid;
 

Here are some basic tips to work with Execution Plan stability, using Baselines.
==========================================================

We use the below queries to see the available execution plans and see which plan was running fine.

1.
      @?/rdbms/admin/awrsqrpi.sql  ---> This will generate the html page for the required query based on the SQLID and its awr history.

or

2.      dbms_xplan.display_awr()

Ex: select * from TABLE(dbms_xplan.display_awr('47qjdv3ncanhr'));

or

3.       USING GRID Control 12c

 To gather the history of a SQL execution and the plans used during those runs, obtain the SQL Id to be evaluated, connect to the GRID Control 12c:

select the Targets/Databases -->Select the database -->Performance/SQL/Search SQL-->
Check AWR Snapshots -->Enter the SQL ID in the SQL ID filed / Search-->
Verify the executions and the different Hash Plans used.

 The ones with the Smallest Elapsed Times are the best execution Plans for the SQL.
·         If the HASH Plan is still in the Cursor Cache it can be created as a baseline and instructed to run every time that SQL ID is loaded to the Shared Pool.

·         If the HASH Plan is no longer in the Cursor Cache, then it is still possible to load the HASH Plan to a Sql Tuning Set and create a baseline from the STS and assign it the SQL ID as well. Take note of  the Snap ID (from the GRID SQL Search above) for the desired HASH Plan

HASH /SQL plan needed found in the Cursor Cache

Now you know which hash plan hash to be fixed. Now follow the below example. If the needed plan is found in the cursor cache then it is very simple to create a baseline and fixing the plan for the SQL query.

Ex: Determined the Hash Plan: 2601263939 is the best to run against the SQL ID: 47qjdv3ncanhr


1.Create the Baseline:

var v_num number;
exec :v_num:=dbms_spm.load_plans_from_cursor_cache(sql_id =>'47qjdv3ncanhr',plan_hash_value => 2601263939);

OR

Example from Internet for script:

SQL> !cat create_baseline.sql
var ret number
exec :ret := dbms_spm.load_plans_from_cursor_cache(sql_id=>'&sql_id', plan_hash_value=>&plan_hash_value);
 SQL> @create_baseline
Enter value for sql_id: 47qjdv3ncanhr
Enter value for plan_hash_value: 2601263939

2. Verify the baseline got created or not
=================================

SQL> select sql_handle, plan_name, enabled, accepted, fixed from dba_sql_plan_baselines;
SQL_HANDLE                      PLAN_NAME                                              ENA    ACC     FIX
------------------------------             ------------------------------                                     ---         ---         ---
SQL_4bd90f15ef3c1f10           SQL_PLAN_4rq8g2rrms7sh3cc6a555       YES     YES     NO


To see all the details, this will create a file with all the baseline info.:

spool baseline_plan.txt
select * from table(dbms_xplan.display_sql_plan_baseline(sql_handle=>'SQL_4bd90f15ef3c1f10',  format=>'basic'));
verify the spool file to confime the SQL ID and the HASH associated with it.

3.TO MODIFY A SQL PLAN BASELINE

var v_num number;
exec :v_num:=dbms_spm.ALTER_SQL_PLAN_BASELINE (sql_handle =>'SQL_4bd90f15ef3c1f10',plan_name => 'SQL_PLAN_4rq8g2rrms7sh3cc6a555', attribute_name=> 'FIXED',  attribute_value  => 'YES');
Attributes
    ·  enabled (YES/NO) : If YES, the plan is available for the optimizer if it is also marked as accepted.
    ·  fixed (YES/NO) : If YES, the SQL plan baseline will not evolve over time. Fixed plans are used in preference to non-fixed plans.
    ·  autopurge (YES/NO) : If YES, the SQL plan baseline is purged automatically if it is not used for a period of time.
·   plan_name : Used to amend the SQL plan name, up to a maximum of 30 character.
·    description : Used to amend the SQL plan description, up to a maximum of 30 character.
   

Sometimes the required HASH / SQL plan  will not be present in the Cursor Cache, then you have to  load it from a AWR snapshots.

 Steps are as below:
======================================

To load plans to the cursor cache from awr snapshots:

1. -- Drop SQL Tuning Set (STS)

BEGIN
  DBMS_SQLTUNE.DROP_SQLSET(
    sqlset_name => 'SAMPLE_TUNING_SET');
END;

 2. -- Create SQL Tuning Set (STS)

 BEGIN
  DBMS_SQLTUNE.CREATE_SQLSET(
    sqlset_name => 'SAMPLE_TUNING_SET',
    description => 'SQL Tuning Set for loading plan into SQL Plan Baseline');
END;

3.-- Populate STS from AWR using a time duration when the desired plan was used.

Retrieve the begin Snap ID from the same session described in the GRID Contol above or by :   

SELECT SNAP_ID, BEGIN_INTERVAL_TIME, END_INTERVAL_TIME FROM dba_hist_snapshot ORDER BY END_INTERVAL_TIME DESC;

Note: Specify the sql_id in the basic_filter (other predicates are available, see desc dba_hist_snapshot) if necessary.

DECLARE
  cur sys_refcursor;
BEGIN
  OPEN cur FOR
    SELECT VALUE(P)
    FROM TABLE(
       dbms_sqltune.select_workload_repository(begin_snap=>1477, end_snap=>1478,basic_filter=>'sql_id = ''9n82zq1gkpg2t''',attribute_list=>'ALL')
              ) p;
     DBMS_SQLTUNE.LOAD_SQLSET( sqlset_name=> 'SAMPLE_TUNING_SET', populate_cursor=>cur);
  CLOSE cur;
END;
/
4. -- List out SQL Tuning Set contents to check we got what we wanted

SELECT
  first_load_time,
  executions as execs,
  parsing_schema_name,
  elapsed_time  / 1000000 as elapsed_time_secs,
  cpu_time / 1000000 as cpu_time_secs,
  buffer_gets,
  disk_reads,
  direct_writes,
  rows_processed,
  fetches,
  optimizer_cost,
  sql_plan,
  plan_hash_value,
  sql_id,
  sql_text
   FROM TABLE(DBMS_SQLTUNE.SELECT_SQLSET(sqlset_name => 'SAMPLE_TUNING_SET'));

5.-- Finally create the baseline from the STS:

DECLARE
my_plans pls_integer;
BEGIN
  my_plans := DBMS_SPM.LOAD_PLANS_FROM_SQLSET(
    sqlset_name => 'SAMPLE_TUNING_SET',
    basic_filter=>'plan_hash_value = ''1117073691'''
    );
END;
/

6.-- Verify the baseline got created and modify it if necessary

select sql_handle, plan_name, enabled, accepted, fixed from dba_sql_plan_baselines;
exec :v_num:=dbms_spm.ALTER_SQL_PLAN_BASELINE (sql_handle =>'SQL_ab2ab5c194ee0fc8',plan_name => 'SQL_PLAN_aqapps6afw3y81722054c', attribute_name=> 'FIXED',  attribute_value  => 'YES');

7.-- Verify all details for the new Baseline:

spool baseline_plan.txt
select * from table(
    dbms_xplan.display_sql_plan_baseline(
        sql_handle=>'SQL_ab2ab5c194ee0fc8',
        format=>'basic'));


using below query to see which all plan_hash_value are being used by my sql_id...


select ss.snap_id,
ss.instance_number node,
begin_interval_time,
sql_id,
plan_hash_value,
nvl(executions_delta,0) execs,
round((elapsed_time_delta/decode(nvl(executions_delta,0),0,1,executions_delta))/1000000,2) avg_etime,
round((buffer_gets_delta/decode(nvl(buffer_gets_delta,0),0,1,executions_delta)),2) avg_lio
from DBA_HIST_SQLSTAT S,
DBA_HIST_SNAPSHOT SS
where sql_id = 'xxxxxxxx'
and ss.snap_id = S.snap_id
and ss.instance_number = S.instance_number
and executions_delta > 0
order by 1 desc, 2, 3;


HOW TO GET 11G TO USE THE 10G EXECUTION PLAN
The next challenge was to get SQL with exactly the same SQL text as the application used (exactly the same SQL_ID) and set a SQL plan baseline. I work from Australia and the client’s team is based in North America. Therefore, I couldn’t just call the developer and ask to re-execute the SQL. I had to come with a method that would allow me to find the exact SQL text without executing it. The SQL’s execution took a lot of resources Oracle captured SQL in AWR repository. (The client had a Diagnostic license.) I used the following statement to get the SQL and associated “good” execution plan in a shared pool.

alter session set optimizer_features_enable='10.2.0.4';
 declare
  v_sql varchar2(8000);
  c NUMBER;
  begin
  select sql_text into v_sql from DBA_HIST_SQLTEXT where sql_id='djkbyr8vkc64h';
  c := dbms_sql.open_cursor;
  dbms_sql.parse(c, v_sql, dbms_sql.NATIVE);
  dbms_sql.close_cursor(c);
 end;

I confirmed that the “good” execution plan had been used by the following SQL:
  select sql_id,LAST_LOAD_TIME,PLAN_HASH_VALUE, exact_matching_signature, sysdate from v$sqlarea where sql_id='djkbyr8vkc64h';
   
  SQL_ID LAST_LOAD_TIME PLAN_HASH_VALUE EXACT_MATCHING_SIGNATURE SYSDATE
  ------------- ------------------- --------------- ------------------------ -------------------
  djkbyr8vkc64h 2012.11.12 01:25:51 810205201 14465951278806438046 2012.11.12 01:26:04
The final bit was to create a SQL plan baseline based on the pair of SQL_ID and PLAN_HASH_VALUE. From there, it was too easy. :)
  declare
  n number;
  begin
  n:=DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE (
  sql_id => 'djkbyr8vkc64h',
  plan_hash_value => '810205201'
  FIXED => 'YES',
  ENABLED => 'YES');
  dbms_output.put_line(n);
 end;
 /  
 PL/SQL procedure successfully completed.
To validate that the base line was created for the SQL:
  col version for a10
  col SQL_HANDLE for a25
  col CREATED for a30
  col sb.last_executed for a30
  SELECT
  sb.sql_handle,sb.plan_name,
  sb.origin,sb.version,
 cast(sb.created as date) created,
 cast(sb.last_executed as date) last_executed,
 sb.enabled,
 sb.accepted FROM dba_sql_plan_baselines sb
 WHERE  sb.signature = 14465951278806438046;  

 SQL_HANDLE PLAN_NAME ORIGIN VERSION CREATED LAST_EXECUTED ENA ACC
 ------------------------- ------------------------------ -------------- ---------- ------------------------------ ------------------- --- ---
 SQL_c8c1620b4f5d909e SQL_PLAN_cjhb21d7pv44y27600b06 MANUAL-LOAD 11.2.0.3.0 2012.11.12 01:31:58 YES YES
Just in case we would need to rollback the change, we just need to run the following PL/SQL block:
  set output on
   DECLARE
   i NATURAL;
    BEGIN
    i := DBMS_SPM.ALTER_SQL_PLAN_BASELINE (
    sql_handle=>'SQL_c8c1620b4f5d909e',
    PLAN_NAME=> 'SQL_PLAN_cjhb21d7pv44y27600b06',
    attribute_name=>'enabled',attribute_value=>'no');
    dbms_output.put_line(i);
   END;
  /

My query is picking a bad execution plan , how to fix it to use the good execution plan available ( using Oracle baselines ) ?

Using Oracle baselines you can fix the sql plan for a SQLID:                                          
SQL plan management is a preventative mechanism that records and evaluates the execution plans of SQL statements over time. This mechanism can build a SQL plan baseline, which is a set of accepted plans for a SQL statement. The accepted plans have been proven to perform well.
The goal of SQL plan baselines is to preserve the performance of corresponding SQL statements, regardless of changes in the database. Examples of changes include:
·         New optimizer version
·         Changes to optimizer statistics and optimizer parameters
·         Changes to schema and metadata definitions
·         Changes to system settings
·         SQL profile creation
SQL plan baselines cannot help in cases where an event has caused irreversible execution plan changes, such as dropping an index.
The SQL tuning features of Oracle Database generate SQL profiles that help the optimizer to produce well-tuned plans. However, this mechanism is reactive and cannot guarantee stable performance when drastic database changes occur. SQL tuning can only resolve performance issues after they have occurred and are identified. 

For example, a SQL statement may become high-load because of a plan change, but SQL tuning cannot solve this problem until after the plan change occurs.


To know more details, follow the below link


To identify blocking session and killing it after approval

Blocking session occurs when one session acquired an exclusive lock on an object and doesn't release it,another session (one or more) want to modify the same data.First session will block the second until it completes its job.

Mostly DML statement caused this issue

From the view of the user It look like the application is completely hanged  while waiting for the first session to release its lock.

In most of case, session lock release automatically when we get mutliple email regarding session lock
DBA need to investigate and identify block session on database and inform to IM Team/Service management Team with complete SQL details .

These details will help  IM Team/Service management team to get approval shop direct or application team .Once IM Team/Service management team get approval from shop direct or application team
The will inform to DBA to kill session