Thursday 19 September 2024

Oracle DB RAC Server Reboot Process

 ORACLE RAC DATABASE SERVER REBOOT PROCESS

-----------------------------------------

Current RAC 4 node Setup

• FOR EASYDB (4-NODE RAC DATABASE on ASM) with four Node Standby Database


➢ DOWN PROCESS:


1) Check log sequence number of all four nodes and make sure it is in sync (DC & DR).


SQL> SELECT MAX(SEQUENCE#),THREAD# FROM V$LOGHIST GROUP BY THREAD#; (both DC & DR).


2) DEFER dataguard (DC side).


SQL> ALTER SYSTEM SET log_archive_dest_state_2='DEFER' SID='*';


3) Stop MRP process (DR side).


SQL> alter database recover managed standby database cancel;


4) Down all Instances.


Login with ‘oracle’ session on master node i.e., EASYNODE1:


EASYNODE1:/home/oracle#srvctl stop database –d EASYDB (both DC & DR).


5) Dismount diskgroups such as EASYDBDATA_DG1, EASYDBDATA_DG2,

EASYDBFRA_DGNEW, EASYDBREDO_DGNEW, and EASYDBTEMP_DGNEW.


Login with ‘orarac’ session and connect to ‘sqlplus / as sysasm’:


SQL> alter diskgroup EASYDBDATA_DG1 dismount;

SQL> alter diskgroup EASYDBDATA_DG2 dismount;

SQL> alter diskgroup EASYDBFRA_DGNEW dismount;

SQL> alter diskgroup EASYDBREDO_DGNEW dismount;

SQL> alter diskgroup EASYDBTEMP_DGNEW dismount;


6) Stop ASM service.


Login with ‘orarac’ session on master node i.e., EASYNODE1:


EASYNODE1:/home/orarac#srvctl stop asm


7) Stop cluster service and handover to UNIX team for reboot the nodes.


Login with ‘orarac’ session on master node i.e., EASYNODE1:


EASYNODE1:/home/orarac#srvctl stop asm

EASYNODE1:/home/orarac#crsctl stop cluster all


or 


stop cluster/crs on all node by root user after shutting down database .


cd /GRIDHOME/bin

./crsctl stop crs - run on each node or one by one as per requirement



➢ UP PROCESS:


1) After get back the servers from UNIX team, First check binary mount points


such as ‘/oracle’ & ‘/asm_grid’ from all 4 nodes.


EASYNODE1:/home/dxc_subhankar#df –gt


2) Start cluster services if it is not started automatically


Login with ‘orarac’ session:


EASYNODE1:/home/orarac#crsctl start cluster all


3) Start ASM services.


Login with ‘orarac’ session on master node i.e., EASYNODE1:

EASYNODE1:/home/orarac#srvctl start asm


4) Mount diskgroups such as EASYDBDATA_DG1, EASYDBDATA_DG2, EASYDBFRA_DGNEW,EASYDBREDO_DGNEW, and EASYDBTEMP_DGNEW.


Login with ‘orarac’ session and connect to ‘sqlplus / as sysasm’:


SQL> alter diskgroup EASYDBDATA_DG1 mount;

SQL> alter diskgroup EASYDBDATA_DG2 mount;

SQL> alter diskgroup EASYDBFRA_DGNEW mount;

SQL> alter diskgroup EASYDBREDO_DGNEW mount;

SQL> alter diskgroup EASYDBTEMP_DGNEW mount;


or 

start the clusterware if it is not started from root user

cd $ORACLE_GRID/bin

./crsctl start crs -- run it on all node or one by one based on requirement


5) UP all Instances.


Login with ‘oracle’ session on master node i.e., EASYNODE1:

EASYNODE1:/home/oracle#srvctl start database –d EASYDB (both DC & DR).


6) ENABLE dataguard (DC side).


SQL> ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SID='*';


7) Start MRP process (DR side).


SQL> alter database recover managed standby database disconnect from session;


8) Do some alter switch (DC side) and check archives are receiving (DR side).


SQL> alter system switch logfile;


9) Do tnsping to check network connectivity, if everything looks fine handover to application team. 


General Oracle DB Check And Recommendation

 General DB Check And Recommendation

------------------------------------------------


Database Configuration Analysis

------------------------------

1 Optimize Redo Logs

2 Table Partitioning for Big table


Database Performance Analysis.

--------------------------------

3 Check Stale Statistics for Tables

4 Check Fragmented Tables/ Index

5 Check Statistics for SYS tables

6 Check User objects in SYSTEM tablespace 

7 Check Un-indexed foreign keys  

select * from (

select ct.table_name, co.column_name, co.position column_position

from user_constraints ct, user_cons_columns co

where ct.constraint_name = co.constraint_name

and ct.constraint_type = 'R'

minus

select ui.table_name, uic.column_name, uic.column_position

from user_indexes ui, user_ind_columns uic

where ui.index_name = uic.index_name

)


Best Practices

-----------------

8 use Switchover Verify during switch over  

9 Check Temp file mismatch on nodes  

10 OS Watcher need to be installed on each node  

11 Backup Table Deletion from Database

12 Block Corruption Check  

13 Purge Recyclebin regularly

14      check Hardparsing for SQL


Other Recommendation

---------------------

15 check Compatibility parameter

16 Check Sequence Cache

17 DOP Removal from object

18 Check Invalid Object 

19 Check AWR Rentention

20 Check Datafiles count in each Tablespace 

21 Table space usages

22 Undo TS Mismatch on each node  

23 Puging for large table  

24 ASM Space usages  

25 Unusable Indexes

26 Invisible indexes

27 Check Default Degree for table

28      Check ORA-ERRORS in alert log

30 Check Auto extensible for datafiles

31 Check Redo Transport Destination

32 set hidden _use_single_log_writer value to  true if it is not set

33 check archive_lag_target parameter


Oracle DB Role And Reponsibility

 SL No. Most common Activity Performed by Database Team 

1 Server load monitoring and Memory Utilization.

2 File system monitoring and space management (by deleting old archive) as per database growth.

3 Connecting SQL Plus console to perform Database activity.

4 Monitoring Database through  SAM (Simple Automated Manager).

5 Current SQL Activity (To check which sql is running and finding query cost).                                

6 Tablespace Status and its Management (like tablespace creation, allocated sapce ,max extent,fee space)

7 Datafile addition based on growth of database.

8 Parallel process Status.

9 Lock - Transaction Lock check & Release.

10 Lock - DD Lock check & Release.

11 Lock - Table Lock check & Release Using Synonym.

12 checking Latches on database and release.

13 Checking Blocking session by other instance or on same instance and release it.

14 Lock -Unknown locks release.

15 Lock Clear Based on SOL_ID,

16 ABH Sessions and Finacle Branches and Sessions.

17 Resource Status ( Max process,Max sessions and other resource).

18 Events and it's details.

19 User account - Check user account status & privileges.

20 Create  and drop  Database user.

21 Lock and unlock Database User.

22 Grant and Revoke privileges to Database User (like select ,upadate,delete.insert )

23 Table analysis and including partition Table.

24 After Monitoring Database generating Health Status Report.

25 DC and DR Sync Status.

26 Checking Temporary tablespace status.

27 Checking Sequence information.

28 Monitoring Long running sql full table scans.

29 Database objects details(like Object Name,Object Type).

30 Archive gap between DC and DR database.

31 Index Information.

32 ASM Disks Status (like ASM Free Space,ASM Disks Status,ASM DBFile Info,ASM Disk Groups).

33 Release Space from ASM Disk Groups by removing OLD Archive.

34 Wait Session Monitoring.

35 Job Monitoring.

36 Generate Performance Statistics Reports.                                          

37 Table Information in Detail.

38 Clear active inactive sessions before down the database.

39 Clear inactive session after requesting by application Team.

40 Configure Job using Crontab.

41 Running  Rman backup on scheduled time and checking confirmation.

42 Complete database analysis on Sunday.

43 down the database to take cold backup after DC-DR in sync

44 index creation and drop as per CR.

45 Table creation purging and truncation.

46 Table insertion and deletion activity.

47 DC-DR drill activity.

48 TDE Encryption implementation.

49 Masking Activity.

50 Database Migration, Upgradation

51 Patching activity on ASM level and database level.

52 Sql tuning after bad sql observation.

53 Awr report ,ASH report,ADDM report.

54 Generating Global AWR report.

55 sqltrt report for problemtic  Sql_id.

56 Database log monitoring .

57 Report Database refresh on daily basis.

58 Database restoration as per request on quarterly Basis.

59 Db link creation and its management ( like Tns entries ,user creation based on requirement).

60 check listener status.

61 raising oracle support for bug fix and other performance issue.

62 Filesystem Utilization monitoring and audit / listener log purging proactively

63 checking SQL plan for specific SQL if plan changes , need to set profile.

------------------

S.No DAILY ACTIVITY Resource Type

-----------------------------------------------------------------------------

1 Oracle Database instance is running or not L1/L2

2 Database Listener is running or not. L1/L2

3 Check any session blocking the other session L1/L2

4 Check the alert log for an error L1/L2

5 Monitor DBA jobs / DBA Scheduler Jobs running status and take proactive/reactive action.L1/L2

6 Check the Top session using more Physical I/O L1/L2

7 Check the number of log switch per hour L1/L2

8 How_much_redo_generated_per_hour.sql L1/L2

9 Tablespace utilization L1/L2

10 Detect lock objects L1/L2

11 Check the SQL query consuming lot of resources. L2/L3

12 Check the usage of SGA L1/L2

13 Display database sessions using rollback segments L1/L2

14 State of all the DB Block Buffer L2/L3

15 DataGuard Sync Status L1/L2

16 Temp Tablespace Size and utilization L1/L2

17 Long running session L1/L2

18 Database status & DB Role (Mount, Open, Primary /Standby) L1/L2

19 FRA utilization L1/L2

20 Wait event on databases, is it available? L1/L2

21 Ensure the flash backup and RMAN backup status [ In case of any other 

backup strategies respective monitoring should be in place] L1/L2

22 Materialized View Refresh / Refresh Group / Job status monitoring and proactive action.L1/L2

23 Filesystem Utilization monitoring and audit / listener log purging proactively L1/L2


WEEKLY ACTIVITY Resource Type

-----------------------------------------------------------------------

1 Check the objects fragmented       L2

2 Check the Chaining & Migrated Rows L2

3 Check the tables having more Indexes and take consultation as and when require with Next Level.L2

4 Check for unindexed FK L2

5 Check the objects having the more extents and take consultation as and when require with L3.L2

6 Check the free space at O/s Level          L1/L2

7 Check the CPU, Memory usage at O/s level to monitor the threshold and proactively act on it. L1/L2

8 Check for UNUSABLE INDEXES & Invisible indexes L2

9 Check the objects reaching to it’s Max extents              L2

10 Generating Statistics L1


MONTHLY ACTIVITY Resource Type

-------------------------------------------------------------------------------------------------

1 Find Tablespace Status, segment management, initial & Max Extents and Extent Management L2

2 Check default tablespace & temporary tablespace of each user L1/L2

3 Check for fragmentation of Indexes L2

4 Check Patch status of database & bug fixation L2

5 Check for Block corruption periodically L2


General Checks L1/L2

----------------------

➢ Check for Backup Tables in the database and send report to bank 

➢ Check for Compatible Parameter 

➢ Check for Purge Recycle bin

➢ Check for Invalid Objects

➢ Check for User objects in SYSTEM tablespace

➢ Check for Archive Generation 

➢ Check for db_files max number

➢ Check for Max db_files per tablespace

➢ Check for Database Registry Components 

➢ Check for UNDO Tablespace Size Mismatch 

➢ Check for ASM Free Space 

➢ Check for ORA-ERRORS in alert logs 

➢ Auto extensible datafiles



Convert RAC Physical Standby To Snapshot Standby in RAC enviroment

 

 

Convert RAC Physical Standby To Snapshot Standby in RAC enviroment

 Current : Standby Database with 4 node RAC enviroment

Step1 :-  Pre-requiste   

  •  FRA needs to be configured.
  •  Ensure the  ORL  path exist  (Online redo logs are created)

Show parameter db_recovery_files_dest

Show parameter db_recovery_file_dest_size

 

To Check if ORL (Online redo log path exist)

 

SELECT thread#, group#, sequence#, bytes, archived ,l.status, blocksize, member FROM v$log l join v$logfile f using (GROUP#) ORDER BY thread#, group#;

Step 2 :- Once FRA is configured and redo log created on Standby

Stop the db using srvctl command on standby

 srvctl stop database -d ORCLDR -o immediate

Startup db in mount stage

srvctl start database -d ORCLDR -o mount

srvctl status database -d ORCLDR

Confirm its mounted.                                                   

Step 3 : Connect to sqlplus and enable flashback

Creation of guarantee restore point instead of normal restore point

alter system set db_flashback_retention_target=1440;

create restore point before_snapshot_19c guarantee flashback database;

select scn, storage_size, time, name from v$restore_point;

or

Connect /as sysdba

Select status from v$instance;

Ensure it shows Mounted

alter database flashback on;

Alter system set db_flashback_retention_target=<value> scope=both sid='*' ;

Select flashback_on from v$database ;

Select * from v$flash_recovery_area_size ;

Select * from v$restore_point ;

 

 

Step 4 :- Shutdown the db and startup in mount stage

srvctl stop database -d ORCLDR -o immediate

srvctl status database -d ORCLDR

 

srvctl start database -d ORCLDR -o mount

srvctl status database -d ORCLDR

Ensure the instance is mounted

Step 5 :- Connect to sqlplus and convert to Snapshot standby

Connect /as sysdba

Select status from v$instance ; ===> Should show Mounted


Alter database convert to Snapshot standby ;

Alter database open ;

Once done

Select database_role,controlfile_type from v$database ;

 

Step 6 :- Restart the entire db if required

srvctl stop database -d ORCLDR -o immediate

srvctl status database -d ORCLDR

Start the db (Both instance using srvctl)

srvctl start database -d ORCLDR -o open

 

 

------------------Revert Back---------------------------

##once app testing is completed and confirmed by apps team

##shutdown DR , restart it in mount stage and convert to physical standby

                      

srvctl stop database -d ORCLDR -o immediate

srvctl start database -d ORCLDR -o mount

 

alter database convert to physical standby;

 

srvctl stop database -d ORCLDR -o immediate

srvctl start database -d ORCLDR 

 

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION.

 

alter database flashback off;

or

drop RESTORE POINT before_snapshot_19c;

 

Monitor DC -DR sync progress.

 

GRID and Database patch on RAC enviroment with real time

 

 GRID and Database patch on RAC enviroment with real time

ORACLE GRID AND DATABASE PATCHING

 

Dependency:

 

1)      sudo su – root permission is  required

2)      Sometime Server reboot required during  clusterware/Grid patching therefore UNIX/AIX team support is required

3)      Minimum 40 GB free space is required on GRID AND ORACLE HOME

 

 

Patch Information And Enviroment Details

 

Current Environment

-----------------------

Primary Database with 4 Node RAC

Standby database with 4 Node RAC

Current Grid And DB Patch Version = 19.17.0.0.221018 

 

Note:  Grid and DB Patch will be performed on Standby database (DR side) first. it requires approx. 12 hours maintenance window on 4 node RAC no downtime required. Once Patch is completed on Standby database, DB Team will switchover and perform patching Grid and DB on Current standby Database. 

 

We are going to apply July 2023 patch on 4 node environments.

 

Patch Information

 

Patch 35319490 - GI Release Update 19.20.0.0.230718

 

To apply patch Patch 35319490 - GI Release Update 19.20.0.0.230718

 on Grid Home and Oracle Home , Grid Home And Oracle Home must have Oracle Database release 19c installed.

 

          Patch will apply in two phase in Rolling Fashion(one by one) on Standby Database

 

·       On Grid Home

·       On Database Home And database


Pre-requisites

 

1)      Check clusterware and database component status. 

2) Check opatch version and recommended to have latest opatch and download it from oracle support (6880880) and unzip it in Grid and Database home after renaming Old Opatch folder

It is recommended to have OPatch utility version 12.2.0.1.37 or later to apply this patch.

 

$ORACLE_HOME/OPatch/opatch version

$ORACLE_HOME/OPatch/opatch lsinventory

$ORACLE_HOME/OPatch/opatch lspatches


3) Check Free Space in Cluster Home And Database Home(approx free space 40 GB )

  

            Run OPatch System Space Check

----------------------------------------

For Grid Infrastructure home, as home user:

 

Create file /tmp/patch_list_gihome.txt with the following content:

cat /tmp/patch_list_gihome.txt

/asm_grid/19cpatch/35319490/35320081

/asm_grid/19cpatch/35319490/35320149

/asm_grid/19cpatch/35319490/35332537

/asm_grid/19cpatch/35319490/35553096

/asm_grid/19cpatch/35319490/33575402

 

Run the OPatch command to check if enough free space is available in the Grid Infrastructure home:

 

$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt

 

For Oracle home, as home user:

Create file /tmp/patch_list_dbhome.txt with the following content:

cat /tmp/patch_list_dbhome.txt

/asm_grid/19cpatch/35319490/35320081

/asm_grid/19cpatch/35319490/35320149

 

Run OPatch command to check if enough free space is available in the Oracle home:

 

 $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt

 

4) run opatch lsinventory to check consistency and corruption of inventory on Grid and Oracle Home

 

Eg

 

Validation of Grid Inventory

 

<GRID_HOME>/OPatch/opatch lsinventory -detail -oh <GRID_HOME>

 

Validation of Oracle Inventory

 

<ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

 

Run OPatch Conflict Check for Grid and Oracle Home

-------------------------------------------------

 

As the Grid home user:

 

/asm_grid/grid/app/19.3/gridhome1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /asm_grid/19cpatch/35319490/35320081

/asm_grid/grid/app/19.3/gridhome1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /asm_grid/19cpatch/35319490/35320149

/asm_grid/grid/app/19.3/gridhome1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /asm_grid/19cpatch/35319490/35332537

/asm_grid/grid/app/19.3/gridhome1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /asm_grid/19cpatch/35319490/35553096

/asm_grid/grid/app/19.3/gridhome1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /asm_grid/19cpatch/35319490/33575402

 

For Oracle home, as home user:

 

/oracle/app/product/19.0.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /asm_grid/19cpatch/35319490/35320081

/oracle/app/product/19.0.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /asm_grid/19cpatch/35319490/35320149

 

5) run analyze command using opatchauto from root user after setting proper path for grid home And Oracle Home

 

 Pre Check at database Level

 

1)   PATCH confilct check ON Oracle Home

 

2)      Oracle suggested to run below command as par of pre-check in advance .It will help during troubleshooting as well

 

select action,action_time,patch_id,status,description from dba_registry_sqlpatch order by 2;

/

select * from dba_registry_sqlpatch;

 

$ORACLE_HOME/OPatch/opatch version            

$ORACLE_HOME/OPatch/opatch lsinventory

$ORACLE_HOME/OPatch/opatch lspatches

 

Patch Apply on Grid Home

  

PATCH confilct check ON Grid Home

 

Implementation Step (maintenance window Start)

 

1) Take backup of the Grid And Oracle_Home before applying patches

            

       From Root user

     

         Stop crs and database instance on one node

eg

Cd /asm_grid/grid/app/19.3/gridhome1

tar -cvf /oracle/software/grid_home.tar .

du -hP /oracle/softwares/grid_home.tar

 

cd /asm_grid/grid/oraInventory

 

tar -cvf /oracle/software/grid_home_inventory.tar .

du -hP /oracle/softwares/grid_home_inventory.tar

         

          Oracle home backup

     

              cd  /oracle/app/product/19.0.0/dbhome_1

               tar -cvf /oracle/app/product/19.0.0/dbhome_1

 

    df -g /oracle/softwares/oracle home _home.tar


and start crs after grid and database home backup (don’t  start database instance)

 

1            applying Patch On GRID


Run analyze and apply patch

 

/asm_grid/grid/app/19.3/gridhome1/OPatch/opatchauto apply /asm_grid/ASM_PATCH/35319490 -analyze -oh /asm_grid/grid/app/19.3/gridhome1

 

nohup /asm_grid/grid/app/19.3/gridhome1/OPatch/opatchauto apply /asm_grid/19cpatch/35319490 -oh /asm_grid/grid/app/19.3/gridhome1 &

 

 

3    Post Implementation And Validation

 

check status of crs

./crsctl check crs

ps -ef|grep d.bin

 $GRID_HOME/OPatch/opatch lspatches

              $GRID_HOME/OPatch/opatch lsinventory 

 

Patch Apply on Oracle Home And Database


1            PATCH confilct check ON Oracle Home

 

From root user

 

/asm_grid/grid/app/19.3/gridhome1/OPatch/opatchauto apply /asm_grid/ASM_PATCH/35319490 -analyze -oh /asm_grid/grid/app/19.3/gridhome1

 

2            applying Patch On GRID

From root user

 

/oracle/app/product/19.0.0/dbhome_1/OPatch/opatchauto apply /asm_grid/19cpatch/35319490   -oh /oracle/app/product/19.0.0/dbhome_1


Post Implementation And Validation

 

1) Validate PSU using Opatch Lsinventory from oracle user 

 Ps -ef|grep pmon

 Ps -ef|grep tns

 

 $ORACLE_HOME/OPatch/opatch lspatches

 $ORACLE/OPatch/opatch lsinventory

 

set feedback on

set lines 200 pages 200

col owner for a10

col object_name for a30

col oracle_maintained for a5

SELECT owner,object_name,object_type,oracle_maintained FROM dba_objects WHERE status='INVALID'  and owner in ('SYS','SYSTEM')

ORDER BY owner,object_type;

set feedback off

 

select owner, object_name, object_type,status from dba_objects where status like 'INVALID'

/

select count(1) from dba_objects where status like 'INVALID'

/

set lines 200 pages 200

col action_time format a40

col action format a30

col namespace format a10

col comments format a30

col version format a13

set lin 200 head on feed on pagesize 100

/


select i.instance_name,r.* from v$instance i, registry$history r

/

set lines 200 pages 200

col action_time format a40

col DESCRIPTION format a60

column comp_name format a50

 

select comp_id, comp_name,version, status from dba_registry;

select PATCH_ID,ACTION_TIME,STATUS,ACTION,DESCRIPTION from dba_registry_sqlpatch

select action,action_time,patch_id,status,description from dba_registry_sqlpatch order by 2;

/ 

select * from dba_registry_sqlpatch;