ExaData Patching Overview
Patches are provided as complete software images that contain updates for the Linux operating system , Cell Server Software (Storage) , InfiniBand software and other component firmware.
- In ExaData, Patching can be done in rolling and Non-rolling Fashion
- Oracle release patches quarterly .
Exadata has three main component/layer that require Software patching
Storage Server –It contain upto 14 storage server. patchmgr utility can be used for patching Storage cells in a rolling or non-rolling fashion. patchmgr utilizes dcli to push patch software to storage cells. Each storage server can take up to approx two hours. Storage server patches apply operating system, firmware, and driver updates. Current version can checked by imageinfo .Same patch applies to all hardware. Exadata software, OS,IB, ILOM, firmware and New Features.
InfiniBand Switch - All Exadata Machine contain two IB Switch. Patchmgr is used to patch it. It can be access by commands line or ILOM. Only update via Exadata-branded patch
# version
SUN DCS 36p version: 1.3.3-2
Database Server – It contain up 8 database servers. Database server patching has two phases
a) Firmware/OS -patchmgr used to patch Firmware/OS in all of Exdata Database se server . Current version can checked by imageinfo
b) GRID and ORACLE HOME -Opatch utility is used to patch it.
Current version can checked as below
Opatch lspatches
Infiniband switches. Its software comes with Exadata cell storage patch software. So you don’t need to download separate software for IB switches.
Additional components, Ethernet(Cisco) Switch,PDU (Power Distribution Unit),KVM. You generally not need to upgrade these components unless you’ve any issue.
The patching procedure
1) Patching the cells ( Storage servers)
2) Patching the IB switches
3) Patching the Database servers (aka Compute Nodes)
4) Patching the Grid Infrastructure
5) Patching the databases ORACLE_HOMEs
The Rollback procedure
1) Cell Rollback
2)DB nodes Rollback
3) IB Switches Rollback
Troubleshooting
1) Cell patching issue
2) CRS does not restart issue
3) A procedure to add instances to a database
4) OPatch resume
To Apply patch
.
1) MOS Notes
Read the following MOS notes carefully.
Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)Exadata 18.1.12.0.0 release and patch (29194095) (Doc ID 2492012.1)
Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr (Doc ID 1553103.1)
2. We need to download the bundle patch as per your setup requirement, in our case it's QFSDP 19625719
Patch 29181093 – Database server bare metal / domU ULN exadata_dbserver_18.1.12.0.0_x86_64_base OL6 channel ISO image (18.1.12.0.0.190111)
Download dbserver.patch.zip as p21634633_12*_Linux-x86-64.zip, which contains dbnodeupdate.zip and patchmgr for dbnodeupdate orchestration via patch 21634633
QFSDP releases contain the latest software for the following components:
1) Infrastructure
- Exadata Storage Server
- InfiniBand Switch
- Power Distribution Unit
2) Database
- Oracle Database and Grid Infrastructure PSU
- Oracle JavaVM PSU (as of Oct 2014)
- OPatch
- OPlan
3) Systems Management
- EM Agent
- EM OMS
- EM Plug-ins
4) Move bundle patch to one of the compute node and extract it to compute node.
unzip <filename>
5) Now extract all the tar files using tar utility
Command: #cat *.tar.* | tar -xvf -
It will make single directory with patch number called 19625719
It contained patches for following stacks
- Database/Clusterware
- Database Server
- Storage Server
- Infiniband
- PDUs
Cell Patching Pre-Check and Activity
To be taken care before the activity
1) Run File system backup and database full backup (SA + DBA)
2) Take backup of iso file /opt/oracle.SupportTools/diagnostics.iso (SA)
3) Verify hardware failure. Make sure there are no hardware failures before patching
dcli -g ~/dbs_group -l root ‘dbmcli -e list physicaldisk where status!=normal’
3) Release space from /root and /boot filesystems (SA)
4)Run Exacheck and review for any actions to be taken. Run sundiag & ILOM snapshot also (SA)
5) Create proactive SR for the patching and upload Exachk and Sundiag reports (DBA)
6)Install and configure VNC Server on Exadata compute node 1. It is recommended to use VNC or screen utility for patching to avoid disconnections due network issues.
7) Clear or acknowledge alerts on db and cell nodes
[root@dm01db01 ~]# dcli -l root -g ~/dbs_group “dbmcli -e drop alerthistory all"
Cell Patching (Implmentation)
Please note: All commands are fired as root
1)Verify network configuration
Run below in VNC/Reflection Session
On First DB node : <hostname>
dcli -l root -g cell_group '/opt/oracle.cellos/ipconf -verify'
2)Verify ssh access to cells
On first DB node - <hostname>
dcli -g cell_group -l root 'hostname -i'
3) Shutdown services on cells
On first DB node : <hostname>
dcli -g cell_group -l root "cellcli -e alter cell shutdown services all"
4) Unzip the patchfile
From First DB node,
cd <patch path>
p29624194_181000_Linux-x86-64.zip ( Already completed. However Validate once before proceeding).
After Unzip, list the files : ls -ltrh
5) Perform reset force and cleanup using patchmanager
cd <patch path>
./patchmgr -cells /root/cell_group -reset_force
./patchmgr -cells /root/cell_group -cleanup
6) Check Prerequisites
From first DB node : <hostname>
cd <patch path>
./patchmgr -cells /root/cell_group -patch_check_prereq
./patchmgr -cells /root/cell_group -cleanupcell,db alert history.
7)Patching the cells
From first DB node, go to the patch path as below
cd <patch path>
For NON -Rolling patch
./patchmgr -cells /root/cell_group -patch
For Rolling patch
./patchmgr -cells /root/cell_group -patch [- Rolling]
./patchmgr -cells <cell_group> -patch_check_prereq
./patchmgr -cells <cell_group> -patch [- Rolling]
Note
"cd /dbamaint/QFSDP_2019_29626115/29626115/Infrastructure/18.1.15.0.0/ExadataStorageServer_InfiniBandSwitch"
9) Perform cleanup using patchmanager once the patch completed successfully
./patchmgr -cells /root/cell_group -cleanup
Check once all cell services are running or not.
10) From first database node ( hostname) as root user,
dcli -l root -g /root/cell_group "service celld status"
11) Verify imageinfo , should show success and the
dcli -l root -g ~/cell_group "/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aAll | grep 'FW Package Build'"
dcli -l root -g ~/cell_group "cat /etc/enterprise-release"
dcli -l root -g ~/cell_group "cat /etc/redhat-release"
dcli -l root -g ~/cell_group "/opt/oracle.cellos/CheckHWnFWProfile -c strict"
12dcli -l root -g ~/cell_group 'rpm -q ofa-`uname -r` '
dcli -l root -g ~/cell_group "cat /etc/enterprise-release"
dcli -l root -g ~/cell_group "/opt/oracle.cellos/CheckHWnFWProfile -c strict"
Reference Check the most recent set of validation logs for any failures. Check the /var/log/cellos/validations.log
and /var/log/cellos/vldrun*.log files for any failures. The image status is marked as failure when there is a failure
reported in one or more places in these validations. Use the following command to check for failures:
grep -i 'fail' /var/log/cellos/validations.log
If a specific validation failed, then the log will indicate where to look for the logs for that validation. Examine the specific
log /var/log/cellos/validations/validation_name.log to determine the problem, and check against know issues. If
the issue is not listed as a known issue in the My Oracle Support note for the patch, then contact Oracle Support Services for assistance.
IB Switch Upgrade
Please note: All commands are fired as root and should be run in VNC/Reflection
1) Ensure switch firmware must be atleast release 1.3.3-2
2) By default, the patchmgr utility upgrades all the switches. To patch a subset of
switches create a file that lists one switch per line. The following is an example of the
file: please create a file
/root/ibswitches.lst
Not required as the current version is same as of July Qfsdp
[root@<ibswicthes~]# version
SUN DCS 36p version: 2.2.7-1
3) Change to the patchmgr directory
4) Run the pre requisites check for the upgrade
./patchmgr -ibswitches /root/ibswitches.lst -upgrade -ibswitch_precheck -force
If the output shows OVERALL SUCCESS then proceed with upgrade
5) Upgrade the IB switches
./patchmgr -ibswitches/root/ibswitches.lst -upgrade
6) Verify the output of the switches and check the version.
[root@dm01dbadm01 ~]# ssh dm01sw-ib1 version
Patching on Database Machine
Summary
- The patchmgr or dbnodeupdate.sh utility can be used for upgrading, rollback and backup Exadata Compute nodes. patchmgr utility can be used for upgrading Compute nodes in a rolling or non-rolling fashion. Compute nodes patches apply operating system, firmware, and driver updates.
- Launch patchmgr from the compute node that is node 1 that has user equivalence setup to all the Compute nodes. Patch all the compute nodes except node 1 and later patch node 1 alone.
- dbnodeupdate.sh is the shell script that patches every database server individually
- patchmgr is the orchestration tool that starts dbnodeupdate.sh in parallel across many database servers (those in the dbs_group configuration file)
- Before patchmgr, we were using dbnodeupdate.sh manually
- dbnodeupdate.sh has a -M option to remove RPMs to resolve dependencies issues
- patchmgr has the -modify_at_prereq option to remove RPMs to resolve dependencies issues We always use patchmgr, you will see some error messages related to the -M option for dbnodeupdate.sh as it is the real script patching the servers
- To sum up, when you start a ./patchmgr -dbnodes ~/dbs_group command, patchmgr will start the dbnodeupdate.sh script on each node contained in the ~/dbs_group file with the proper option
- then a ./patchmgr -dbnodes ~/dbs_group -modify_at_prereq command will launch many dbnodeupdate.sh -M on each server specified in the ~/dbs_group file
Exadata Database Machine Patching Pre-Check and Activity
- Run exachk for health check on Oracle Exadata Database Machine. more information HealthCheck (Doc ID 1070954.1)
- Verify disk space on Database Machine
- Take of backup Grid Home and Oracle Home and Databases
- Check SSH connectivity among Database Machine
No comments:
Post a Comment