Friday, 20 February 2026

OS upgrade on OCI Oracle DB system in Base Database service using OCI console


OS upgrade on OCI Oracle DB system in Base Database service using OCI console

 

Pre check


1) Before applying, run a "Precheck" to ensure prerequisites are met and ensure that no 3rd party modules/softwares are no installed

2) Back up the database in the DB system prior to attempting an OS update.

3) Do not remove packages from a DB system. However, you might have to remove custom RPMs (packages that were installed after the system was provisioned) for the update to complete successfully.

4) Oracle recommends that you test any updates thoroughly on a non-production system before updating a production system.

5) The image used to launch a DB system is updated regularly with the necessary updates. After you launch a DB system, you are responsible for applying the required OS security updates published through the Oracle public YUM server.

5) To apply OS updates, the virtual cloud network (VCN) in the DB system must be configured to allow access to the YUM repository. For more information, see VCN and Subnets.

Steps

  1. On the DB Systems list page, select the DB system that you want to work with. If you need help finding the list page or the DB system, see List the DB Systems.
  2. On the details page, select the Updates (OS) tab to view the list of available OS upgrades for the DB system.
  3. From the Actions menu for the upgrade you are interested in, select one of the following actions:
    • View details: View the details about this upgrade.
    • Precheck: Check for any prerequisites to ensure that the upgrade can be successfully applied.
    • Apply update: Applies the selected upgrade. Oracle recommends that you run the precheck operation for an upgrade before applying it.


Please refer below doc


https://docs.oracle.com/en/cloud/paas/base-database/update-dbcli/#articletitle

https://docs.oracle.com/en/cloud/paas/base-database/upgrade-dbs/#GUID-5B5D2ED6-5865-489E-BA20-A9E492C7CA12
 

Thursday, 19 February 2026

how to fix opc user expiry issue in base database service for oracle@Azure

Environment: It is oracle@Azure and base database service provisioned 


if OPC user password has been expired and you need to follow below step to fix this issue.


In Step 1, I'm sharing Oracle Doc to login into your instance in rescue mode. Once you login into rescue mode. Then need to execute other command to fix password expiration issue in Step 2 and Step 3.


Step 1

=====

Video Form

=======

1) OCI Compute - How To Reset Forgotten Root Password Using Serial Console For Oracle Linux 7, 8, and 9 [Video] KB101078


URL : fa-etmi-saasfaprod1.fa.ocs.oraclecloud.com/fscmUI/redwood/myknowledge/content/container/main/article?answerId=2940603


Oracle Document

============

OCI: How To Reset Forgotten Root Password Using Serial Console For Oracle Linux 6, 7, 8, and 9 Instances (Doc ID 2489923.1)

URL - mosemp.us.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=196232300826152&id=2489923.1&_afrWindowMode=0&_adf.ctrl-state=uu8heermb_841


Above video is for password reset. Though, you don't need to do password reset. Just eecute below command in Step 2 to fix opc user expiration issue.


Step 2

=====

Once you're login into rescue mode then execute below commands.

# chage -l opc (Check the status of password expiration.)


If it's expired then set it to never

# chage -m 0 -M 99999 -I -1 -E -1 opc

# chage -l opc


Output should like below output once it set to never expire password.

Last password change : 

Password expires : never

Password inactive : never

Account expires : never

Minimum number of days between password change : 0

Maximum number of days between password change : 99999

Number of days of warning before password expires : 7


Step 3

====

If it's OL 8 then run below command as mentioned in Oracle doc step 1. Check if SELinux is enabled then run below SELinux command.


Force an SELinux relable.

bash-4.4# touch /.autorelabel

bash-4.4#

When ready, resume normal bootup.


bash-4.4# sync

bash-4.4#

bash-4.4# sync

bash-4.4#

bash-4.4# exec /usr/sbin/init


A> Reboot the broken node from the OCI console (Not Stop And Start)

B> The serial console will show you the boot process.

C> Interrupt the boot process when prompted "Press any key"

D> Enter the Grub menu, 'e' to edit , select the like starting with "kernel", 'e' to edit , add "init=/bin/bash" at the end of the line:

++ Scroll down to the last line, which starts with initrdefi.

++ Press the left arrow key to get to the end of the long, wrapped line that starts with linuxefi.

++ Press the space bar then add init=/bin/bash to the end of the line

++ Press Crtl-x to start the instance.

** Important **

For troubleshooting ssh connectivity issues, if grub asks for password, below values may be provided:

Username: root

Password: grub######


eg

we could successfully change the password of OPC User as below

you should have correct root Password for editing the boot menu.

we append init=bin/bash

it is importatant to remount,rw / then it is possible to set a new password for opc and logon 

[root@vmlhmdbc002 /]# cp /etc/shadow /etc/shadow.0

cp: cannot create regular file '/etc/shadow.0': Read-only file system

[root@vmlhmdbc002 /]# mount -o remount,rw /

[root@vmlhmdbc002 /]# cp /etc/shadow /etc/shadow.0

 

[root@vmlhmdbc002 /]# passwd opc

Changing password for user opc.

New password: 

Retype new password: 

passwd: all authentication tokens updated successfully.

 

[root@vmlhmdbc002 /]# touch /.autorelabel

[root@vmlhmdbc002 /]# sync

[root@vmlhmdbc002 /]# sync

[root@vmlhmdbc002 /]# exec /usr/sbin/init


Test:

ssh -i key_orabasec001.pem opc@vmleasyc002.ocidefault.ocieasyadbba.oraclevcn.com

[opc@vmleasyc002 ~]$ whoami

opc

opc@vmleasyc002 ~]$  chage -l opc

Last password change                                    : Feb 18, 2026

Password expires                                        : never

Password inactive                                       : never

Account expires                                         : never

Minimum number of days between password change          : 0

Maximum number of days between password change          : 99999

Number of days of warning before password expires       : 7

 

Sharing couple of Oracle Docs. Please follow below Oracle docs to attach boot volume to another instance and fix OPC password issue.

Note : Take a backup of impacted node prior to do any activity for safer side.

1) How to Reset the Password on OCI Oracle Linux Instances?

KB114663

URL: fa-etmi-saasfaprod1.fa.ocs.oraclecloud.com/fscmUI/redwood/myknowledge/content/container/main/article?answerId=2408898

2) OPC password expired in OCICKB31429

URL : fa-etmi-saasfaprod1.fa.ocs.oraclecloud.com/fscmUI/redwood/myknowledge/content/container/main/article?answerId=2510982


Wednesday, 18 February 2026

How to check/apply patches in OCI DB System (Base Database/DB System)


 You cannot apply an RU until it is published for your database in the OCI Console. Manual patching is not supported and recommended in Base Database Service

Oracle releases RUs quarterly. For Oracle Base Database (OCI managed), there is a standard qualification lag before an RU appears in the Console; you cannot manually apply it before Oracle publishes it for your db system

Best practices to apply patch to minimize downtime

On production Environment

1) apply patch on GI on standby database if applicable 

2) perform switchover and apply patch on GI on new standby

3) apply patch on db home on standby database

4) perform switchover and apply patch on db home on new standby

5) complete validation


- How to check/apply patches in OCI (Base Database/DB System)

- Use the OCI Console:

- For DB Systems (Grid Infrastructure / Database): docs.oracle.com/en/cloud/paas/base-database/update-dbs/

- For Database homes/databases: docs.oracle.com/en/cloud/paas/base-database/update-db/

- In the Console, navigate to your database or DB system, open the Patching/Updates page,

   review the Available updates, and schedule/apply the RU when the desired version is listed.


Precautions for any patch activity

------------------------------------------------------ 

Patch first in non-production, verify backups/restore, schedule a maintenance window, and review the patch README. Ensure datapatch completes successfully and validate with DBA_REGISTRY_SQLPATCH.

It's strongly recommended to test it non-production environment before implementing on the production database This helps minimize the risk of disruptions, system crashes, and security issues that can arise from untested patches. 


Patch validation

From root user

/opt/oracle/dcs/bin/dbcli describe-component

/opt/oracle/dcs/bin/dbcli   describe-latestpatch

sudo su - grid

$ORACLE_HOME/OPatch/opatch lspatches

$ORACLE_HOME/bin/kfod op=patchlvl

sudo su - oracle

$ORACLE_HOME/OPatch/opatch lspatches

$ORACLE_HOME/bin/kfod op=patchlvl


Output of the following

       - for the CDB:

sqlplus / as sysdba

set markup html on

spool /tmp/CDB.html

show con_name;

show pdbs;

select dbid,name,db_unique_name,open_mode,database_role,switchover_status from v$database;

select comp_id, comp_name, version, status, schema from dba_registry order by status;

select * from dba_registry_sqlpatch order by action_time;

select * from dba_objects where status='INVALID';

select * from v$option order by parameter;

select * from pdb_plug_in_violations;

select message from pdb_plug_in_violations where type like '%ERR%' and status <> 'RESOLVED';

col action_time for a28

col action for a10

col version for a8

col comments for a30

col status for a10

set line 999 pages 999

select patch_id, patch_type,source_version, target_version, status, Action,Action_time from dba_registry_sqlpatch order by action_time;

 

set pages 999

set pagesize 999

 select i.instance_name,

       i.version "Inst Version",

       to_char(h.action_time, 'DD-MON-YYYY HH24:MI:SS') as when,

       h.action,

       h.namespace,

       h.version "Patch Version",

       h.id,

       h.comments,

       h.bundle_series

from   sys.registry$history h,

       v$instance i

order  by h.action_time; 

       spool off

       

       - for the PDB:

       set markup html on

       spool /tmp/PDB.html

       show con_name;

       select comp_id, comp_name, version, status, schema from dba_registry order by status;

       select * from dba_registry_sqlpatch order by action_time;

       select * from dba_objects where status='INVALID';

       select * from v$option;

       spool off




Thursday, 12 February 2026

password has been expired and need to fix this issue in OCI db system for Base Database Service

if  password has been expired and need to fix this issue.

In Step 1, Please login into your instance in rescue mode. Once you login into rescue mode. Then need to execute other command to fix password expiration issue in Step 2 ans Step 3.

Step 1

=====

Video Form

=======

1) OCI Compute - How To Reset Forgotten Root Password Using Serial Console For Oracle Linux 7, 8, and 9 [Video] KB101078

URL : fa-etmi-saasfaprod1.fa.ocs.oraclecloud.com/fscmUI/redwood/myknowledge/content/container/main/article?answerId=2940603


Oracle Document

============

OCI: How To Reset Forgotten Root Password Using Serial Console For Oracle Linux 6, 7, 8, and 9 Instances (Doc ID 2489923.1)

URL - https://mosemp.us.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=196232300826152&id=2489923.1&_afrWindowMode=0&_adf.ctrl-state=uu8heermb_841


Above video is for password reset. Though, you don't need to do password reset. Just execute below command in Step 2 to fix opc user expiration issue.

Step 2

=====

Once you're login into rescue mode then execute below commands.

# chage -l opc (Check the status of password expiration.)

If it's expired then set it to never

# chage -m 0 -M 99999 -I -1 -E -1 opc

# chage -l opc

Output should like below output once it set to never expire password.

Last password change : 

Password expires : never

Password inactive : never

Account expires : never

Minimum number of days between password change : 0

Maximum number of days between password change : 99999

Number of days of warning before password expires : 7


Step 3

====

If it's OL 8 then run below command as mentioned in Oracle doc step 1. Check if SELinux is enabled then run below SELinux command.

Force an SELinux relable.

bash-4.4# touch /.autorelabel

bash-4.4#

When ready, resume normal bootup.

bash-4.4# sync

bash-4.4#

bash-4.4# sync

bash-4.4#

bash-4.4# exec /usr/sbin/init


Friday, 6 February 2026

bug in DB RU 19.29 and 19.30 (January 2026)


Bug in DB RU 19.29 and 19.30 (January 2026)

Users is experiencing difficulties in downloading the Database Release Update 19.30.0.0.260120 (Patch 38632161) shortly after release.it is known issue and will be available shortly

 Oracle has temporarily withdrawn to download DB RU 19.29 and 19.30 (January 2026) from website

 both 19.29 and 19.30 are currently impacted by a critical bug (Bug 34352668), specifically affecting RAC (Real Application Cluster) environments. due to this bug ,database block and redo log corruption noticed after rolling updates to 19.29 (e.g., RU 19.28 to 19.29) on RAC databases. 

So it is encouraged to apply 19.30 if possible when it is available for download .

New 19.30 DB RU with the proper fix will be released shortly

It is also advised to postpone direct updates to 19.29 till new DB RU release 19.30 published

As a mitigation, set the hidden parameter “_gcs_recoverable_asserts” on any one RAC instance. 

If Oracle Data Guard is in use, apply this setting on both the primary and standby databases.


Recommendation: Oracle has temporarily paused the release of 19.30 and advises against applying 19.29 or 19.30 in RAC environments until a new patched version is available.

This issue does not affect the following:

Single instance RAC databases

Non-RAC databases

Clusterware or ASM instances

To check for lock state consistency across all instances

Please run below command

SQL> select sum(value) from gv$ges_statistics where name like 'number of triggered LSRs%';

Please verify fixes are in-place 

opatch lsinventory -bugs_fixed | egrep '^(34352668|38854064|38884142)'

To mitigate the issue, run the following SQL on any one RAC instance. 

If you use Oracle Data Guard, run these on both the primary and the standby databases.

Note : Downtime is not required 

SQL> ALTER SYSTEM SET "_gcs_recoverable_asserts" = 3 SCOPE = BOTH SID = '*';

SQL> ALTER SYSTEM RESET "_gcs_recoverable_asserts" SCOPE = BOTH SID = '*';


Please verify as below


SELECT

   a.ksppinm AS parameter,

   b.ksppstvl AS value,

   b.ksppstdf AS is_default,

   a.ksppdesc AS description

FROM

   x$ksppi a,

   x$ksppcv b

WHERE a.indx = b.indx AND a.ksppinm = '_gcs_recoverable_asserts';


Please verify fixes are in-place 

opatch lsinventory -bugs_fixed | egrep '^(34352668|38854064|38884142)'


Oracle Database 19c Proactive Patch Information (MOS KM Doc ID 2521164.1) - R19c Database Release Update Revision (Revision)
Critical Patch Update (CPU) Program Jan 2025 Patch Availability Document (DB-only) (MOS KM Doc ID KB160593) - DETAILS
Oracle Database 19c Release Update January 2025 Known Issues (MOS KM Doc ID 19202501.9) - Purpose
Oracle Database 19c Release Update January 2025 Known Issues (MOS KM Doc ID 3113613.1) - Purpose
Critical Patch Update (CPU) Program Apr 2025 Patch Availability Document (DB-only) (MOS KM Doc ID 3070732.1) - 6 Modification History
[OCI]: Primary Note For Oracle Database Cloud Service In OCI (Oracle Cloud Infrastructure) Environment (MOS KM Doc ID KB33189) - Database Patching
Oracle Database 19c Release Update January 2024 Known Issues (MOS KM Doc ID 19202401.9) - Purpose
Oracle Database 19c Release Update January 2024 Known Issues (MOS KM Doc ID 3113605.1) - Purpose

Tuesday, 3 February 2026

DBaaS/DBCS: 19c: January 2026 patches 19.30.0.0.0 -OCI Oracle db system in base database service


DBaaS/DBCS: 19c: January 2026 patches 19.30.0.0.0  -OCI Oracle db system in base database service

There is no announced ETA  for 19c: January 2026 patches 19.30.0.0.0  in OCI Oracle db system in base database service
OCI patch images typically appear about 2–4 weeks after the quarterly release, and any delay/ETA will be posted in the quarter’s Database PAD.

To track availability for OCI Oracle db systems, check the OCI Console Patching page or run dbcli describe-component on the DB system

you can find patching detail below

Oracle Database 19c Proactive Patch Information (MOS KM Doc ID 2521164.1) - R19c Database Release Update Revision (Revision)

Critical Patch Update (CPU) Program Jan 2025 Patch Availability Document (DB-only) (MOS KM Doc ID KB160593) - DETAILS

Oracle Database 19c Release Update January 2025 Known Issues (MOS KM Doc ID 19202501.9) - Purpose

Oracle Database 19c Release Update January 2025 Known Issues (MOS KM Doc ID 3113613.1) - Purpose

Critical Patch Update (CPU) Program Apr 2025 Patch Availability Document (DB-only) (MOS KM Doc ID 3070732.1) - 6 Modification History

[OCI]: Primary Note For Oracle Database Cloud Service In OCI (Oracle Cloud Infrastructure) Environment (MOS KM Doc ID KB33189) - Database Patching

Oracle Database 19c Release Update January 2024 Known Issues (MOS KM Doc ID 19202401.9) - Purpose

Oracle Database 19c Release Update January 2024 Known Issues (MOS KM Doc ID 3113605.1) - Purpose

Friday, 30 January 2026

Steps to fix a lost Transparent Data Encryption (TDE) wallet password in an OCI Base Database Service system



 Steps to fix a lost Transparent Data Encryption (TDE) wallet password in an OCI Base Database Service system


Note : Ideally the TDE wallet password is the same as SYS password when you first provision the DB system.


To fix a lost Transparent Data Encryption (TDE) wallet password in an OCI Base Database Service system, Please use the OCI Console to update the password directly

Steps to Fix Lost TDE Wallet Password

----------------------------------------

The option is provided to change the TDE password, when you already know the existing password

Using OCI Console (Recommended)

Go to the Oracle Cloud Infrastructure (OCI) Console.

Navigate to the DB system and select the specific database.

Select Manage passwords from the Actions menu.

Select Update TDE wallet password.

Enter a new password. 


Plan b

If you do not have the old TDE wallet password and auto-login wallet is available

you need to verify existing password and merge the existing keystore into newly created empty wallet

Recovery Steps (if auto-login wallet is available)


Please follow these steps

1) Determine the TDE wallet location on the source

- Login to the source DB host as oracle.

- Find the wallet directory from sqlnet.ora:

  cat $ORACLE_HOME/network/admin/sqlnet.ora | grep ENCRYPTION_WALLET_LOCATION

- On OCI DB Systems the wallet is typically under:

 /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME

- If needed, get db_unique_name:

sqlplus "/ as sysdba"

show parameter db_unique_name


2) Verify candidate passwords against the wallet

 

- Using orapki (prompts for the wallet password):

orapki wallet display -wallet /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME/ewallet.p12 -summary

- Or using mkstore (also prompts for the password):

mkstore -wrl /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME -list

- If the password is correct, the wallet contents (master key info) will display. If incorrect, you will get a PKI/“incorrect password” error


3) Merge the existing keystore into newly created empty wallet

>>Create a backup of your current wallet

>> SELECT wrl_type, wrl_parameter, status, wallet_type FROM V$ENCRYPTION_WALLET;

As AUTOLOGIN is Yes, you can merge the wallet, please follow these steps.

a. Create a new empty wallet at some other location than the original wallet.

SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '<New location for wallet>' IDENTIFIED BY <any wallet password>;

b. Merge the existing keystore into newly created empty wallet. Here for the first Keystore there is no need to specify the password as it's Auto-Login.

SQL> ADMINISTER KEY MANAGEMENT MERGE KEYSTORE '<Original Keystore location>' INTO EXISTING KEYSTORE '<Newly created wallet location>' IDENTIFIED BY <Newly created wallet password> WITH BACKUP;

c. Create an Auto-Login wallet for this Merged Keystore.

NOTE: Here the location of the Merged wallet needs to be specified i.e. the location of newly created wallet

SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE '<Newly created wallet location>' IDENTIFIED BY <Newly created wallet password>;

d. At this point test this newly Merged wallet by modifying the wallet location in sqlnet.ora file to point to this new wallet.

$ cat <Directory>/sqlnet.ora

ENCRYPTION_WALLET_LOCATION=

(SOURCE = (METHOD = FILE)

(METHOD_DATA =

(DIRECTORY = <Merged Wallet location>))) ------ Pointing to newly Merged wallet

SQL> select * from v$encryption_wallet;

e. If the wallet is open and the database is accessible, Copy the wallet files to the default location (after taking a backup of it) and correct the path in sqlnet.ora file.


Reference: Quick TDE Setup and FAQ (Doc ID 1251597.1)

TDE Recovery Scenarios (Doc ID 3011213.1) >> It describes all the different scenarios

Primary Note For Transparent Data Encryption ( TDE ) (Doc ID 1228046.1) 

https://docs.oracle.com/en/database/oracle/oracle-database/21/asoag/managing-keystore-and-tde-master-encryption-key.html


If No Auto-Login Wallet Exists

-------------------------------

If no auto-login wallet file (cwallet.sso) is available for the old wallet, and the password for the ewallet.p12 file is lost/damaged, the encrypted data cannot be accessed, it may not be possible to open the wallet. 

In this scenario, you must restore the ewallet.p12 file from a previous backup to a time when the password was known. 

if the wallet merge steps fail, creating a new OCI Database System (DB System) from an automatic backup is the primary recovery path, provided automatic backups are enabled and the wallet is in AUTOLOGIN status. 

Create a DB System from a Backup

https://docs.oracle.com/en/cloud/paas/base-database/create-dbs-from-backup/index.html#articletitle


Regards

thanks you


Saturday, 18 October 2025

OCI: Oracle Base Database Service Training

                                    OCI:  Oracle Base Database Service Training



  • Oracle Base Database Service overview
  • Create an Base Database Service
  • Manage an Oracle Base Database Service
  • Database lifecycle management on a virtual machine DB system
  • Database metrics and management interfaces
  • Demo: Create a VCN
  • Demo: Create a VM DB system
  • Demo: Create a custom database software image
  • Demo: Changing the shape of a VM DB system
  • Demo: Scale the storage for a VM DB system
  • Demo: Patch a virtual machine DB system
  • Demo: Changing the license type of a VM DB system
  • Demo: Scale the storage for a VM DB system
  • Demo: Patch a virtual machine DB system
  • Demo: Changing the license type of a VM DB system
  • Demo: Create PDB in an existing DB system
  • Demo: Configuring automatic backups for a database on a VM DB system
  • Demo: Enable Data Guard for a VM DB system

https://docs.oracle.com/en/cloud/paas/base-database/data-guard/index.html
https://docs.public.content.oci.oraclecloud.com/en-us/iaas/releasenotes/database/basedb-cross-region-dg-virtual-vault.htm






Friday, 11 July 2025

Issue while precheck on Oracle Base Database Service and observed Object storage connectivity issue on DB system in OCI

 Subject : Issue while precheck on Oracle Base Database Service and observed Object storage connectivity issue on DB system 


Summary : It is observed that Grid and Database patching precheck is failing on a DB system or two-node RAC DB system due to Object Storage connectivity issues while downloading artifacts . 

         

          Expected error : 1) Error : Failed to download patchmetadata from objectstore!

   2) DCS-10406:Failed to connect to Object Storage while downloading /latest/gicsControl.json


if it is happening on two-node RAC DB system  ,We need to verify OSS connectivity on both node


Check Each Node Individually:


1) Log in to each node of your 2-node RAC system individually.

2) On each node, run the below command to verify connectivity to Oracle Cloud Infrastructure (OCI) Object Storage.

   

cd /opt/oracle/dcs/bin 

/opt/oracle/dcs/bin/dbcli describe-component 

/opt/oracle/dcs/bin/dbcli describe-latestpatch 


  

An example of a successful output would include a list of available patches.

A failure output will display an error like: "DCS-10032:Resource patch metadata is not found.Failed to download patchmetadata from objectstore".



Diagnose and Troubleshoot Connectivity Issues

-----------------------------------------------

To check if your Oracle Database (DB) system in Oracle Cloud Infrastructure (OCI) can connect to OCI Object Storage, 


consider the following methods:


1. Network connectivity validation


Test with curl: Access your DB system via SSH and run a curl command to the Object Storage API endpoint for the region your DB system is in.


For example, 


if your DB system is in the Ashburn region, use: 


curl https://objectstorage.<region>.oraclecloud.com


eg


curl https://objectstorage.us-ashburn-1.oraclecloud.com

curl https://objectstorage.me-abudhabi-3.oraclecloud26.com


If the connection is successful, 


you'll receive a JSON object response, like: {"code":"NotAuthorizedOrNotFound","message":"Authorization failed or requested resource not found."}.


A timeout or hanging SSH session indicates a network connectivity issue.



Possible causes

----------------

Network Configuration Problems:

Incorrectly configured Virtual Cloud Network (VCN) or subnet settings, preventing access to the OCI Services Network.

Firewall rules (either within the DB system's host OS or VCN security lists) blocking Object Storage traffic.

DNS resolution issues for Object Storage endpoints.

VPNs or web proxy servers interfering with connectivity.



Recommendation

---------------


Please verify below configuration to fix connectivity issue for Object Storage:


- Ensure Service gateway points to all services.

- Route rule has service gateway and all services.

- Egress has rule to all services.

- DNS (if custom), test with internet DNS.( if two node verify from both node)

  ls -ltr /etc/resolve.conf

  cat  /etc/resolve.conf 



- Please check if dns is also working fine ( if two node verify from both node)

  eg nslookup objectstorage.me-abudhabi-3.oraclecloud26.com 

- Please check if you able to connect to objectstorage from dbsystem using IP address?( if two node verify from both node)

- Please verify ( if two node verify from both node)

  

 (a) vi /etc/hosts


(b) nslookup to objectstorage


(c) tcpdump from both nodes while you test connecting to object storage and perform nslookup


- Firewall Configuration: Ensure firewalls (both operating system and network) are not blocking outgoing connections to the Object Storage endpoints.


    systemctl status iptables

    systemctl status firewalld


Please refer below doc

=======================

Troubleshoot Network Connectivity Failures

-----------------------------------------

https://docs.oracle.com/en/cloud/paas/base-database/troubleshoot-network/index.html#articletitle


https://docs.oracle.com/en/cloud/paas/base-database/vcn-subnets/index.html#GUID-28682953-F86D-41DA-9FDF-7B53D1E5BE68


Regions and Availability Domains

-------------------------------

https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm


Validate VCN&Subnet and Security list

------------------------------------

https://docs.oracle.com/en-us/iaas/dbcs/doc/vcn-and-subnets.html


https://docs.oracle.com/en-us/iaas/dbcs/doc/security-rules-db-system.html


Saturday, 12 April 2025

Update the Operating System of a DB System in Oracle Cloud Infrastructure (OCI) using DBCLI Command Step By Step

 

Update the Operating System of a DB System in Oracle Cloud Infrastructure (OCI) using DBCLI Command


 

In this Article, we will discuss about how to update the Operating System of Two node DB System in Oracle Cloud Infrastructure (OCI) using DBCLI Command in Rolling Fashion (One by One)

The current configuration is Two Node DB System

EASYDB1 And EASYDB2

Note:

1)    Some OS update operations require a reboot after update is complete. Use the dbcli get-availableospatches Command to confirm it

2)    You can use the -l (--local) flag to update the server components only in the current node(One By One):

3)    Oracle recommends rebooting the DB system if any kernel update is present in OS update.

4)    Oracle does not recommend installing OS packages or dependencies that are not part of the version lock that Oracle provides

 

Prerequisites

 

  • Back up the database in the DB system prior to attempting an OS update.
  • Do not remove packages from a DB system. However, you might have to remove custom RPMs (packages that were installed after the system was provisioned) for the update to complete successfully.
  • Oracle recommends that you test any updates thoroughly on a non-production system before updating a production system.

 

Apply OS Patch on EASYDB1(Node1)

 

Login DB System using Root User

1)     check installed update versions a

Sudo su –

Cd /opt/oracle/dcs/bin/dbcli

dbcli describe-component

2)     Check system status before patching

dbcli describe-system

3)     identify updates you want to apply to the OS.

Note that if the rebootIsRequired field is "true", you must reboot the DB system

dbcli get-availableospatches

4)     to get the output in JSON, use the following command.

dbcli get-availableospatches -j

5)    run a precheck on Node1


dbcli update-server -c os -p -l

6)    Check job details.

dbcli describe-job -i <Job_id>

If the precheck is successful and uncovers no issues that prevent a successful update operation, you can update the OS. If the precheck is not successful, address the issues identified by the precheck before trying to update the OS.

 

Note:

You can use the -l (--local) flag to update the server components only in the current node.


7) update the OS on Node1

dbcli update-server -c os -l


Note:
You can use the -l (--local) flag to update the server components only in the current node.

Note:

f the OS update requires a reboot, reboot the server after the update operation is complete.

 

Verification

dbcli describe-component

dbcli describe-system

crsctl stat res -t

Ensure:

- All resources are ONLINE

- System status is Ready

- No failed components

 

 

Apply OS Patch on EASYDB2(Node2)

 

1) Check system status before patching

dbcli describe-system

 

 pre-check and Apply OS Patch as Node2 and do verification as we have done for node1.

2) Node wise pre check

dbcli update-server -c os -p -l

 

3) Node wise Os patch apply

dbcli update-server -c os -l

 

After Node 2 Reboots, Validate

 

 Verify patch success on Node2 as below

dbcli describe-component

dbcli describe-system

crsctl stat res -t

Optional: Verify Kernel Version After Patching (Both Nodes)

 

uname -r

You should see an updated kernel version (if a new UEK patch was applied).

Please refer below for more Details

https://docs.oracle.com/en/cloud/paas/base-database/cli-reference/index.html#GUID-1B083322-AAE3-47AB-A5E6-FF19EA81D8B3

https://docs.oracle.com/en/cloud/paas/base-database/update-dbcli/index.html#GUID-2E59FE04-7716-4A66-B6F1-9AA622CDD33C

https://docs.oracle.com/en/cloud/paas/base-database/cli-reference/index.html#GUID-A6FA643C-99E2-4CCD-B376-A0D4CC508C4B

https://docs.oracle.com/en/cloud/paas/base-database/update-dbcli/#articletitle



Wednesday, 20 November 2024

oracle database 23ai new features for administrators

Oracle Database 23ai brings over 300 new features, focusing heavily on AI integration, simplifying developer operations, and strengthening database administration. Key administration highlights include built-in AI SQL firewalls, True Cache for application scaling, AI-driven auto-indexing, and simplified Schema-level privilege

The most impactful enhancements for Database Administrators (DBAs) are detailed below:


Security & Governance

  • Oracle SQL Firewall: Built directly into the database kernel, this inspects incoming connections and SQL statements against an allowlist of approved commands, actively blocking unauthorized execution and potential SQL injection attacks.
  • Schema Privileges: You can now grant access to all existing and future objects within a single schema using a single command, vastly reducing the administrative overhead of individual privilege management.
  • Passwordless Authentication: Streamlines security compliance by supporting biometric logins (Face ID, fingerprints), hardware security keys (YubiKey), and OAuth-based logins.
  • Immutable Backups: RMAN now supports immutable storage integration with OCI Object Storage, preventing backups from being deleted or modified during a ransomware event


 Performance & Caching
  • Oracle True Cache: A diskless, read-only cache layer you can deploy at the middle tier. It automatically maintains data consistency with the primary database, improving application response times and reducing the workload on database servers.
  • AI-Driven Auto-Index Optimization: The database can now learn query patterns and dynamically create or drop indexes based on workload, mitigating under-indexing (which slows queries) and over-indexing (which slows updates).

Operational Management
  • Data Pump Enhancements: Troubleshooting is much faster with three new views (V$DATAPUMP_PROCESS_INFO, V$DATAPUMP_PROCESSWAIT_INFO, and V$DATAPUMP_SESSIONWAIT_INFO) which expose Data Pump processes and wait events to identify bottlenecks quickly.
  • Read-Only PDBs: Administrators can make individual Pluggable Database (PDB) users or entire connected sessions strictly read-only, preventing unwanted write operations.
  • AutoUpgrade Utility Enhancements: Upgrading to 23ai from versions 19c or 21c is highly streamlined using the updated AutoUpgrade tool, and patching is simplified with proactively provided monthly updates
 AI Workloads & Operations
  • AI Vector Search: Allows the database to store and query semantic meanings of unstructured data (text, images) alongside relational predicates, all without needing an external vector database.
  • SQL AI Syntax: You can interact with and query the database directly using natural language queries, speeding up day-to-day administrative exploration

Oracle Database 23ai brings critical enhancements to Real Application Clusters (RAC) administration, focusing on patching, containerization, and automated disaster recovery. For database administrators, these updates significantly streamline cluster operations and improve high availability without requiring major infrastructure overhauls

Key RAC features for administrators in 23ai include:
  • Rolling Patching for Complex Changes: Administrators can now perform rolling patches for complex grid infrastructure modifications, minimizing downtime and human intervention.
  • DBMS_ROLLING Integration with Transparent Application Continuity (TAC): This integration allows for completely seamless, uninterrupted database operations during rolling upgrades or patching, routing sessions transparently without application disruption.
  • Reduced Grid Infrastructure Disk Footprint: The GI disk footprint has been reduced by more than 50% compared to previous versions. This enables up to 33% faster initial installations and significantly quicker "add node" operations.
  • Lightweight Container Support: The highly optimized disk footprint makes deploying RAC environments in containers—such as Docker or Podman—faster, lighter, and much more practical for testing and dev or microservices architectures.
  • RAC Fast Pluggable Database Open: Administrators can now manage pluggable database (PDB) opening and closing operations in RAC environments much faster, localizing PDB status changes without impacting the whole CDB instance.
  • True Cache Integration: For read-heavy, latency-sensitive applications, administrators can leverage Oracle True Cache, a middle-tier, in-memory cache that automatically scales with RAC environments without requiring complex application change

Oracle Database 23ai introduces several architectural changes and features specifically related to memory structures that assist administrators in optimizing performance for AI and mission-critical workloads

New Memory Structures & Frameworks
  • Managed Global Area (MGA): This is a significant addition to the Oracle Database Architecture. The MGA is a memory framework designed to share and coordinate memory across a set of trusted Oracle processes, improving efficiency in modern, high-concurrency environments.
  • Oracle True Cache: A new, high-performance, middle-tier caching solution. It functions as a read-only, memory-consistent replica of the primary database, satisfying queries directly from its own buffer cache to reduce the load on the main database server.
  • AI Vector Search Memory speed: Support for high-speed vector searches in memory. Specifically on Exadata platforms, this includes "Memory Speed Vector Search," which leverages memory-optimized structures to handle multi-dimensional vector mappings for AI similarity searches
Memory-Related Performance Improvements 
  • In-Memory Columnar Speed for JSON: The database now provides In-Memory Columnar speed for JSON queries, allowing JSON data to be stored and processed in the In-Memory Column Store using optimized columnar formats.
  • Fast Ingest (Memoptimized Rowstore) Enhancements: Improvements to the Memoptimized Rowstore now include support for direct in-memory column store population and fast flush using direct writes, which streamlines high-frequency data ingestion.
  • Large Column Fetch Efficiency: Internal protocol changes have made fetching large columns more efficient for modern drivers, reducing the memory overhead during large-scale data transfer
Administrative Controls
  • Automatic KEEP Object Load: On Exadata systems, administrators can benefit from the automatic loading of "KEEP" objects into Flash Cache, ensuring critical data remains in the fastest memory-adjacent storage.
  • Lock-Free Column Value Reservations: This feature allows for the creation of rules for data row locking, which improves memory throughput and reduces contention in high-volume transaction environments.
  • Memory Usage Monitoring: New administrative scripts and documentation are available to monitor and manage the System Global Area (SGA) and Program Global Area (PGA) more effectively
 

1. AI & Performance Management
  • AI Vector Search: Allows direct storage, indexing, and querying of vector data alongside relational data, enabling semantic search capabilities without moving data to specialized vector databases.
  • Automatic SQL Plan Management (SPM): The database actively monitors and auto-repairs suboptimal query execution plans to improve stability.
  • True Cache: A memory-only, read-only cache that offloads read workloads and satisfies queries locally

Upgrading to Oracle Database 23ai requires a source version of Oracle 19c or 21c. The upgrade process is streamlined using the Oracle AutoUpgrade Utility, which automates pre-checks, fixups, and post-upgrade steps, replacing the older DBUA tool
Upgrade Considerations
  • Direct Path: You can upgrade directly from Oracle 19c or 21c to 23ai. Direct upgrades from older releases (like 11g or 12c) require an intermediate upgrade step.
  • AutoUpgrade Process: The recommended method involves executing the AutoUpgrade Utility in -mode analyze to generate reports and fix potential pre-upgrade issues before pushing to deployment.
  • Client and Grid Infrastructure: If running RAC, Grid Infrastructure must be upgraded prior to the database upgrade. It is also highly recommended to upgrade client software for full interoperability


Oracle Database 23ai introduces several performance monitoring and administrative enhancements focused on AI-driven automation, real-time diagnostics, and streamlined data management. 

Key Performance Monitoring Features
  • Automatic SQL Plan Management (SPM): The system can now automatically detect and correct suboptimal SQL plans by providing real-time optimization.
  • Real-Time SQL Monitoring & Statistics: Enhanced tools for tracking query performance as it happens, including improved diagnostics and real-time statistics to identify bottlenecks quickly.
  • SQL Performance Watch: A new feature to help DBAs monitor and maintain consistent performance during and after database upgrades.
  • True Cache: A middle-tier caching solution that satisfies queries using a read-only replication of the primary database's buffer cache, significantly improving application response times and reducing server load.
  • Enhanced Observability: New observability dashboards and telemetry provide proactive alerting and resource monitoring
Administrative & Performance Enhancements
  • Lock-Free Column Value Reservations: Allows developers and administrators to create rules for row locking, improving throughput and end-user experience for high-concurrency workloads.
  • Priority Transactions: Enables automatic prioritization of urgent transactions to prevent them from being blocked by lower-priority row access.
  • Data Pump Performance Views: Three new views—(G)V$DATAPUMP_PROCESS_INFO, (G)V$DATAPUMP_PROCESSWAIT_INFO, and (G)V$DATAPUMP_SESSIONWAIT_INFO—help administrators identify bottlenecks and troubleshoot Data Pump operations more effectively.
  • Automatic Transaction Quarantine: Automatically handles problematic transactions that might otherwise impact system stability.
  • Unrestricted Parallel DMLs: Improvements to parallel data manipulation language (DML) and direct loads to enhance data processing speed.Flashback Log Placement: Administrators can now manage Flashback database logs outside the Fast Recovery Area (FRA) for better storage flexibility


To compare performance and resolve issues after an upgrade to Oracle Database 23ai,


To compare performance and resolve issues after an upgrade to Oracle Database 23ai, you should use Oracle's built-in performance validation tools. These tools allow you to compare a "before" (pre-upgrade) baseline with an "after" (post-upgrade) trial to pinpoint regressions

Core Tools for Comparison
  • SQL Performance Analyzer (SPA): This is the primary tool for identifying SQL regressions. It captures a representative workload into a SQL Tuning Set (STS) from your pre-upgrade environment and re-executes it in 23ai to generate a comparison report.
  • SQL Performance Watch: A feature within OCI Database Management and Enterprise Manager that helps automate the comparison of workload performance across two time periods (pre- and post-upgrade).
  • Database Replay: For high-load environments, this tool captures actual production concurrency and re-plays it on the upgraded 23ai system to identify system-level bottlenecks that single SQL execution might miss
Comparison Workflow
  1. Capture Baseline: Use your pre-upgrade system (e.g., 19c) to capture a representative workload during peak time into an STS.
  2. Execute Post-Upgrade Trial: Import the STS into the 23ai database and run a "trial" execution. Tools like SQL Performance Analyzer will generate the execution plans and statistics for the new version.
  3. Generate Comparison Report: Run a comparison task to view the net impact. Focus on metrics like Elapsed Time, CPU time, and changes in execution plan hash values.
  4. Identify Regressions: Review the report for SQL statements with "Regressed" status. These are queries that performed better in the previous version
Post-Upgrade Performance Fixes
If you find performance issues after the move to 23ai:
  • Gather Fixed Object Statistics: Run DBMS_STATS.GATHER_FIXED_OBJECTS_STATS after the system has run a representative workload in 23ai. This helps the optimizer understand the internal memory structures of the new version.
  • Gather Dictionary Statistics: Use DBMS_STATS.GATHER_DICTIONARY_STATS to ensure the data dictionary metadata is accurately represented for the 23ai optimizer.
  • SQL Tuning: Use the SQL Tuning Advisor or SQL Plan Management (SPM) to force the 23ai engine to use a better-performing plan if a regression is found.
  • Check Initialization Parameters: Ensure parameters that affect performance haven't changed defaults significantly during the upgrade


How to transfer optimizer statistics and execution plans from your source database to your 23ai target database


 you can absolutely transfer both optimizer statistics and execution plans from your source database to your 23ai target database. This is a standard and highly recommended practice to prevent performance regressions following an upgrade

Method 1: Transferring Optimizer Statistics
You can move statistics by exporting them from the source data dictionary to a temporary staging table, moving that table via Data Pump, and importing it into the target.
1. Create a staging table on the Source:
EXEC DBMS_STATS.CREATE_STAT_TABLE('SYSTEM', 'MY_STAT_TABLE');

2. Export statistics from the Source (Schema Level):
EXEC DBMS_STATS.EXPORT_SCHEMA_STATS(ownname=>'YOUR_SCHEMA', stattab=>'MY_STAT_TABLE', statid=>'SOURCE_STATS', ownname_old=>'SYSTEM');

(Note: You can also export by table or database using EXPORT_TABLE_STATS or EXPORT_DATABASE_STATS.)
3. Move and Import to Target:
  • Use Oracle Data Pump (expdp/impdp) to export MY_STAT_TABLE from the source, transfer the .dmp file, and import it into the target database.
  • Once imported, apply the statistics to your 23ai target's data dictionary
sql
EXEC DBMS_STATS.IMPORT_SCHEMA_STATS(ownname=>'YOUR_SCHEMA', stattab=>'MY_STAT_TABLE', statid=>'SOURCE_STATS', ownname_old=>'SYSTEM');

Method 2: Transferring Execution Plans (SQL Plan Management)
To lock in exact execution plans that were performing well on your source version, you can capture them as SQL Plan Baselines and migrate them. [1]
1. Load plans into a SQL Tuning Set (STS) on the Source:

sql
BEGIN
  DBMS_SQLTUNE.CREATE_SQLSET(sqlset_name => 'GOOD_PLANS_STS');
END;
/

DECLARE
  l_cursor DBMS_SQLTUNE.SQLSET_CURSOR;
BEGIN
  OPEN l_cursor FOR
    SELECT VALUE(p) FROM TABLE(DBMS_SQLTUNE.SELECT_CURSOR_CACHE('sql_id = ''YOUR_SQL_ID''')) p;
  DBMS_SQLTUNE.LOAD_SQLSET(sqlset_name => 'GOOD_PLANS_STS', populate_cursor => l_cursor);
END;
/

2. Transport and Unpack:
  • Package the STS into a staging table using DBMS_SQLTUNE.PACK_STGTAB_SQLSET and export it using Data Pump.
  • Import the Data Pump file into your 23ai target database and unpack it using DBMS_SQLTUNE.UNPACK_STGTAB_SQLSET. [1]
3. Evolve into Baselines on the Target:
  • Once the SQL Tuning Set is on your 23ai target, you can easily load those historical plans as active baselines: [1]

DECLARE l_plans PLS_INTEGER; BEGIN l_plans := DBMS_SPM.LOAD_COV_SQLSET(sqlset_name => 'GOOD_PLANS_STS'); END; /


Pro-Tip for 23ai Post-Upgrade Performance
Before importing your source statistics, ensure that your 23ai database has its fixed object statistics properly gathered. Because the 23ai data dictionary and internal memory architectures (fixed tables) differ from older releases, running this after upgrading prevents critical performance degradation
sql
EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;



For more details

https://learn.oracle.com/ols/course/oracle-database-23ai-new-features-for-administrators/127088/133283