Issue while precheck on Oracle Base Database Service and observed Object storage connectivity issue on DB system in OCI

 Subject : Issue while precheck on Oracle Base Database Service and observed Object storage connectivity issue on DB system 

Summary : It is observed that Grid and Database patching precheck is failing on a DB system or two-node RAC DB system due to Object Storage connectivity issues while downloading artifacts . 

         

          Expected error : 1) Error : Failed to download patchmetadata from objectstore!

   2) DCS-10406:Failed to connect to Object Storage while downloading /latest/gicsControl.json

if it is happening on two-node RAC DB system  ,We need to verify OSS connectivity on both node

Check Each Node Individually:

1) Log in to each node of your 2-node RAC system individually.

2) On each node, run the below command to verify connectivity to Oracle Cloud Infrastructure (OCI) Object Storage.

   

cd /opt/oracle/dcs/bin 

/opt/oracle/dcs/bin/dbcli describe-component 

/opt/oracle/dcs/bin/dbcli describe-latestpatch 

  

An example of a successful output would include a list of available patches.

A failure output will display an error like: "DCS-10032:Resource patch metadata is not found.Failed to download patchmetadata from objectstore".

Diagnose and Troubleshoot Connectivity Issues

-----------------------------------------------

To check if your Oracle Database (DB) system in Oracle Cloud Infrastructure (OCI) can connect to OCI Object Storage, 

consider the following methods:

1. Network connectivity validation

Test with curl: Access your DB system via SSH and run a curl command to the Object Storage API endpoint for the region your DB system is in.

For example, 

if your DB system is in the Ashburn region, use: 

curl https://objectstorage.<region>.oraclecloud.com

eg

curl https://objectstorage.us-ashburn-1.oraclecloud.com

curl https://objectstorage.me-abudhabi-3.oraclecloud26.com

If the connection is successful, 

you'll receive a JSON object response, like: {"code":"NotAuthorizedOrNotFound","message":"Authorization failed or requested resource not found."}.

A timeout or hanging SSH session indicates a network connectivity issue.

Possible causes

----------------

Network Configuration Problems:

Incorrectly configured Virtual Cloud Network (VCN) or subnet settings, preventing access to the OCI Services Network.

Firewall rules (either within the DB system's host OS or VCN security lists) blocking Object Storage traffic.

DNS resolution issues for Object Storage endpoints.

VPNs or web proxy servers interfering with connectivity.

Recommendation

---------------

Please verify below configuration to fix connectivity issue for Object Storage:

- Ensure Service gateway points to all services.

- Route rule has service gateway and all services.

- Egress has rule to all services.

- DNS (if custom), test with internet DNS.( if two node verify from both node)

  ls -ltr /etc/resolve.conf

  cat  /etc/resolve.conf 

- Please check if dns is also working fine ( if two node verify from both node)

  eg nslookup objectstorage.me-abudhabi-3.oraclecloud26.com 

- Please check if you able to connect to objectstorage from dbsystem using IP address?( if two node verify from both node)

- Please verify ( if two node verify from both node)

  

 (a) vi /etc/hosts

(b) nslookup to objectstorage

(c) tcpdump from both nodes while you test connecting to object storage and perform nslookup

- Firewall Configuration: Ensure firewalls (both operating system and network) are not blocking outgoing connections to the Object Storage endpoints.

    systemctl status iptables

    systemctl status firewalld

Please refer below doc

=======================

Troubleshoot Network Connectivity Failures

-----------------------------------------

https://docs.oracle.com/en/cloud/paas/base-database/troubleshoot-network/index.html#articletitle

https://docs.oracle.com/en/cloud/paas/base-database/vcn-subnets/index.html#GUID-28682953-F86D-41DA-9FDF-7B53D1E5BE68

Regions and Availability Domains

-------------------------------

https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm

Validate VCN&Subnet and Security list

------------------------------------

https://docs.oracle.com/en-us/iaas/dbcs/doc/vcn-and-subnets.html

https://docs.oracle.com/en-us/iaas/dbcs/doc/security-rules-db-system.html

Update the Operating System of a DB System in Oracle Cloud Infrastructure (OCI) using DBCLI Command Step By Step

 

Update the Operating System of a DB System in Oracle Cloud Infrastructure (OCI) using DBCLI Command

 

In this Article, we will discuss about how to update the Operating System of Two node DB System in Oracle Cloud Infrastructure (OCI) using DBCLI Command in Rolling Fashion (One by One)

The current configuration is Two Node DB System

EASYDB1 And EASYDB2

Note:

1)    Some OS update operations require a reboot after update is complete. Use the dbcli get-availableospatches Command to confirm it

2)    You can use the -l (--local) flag to update the server components only in the current node(One By One):

3)    Oracle recommends rebooting the DB system if any kernel update is present in OS update.

4)    Oracle does not recommend installing OS packages or dependencies that are not part of the version lock that Oracle provides

 

Prerequisites

 

• Back up the database in the DB system prior to attempting an OS update.
• Do not remove packages from a DB system. However, you might have to remove custom RPMs (packages that were installed after the system was provisioned) for the update to complete successfully.

• Oracle recommends that you test any updates thoroughly on a non-production system before updating a production system.

 

Apply OS Patch on EASYDB1(Node1)

 

Login DB System using Root User

1)     check installed update versions a

Sudo su –

Cd /opt/oracle/dcs/bin/dbcli

dbcli describe-component

2)     Check system status before patching

dbcli describe-system

3)     identify updates you want to apply to the OS.

Note that if the rebootIsRequired field is "true", you must reboot the DB system

dbcli get-availableospatches

4)     to get the output in JSON, use the following command.

dbcli get-availableospatches -j

5)    run a precheck on Node1

dbcli update-server -c os -p -l

6)    Check job details.

dbcli describe-job -i <Job_id>

If the precheck is successful and uncovers no issues that prevent a successful update operation, you can update the OS. If the precheck is not successful, address the issues identified by the precheck before trying to update the OS.

 

Note:

You can use the -l (--local) flag to update the server components only in the current node.

7) update the OS on Node1

dbcli update-server -c os -l

Note:
You can use the -l (--local) flag to update the server components only in the current node.

Note:

f the OS update requires a reboot, reboot the server after the update operation is complete.

 

Verification

dbcli describe-component

dbcli describe-system

crsctl stat res -t

Ensure:

- All resources are ONLINE

- System status is Ready

- No failed components

 

 

Apply OS Patch on EASYDB2(Node2)

 

1) Check system status before patching

dbcli describe-system

 

 pre-check and Apply OS Patch as Node2 and do verification as we have done for node1.

2) Node wise pre check

dbcli update-server -c os -p -l

 

3) Node wise Os patch apply

dbcli update-server -c os -l

 

After Node 2 Reboots, Validate

 

 Verify patch success on Node2 as below

dbcli describe-component

dbcli describe-system

crsctl stat res -t

Optional: Verify Kernel Version After Patching (Both Nodes)

 

uname -r

You should see an updated kernel version (if a new UEK patch was applied).

Please refer below for more Details

https://docs.oracle.com/en/cloud/paas/base-database/cli-reference/index.html#GUID-1B083322-AAE3-47AB-A5E6-FF19EA81D8B3

https://docs.oracle.com/en/cloud/paas/base-database/update-dbcli/index.html#GUID-2E59FE04-7716-4A66-B6F1-9AA622CDD33C

https://docs.oracle.com/en/cloud/paas/base-database/cli-reference/index.html#GUID-A6FA643C-99E2-4CCD-B376-A0D4CC508C4B

https://docs.oracle.com/en/cloud/paas/base-database/update-dbcli/#articletitle