Issue while precheck on Oracle Base Database Service and observed Object storage connectivity issue on DB system in OCI

 Subject : Issue while precheck on Oracle Base Database Service and observed Object storage connectivity issue on DB system 

Summary : It is observed that Grid and Database patching precheck is failing on a DB system or two-node RAC DB system due to Object Storage connectivity issues while downloading artifacts . 

         

          Expected error : 1) Error : Failed to download patchmetadata from objectstore!

   2) DCS-10406:Failed to connect to Object Storage while downloading /latest/gicsControl.json

if it is happening on two-node RAC DB system  ,We need to verify OSS connectivity on both node

Check Each Node Individually:

1) Log in to each node of your 2-node RAC system individually.

2) On each node, run the below command to verify connectivity to Oracle Cloud Infrastructure (OCI) Object Storage.

   

cd /opt/oracle/dcs/bin 

/opt/oracle/dcs/bin/dbcli describe-component 

/opt/oracle/dcs/bin/dbcli describe-latestpatch 

  

An example of a successful output would include a list of available patches.

A failure output will display an error like: "DCS-10032:Resource patch metadata is not found.Failed to download patchmetadata from objectstore".

Diagnose and Troubleshoot Connectivity Issues

-----------------------------------------------

To check if your Oracle Database (DB) system in Oracle Cloud Infrastructure (OCI) can connect to OCI Object Storage, 

consider the following methods:

1. Network connectivity validation

Test with curl: Access your DB system via SSH and run a curl command to the Object Storage API endpoint for the region your DB system is in.

For example, 

if your DB system is in the Ashburn region, use: 

curl https://objectstorage.<region>.oraclecloud.com

eg

curl https://objectstorage.us-ashburn-1.oraclecloud.com

curl https://objectstorage.me-abudhabi-3.oraclecloud26.com

If the connection is successful, 

you'll receive a JSON object response, like: {"code":"NotAuthorizedOrNotFound","message":"Authorization failed or requested resource not found."}.

A timeout or hanging SSH session indicates a network connectivity issue.

Possible causes

----------------

Network Configuration Problems:

Incorrectly configured Virtual Cloud Network (VCN) or subnet settings, preventing access to the OCI Services Network.

Firewall rules (either within the DB system's host OS or VCN security lists) blocking Object Storage traffic.

DNS resolution issues for Object Storage endpoints.

VPNs or web proxy servers interfering with connectivity.

Recommendation

---------------

Please verify below configuration to fix connectivity issue for Object Storage:

- Ensure Service gateway points to all services.

- Route rule has service gateway and all services.

- Egress has rule to all services.

- DNS (if custom), test with internet DNS.( if two node verify from both node)

  ls -ltr /etc/resolve.conf

  cat  /etc/resolve.conf 

- Please check if dns is also working fine ( if two node verify from both node)

  eg nslookup objectstorage.me-abudhabi-3.oraclecloud26.com 

- Please check if you able to connect to objectstorage from dbsystem using IP address?( if two node verify from both node)

- Please verify ( if two node verify from both node)

  

 (a) vi /etc/hosts

(b) nslookup to objectstorage

(c) tcpdump from both nodes while you test connecting to object storage and perform nslookup

- Firewall Configuration: Ensure firewalls (both operating system and network) are not blocking outgoing connections to the Object Storage endpoints.

    systemctl status iptables

    systemctl status firewalld

Please refer below doc

=======================

Troubleshoot Network Connectivity Failures

-----------------------------------------

https://docs.oracle.com/en/cloud/paas/base-database/troubleshoot-network/index.html#articletitle

https://docs.oracle.com/en/cloud/paas/base-database/vcn-subnets/index.html#GUID-28682953-F86D-41DA-9FDF-7B53D1E5BE68

Regions and Availability Domains

-------------------------------

https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm

Validate VCN&Subnet and Security list

------------------------------------

https://docs.oracle.com/en-us/iaas/dbcs/doc/vcn-and-subnets.html

https://docs.oracle.com/en-us/iaas/dbcs/doc/security-rules-db-system.html