EasyReliableDBA: Managing Database on ExaCC (X10M) part2

Q1: What makes ExaCC X10M performance tuning different from traditional on-premises Exadata or generic cloud infrastructure?

Answer: The primary shift lies in compute density and memory architecture. X10M uses AMD EPYC processors, providing massively higher core counts per socket. Performance tuning shifts from being CPU-bound to managing massive concurrency and NUMA nodes. Furthermore, X10M introduces XRMEM (Exadata RDMA Memory) as a caching layer in front of NVMe Flash. Data is fetched via Remote Direct Memory Access (RDMA) bypassing the storage software stack completely, achieving sub-10 microsecond latencies.

Q2: How do you identify if a performance bottleneck is related to the ExaCC RoCE network versus the storage cells?

Answer: I analyze the Automatic Workload Repository (AWR) and check Top 10 Foreground Wait Events.

If I see high waits on cell single block physical read or cell smart table scan, the bottleneck is likely at the storage cell or SQL execution plan level.

If I see high gc cr request or gc current block 2-way/3-way with high average wait times (>1-2ms), the issue points to RAC Interconnect congestion on the RoCE fabric or severe hot blocks. I would cross-verify this by checking the cell OS metrics using cellcli or checking for dropped packets on the RoCE switches.

Q3: A query is underperforming on ExaCC X10M. How do you verify if it is utilizing Smart Scan?

Answer: I check the execution plan or query V$SQL_SYSSTAT / V$SESSTAT. Specifically, I look for:

cell smart table scan or cell smart index fast full scan in the execution plan.

The ratio between "cell physical IO bytes eligible for predicate offload" and "cell physical IO bytes returned by smart scan" in V$SQL. If the bytes returned are significantly lower than eligible bytes, the Smart Scan is efficient. If they are equal, the storage cell is returning all data without filtering, indicating poor predicate efficiency.

Real-World Challenges on ExaCC X10M

NUMA Node Misalignment: Because X10M AMD processors have a high number of cores and multiple NUMA domains per socket, poorly configured Oracle initialization parameters (like _enable_NUMA_optimization) or incorrect VM sizing can lead to excessive cross-node memory latencies inside the database node.

XRMEM Cache Invalidation: Heavy OLTP DML workloads on specific tables can cause rapid cache invalidations in the XRMEM/Flash Cache layer, causing performance to drop back to physical NVMe disk speeds.

Smart Scan Disabling Factors: Unintentional use of functions in WHERE clauses that prevent predicate offloading, or choosing wrong data types, causes queries to switch to regular block shipping, crippling ExaCC's processing advantages.

Troubleshooting Checklist

Step 1: Check System Health via Cloud Control / OCI Console. Ensure no storage cells are degraded and no RoCE links are down.

Step 2: Generate AWR / ASH Reports. Focus on the Top 5 Timed Foreground Events. Look for Exadata-specific waits like cell single block physical read, cell smart table scan, and RAC waits like gc current block receive.

Step 3: Analyze Storage Performance via CellCLI. Run list metriccurrent where alertState='warning' and analyze Cell Server IO throughput and IOPS to see if IO throttling (IORM) is triggering.

Step 4: Check Execution Plan Details. Use DBMS_XPLAN.DISPLAY_CURSOR(format=>'+ADVANCED') to check for PREDICATE INFO. Ensure filters are actually being pushed down to the storage layer.

Performance Analysis Test Case

Scenario

A critical reporting query (Monthly Financial Aggregation) takes 45 minutes to run on the newly migrated ExaCC X10M platform, failing to meet the 5-minute SLA.

Test Execution & Diagnostics

Run the Query & Capture SQL ID: select sql_id, child_number, sql_text from v$sql where sql_text like '%Monthly Financial Aggregation%'; (Assume SQL ID: 7g1h2j3k4m5p6).

Generate SQL Monitor Report:

sql

SELECT DBMS_SQLTUNE.REPORT_SQL_MONITOR(sql_id => '7g1h2j3k4m5p6', type => 'TEXT') FROM DUAL;

Diagnostic Findings:

The SQL Monitor shows high database time spent on direct path read rather than cell smart table scan.
The Predicate Information section reveals an implicit data type conversion: a VARCHAR2 column is compared against a NUMBER bind variable (TO_NUMBER(ACCOUNT_ID) = :B1).
This implicit conversion completely disabled Exadata Smart Scan Offloading, forcing the storage cell to ship every raw database block over the network to the database node compute layer.

Resolution & Validation

Fix: Modified the application code to match data types properly, ensuring the bind variable passed was a VARCHAR2.

Retest Result: The execution plan switched to CELL SMART TABLE SCAN.

Metrics Comparison:

Metric	Before Fix (No Smart Scan)	After Fix (Smart Scan Active)
Execution Time	45 minutes	1.8 minutes
Database Wait Event	`direct path read` (High)	`cell smart table scan`
Interconnect Traffic	12 GB/sec shipped	150 MB/sec shipped

Part 1: Project Details (The Context)

Project Name: Mission-Critical Financial Core Migration to ExaCC X10M.

Infrastructure: ExaCC X10M Gen2 4-Node RAC Compute Cluster (VM Cluster) linked to 6 Exadata Storage Servers via dual 100Gbps RoCE networks.

Software Stack: Grid Infrastructure 19c (19.22+) with ASM, Oracle Database 19c (Multitenant Architecture) running container/pluggable databases (CDB/PDB).

Workload Pattern: High-frequency Online Transaction Processing (OLTP) paired with heavy batch analytics.

Part 2: Top Interview Questions & Answers

Q1: On ExaCC X10M, how does CSSD locate the Voting Disks at startup if they are stored in ASM Diskgroups before the ASM instance is mounted?

Answer: This avoids a "chicken-and-egg" loop via ASM Disk Header Metadata. During Cluster Synchronization Services (CSSD) bootstrap, it skips the ASM instance layer. It scans disk discovery paths using raw OS commands (/dev/nvme* or multipath devices).

The CSSD process reads the physical block headers of the disks using integrated logic (similar to how kfed extracts header information). The metadata fields explicitly specify the precise physical offset and sectors where the Voting Files reside. This allows CSSD to read/write to the Voting Disks and establish cluster quorum before launching the ASM instance.

Q2: What architecture replaces InfiniBand on ExaCC X10M for the private interconnect, and how does it prevent Cache Fusion latency spikes?

Answer: ExaCC X10M utilizes RoCE (RDMA over Converged Ethernet) on 100Gbps network interfaces. It eliminates TCP/IP stack overhead by executing Remote Direct Memory Access (RDMA) directly over hardware. When a Global Cache Transfer occurs, the LMS background process leverages RDMA to read data blocks straight from a remote instance’s buffer cache into the requesting instance's SGA. This maintains microsecond-level latency, rendering network bandwidth a non-bottleneck for Cache Fusion.

Q3: During an ExaCC VM Cluster patching event, how do you verify and execute a rolling upgrade across Grid Infrastructure and Database layers?

Answer: Patching on ExaCC is managed via the OCI Console / dbaascli toolsets.

Run pre-checks using Opatch and cloud tooling to verify inventory consistency across nodes via opatch lsinventory -all_nodes.

Execute the Grid Infrastructure patch first in a rolling manner, taking down one node at a time. The remaining nodes absorb the workload via SCAN and VIP failover.

Once Grid Infrastructure is successfully patched, the RDBMS home is patched rolling, sequentially stopping instances via srvctl stop database -d <db_name> -n <node> to minimize downtime.

Q4: If an ASM Disk Group becomes dismounted on only one node of an ExaCC RAC cluster, what are your immediate troubleshooting steps?

Answer:

Check if the diskgroup status shows DISMOUNTED on that specific node while remaining MOUNTED on others by querying GV$ASM_DISKGROUP.

Inspect the local ASM alert log (alert_+ASM1.log) for error sequences like ORA-15032 or ORA-15080 (I/O error to metadata block).

Validate OS-level block device access and permissions for /dev/nvme* devices. Since ExaCC uses virtual disks mapped from Exadata Storage cells, check cell connectivity from that compute node via cellcli or by reviewing cell server alerts to see if a specific storage network path dropped.

Part 3: Architecture Challenges & Troubleshooting Logs

Challenge 1: Node Eviction / Split-Brain due to RoCE Network Interconnect Hiccups

The Issue: Heavy network congestion or transient switch failures trigger missing heartbeats between cluster nodes.

The Log Indicators:

Grid Infrastructure Alert Log (alert.log):

text

2026-09-29T14:10:05.123+00:00
CRS-1605: CSSD detected a network heartbeat failure with node exacc-node2.
CRS-1612: Network heartbeat to cluster node exacc-node2 missed 50% of the timeout.
CRS-1632: Node exacc-node2 is being evicted from the cluster.

CSSD Daemon Log (ocssd.log):

text

clscrimSend: Failed to send message via RoCE interconnect to node 2 [status: 114].
clssnmvWorkerThread: Evicting node 2; reason: missing network heartbeats.

Troubleshooting Resolution:

Collect automated diagnostic logs via diagcollection.pl.
Check node health and interface metrics using ping -6 (ExaCC internal networks frequently use IPv6) or traceroute over the RoCE network interfaces (bond1 or equivalent).
Run srvctl status nodeapps and examine the Cluster Health Monitor (CHM) data repository using oclumon dumpnodeview to analyze OS metrics immediately leading up to the eviction.

Challenge 2: ASM Disk Offline / Rebalance Failures due to Exadata Cell Resync Delays

The Issue: An Exadata Grid Disk drops offline due to a flash card component alert on a storage cell. When it recovers, automatic rebalancing stalls due to high operational I/O workload.

The Log Indicators:

ASM Instance Alert Log (alert_+ASM1.log):

text

ORA-15042: ASM disk "0002" is missing from group "DATA"
WARNING: offline disk 2.3905973435 in group DATA
State changed to REBALANCING for diskgroup DATA
SUCCESS: rebalance completed for group DATA (ID 1) with 0 errors (stalled)

Troubleshooting Resolution:

Identify the failing storage cell disk from the compute node:

sql

SELECT path, header_status, mode_status FROM v$asm_disk WHERE status = 'OFFLINE';

Use cellcli on the storage server to verify grid disk connectivity:
bash
cellcli -e "LIST GRIDDISK WHERE status='inactive'"
Safely accelerate the storage sync process by tuning the allocation unit distribution speed using ASM_POWER_LIMIT:
sql
ALTER DISKGROUP DATA REBALANCE POWER 32;

Part 4: Production Test Case

Objective: Simulate a Safe Node Isolation Failure & Automatic Recovery

+------------------------------------------------------------+

|                  TEST CASE SPECIFICATION                   |
+------------------------------------------------------------+

| Test ID:     TC-EXACC-042                                  |
| Component:   Oracle Grid Infrastructure & RAC Database     |
| Layer:       Network Interconnect (RoCE Fencing)           |
+------------------------------------------------------------+

text

               [ SCAN / Public Client Network ]

                     |                 |
            +--------+--------+--------+--------+

            |                                   |
     +------------+                      +------------+

     | EXACC NODE1|                      | EXACC NODE2|
     +------------+                      +------------+

            |                                   |
            +======= [ RoCE Interconnect ] =====+  <-- Disrupted during test

            |         (Heartbeat Channel)       |
            |                                   |
     +------------------------------------------------+

     |       Exadata Storage Cells (Voting Disks)     |
     +------------------------------------------------+

Step-by-Step Test Procedure

bash

# ----------------------------------------------------------------------
# STEP 1: Capture Pre-Failure Healthy State
# ----------------------------------------------------------------------
# Run from Node 1 to ensure all cluster resources are running clean
crsctl check cluster -all
srvctl status database -d PRODDB
asmcmd lsdsk -t -g DATA

# ----------------------------------------------------------------------
# STEP 2: Induce Artificial Interconnect Failure (Simulated Drop)
# ----------------------------------------------------------------------
# For instructional or simulation environments, block private RoCE traffic 
# via target interfaces (e.g., re1 / bond1) using ip tables or routing rules.
# Run on Node 2 as root:
ip link set dev bond1 down

# ----------------------------------------------------------------------
# STEP 3: Monitor Real-Time System Eviction Actions
# ----------------------------------------------------------------------
# Monitor Node 1 alert logs to observe cluster management behavior
tail -f $ORACLE_BASE/diag/crs/$(hostname)/crs/trace/alert.log

# Expected Result: 
# Node 1 and Node 2 notice missing heartbeats. Both attempt to lock 
# the voting disks in ASM. Node 1 secures the majority vote. 
# Node 2 is issued an eviction signal via OPROCD / CSSD.

# ----------------------------------------------------------------------
# STEP 4: Observe Clean Database Client Connection Failover
# ----------------------------------------------------------------------
# Run client verification loop during failover window
sqlplus user/pass@//scan-listener.exacc.local/PROD_SERVICE
# Result: Connections running via Node 2 failover to Node 1 seamlessly via VIP.

# ----------------------------------------------------------------------
# STEP 5: Post-Test Environment Restoration
# ----------------------------------------------------------------------
# Restore network path link status on Node 2
ip link set dev bond1 up

# Verify Node 2 rejoins cluster framework after automatic system reboot
crsctl wait_for_startup
crsctl check cluster

Part 1: Project Detail (Context & Architecture)

Project Description:
Migration and lifecycle management of a mission-critical financial ledger platform onto a Quarter-Rack Exadata Cloud@Customer (ExaCC) X10M system. The objective was to replace aging on-premises infrastructure with a secure, highly scalable, and highly available hybrid cloud platform using AMD EPYC-based database servers and RoCE (RDMA over Converged Ethernet) network fabric.

Infrastructure Details: ExaCC X10M, 2x Database Nodes (Guest VM domain), 3x Exadata Storage Servers.

Storage Architecture (Customer-Managed Scope):

+GRID: High redundancy disk group containing the Oracle Cluster Registry (OCR) and Voting Disks.
+DATAC1: Normal redundancy disk group using Exadata Grid Disks for database files, using Exadata Smart Flash Cache.
+RECOC1: Normal redundancy disk group dedicated to Fast Recovery Area (FRA) and archive logs.

Role Scope: Responsibility was limited to the VM guest layer. This included provisioning and resizing ASM disk groups, managing Grid clusterware services (CRS, CSS, EVMD), applying GI Release Updates (RUs) via OCI Control Plane or CLI, configuring ASM scoped security, and optimizing I/O resource management at the guest level.

Part 2: Technical Challenges & Mitigation

Challenge 1: The "Shared Responsibility" Blindspot

Issue: Because storage cells are managed exclusively by Oracle via the OCI Control Plane, traditional on-premises commands like cellcli cannot be run directly by the customer. When an underlying Exadata storage server experienced a predictive flash drive failure, the guest cluster did not drop the disks automatically, leading to slow I/O out of the affected storage cells.

Mitigation: Configured customized Exachk automated cron workflows and integrated guest OS monitoring with Oracle Enterprise Manager (OEM). Used the OCI CLI at the customer level to query the health of the underlying storage servers through oci db exadata-infrastructure get commands, allowing proactive maintenance requests to Oracle before failures impacted ASM redundancy.

Challenge 2: RoCE Network Packet Interventions

Issue: During peak end-of-month batch processing, Inter-Instance RAC interconnect traffic experienced intermittent packet drops. On ExaCC X10M, the interconnect utilizes high-speed RoCE (RDMA over Converged Ethernet).

Mitigation: The issue was traced to mismatched Maximum Transmission Unit (MTU) sizes at the guest OS level (bond0.2000 or equivalent RoCE subinterfaces). The guest OS was set to 1500 instead of 9000 (Jumbo Frames). Rectified this by executing chdev / modifying ifcfg parameters inside the customer-managed VM domain to ensure alignment with Oracle's 9000 MTU underlying network standard.

Part 3: Troubleshooting Scenarios & Guide

Scenario 1: Eviction of a Database Node due to Clusterware Network Heartbeat Miss

Symptoms: Node 2 unexpectedly reboots. crsd.log and ocssd.log reveal network split-brain isolation and eviction.

Troubleshooting Steps:

Log in to the surviving Node 1. Check cluster status: crsctl check cluster -all
Review the diag/crs/<node>/crs/trace/ocssd.log on both nodes. Search for the word eviction or heartbeat.
Verify the status of the customer-managed private network interfaces using: oifcfg getif
Run a ping test across the RoCE interconnect fabric using specialized options to rule out MTU clipping:
bash
ping -I bond0 -s 8972 <IP_of_Node2_Interconnect>
Resolution: Identified that an aggressive third-party security agent installed in the customer-managed OS space was briefly locking network sockets during a system scan. Whitelists were updated to exclude GI binaries ($GRID_HOME/bin).

Scenario 2: ASM Disk Group Rebalance Hung (ORA-15041: diskgroup space exhausted)

Symptoms: Adding new grid disks via the OCI console fails during the ASM rebalance phase. The rebalance stays at 0% remaining indefinitely.

Troubleshooting Steps:

Query V$ASM_OPERATION to locate the active rebalance process:
sql
SELECT group_number, operation, state, power, actual, SOFAR, EST_MINUTES FROM v$asm_operation;
If SOFAR is static, check the ASM alert log (alert_+ASM1.log) for space allocation imbalances.
Root Cause: In ExaCC, if one grid disk fills up disproportionately before the rebalance can distribute data uniformly, the operation fails with ORA-15041.
Resolution: Temporarily set the ASM rebalance power to a higher value to force aggressive processing, and verified that ASM_POWER_LIMIT was supported by the system's current CPU headroom:
sql
ALTER DISKGROUP DATAC1 REBALANCE POWER 12;

Part 4: Comprehensive Interview Questions & Answers

Q1: What is the exact boundary of customer-managed scope regarding ASM on an Exadata Cloud@Customer X10M deployment?

Answer: In ExaCC X10M, the physical storage cells, physical cell disks, and creation of initial LUNs/Grid Disks are managed by Oracle. The Customer Managed Scope begins within the DomU (Guest VM). The customer has SYSASM rights inside the VM to create, alter, drop, mount, and dismount ASM Disk Groups using the provided Grid Disks. The customer controls parameters like ASM_DISKSTRING, disk group redundancy, ASM rebalance power, and lifecycle tasks like adding or removing disks from a group once Oracle has exposed them to the VM layer.

Q2: On ExaCC X10M, what does the ASM_DISKSTRING parameter point to, and how does it differ from traditional on-prem Exadata?

Answer: On traditional on-premises Exadata, ASM_DISKSTRING points to paths like /dev/exadata/o/*. In an ExaCC environment, virtualization abstraction is handled via KVM/SR-IOV. The diskstring generally points to path configurations mapping to virtualized block devices or network-block targets exposed to the guest VM layer, such as /dev/vdb* or /dev/nvme* matching the paths provisioned by the Exadata infrastructure manager software.

Q3: How do you perform a Grid Infrastructure Rolling Patch Upgrade on an ExaCC X10M cluster while adhering to the customer-managed scope?

Answer: Grid Infrastructure patching on ExaCC is orchestrated through the OCI Console, OCI API, or the dbaascli utility on the VM nodes. As a customer, you do not manually download the patch from My Oracle Support and execute opatchauto natively unless instructed by a specific workaround. Instead, you:

Verify prerequisites by selecting Precheck in the OCI Control Plane.

Choose Roll Patching through the UI or via dbaascli grid patch command. This automates the process of draining sessions, putting nodes in maintenance mode, patching the local $GRID_HOME, and moving to the next node automatically to maintain High Availability.

Q4: If an ASM disk shows an status of UNKNOWN or CANDIDATE in V$ASM_DISK, what does it imply in an ExaCC context, and how do you consume it?

Answer: It means that new storage capacity or a new Grid Disk has been provisioned and allocated to the VM Cluster via the OCI Control Plane, but it has not yet been assigned to an existing ASM disk group. To consume it, you use the SQL ALTER DISKGROUP syntax inside the ASM instance:

sql

ALTER DISKGROUP DATAC1 ADD DISK '/dev/disk_path' NAME DATAC1_NEW_DISK REBALANCE POWER 4;

Use code with caution.

Part 5: Detailed Practical Test Case

Test Case Objective: Scale-Out Storage Capacity by Adding a Grid Disk to +DATAC1

This test case demonstrates how a customer handles storage scale-out within their boundary on ExaCC.

+-------------------------------------------------------------+

|                     OCI Control Plane                       |
|           Scale Storage -> Allocates Grid Disk              |
+------------------------------+------------------------------+
                               |
                               v (Exposed to VM)
+-------------------------------------------------------------+

|                     Customer Guest VM                       |
|  1. Scan New Storage   -->  2. Alter ASM Disk Group         |
|  (udev / lsblk)             (ADD DISK & REBALANCE)          |
+-------------------------------------------------------------+

Step 1: Pre-requisites & Verification

Check the current disk group configuration and ensure there are no active, uncompleted rebalances:

sql

-- Connect as SYSASM
sqlplus / as sysasm

SELECT name, state, type, total_mb, free_mb FROM v$asm_diskgroup WHERE name = 'DATAC1';
SELECT operation, state, power FROM v$asm_operation;

Use code with caution.

Step 2: Identify the Newly Allocated OS Disk Device

After triggering the disk addition via the OCI console, verify the device is visible at the guest Linux level:

bash

# Run as root user on both nodes
lsblk
# Look for the new device name, for example: /dev/nvme3n1

Use code with caution.

Step 3: Add the New Disk to the Target Disk Group

Execute the command from the primary database node within the ASM instance. Set the rebalance power to an optimal level (4 to 8 to minimize application overhead during production hours):

sql

ALTER DISKGROUP DATAC1 ADD DISK '/dev/nvme3n1' NAME DATAC1_EXT_04 REBALANCE POWER 4;

Use code with caution.

Step 4: Monitor the Progress and Success Criteria

Monitor the rebalance until the row returns no rows, confirming the process completed successfully:

sql

SELECT group_number, operation, state, sofar, est_minutes FROM v$asm_operation;

Use code with caution.

Verify the disk status in the group shows as ONLINE:

sql

SELECT header_status, path, name FROM v$asm_disk WHERE disk_number = 'DATAC1_EXT_04';
-- Expected Output for header_status: MEMBER

Part 1: Project Blueprint & End-to-End Migration Strategy

Project Detail (Sample Production Blueprint)

Objective: Migrate and consolidate a mission-critical, high-throughput core banking application from an on-premises, legacy Oracle Exadata X7-2 environment into an isolated ExaCC X10M AMD EPYC-based VM Cluster.

Source Environment: 4-Node RAC, Oracle Database 19c (19.12), On-Premises Exadata X7-2, Total DB size: 35 TB.

Target Environment: 4-Node RAC VM Cluster on ExaCC X10M running Oracle Database 19c (19.22+), integrated with the Oracle Autonomous Recovery Service.

SLA Requirement: Near-zero downtime (Maximum acceptable business interruption: < 15 minutes).

                      [ MIGRATION WORKFLOW ARCS ]
+-------------------------+               +-------------------------------+

|   On-Premises Source    |  Data Guard   |         Target ExaCC          |
|      Exadata X7-2       | ------------> |             X10M              |
|  (Oracle 19c / 19.12)   | (Physical/Net)|      (Oracle 19c / 19.22+)    |
+-------------------------+               +-------------------------------+
                                                          |
                                                          | Local Refresh
                                                          v
                                          +-------------------------------+

                                          |          UAT / Stage          |
                                          |    (Sparse Snapshots / OCI)   |
                                          +-------------------------------+

Step-by-Step Migration and Rollback Playbook

Preparation: Provision a custom database software image via the OCI console incorporating the target minor version. Set up secure network routing between on-premises data centers and the ExaCC infrastructure control planes.

Initial Seeding: Configure Oracle Data Guard from the on-premises source to the ExaCC target using an RMAN duplicate via an active database network stream.

Synchronization: Maintain synchronization until lag matches zero. Convert the target standby to a snapshot standby for performance validation and application smoke testing.

Switchover Execution: Revert the target back to a physical standby. Execute a controlled Data Guard switchover via dgmgrl. Update localized DNS strings or connection strings using connection managers.

Rollback Plan: In the event of catastrophic data anomalies post-cutover, data synchronization is maintained in reverse to the old on-premises cluster. A clean switchover command can safely route traffic back to the source.

Testing and Validation Strategy

Data Validation Check: Generate system-wide cryptographic checksums (DBMS_ROWID slices or custom hash comparisons across financial ledgers) before and after cutover to ensure row-level consistency.

Performance Baselines: Utilize automated AWR Compare Period Reports to evaluate the transaction profile under simulated stress, comparing the historical X7 workload directly against the new X10M compute fabric.

Part 2: Core Engineering Interview Questions & Answers

Section A: Database Creation & Cloning

Q1: How do you provision a Container Database (CDB) on ExaCC X10M using infrastructure automation tools, and how does it differ from traditional systems?

Answer: On ExaCC X10M, database provisioning is driven by the Cloud Control Plane via the OCI console, OCI CLI, or Terraform. Oracle Cloud Operations manages the hypervisor and base system layers, while you call automated orchestration workflows to initialize database homes and container roots. Unlike generic architectures, this method integrates seamlessly with Exadata-aware systems, automatically placing files onto high-performance ASM disk groups while applying optimal DB_BLOCK_SIZE parameters for Hybrid Columnar Compression (HCC).

Q2: Explain the mechanics of Space-Efficient Snapshot Cloning within an ExaCC X10M environment.

Answer: ExaCC uses the underlying storage fabric to orchestrate sparse snapshot clones. You first create a read-only Snapshot Master deployment from your source database. When establishing target child instances, ExaCC relies on the SPARSE ASM disk group. The clone links straight to the master's block addresses; all subsequent runtime modifications are isolated exclusively inside the SPARSE group. This enables swift provisioning of multi-terabyte QA and test databases with minimal storage consumption.

Section B: Refresh & Upgrade

Q3: Walk through the exact architectural process to execute an in-place Grid Infrastructure (GI) update on an ExaCC X10M VM Cluster.

Answer:

Navigate to the OCI Console, locate Exadata VM Clusters, select the specific cluster, and click Updates (GI).

Run a Precheck to uncover prerequisite gaps or missing OS-level components.

Choose Apply Grid Infrastructure update. The automated control plane performs an online, rolling update across the cluster nodes, keeping the database engine online while upgrading the underlying high-performance clusterware components node-by-node.

Q4: How do you leverage RMAN duplicate or transportable tablespaces for a cross-platform database refresh onto an ExaCC architecture?

Answer: If refreshing from an on-premises platform with a different endian format, use Cross-Platform Transportable Tablespaces (XTTS) with RMAN incremental backups to minimize down times. For same-endian environments (Linux x86-64), execute an online network-based RMAN DUPLICATE TARGET DATABASE TO ... FROM ACTIVE DATABASE command, optimizing data delivery with section sizes and parallel channel allocations to saturate the high-bandwidth ExaCC network pipes.

Section C: Migration

Q5: What migration methodologies provide the lowest downtime when moving a 35 TB legacy database onto ExaCC X10M?

Answer: The primary approaches depend on acceptable down times and application complexity:

Oracle Data Guard / Active Data Guard: Best for minimizing risk; handles seed synchronization smoothly and reduces downtime to a simple, deterministic switchover window.

Oracle GoldenGate: Excellent when zero downtime is requested or migration involves a cross-version upgrade (e.g., 11g to 19c), allowing ongoing real-time replication.

Data Pump with Transportable Tablespaces (XTTS): Suitable when re-architecting the storage block layouts or prefixing compression modifications on the target.

Part 3: Real-World Challenges & Troubleshooting

Operational Scenario	Root Cause	Remediation & Fix
Grid Infrastructure Update Failures	Misconfigured local custom network parameters or missing security certificates on the local Control Plane Server (CPS) preventing OCI handshakes.	Download the CPS Offline Diagnostic Report via OCI. Clear out corrupted wallet items and restart local cloud agent daemons to restore communication.
ExaCC Sparse Clone Growth	Excessive index rebuilding or structural maintenance routines inside the clone database, saturating the `SPARSE` disk group storage.	Isolate and restrict heavy batch processing jobs on sparse dev-test platforms. Monitor block deviations using the `V$ASM_DISK` and `V$ASM_FILE` system views.
Data Guard Network Throttling	Misaligned MTU values or improper TCP buffer allocations traversing network firewalls between on-premises source environments and ExaCC.	Standardize jumbo frames (MTU 9000) across all connection pathways. Fine-tune Oracle Net layer profiles by boosting `SDU` values to `65536` in network descriptor files.

Part 4: Technical Test Case Matrix

Test Case 1: Snapshot Clone Validation

Description: Verify the integrity and storage consumption of a space-efficient snapshot clone.

Execution Command:

sql

-- Check ASM disk group metrics before and after clone creation
SELECT name, total_mb, free_mb, usable_file_mb FROM v$asm_diskgroup WHERE name = 'SPARSE';

Use code with caution.

Expected Outcome: The newly provisioned target database instance opens successfully in READ WRITE mode, while the storage consumed inside the SPARSE disk group matches only the block-level changes introduced post-clone execution.

Test Case 2: In-Place Minor Version Database Upgrade

Description: Apply an out-of-place or in-place minor database release update via OCI APIs.

Execution Command:

bash

# Execute update via OCI CLI interface targeting the specific DB Home
oci db database update --database-id <db_ocid> --db-home-id <new_db_home_ocid>

Use code with caution.

Expected Outcome: The automated workflow successfully patches the database homes across all VM nodes sequentially, ensuring cluster service continuity throughout the process.

Project Detail: ExaCC Multitenant Migration & Consolidation

When asked about your project experience, you can use this structured blueprint.

Project Title: High-Density Database Consolidation on Exadata Cloud@Customer.

Objective: Consolidate 45 legacy standalone Oracle 19c databases from on-premises commodity hardware into a 4-node ExaCC Quarter Rack running Oracle 19c Multitenant (2 CDBs for Production, 4 PDBs per CDB initially, scaling up based on workload).

Architecture:

Grid Infrastructure (GI) managed via OCI Control Plane.
Databases deployed as Multi-Node Oracle RAC.
Custom Database Services mapped to specific PDBs to isolate application workloads and manage Connection Load Balancing (CLB) and Runtime Load Balancing (RLB).
Storage managed via Exadata Storage Servers (GRID disks partitioned into DATA and RECO diskgroups).

❓ Top Interview Questions & Answers

Q1: How do you create and manage Database Services for a PDB in an ExaCC RAC environment?

Answer: On ExaCC, you should never use standard SQL (ALTER SYSTEM SET SERVICE_NAMES...) to manage RAC services. Instead, use the OCI Console / OCI CLI (preferred for cloud automation) or srvctl at the OCI OS level.
To create a service via CLI:

bash

srvctl add service -db <cdb_unique_name> -service <service_name> -pdb <pdb_name> -preferred <inst1,inst2> -available <inst3,inst4> -tafpolicy TAF_POLICY
srvctl start service -db <cdb_unique_name> -service <service_name>

Use code with caution.

Q2: How does the OCI Control Plane interact with local CDB/PDB management on ExaCC?

Answer: The OCI Control Plane communicates with the ExaCC Virtual Machines via the Exadata Infrastructure Management Agent (ovmagent). When you trigger a PDB creation, cloning, or deletion from the OCI Console, the cloud control plane sends REST APIs to the agent, which executes the corresponding dbcli or Oracle SQL commands locally. Manual interventions via SQL*Plus that bypass the control plane can cause state drift, making the OCI Console display inaccurate database states.

Q3: How do you restrict resource consumption between PDBs within the same ExaCC CDB?

Answer: Resource management is handled at two levels:

Instance Caging / DB Resource Manager (DBRM): Using CDB Resource Plans to limit CPU utilization per PDB using SHARES and UTILIZATION_LIMIT.

Exadata I/O Resource Manager (IORM): Allocating I/O shares or limits to specific PDBs directly at the Exadata storage cells using inter-database and intra-database plans.

⚡ Challenges & Troubleshooting

1. Challenge: OCI Console "Sync" Issues (State Drift)

Symptom: You dropped or modified a PDB/Service using SQL*Plus, and now the OCI Cloud Console shows the PDB as "Failed," "Updating," or out-of-sync.

Troubleshooting:

Check the local agent logs: /var/opt/oracle/log/dbaasapi/dbaasapi.log.
Use the local cloud management tool to check status: dbcli list-databases.
Fix: Avoid manual drops. If drift occurs, you may need to run dbcli update-database or contact Oracle Support to sync the OCI backend metadata repository.

2. Challenge: PDB Connection Failures via SCAN

Symptom: Applications get ORA-12514: TNS:listener does not currently know of service requested in connect descriptor when connecting to a newly created PDB.

Troubleshooting:

Check if the PDB service is registered with the local listener and SCAN listener: lsnrctl status and srvctl status service -db <cdb>.
Ensure the PDB is actually in READ WRITE open mode across all RAC nodes: SELECT name, open_mode FROM v$pdbs;.
Verify the parameter local_listener and remote_listener point correctly to the local VVIP and SCAN VIPs.

🧪 Test Case: Creating, Isolating, and Testing a New PDB Service

Scenario

An application team requires a new PDB (PDB_FINANCE) with a dedicated service (SRV_FIN_PROD) that must only run on Nodes 1 and 2 of a 4-node ExaCC cluster to keep its workload isolated.

Step 1: Create the PDB (via OCI Console or dbcli)

bash

# Using local dbcli if bypassing console for rapid testing
dbcli create-pdb -dbId <cdb_id> -pdbName PDB_FINANCE

Use code with caution.

Verify it is open on all nodes:

sql

SQL> ALTER PLUGGABLE DATABASE PDB_FINANCE OPEN ALL;

Use code with caution.

Step 2: Configure Dedicated Service via srvctl

Create the service to prefer Node 1 (exadb1) and Node 2 (exadb2).

bash

srvctl add service -db cdb_exacc -service SRV_FIN_PROD -pdb PDB_FINANCE -preferred cdb_exacc1,cdb_exacc2 -available cdb_exacc3,cdb_exacc4 -clbgoal LONG -rlbgoal SERVICE_TIME
srvctl start service -db cdb_exacc -service SRV_FIN_PROD

Use code with caution.

Step 3: Validate Routing & Load Balancing (The Test)

From a client machine, configure the tnsnames.ora using the ExaCC Client Network SCAN address:

text

FIN_PROD_SCAN =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ://clientnet.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = SRV_FIN_://clientnet.com)
    )
  )

Use code with caution.

Run a validation loop to verify connections only hit instances 1 and 2:

bash

for i in {1..10}; do 
  sqlplus -S app_user/password@FIN_PROD_SCAN <<EOF
  SELECT instance_name FROM v\$instance;
  EXIT;
EOF
done

Use code with caution.

Expected Outcome: The output loop should show alternating results between cdb_exacc1 and cdb_exacc2, never hitting nodes 3 or 4.

Q1: What makes administering a RAC database on ExaCC different from a traditional on-premises Exadata setup?

Answer: On traditional Exadata, the DBA controls the entire stack (OS, firmware, hardware). On ExaCC, responsibility is split. Oracle manages the physical infrastructure, hypervisor (KVM), and Exadata Storage Servers (via RoCE/InfiniBand networks). The customer DBA has root and oracle access only inside the VM Guest Domain (DomU). Administrative tasks like provisioning, scaling OCPUs, and initial patch applications are abstracted through the OCI Console / OCI CLI, rather than pure manual command-line scripts.

Q2: How does the Smart Scan feature change your performance tuning strategy on ExaCC?

Answer: Smart Scan offloads intensive SQL operations (like full table scans and joins) directly from the compute nodes to the Cell Storage servers. When tuning, instead of focusing solely on reducing disk I/O, you look at AWR reports to verify if queries are actually utilizing Smart Scan. You track metrics like cell physical IO bytes eligible for smart scan versus cell physical IO bytes returned by smart scan. If a query isn't offloading, you check for inhibitory factors such as scalar subqueries in the WHERE clause, mismatches in row piece sizes, or disabled cell_offload_processing.

Q3: How do you handle RAC node evictions differently on an ExaCC environment?

Answer: While traditional troubleshooting requires checking Clusterware alert logs (crsd.log, ocssd.log), on ExaCC, a node eviction can stem from network drops or hypervisor glitches managed by Oracle. DBAs analyze the guest VM logs first. If the eviction is tied to a shared storage heartbeat timeout or a physical network issue outside DomU, you must look at Oracle Auto Service Request (ASR) logs or immediately file a Severity 1 Service Request (SR) with Oracle Cloud Support since they own the underlying Dom0 (hypervisor layer) and physical switches.

2. Operational Challenges

Co-managed Maintenance Windows: DBAs must orchestrate GI/Database patching schedules around Oracle's infrastructure maintenance windows (Grid Infrastructure must be kept compatible with the underlying Storage Cell software versions maintained by Oracle).

Limited Root Control over Infrastructure: Because you do not have access to Dom0 or physical cell servers, you cannot run direct utilities like CellCLI natively across storage layers unless abstracting via cloud tools or reviewing metrics pulled via V$CELL views.

Grid Network Isolation: Troubleshooting gc buffer busy wait or network heartbeats requires distinct handling because the private interconnect runs over virtualized SR-IOV (Single Root I/O Virtualization) interfaces mapping back to physical RoCE fabrics.

3. Troubleshooting Matrix

Issue	Root Cause Analysis	Troubleshooting & Resolution
High `gc buffer busy acquire`	Hot blocks being requested concurrently across multiple RAC instances over the private interconnect.	Query `V$SESSION` and `V$SEGMENT_STATISTICS` to identify the hot object. Implement application partitioning, hash partitioning on tables, or increase sequence caches.
Storage Offload Failure	The query did not trigger Smart Scan, leading to high CPU on database nodes.	Check `V$SQL_PLAN` to verify if `TABLE ACCESS FULL` is present. Ensure `cell_offload_processing = TRUE`. Check for un-indexed functions or data type mismatches that inhibit offloading.
ExaCC VM Local Disk Full	Log/Trace pile-up inside the `/u01` mount point on a DomU node.	Run `adrci` to purge old diagnostic trace files. Check and rotate audit trails (`DB_AUDIT_TRAIL`) or grid infrastructure trace logs regularly.

4. Enterprise Project Scenario

Project Title: Legacy Core Banking Migration to Exadata Cloud@Customer (ExaCC)

Objective: Migrate a 40TB mission-critical OLTP/Batch database from an aging on-premises AIX platform to a 4-node ExaCC X9M RAC Cluster to clear batch bottlenecks and ensure 99.99% availability.

Implementation Strategy: Used Oracle Zero Downtime Migration (ZDM) utilizing physical online migration via fallback Data Guard. Data was instantiated using RMAN backup sets directly sent to OCI Object Storage, synced via Active Data Guard over a dedicated FastConnect pipeline, and cut over during a 15-minute maintenance window.

Outcome: Batch execution runtimes dropped by 65% due to Exadata Smart Flash Log write performance and Storage Index offloading.

5. Validation Test Case

Use this systematic test case to validate the High Availability (HA) failover and load balancing performance of the ExaCC RAC database after infrastructure upgrades.

Test Case ID: TC-EXACC-HA-004

Test Objective: Validate seamless client application failover and Single Client Access Name (SCAN) load balancing during an unexpected single-node crash.

Prerequisites:

A 3-node ExaCC RAC database (exadb1, exadb2, exadb3).

Client application using an Application Continuity (AC) enabled connection string:

text

(DESCRIPTION=(CONNECT_TIMEOUT=15)(RETRY_COUNT=20)(RETRY_DELAY=3)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=exa-scan.cloud.local)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=app_srvc_ac.cloud.local)))

Use code with caution.

Swingbench running an active OLTP workload simulating 100 concurrent transactions.

Execution Steps:

Verify Baseline State: Ensure sessions are evenly distributed across all 3 nodes.

sql

SELECT inst_id, count(*) FROM gv$session WHERE username='APP_USER' GROUP BY inst_id;

Use code with caution.

Simulate Hard Failure: Log into node 3 as root and abruptly abort the instance using an immediate cluster command to simulate a hardware panic:

bash

crsctl stop crs -f

Use code with caution.

Monitor Active Sessions: Watch the application console and database topology concurrently.

Expected Results:

The application should not throw an ORA-03113 or connection lost error.

Application Continuity captures the in-flight transactions on node 3, masks the failure, and replays them on node 1 or node 2.

gv$session shows all connections safely redistributed away from inst_id = 3 within seconds.

Project Profile: Dynamic Compute Resource Scaling on ExaCC

Platform: Oracle Exadata Cloud at Customer (ExaCC) X9M / X10M

Architecture: Multi-VM Cluster Configuration (Grid Infrastructure + Oracle RAC)

Objective: Implement online CPU vertical scaling (OCPU/ECPU) and horizontal node expansion to handle unpredictable financial month-end processing spikes without database downtime. [1, 2, 3]

Interview Questions & Answers

Q1: How do you perform CPU scaling in an ExaCC VM Cluster, and is it disruptive?

Answer: CPU scaling on ExaCC can be done online without restarting the VMs or the Oracle instances. It can be initiated manually via the OCI Console (by updating the OCPU/ECPU count per VM) or programmatically via OCI CLI / Terraform scripts. The underlying OCI Control Plane dynamically updates the hypervisor allocations (cgroup configurations) inside the Exadata Database Nodes. [1, 2]

Q2: What is the minimum scaling increment, and what is a prerequisite for scaling up?

Answer: The minimum increment depends on the platform: 1 OCPU for traditional shapes or 4 ECPUs for newer X11M/X10M architectures. A critical prerequisite is ensuring that the Exadata Infrastructure has enough unallocated physical CPU cores in the pool. Additionally, the parameters common_dcs_agent_bindHost and common_dcs_agent_port must be correctly configured in cprops.ini for the OCI management agent (dcsagent) to sync successfully. [1, 2]

Q3: What parameters inside the Oracle Database need to change when scaling CPUs?

Answer: When scaling CPUs up or down, Oracle's CPU_COUNT database initialization parameter updates automatically if set to default. However, you must verify that RESOURCE_MANAGER_PLAN is enabled so the database can redistribute workloads according to the new compute capacity. If PGA_AGGREGATE_TARGET and SGA_TARGET are reliant on CPU-bound calculations, they may need manual adjustments to prevent spilling to the temporary tablespace during heavy workloads. [1]

Production Challenges

DCS Agent Sync Hangs: During a scale-up request, the OCI Control Plane relies on the local dcsagent running inside the guest VM. If the agent is unresponsive or blocked by network security rules, the operation hangs in an UPDATING state. [1]

Clusterware Eviction on Node Additions: When scaling horizontally by adding a new compute node to the VM Cluster, high network latency on the private interconnect (InfiniBand/RoCE) can cause transient split-brain scenarios, leading to node eviction of existing nodes during cluster reconfiguration. [1]

Local Storage Depletion (/u01): Scaling operations or adding new nodes generate substantial logging in /opt/oracle/dcs/log/ and AHF (Autonomous Health Framework) traces. If /u01 or /var hits 100% capacity, the automated scaling workflow fails immediately. [1]

Troubleshooting Guide

Scenario: CPU Scaling Operation Fails or Hangs in OCI Console

Step 1: Check the Agent Status. Log into the VM node as root and check the OCI management service:

bash

systemctl status dbcsagent   # For newer ExaCC generations
# or
initctl status initdcsagent

Use code with caution.

Step 2: Inspect the Log Files. Examine the core framework logs for exact step execution details:

bash

tail -n 100 /opt/oracle/dcs/log/dcsagent.log

Use code with caution.

Step 3: Fix Permission Issues. If the Autonomous Health Framework (AHF) or grid traces are locked, the scaling program might fail to collect prerequisites:

bash

chown -R grid:oinstall /opt/oracle/dcs/log/

Use code with caution.

Step 4: Verify Local Storage Availability. Ensure there is enough space on the local volume: [1]

bash

df -h /u01

Use code with caution.

Test Case: Online Compute Scale-Up Verification

Test Case ID	TC-EXACC-SCALE-01
Test Scenario	Validate online vertical scaling of OCPUs in a multi-VM RAC Cluster under synthetic load.
Prerequisites	OCI CLI access configured; `dbcsagent` active on all nodes; available unallocated pool on Exadata Infrastructure.
Execution Steps	1. Log into the database and check current allocation: `SHOW PARAMETER cpu_count;` 2. Execute OCI CLI scale command: `oci db vm-cluster update --vm-cluster-id <id> --ocpu-count <new_count>` 3. Monitor the OCI console status transition from `UPDATING` to `AVAILABLE`. 4. Re-run `SHOW PARAMETER cpu_count;` on all RAC instances.
Expected Result	Core count increases dynamically without database instance crashes, connection drops, or node reboots.

Part 1: Project Details & Context

Project Title: Automated ExaCC Cloud Infrastructure Maintenance Lifecycle Management.

Objective: End-to-end automation of infrastructure patching alerts (GI, Dom0, Storage Cells), integration with an internal IT service management (ITSM) system like ServiceNow, execution of pre-maintenance readiness health checks, application of rolling patch sequences with minimal disruption, and automated post-maintenance smoke testing.

Scope: ExaCC X8/X9M/X10M architectures across multiple production and non-production clusters.

Part 2: Interview Questions & Answers

Q1: How does ExaCC notify customers of upcoming planned infrastructure maintenance, and how do you track it?

Answer: Oracle provides notifications via multiple streams:

OCI Console & Announcements: Displayed under OCI Announcements and the Exadata Infrastructure Details page.

OCI Events Service & Notification Service (ONS): Triggers when an event type like com.oraclecloud.infrastructure.maintenance.scheduled occurs.

OCI CLI / REST APIs: The query oci db autonomous-exadata-infrastructure list-maintenance-runs or equivalent commands for co-managed ExaCC fetch upcoming maintenance schedules.

Q2: What is the customer's role when Oracle initiates planned maintenance on an ExaCC rack?

Answer: ExaCC works under a co-managed model. Oracle schedules and runs the infrastructure patching (Dom0, Storage Cells, InfiniBand/RoCE switches). The customer is responsible for:

Acknowledging or rescheduling the maintenance window via the OCI Console if it conflicts with peak business hours.

Running pre-patching health checks (like exachck).

Ensuring that the application connections support Transparent Application Failover (TAF) or Application Continuity (AC) so that a rolling node reboot does not drop live traffic.

Q3: Oracle scheduled an infrastructure patch window, but you notice a major reporting batch window overlaps it. How do you handle this programmatically?

Answer: I would write a script leveraging the OCI CLI to find the scheduled maintenanceRunId. Using the oci db maintenance-run update command with the --time-scheduled parameter, I can update the start window to an approved future date/time within the allowed Oracle patching buffer window (usually up to 30 days from the original schedule).

Part 3: Operational Challenges

Drain Interruption Failures: During rolling node reboots (e.g., hypervisor/Dom0 updates), the instance on that node must cleanly drain and failover. If active, un-draining sessions or non-RAC-optimized legacy applications hang, it stalls Oracle’s automated automation framework, leading to an extended maintenance window or ungraceful termination.

Mismatched Component Software Versions: If the customer database components (e.g., Grid Infrastructure or DB Home) are running highly legacy versions, they might conflict with the newly patched underlying storage cell or hypervisor software versions updated by Oracle.

ASR (Auto Service Request) Alerts: During planned storage cell rolling reboots, false-positive alerts can be triggered via Oracle Auto Service Request (ASR) if maintenance mode isn't correctly isolated, flooding your internal alerting system.

Part 4: Troubleshooting Playbook

Symptom: Maintenance status shows FAILED or REQUIRES_ATTENTION in the OCI Console.

Step 1: Check the OCI Activity Log. Run the OCI CLI command:

bash

oci db maintenance-run get --maintenance-run-id <run_id>

Use code with caution.

Step 2: Inspect Local Cluster Logs. Log into the VM nodes and check the Clusterware alert log (alert.log of GI) to see if node eviction or resource timeouts occurred:

bash

adrci -exec "set homepath diag/crs/$(hostname)/crs; show alert"

Use code with caution.

Step 3: Check Storage Cell Status. If the cell patch hangs, verify storage cell accessibility from the remaining nodes using cellcli:

bash

cellcli -e "list cell detail"
cellcli -e "list griddisk attributes name, status"

Use code with caution.

Step 4: Engage Oracle Cloud Operations. Since infrastructure maintenance is executed by Oracle automation, if the hang happens at the Dom0/hypervisor layer, file a Severity 1 Service Request (SR) immediately via the OCI Support Portal, citing the specific Maintenance Run ID.

Part 5: Comprehensive Test Case

This test case validates that an OCI Maintenance notification is correctly captured, parsed, and prepared for processing.

Test Case ID: TC-EXACC-MAINT-001

Test Title: Automated Interception and Validation of ExaCC Infrastructure Planned Maintenance Notification.

Pre-conditions: OCI CLI is configured with the appropriate IAM policies to read infrastructure maintenance runs. An active, mock, or actual planned maintenance schedule exists in the OCI tenancy.

Execution Steps:

Simulate or wait for an OCI announcement email/event payload.
Execute the OCI CLI script to pull current maintenance logs for the specific Exadata Infrastructure component.
Verify that the script successfully extracts the maintenanceRunId, the planned start time (timeScheduled), and target version.

Expected Result: The script returns a JSON payload showing a status of SCHEDULED, extracts the date/time parameters, and pipes them cleanly into the internal change control system.

Test Script Example (Bash/CLI)

bash

#!/bin/bash
# Fetch upcoming ExaCC Infrastructure Maintenance runs that are scheduled
COMPARTMENT_OCID="ocid1.compartment.oc1..example"

echo "Checking for planned ExaCC maintenance..."
MAINT_RUNS=$(oci db maintenance-run list \
    --compartment-id "$COMPARTMENT_OCID" \
    --lifecycle-state "SCHEDULED" \
    --query "data[?targetResourceType=='ExadataInfrastructure'].{ID:id, Target:targetResourceId, ScheduledTime:timeScheduled, Action:action}" \
    --output json)

if [ "$MAINT_RUNS" == "[]" ] || [ -z "$MAINT_RUNS" ]; then
    echo "SUCCESS: No upcoming planned infrastructure maintenance runs found."
    exit 0
else
    echo "ATTENTION: Upcoming maintenance run detected!"
    echo "$MAINT_RUNS" | jq .
    # Further automation logic here: trigger pre-patching health checks (exachk)
    exit 1
fi

Core Strategy for ExaCC Infrastructure Maintenance

Coordinating Exadata Cloud@Customer (ExaCC) infrastructure maintenance requires a distinct separation of responsibilities between Oracle (Infrastructure Layer) and the Customer (VM/Database Layer). While Oracle owns and updates physical hardware, cell storage, network switches, and the hypervisor, the customer owns the scheduling, pre-checks, and application failover validation.

Part 1: Interview Questions & Answers

Q1: How do you coordinate an Oracle-managed infrastructure patch on ExaCC? What is your role vs. Oracle’s role?

Answer: ExaCC infrastructure maintenance (covering storage servers, database nodes/hypervisors, and RoCE/InfiniBand switches) is driven by Oracle but coordinated with the customer through the OCI Console.

Oracle’s Role: Proposes maintenance windows, executes firmware updates, rolls updates across storage cells, and updates the physical hypervisors one node at a time to maintain high availability.
Customer's Role: Acknowledges or reschedules the window in OCI, runs pre-patch validation scripts (like exachk), ensures cluster draining via srvctl, manages database capacity during single-node operations, and validates application connectivity after the patch.

Q2: Oracle is patching the Dom0 (hypervisor) layer on a multi-node ExaCC rack. How do you ensure zero downtime for the databases?

Answer: ExaCC applies updates rolling fashion across database nodes. To ensure zero downtime:

Verify that Oracle RAC is healthy across all nodes and services are configured for failover.
Use srvctl relocate service to gracefully drain sessions off the node scheduled for maintenance before Oracle takes it down.
Ensure the surviving nodes have sufficient CPU and memory capacity to handle the consolidated workload.
Rely on Application Continuity (AC) or Fast Application Notification (FAN) to replay uncommitted transactions seamlessly on surviving instances.

Q3: What critical step must you take before a Grid Infrastructure (GI) or OS update managed through the OCI console?

Answer: You must always run a Precheck operation via the OCI console or OCI API. This ensures that the GI home and /u01 directory have adequate space (at least 15GB free space for GI and /u02 for DB homes) and verifies that there are no underlying network or cluster split-brain risks.

Part 2: Real-World Maintenance Challenges

Capacity Squeezes During Rolling Patches: During a hypervisor or DB node reboot, your 4-node cluster becomes a 3-node cluster. If your memory allocation (SGA/PGA) on the surviving nodes exceeds physical limits when workloads shift, nodes will crash due to OOM (Out of Memory) conditions.

Oracle Grid Infrastructure Local Evictions: If local custom agents or unapproved third-party software are deployed inside the client VMs, they can cause CPU spikes or conflict with cluster heartbeat signals during rolling updates, causing abrupt node evictions.

Mismatched Software and Firmware Tiers: If the client-managed DB/GI layers are heavily outdated compared to the new Oracle-managed infrastructure firmware, query offloading (Smart Scan) features can degrade or fail completely.

Part 3: Troubleshooting Playbook

Issue / Symptom	Root Cause	Resolution Strategy
OCI Precheck Fails with space errors	Insufficient space in `/u01` or `/u02` directories.	Clear old trace files, diagnostic logs, and core dumps using `adrci`. Ensure `15GB+` free space.
Grid Infrastructure fails to start post-maintenance	Custom configurations or modified files (`/etc/hosts`, `sshd_config`) overwritten or blocked by the new image.	Roll back manual VM customizations. Coordinate with Oracle Support via the automated ASR / SR system to audit custom profile flags.
Smart Scans Stop Working post-storage patch	Storage cell IP communication blockages or modified `cellip.ora` file.	Verify cell accessibility using `cellcli` or audit `cat /etc/oracle/cell/network-config/cellip.ora`.

Part 4: Project Profile: ExaCC Infrastructure Modernization

Project Title: High-Availability Infrastructure Maintenance & Automated Patching Lifecycle on ExaCC.

Project Description: Managed the end-to-end coordination, pre-validation, and execution of quarterly Oracle Exadata Cloud@Customer full-stack infrastructure updates (including Storage Cells, InfiniBand/RoCE network infrastructure, and Dom0 hypervisors) across production environments.

Key Achievements: Achieved 100% database availability during 4 consecutive rolling infrastructure patch cycles, successfully draining and balancing active connections for over 50 mission-critical pluggable databases (PDBs).

Part 5: Production Maintenance Validation Test Case

TEST CASE ID: TC-EXACC-MAINT-042
TITLE: Validation of Rolling DB Node Hypervisor Patching with Zero Application Downtime
PRE-CONDITIONS:
  1. Oracle ExaCC 2-Node or 4-Node Cluster is fully operational.
  2. TAFT/Application Continuity is enabled on the client side.
  3. Precheck status shows "Passed" in the OCI Console.

EXECUTION STEPS:
  1. Trigger the OCI Maintenance window or permit Oracle to initiate the scheduled patch rolling cycle.
  2. Continuously run cluster status command: 'crsctl stat res -t'
  3. Manually relocate application services away from Node 1:
     $ srvctl relocate service -db <db_name> -service <svc_name> -currentnode <node1> -targetnode <node2>
  4. Monitor application transaction throughput and error rates via APM logs during Node 1 reboot.
  5. Once Node 1 returns online and up-to-date, verify cluster synchronization.
  6. Repeat steps for surviving nodes.

EXPECTED RESULT:
  - Oracle infrastructure updates successfully.
  - No active application transactions drop or experience HTTP 500 errors.
  - Surviving nodes stay under 80% CPU/Memory usage throughout the rolling transition.

Part 1: Project Details & Context (To Set Your Interview Narrative)

Project Title: Enterprise Core Database Migration and Observability Framework on ExaCC
Role: Senior Infrastructure & Lead Oracle Database Cloud Architect
Objective: Migrated and managed mission-critical, high-throughput OLTP and heavy Analytics workloads from on-premises legacy hardware to an ExaCC X8M/X9M environment. Implemented end-to-end full-stack observability across VM Clusters, Databases (CDB/PDB), Storage Cells (Exadata Storage Software), and cluster services.

Architecture Stack Covered:

Compute Tier: OCI VM Cluster running Grid Infrastructure (GI) and Oracle RAC.

Storage Tier: Exadata Cell Servers (ILOM, CellCLI, Grid disks, Flash Cache).

Network Tier: RoCE (RDMA over Converged Ethernet) / InfiniBand fabrics.

Control Plane: Oracle Cloud Infrastructure (OCI) Agent (dbcsagent) bridging customer-managed VMs and Oracle's Cloud Control Plane.

Part 2: Interview Questions & Answers

Q1: Under the Shared Responsibility Model of ExaCC, what are you responsible for monitoring vs. what Oracle monitors?

Answer:

Customer Responsibility: Virtual Machines (VM OS layer), Grid Infrastructure (GI/CRS), Automated Storage Management (ASM), Oracle Databases (CDBs/PDBs), local applications, and security access logs.

Oracle Responsibility: Physical hardware (Exadata Database and Storage Servers), RoCE/InfiniBand internal switches, Power Distribution Units (PDUs), physical hypervisors (KVM), and the physical OCI infrastructure. Oracle handles Automated Service Requests (ASR) for hardware replacements.

Q2: How do you monitor Exadata Storage Server health and performance metrics directly from the VM cluster since you do not have root/celladmin access to the physical storage cells?

Answer: On ExaCC, we use ExaCLI. It allows authorized database cluster users (cloud_user_clustername) to execute restricted monitoring commands remotely against the storage cells.

Command to check cell disk health: exacli -c cell_ip -e "LIST CELLDISK"

Command to check active warnings: exacli -c cell_ip -e "LIST ALERTHISTORY WHERE alertState='active'"

Q3: Which tools are critical for validating the overall deployment health and patch level compliance of an ExaCC stack?

Answer:

Autonomous Health Framework (AHF / Exachk): Run regularly to validate the entire hardware/software stack configuration against Oracle best practices, catching configuration drifts or pending bugs.

OCI Management Console / OCI CLI: Used to monitor high-level cloud metrics and state changes of the VM cluster and OCI database agents.

Enterprise Manager (OEM) Cloud Control: Plugs deep into RAC, ASM, and the databases for historical performance, performance hubs, and fine-grained service alerting.

Q4: How do you track the health and metrics of Pluggable Databases (PDBs) via OCI native tools?

Answer: By enabling Database Management (Basic or Full) via the OCI console or OCI API. This unlocks the Performance Hub inside the OCI cloud dashboard, allowing the tracking of Active Session History (ASH) analytics, SQL monitoring, workload volumes, and blocking sessions directly within the cloud interface.

Part 3: Real-World Operational Challenges

OCI Database Agent (dbcsagent) Hangs: The control plane agent on the VM cluster occasionally hangs or loses communication with OCI. When this happens, database actions (like scaling CPU, patching, or taking OCI backups) fail from the cloud console, even though the database is running locally.

Flash Cache Pollution & Latency Drops: Large ad-hoc analytical queries or faulty reporting applications running full table scans can bypass Smart Scan and thrash the Exadata Smart Flash Cache if storage indexes are poorly managed.

Grid Infrastructure (GI) Upgrade/Patching Blockers: Orphaned OS trace files or incorrect directory permissions (e.g., AHF trc file ownership mapped to root instead of grid) can cause rolling cluster patching routines to abruptly crash mid-way.

Part 4: Troubleshooting Playbook

Scenario A: Storage / ASM Disks dropped or throwing performance alerts

Check: Run exacli from the compute node to check if a specific physical or grid disk has gone into a "predictive failure" or "poor performance" state.

Resolution: Verify ASM rebalance status via v$asm_operation. Ensure Oracle ASR has triggered a ticket automatically.

Scenario B: Control Plane operation (e.g., CPU scaling or PDB creation) fails with timeout errors

Check: Log into the affected VM node via SSH as opc or root. Check the status of the cloud management agent:

bash

systemctl status dbcsagent

Use code with caution.

Resolution: Review /var/log/oracle/dbcsagent/dbcsagent.log. If it's unresponsive or out of memory, safely restart the service using systemctl restart dbcsagent.

Scenario C: Cluster Interconnect / RoCE Network packet drops

Check: Run pinggi or check oswatcher / traceroute across the private RoCE interfaces. Inspect GI alert logs (alert.log for CRS) for eviction warnings.

Part 5: QA Testing & Validation Case

Below is a technical testing scenario used during the project validation phase to ensure the monitoring framework alerts on real infrastructure issues.

xml

<CreativeWritingPad title="Test Case TC-EXACC-MON-004: Storage Cell Communication Alerting & Failover Validation">
TEST CASE ID: TC-EXACC-MON-004
COMPONENT: Exadata Storage Tier & ASM Disk Group Health
OBJECTIVE: Verify that the monitoring architecture (OEM, OCI Metrics, and local crsctl) correctly identifies, alerts, and handles a simulated storage channel connection drop without database service disruption.

PREREQUISITES:
1. ExaCC VM Cluster is healthy, running a 2-node RAC database.
2. ExaCLI is configured and accessible from compute nodes.
3. OEM Cloud Control / OCI Database Management agent is active.

TEST EXECUTION STEPS:
1. Validate baseline health: Execute 'crsctl check cluster -all' and ensure all nodes and disk groups are ONLINE.
2. From Compute Node 1, run ExaCLI to confirm storage cell status: 
   exacli -c [Cell_IP] -e "LIST CELL DETAIL"
3. Simulate an intermittent network path drop to one Storage Cell on the isolated client network boundary (done in coordination with network administrators/Oracle support via port filtering or dry-run path disablement).
4. Monitor system logs in real-time:
   - Tail the ASM alert log: tail -f /u01/app/grid/diag/asm/+asm/+asm1/trace/alert.log
   - Monitor the GI alert log for transport layer warnings.
5. Check OCI Console / OEM Cloud Control for critical alert ingestion.

EXPECTED RESULTS:
1. ASM drops the affected grid disk path gracefully without dismounting the entire Disk Group (due to High/Normal redundancy configurations).
2. The Database remains 100% online; RAC handles I/O failover transparently to alternative paths.
3. OEM/OCI Event Engine triggers a Critical Alert within 5 minutes: "ORA-15078: ASM disk group mismatch / Cell Communication Timeout".
4. ExaCLI command 'LIST ALERTHISTORY' shows an active, un-cleared alert entry.
</CreativeWritingPad>

Q1: What is the difference between Quarterly Maintenance and Monthly Security Maintenance in ExaCC X10M, and who is responsible for executing it?

Answer:

Responsibility Split: The underlying physical infrastructure patching is Oracle-managed. The customer manages the software layers inside the Guest VMs (Grid Infrastructure, Database Homes).

Quarterly Maintenance: Scheduled every 3 months. It includes full infrastructure software updates, firmware upgrades, and bug fixes for the entire Exadata stack.

Monthly Security Maintenance: Executed on a monthly cadence. It specifically targets critical security vulnerabilities (typically targeting CVSS scores greater than 7 or critical package updates).

Q2: How does Oracle perform Monthly Security Maintenance on X10M Database (Compute) Servers vs. Storage Servers without causing application downtime?

Answer:
Oracle utilizes zero-downtime mechanisms to ensure high availability:

Database/Compute Servers: Updates are applied online without a reboot. Oracle uses technologies like Ksplice to patch the Linux kernel and packages in memory, meaning your running virtual machines (VMs) and database instances face zero disruption.

Storage Servers: Updates are applied in a rolling manner, taking one storage cell offline at a time. While a cell is being updated, Oracle ASM handles data redundancy using normal or high redundancy disk groups, ensuring database I/O remains unaffected.

Q3: What happens during the 21-day Monthly Security Maintenance window? Can a customer reschedule it?

Answer:

The Window: Oracle publishes the maintenance schedule at least one week in advance. The designated window usually stretches across a 21-day period.

Rescheduling: Yes, customers can log into the OCI Console, navigate to the Exadata Infrastructure page, and reschedule the maintenance window to a time that suits their business operations, provided it falls within the permitted cycle.

2. Operational & Configuration Questions

Q4: How do you prevent your primary OCI Cloud Administrator from being overwhelmed by ExaCC infrastructure patch notifications?

Answer:
You configure Maintenance Contacts at the Exadata Infrastructure level.

You can designate up to 10 email addresses (such as DBAs, SysAdmins, or specific team distribution lists) directly in the OCI Console.

These contacts will receive operational infrastructure notifications directly regarding upcoming monthly and quarterly updates without cluttering the global cloud administrator's inbox.

Q5: If an out-of-cycle critical security vulnerability affects the guest VM OS (which is customer-managed), how should you handle it before the next monthly Oracle release cycle?

Answer:

While Oracle updates the underlying infrastructure, the Guest VM operating system is customer-managed.

For urgent out-of-cycle OS security updates, you patch the compute nodes manually using the patchmgr tool driven from a separate deployment server or a primary node with root SSH access.

You configure local YUM repositories or OCI OS Management Service to download the fix, run a system precheck, and execute a rolling update across the VM cluster nodes.

3. Troubleshooting & Best Practices Questions

Q6: What is the most critical tool to run before and after any ExaCC X10M infrastructure maintenance window, and why?

Answer:
The most critical tool is Exachk (Exadata Health Check).

Pre-maintenance: Run Exachk to ensure the system is in a healthy state, verify there are no active disk failures, ASM rebalances, or cluster communication issues that could cause a rolling update to fail or hang.

Post-maintenance: Run Exachk again to validate the entire platform configuration stack, ensure no services were left offline, and confirm the environment adheres to Oracle’s Best Practices.

Q7: What is the best strategy for orchestrating large-scale user-managed updates (Grid Infrastructure/Database patches) alongside Oracle's monthly infrastructure maintenance?

Answer:
The best practice is to leverage Exadata Fleet Update (EFU). EFU allows you to group VM clusters and Database Homes into collections, define maintenance campaigns, run fleet-wide prechecks, and cycle through rolling updates across the entire enterprise estate efficiently, reducing manual overhead and configuration drift.

Interview Question & Sample Answer

Question: How do you configure and handle email notification (-l notification) tracking when executing infrastructure or compute node patching via patchmgr in an Exadata/ExaCC X10M environment?

Answer:
During large-scale infrastructure maintenance or compute node updates in Exadata X10M/ExaCC, we use the patchmgr update utility. To ensure we don't have to manually watch terminal logs for hours, patchmgr provides native SMTP email alerting hooks using specific flags.

When launching the utility from the driving system, we pass the mail notifications arguments to keep the operations, infrastructure, or DBA teams updated in real time.

Key `patchmgr` Syntax for Notifications

Depending on the version of the patchmgr utility being deployed on the driving system, the parameters used to trigger email tracking updates are structured as follows:

--smtp_to (or legacy -l option): Specifies the space-separated list of target destination email addresses that will receive the status updates.
--smtp_from: Sets the official sending email address from the driving system.
--smtp_server: Identifies the internal SMTP relay server IP address or hostname.

Example Command (Rolling DB Node Update with Notifications):

bash

./patchmgr --dbnodes dbnode_group --upgrade --rolling --smtp_to "dba_team@company.com alerts@company.com" --smtp_from "exacc-x10m-driving-node@company.com"

What Information Does the Notification Contain?

Interviewers want to know what these automated notifications actually track. The generated patchmgr summaries provide visibility into three distinct phases of the patch execution lifecycle:

Pre-check Phase (--precheck / --patch_check_prereq): Sends out alerts listing any validation failures (such as asymmetric cluster status, grid issues, or free space limitations) before actual implementation begins.
Execution Progress: During a rolling update, patchmgr emails the team as it sequentially drains connections, takes the node down, runs the update, reboots, verifies system health, and moves to the next node.
Final Summary Report: A complete overview detailing the final patch state of every node, software versions applied, elapsed duration, and any non-critical post-patch warnings.

Pro-Tips for ExaCC X10M Infrastructure Context

Live Updates: Emphasize that starting with recent Exadata software layers (like 24.1+), database node updates support online --liveupdate options. Tracking notifications for live runs is a critical operational check.
Network Pre-requisites: Mention that if an email notification fails to send, the DBA team must verify that the driving management server has explicit port 25/587 network access to the corporate SMTP relay.

Q: "During network fluctuations or standard infrastructure updates on an ExaCC X10M environment, how do you modify the ora_net_ping internal configuration to increase retries and timeout thresholds to prevent false positives or split-brain evictions?"

Interview Answer (Model Response)

In an Oracle ExaCC X10M environment, the ora_net_ping utility is used internally by Oracle's Cluster Health Monitor (CHM) and Exadata infrastructure components to validate inter-node and node-to-storage network connectivity across the high-speed RoCE (RDMA over Converged Ethernet) network fabric.

Because ExaCC X10M operates under a shared-management model, these internal ping utilities are managed directly at the infrastructure tier. To safely increase ora_net_ping retries and timeout limits, the following approach must be taken:

1. Identify the Configuration Target

The network latency baseline of the X10M RoCE network is exceptionally low (sub-millisecond). If ora_net_ping is timing out, it usually points to an underlying RoCE frame drop or a temporary node saturation during rolling cluster maintenance. The thresholds are governed in two places:

The GI / CSS Cluster Heartbeat Tier: Controlled via misscount and nettimeout.

The ora_net_ping internal script/binary wrappers: Governed by environment variables or configuration files located in the Grid Infrastructure home ($GRID_HOME/log/<node>/chm/).

2. Process for Adjustment on ExaCC

Because the physical infrastructure and hypervisors are Oracle-managed, customers cannot directly log into the compute node's bare metal host (Dom0) to modify low-level kernel ping scripts. Adjustments are handled via the following steps:

Verify Current Timeouts via GI:
Check the general network timeout parameters within Grid Infrastructure using:

bash

crsctl get css misscount
crsctl get css nettimeout

Use code with caution.

Modify via crsctl (If applicable to the network layer):
If the pings are causing eviction behavior, increase the nettimeout (default is typically 30 seconds) to tolerate wider network blips:

bash

crsctl set css nettimeout <value_in_seconds> -all

Use code with caution.

Engage Oracle Support for Internal Infrastructure Tooling:
For changing the specific parameters inside the ora_net_ping service configuration itself (which operates at the Exadata storage/compute interface layer), you must open a Service Request (SR). Oracle Field Engineers modify these configurations inside the Dom0 management layer or update the cloud automation daemon configurations, as manual modification inside the customer VM does not persist across automated infrastructure patching cycles.

Key Follow-Up Discussion Points

RoCE Fabric Specifics: Emphasize that because the X10M uses RoCE (RDMA over Converged Ethernet) instead of the older InfiniBand fabric found in older generations, network timeouts must be evaluated against switch Priority Flow Control (PFC) configurations rather than traditional TCP/IP behaviors.

Custom Action Timeouts: Remind the interviewer that under the Enhanced Infrastructure Maintenance Controls for ExaCC, customers can adjust the general Custom Action Timeout up to 120 minutes via the OCI Console, allowing applications and internal node checks extra time to drain and stabilize before a node is aggressively fenced or updated.

Q: How do you disable or clear the ping target configuration in an Oracle Exadata Cloud@Customer (ExaCC) X10M or Oracle RAC environment, and why is this configuration used?

Answer:

1. How to Disable the Ping Target

To completely disable or clear the configured ping targets in Oracle Grid Infrastructure, you must run the srvctl command as the root or Grid user and pass an empty string ("") to the parameter. Depending on your Oracle Grid Infrastructure version and setup, you can do this at the network level or the node applications level:

At the Network level (Recommended for newer versions):
bash
srvctl modify network -netnum <network_number> -pingtarget ""
At the Nodeapps level:
bash
srvctl modify nodeapps -pingtarget ""
2. Why the Ping Target is Used

Public Network Monitoring: Introduced to assist in virtualized environments (like ExaCC). Virtual Guest VMs often cannot natively detect a physical upstream network link or switch failure.
VIP Failover Automation: Grid Infrastructure continuously pings the designated target (typically the default gateway switch) via the public network. If the ping fails or times out significantly, Oracle Clusterware determines the local node has lost public access and automatically fails over the Virtual IP (VIP) to a surviving node.

3. Why You Might Need to Disable or Modify It

False Failovers: If the upstream network switch or gateway firewall drops ICMP ping traffic due to strict security policies, it will cause unwarranted node evictions or constant VIP failovers.
Network Infrastructure Changes: When migrating subnets or updating gateway routers, the old target IPs become unreachable, forcing an administrator to either update the target list or temporarily clear it.

Q1: What is a Quorum Disk in Exadata/ExaCC, and why do we need to add an extra one?

Answer:
A quorum disk is a specialized disk created on database servers (compute nodes) that does not store user data but instead hosts copies of Clusterware Voting Files and the Oracle ASM Partner Status Table (PST).

We add or configure them primarily in systems with fewer than 5 storage servers (such as Quarter Racks or small Elastic configurations) that utilize High Redundancy disk groups. High Redundancy requires 5 distinct failure groups. In environments with only 3 storage servers, the database nodes are leveraged via iSCSI over the RoCE Network Fabric to act as the 4th and 5th quorum failure groups, preventing cluster eviction if multiple storage cells fail.

Q2: Which utility is used to manage and add quorum disks on the database nodes?

Answer:
The quorumdiskmgr utility (located at /opt/oracle.SupportTools/quorumdiskmgr) is used exclusively by the root user on the database servers to manage the lifecycles of quorum configurations, targets, and devices.

Q3: Can you walk me through the end-to-end command workflow to create and add a new quorum disk?

Answer:
The process is executed in a multi-step workflow spanning the OS layer (root), iSCSI network target mapping, and finally the Oracle Grid Infrastructure layer (grid / SYSASM):

Step 1: Check existing configuration & Network interfaces

Run the utility to find the active RoCE network interfaces (e.g., re0, re1 on X10M):

bash

/opt/oracle.SupportTools/quorumdiskmgr --list --config

Step 2: Create the Quorum Disk Configuration

Define the ASM binary owner (oragrid/grid), group, and network interfaces on all database nodes:

bash

/opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=grid --group=asmadmin --network-iface-list="re0, re1"

Step 3: Create the iSCSI Target

Generate the target instance mapped to a specific ASM diskgroup (e.g., DATAC1):

bash

/opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-diskgroup=DATAC1 --visible-to-ips="<Node1_RoCE_IP>, <Node2_RoCE_IP>"

Step 4: Create the iSCSI Device

Create the actual virtual block device on the database servers:

bash

/opt/oracle.SupportTools/quorumdiskmgr --create --device --target-node=<Host_Name> --asm-diskgroup=DATAC1

Verification: Ensure the path /dev/exadata_quorum/QD_DATAC1_* is visible and has the correct permissions.

Step 5: Add the Quorum Disk to Oracle ASM

Log into ASM via SQL*Plus as SYSASM, verify asm_diskstring includes the quorum path, and add the disk to the respective group:

sql

ALTER DISKGROUP DATAC1 ADD QUORUM FAILGROUP FG_QD_NODE1 DISK '/dev/exadata_quorum/QD_DATAC1_idxxx';

Q4: How does the network layer differ in ExaCC X10M compared to older generations (like X8M/X9M) when configuring quorum disks?

Answer:
The underlying command structures remain largely uniform, but ExaCC X10M relies natively on a 100 Gbps RoCE (RDMA over Converged Ethernet) Network Fabric instead of InfiniBand.

When configuring the --network-iface-list option in quorumdiskmgr, you must pass the RoCE interfaces (typically re0 and re1) instead of the older InfiniBand names (ib0, ib1).

Q5: What critical verification step must you perform after adding the quorum disk to ensure cluster stability?

Answer:
After the ASM disk group finishes rebalancing, you must verify that the Oracle Clusterware voting files have moved or re-allocated appropriately onto the newly provided quorum paths. This is verified using the Grid Infrastructure command line:

bash

crsctl query css votedisk

Expected Outcome: The list should display the voting disks mapped successfully to both the traditional Exadata storage cell grid disks and the newly added /dev/exadata_quorum/ paths. If it does not adjust automatically, execute crsctl replace votedisk +DATAC1 to redistribute them.

Q1: What are HugePages, and why are they mandatory for Oracle databases on ExaCC?

Answer: HugePages are pre-allocated memory blocks at the OS kernel level. By default, Linux uses 4 KB pages, but HugePages uses 2 MB pages (on x86_64).

Why it's used: Large Oracle databases running on ExaCC manage hundreds of gigabytes or terabytes of System Global Area (SGA). Managing 4 KB pages creates millions of page table entries, causing significant Memory Management Unit (MMU) and CPU overhead. HugePages drastically reduces the size of the page table, prevents swapping, and locks the SGA into physical RAM. [
Exadata Mandate: In modern Oracle Database Release Updates (DBRUs like 19.27 and 23.8+), standard 4 KB small pages are no longer supported for the SGA on Exadata hardware; using HugePages is strictly enforced.

Q2: What is the difference between standard HugePages and Transparent HugePages (THP)? Which one should be enabled on ExaCC?

Answer:

Static/Standard HugePages: Are pre-allocated by the system administrator at boot or runtime and remain locked for specific allocation (like Oracle SGA).
Transparent HugePages (THP): Are dynamically allocated and managed at runtime by a background Linux kernel thread (khugepaged).
ExaCC Recommendation: Historically, THP was strictly disabled because it caused runtime latency, node reboots, and RAC instabilities. However, starting with Oracle 23ai and Exadata System Software 25.1 on UEK7, Oracle recommends keeping static HugePages for the SGA, while enabling THP set to madvise (transparent_hugepage=madvise). This allows the OS to dynamically optimize large Oracle executable .text memory regions without impacting the SGA.

2. Configuration & Parameter Questions

Q3: What are the different values for the USE_LARGE_PAGES database parameter? Which one is the Exadata best practice?

Answer: The database parameter USE_LARGE_PAGES controls how the Oracle instance utilizes the OS-allocated HugePages:

FALSE: The database will not use HugePages at all.
TRUE: The database will attempt to use HugePages. If there aren't enough HugePages to fit the entire SGA, it will allocate what it can and backfill the remainder using standard 4 KB pages. (Note: This is not recommended for Exadata because mixed page sizes degrade performance).
ONLY: The database will only start if the entire SGA can fit completely inside the allocated HugePages. If there is insufficient allocation, the database instance fails to start.
AUTO_ONLY: The database attempts to use HugePages. If insufficient, it instructs the kernel to dynamically scale up vm.nr_hugepages at startup. If the OS cannot find contiguous free memory to do so, the startup fails.

Exadata Best Practice: Oracle mandates setting USE_LARGE_PAGES=ONLY to guarantee maximum performance and prevent mixed-page allocations.

Q4: How is memory reserved for HugePages by default on ExaCC deployments?

Answer: For automated environments like Exadata Cloud@Customer (post-release 17.1.5), the infrastructure automatically provisions memory pools based on the workload type chosen during VM cluster sizing:

OLTP (Online Transaction Processing) deployments: Automatically reserve 70% of the physical memory for HugePages.
DSS / Data Warehouse deployments: Automatically reserve 50% of the physical memory for HugePages.
The initial starter database uses USE_LARGE_PAGES=ONLY, while additional manual databases can hook into the remaining pool.

Q1: What is the primary benefit of Huge Pages in ExaCC?

Answer: Huge Pages reduce CPU and memory management unit (MMU) overhead by grouping standard 4 KB pages into larger 2 MB blocks. This lowers page table entries required for large System Global Area (SGA) allocations, preventing performance degradation and excessive swapping.

Q2: How do you configure Huge Pages at the OS level in ExaCC?

Answer: System administrators calculate the required pool size by dividing aggregate SGA sizes by 2 MB and modifying the vm.nr_hugepages kernel parameter in /etc/sysctl.conf (or via /etc/sysctl.d/). Changes are applied using the sysctl -p command. [

Q3: What is the recommended setting for the USE_LARGE_PAGES database parameter on Exadata/ExaCC?

Answer: The recommended setting is ONLY (or AUTO_ONLY depending on specific baseline rules), which ensures that the instance will fail to start rather than fallback to standard 4 KB small pages if insufficient huge pages are available. Starting with modern Oracle database releases on Exadata, small pages are restricted for the SGA, making USE_LARGE_PAGES=ONLY mandatory.

Q4: How do default allocations for Huge Pages behave in Exadata Cloud environments?

Answer: Post-release 17.1.5 defaults allocate a specific percentage of memory for Huge Pages based on workload type—reserving roughly 70% of memory for OLTP (Online Transaction Processing) databases and 50% for Decision Support / Data Warehouse deployments.

Q5: How do you verify Huge Page usage and allocation on the database node?

Answer: Run the command cat /proc/meminfo | grep Huge in the operating system shell to check total, free, and reserved huge page metrics.

Q1: What defines an Exadata Elastic Configuration, and how does it differ from traditional fixed racks?

Answer: Traditionally, Oracle Exadata came in fixed structural shapes like Quarter Rack, Half Rack, or Full Rack. Exadata Elastic Configuration allows enterprises to customize the exact number of Database (Compute) Servers and Storage Servers inside the machine.

Instead of sticking to rigid ratios (e.g., a Full Rack having 8 DB nodes and 14 storage cells), you can scale compute and storage independently to meet specific workload needs—such as adding extra storage nodes for massive analytical data warehouses without paying for extra database compute licenses.

Q2: In Exadata Cloud at Customer (ExaCC), what are the mandatory network requirements to provision a standard 2-Node VM Cluster?

Answer: A total of 7 public-facing (client-side) IP addresses are strictly mandatory to successfully configure and run a standard 2-node ExaCC VM cluster:

2 Fixed IPs: Assigned to the physical client network (1 per VM Node).
2 Virtual IPs (VIPs): Utilized for cluster infrastructure (1 per VM Node).
3 Single Client Access Name (SCAN) IPs: Used to distribute client connection requests dynamically across the entire cluster.

Q3: What is the "Locked Configuration Trap" in ExaCC deployment?

Answer: The "Locked Configuration Trap" refers to a network metadata constraint in Oracle Cloud Infrastructure (OCI). Once an ExaCC VM Cluster is fully instantiated, you cannot modify its core network metadata (such as VLAN IDs, client subnets, or Netmasks) directly.

The Solution: If a critical mistake is made or network restructuring is required, you must completely terminate and destroy all child VM clusters and hosted databases to reconfigure the base network infrastructure.

Storage & Performance Optimization Questions

Q4: How do Cell Disks and Grid Disks relate to each other in Exadata storage architecture?

Answer: They represent two separate logical layers built on top of physical storage:

Cell Disk: A logical volume created directly out of a physical disk drive that has been discovered and formatted by the Exadata Storage Server software.
Grid Disk: Created on top of Cell Disks. Grid Disks are what Exadata presents to Oracle Automatic Storage Management (ASM) as raw ASM disks. To maximize performance, space is allocated sequentially starting from the outer, faster-spinning tracks of the Cell Disk and moving inward.

Q5: What are the primary communication protocols that handle data movement between Exadata DB Nodes and Storage Cells?

Answer: Exadata relies on iDB (Intelligent Database) and LIBCELL protocols:

iDB Protocol: A specialized network protocol operating over the high-speed InfiniBand or RoCE network. It carries commands and metadata between the database instances and the storage cells, allowing operations like Smart Scan offloading.
LIBCELL (Library Cell): A software library linked directly into the Oracle Database kernel. It overrides standard operating system read/write system calls, allowing the kernel to transparently talk to storage servers over network protocols instead of standard block I/O.

Q6: Explain how a Storage Index works in memory and whether it can be manually tuned.

Answer: A Storage Index automatically tracks the minimum and maximum values of up to eight columns for every 1MB chunk of storage (called storage regions).

How it works: When a query contains a WHERE clause, the storage server checks the index. If the criteria fall outside the min/max range, that entire 1MB chunk is skipped, massively reducing physical disk I/O.
Manual Tuning: No manual intervention is possible. Storage Indexes are built, maintained, and updated entirely in memory by the Exadata storage software and are never written to disk.

Administration & Operations Questions

Q7: What is the step-by-step procedure to properly shut down a complete physical Exadata Machine?

Answer: To safely power down an Exadata rack, the dependencies must be removed from the top layer down to the physical hardware:

Stop the Database instances and their respective Listeners.
Stop the Grid Infrastructure / Cluster software on all nodes.
Shut down the Database Servers (Compute nodes) via the OS.
Shut down the Cell Storage Servers.
Shut down all the network switches (InfiniBand/RoCE and Cisco switches).
Remove power directly from the Power Distribution Units (PDUs).
(Note: The startup procedure is performed in the exact reverse order).

Q8: Which primary utilities are used to perform health checks and manage configurations?

Answer:

OEDA (Oracle Exadata Deployment Assistant): Used for initial network allocation, parameter generation, and initial machine deployment.
Exachk / Exacheck: The primary automated health check tool designed to scan the entire Exadata stack for configuration drift, security vulnerabilities, and best practice deviations.
CellCLI & DCLI: CellCLI manages a single storage cell locally, while DCLI (Distributed Command Line Interface) allows administrators to replicate commands across multiple storage cells or database nodes simultaneously.
ASR (Auto Service Request): Automatically monitors hardware faults and instantly logs a Service Request (SR) with Oracle Support when hardware failure occurs.

EasyReliableDBA

Tuesday, 29 September 2026

Managing Database on ExaCC (X10M) part2

No comments:

Post a Comment

Search This Blog