Thursday, 19 November 2026

Exadatacc.x10m.xl Architecture Detail and Troubleshooting Steps part2

 



Oracle Exadata Database Machine X10M

Q: sys and system password rotation in exacc and best practice


In ExaCC (Oracle Exadata Database Service on Cloud@Customer / Exadata Cloud at Customer), changing or rotating the SYS and SYSTEM administrative passwords requires using Oracle's managed utility rather than raw manual edits or standard manual orapwd commands. 

How to Change SYS and SYSTEM Passwords in ExaCC
1. Changing the SYS Password
Use the dbaascli utility on the database node to handle SYS password updates and keep cluster and backup synchronizations intact:
  • Standard / RAC Databases:
    Run the utility as the appropriate database administrator user on the VM:
    bash
    sudo dbaascli database changepassword --dbname <database_name> --user SYS
    
    Follow the prompt to supply the new password. 
  • Data Guard Environments:
    Execute the preparation workflow on the primary database node to generate a sync blob, transfer it to the standby nodes, and apply it there to prevent broker/authentication failures:
    1. Primary node: dbaascli database changePassword --dbName <dbname> --user SYS --prepareStandbyBlob true --blobLocation <location_to_create_blob_file>
    2. Copy the blob file to the standby VM(s) and assign oracle ownership.
    3. Standby nodes: dbaascli database changePassword --dbName <dbname> --user SYS --standbyBlobFromPrimary <location_of_copied_blob_file> 
2. Changing the SYSTEM Password
  • Update the SYSTEM user via SQL*Plus connected to CDB$ROOT as SYSDBA:
    sql
    ALTER USER SYSTEM IDENTIFIED BY "<new_complex_password>";
    

  • Refresh any application connections, PDB mappings, or scheduled scripts referencing the old SYSTEM credentials. 

Best Practices for SYS and SYSTEM Rotation in ExaCC
  • Rely on dbaascli, Not Manual Edits: Avoid manual orapwd or file-level swaps for SYS, as ExaCC's internal backup and agent utilities depend on dbaascli to sync wallet and authentication states. 
  • Respect Data Guard Protocols: Always use the dedicated Data Guard workflow parameters in dbaascli to sidestep ORA-46952 or transport authorization drops.
  • Secure Credentials: Refrain from hard-coding administrative passwords in scripts; leverage Oracle Wallets or secret stores instead. 
  • Enforce Policy Standards: Apply strong password complexity rules (length, mix of characters) meeting Exadata security baselines.
  • Patch and Validate: Ensure DB homes are updated to prevent metadata tracking or expiration discrepancies for core accounts. 


1. Why you cannot just use ALTER USER
In a non-cloud environment, a DBA alters the password via SQL*Plus (ALTER USER sys IDENTIFIED BY...). In ExaCC, doing this manually breaks cloud automation framework workflows. ExaCC relies on internal scripts, backup automation (bkup_assistant), cloud monitoring tools, and Data Guard frameworks that cache credentials. Changing it exclusively via SQL leaves the automation with stale credentials, causing failed backups and critical infrastructure errors. 
2. The Correct Methods to Rotate
To safely rotate the SYS password, you must use either the OCI Cloud Console (UI) or the DBAASCLI utility from the virtual machine command line.
  • Method A: Via the OCI Console (UI)
    1. Log into your OCI Cloud Console.
    2. Navigate to Oracle Database -> Exadata Cloud@Customer -> VM Clusters.
    3. Click on your specific database, select Manage Passwords under the Database Actions or More Actions menu.
    4. Provide the new password. This triggers a backend cloud job that changes the password at the Database level, updates the internal configuration/wallet caches, and auto-propagates it across all RAC nodes and Data Guard standbys. 
  • Method B: Via the Command Line Interface (dbaascli)
    Log into your ExaCC Database node VM as the opc or oracle user and invoke the cloud tooling:
    bash
    # Run as opc/root or switch to oracle user
    dbaascli database changepassword --dbname <your_db_name> --user SYS
    
    The utility will securely prompt for the new password, alter the database, and automatically refresh dependent background cloud agents (e.g., bkup_assistant). 

 Interview Question and Answer
Interviewer: "How do you handle SYS and SYSTEM password rotation in an Oracle Exadata Cloud at Customer (ExaCC) environment, and what are the operational best practices you follow to avoid downtime or service disruption?"
Recommended Candidate Answer:
"When managing an ExaCC environment, the core rule is that you must never rotate the SYS database password purely via standard SQL*Plus ALTER USER commands. 
My personal best practices checklist for this rotation includes:
  • Enforcing Cloud Complexity Filters: ExaCC enforces strong complexity rules natively. I ensure our team picks passwords meeting the cloud requirements (typically a minimum of 15 characters, including 2 uppercase, 2 lowercase, 2 numbers, and 2 special characters). 
  • Decoupling Data Guard Transport: I configure Data Guard to use a dedicated REDO_TRANSPORT_USER rather than relying on the SYS account. This way, when SYS undergoes periodic compliance rotations, log shipping is never interrupted. 
  • Using Oracle Wallets for Scripts: Any client-side scripts, local cron jobs, or third-party application monitoring tools are strictly barred from hardcoding cleartext passwords. They are mapped using secure Oracle Wallets.
  • Blackout Management: Before performing mass automated rotations, I ensure Enterprise Manager (OEM) or other monitoring agents are put into a controlled blackout window to avoid false alerting loops." 



Q: okv key rotation in exacc and best practice


Rotating encryption keys in Oracle Key Vault (OKV) for Exadata Cloud@Customer (Exa@CC) involves managing TDE (Transparent Data Encryption) master encryption keys at the database level while maintaining OKV server and endpoint infrastructure certificates.
How Key Rotation Works in Exa@CC with OKV
  • TDE Master Keys: You cannot initiate or rotate the TDE master encryption key centrally from the OKV management console. You must execute rotation directly inside the database using SQL/Administer Key Management commands: 
    sql
    ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "<password>" [WITH BACKUP];
    

  • OKV Component & Certificate Rotations: OKV servers and endpoints (database nodes and ZFS appliances) require regular certificate and software/endpoint rotation. If endpoint or CA certificates expire, communication between OKV and the database drops, causing an outage.
  • Re-enrollment: After patching OKV nodes or when endpoint certificates need renewal, re-enrolling the database endpoint or supplying updated okvclient.jar binaries is often required.

Best Practices for OKV Key & Certificate Rotation in Exa@CC
  • Execute TDE Key Rotation from the Database: Always run ADMINISTER KEY MANAGEMENT commands locally on the database instance rather than looking for a centralized push button in OKV.
  • Retain Old Key Versions / Backups: Never delete or purge old TDE master keys immediately. Keep old key versions active or safely backed up in OKV wallets so historical backups and RMAN point-in-time recoveries remain decryptable. 
  • Track Certificate Expiration Deadlines: Monitor OKV server and endpoint certificate validity windows strictly. Do not miss renewal deadlines, as expired endpoint certificates break the KMIP/TLS communication channel between Exa@CC databases and OKV.
  • Use Company / Intermediate CAs: Configure OKV with an Intermediate CA certificate signed by your enterprise CA (such as Keyfactor) rather than relying solely on default self-signed lifecycles that are harder to track at scale.
  • Test in Staging First: Before rolling out major OKV updates, client jar updates, or re-enrolled endpoints across production Exa@CC RAC clusters, validate database startup, TDE operations, and RMAN/Data Guard workflows in a non-production environment. 
  • Maintain Local Offline Backups: Always keep a secure, offline local backup or software keystore/wallet fallback accessible in case network communication with the OKV cluster is temporarily interrupted during maintenance



Q1: When you rotate a TDE Master Key in an ExaCC database, does it re-encrypt all the underlying data on disk?
Answer: No. Rotating the TDE master key only generates a new master key inside OKV. This new master key is then used to re-encrypt the Tablespace Encryption Keys (TEK). The actual payload data residing in your Exadata storage cells remains untouched, meaning the process is near-instantaneous and does not cause a performance penalty on the database.
Q2: What happens if an ExaCC database endpoint certificate expires before it can be rotated?
Answer: The database endpoint will lose its mutual TLS (mTLS) connection with the OKV server. If the database attempts to open an encrypted tablespace or fetch a key for a new session, it will fail, causing application outages. To fix this, you must perform a manual Endpoint Re-Enrollment by downloading a new okvclient.jar file and registering the node again.
Q3: Why must we preserve old keys in OKV after a successful key rotation?
Answer: TDE master keys operate on a versioning lineage. While new data and new logs are encrypted using the latest active key, older database backups, exported datafiles, and archived redo logs remain encrypted with the older master keys. If you purge old keys from OKV, your historical backups become permanently unrecoverable.
Q4: How do you rotate a database TDE master key manually if cloud tooling is unavailable?
Answer: You log into the container database (CDB) as an administrative user and execute the SQL command:
sql
ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "wallet_password" KEY STORE CLOSE ON EXIT;
For multitenant architectures, this is typically executed at the CDB level, though independent PDB key management can be configured depending on isolation requirements.


Question : AWR Interpretation


Real-World Project Context & Use Case
  • Project Profile: Core Data Warehousing & Real-Time Analytical Platform (OLAP) for a global financial institution processing 50+ million transactions daily. 
  • Architecture: ExaCC X10M Quarter Rack running Oracle Database 19c/21c (Multitenant Architecture).
  • The Problem: A critical End-of-Day (EOD) portfolio risk-assessment ledger query unexpectedly experienced a severe performance degradation, spiking execution times from 45 seconds to over 45 minutes. 

 Step 1: Identifying the Degradation in AWR
When the slowdown occurred, an Automatic Workload Repository (AWR) report was generated for the historical snapshot window.
1. Top Foreground Wait Events
In an Exadata environment, we look for indicators showing whether data is processing efficiently in the storage tier via Smart Scans or stalling in the compute tier. 
Wait EventWaitsTotal Wait Time (sec)Avg Wait% DB TimeDiagnostic Interpretation
cell smart table scan1,2004537ms1.2%Expected. Exadata offloading reads cleanly to flash/storage.
db file sequential read4,500,0002,1000.46ms78.5%The Smoking Gun. The query dropped out of Smart Scan and started forcing single-block index lookups into XRMEM/Flash.
2. SQL Ordered by Elapsed Time & Get Metrics
Under SQL Ordered by Elapsed Time, SQL_ID: 87am9x1z7qp2w was responsible for 88% of DB Time.
  • Buffer Gets (Logical Reads): Spiked to 140,000,000.
  • Physical Reads: 35,000,000.
  • Exadata CPU Offload Efficiency: Changed from 95% down to 0%.

 Step 2: Execution Plan Regression Analysis
Using dbms_xplan.display_awr('87am9x1z7qp2w'), we compared the Baseline (Good) Plan against the Regressed (Bad) Plan.
The Complex Query Structure
sql
SELECT /*+ MONITOR */ ledger_id, SUM(balance) 
FROM financial_ledgers f 
JOIN account_dimensions a ON f.acct_key = a.acct_key
WHERE a.region = 'NORTH_AMERICA' AND f.trade_date = :b1
GROUP BY ledger_id;
Plan Comparison
text
=========================================
GOOD PLAN (Child 0) - Exadata Smart Scan
=========================================
----------------------------------------------------------------------------------------------------------

| Id  | Operation                           | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT                    |                   |       |       | 4200 (100) |          |
|   1 |  HASH GROUP BY                      |                   |   120 |  4800 | 4200   (2) | 00:00:01 |
|*  2 |   HASH JOIN                         |                   |  450K |   16M | 4150   (1) | 00:00:01 |
|   3 |    TABLE ACCESS FULL                | ACCOUNT_DIMENSIONS|  5000 |  120K |   50   (0) | 00:00:01 |
|*  4 |    CELL SMART TABLE SCAN            | FINANCIAL_LEDGERS |   50M |  1.2G | 4000   (1) | 00:00:01 |
----------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
   4 - storage(:b1 = "F"."TRADE_DATE") -- Passed directly to Exadata Storage Cells!
Use code with caution.
text
=========================================
REGRESSED PLAN (Child 1) - Index Stall
=========================================
----------------------------------------------------------------------------------------------------------

| Id  | Operation                           | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT                    |                   |       |       |  2M  (100) |          |
|   1 |  HASH GROUP BY                      |                   |   120 |  4800 |  2M    (2) | 00:09:30 |
|*  2 |   NESTED LOOPS                      |                   |     2 |    80 |  2M    (1) | 00:09:30 |
|   3 |    TABLE ACCESS FULL                | ACCOUNT_DIMENSIONS|  5000 |  120K |   50   (0) | 00:00:01 |
|*  4 |    TABLE ACCESS BY INDEX ROWID BATCHED| FINANCIAL_LEDGERS |     1 |    24 |   450  (0) | 00:00:01 |
|*  5 |     INDEX RANGE SCAN                | IDX_LEDGER_ACCT   |     1 |       |    12  (0) | 00:00:01 |
----------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
   5 - access("F"."ACCT_KEY" = "A"."ACCT_KEY")
Use code with caution.
Why it Degraded
Stale optimizer statistics on ACCOUNT_DIMENSIONS led the Cost-Based Optimizer (CBO) to believe that only 2 rows would return from the join. Because the cardinality estimate dropped to 2 rows, the optimizer abandoned the high-throughput Cell Smart Table Scan (Good Plan) and picked a Nested Loops with Index Range Scan (Bad Plan).
In reality, the join returned 450,000 rows, causing the database to loop millions of times over the index (db file sequential read), completely bypassing Exadata storage offloading. 

 Resolution Strategies
  1. Immediate Mitigation: Forced the database to use the baseline plan using an SPM (SQL Plan Baseline) or an SQL Profile fetched from the valid AWR snapshot histories.
  2. Permanent Fix: Gathered updated statistics and locked the preferred execution profiles using Adaptive Plan mechanisms. 

 OEM (Enterprise Manager) Performance Monitoring
To prevent this degradation from flying under the radar again, the following setup was configured inside Oracle Enterprise Manager (OEM) Cloud Control:
  1. Real-Time SQL Monitoring (RTSM): Because the query contains the /*+ MONITOR */ hint and runs parallel or consumes over 5 seconds of CPU/IO, it triggers RTSM automatically. DBAs can visually track row-by-row progression through the execution plan nodes live.
  2. Performance Hub (PerfHub): Used to isolate the exact moment of plan divergence. The "Selected SQL" view maps execution history over time, showing visually when the plan shifted from Child 0 to Child 1 along with corresponding spikes in user wait-time components. 
  3. Metric Alerts Configuration: Dedicated alerts were implemented to monitor Exadata-specific operational metrics:
    • Exadata Storage Cell Alert: Triggers if cell flash cache bypass percentages cross custom thresholds.
    • SQL Execution Plan Change Alert: Notifies the DBA team immediately via notification groups if critical analytical SQL_IDs alter their PLAN_HASH_VALUE.

or

Executive Summary
Deploying complex database workloads on Oracle Exadata Cloud@Customer (ExaCC) X10M introduces unmatched scale via 4th Gen AMD EPYC processors, DDR5 DRAM, and ultra-low latency Exadata RDMA Memory (XRMEM). However, complex query performance degradation often surfaces not due to resource constraints, but from architectural mismatches like failing to trigger Smart Scans, storage index invalidation, or network-fabric serialization. 
This framework breaks down how to interpret Automatic Workload Repository (AWR) profiles, dissect Execution Plans, and mitigate performance degradation based on a real-world enterprise analytics migration. 

Project Context & System Architecture
Financial Data Warehouse Migration Project
  • Objective: Consolidate a legacy on-premises reporting database into an ExaCC X10M Elastic Quarter Rack. 
  • Workload: Complex reporting, recursive financial rollups, multi-table hash joins, and deep analytical scans.
  • The Issue: Post-migration, a set of critical end-of-month reconciliation queries experienced a 4x performance degradation, stalling CPU queues and inflating ETL windows.

AWR Interpretation: Spotting Exadata-Specific Bottlenecks
When queries degrade on an ExaCC X10M platform, classic wait events present unique Exadata signatures. Analyze these specific sections of the AWR report:
1. Top 10 Foreground Wait Events
  • cell smart table scan: The primary indicator of healthy Exadata offloading. If this event displays high wait counts but low average wait times (<1.5ms), Smart Scan is functioning effectively over the 100Gbps RoCE fabric.
  • cell single block physical read: Indicates traditional random I/O bypassing Smart Flash Cache and XRMEM. High waits here signal index misconfiguration or non-selective queries. 
  • cell smart flash idle / cell smart table scan flushing: If elapsed times spike here, it indicates storage cells are performing normal-flushing operations, forcing queries to wait for dirty blocks to write to disk.
2. Exadata Performance Statistics Section
Cross-reference the Exadata Statistics page to verify offload efficiency:

  • Healthy Target: > 80% for analytical queries.
  • Degraded Baseline: < 10%, which indicates massive volumes of raw, uncompressed blocks are traveling across the network fabric, forcing database nodes to process filtering locally.

Execution Plan Analysis: Smart Scan vs. Host Processing
To diagnose the degradation, extract the historical execution plan using the AWR repository:
sql
SELECT * FROM TABLE(dbms_xplan.display_awr('g81cbrq5yamf5', format => '+BASIC +NOTE +PRED +PSTATS'));
Degraded Execution Plan (The Anti-Pattern)
text
-----------------------------------------------------------------------------------------------

| Id  | Operation                           | Name               | Rows  | Bytes | Cost (%CPU)|
-----------------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT                    |                    |       |       |  905K(100)|
|*  1 |  HASH JOIN                          |                    |  157K |   22M |  905K  (2) |
|   2 |   TABLE ACCESS FULL                 | GL_LEDGER          |  4M   |  120M |  210K  (1) |
|*  3 |   TABLE ACCESS BY INDEX ROWID BATCHED| GL_BALANCES        |   12M |  810M |  690K  (3) |
|*  4 |    INDEX RANGE SCAN                 | GL_BAL_IDX01       |  40M  |       |  12K   (2) |
-----------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter("FISCAL_YEAR"=2026)
  • The Bug: The optimizer chose an INDEX RANGE SCAN. Because it uses an index lookup, it performs single-block reads, completely bypassing Exadata Smart Scan offloading. 

Optimized Execution Plan (The Target)
text
-----------------------------------------------------------------------------------------------

| Id  | Operation                           | Name               | Rows  | Bytes | Cost (%CPU)|
-----------------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT                    |                    |       |       |  42K (100)|
|*  1 |  HASH JOIN                          |                    |  157K |   22M |  42K   (2) |
|   2 |   TABLE ACCESS STORAGE FULL         | GL_LEDGER          |  4M   |  120M |  10K   (1) |
|*  3 |   TABLE ACCESS STORAGE FULL         | GL_BALANCES        |   12M |  810M |  31K   (2) |
-----------------------------------------------------------------------------------------------
Note
-----
   - cell smart table scan                                                                      
   - storage structure indexes applied

  • The Fix: Changing the operation to TABLE ACCESS STORAGE FULL allows Oracle to offload filtering directly to the storage cells. It filters rows at the storage layer and sends only the matching dataset back to the database servers. 

Core Engineering Challenges & Mitigations
ChallengeRoot CauseExaCC X10M Solution / Mitigation
Storage Index InvalidationOut-of-date or ungathered global object statistics caused the optimizer to miscalculate data distribution.Implement Real-Time Statistics Gathering and enforce Hybrid Columnar Compression (HCC) to optimize the storage index structure.
PGA/Temp SpillingMassive multi-table hash joins exceeded the configured PGA_AGGREGATE_TARGET, spilling intermediate join tables to disk.Leverage X10M's vast DDR5 RAM pool. Transition the instance from PGA_AGGREGATE_TARGET to PGA_AGGREGATE_LIMIT and expand the automated workareas.
Implicit Type ConversionPredicate clauses contained string parameters compared against numeric columns, disabling Smart Scan offloading.Refactor application code to enforce type consistency or introduce Virtual Columns to maintain clean evaluation logic.

Step-by-Step Execution Plan
FLOWCHART diagram
  1. Isolate Snapshots: Run awrrpt.sql spanning the exact 30–60 minute degradation window.
  2. Verify DB Time: Ensure \(\text{DB Time} \gg \text{Elapsed Time}\) to confirm active serialization bottlenecks rather than external network or client-side delays.
  3. Audit the SQL ID: Extract raw text and runtime history via awrsqrpt.sql.
  4. Enforce Smart Scans: Use SQL Patches or profiles to inject /*+ FULL(t) CONFIDE_ON_STORAGE */ hints to force Smart Scan execution without changing core code.
  5. Pin Plan Baselines: Lock down stable plans using SQL Plan Management (SPM) to prevent future optimization flips


or

In Oracle Exadata Cloud@Customer (ExaCC) X10M, optimizing database performance requires a specialized approach. Because the X10M platform shifts from Intel to massive core-count AMD EPYC processors and replaces PMEM with Exadata RDMA Memory (XRMEM), traditional AWR interpretation and Explain Plans must be evaluated through an "Exadata-aware" lens.

1. AWR Interpretation in ExaCC X10M
An Automatic Workload Repository (AWR) report on ExaCC contains dedicated Exadata Performance Statistics sections. When interpreting these reports, you should focus on how effectively the database is offloading work to the storage tier.
Key Metrics to Evaluate
  • Exadata Smart IO Statistics: Look for cell physical IO bytes eligible for smart scan versus cell physical IO bytes saved by storage index. On X10M, high efficiency means a large percentage of IO is filtered out completely before hitting the network.
  • Top Foreground Wait Events:
    • cell single block physical read: Indicates traditional OLTP index lookups. In X10M, these should be hyper-fast due to RoCE network fabric and XRMEM caching.
    • cell smart table scan: Indicates the database is successfully offloading queries via Smart Scan.
    • cell multiblock physical read: This is an anti-pattern on Exadata. It indicates a full table scan that failed to offload to the storage cells.

2. Explain Plan in ExaCC X10M
An Explain Plan or actual Execution Plan (via DBMS_XPLAN.DISPLAY_CURSOR) details the steps Oracle intends to take. To verify if a query is taking advantage of X10M’s architecture, look for specific database operations:
  • TABLE ACCESS STORAGE FULL or INDEX STORAGE FAST FULL SCAN: The keyword STORAGE confirms that the operation will be offloaded to the Exadata storage cells. If it simply says TABLE ACCESS FULL, the offload is not happening.
  • Predicate Information: Under the plan, look for storage(...) predicates. This proves that filter conditions (e.g., WHERE status = 'ACTIVE') are being evaluated directly inside the storage servers, preventing raw data from clogging the 100 Gbps RoCE network.

3. Key Use Cases
Use CaseAWR IndicatorExplain Plan Indicator
High-Throughput Analytics & ReportingHigh values for cell physical IO bytes eligible for smart scan.TABLE ACCESS STORAGE FULL with storage(...) predicates in the plan output.
Ultra-Low Latency OLTPProminent cell single block physical read wait times under < 100 microseconds, proving XRMEM cache hits.Clean index lookups (INDEX UNIQUE SCAN) bypassing storage scans entirely.
Consolidated Mixed WorkloadsHigh CPU utilization but steady I/O throughput managed by IORM (I/O Resource Management) metrics.Plans showing consistent join methods (like fast hash joins) even during peak workloads.

4. Challenges and Pitfalls
While ExaCC X10M is incredibly powerful, certain configuration mistakes can limit its potential:
  • Smart Scan Prevention: If a query uses features like serial PL/SQL functions in the WHERE clause, or if the table headers are un-synchronized, Exadata will fallback to a standard cell multiblock physical read. You will see this flag in the AWR report, and the STORAGE keyword will disappear from the execution plan.
  • The "CPU Blindspot" (AMD High-Core Management): Because X10M scales up to thousands of cores, a poorly written query using row-by-row processing (CURSORS without bulk collects) might not show up as an IO bottleneck in AWR. Instead, it will silently consume vast amounts of DB CPU time. DBAs must actively look at the SQL ordered by CPU Time section in AWR.
  • Temp Space Spills: If large parallel queries require sorting that exceeds the PGA size, data spills to the temporary tablespace. On AWR, this shows up as direct path write temp and direct path read temp. This bypasses many Exadata caching benefits and indicates that PGA_AGGREGATE_TARGET needs adjustment.

or

The X10M architecture features massive upgrades—including AMD EPYC processors (up to 96 cores per database server), PCIe Gen 5, DDR5 memory, and Exadata RDMA Memory (XRMEM) replacing PMEM. Because of this raw power, performance bottlenecks shift from standard hardware limitations to configuration, application efficiency, or storage cell offloading. 

1. The Core Baseline: DB Time vs. Host CPU
Always start at the top of the AWR report with the Report Summary. 
  • DB Time vs. Elapsed Time: If DB Time is significantly greater than Elapsed Time × Number of Cores, your system is overloaded. On an X10M, which has massive core counts, it takes a highly parallel or highly inefficient workload to saturate the host CPU
  • Load Profile: Check the Logical Reads vs. Physical Reads. A massive volume of physical reads indicates either smart scans are running or you are missing out on the X10M's vast flash cache capacities. 

2. Primary Top Foreground Wait Events
Look at the Top 10 Foreground Events to see where the database is spending time: 
  • cell single block physical read: Indicates an OLTP-style single-block read from the storage cells. In X10M, if these are hitting the XRMEM cache, latencies should be sub-100 microseconds. If the latency is in milliseconds, the data is coming from hard disks or flash cache misses. 
  • cell smart table scan / cell smart index scan: This is a good sign; it means Exadata is offloading data processing to the storage layer. However, if this event dominates DB time, verify if the SQL is actually reducing data volume or wasting cell CPU resources.
  • cell multiblock physical read: Indicates standard multi-block I/O that failed to trigger a Smart Scan (e.g., due to an un-indexed serial query or initialization parameter mismatch like cell_offload_processing = false).

3. Exadata-Specific Sections (The Smoking Guns)
Scroll past the standard SQL statistics to the Exadata Statistics sections: 
 Performance Summary & Smart I/O
  • Offload Efficiency: Look for the ratio of Cell Physical IO Bytes Eligible for Smart Scan vs. Cell Physical IO Bytes Returned by Smart Scan. In a healthy data warehouse/analytics workload on X10M, you want to see an offload efficiency > 90% (meaning 90%+ of the data was filtered directly at the storage server and not sent across the network). 
 XRMEM Cache Performance (X10M Specific)
X10M uses Exadata RDMA Memory (XRMEM). Review the XRMEM Cache Performance Statistics: 
  • XRMEM Read Hits: High hits indicate excellent OLTP performance using ultra-fast remote memory.
  • XRMEM Read Misses: High misses mean the workload is spilling down into the Smart Flash Cache or capacity disks. 
 Smart Flash Cache & Flash Log
  • Flash Cache Hit %: X10M features extreme flash storage. Look at the Flash Cache Read Hits. OLTP environments should exhibit high cache hit ratios here. 
  • Cell Smart Flash Log: Check for write log latencies. X10M uses flash logging to eliminate log file sync bottlenecks. If you see high cell log file sync times, check the storage servers for write saturation or outlier disks. 

4. Outlier & Health Diagnostics
ExaCC runs on a shared infrastructure where one noisy neighbor or bad disk can hurt cluster performance. 
  • Exadata Server Health Report: Ensure there are no open alerts or offline disks listed. A single degraded disk can disrupt I/O balance across the cell matrix. 
  • Top Databases by I/O: Look at this section to ensure that specific PDBs are not starving others of I/O resources, and verify that IORM (I/O Resource Manager) configurations are actively managing workloads according to your business rules. 

Quick Diagnostic Cheat Sheet
Symptom in AWRCommon Root CauseX10M Resolution Strategy
High cell smart table scan wait timeMassive full table scans with poor storage indexing.Ensure Storage Indexes are built; evaluate if queries require partitioning.
High db file sequential read or cell single block...Storage layer cache misses or unoptimized index lookups.Check if tables/indexes should be pinned in the Keep Cache or if XRMEM is saturated.
Low Offload Efficiency %Code patterns preventing offloading (e.g., using ROWNUM, complex PL/SQL wrapper functions).Rewrite SQL to utilize native SQL operators that can be natively offloaded to cell CPUs