EasyReliableDBA

Creating First VM Cluster Network on Oracle Exadata Database Service on Cloud@Customer

As an Exadata Cloud@Customer (ExaC@C) DBA Expert, managing the environment requires a distinct understanding of what you control inside the Guest VMs (DomU) versus what Oracle manages or what requires specialized tools like dbaascli.

Below is the definitive, production-ready command and diagnostic reference manual organized by daily operational workflows.

1. Daily Health Check & Status Verification Commands

Run these at the start of your shift to verify infrastructure, clusterware, and software layers.

Component	Command	Purpose
Exadata Version	`imageinfo`	Verifies current DomU image, kernel, and Exadata software release.
Patch History	`imagediff -b`	Checks the baseline and history of infrastructure updates.
Grid Infrastructure	`crsctl check crs` or `crsctl stat res -t`	Checks the health and status of all clusterware resources.
ASM Status	`asmcmd lsdg`	Verifies ASM diskgroup mount status, total capacity, and free space.
Database Status	`srvctl status database -d <db_unique_name>`	Verifies running instances across all VM cluster nodes.
Local Node Alerts	`dbmcli -e "list alert history where severity='critical' or active='true'"`	Checks VM-level hardware/software alert thresholds.
Storage Alerts	`dcli -g cell_group -l celladmin "cellcli -e list alert history"`	Scans all storage cells for critical or active alerts (Read-Only on ExaCC).

2. Performance Monitoring & Diagnostics

Use these commands to pinpoint CPU bottlenecks, memory depletion, network latencies, or I/O contention.

Mass Execution via DCLI: Run an OS check across all database nodes simultaneously:
bash
dcli -g dbs_group -l oracle "uptime; free -g"
Storage Cell Performance Metrics:

bash

# Check flash cache utilization and hit ratios across cells
dcli -g cell_group -l celladmin "cellcli -e list metriccurrent where name like 'FC_.*'"

# Check physical disk I/O load (IOPS / MBPS)
dcli -g cell_group -l celladmin "cellcli -e list metriccurrent where name like 'CD_IO_.*'"

Real-time I/O Overviews:

bash

# Display advanced physical device I/O statistics
iostat -xnz 1 10

ExaWatcher Metrics:

Navigate to /opt/oracle.ExaWatcher/archive to inspect historical snapshots of system performance (vmstat, mpstat, netstat) automatically gathered by the daemon.

3. ExaCC Tooling & Lifecycle Operations (dbaascli)

On ExaCC, traditional manual administration changes (like manual srvctl database creations or structural parameters changes) can break the OCI Control Plane sync. Always use dbaascli for lifecycle tasks.

Tooling Health Check:

bash

# Validates if the guest VM cloud automation framework is healthy
dbaascli diag checkhealth

Database Control:

bash

# Safe start/stop of instances through cloud automation
dbaascli database start --database <db_name>
dbaascli database stop --database <db_name>

PDB Management:

bash

# List or open pluggable databases cleanly

dbaascli pdb list --database <db_name>
dbaascli pdb open --pdb <pdb_name> --database <db_name>

Oracle Home Patching Pre-checks:

bash

# Test patch compliance before rolling deployment
dbaascli dbpatchm --run --action precheck --dbhome <db_home_name>

4. Advanced Troubleshooting & Diagnostic Bundles

When an incident hits, collect full data traces using standard Oracle Autonomous Health Framework (AHF) and TFA tools.

Generate an Exachk Report: Execute this monthly or right before/after infrastructure maintenance to review security and stability compliance against Oracle Maximum Availability Architecture standards:
bash
ahf check exachk -u -a
Collect TFA Diagnostic Bundles: Collect all cluster logs, OS logs, and database traces matching a specific timeframe:

bash

tfactl diagcollection --from "Oct/30/2026 22:00:00" --to "Oct/31/2026 01:00:00" --database <db_name>

Verify Interconnect Network:

# Validate cluster interconnect traffic stability across private RDS bonds
skgxpinfo

5. Critical Log and Trace File Locations

Knowing where files reside speeds up root-cause analysis during high-priority incidents.

Database & Grid Infrastructure Logs

Database Alert Log: /u01/app/oracle/diag/rdbms/<db_name>/<instance_name>/trace/alert_<instance_name>.log
GI Clusterware Alert Log: /u01/app/19.0.0.0/grid/log/<hostname>/alert<hostname>.log (Path varies slightly based on the current active GI Home release version).
CRS Daemon Trace Logs: /u01/app/oracle/diag/crs/<hostname>/crs/trace/

Exadata Tooling & OCI Control Plane Logs

Cloud Tooling Framework Log: /var/log/oracle/dbaascli.log (Primary log for dbaascli execution tracing).
OCI Agent & Orchestration Logs: /var/log/oracle/ocisubs.log and /var/log/oracle/dcs/agent/dcs-agent.log.
DBMCLI Logs: /opt/oracle/dbserver/dbms/deploy/config/alertlog/

Cell Alert Log: /opt/oracle/cell/cellsrv/deploy/log/alert.log
Management Server (MS) Log: /opt/oracle/cell/log/ms/ms_logs.log

Are you currently investigating a specific operational issue, such as an OCI control plane sync failure, ASM disk eviction, or an I/O performance slowdown? I can provide the explicit step-by-step diagnostic workflow for that scenario.

or

As an Exadata Cloud@Customer (ExaCC) DBA Expert, you manage a highly specialized cloud-adjacent ecosystem where cloud tooling (dbaascli) coexists with traditional Exadata infrastructure tools (dcli, cellcli, dbmcli).

Below is your production-ready, master reference guide containing the most critical commands, diagnostics, and log locations used daily for ExaCC management.

1. Daily Health Checks & Version Verification

Run these commands across compute nodes and storage cells daily to monitor infrastructure health and verify active patch versions.

Check infrastructure versions & histories:

bash

# Run on DB Nodes
imageinfo
imagediff

# Run on Cell Nodes via dcli from DB Node
dcli -g cell_group -l root "imageinfo"

Verify Clusterware & ASM Status:

bash

crsctl check cluster -all
crsctl stat res -t
asmcmd lsdsk -t

Overall Storage Cell & Alert Status:

bash

# Execute via dcli to hit all storage cells simultaneously
dcli -g cell_group -l celladmin "cellcli -e list cell detail"
dcli -g cell_group -l celladmin "cellcli -e list alerthistory where severity='critical' or active='true'"

Run Cloud Tooling Health Check:

bash

# Check OCI Cloud Tooling configuration and status on the Guest VM
dbaascli diag chkstatus

2. Performance Monitoring & Tuning

ExaCC performance issues usually involve Flash Cache efficiency or cell I/O saturation.

Monitor Smart Flash Cache Efficiency:

bash

# Check how hard your flash cache is working across cells
dcli -g cell_group -l celladmin "cellcli -e list metriccurrent where name like 'FC_.*'"

Check Real-time Cell I/O & IOPS:

bash

# Identify cells experiencing high I/O latency or load
dcli -g cell_group -l celladmin "cellcli -e list metriccurrent where name like '.*_IO_.*'"

ExaWatcher (Native Exadata OS Performance Daemon):

bash

# Generate charts or collect historical OS metrics (CPU, Memory, Network)
/opt/oracle.ExaWatcher/GetExaWatcherResults.sh -t "2026-10-30 08:00:00" -d 2h

3. Troubleshooting & Diagnostics Commands

When databases throw errors or tooling fails, use these commands to extract raw diagnostics or identify failed hardware.

Gather ExaCC Cloud Tooling Diagnostics:

bash

# Gathers Guest VM logs for Oracle OCI automation tooling
dbaascli diag collectlogs

Autonomous Health Framework (AHF/TFA):

bash

# Run a comprehensive diagnostic collection across nodes for a specific time window
ahf dsc -tfactl diagcollection -from "Oct/30/2026 06:00:00" -to "Oct/30/2026 08:00:00"

Identify Disk Failures on Cells:

bash

# Isolate degraded or prediction-failing physical/grid disks
dcli -g cell_group -l celladmin "cellcli -e list physicaldisk where status not 'normal'"
dcli -g cell_group -l celladmin "cellcli -e list griddisk where status not 'active'"

4. Critical Log Locations Map

Because ExaCC splits administration between standard Oracle Grid/DB layers, Exadata hardware, and OCI automated tooling, logs are separated across specific environments:

Component	Target Server Type	Absolute Log Path
OCI Cloud Tooling Log	Guest DB VM	`/var/log/oracle/dbaas/`
Dbaascli Actions Log	Guest DB VM	`/var/log/oracle/dbaas/dbaascli.log`
ExaCheck Automated Reports	Guest DB VM	`/opt/oracle.SupportTools/exachk/`
Grid Infrastructure Alert Log	Guest DB VM	`$GRID_BASE/diag/crs/<node>/crs/trace/alert.log`
Database Alert Log	Guest DB VM	`$ORACLE_BASE/diag/rdbms/<db_unique_name>/<instance>/trace/alert.log`
Storage Cell Server Alert Log	Storage Cell Node	`/opt/oracle/cell/cellsrv/deploy/log/alert.log`
Management Server (MS) Log	Storage Cell Node	`/opt/oracle/cell/log/diag/asm/cell/<cell_name>/trace/ms-odl.trc`

5. Automated System Configuration Verification

Run this system health utility monthly or prior to quarterly patching cycles.

Execute System-Wide Exachk:

bash

# Run from the primary database node as root/oracle to catch security or config drifts
/opt/oracle.SupportTools/exachk/exachk

1. Daily Operational Commands

These commands are used to check standard cluster status, grid infrastructure, and database health.

Cluster Status: Check the status of all cluster resources.

bash

crsctl stat res -t

Grid Infrastructure Stack: Verify the high-level health of the clusterware stack.

bash

crsctl check crs

ASM Disk Group Status: Check free/used space and status of ASM disk groups.

bash

asmcmd lsdg

Database & Instance Status: View the status of all databases managed by GI.

bash

srvctl status database -d <db_unique_name>

PDB Status: Quickly check Pluggable Database states via SQL*Plus.

sql

SELECT name, open_mode FROM v$pdbs;

2. Health Check Commands

Run these regularly to ensure the Exadata hardware and software layers are operating within normal parameters.

Exachk Utility: The absolute source of truth for ExaCC health compliance. Run this from the tool's directory (typically /opt/oracle.SupportTools/exachk).

bash

./exachk

Storage Server (Cell) Health: Check the status of the Exadata storage cells from a compute node using dcli.

bash

dcli -g dbs_group -l celladmin "cellcli -e list cell detail"

Grid Disks Status: Ensure all physical and grid disks are online.

bash

dcli -g dbs_group -l celladmin "cellcli -e list griddisk attributes name, status"

Flash Cache & Log Status: Verify Exadata Smart Flash Cache and Flash Log are functional.

bash

dcli -g dbs_group -l celladmin "cellcli -e list flashcache detail"
dcli -g dbs_group -l celladmin "cellcli -e list flashlog detail"

3. Performance Monitoring Commands

Use these commands to isolate performance bottlenecks across compute, network, and storage layers.

Exadata Cell Top (CellOS): Identify which cells or databases are driving the highest I/O load.

bash

dcli -g dbs_group -l celladmin "cellcli -e list metriccurrent where name like 'CL_.*'"

Storage Cell I/O Latency: Check real-time small/large read/write latencies.

bash

dcli -g dbs_group -l celladmin "cellcli -e list metriccurrent where name short_name like 'GD_IO_RQ_X_SM__SEC'"

OS Watcher Analyzer: ExaCC comes with OSWatcher pre-installed. Generate performance charts using the analyzer.

bash

java -jar oswba.jar -i /opt/oracle.oswatcher/osw/archive

Real-Time System Stats: Monitor standard OS performance on compute nodes.

bash

top -b -n 1 | head -n 20
iostat -xz 1 5
vmstat 1 5

4. Troubleshooting Commands

When an incident occurs (e.g., node eviction, performance degradation, network drop), use these triage commands.

TFA (Trace File Analyzer): Collect all relevant logs across the entire cluster for a specific time window to send to Oracle Support.

bash

tfactl diagcollect -from "yyyy-mm-dd hh:mm:ss" -to "yyyy-mm-dd hh:mm:ss"

Opatch Health & Inventory: Check for patch conflicts or current inventory.

bash

$ORACLE_HOME/OPatch/opatch lsinventory

CRS Visual Eviction Triage: Find out why a node evicted or crashed.

bash

tfactl analyze -search "eviction" -last 1d

5. Critical Log Locations

On ExaCC, diagnostic logs are split between the standard Oracle Database/GI paths and the Exadata-specific hardware/storage layers.

Database & Grid Infrastructure Logs

Database Alert Log: $ORACLE_BASE/diag/rdbms/<db_unique_name>/<instance_name>/trace/alert_<instance_name>.log

Clusterware (CRS) Alert Log: $GRID_BASE/diag/crs/<hostname>/crs/trace/alert.log

Grid Infrastructure Management Repository (GIMR/CHM): $GRID_BASE/diag/crs/<hostname>/crs/trace/ochmd.log

Exadata Storage Cell Logs (Accessible via celladmin user on cells)

Cell Server Alert Log: /opt/oracle/cell/cellsrv/deploy/log/alert.log

Management Server Log: /opt/oracle/cell/log/ms/ms-odl.log

Restart Server Log: /opt/oracle/cell/log/rs/rs.log

Cloud Automation Layers (OCI-C / ExaCC Control Plane)

DCS Agent Log (Database Cloud Service operations like backup/patching via OCI Console):

text

/opt/oracle/dcs/log/dcs-agent.log

1. Daily Health Checks & Status Verification

Use these commands first thing every day to ensure the infrastructure is healthy.

Check clusterware and CRS status:

bash

crsctl check crs
crsctl stat res -t

Verify Grid Infrastructure & Database software versions:

bash

ahf version

Verify Exadata storage server (Cell) status from a DB node:

bash

exacli -e "list cell attributes name, status"

Check overall Gi status and basic node health (as root):

bash

tfactl status

2. Performance Monitoring & Tuning

Run these commands to isolate CPU, memory, I/O, or network bottlenecks.

Monitor Exadata-specific real-time metrics (Flash cache, IOPS, throughput):

bash

# Check cell-level IOPS/Throughput via Exacli
exacli -e "list metriccurrent where objectType='CELL'"

# Monitor active database requests on cells
exacli -e "list activerequest"

Monitor OS-level resource utilization (CPU, memory, disk I/O):

bash

# Continuous OS performance monitor
oswatcher

# Check current real-time system stats
top -b -n 1 | head -n 20
vmstat 1 5
iostat -xz 1 5

Check ASM disk group space and performance:

bash

asmcmd lsdg
asmcmd iostat

3. Troubleshooting & Diagnostics

Use these tools when a node evicts, a database hangs, or performance drops sharply.

Generate a comprehensive Oracle AHF (TFA) diagnostic bundle:

bash

# Generate a summary of issues for the last 4 hours
tfactl diagcollection --from "4h ago"

# Run a proactive health check tailored for Exadata
orachk

Troubleshoot cluster network and interconnect issues:

bash

# Check network interface statistics
ip -s link

# Check cluster interconnect status via CRS
oifcfg getif

Examine detailed Exadata Cell alerts:

bash

exacli -e "list alerthistory where alertType='State' and severity='critical'"

4. Critical Log Locations & Paths

On ExaCC, paths conform to standard Oracle Flexible Architecture (OFA), but remember that Oracle manages the infrastructure layer, while you manage the VM layer.

Grid Infrastructure & Clusterware Logs

GI Alert Log: /u01/app/grid/diag/crs/<hostname>/crs/trace/alert.log

Clusterware Daemon Logs (CSSD, CRSD): /u01/app/grid/diag/crs/<hostname>/crs/trace/

Eviction/Reboot logs: /var/log/messages and /u01/app/grid/diag/crs/<hostname>/crs/trace/ocssd.trc

Database Logs

Database Alert Log: /u01/app/oracle/diag/rdbms/<db_unique_name>/<SID>/trace/alert_<SID>.log

Listener Log: /u01/app/grid/diag/tnslsnr/<hostname>/<listener_name>/trace/<listener_name>.log

Automatic Diagnostic Repository (ADR) & AHF Locations

TFA Repository/Output Path: /opt/oracle/dcs/oracle.ahf/data/<repository>

OSWatcher Archive Logs: /u01/app/grid/repository/oracle.oswatcher/osw/archive/ (or under /opt/oracle)

Direct Command Comparison Matrix

Objective	Command / Tool	Level	Primary Purpose
Grid Status	`crsctl stat res -t`	VM Node	Displays real-time status of all cluster resources.
Storage Health	`exacli -e "list cell"`	Storage	Checks availability and state of Exadata storage servers.
Log Gathering	`tfactl diagcollection`	Automation	Bundles logs across DB, GI, and OS for Oracle Support.
Space Check	`asmcmd lsdg`	ASM	Identifies free, usable space across Exadata diskgroups.
Real-time Performance	`top` / `exacli`	OS / Cell	Isolates CPU hogs or high-latency cells immediately.

1. How to Check ExaCC Server Uptime Like an Expert

To determine how long your ExaCC components have been running without causing a performance impact, you must check both the Database Compute Nodes and the Storage Servers.

A. Database Compute Nodes (VMs)

Execute standard Linux uptime queries using the Distributed Command Line Interface (dcli) from one database node to check all cluster nodes simultaneously:

bash

dcli -g dbs_group -l oracle "uptime"

Expert Tip: To see exactly when the systems were booted up rather than a dynamic timer, append the -s flag:

bash

dcli -g dbs_group -l oracle "uptime -s"

B. Storage Cells (Remote Monitoring via ExaCLI)

On ExaCC, you do not log directly into the storage cells via SSH. Instead, you use the exacli utility pre-authenticated with your cloud user identity:

bash

exacli -l cloud_user_[clustername] -e "LIST CELL DETAIL"

Look specifically for the upTime attribute in the text block output. To isolate this across all cells using dcli:

bash

dcli -g cell_group -l oracle "exacli -e 'LIST CELL ATTRIBUTES name, upTime, status'"

2. Daily Health Check Commands Cheat Sheet

Run these foundational health checks every day to catch infrastructure and software issues proactively.

A. OCI Cloud Agent & Control Plane Status

Ensure that the internal VM automation tool kit can communicate with the Oracle Cloud Infrastructure (OCI) plane:

bash

# Check the status of the DBaaS provisioning engine agent
dbaascli agent status

# Check the Autonomous Health Framework (AHF) status
ahfctl statusahf

B. Grid Infrastructure & Cluster Status

Verify that the clusterware layer is completely stable across all nodes:

bash

# Check absolute health of CSS and CRS daemons
crsctl check cluster -all

# View high-level summarized resource state across all nodes
crsctl stat res -t

C. Storage Engine Health

Use exacli (or dcli targeting your cell groups) to watch out for active hardware alerts or failing storage drives:

bash

# List all active critical or warning exceptions
exacli -e "LIST ALERTHISTORY WHERE severity='critical' AND alertType='State'"

# Check structural statuses of physical drives and software griddisks
exacli -e "LIST PHYSICALDISK ATTRIBUTES name, status, luns"
exacli -e "LIST GRIDDISK ATTRIBUTES name, status, size"

D. Automatic Database Diagnostic (ADDM) & System Benchmarks

Run a localized system framework check via the automated tool kit daemon:

bash

# Check daemon health status for standard baseline collection
exachk -d status

1. Check ExaCC Server Uptime (Expert Approach)

To check the uptime across all nodes in a group using the root user, run the following command. The -q (quiet) flag is often added to format the output cleanly.

bash

dcli -I root -g similiar 'uptime'

Expert Tip: If you want to sort or filter the output to quickly spot a node that recently rebooted, pipe the output to evaluate the days/hours:

bash

dcli -I root -g similiar 'uptime' | awk '{print $1, $4, $5}'

2. Daily Operations & Performance Monitoring

These commands help you monitor active performance and resource utilization across your ExaCC infrastructure daily.

CPU & Memory Utilization: Check the current system load and memory bottlenecks.
bash
dcli -I root -g similiar 'top -b -n 1 | head -n 20'
Storage Space (OS Level): Monitor root and diagnostic partition space to prevent full disks.
bash
dcli -I root -g similiar 'df -h /'
Active Network Connections: Monitor network socket usage.
bash
dcli -I root -g similiar 'ss -s'

3. Infrastructure Health & Hardware Checks

Run these health check commands to verify hardware integrity and alert status across the cells.

Cell Server Status: Verify if the cell services (cellsrv, ms, rs) are running normally.
bash
dcli -I root -g similiar 'cellcli -e list cell attributes gsn, status'
Open Alerts: Check for any active hardware or software alerts on the storage servers.
bash
dcli -I root -g similiar 'cellcli -e "list alerthistory where alertState==\"active\""'
Physical Disk Health: Instantly check the status of all physical hard drives and flash cards.

bash

dcli -I root -g similiar 'cellcli -e list physicaldisk attributes name, status, lunit'

Grid Disk Status: Ensure all grid disks are "online" and available to ASM.

bash

dcli -I root -g similiar 'cellcli -e list griddisk attributes name, status'

4. Expert dcli Execution Flags Reference

-I root: Authenticates as the root user using pre-configured SSH keys.
-g similiar: Targets a custom group file named similiar containing the list of IP addresses or hostnames.
-x: Explodes or runs the command concurrently instead of sequentially (great for saving time across large clusters).
-l: Allows you to specify a different user if you ever need to run as oracle or grid.

To check the uptime of all Exadata Cloud@Customer (ExaCC) database servers simultaneously as an expert using dcli, you can use the following optimized command:

bash

dcli -g dbs_group -l root "uptime"

Expert Syntax Breakdown & Variations

Group File (-g): Replace dbs_group with the path to your text file containing the list of database server hostnames or IP addresses (e.g., /home/oracle/dbs_group or the default /opt/oracle.SupportTools/onecommand/dbs_group).
User Login (-l): Use -l root instead of -I root. In dcli, -l specifies the remote login user.

If you want a cleaner, more structured output that filters out everything except the hostname and the uptime duration, you can pipe the command through awk:

bash

dcli -g dbs_group -l root "uptime" | awk -F':' '{print $1 $2 ":" $3}'

Alternative: Checking Storage Cells Uptime

If you are an infrastructure administrator and also need to check the uptime of the underlying ExaCC storage servers (cells), switch the group file to your cell group:

bash

dcli -g cell_group -l root "uptime"

Troubleshooting Tip

If you encounter SSH key exchange or connectivity issues while running this across nodes, you can force dcli to show detailed SSH output by appending the unhashed option:

bash

dcli -g dbs_group -l root -x "uptime"

The -I flag tells dcli to log in as a specific user (in this case, root), and the -g flag specifies a file containing a list of target servers (group file).

Here is the proper syntax and a breakdown of how an expert uses this command on an Exadata Cloud@Customer (ExaCC) or on-premises Exadata environment:

Core Syntax

bash

dcli -g <group_file> -l <user> <command>

Note: While -I works in certain versions to specify the login user, -l (lowercase L) is the standard, universally supported flag across all Exadata software versions to define the login user.

Real-World Expert Examples

1. Checking Cell Node Status (Standard Group File)

If you have a file named cell_group containing your storage cell hostnames, you would run:

bash

dcli -g cell_group -l root "cellcli -e list cell"

2. Checking Free Space Across All Storage Cells

To check the physical and logical space across your Exadata storage tier:

bash

dcli -g cell_group -l root "cellcli -e list celldisk attributes name, size, freespace"

3. Checking Flash Cache Status and Alert Logs

To see if any cells have active alerts:

bash

dcli -g cell_group -l root "cellcli -e list alerthistory where alertType='State' and severity='critical'"

4. Running a Command on Database Nodes (instead of Cells)

If you want to run a command across all DB nodes (e.g., checking memory usage), you point to your dbs_group file:

bash

dcli -g dbs_group -l root "free -g"

Quick Troubleshooting Checklist for Experts

SSH Key Exchange: dcli relies on passwordless SSH. If it prompts for a password, run dcli -g cell_group -k first to push the SSH keys to the target nodes.
Quotation Marks: Always wrap the target command in double quotes (" ") so that pipes (|), redirects (>), or variables ($) are evaluated on the remote nodes, not your local terminal.
Aborting Hanging Commands: Use the -t flag to set a timeout (e.g., -t 10 for 10 seconds) if you suspect a remote node is unresponsive, preventing the dcli command from hanging indefinitely.

Q1: What is the primary purpose of the dbaascli utility in ExaCC, and how does dbhome fit into it?

Answer:
dbaascli (Database as a Service Command Line Interface) is the official cloud tooling wrapper provided by Oracle to manage database lifecycle operations in ExaCC/ExaCS.

While the Oracle Cloud Infrastructure (OCI) web console is used for most tasks, dbaascli executes the actual changes under the hood on the VM nodes. The dbhome group of commands specifically handles the automation, tracking, creation, deletion, and inventory auditing of ORACLE_HOME environments across the RAC compute nodes.

Q2: How do you identify all registered Oracle Homes and their respective versions on an ExaCC node using dbaascli?

Answer:
You log in to the virtual machine as the opc user, switch to root, and execute:

bash

# dbaascli dbhome info

Behavior: If you press Enter when prompted, it lists all registered Oracle Homes with their names, paths, and active status. You can also pass a specific home name to extract detailed information for just that target environment.

Q3: Explain how you would create a new Oracle Database Home using dbaascli. What happens behind the scenes?

Answer:
To provision a brand-new, ready-to-use Oracle Home, you run:

bash

# dbaascli dbhome create --version 19.0.0.0

Behind the scenes: dbaascli interacts with the local Cloud Software Library (cswlib). It automatically extracts the latest certified Release Update (RU) image bundle onto the ACFS/local file system, sets up the required inventory pointers across all cluster nodes, and configures the default permissions.

Q4: What is the difference between "In-Place" and "Out-of-Place" patching using dbaascli? Which commands are used?

Answer:
The two methodologies differ significantly in risk profile and downtime:

In-Place Patching (dbaascli dbhome patch): Applies the Release Update directly onto the existing Oracle Home. Databases running out of this home will experience rolling or non-rolling downtime while binaries are swapped and datapatch runs.

Out-of-Place Patching (dbaascli database move): The preferred cloud approach. You first provision a new separate ORACLE_HOME with the higher patch level via dbhome create. You then migrate the live database instances using:

bash

# dbaascli database move --oracleHome /u02/app/oracle/product/19.0.0.0/dbhome_2 --dbname cdb1

This minimizes production downtime to the duration of an instance restart.

Q5: How do you clean up an obsolete or unlinked Oracle Home in ExaCC safely? What does the cloud tool validate before doing this?

Answer:
To drop an unused Oracle Home, you must use the purge command:

bash

# dbaascli dbhome purge

Validations: Before deleting anything, dbaascli checks the central OCI agent metadata and the local cluster configurations to ensure no active database instances are running or linked to that target directory. If it is completely unlinked, it purges the binaries from the OS layer and scrubs the home from the platform registry.

Q6: If a dbaascli database move or dbhome patch command fails halfway through (e.g., due to a space error during datapatch), how do you handle it?

Answer:

Do not manually run local sql scripts or fix inventory paths by hand, as this breaks OCI tracking sync.

Review the automated execution logs generated in /var/opt/oracle/log/dbaascli/ or /var/opt/oracle/log/exadbcms/ to isolate the error (e.g., tablespace exhaustion).

Remediate the root cause (e.g., extend the tablespace or fix file systems).

Resume the exact cloud automation session using the --resume flag and the failed session ID reported by the utility:

bash

# dbaascli database move --oracleHome <path> --dbname <db> --sessionID <ID> --resume

Summary Checklist for Interactivity

Action	Target Command	Core Parameter Required
Check Environment	`dbaascli dbhome info`	None (interactive or specific Home Name)
Deploy Binaries	`dbaascli dbhome create`	`--version` (e.g., `19.0.0.0`)
In-Place RU Apply	`dbaascli dbhome patch`	`--oracleHome`
Migrate DB to Home	`dbaascli database move`	`--dbname` and `--oracleHome`
Drop Unused Binaries	`dbaascli dbhome purge`	Home Name or Home Path

The /bin/dbaascli utility is a cloud tooling command-line interface used to manage lifecycle and administration tasks for Oracle databases directly from an Exadata Cloud@Customer (ExaCC) or Exadata Cloud Infrastructure (ExaCS) virtual machine.

To run most of these commands, you must connect to a database compute node as the opc user and switch to the root user (sudo -s).

The primary dbaascli database subcommands available on ExaCC include:

1. Check Database Status

Displays the database open mode, software release edition, and running components.

bash

/bin/dbaascli database status --dbname <database_name>

2. Create a Database

Provisions a Container Database (CDB) or a Non-CDB inside a pre-existing Oracle Home.

Create a CDB:

bash

/bin/dbaascli database create --dbName <database_name> --oracleHome <path_to_oracle_home>

Use code with caution.

Create a Non-CDB:

bash

/bin/dbaascli database create --dbName <database_name> --oracleHome <path_to_oracle_home> --createAsCDB false

3. Start, Stop, and Restart (Bounce)

Controls the database instance state across the Exadata cluster nodes.

Start: /bin/dbaascli database start --dbname <database_name>

Stop: /bin/dbaascli database stop --dbname <database_name>

Bounce: /bin/dbaascli database bounce --dbname <database_name>

4. Manage Backups

Configures, triggers, or views details about the automated cloud backup framework.

Get Backup Config:

bash

/bin/dbaascli database backup --getConfig --dbname <database_name>

5. Modify Initialization Parameters

Updates or resets SPFILE static/dynamic parameters in batch mode.

bash

/bin/dbaascli database modifyParameters --dbname <database_name> --setParameters "parameter1=value1,parameter2=value2" --allowBounce

6. Change Database Passwords

Updates administrative credentials like SYS or the Transparent Data Encryption (TDE) wallet password.

bash

/bin/dbaascli database changepassword --dbname <database_name>

Tip: To see the full checklist of arguments and real-time help syntax for any command block, append --help to your query:

bash

/bin/dbaascli database --help
/bin/dbaascli database create --help


In Oracle Exadata Database Service on Cloud@Customer (ExaCC), the /bin/dbaascli diag command group is used as root to diagnose, run health checks, and gather logs from the Guest VM cloud tooling stack. [1, 2]
The utility supports two principal subcommands for troubleshooting lifecycle or management automation failures: [1]
1. dbaascli diag collect
This command initiates automated diagnostic log collection across your ExaCC environment. Running it on a single node seamlessly coordinates log retrieval from all cluster nodes. [1, 2]
Basic Execution:bash
# dbaascli diag collect

Advanced Usage with Options:bash
# dbaascli diag collect --components db|gi|os|dbaastools|all --startTime YYYY-MM-DDTHH:MM:SS --endTime YYYY-MM-DDTHH:MM:SS --destLocation /path/to/output

Key Flags:--components: Filters the scope of collection. Accepts db, gi (Grid Infrastructure), os, dbaastools, or all.
--destLocation: The target directory on the Guest VM to drop the generated archive. The default directory is /var/opt/oracle/dbaas_acfs.
--waitForCompletion: Defaults to true. Set to false if you want the job to process asynchronously in the background while monitoring progress via dbaascli job getStatus. [1, 2]
2. dbaascli diag healthCheck
This command invokes a programmatic cloud tooling configuration review across all nodes to identify systemic configuration drift, permissions errors, or corrupted cloud infrastructure tracking repositories. [1, 2]
Execution:bash
# dbaascli diag healthCheck
Use code with caution.
Output files generated by this check are automatically staged inside compressed .tar.gz archives within your destination location, 
ready to accompany an Oracle Support Request (SR). 
The /bin/dbaascli dbhome command is used to manage Oracle Database homes in an Oracle Exadata Cloud@Customer (ExaCC) environment. It allows database administrators to perform lifecycle management tasks directly from the command line.
Here is a grouped list of the primary operations you can perform with this command:
Creation and Deletion
create: Installs a new Oracle Database home.
purge: Removes a specific database home from the system.
Patching and Updates
patch: Applies Release Updates (RUs) or interim patches to a database home.
rollback: Rolls back a previously applied patch.
update: Updates the database home software version.
Information and Status
get-details: Displays specific configuration information for a database home.
list: Shows all database homes provisioned on the cluster.



Interviewer: "We are facing an intermittent node eviction or severe OS performance degradation on an ExaCC database node (or storage cell). How would you collect and analyze the operating system metrics for that specific window?"
Candidate (Your Answer):
"I would use ExaWatcher, specifically by executing the utility script GetExaWatcherResults.sh. It automatically aggregates OS and network performance data (like iostat, vmstat, top, and cellsrvstat) into an extracted bundle. I would extract the metrics covering the exact problematic timeframe and review the automatically generated HTML charts to identify spikes in CPU, memory saturation, or I/O latency." [1, 2, 3, 4]
 Top Technical Questions & Answers
Q1: Where is the script located and how do you execute it?
Answer: The script is located in the /opt/oracle.ExaWatcher/ directory on both Database (Compute) nodes and Storage Cells. It must be executed as the root user. [1, 2, 3]
bash
cd /opt/oracle.ExaWatcher/
./GetExaWatcherResults.sh --from MM/DD/YYYY_HH:MM:SS --to MM/DD/YYYY_HH:MM:SS
Q2: What are the primary arguments used with GetExaWatcherResults.sh?
Answer:
--from: The start time of the incident (Required).
--to: The end time of the incident (Highly recommended to narrow the output).
--resultdir: Optional parameter to specify a custom target directory if the default partition lacks space. [1, 2, 3]
Q3: What is the default output format, and where is it saved?
Answer: The script packages the data into a compressed tarball/bzip2 archive (.tar.bz2). By default, it drops the file into /opt/oracle.ExaWatcher/archive/ExtractedResults/. [1, 2, 3]
Q4: How do you read or analyze the output of the script?
Answer:
Uncompress and untar the generated file (tar -xvf ExaWatcher_...tar).
Inside the extracted directory, look for a folder named Charts.ExaWatcher.<hostname>.
Open the index.html file in any standard web browser. It populates interactive charts mapping out CPU utilization, I/O summary, memory metrics, and cell server statistics for that specific timeframe. [1, 2, 3, 4, 5]
Q5: What is the underlying data collection interval of ExaWatcher compared to standard OSWatcher?
Answer: ExaWatcher is customized for high-frequency Exadata workloads. While standard OSWatcher typically captures metrics every 30 to 60 seconds, ExaWatcher captures data at a granular 5-second interval, making it highly effective at catching sudden microsecond CPU spikes or transient I/O hangs. [1]
 Pro-Tips to Sound Like an Expert (ExaCC Specific)
Space Management Warning: Mention that running GetExaWatcherResults.sh over a long period (e.g., 24+ hours) can generate massive files. Emphasize that you always check disk space (df -h) on /opt before running it, or redirect the output using --resultdir to a safe mount point like /u02 or /tmp. [1, 2, 3]
Storage Cells vs DB Nodes: Clarify that ExaWatcher runs on both layers. If the issue is a slow query, you run it on the DB node; if the issue is high cell disk service times, you run it directly on the Storage Node to capture cellsrvstat details. [1, 2]


Below is the standard interview question breakdown, what the output means, and why it is critical for ExaCC environments.
 The Interview Question
Question: What is the purpose of the /sbin/vgdisplay command on an ExaCC Database Node, and what key metrics do you look for when managing database storage space?
Answer:
The /sbin/vgdisplay command is used to display detailed attributes of Volume Groups (VG) configured on the Linux OS layer of the ExaCC Database Node. While the actual database files reside on ASM (Automatic Storage Management) inside Exadata Grid Infrastructure, local paths like /, /u01, /oracle, and the grid homes are managed via LVM. [1, 2, 3, 4]
When running this command, an administrator primarily looks for the Free PE (Physical Extents) metric to determine if there is enough unallocated space in the volume group to safely extend a critical system or software filesystem (like /u01) when it runs out of space. [1, 2]
 Key Metrics Evaluated in the Output
When you run /sbin/vgdisplay <volume_group_name>, the interviewer will expect you to know what these crucial fields represent: [1]
Field Name Description & Relevance to ExaCC
VG Name The name of the volume group (commonly VGExaDb or rootvg on Exadata nodes).
PE Size The size of each Physical Extent block (typically 4MB or larger). This is the smallest unit LVM can allocate.
Total PE The total number of allocatable blocks inside this volume group.
Alloc PE / Size The amount of space currently dedicated to existing Logical Volumes (like /, /var, /u01).
Free PE / Size The most critical metric. It shows the remaining unallocated space in the pool that you can use to extend filesystems online using lvextend.
 Scenario-Based Follow-Up Questions
Interviewers will often pivot from the basic command to a real-world scenario to test your system administration skills:
Scenario A: The /u01 filesystem on an ExaCC node is 99% full, causing patch deployment or trace generation to fail. How do you resolve this using LVM?
Check Available Space Pool: Run /sbin/vgdisplay to see if there is unallocated space under Free PE / Size. [1, 2]
Extend the Logical Volume: If free space exists, extend the logical volume online (e.g., adding 20 GB):bash
sudo /sbin/lvextend -L +20G /dev/VGExaDb/LVOraHome

Resize the Filesystem: Expand the underlying filesystem so the OS recognizes the new space.For XFS (Default on modern Exadata images):bash
sudo /sbin/xfs_growfs /u01
For Ext4:
bash
sudo /sbin/resize2fs /dev/VGExaDb/LVOraHome

Scenario B: What is the difference between vgdisplay, pvdisplay, and lvdisplay?
pvdisplay: Shows information about the raw physical storage devices/partitions (/dev/nvme* or /dev/sd*) assigned to LVM.
vgdisplay: Shows the combined, centralized storage pool (Volume Group) details.
lvdisplay: Shows individual slice/partition metrics (Logical Volumes) created out of the volume group that hold the actual file systems. [1, 2]

Q: What is the imageinfo command used for in Exadata/ExaCC?A: It is executed with root or sudo privileges on a compute (database) node to check the exact Exadata software version, OS kernel version, activation timestamp, and whether the image status is successful. [1, 2]
Q: Where do you run imageinfo—on the database node or the storage cell?A: You run imageinfo on the database compute node. To check version details on a storage cell server instead, you use cellcli commands like cellcli -e "list cell details". [1]
Q: What key details does imageinfo output?
A: It outputs the active Exadata image version, tracking components, GI/database node pairing information, and rollback/success status of the software image. [1]

Question 1: What is the purpose of the /usr/bin/exachk tool, and why is it critical in an ExaCC environment?
Answer:
exachk is a holistic health check tool designed specifically for Oracle Engineered Systems like Exadata. In an ExaCC environment, it acts as the primary tool to ensure the entire system aligns with Oracle’s Maximum Availability Architecture (MAA) and best practices. [1]
It performs configuration and software health audits across all layers: []
Database Servers (Compute Nodes): OS parameters, Grid Infrastructure, ASM, and RDBMS configurations.
Storage Servers (Cells): Cell software, physical/grid disks, flash cache, and IORM setup.
Network Components: RDMA Network Fabric / RoCE switches and InfiniBand switches. [1, 2, 3]
Why it's critical for ExaCC: It generates a comprehensive HTML health report detailing warnings, critical alerts, and failure points. Running exachk is mandatory before and after lifecycle operations like quarterly infrastructure or database patching to ensure the cloud system remains stable. [1]
Question 2: As an Exadata/Cloud DBA, at what specific times are you required to execute exachk?
Answer:
You should run exachk during the following crucial milestones:
Pre-Patching / Pre-Upgrade Readiness: To identify software dependencies, configuration anomalies, or known critical issues that could cause an upgrade or patch to fail. [1, 2]
Post-Patching Verification: To verify that the environment is in a healthy, operational state and no cluster parameters were inadvertently altered. [1]
Routine Maintenance (Quarterly/Monthly): To preemptively capture configuration drifts, hardware warnings, or performance degradations. []
Before Opening a Service Request (SR) with Oracle: Oracle Support almost always requests a fresh exachk run in debug mode to accelerate troubleshooting. [1]
Question 3: How do you execute exachk, and what are the most common command-line options you use?
Answer:
Typically, exachk is run out of the Autonomous Health Framework (AHF) path or standard /usr/bin/exachk directory. It is recommended to run it as the root user so it can discover and execute checks seamlessly across the network fabric and storage layers. [1, 2, 3]
bash
# Standard interactive execution (discovers all components automatically)
./exachk
Crucial Command Line Options:
-a: Runs all default best-practice checks and recommended patch checks.
-debug: Runs the utility in verbose mode. It packages a debug log inside the final .zip file for Oracle Support.
-localonly: Forces the tool to run checks only on the local compute node where it is executed, skipping network discovery of other cells or nodes.
-clusternodes <nodes> / -cells <storage_cells>: Allows you to limit the scan to a specific subset of compute or storage nodes. [1, 2, 3, 4]
Question 4: How does exachk execute tasks on remote components like Storage Cells and Network Switches? What configurations are required?
Answer:
exachk uses passwordless SSH connections initiated from the primary compute node to communicate with remote components. [1]
For Storage Servers (Cells), it communicates via root or an elevated execution user (defined by the RAT_CELL_SSH_USER environment variable).
For Network Switches (InfiniBand/RoCE), it connects using root or nm2user. 
If passwordless SSH is broken or blocked between the compute nodes and the storage cells, exachk will fail to run the storage layer audits, resulting in incomplete reports.
Question 5: If an exachk report shows a "FAIL" status for an OS kernel parameter or an initialization parameter on ExaCC, how do you handle it?
Answer:
Analyze the Recommendation: Review the detailed mitigation steps provided inside the exachk HTML report. []
ExaCC Shared Responsibility Context: Because ExaCC is a hybrid cloud model, I must distinguish between customer-managed layers and Oracle-managed layers.If it is a Database/RDBMS parameter (e.g., db_block_checking), I can fix it directly as the oracle user via ALTER SYSTEM.
If it is a Grid Infrastructure/OS parameter inside the DomU (Guest VM), I can fix it as the root user using standard procedures.
If the failure points to the underlying hardware, hypervisor (Dom0), or storage cells, I must open an Oracle Service Request (SR), as those components are managed solely by Oracle Cloud Operations. [1, 2]
Question 6: What is the relationship between exachk and orachk?
Answer:
Both tools belong to the Oracle Autonomous Health Framework (AHF).
orachk is utilized for standard, non-engineered Oracle hardware ecosystems (e.g., standalone servers, standard RAC clusters, ODA).
exachk is explicitly compiled with extra awareness for Exadata architectures,

Field Name	Description & Relevance to ExaCC
VG Name	The name of the volume group (commonly `VGExaDb` or `rootvg` on Exadata nodes).
PE Size	The size of each Physical Extent block (typically 4MB or larger). This is the smallest unit LVM can allocate.
Total PE	The total number of allocatable blocks inside this volume group.
Alloc PE / Size	The amount of space currently dedicated to existing Logical Volumes (like `/`, `/var`, `/u01`).
Free PE / Size	The most critical metric. It shows the remaining unallocated space in the pool that you can use to extend filesystems online using `lvextend`.

 extending its capabilities to audit storage cells, Exadata storage software, 
InfiniBand/RoCE fabrics, and virtualization components unique to the Exadata framework. 


In an Oracle Exadata Cloud at Customer (ExaCC) interview, understanding the separation of duties, user access, and resource isolation models is critical. dbaadmin, DRM, and exadba represent foundational concepts across OS-level administration, Cloud automation, and database resource management. 
1. oracle vs. grid vs. exadba vs. dbaadmin (OS-Level User Access)
Question: What is the difference between the exadba user and the dbaadmin user in an ExaCC environment? Who owns the software binaries?
Answer:oracle / grid: These traditional users own the Oracle Database and Grid Infrastructure software binaries inside the VM. However, direct ssh access as oracle or grid is blocked for customers by default.
exadba: This is the default customer operating system user used to connect to the ExaCC virtual machines (VMs) via SSH. It has partial sudo privileges allowing you to switch to the oracle or grid users to perform standard database administration tasks (e.g., using SQL*Plus, srvctl, crsctl).
dbaadmin: This is a specialized internal automation user managed by Oracle's cloud tooling (like the OCI agent). It is utilized to trigger automated lifecycle management tasks such as cloud-based patching, automated backups, and scaling from the Oracle Cloud Infrastructure (OCI) console.
Question: As an Exadata DBA, which OS user do you use to perform daily operations on ExaCC, and how do you access it?
Answer: You use the exadba user. You access it by logging into the database nodes via SSH using the customer-generated private key associated with the ExaCC VM cluster. Once logged in, you can execute sudo su - oracle or sudo su - grid to administer the database and clusterware environment.
2. DRM (Instance / Cluster Management vs. Database Resource Management)
Depending on the context of the question, DRM in an Exadata/ExaCC interview can stand for Distributed Read Management (Instance/RAC level) or Database Resource Manager (Workload management). 
Context A: Distributed Read Management (RAC Architecture)
Question: What is the role of DRM (Distributed Read Management / Dynamic Resource Management) in an Oracle RAC environment running on Exadata?
Answer: Distributed Read Management (DRM)—often referred to as Dynamic Resource Management—is a background mechanism in Oracle RAC that automatically re-masters database blocks. If Node 1 frequently accesses a set of blocks that are currently mastered on Node 2, DRM dynamically shifts the mastership of those resources to Node 1. This minimizes cluster interconnect traffic across the high-speed network, reducing latency.
Context B: DBRM (Database Resource Manager) vs. IORM
Question: How does DBRM (Database Resource Manager) differ from IORM in Exadata, and how do they work together?
Answer:DBRM (Database Resource Manager): Operates at the database instance level on the compute nodes. It controls CPU distribution, parallel execution servers, and session limits among different consumer groups within a single database.
IORM (I/O Resource Manager): Operates at the Exadata Storage Server level (Cells). It regulates physical disk and flash I/O allocation between different databases or inter-database consumer groups.
Integration: DBRM passes the consumer group information inside the data blocks via the iDB protocol down to the storage cells. IORM maps this tag and ensures that the storage cells prioritize or throttle the physical I/O according to the defined resource plan. 
3. Scenario-Based Scenario Summary Matrix
Feature/Component Primary Responsibility Target Location Managed By
exadba Customer CLI administration (DBA tasks) Compute Node (VM) Customer via SSH
dbaadmin Internal cloud automation agent execution Compute Node (VM) OCI Cloud Tooling
DRM (RAC) Dynamic block remastering to optimize interconnect Cluster-wide Oracle Grid Infrastructure
DBRM Managing CPU & parallelism for resource groups Compute Node / DB Instance Database Administrator

EasyReliableDBA

Wednesday, 2 September 2026

Exacc Commands and Utilities

or

Search This Blog

Feature/Component	Primary Responsibility	Target Location	Managed By
`exadba`	Customer CLI administration (DBA tasks)	Compute Node (VM)	Customer via SSH
`dbaadmin`	Internal cloud automation agent execution	Compute Node (VM)	OCI Cloud Tooling
`DRM` (RAC)	Dynamic block remastering to optimize interconnect	Cluster-wide	Oracle Grid Infrastructure
`DBRM`	Managing CPU & parallelism for resource groups	Compute Node / DB Instance	Database Administrator