Exacc Commands and Utilities

Q1: What is the primary purpose of the dbaascli utility in ExaCC, and how does dbhome fit into it?

Answer:
dbaascli (Database as a Service Command Line Interface) is the official cloud tooling wrapper provided by Oracle to manage database lifecycle operations in ExaCC/ExaCS. [1, 2]

While the Oracle Cloud Infrastructure (OCI) web console is used for most tasks, dbaascli executes the actual changes under the hood on the VM nodes. The dbhome group of commands specifically handles the automation, tracking, creation, deletion, and inventory auditing of ORACLE_HOME environments across the RAC compute nodes. [1, 2, 3]

---

Q2: How do you identify all registered Oracle Homes and their respective versions on an ExaCC node using dbaascli?

Answer:
You log in to the virtual machine as the opc user, switch to root, and execute: [1]

bash

# dbaascli dbhome info

Use code with caution.

Behavior: If you press Enter when prompted, it lists all registered Oracle Homes with their names, paths, and active status. You can also pass a specific home name to extract detailed information for just that target environment. [1]

---

Q3: Explain how you would create a new Oracle Database Home using dbaascli. What happens behind the scenes?

Answer:
To provision a brand-new, ready-to-use Oracle Home, you run: [1, 2]

bash

# dbaascli dbhome create --version 19.0.0.0

Use code with caution.

Behind the scenes: dbaascli interacts with the local Cloud Software Library (cswlib). It automatically extracts the latest certified Release Update (RU) image bundle onto the ACFS/local file system, sets up the required inventory pointers across all cluster nodes, and configures the default permissions. [1, 2, 3]

---

Q4: What is the difference between "In-Place" and "Out-of-Place" patching using dbaascli? Which commands are used?

Answer:
The two methodologies differ significantly in risk profile and downtime:

In-Place Patching (dbaascli dbhome patch): Applies the Release Update directly onto the existing Oracle Home. Databases running out of this home will experience rolling or non-rolling downtime while binaries are swapped and datapatch runs. [1]
Out-of-Place Patching (dbaascli database move): The preferred cloud approach. You first provision a new separate ORACLE_HOME with the higher patch level via dbhome create. You then migrate the live database instances using:
bash
# dbaascli database move --oracleHome /u02/app/oracle/product/19.0.0.0/dbhome_2 --dbname cdb1
Use code with caution.
This minimizes production downtime to the duration of an instance restart. [1, 2, 3, 5]

---

Q5: How do you clean up an obsolete or unlinked Oracle Home in ExaCC safely? What does the cloud tool validate before doing this?

Answer:
To drop an unused Oracle Home, you must use the purge command: [1]

bash

# dbaascli dbhome purge

Use code with caution.

Validations: Before deleting anything, dbaascli checks the central OCI agent metadata and the local cluster configurations to ensure no active database instances are running or linked to that target directory. If it is completely unlinked, it purges the binaries from the OS layer and scrubs the home from the platform registry. [1]

---

Q6: If a dbaascli database move or dbhome patch command fails halfway through (e.g., due to a space error during datapatch), how do you handle it?

Answer:

Do not manually run local sql scripts or fix inventory paths by hand, as this breaks OCI tracking sync.
Review the automated execution logs generated in /var/opt/oracle/log/dbaascli/ or /var/opt/oracle/log/exadbcms/ to isolate the error (e.g., tablespace exhaustion).
Remediate the root cause (e.g., extend the tablespace or fix file systems).
Resume the exact cloud automation session using the --resume flag and the failed session ID reported by the utility:
bash
# dbaascli database move --oracleHome <path> --dbname <db> --sessionID <ID> --resume
Use code with caution.
[1, 2, 3]

---

Summary Checklist for Interactivity

Action	Target Command	Core Parameter Required
Check Environment	dbaascli dbhome info	None (interactive or specific Home Name)
Deploy Binaries	dbaascli dbhome create	--version (e.g., 19.0.0.0)
In-Place RU Apply	dbaascli dbhome patch	--oracleHome
Migrate DB to Home	dbaascli database move	--dbname and --oracleHome
Drop Unused Binaries	dbaascli dbhome purge	Home Name or Home Path

 The /bin/dbaascli utility is a cloud tooling command-line interface used to manage lifecycle and administration tasks for Oracle databases directly from an Exadata Cloud@Customer (ExaCC) or Exadata Cloud Infrastructure (ExaCS) virtual machine. [1, 2]

To run most of these commands, you must connect to a database compute node as the opc user and switch to the root user (sudo -s). []

The primary dbaascli database subcommands available on ExaCC include: [1]

1. Check Database Status

Displays the database open mode, software release edition, and running components. [1]

bash

/bin/dbaascli database status --dbname <database_name>

Use code with caution.

2. Create a Database

Provisions a Container Database (CDB) or a Non-CDB inside a pre-existing Oracle Home. [1, 2]

Create a CDB:
bash
/bin/dbaascli database create --dbName <database_name> --oracleHome <path_to_oracle_home>
Use code with caution.
Create a Non-CDB:
bash
/bin/dbaascli database create --dbName <database_name> --oracleHome <path_to_oracle_home> --createAsCDB false
Use code with caution.

3. Start, Stop, and Restart (Bounce)

Controls the database instance state across the Exadata cluster nodes. [1]

Start: /bin/dbaascli database start --dbname <database_name>
Stop: /bin/dbaascli database stop --dbname <database_name>
Bounce: /bin/dbaascli database bounce --dbname <database_name>

4. Manage Backups

Configures, triggers, or views details about the automated cloud backup framework. [1]

Get Backup Config:
bash
/bin/dbaascli database backup --getConfig --dbname <database_name>
Use code with caution.

5. Modify Initialization Parameters

Updates or resets SPFILE static/dynamic parameters in batch mode. [1]

bash

/bin/dbaascli database modifyParameters --dbname <database_name> --setParameters "parameter1=value1,parameter2=value2" --allowBounce

Use code with caution.

6. Change Database Passwords

Updates administrative credentials like SYS or the Transparent Data Encryption (TDE) wallet password. [1]

bash

/bin/dbaascli database changepassword --dbname <database_name>

💡 Tip: To see the full checklist of arguments and real-time help syntax for any command block, append --help to your query: 

bash

/bin/dbaascli database --help
/bin/dbaascli database create --help


In Oracle Exadata Database Service on Cloud@Customer (ExaCC), the /bin/dbaascli diag command group is used as root to diagnose, run health checks, and gather logs from the Guest VM cloud tooling stack. [1, 2]
The utility supports two principal subcommands for troubleshooting lifecycle or management automation failures: [1]
1. dbaascli diag collect
This command initiates automated diagnostic log collection across your ExaCC environment. Running it on a single node seamlessly coordinates log retrieval from all cluster nodes. [1, 2]
Basic Execution:
bash
# dbaascli diag collect
Use code with caution.
Advanced Usage with Options:
bash
# dbaascli diag collect --components db|gi|os|dbaastools|all --startTime YYYY-MM-DDTHH:MM:SS --endTime YYYY-MM-DDTHH:MM:SS --destLocation /path/to/output
Use code with caution.
 [1]
Key Flags:
--components: Filters the scope of collection. Accepts db, gi (Grid Infrastructure), os, dbaastools, or all.
--destLocation: The target directory on the Guest VM to drop the generated archive. The default directory is /var/opt/oracle/dbaas_acfs.
--waitForCompletion: Defaults to true. Set to false if you want the job to process asynchronously in the background while monitoring progress via dbaascli job getStatus. [1, 2]
2. dbaascli diag healthCheck
This command invokes a programmatic cloud tooling configuration review across all nodes to identify systemic configuration drift, permissions errors, or corrupted cloud infrastructure tracking repositories. [1, 2]
Execution:
bash
# dbaascli diag healthCheck
Use code with caution.
Output files generated by this check are automatically staged inside compressed .tar.gz archives within your destination location, 
ready to accompany an Oracle Support Request (SR). [1, 2]



The /bin/dbaascli dbhome command is used to manage Oracle Database homes in an Oracle Exadata Cloud@Customer (ExaCC) environment. It allows database administrators to perform lifecycle management tasks directly from the command line.
Here is a grouped list of the primary operations you can perform with this command:
Creation and Deletion
create: Installs a new Oracle Database home.
purge: Removes a specific database home from the system.
Patching and Updates
patch: Applies Release Updates (RUs) or interim patches to a database home.
rollback: Rolls back a previously applied patch.
update: Updates the database home software version.
Information and Status
get-details: Displays specific configuration information for a database home.
list: Shows all database homes provisioned on the cluster.



Interviewer: "We are facing an intermittent node eviction or severe OS performance degradation on an ExaCC database node (or storage cell). How would you collect and analyze the operating system metrics for that specific window?"
Candidate (Your Answer):
"I would use ExaWatcher, specifically by executing the utility script GetExaWatcherResults.sh. It automatically aggregates OS and network performance data (like iostat, vmstat, top, and cellsrvstat) into an extracted bundle. I would extract the metrics covering the exact problematic timeframe and review the automatically generated HTML charts to identify spikes in CPU, memory saturation, or I/O latency." [1, 2, 3, 4]
❓ Top Technical Questions & Answers
Q1: Where is the script located and how do you execute it?
Answer: The script is located in the /opt/oracle.ExaWatcher/ directory on both Database (Compute) nodes and Storage Cells. It must be executed as the root user. [1, 2, 3]
bash
cd /opt/oracle.ExaWatcher/
./GetExaWatcherResults.sh --from MM/DD/YYYY_HH:MM:SS --to MM/DD/YYYY_HH:MM:SS
Use code with caution.
Q2: What are the primary arguments used with GetExaWatcherResults.sh?
Answer:
--from: The start time of the incident (Required).
--to: The end time of the incident (Highly recommended to narrow the output).
--resultdir: Optional parameter to specify a custom target directory if the default partition lacks space. [1, 2, 3]
Q3: What is the default output format, and where is it saved?
Answer: The script packages the data into a compressed tarball/bzip2 archive (.tar.bz2). By default, it drops the file into /opt/oracle.ExaWatcher/archive/ExtractedResults/. [1, 2, 3]
Q4: How do you read or analyze the output of the script?
Answer:
Uncompress and untar the generated file (tar -xvf ExaWatcher_...tar).
Inside the extracted directory, look for a folder named Charts.ExaWatcher.<hostname>.
Open the index.html file in any standard web browser. It populates interactive charts mapping out CPU utilization, I/O summary, memory metrics, and cell server statistics for that specific timeframe. [1, 2, 3, 4, 5]
Q5: What is the underlying data collection interval of ExaWatcher compared to standard OSWatcher?
Answer: ExaWatcher is customized for high-frequency Exadata workloads. While standard OSWatcher typically captures metrics every 30 to 60 seconds, ExaWatcher captures data at a granular 5-second interval, making it highly effective at catching sudden microsecond CPU spikes or transient I/O hangs. [1]
💡 Pro-Tips to Sound Like an Expert (ExaCC Specific)
Space Management Warning: Mention that running GetExaWatcherResults.sh over a long period (e.g., 24+ hours) can generate massive files. Emphasize that you always check disk space (df -h) on /opt before running it, or redirect the output using --resultdir to a safe mount point like /u02 or /tmp. [1, 2, 3]
Storage Cells vs DB Nodes: Clarify that ExaWatcher runs on both layers. If the issue is a slow query, you run it on the DB node; if the issue is high cell disk service times, you run it directly on the Storage Node to capture cellsrvstat details. [1, 2]



Below is the standard interview question breakdown, what the output means, and why it is critical for ExaCC environments.
📋 The Interview Question
Question: What is the purpose of the /sbin/vgdisplay command on an ExaCC Database Node, and what key metrics do you look for when managing database storage space?
Answer:
The /sbin/vgdisplay command is used to display detailed attributes of Volume Groups (VG) configured on the Linux OS layer of the ExaCC Database Node. While the actual database files reside on ASM (Automatic Storage Management) inside Exadata Grid Infrastructure, local paths like /, /u01, /oracle, and the grid homes are managed via LVM. [1, 2, 3, 4]
When running this command, an administrator primarily looks for the Free PE (Physical Extents) metric to determine if there is enough unallocated space in the volume group to safely extend a critical system or software filesystem (like /u01) when it runs out of space. [1, 2]
🔍 Key Metrics Evaluated in the Output
When you run /sbin/vgdisplay <volume_group_name>, the interviewer will expect you to know what these crucial fields represent: [1]
Field Name
Description & Relevance to ExaCC
VG Name
The name of the volume group (commonly VGExaDb or rootvg on Exadata nodes).
PE Size
The size of each Physical Extent block (typically 4MB or larger). This is the smallest unit LVM can allocate.
Total PE
The total number of allocatable blocks inside this volume group.
Alloc PE / Size
The amount of space currently dedicated to existing Logical Volumes (like /, /var, /u01).
Free PE / Size
The most critical metric. It shows the remaining unallocated space in the pool that you can use to extend filesystems online using lvextend.
💡 Scenario-Based Follow-Up Questions
Interviewers will often pivot from the basic command to a real-world scenario to test your system administration skills:
Scenario A: The /u01 filesystem on an ExaCC node is 99% full, causing patch deployment or trace generation to fail. How do you resolve this using LVM?
Check Available Space Pool: Run /sbin/vgdisplay to see if there is unallocated space under Free PE / Size. [1, 2]
Extend the Logical Volume: If free space exists, extend the logical volume online (e.g., adding 20 GB):
bash
sudo /sbin/lvextend -L +20G /dev/VGExaDb/LVOraHome
Use code with caution.
 [1]
Resize the Filesystem: Expand the underlying filesystem so the OS recognizes the new space.
For XFS (Default on modern Exadata images):
bash
sudo /sbin/xfs_growfs /u01
Use code with caution.
For Ext4:
bash
sudo /sbin/resize2fs /dev/VGExaDb/LVOraHome

Scenario B: What is the difference between vgdisplay, pvdisplay, and lvdisplay?
pvdisplay: Shows information about the raw physical storage devices/partitions (/dev/nvme* or /dev/sd*) assigned to LVM.
vgdisplay: Shows the combined, centralized storage pool (Volume Group) details.
lvdisplay: Shows individual slice/partition metrics (Logical Volumes) created out of the volume group that hold the actual file systems. [1, 2]

Q: What is the imageinfo command used for in Exadata/ExaCC?
A: It is executed with root or sudo privileges on a compute (database) node to check the exact Exadata software version, OS kernel version, activation timestamp, and whether the image status is successful. [1, 2]
Q: Where do you run imageinfo—on the database node or the storage cell?
A: You run imageinfo on the database compute node. To check version details on a storage cell server instead, you use cellcli commands like cellcli -e "list cell details". [1]
Q: What key details does imageinfo output?
A: It outputs the active Exadata image version, tracking components, GI/database node pairing information, and rollback/success status of the software image. [1]

Question 1: What is the purpose of the /usr/bin/exachk tool, and why is it critical in an ExaCC environment?
Answer:
exachk is a holistic health check tool designed specifically for Oracle Engineered Systems like Exadata. In an ExaCC environment, it acts as the primary tool to ensure the entire system aligns with Oracle’s Maximum Availability Architecture (MAA) and best practices. [1]
It performs configuration and software health audits across all layers: []
Database Servers (Compute Nodes): OS parameters, Grid Infrastructure, ASM, and RDBMS configurations.
Storage Servers (Cells): Cell software, physical/grid disks, flash cache, and IORM setup.
Network Components: RDMA Network Fabric / RoCE switches and InfiniBand switches. [1, 2, 3]
Why it's critical for ExaCC: It generates a comprehensive HTML health report detailing warnings, critical alerts, and failure points. Running exachk is mandatory before and after lifecycle operations like quarterly infrastructure or database patching to ensure the cloud system remains stable. [1]
Question 2: As an Exadata/Cloud DBA, at what specific times are you required to execute exachk?
Answer:
You should run exachk during the following crucial milestones:
Pre-Patching / Pre-Upgrade Readiness: To identify software dependencies, configuration anomalies, or known critical issues that could cause an upgrade or patch to fail. [1, 2]
Post-Patching Verification: To verify that the environment is in a healthy, operational state and no cluster parameters were inadvertently altered. [1]
Routine Maintenance (Quarterly/Monthly): To preemptively capture configuration drifts, hardware warnings, or performance degradations. []
Before Opening a Service Request (SR) with Oracle: Oracle Support almost always requests a fresh exachk run in debug mode to accelerate troubleshooting. [1]
Question 3: How do you execute exachk, and what are the most common command-line options you use?
Answer:
Typically, exachk is run out of the Autonomous Health Framework (AHF) path or standard /usr/bin/exachk directory. It is recommended to run it as the root user so it can discover and execute checks seamlessly across the network fabric and storage layers. [1, 2, 3]
bash
# Standard interactive execution (discovers all components automatically)
./exachk
Use code with caution.
Crucial Command Line Options:
-a: Runs all default best-practice checks and recommended patch checks.
-debug: Runs the utility in verbose mode. It packages a debug log inside the final .zip file for Oracle Support.
-localonly: Forces the tool to run checks only on the local compute node where it is executed, skipping network discovery of other cells or nodes.
-clusternodes <nodes> / -cells <storage_cells>: Allows you to limit the scan to a specific subset of compute or storage nodes. [1, 2, 3, 4]
Question 4: How does exachk execute tasks on remote components like Storage Cells and Network Switches? What configurations are required?
Answer:
exachk uses passwordless SSH connections initiated from the primary compute node to communicate with remote components. [1]
For Storage Servers (Cells), it communicates via root or an elevated execution user (defined by the RAT_CELL_SSH_USER environment variable).
For Network Switches (InfiniBand/RoCE), it connects using root or nm2user. [1]
If passwordless SSH is broken or blocked between the compute nodes and the storage cells, exachk will fail to run the storage layer audits, resulting in incomplete reports.
Question 5: If an exachk report shows a "FAIL" status for an OS kernel parameter or an initialization parameter on ExaCC, how do you handle it?
Answer:
Analyze the Recommendation: Review the detailed mitigation steps provided inside the exachk HTML report. []
ExaCC Shared Responsibility Context: Because ExaCC is a hybrid cloud model, I must distinguish between customer-managed layers and Oracle-managed layers.
If it is a Database/RDBMS parameter (e.g., db_block_checking), I can fix it directly as the oracle user via ALTER SYSTEM.
If it is a Grid Infrastructure/OS parameter inside the DomU (Guest VM), I can fix it as the root user using standard procedures.
If the failure points to the underlying hardware, hypervisor (Dom0), or storage cells, I must open an Oracle Service Request (SR), as those components are managed solely by Oracle Cloud Operations. [1, 2]
Question 6: What is the relationship between exachk and orachk?
Answer:
Both tools belong to the Oracle Autonomous Health Framework (AHF).
orachk is utilized for standard, non-engineered Oracle hardware ecosystems (e.g., standalone servers, standard RAC clusters, ODA).
exachk is explicitly compiled with extra awareness for Exadata architectures,

extending its capabilities to audit storage cells, Exadata storage software,
InfiniBand/RoCE fabrics, and virtualization components unique to the Exadata framework.

In an Oracle Exadata Cloud at Customer (ExaCC) interview, understanding the separation of duties, user access, and resource isolation models is critical. dbaadmin, DRM, and exadba represent foundational concepts across OS-level administration, Cloud automation, and database resource management. [1]

---

1. oracle vs. grid vs. exadba vs. dbaadmin (OS-Level User Access)

Question: What is the difference between the exadba user and the dbaadmin user in an ExaCC environment? Who owns the software binaries?

Answer:
• oracle / grid: These traditional users own the Oracle Database and Grid Infrastructure software binaries inside the VM. However, direct ssh access as oracle or grid is blocked for customers by default.
• exadba: This is the default customer operating system user used to connect to the ExaCC virtual machines (VMs) via SSH. It has partial sudo privileges allowing you to switch to the oracle or grid users to perform standard database administration tasks (e.g., using SQL*Plus, srvctl, crsctl).
• dbaadmin: This is a specialized internal automation user managed by Oracle's cloud tooling (like the OCI agent). It is utilized to trigger automated lifecycle management tasks such as cloud-based patching, automated backups, and scaling from the Oracle Cloud Infrastructure (OCI) console.

Question: As an Exadata DBA, which OS user do you use to perform daily operations on ExaCC, and how do you access it?

Answer: You use the exadba user. You access it by logging into the database nodes via SSH using the customer-generated private key associated with the ExaCC VM cluster. Once logged in, you can execute sudo su - oracle or sudo su - grid to administer the database and clusterware environment.

---

2. DRM (Instance / Cluster Management vs. Database Resource Management)

Depending on the context of the question, DRM in an Exadata/ExaCC interview can stand for Distributed Read Management (Instance/RAC level) or Database Resource Manager (Workload management). [1]

Context A: Distributed Read Management (RAC Architecture)

Question: What is the role of DRM (Distributed Read Management / Dynamic Resource Management) in an Oracle RAC environment running on Exadata?

Answer: Distributed Read Management (DRM)—often referred to as Dynamic Resource Management—is a background mechanism in Oracle RAC that automatically re-masters database blocks. If Node 1 frequently accesses a set of blocks that are currently mastered on Node 2, DRM dynamically shifts the mastership of those resources to Node 1. This minimizes cluster interconnect traffic across the high-speed network, reducing latency.

Context B: DBRM (Database Resource Manager) vs. IORM

Question: How does DBRM (Database Resource Manager) differ from IORM in Exadata, and how do they work together?

Answer:
• DBRM (Database Resource Manager): Operates at the database instance level on the compute nodes. It controls CPU distribution, parallel execution servers, and session limits among different consumer groups within a single database.
• IORM (I/O Resource Manager): Operates at the Exadata Storage Server level (Cells). It regulates physical disk and flash I/O allocation between different databases or inter-database consumer groups.
• Integration: DBRM passes the consumer group information inside the data blocks via the iDB protocol down to the storage cells. IORM maps this tag and ensures that the storage cells prioritize or throttle the physical I/O according to the defined resource plan. [1, 2, 3, 4]

---

3. Scenario-Based Scenario Summary Matrix

Feature/Component	Primary Responsibility	Target Location	Managed By
exadba	Customer CLI administration (DBA tasks)	Compute Node (VM)	Customer via SSH
dbaadmin	Internal cloud automation agent execution	Compute Node (VM)	OCI Cloud Tooling
DRM (RAC)	Dynamic block remastering to optimize interconnect	Cluster-wide	Oracle Grid Infrastructure
DBRM	Managing CPU & parallelism for resource groups	Compute Node / DB Instance	Database Administrator