EasyReliableDBA

Linux Administration: OS Version Check

Example Commands

To verify the operating system and kernel version on an ExaCC or Exadata database/storage node, run:

bash

# Check OS release details
cat /etc/os-release

# Check kernel version
uname -r

Test Case Scenario

Objective: Validate that the environment has been successfully upgraded to Oracle Linux 8 (minimum requirement for modern Exadata X10M features and newer database compatibility).

Expected Output for /etc/os-release:

text

NAME="Oracle Linux Server"
VERSION="8.x"
ID="ol"
ID_LIKE="fedora"
VERSION_ID="8.x"

Expected Output for uname -r: A UEK6 or later kernel string (e.g., 5.4.32-... or higher depending on the specific Exadata software image bundle).

Question: How do you verify the current OS and kernel versions on Exadata X9M or X10M infrastructure, and why is knowing the baseline Linux version critical when moving from X9M to X10M architectures?

Answer:
You check the OS using cat /etc/os-release and the kernel using uname -r. Knowing the baseline version is critical because Exadata X10M introduces newer AMD EPYC processor architectures that require a minimum of Oracle Linux 8 and UEK 6 for proper low-level hardware support, linear core scaling, and optimal resource management. Upgrading the system software bundle handles this OS and kernel alignment uniformly across Dom0 and DomU guests.

Question:In an Exadata X9M or X10M environment, how do you verify and troubleshoot the RDMA over Converged Ethernet (RoCE) network fabric interface bonding and reachability at the Linux OS level if a storage-cell heartbeat warning triggers?

Context & Answer Framework

Exadata X9M/X10M systems rely on dual-port RoCE network cards for ultra-low latency cluster interconnect and storage access via the iDB protocol. Unlike standard TCP/IP networking, RoCE relies heavily on DCB (Data Center Bridging) and PFC (Priority Flow Control) configurations mapped via specific Linux network interface bonds (bondeth or native ip link).

Check Bond and Interface Status:
Use modern iproute2 commands to check state and link layer parameters:

bash

ip -s link show
cat /proc/net/bonding/bond-roce

Verify RDMA Device Health:
Use RDMA core utilities to ensure the HCA (Host Channel Adapter) is active:

bash

ibv_devinfo -v
rdma link show

OS Version Nuance (X9M vs X10M):

X9M typically runs Oracle Linux 7 with UEK5, utilizing legacy network.service or early NetworkManager profiles with older ibacm daemons.
X10M runs Oracle Linux 8 with UEK6, utilizing systemd-networkd or refined NetworkManager with enhanced kernel linear scaling on high core counts.

Scenario : An alert shows degraded RDMA throughput on database server node db01 after a switch maintenance window on the internal RoCE fabric. You need to run a validation test case to isolate whether the issue is an OS interface failure or a fabric configuration mismatch.

Step-by-Step Test Case Implementation

Test Step 1: Validate physical and bonded link state

bash

# Check if both underlying interfaces of the RoCE bond are up
ip link show up

Expected Output: Interfaces like eth5 and eth6 must show UP and master-assigned to the RDMA bond interface.

Test Step 2: Test RDMA connectivity across the fabric
Run the RDMA-core latency/bandwidth benchmark tool between the database node and a storage cell:

bash

# On the storage cell or target node, start the server component:

rping -s -v -a <local_roce_ip>

# On the database node (db01), run the client connection test:

rping -c -v -a <cell_roce_ip> -C 10

Pass Criteria: rping completes successfully with 10 iterations and zero packet/message drops, confirming that PFC and RoCE queues are passing traffic correctly across the fabric layer.

Test Step 3: Inspect kernel log for InfiniBand/RDMA stack errors

bash

dmesg | grep -iE 'rdma|roce|mlx5|bond'

Pass Criteria: Absence of mlx5_core link-down errors, driver timeouts, or local ACK timeout warnings from the Mellanox/NVIDIA adapter layer.

Question "A package installation failed with a dependency error on a production Red Hat server during a routine update. How do you troubleshoot and resolve this using RPM?"

Answer Structure

Identify the failure: Check the exact missing dependency or file conflict.

Inspect the RPM database: Query installed packages to see what version or conflicting file exists.

Resolve the issue: Use force flags carefully or let a high-level tool handle the dependency tree.

Practical Example & Test Case

Scenario

You try to install a local package nginx-1.20.1-1.el7.x86_64.rpm, but the system throws a failed dependencies error or a file collision error.

1. Test Case: Simulating and Diagnosing the Issue

Run the RPM query or test install command to see what is wrong without making changes.

Check dependencies without installing (Test Mode):

bash

rpm -ivh --test nginx-1.20.1-1.el7.x86_64.rpm

Output error example:

text

error: Failed dependencies:
        libcrypto.so.10()(64bit) is needed by nginx-1:1.20.1-1.el7.x86_64

Identify which package provides the missing file/library:

bash

repoquery --whatprovides libcrypto.so.10
# Or using rpm directly if installed elsewhere:
rpm -qf /usr/lib64/libcrypto.so.10

2. Fixing the Daily Issue

Instead of using low-level rpm -ivh which fails on missing dependencies, switch to the high-level package manager (dnf or yum) which automatically pulls dependencies from repositories.

Let DNF/YUM resolve and install the local RPM with dependencies:

bash

dnf localinstall nginx-1.20.1-1.el7.x86_64.rpm

Alternative: Handling a "file conflicts" error
If the error says a file is already owned by another package:

text

error: file /etc/nginx/nginx.conf from install of nginx conflicts with file from package old-nginx-1.18

Solution: Replace the conflicting file or upgrade properly using the refresh flag:
bash
rpm -Uvh --replacefiles nginx-1.20.1-1.el7.x86_64.rpm
Question: How do you use the ping command to check for network delay, and how do you handle a scenario where ping shows high delay or intermittent response?

Answer:
"I use the ping command with specific flags to control packet size and interval to measure network delay accurately. If a server shows high delay, I check packet loss, trace the network path using traceroute, and inspect interface errors using ip -s link."

Example Usage with Delay Control

To add a delay (interval) between each ping packet, use the -i flag (in seconds). To set a deadline or packet count, use -c.

bash

ping -c 5 -i 2 8.8.8.8

-c 5: Send only 5 packets and stop.

-i 2: Wait 2 seconds between sending each packet (default is 1 second).

8.8.8.8: The target IP address (Google Public DNS).

Sample Output

text

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=14.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=15.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=45.1 ms  # Spike in delay
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=14.5 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=117 time=13.8 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 8011ms
rtt min/avg/max/mdev = 13.801/20.520/45.102/12.351 ms

Test Case for Troubleshooting High Ping Delay

When an application complains about lag or time-outs, execute this structured test plan.

Test Case 1: Local Loopback Test

Action: ping -c 4 127.0.0.1

Expected Result: time < 1ms

Purpose: Verifies that the local network stack and kernel interface are functioning properly. High local delay indicates a local CPU or OS resource bottleneck.

Test Case 2: Gateway Reachability Test

Action: ping -c 5 $(ip route | grep default | awk '{print $3}')

Expected Result: time < 5ms on a wired LAN.

Purpose: Isolates whether the delay originates inside your local network (switch, local router, or physical cable) or outside on the internet.

Test Case 3: Path Trace Analysis

Action: traceroute 8.8.8.8 (or mtr 8.8.8.8 for real-time diagnostics)

Expected Result: Clear sequential hops with stable response times.

Purpose: Pinpoints the exact network hop or router interface where the packet delay spikes or packet loss begins.

Daily Oracle Key Vault (OKV) key rotation on Exadata Database Machine (Exacc) involves updating cryptographic wallet secrets on Linux database nodes to ensure compliance and security without disrupting database services.

Overview of OKV Key Rotation on Exacc

Purpose: Replaces aging master encryption keys (MEK) or credentials stored in Oracle Key Vault used for Transparent Data Encryption (TDE).

Frequency: Typically performed based on corporate security policy (e.g., monthly or annually) or as a routine daily administrative validation/check.

Exacc Context: Exadata Cloud@Customer (Exacc) runs Oracle Grid Infrastructure and Database homes on Oracle Linux, where database servers communicate with an external OKV server via client configuration files (okvclient.ora and wallet.sso).

Step-by-Step Example: Rotating/Updating OKV TDE Keys

This example shows how a Linux administrator interacts with the environment to refresh or rotate keys using the Oracle Autonomous Health Framework or SQL-level commands tied to the OKV wallet backend.

Verify Current Wallet Status via SQL Plus:
Connect to the pluggable database (PDB) or container database (CDB) as SYSDBA or a user with administrative privileges to check the key status.

sql

SELECT KEY_ID, TAG, STATE, CON_ID FROM V$ENCRYPTION_KEYS;

Set a New Master Encryption Key:
Generate and set a new key from the connected OKV keystore.

sql

ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "WalletPassword" WITH TAG 'OKV_Rotation_2026' TO CURRENT;

Verify Key Rotation on Linux OS Level:
Ensure the local client environment points to the valid OKV configuration directory (typically under /etc or /var/lib/okv or the designated wallet location in the Oracle Grid/DB home).

bash

ls -la /opt/oracle/okv/

Check the okvclient.ora parameters to confirm connectivity endpoints to the OKV server remain intact after rotation:

bash

cat /opt/oracle/okv/okvclient.ora

Question: "How do you handle or troubleshoot an OKV master encryption key rotation failure on an Exacc Linux node where the database loses sync with the Oracle Key Vault?"

Answer:

Diagnosis: Check the alert log and the TDE wallet trace files located in $ORACLE_BASE/diag/rdbms/.../trace or the OKV client logs (/var/log/okv or custom paths) for authentication errors (e.g., ORA-28354: wallet does not exist or communication timeouts).

Network & Endpoint Check: Validate that the Exacc compute node can reach the OKV cluster port using network utilities:

bash

nc -zv <okv-server-ip> 5696

Credential Verification: Ensure the wallets.sso and okvclient.ora files have correct file permissions (600 or 400 owned by oracle:oinstall) and that the API password/certificate bindings have not expired on the OKV server side. Re-sync or re-enroll the client using the okvutil command if the certificate is corrupted:

bash

okvutil list -t /etc/openssl/certs/

Question > Migration

Migrating an Oracle Exadata Database Service from an on-premises or older X9M infrastructure to an X10M Exadata Cloud@Customer (ExaCC) environment uses automated utilities like Oracle Zero Downtime Migration (ZDM) or the Exadata Cloud Infrastructure Migration Automation Utility.

Migration Overview

Source: Exadata X9M infrastructure.

Target: Exadata X10M ExaCC (Cloud@Customer) infrastructure.

Method: Oracle Data Guard or ZDM physical online migration for minimal downtime.

Example Workflow & Steps

Pre-Checks and Setup

Download and configure the migration utility or ZDM control plane on a separate Linux host.
Validate network connectivity, SSH equivalence, and firewall ports (e.g., port 1521, 22) between source X9M and target X10M.

Provision Target Environment

Deploy the X10M VM clusters and empty placeholder databases with identical character sets and compatibility parameters via OCI console or CLI.

Establish Replication

Configure an Oracle Data Guard standby association from the X9M source database to the X10M target database.

Switchover and Cutover

Perform a Data Guard switchover to promote the X10M database to primary with near-zero downtime.

Test Cases

TC01: Network & Port Connectivity

Action: Run nc -zv <target_scan_ip> 1521 from source X9M nodes.
Expected Result: Connection succeeds, confirming listener access across racks.

TC02: Data Guard Synchronization

Action: Check lag on the target standby using SELECT dest_idp, status, applied_seq# FROM v$archived_log;.
Expected Result: Zero or minimal lag (applied_seq# matches source primary).

TC03: Post-Migration Application Smoke Test

Action: Point application connection strings to the new X10M SCAN listener and execute read/write transactions.
Expected Result: Transactions commit successfully with expected X10M performance metrics.

Q: What is the primary advantage of migrating from an X9M to an X10M ExaCC platform using Data Guard?

A: It allows near-zero downtime cutover by synchronizing data continuously via redo logs before executing a final switchover during a brief maintenance window.

Q: How do you handle storage and database configuration parity during an X9 to X10 ExaCC migration?
A: You use automated templates via OCI migration tools or Oracle ZDM to ensure identical database names, initialization parameters, and encryption settings are pre-provisioned on the X10M target before data synchronization begins.

Daily Task: Resolving CPU/Memory Bottlenecks on ExaCC Database Nodes

When a database node (DomU/Guest VM) experiences high load averages or performance degradation, a Linux administrator follows a structured troubleshooting workflow.

1. Identify the Resource Drain

Run top or htop to identify if the CPU is bottlenecked by user processes (us), system/kernel tasks (sy), or waiting on I/O (wa).

Run vmstat 1 10 to check for CPU context switching (cs) and memory swapping (si/so).

2. Check for HugePages Misconfiguration

ExaCC relies heavily on HugePages to manage large SGA memory footprints for Oracle Databases.

Run cat /proc/meminfo | grep -i Huge to verify if HugePages are fully utilized or if the system is misconfigured, forcing the OS to use standard 4KB pages and causing high system CPU usage (sys%).

3. Analyze Storage & Network I/O

Use iostat -xz 1 10 to check disk latency and utilization.

Because ExaCC uses RoCE (RDMA over Converged Ethernet) or InfiniBand, run ibv_devinfo or roce_admin tools to check the health of the high-speed interconnect links.

🛠️ Simulated Test Case: The "Ghost" CPU Spike

Scenario

An ExaCC Guest VM (Database Node) is showing a CPU utilization of 98%, causing application timeouts. The Database Administrator (DBA) insists there are no heavy queries running in Oracle.

Investigation & Steps

Check OS Metrics: The admin runs top and notices that sys% (system CPU) is at 70%, while user% is only 15%.

Identify the Culprit Process: Sorting top by CPU reveals that multiple Oracle foreground processes (oracle<SID>) are consuming high CPU, but they are stuck in kernel space.

Trace System Calls: The admin runs strace -c -p <PID> on one of the spinning processes. The output shows millions of calls to gettimeofday() or page faults.

Root Cause Analysis: Checking /proc/meminfo reveals:

text

HugePages_Total:    512
HugePages_Free:     0
HugePages_Rsvd:     0

The DBA recently increased the Oracle SGA size, but the Linux HugePages count was not increased to match it. The OS is allocating regular 4KB pages for the database, causing massive page table overhead and killing CPU performance.

Resolution

Calculate required HugePages based on the new SGA size.

Edit /etc/sysctl.conf and update vm.nr_hugepages.

Run sysctl -p to dynamically apply changes (or schedule a maintenance window if memory fragmentation prevents allocation).

Interview Questions & Answers

Q1: A database node on ExaCC is experiencing high "iowait" (wa%) in top. How do you determine if the issue is inside the local Linux VM or on the Exadata Storage Cells?

Answer:

First, I check iostat -xz 1 on the local node. If the service time (svctm) and average wait time (await) are exceptionally high, it indicates an I/O bottleneck.

To isolate the storage layer, I look at the Exadata-specific metrics. I will use the Oracle cellcli utility (if accessible) or work with the DBA to review Automatic Workload Repository (AWR) reports specifically looking for "cell single block physical read" latencies.

If flash log or hard disk metrics on the storage cells show high latencies, the bottleneck is on the storage servers (Cell Nodes). If cell latencies are low but Linux iowait is high, the issue is likely a network throttling or queuing issue on the local ExaCC VM network interfaces.

Q2: What is the significance of HugePages in an ExaCC Linux environment, and what happens if they are misconfigured?

Answer:

Significance: ExaCC hosts massive Oracle databases. HugePages increases the default page size from 4KB to 2MB (or 1GB). This drastically reduces the size of the OS page table, saving gigabytes of memory and lowering CPU overhead for memory mapping.

Misconfiguration Impact: If HugePages are under-allocated, Oracle will fallback to standard 4KB pages. This causes the Linux kernel to spend massive amounts of CPU cycles managing the page table, leading to high system CPU utilization (sys%) and severe database performance degradation.

Q3: How do you monitor and troubleshoot the private interconnect network (RoCE/InfiniBand) performance on an ExaCC compute node?

Answer:

I use ibstatus or ibv_devinfo to verify that the physical ports are active and running at the correct speed (e.g., 100 Gbps for RoCE).

For active performance monitoring, I use osar or ifconfig to check for packet drops, framing errors, or overruns on the specific interfaces (like re0 or bondeth0).

I also check /var/log/messages or dmesg for any link flapping events or RDMA-related errors that indicate a hardware or switch-level issue.

Q4: If an ExaCC Linux node becomes completely unresponsive due to a performance freeze, how do you collect diagnostic data?

Answer:

Since ExaCC is a managed cloud infrastructure, I would utilize the Oracle Cloud Infrastructure (OCI) console or CLI to check the node status.

If the OS is frozen but the hypervisor layer is functional, I would attempt to trigger a non-maskable interrupt (NMI) or use the OCI Serial Console to capture the current state.

If the node panics or reboots, I would analyze the Kdump crash dump file located in /var/crash/ using the crash utility once the node recovers to inspect the kernel stack trace at the exact moment of the freeze.

Linux Admin Daily Tasks & OS Troubleshooting on Exadata CC

Exadata Cloud at Customer (ExaCC) combines Oracle Exadata hardware with cloud management. Linux administrators handle standard OS tasks alongside ExaCC-specific infrastructure checks.

Daily Linux Administration Tasks on ExaCC

Check System Health: Run uptime, dmesg -T, and inspect /var/log/messages for hardware or OS warnings.

Monitor Storage: Verify ASM disks and local filesystems using df -h and lsblk.

Check InfiniBand/RDMA Status: Run ibstat to ensure the InfiniBand fabric links are active.

Review Cluster Services: Check Oracle Grid Infrastructure and daemon status using crsctl check crs.

OS Issue Example, Test Case, and Interview Q&A

Scenario: High CPU Usage Caused by a Runaway Process

The Issue

An ExaCC compute node shows sluggish performance. CPU utilization hits 100% on user space.

Test Case / Reproduction Steps

Run top or htop to identify the process consuming the highest CPU.

Simulate a runaway process for testing using a simple loop:

bash

while :; do :; done &

Note the Process ID (PID) from the top output.

Troubleshooting & Resolution

Identify the PID and command name.

Check if the process belongs to an Oracle database instance or an unauthorized user script.

If safe to terminate, run:

bash

kill -15 <PID>

Use code with caution.

If the process does not stop, force-kill it:

bash

kill -9 <PID>

Use code with caution.

Verify CPU usage drops back to normal levels.

Interview Questions and Answers

Q1: What is a key difference in managing storage on an ExaCC compute node compared to a standard standalone Linux server?

Answer: On ExaCC, local OS disks use standard Linux LVM, but database data storage is managed via Automatic Storage Management (ASM) across Exadata Storage Cells, not standard Linux filesystems.

Q2: How do you verify network bonding and high availability on an ExaCC compute node?

Answer: You check the status of the network interface bonding files in /proc/net/bonding/ or run ip -s link to ensure both primary and backup interfaces are active and passing traffic.

Q3: How do you troubleshoot an unresponsive ExaCC compute node when SSH access fails?

Answer: Use the ILOM (Integrated Lights Out Management) or the cloud control plane console to access the serial console and inspect kernel panics or boot logs.

Linux administration daily tasks for Exadata Database Machine (ExaCC - Exadata Cloud@Customer) involve monitoring Exadata infrastructure, checking Exadata storage cells, tracking compute node health, and troubleshooting OS issues.

Common Daily Tasks on ExaCC OS

Check Node Health: Run dbnodeupdate.sh -m check to see if the system is ready for updates.

Monitor Storage Cells: Use cellcli to check disk health with LIST CELL DETAIL and LIST DISKGROUP DETAIL.

Inspect Hardware Logs: Check /var/log/messages, dmesg, and ipmitool sensor data for hardware faults.

Verify Clusterware: Run crsctl check crs to confirm Oracle High Availability services are running.

Interview Question: OS Issue on ExaCC

Question:

An ExaCC compute node shows high CPU utilization and slow response times. How do you troubleshoot this OS issue, and what is a practical test case?

Answer:

Identify the Bottleneck: Run top or htop to find the processes using the most CPU or memory.

Check I/O Wait: Run iostat -xz 1 10 to see if Exadata storage cells or local disks have high latency or disk bottlenecks.

Inspect Logs: Check /var/log/messages and /u01/app/grid/diag/... for alert logs.

Examine Exadata Health: Use cellcli to verify that storage cells are not degraded or rebuilding.

Test Case Scenario:

Scenario: A runaway background process consumes 100% CPU on a compute node.

Action:

Log in via SSH.
Run ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head -n 10 to find the offending PID.
Terminate safely using kill -15 <PID>. If it does not stop, use kill -9 <PID>.

Configuring and troubleshooting Network File System (NFS) on Oracle Exadata Database Machine (ExaCC - Exadata Cloud@Customer) requires strict adherence to Oracle’s high-availability and networking standards.

Pre-Checks and Considerations on ExaCC

Network Isolation: Ensure NFS traffic runs over the designated client or backup network fabrics (virtual local area networks - VLANs) rather than the private InfiniBand/RoCE cluster network unless specifically architected.

Firewall and Ports: Verify that required ports (2049 for NFS, 111 for rpcbind/portmapper) are open through any local or corporate firewalls. [1]

Mount Options: Use bg (background) instead of fg for non-critical mounts, and include _netdev in /etc/fstab so the system waits for network activation during boot. [1]

Version Control: Prefer NFSv3 or NFSv4 depending on your security and locking requirements; note that NFSv4 requires TCP port 2049 and handles state differently than v3. [1]

Configuration Example

1. On the NFS Server Side

Install packages and define the export in /etc/exports: [1]

bash

# Install utilities
sudo yum install -y nfs-utils

# Create export directory
sudo mkdir -p /mnt/exacc_share

# Edit /etc/exports
/mnt/exacc_share  192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)

# Export and start service
sudo exportfs -ra
sudo systemctl enable --now nfs-server

Use code with caution.

2. On the ExaCC Client Side

Verify visibility and mount the share: [1, 2]

bash

# Check available exports from server
showmount -e 192.168.1.100

# Mount the share manually
sudo mount -t nfs -o rw,sync,hard,intr 192.168.1.100:/mnt/exacc_share /local_mount

Use code with caution.

Test Case Scenario

Objective: Validate read/write capability and persistent recovery after network interruption.

Test Steps:

Create a test file on the client mount point: echo "ExaCC NFS Test" > /local_mount/test.txt
Verify the file exists on the NFS server side under /mnt/exacc_share/test.txt.
Simulate network drop or restart the nfs-server service on the host.
Run df -h or access /local_mount to observe hard-mount wait behavior or recovery once the service resumes.

Interview Questions & Answers

Q1: Why do we use no_root_squash in enterprise database setups, and what is the risk?

Answer: It allows the root user on the client machine to retain root privileges over the shared files. This is often required for database backups or administrative tools running as root that need to manipulate file ownership. The risk is security compromise: if a client node is breached, the attacker has root-level control over the shared storage data. [1]

Q2: What is the difference between a hard mount and a soft mount in NFS?

Answer: A hard mount (default/recommended for critical data) makes the operating system hang/retry indefinitely if the NFS server stops responding, protecting data integrity. A soft mount returns an error to the calling process after a timeout, which can cause silent data corruption or application crashes if writes fail midway. [1]

Q3: How do you troubleshoot an NFS mount hanging on an ExaCC node?

Answer: First check network routing and port connectivity using ping and nc -zvw3 <server_ip> 2049. Verify RPC services using rpcinfo -p <server_ip>. Check system logs via dmesg | tail or /var/log/messages for blocked NFS tasks. [1, 2]

A common Linux sudoers issue on Exadata Cloud@Customer (ExaCC) involves syntax errors or incorrect file permissions in /etc/sudoers (or files in /etc/sudoers.d/), which locks out administrative users from executing root commands.

ExaCC Context & Considerations

ExaCC Architecture: Exadata Cloud@Customer runs Oracle Grid Infrastructure and Database services on dedicated Exadata infrastructure managed via specific oracle/grid users and root.

Control Plane vs. Data Plane: Direct OS access is partitioned. Breaking sudo on dom0 (domU/guest VM) can block Oracle Cloud automation or patching scripts if they rely on specific passwordless sudo configurations for opc or oracle users.

Never use standard editors: Always edit configuration files using visudo, which checks syntax before saving.

Pre-checks

Verify if you still have an active root shell or another session open with root privileges.

Check the current permissions of the sudoers file: it must be 0440 and owned by root:root.

Inspect system logs (/var/log/secure or journalctl -u sudo) for syntax rejection details.

Example & Test Case

The Issue (Syntax Error in /etc/sudoers.d/custom)

An administrator accidentally adds a line with a typo:

text

opc ALL=(ALL) NOPASSWD: /bin/yum update

Use code with caution.

If /bin/yum does not match the exact restricted path or contains a typo like /bin/yumg, or a missing comma/alias, running sudo visudo -c would have caught it.

Test Case

Trigger: Run sudo -l or a restricted command as a non-root user (e.g., opc).

Error Output:

text

>>> /etc/sudoers: syntax error near line 45 <<<
sudo: parse error in /etc/sudoers near line 45
sudo: no valid sudoers sources found, quitting

Use code with caution.

Resolution / Recovery:

Log in via an alternate root-level access mechanism or ILOM/serial console if locked out completely.
Fix permissions or syntax via a recovery mount or emergency mode if remote root access is completely dropped:
bash
chmod 440 /etc/sudoers chown root:root /etc/sudoers
Use code with caution.

Interview Question and Answer

Question: "You edited the /etc/sudoers file on an ExaCC compute node, and now no one can run sudo. How do you troubleshoot and fix this without rebooting the system if you are completely locked out of sudo?"

Answer:

"If sudo is broken and I have an existing root shell open, I will immediately run visudo -c to find the syntax error, correct the file, and verify permissions are 0440. If I am completely logged out of root and non-root users get parse errors, I must use out-of-band management such as the cloud console, ILOM, or a hypervisor-level rescue mechanism to boot into single-user mode or mount the root filesystem, correct the syntax or permissions on /etc/sudoers, and restart standard access."

Oracle Client installation on Exadata requires strict adherence to Oracle’s Distributed Command Execution (DCE) framework and Quarterly Maintenance Patch (QFSDP) alignment. Because Exadata leverages specialized architecture like InfiniBand/RoCE networks and Exadata Storage Servers, a Client installation must not disrupt existing Grid Infrastructure (GI) or Database (DB) homes.

Here is a comprehensive breakdown of the pre-checks, considerations, installation steps, and interview scenarios.

📋 Pre-checks and Critical Considerations

Before installing the Oracle Database Client on an Exadata compute node, you must verify system compatibility and resource isolation.

Software & Architecture Alignment

Version Match: Ensure the Oracle Client version matches or is compatible with the existing Exadata Database Home version.

Inventory Verification: Check the global inventory location (/etc/oraInst.loc) to ensure the oracle user has write permissions.

Patch Level: Verify if the client requires specific one-off patches to match the Exadata Bundle Patch (BP) or Quarter Full Stack Download Patch (QFSDP) level.

Resource & Path Isolation

ORACLE_BASE Isolation: Set a distinct ORACLE_BASE and ORACLE_HOME for the client. Never overwrite or merge with the Database or Grid Infrastructure directories.

Storage Allocation: Ensure the target local file system (usually /u01) has at least 5–10 GB of free space. Do not install the client on clustered file systems (ACFS/OCFS2) unless explicitly required for shared tools.

User and Permissions: Execute the installation strictly as the oracle software owner, never as root or grid.

⚙️ Step-by-Step Installation Example (Silent Mode)

Exadata environments rarely use graphical user interfaces (GUI/X11). Silent installation using a response file is the standard production method.

Step 1: Download and Stage the Software

Stage the Oracle Client installation zip file in a temporary directory (e.g., /u01/stage).

bash

mkdir -p /u01/stage
# Securely copy the client zip file to the staging directory
cd /u01/stage
unzip LINUX.X64_193000_client.zip

Use code with caution.

Step 2: Configure the Response File

Navigate to the response directory and edit client_install.rsp. Ensure the following parameters are explicitly defined:

ini

ORACLE_SUITE_NAME=Oracle Client
ORACLE_HOME=/u01/app/oracle/product/19.0.0/client_1
ORACLE_BASE=/u01/app/oracle
oracle.install.client.installType=Administrator

Use code with caution.

(Note: "Administrator" type installs all required tools like SQLPlus, TNSPING, and development libraries).*

Step 3: Run the Silent Installation

Execute the installer using the modified response file.

bash

/u01/stage/client/runInstaller -silent -responseFile /u01/stage/client/response/client_install.rsp -ignorePrereq

Use code with caution.

Step 4: Run Root Scripts

Once the installer prompts you, switch to the root user to execute the configuration script:

bash

# As root user
/u01/app/oracle/product/19.0.0/client_1/root.sh

Use code with caution.

🧪 Verification and Test Case

After installation, you must verify that the client can communicate over the Exadata internal network or client network using local naming parameters.

1. Environment Setup

Configure the client profile file (.bash_profile_client or temporary export variables):

bash

export ORACLE_HOME=/u01/app/oracle/product/19.0.0/client_1
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/usr/lib
export TNS_ADMIN=$ORACLE_HOME/network/admin

Use code with caution.

2. Configure tnsnames.ora

Create a test connection string pointing to the Exadata SCAN (Single Client Access Name) listener in $TNS_ADMIN/tnsnames.ora:

text

EXADB_TEST =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ://example.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = exasvc_://example.com)
    )
  )

Use code with caution.

3. Execution of the Test Case

Run these validation commands to confirm network routing and software integrity:

bash

# Test 1: Verify the executable resolves correctly
which sqlplus

# Test 2: Verify network resolution to the Exadata SCAN listener
tnsping EXADB_TEST

# Test 3: Validate end-to-end database connectivity
sqlplus username/password@EXADB_TEST <<EOF
SELECT open_mode, database_role FROM v\$database;
EXIT;
EOF

Use code with caution.

🗣️ Interview Questions and Answers

Q1: Why must we be extremely cautious when defining ORACLE_HOME during an Oracle Client installation on an active Exadata compute node?

Answer: On Exadata, the compute nodes already host highly optimized, mission-critical Database and Grid Infrastructure homes. If an administrator accidentally sets the client installation path to an existing ORACLE_HOME, it will corrupt the production binaries, overwrite critical library links, and potentially cause a node crash or cluster eviction. The client must always reside in a completely isolated subdirectory.

Q2: During a client silent installation on Exadata, the prerequisite check fails due to swapping space or minor kernel parameter mismatches. How should you proceed?

Answer: Exadata compute nodes are pre-engineered systems with hardcoded kernel optimizations tuned specifically by Oracle for database workloads. You should never change system-level kernel parameters or swap space to satisfy a client installer prerequisite. Instead, verify that the OS version is supported, and bypass minor prerequisite warnings by appending the -ignorePrereq flag to the runInstaller command.

Q3: How do you ensure that your newly installed Oracle Client utilizes the high-bandwidth Exadata network instead of the slower public management network?

Answer: To leverage the Exadata architecture efficiently, the client's network configuration (tnsnames.ora) must reference the SCAN host name or VIPs associated with the Client Ethernet/RoCE network interfaces, rather than the management IP or administrative networks. Additionally, ensure that firewall routing allows traffic from the client host to the Exadata client network ports (typically 1521).

Here is a comprehensive guide to understanding, troubleshooting, and answering interview questions regarding Dom0 and DomU on Oracle Exadata database machines (which utilize Xen-based Oracle VM or KVM-based virtualization).

💡 Core Concepts: Dom0 vs. DomU

Dom0 (Domain 0): The privileged management domain run by the hypervisor. It has direct access to the physical hardware (CPU, memory, network cards) and manages the unprivileged guest domains.

DomU (User Domain): The unprivileged guest domains (virtual machines). This is where your Oracle Database instances, Grid Infrastructure, and application software actually run.

⚠️ Common Issue: Communication & Resource Starvation

A frequent issue in Exadata virtualized environments is DomU unresponsiveness or eviction from the cluster caused by resource starvation or communication breakdowns between Dom0 and DomU.

The Problem

If Dom0 experiences high CPU utilization, memory pressure, or disk I/O bottlenecks, it cannot schedule time slices for DomU or process its virtual network traffic. This causes the DomU clusterware to miss heartbeats, leading to node evictions.

📋 Pre-Checks & Critical Considerations

Before making changes or during routine maintenance, always verify the following:

1. Resource Allocation

CPU Overcommit: Ensure you have not overallocated physical CPUs to DomUs. Dom0 needs dedicated, pinned CPU cores to remain responsive.

Memory Ballooning: Never overcommit memory on Exadata. Memory should be statically allocated to DomUs so Dom0 never runs out of physical RAM.

2. Network Configuration

Virtual Bridges: DomU communicates with the Exadata Storage Servers via the InfiniBand/RoCE network through virtual functions (SR-IOV) or bridged interfaces managed by Dom0. Ensure these links are healthy in Dom0.

3. Space Allocation

Dom0 Local Disk: If Dom0's / or /var/log partitions fill up, the entire physical node (including all guest DomUs) can freeze or crash.

🧪 Test Case & Example Scenario

Scenario

A 2-node Exadata Virtual Machine RAC cluster experiences sudden performance degradation. DomU (Node 1) suddenly reboots and gets evicted from the cluster.

Step-by-Step Diagnostic Test Case

Step 1: Check the physical host (Dom0) status
Log into Dom0 via the ILOM or management IP to see if the hypervisor is alive and check resource usage:

bash

# Check Dom0 resource consumption (Xen environment)
xm list
# Or for newer KVM-based Exadata:
virsh list --all

Use code with caution.

Step 2: Investigate Dom0 system logs for errors
Look for hardware alerts or memory issues in Dom0:

bash

dmesg | tail -n 100
cat /var/log/messages | grep -i -E "error|fail|oom"

Use code with caution.

Step 3: Check virtual disk / block device status
If DomU lost connection to its virtual disks managed by Dom0:

bash

# Check if Dom0 is experiencing disk bottlenecks
iostat -xz 1 10

Use code with caution.

Step 4: Check DomU Console Output from Dom0
If the DomU is hung and unreachable via SSH, attach to its console from Dom0 to see the panic message:

bash

# For Xen:
xm console <domU_name>
# For KVM:
virsh console <domU_name>

Use code with caution.

🗣️ Interview Questions & Answers

Q1: A DomU is completely frozen and completely unreachable via SSH, but the physical Exadata storage and network are fine. How do you troubleshoot and fix this from a Linux Administrator perspective?

Answer:
First, I would log into Dom0 (the host hypervisor) since it controls the virtual machine.

I will run xm list (Xen) or virsh list (KVM) to check if the DomU state is hanging, crashed, or blocked.

I will check Dom0's /var/log/messages and dmesg to see if there are underlying physical hardware issues or Out-Of-Memory (OOM) killer events affecting the hypervisor.

I will attempt to connect to the virtual console using xm console <DomU> or virsh console <DomU> to view the last kernel panic or error message on the screen.

If it is completely non-responsive and impacting production, I will issue a hard reboot from Dom0 using xm destroy <DomU> followed by xm create <DomU> (or virsh destroy/start).

Q2: Why is it critical to restrict resource allocation (CPU/Memory) in DomU, and what happens if Dom0 runs out of CPU cycles?

Answer:
Dom0 is responsible for all I/O virtualization, network routing (InfiniBand/RoCE handling), and managing the lifecycle of the DomUs. If DomUs are allowed to consume 100% of physical CPU resources because Dom0 resources weren't properly pinned or reserved, Dom0 will starve.

When Dom0 starves, it cannot process heartbeat signals or network packets for the guest domains. In an Exadata RAC environment, this causes the Oracle Clusterware inside DomU to assume a network or disk split-brain scenario, resulting in an automatic node eviction (fencing) and forced reboot of the DomU.

Q3: How do you check the health and configuration of network plumbing between Dom0 and DomU on Exadata?

Answer:
Exadata uses high-speed InfiniBand or RoCE networks.

In Dom0, I would check the status of the physical network bonds and bridges using ip link show, ifconfig, or ovs-vsctl show (if Open vSwitch is used).

I would check /var/log/messages in Dom0 for link-down events on the physical ports (mlx4_core or mlx5_core drivers).

In DomU, I would check if the virtual interfaces are up and verify that the Single Root I/O Virtualization (SR-IOV) virtual functions are correctly attached and communicating by running ibv_devinfo or checking the Exadata-specific alert logs via dbmcli.

Question: You modified /etc/sudoers on an Exadata database node and now no one can use sudo. How do you troubleshoot and fix this without losing access or rebooting the node?

Answer:

The Problem: A syntax error in /etc/sudoers breaks the sudo command immediately because Exadata's security framework relies on valid parsing.

The Fix: Log in as root directly (via serial console, ILOM, or emergency SSH key if available), run visudo -c to find and correct the syntax error, and restore correct permissions (0440) to /etc/sudoers.

Example & Test Case

Example Scenario

An administrator attempts to grant grid user permissions to run specific Exadata diagnostic scripts via an incorrect syntax line in /etc/sudoers:

text

grid ALL=(ALL) NOPASSWD /opt/oracle/exapatch # Missing colon or invalid path syntax

Use code with caution.

Test Case

Trigger: Run sudo -u root /opt/oracle/exapatch as the grid user.

Failure Output:

text

>>> /etc/sudoers: syntax error near line 115 <<<
sudo: parse error in /etc/sudoers near line 115
sudo: no valid sudoers sources found, quitting

Use code with caution.

Resolution Validation: Log in via root, run visudo to fix the typo, save as /etc/sudoers.tmp, verify with visudo -c, and confirm functionality by re-running the test command successfully.

Pre-checks & Considerations

Pre-checks:

Always run visudo -c to check for syntax errors before closing the editor.
Verify file permissions of /etc/sudoers are strictly set to 440 and owned by root:root.
Ensure pam_loginuid.so or security modules are not blocking session initialization on Exadata dom0/domU.

Considerations:

Exapatch/Patching: Do not use temporary or loose permissions; automated Exadata patching utilities (patchmgr) expect standard secure permissions.
LDAP/Active Directory: If Exadata is integrated with corporate LDAP/ODSEE for authentication, check network timeouts or bind DN permissions if sudo hangs instead of throwing syntax errors.
Immutable Files: Never delete /etc/sudoers; keep a tested backup or active root terminal open during changes.

A common Linux NFS (Network File System) mount hang on an Oracle Exadata database node occurs when an unresponsive remote NFS server causes hard-mounted I/O requests to lock up database background processes or shell sessions.

Pre-Checks

Verify network reachability using ping and check if the remote server exports the path using showmount -e <NFS_Server_IP>.

Confirm RPC daemons and portmappers are running with rpcinfo -p <NFS_Server_IP>.

Check active kernel mount options via nfsstat -m or by reviewing /etc/fstab.

Ensure the local mount point directory exists and is empty before mounting. [1, 2, 3, 4, 5]

Considerations for Exadata

Never use hard mounts without timeo and retrans settings on Exadata database servers for non-critical shared storage, as a hanging NFS server will cause the entire database node or administrative scripts to hang indefinitely.

Use bg (background) and soft or proper hard,intr options so signals can interrupt hung NFS requests.

Network Isolation: Exadata uses dedicated interfaces (like client and private InfiniBand/RoCE networks). Ensure NFS traffic routes through the correct client network interface, not the high-performance private interconnect meant for ASM/RAC cluster communication.

Interview Q&A, Example, and Test Case

Question: An Oracle Exadata compute node experiences a system/shell hang when trying to access an NFS mount point. How do you troubleshoot and resolve this OS-level issue?

Answer Framework:

Identify the Hang: Run df -h or ls /mountpoint. If the command hangs without returning, the NFS mount is unresponsive.

Check Mount Type: Look at /proc/mounts or run nfsstat -m to see if it was mounted using hard mode without a low timeout value.

Trace the Issue: Use tcpdump or check dmesg | tail to spot RPC timeouts or "server not responding" messages.

Immediate Mitigation: Force unmount the hung share safely using umount -f -l /mountpoint (lazy and forced unmount).

Permanent Fix: Remount using optimized options: mount -t nfs -o rw,hard,intr,rsize=32768,wsize=32768,timeo=600,retrans=2 SERVER:/export /mountpoint. [1]

Example & Test Case

Scenario: Mounting an external backup share (192.168.10.50:/backups) on an Exadata database node (/mnt/backup).

The Faulty Command (Causing Hangs):

bash

mount -t nfs 192.168.10.50:/backups /mnt/backup

Use code with caution.

The Correct Test Case Command (With Safe Parameters):

bash

mount -t nfs -o rw,hard,intr,timeo=50,retrans=3 192.168.10.50:/backups /mnt/backup

Use code with caution.

Validation Test:
Run mount | grep nfs to confirm the active state, and run a write test touch /mnt/backup/test_file to verify operational read/write capability. [1]

If you'd like, let me know:

What specific error message or behavior you are seeing (e.g., Permission denied, RPC: Program not registered, or complete shell freeze)

Question: How do you troubleshoot an NFS mount failure on an Exadata compute node?

Answer:

Check network reachability to the storage server using ping or traceroute.

Verify that the RPC portmapper is active via rpcinfo -p <nfs_server_ip>.

Test the mount manually with verbose flags to isolate kernel or protocol version mismatches. [1, 2, 3, 4, 5]

Real-World Example & Test Case

Scenario

An Exadata database node (Database Server) needs to mount an external backup NFS share (192.168.10.50:/backup_share), but the mount command hangs or returns a mount.nfs: Connection timed out error.

Test Case Steps

Test Network and Port Accessibility:

bash

ping -c 3 192.168.10.50
telnet 192.168.10.50 2049

Use code with caution.

Expected: If telnet fails, a local or network firewall (or Exadata iptables/firewalld) is blocking TCP port 2049. [1, 2]

Check RPC Services on the Server:

bash

rpcinfo -p 192.168.10.50

Use code with caution.

Expected: Must list nfs, mountd, and portmap/rpcbind. If missing, start the NFS services on the server. [1, 2]

Verify Exports from the Client:

bash

showmount -e 192.168.10.50

Use code with caution.

Expected: Should output the allowed export list. If it throws RPC: Program not registered, the NFS daemon on the server side requires a restart. [1, 2]

Execute Manual Mount with Explicit Options (Exadata Best Practice):

bash

mount -t nfs -o rw,hard,noresvport,vers=3 192.168.10.50:/backup_share /mnt/nfs_backup

Use code with caution.

Note: Using noresvport is critical for high-availability environments like Exadata to prevent disconnections during failovers.

Oracle Exadata updates individual software packages and dependencies via RPMs managed automatically through the patchmgr utility or manually via standard RPM commands during node maintenance. [1, 2]

Exadata RPM Installation Example

In standard Exadata maintenance, individual RPMs or custom packages are installed using --additional_rpms via patchmgr, or directly via Linux rpm / yum / dnf commands on compute or storage nodes. [1, 2]

Example: Installing or Reinstalling an RPM Manually

If an RPM is corrupted or missing on an Exadata compute node, you verify and reinstall it using the Red Hat package manager: [1]

bash

# Check current RPM status
rpm -qa | grep exadata-dbmmgmt

# Reinstall or install the specific RPM package
rpm -ivh --force /scratch/u01/patchmgr/test/exadata-dbmmgmt-19.2.8.0.0-1.noarch.rpm

Use code with caution.

Example: Using patchmgr for Batch RPM Deployments

When updating multiple compute nodes via patchmgr, additional custom or required RPMs are passed using the additional RPMs parameter: [1]

bash

./patchmgr -upgrade -dbnodes /scratch/u01/patchmgr/dbnodes \
  --repo /scratch/u01/patchmgr/builds/exadata_iso.zip \
  --additional_rpms /scratch/u01/patchmgr/custom_rpms/ \
  --log_dir auto

Use code with caution.

Interview Q&A and Test Cases

Q1: How do you handle a missing or broken RPM on an Exadata compute node?

Answer: Identify the broken package using rpm -qa, locate the matching version from the official Exadata image patch bundle/ISO, and reinstall it using rpm -ivh --force or yum/patchmgr to restore functionality without breaking system dependencies. [1, 2]

Q2: What is a critical test case after installing or updating an Exadata RPM package?

Test Case Name: TC_EXADATA_RPM_VERIFY_CLI

Objective: Validate that management and database services respond correctly post-installation.

Steps:

Run dbmcli -e list dbm or cellcli -e list cell depending on whether it is a compute or cell node.
Check log files under /var/log/oracle/ or the specific component log directory for errors.
Execute Exacheck to ensure system health baselines remain green. [1]

If you'd like, I can provide:

Installing custom RPM packages on Oracle Exadata database servers requires keeping changes minimal and avoiding alterations to the kernel or RDMA network fabric. [1]

Exadata RPM Installation Guide

Prerequisites

Download the target .rpm package to a local directory (e.g., /tmp or /home/oracle).

Review official guidelines in the Oracle Exadata Database Machine Maintenance Guide. [1, 2, 3]

Example: Installing a Custom or Support RPM

Copy or download the RPM package to the target server:

bash

# cd /tmp
# ls -l custom-package.rpm

Use code with caution.

Install the package using yum to automatically resolve dependencies safely:

bash

# yum -y localinstall custom-package.rpm

Use code with caution.

Alternatively, if using standard rpm:

bash

# rpm -ivh custom-package.rpm

Use code with caution.

Test Case: Verification and Validation

Objective: Verify that the RPM package installed correctly, registers with the RPM database, and does not break Exadata system constraints.

Steps:

Check if the package is registered properly:
bash
# rpm -qa | grep custom-package
Use code with caution.
Inspect package installation scripts or post-install triggers if behavior is unexpected:
bash
# rpm -qp --scripts custom-package.rpm
Use code with caution.
Validate Exadata health and image status post-installation:
bash
# imageinfo
Use code with caution.

Expected Result: The command rpm -qa returns the exact package name and version, imageinfo reports a healthy system state, and core Exadata management utilities (like dbmcli or cellcli) remain fully operational.

The sudoers file (/etc/sudoers) controls which users and groups can run commands with administrative or root privileges in Linux.

Interview Q&A: Sudoers File

Question: What is the sudoers file and how do you edit it safely?

Answer: The /etc/sudoers file configures user permissions for the sudo command. You must always edit it using the visudo command. visudo locks the file to prevent simultaneous edits and runs a syntax check before saving to prevent locking yourself out of root access. [1]

Example sudoers Configuration

To grant a user or group specific privileges, entries follow this basic syntax:

text

# user/group   machine = (runas_user)   commands
john           ALL=(ALL:ALL)            ALL
%wheel         ALL=(ALL)                /bin/systemctl restart nginx

Use code with caution.

john: The specific username allowed to run the command.

ALL=(ALL:ALL): The user can run commands from any terminal/host, acting as any user and any group.

%wheel: The percent sign indicates a group name (like the wheel group in RHEL/CentOS).

/bin/systemctl restart nginx: Restricts the group to only run this exact command, rather than full root access. [1]

Test Cases for sudoers Configuration

Test ID	Scenario / Action	Expected Result
TC_01	Authorized user runs an allowed command with `sudo`.	Prompt requests the user's password; command executes with root privileges.
TC_02	Unauthorized user runs a command with `sudo`.	Access is denied with a "not in the sudoers file" error, and an incident is logged to `/var/log/auth.log`.
TC_03	User enters an incorrect password three times during `sudo`.	Authentication fails, command aborts, and a failure attempt is logged.
TC_04	Saving a `sudoers` file with a syntax error using `visudo`.	`visudo` rejects the save, displays an error line number, and asks to fix the mistake.
TC_05	Restricted user runs a command outside their permitted list (e.g., `rm -rf /` instead of allowed `systemctl`).	Access is denied and logged as an unauthorized command attempt.

Oracle Exadata does not have a built-in native RSA SecurID authentication agent, but you can integrate RSA multifactor authentication (MFA) at the operating system level (Linux PAM) on the database and compute nodes.

Overview of Integration

OS-Level Integration: Exadata compute nodes run Oracle Linux. You can install and configure the RSA Authentication Agent for PAM (Pluggable Authentication Modules) to handle SSH and system logins via SecurID.

Co-Managed vs. On-Premises: On on-premises Exadata racks, you have root access to install standard Linux-compatible security agents. On cloud deployments (like Exadata Cloud Service or Database@Azure), restrictions may apply to modifying base OS system files or PAM stacks on hypervisor/managed layers, though you generally manage the guest VMs (DOMUs). [1]

General Setup Steps (On-Premises / Guest VMs)

Register the Host: Register each Exadata compute node as an RSA Authentication Agent host on your RSA Authentication Manager.

Install the Agent: Download and install the official Linux PAM agent package from RSA onto the Exadata database compute nodes.

Configure PAM: Modify /etc/pam.d/sshd and related PAM configuration files to include the RSA PAM module (pam_rsa.so).

Update SSH Configuration: Ensure /etc/ssh/sshd_config allows keyboard-interactive or PAM-based authentication methods required for token/MFA prompts.

The root.sh script is a post-installation or patching utility provided by Oracle (found in the $ORACLE_HOME or Grid Infrastructure home) that must be executed with superuser privileges to configure system binaries, permissions, and environment files. [1, 2, 3]

What root.sh Does

Sets Permissions: Adjusts ownership and permissions for binaries, libraries, and directories that require root access. [1]

Creates Local Binaries: Copies essential executables (like environment setting scripts or database control tools) to default system paths such as /usr/local/bin. [1, 2]

Configures Clusterware/Restart: Handles unlocking and relocking home directories during Oracle Grid Infrastructure or database patching.

How to Run root.sh

Navigate to your Oracle home directory:

bash

cd /path/to/oracle/product/version/db_1

Use code with caution.

Execute the script:

bash

./root.sh

The primary configuration file for managing administrative privileges in Linux is located at /etc/sudoers. [1, 2]

Editing the Sudoers File Safely

Always use visudo: Never edit /etc/sudoers with a standard text editor like nano or vi directly. Run sudo visudo instead.

Syntax Checking: The visudo command locks the file and tests your syntax before saving. This prevents you from locking yourself out of administrative access due to a typo. [1, 2, 3]

Granting Privileges

Using Groups: The safest way to manage permissions is adding users to an admin group, such as wheel (Fedora/RHEL/Arch) or sudo (Debian/Ubuntu). Ensure the group line is uncommented in visudo:

text

%wheel ALL=(ALL:ALL) ALL

Use code with caution.

Adding a User to a Group: Run sudo usermod -a -G wheel username (replace wheel with sudo if on Ubuntu).

Direct User Rule: To grant an individual user full root rights, add this line in visudo:

text

username ALL=(ALL:ALL) ALL

Use code with caution.

[1, 2, 3, 4, 5]

Modular Configuration (/etc/sudoers.d)

Best Practice: Instead of altering the main /etc/sudoers file, create custom configuration files inside the /etc/sudoers.d/ directory.

File Naming: Create a file named after the user or rule (e.g., /etc/sudoers.d/username) using sudo visudo /etc/sudoers.d/username.

Permissions: Files in this directory must have strict permissions (such as 0440), meaning they are read-only and owned by root.

Oracle Linux system administration relies heavily on native Red Hat Enterprise Linux (RHEL) command-line utilities, alongside specialized Oracle-specific tools. [1, 2]

The primary commands essential for managing an Oracle Linux system are categorized by their administrative functions below.

📦 Package & Subscription Management

Oracle Linux uses dnf (or yum on older versions) to manage software installations and security patches.

dnf update – Updates all installed packages to their latest versions.

dnf install <package> – Installs a specific package from the Oracle Linux repositories.

dnf check-update – Checks for available security patches and software updates.

⚙️ System & Service Control

Oracle Linux uses systemctl to administer system daemons, services, and the built-in firewall. [1]

systemctl status <service> – Checks if a background service is running.

systemctl start <service> / systemctl stop <service> – Starts or stops a service immediately.

systemctl enable <service> / systemctl disable <service> – Configures whether a service automatically launches at boot time.

systemctl status firewalld – Verifies the current state of the native Linux firewall. [1, 2]

👤 User & Access Administration

Administrative tasks require evaluating privileges using sudo or switching to the root profile. [1, 2]

sudo <command> – Executes a single command with root-level privileges.

sudo -s – Elevates the current terminal session to a root shell using your own credentials.

useradd <username> – Provisions a new user account and creates a corresponding private group.

passwd <username> – Updates or sets the password for a local user account. [1, 2]

📊 Performance & Resource Monitoring

When hosting heavy enterprise workloads like an Oracle Database, tracking live CPU and memory metrics is critical. [1]

top – Displays an interactive, real-time dashboard of system resource usage and active processes.

free -m – Displays total, used, and available physical memory (RAM) and swap space in megabytes.

df -h – Shows disk space consumption across all mounted filesystems in human-readable formats. [1, 2]

🛠️ Oracle-Specific Management Tools

If you are managing specialized Oracle cloud or automation infrastructures, you will interact with proprietary command utilities: [1, 2]

spacecmd – The command-line tool used to interact with and manage Oracle Linux Manager infrastructures.

ol-automation-manager – Associated commands used to orchestrate enterprise deployment workflows within the Oracle Linux Automation Manager environment

Q: Can you install third-party security agents like RSA on Oracle Exadata compute nodes?

A: Yes, but with strict caveats. Because customers have root access to Exadata database compute nodes (domU / VM clusters), they can install third-party security, monitoring, or authentication agents. However, Oracle does not officially certify or support non-Oracle software. If an issue arises, the vendor (e.g., RSA) or the customer is responsible for troubleshooting. Furthermore, agents running at the root level must be carefully validated so they do not interfere with Exadata-specific processes, storage protocols (iDB), or rolling patching frameworks. [1, 2]

Real-World Example

Scenario: A corporate security policy mandates that all administrative SSH logins to database machines require two-factor authentication via the RSA SecurID Authentication Agent for Pluggable Authentication Modules (PAM).

Implementation: The RSA PAM agent is installed on the Exadata DomU Linux compute nodes and integrated with /etc/pam.d/sshd so that any interactive SSH session triggers an RSA passcode challenge before granting shell access.

Test Case for Validation

Test ID	Test Objective	Steps	Expected Result
TC_SEC_01	Verify SSH access with RSA Agent active	1. Attempt SSH login to Exadata DB node using valid credentials. 2. Provide the requested RSA SecurID one-time passcode.	Login succeeds and user is granted prompt access. Exadata alert logs show no anomalous latency or dropped connections.
TC_SEC_02	Verify behavior during Exadata rolling maintenance	1. Put the storage/compute node in maintenance mode. 2. Apply a minor Grid Infrastructure or image patch via `patchmgr`.	The third-party agent does not lock critical system binary files, block root-level scripts, or cause kernel panics/hangs during patching.

If you'd like, let me know:

Are you planning to install the agent on compute nodes (DomU) or try modifying storage cells?
What specific version of Exadata / Oracle Linux are you targeting?

I can help detail performance mitigation steps or specific port requirements.

mps-asia.com
Exadata: Guide to Testing Compatibility of Third-Party Security ...
7 Aug 2024 — Third-party security agents, referred to as security agents from here on, can have a significant impact on Exadata operations. Mos...
Oracle A-Team Chronicles
Exadata 3rd Party Agent and Security Agent Compatibility and ...
Oracle Database is the world's leading converged, multi-model database management system. Its performance, scalability, and

"A critical production server is running dangerously low on disk space in the /var directory, causing services to fail. You cannot add a new physical disk or reboot the server. How do you identify the cause, clear up space safely, and implement a temporary safeguard to keep the system running?"

Expected Answer (The Practical Approach)

An experienced Linux administrator will follow a methodical triage process:

Analyze: Check the current disk space metrics and identify the specific subdirectories or massive files consuming space.

Remediate: SAFELY clean up files. Never blindly delete active logs; truncate them instead to release disk descriptors without breaking active application processes.

Automate/Safeguard: Set up a log rotation strategy or an automated script to ensure the disk does not fill up immediately again.

Practical Example & Command Breakdown

Step 1: Identify the Root Cause

First, check filesystem usage using the human-readable disk free command:

bash

df -h /var

Use code with caution.

Next, navigate to the directory and find the top 5 largest directories or files using du (disk usage) sorted by size:

bash

du -ah /var | sort -rh | head -n 5

Use code with caution.

Assume the output shows a 45GB file located at /var/log/nginx/access.log.

Step 2: Safe Remediation (Do Not Use rm)

If you delete an active log file using rm, the process writing to it (e.g., Nginx) will keep the file descriptor open. The space will not be released to the OS until the service is restarted, and you run the risk of crashing the application.

Instead, truncate the file to zero out its size instantly while keeping the descriptor open:

bash

truncate -s 0 /var/log/nginx/access.log
# OR alternative method:
> /var/log/nginx/access.log

Use code with caution.

Step 3: Check for "Ghost" Deleted Files

If df -h still shows high utilization after deleting files normally, find unlinked files that are still being held in memory by active processes:

bash

lsof +L1
# OR
lsof | grep deleted

Use code with caution.

Fix: Gracefully reload or restart the specific Process ID (PID) discovered in the command output to completely free the space.

Test Case for the Interviewee

To verify a candidate's hands-on competency, an interviewer can present this specific diagnostic test case:

Scenario Context

A rogue background process is actively writing to a hidden directory, and df -h shows /var is at 100% capacity.

Test Execution & Commands

To simulate or resolve this exact behavior in a test environment:

bash

# 1. Verify the overall failure state
df -h /var

# 2. Track down the heavy directories without getting lost in subfolders
du -h --max-depth=1 /var | sort -hr

# 3. If a file was deleted but space wasn't reclaimed, find the culprit PID
lsof /var | grep -i deleted

# 4. Gracefully restart the service found by lsof to reclaim space
systemctl reload <service_name>

Use code with caution.

Here is your comprehensive interview guide for testing network latency on Oracle Exadata, tailored for a Linux Administration or Database Infrastructure role.

The Direct Answer

To measure network latency on Oracle Exadata, administrators primarily use ping, traceroute, and exachk, alongside specialized InfiniBand/RoCE tools like ibping or roce_admin. Network latency in Exadata is critical because the storage nodes and database nodes rely on a high-speed, low-latency backbone (historically InfiniBand, and RoCE in newer X8M+ generations) to achieve extreme performance.

Key Concepts to Know

InfiniBand / RoCE: The private networks connecting Exadata database servers (compute nodes) to cell servers (storage nodes).

Latency Thresholds: Private network latency should typically be sub-millisecond (often under 0.2 ms or 200 microseconds).

Packet Dropping: Higher latency or spikes usually indicate bad cables, faulty switch ports, or misconfigured MTU settings (Jumbo Frames).

An RPM dependency or conflict issue on an Oracle Exadata storage cell or database node can be resolved by identifying broken dependencies with rpm -qa or yum, and overriding or cleaning up conflicting files carefully without breaking Exadata-specific packages (cellos, kernel-uek).

Interview Question: How Do You Troubleshoot and Resolve an RPM Dependency or Installation Conflict on an Exadata Node?

Problem Scenario

During an Exadata cell or database server software update, an RPM installation or upgrade fails due to a blocked dependency or a conflicting file error (e.g., file /opt/oracle/cell/bin/... conflicts with file from package...).

Test Case & Example

Symptom: Running an RPM update fails with a conflict error.

bash

# rpm -ivh cell-custom-package-19.x.rpm
error: unpacking of archive failed on file /opt/oracle/cell/bin/cellcli; conflicting with preexisting file

Use code with caution.

Root Cause: An existing file was modified or left behind by a previous failed patch or manual file placement, causing the RPM database to get out of sync with the file system.

Step-by-Step Resolution / Test Case Action

Check package and verify RPM database:

bash

rpm -V cell-custom-package

Use code with caution.

Inspect the conflict and query ownership:

bsh

rpm -qf /opt/oracle/cell/bin/cellcli

Use code with caution.

Force or replace the conflicting file safely (if verified it belongs to the new package):

bash

rpm -ivh --replacefiles cell-custom-package-19.x.rpm

Use code with caution.

If it is a dependency issue (missing library):
Use Yum/DNF to auto-resolve dependencies from the local Exadata repository:

bash

yum localupdate cell-custom-package-19.x.rpm

Use code with caution.

Core Interview Question: Networking & Configuration Files

Question

What are cellinit.ora and cellip.ora, and where are they located in an Exadata environment?

Answer

These configuration files handle the network binding and discovery between database compute nodes and Exadata storage cells. [1, 2]

cellinit.ora: Contains the local management and IP configuration for the cell node.

cellip.ora: Lists the IP addresses or InfiniBand/RoCE endpoints of the storage cells that the database server is allowed to access.

Location: /etc/oracle/cell/network-config/ [1, 2]

Example Test Case & Troubleshooting

Scenario / Test Case

A database compute node cannot communicate with a specific Exadata storage cell. How do you troubleshoot this at the Linux OS and Exadata network layer?

Step-by-Step Verification Example

Check cellip.ora configuration on the Database Node:
Verify that the target cell's IP address is properly listed.

bash

cat /etc/oracle/cell/network-config/cellip.ora

Use code with caution.

Expected Output Example:

text

cell="192.168.10.15:5042"

Use code with caution.

Test Network Connectivity (InfiniBand / Private Network):
Exadata uses the iDB (Intelligent Database) protocol over port 5042. Test TCP/Port reachability using nc or telnet:

bash

nc -zv 192.168.10.15 5042

Use code with caution.

Check InfiniBand Interface Status:
Verify the state of the InfiniBand interfaces (ib0 or re0) on Linux:

bash

ip link show
ibstat

Use code with caution.

Review Storage Cell Services (cellcli):
Log into the storage cell via SSH and check if the core cell software (cellsrv) is running:

bash

cellcli -e "list cell detail"

Use code with caution.

[1, 2, 3]

Question:

"Describe a critical Linux OS issue you faced on an Exadata database or storage cell server, how you diagnosed it, and what your test case or resolution steps were."

Answer Structure (STAR Method):

The Problem: The root partition on an Exadata compute node filled up completely, leading to an unresponsive Oracle Clusterware stack (ohasd failing to write logs) and alerts in Enterprise Manager.

Diagnosis: Logged in via ILOM/console, ran df -h to see Use% at 100% on /, and used du -sh /* to isolate large directories. Found that /var/log or /var/crash contained massive core/vmcore files from a previous kernel panic. [1]

Resolution: Cleaned up outdated dumps, rotated logs, and adjusted kdump configuration or log rotation policies to prevent recurrence.

Example Scenario & Test Case

Scenario

An Exadata database node experiences a sudden cluster node eviction because the local root filesystem ran out of inode or disk space, preventing ASM and Grid Infrastructure daemon writes.

Step-by-Step Test / Troubleshooting Case

Check Disk Space Allocation

bash

df -h

Use code with caution.

Finding: /dev/mapper/VGExaDb-LVDbRoot shows 100% capacity used.

Identify Space Hogs

bash

du -h --max-depth=1 /var
du -h --max-depth=2 /var/crash

Use code with caution.

Finding: Large vmcore or core.* files generated by unexpected daemon crashes.

Safe Cleanup Action

bash

# Remove old core files older than 7 days safely
find /var/crash -type f -mtime +7 -exec rm -f {} \;
# Truncate large active log files if needed without restarting services
cat /dev/null > /var/log/messages

Use code with caution.

Verify System Health via Exadata Tools
Run the health check utility to ensure no underlying hardware or cell communication issues persist:

bash

exacheck -c db

Guide to Advanced Linux Command Mastery, Part 2

Direct Answer

As an Oracle Exadata Linux Administrator, your daily responsibility is maintaining high availability, optimal performance, and stability across database nodes (compute nodes) running Oracle Linux. While storage cells run Exadata Storage Server software, the compute nodes are standard Linux environments hardened by Oracle, requiring precise OS-level troubleshooting that accounts for underlying InfiniBand/RDMA over Converged Ethernet (RoCE) networks and clusterware.

Daily OS Administration Tasks on Exadata

Health & Hardware Monitoring: Checking /var/log/messages, dmesg, and dbserver.ctl for hardware or OS faults.

Storage & Space Management: Monitoring root (/), /u01, and /var filesystems to prevent Oracle Clusterware evictions due to full disks.

Performance Tracking: Analyzing CPU, memory, and I/O using vmstat, iostat, top, and Exadata-specific tools like exatop.

Kernel & Patch Management: Applying Exadata Quarterly Updates (QDUs) to update the Oracle Linux kernel and firmware using patchmgr.

Network Verifications: Monitoring RoCE/InfiniBand interfaces (mlx4_bond0 or re0) for packet drops or link flapping.

Real-World OS Issue: Node Eviction Due to Memory Starvation (HugePages Misconfiguration)

The Scenario

An Exadata compute node suddenly reboots or is evicted from the cluster. Oracle Grid Infrastructure (CRS) terminates because the OS became unresponsive or ran out of allocatable memory, triggered by standard Linux HugePages not matching the Oracle Database SGA requirements.

Scenario Breakdown

The Root Cause: Oracle Database instances are configured to use HugePages. However, after an OS reboot or database initialization parameter change, the vm.nr_hugepages value in /etc/sysctl.conf is lower than what the database SGA demands. The database falls back to allocating standard 4KB pages, causing the OS page table to swell exponentially, starving the OS of memory and freezing the node.

Complete Interview Test Case

Interview Question

"Can you describe a challenging Linux OS issue you faced on an Exadata compute node, how you diagnosed it, and what steps you took to resolve it permanently?"

EasyReliableDBA

Wednesday, 26 August 2026

Linux Administration for Ansible

Search This Blog