Oracle linux 8 administration for Ansible

Q. How to migrate from x9m to x10m

Migrating a database from Oracle Exadata Database Service on Cloud@Customer (ExaCC) X9M to an X10M (or newer X10/X10M-based infrastructure) is best achieved using an Oracle Data Guard switchover approach for near-zero downtime. 

---

Pre-Check Considerations

Version & Patch Levels: Ensure the target ExaCC X10M environment runs the exact same (or higher) Oracle Database and Grid Infrastructure release/patch level as the source X9M. 
Network & Connectivity: Verify bidirectional SQL*Net (1521) and SSH connectivity between the X9M source nodes and the target X10M VM cluster nodes for the oracle user. 
Database Parameters: Confirm DB_NAME matches on both systems, while DB_UNIQUE_NAME must be unique for the standby. 
Archivelog Mode: The source database must be operating in ARCHIVELOG mode. 
TDE & Passwords: Transparent Data Encryption (TDE) wallets must be active and synchronized, and SYS passwords must match on both systems. 

---

Step-by-Step Migration Process (Data Guard Method)

Step 1: Provision Target Infrastructure

Log in to the Oracle Cloud Infrastructure Console.
Create the new Exadata X10M Infrastructure and VM Cluster matching your required shape limits.
Ensure matching storage allocations for /u01/oradata and Fast Recovery Area (FRA). 

Step 2: Create a Data Guard Standby Database

Use the OCI Console, CLI, or Oracle Zero Downtime Migration (ZDM) to instantiate a physical standby database on the target X10M VM cluster. 

If configuring manually or via ZDM CLI, initiate the standby creation:

bash

# Example ZDM physical online migration setup command structure
zdmcli migrate database -sourcesource-db \
  -sourcenode source-node-listener \
  -targetnode x10m-node-listener \
  -targetdatabasename target_db \
  -method dgs

Step 3: Monitor Synchronization

Verify that Redo transport and apply lag are down to zero by running the following on the source primary database:

sql

SELECT dest_id, status, synchronizations_state, recovery_mode 
FROM v$archive_dest_status WHERE dest_id = 2;

Step 4: Perform Switchover Cutover

Perform the switchover to transition the X10M standby to the primary database role: 

Via OCI Console: Navigate to the Database details page, select Data Guard, click the action menu on the standby database, and choose Switchover.
Via SQL*Plus (Manual): 

sql

-- On target X10M standby database:
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;

-- On source X9M primary database:
ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY WITH SESSION SHUTDOWN;
STARTUP;

Step 5: Post-Migration Cleanup

Re-point client applications and connection strings (tnsnames.ora) to the new X10M database listener endpoints.
Verify application connectivity and workload performance on X10M.
Terminate the old database instances and de-provision the X9M VM cluster and infrastructure. 

---

Post-Migration Issues, Troubleshooting, & Challenges

Lag / Network Saturation: High redo log generation rates can saturate the interconnect network, causing Data Guard apply lag. Fix: Tune LOG_ARCHIVE_BUF_SIZE and increase redo log file sizes.
TDE Wallet Mismatch: If encrypted tablespaces fail to open on the target, the auto-login wallet (cwallet.sso) was not copied or synced properly. Fix: Manually copy and reopen the wallet on X10M nodes: ALTER SYSTEM SET WALLET OPEN IDENTIFIED BY "password";
Invalid Objects / Patch Discrepancies: If target Oracle Home had a higher patch bundle, datapatch should run automatically, but failures can leave components invalid. Fix: Run @?/rdbms/admin/utlrp.sql to recompile invalid objects.

Q. What is Oracle Linux 8

Oracle Linux 8 is an enterprise Linux operating system built for open-source performance and optimized for Oracle databases. 

Oracle Linux 8 - Documentation

What is Oracle Linux 8?

Enterprise OS: Free enterprise operating system based on Red Hat Enterprise Linux (RHEL) source code.
Dual Kernel: Includes the Red Hat Compatible Kernel (RHCK) and the tuned Unbreakable Enterprise Kernel (UEK).
Application Stream: Uses DNF package manager and module streams to manage software versions flexibly. 

Core Architecture and Features

Kernel: UEK provides advanced scaling, performance optimizations, and security features specifically tested for Oracle workloads.
System Management: Uses systemd to control services and system boot states.
Containers & Cloud: Built-in support for container tools like Podman, Buildah, and Skopeo, plus cloud-native tooling. 

Basic Administration Concepts

Package Management: Use dnf install <package> to add software and dnf update to apply patches.
Service Control: Use systemctl start <service> or systemctl enable <service> to manage background daemons.
Firewall Configuration: Use firewall-cmd to manage network ports and security zones.

Oracle Linux 8 uses the dnf package manager, systemd for service management, and nmcli/ip for networking, mirroring Red Hat Enterprise Linux (RHEL) 8. Administering the system effectively requires mastering a few foundational categories of commands. 

Here is a categorized guide to basic Oracle Linux 8 administration commands and the procedural steps to complete common tasks.

---

1. User & Group Management

Oracle Linux uses standardized commands to handle identity and access. Administrative privileges are controlled by the wheel group using sudo. 

useradd <username>: Creates a new user account.
passwd <username>: Sets or changes a user's password.
usermod -aG wheel <username>: Adds a user to the wheel group to grant sudo (root) access.
groupadd <groupname>: Creates a new group.
userdel -r <username>: Deletes a user along with their home directory. 

Steps to Create an Admin User:

Open the terminal and switch to root: sudo su -.
Create the user: useradd john.
Assign a secure password: passwd john.
Grant admin rights: usermod -aG wheel john. 

---

2. Package Management (dnf)

Oracle Linux 8 replaced yum with dnf as the default package manager, though yum still works as an alias. 

dnf check-update: Checks for available package updates.
dnf update: Installs all available system upgrades and security patches.
dnf install <package_name>: Installs a specific application or package.
dnf remove <package_name>: Uninstalls a package.
dnf search <keyword>: Searches the repository for matching software packages.

---

3. Service Management (systemctl)

Services, daemons, and system targets are fully controlled via systemd. 

systemctl start <service>: Starts a background service immediately.
systemctl stop <service>: Stops a running service.
systemctl restart <service>: Stops and restarts a service.
systemctl status <service>: Displays the current running state and recent logs of a service.
systemctl enable <service>: Configures a service to start automatically at system boot.
systemctl disable <service>: Prevents a service from starting at boot.
systemctl list-units --type=service: Lists all active services on the system. 

---

4. Networking Management

Oracle Linux 8 relies on NetworkManager. Use nmcli (command line) or the standard ip utility for network configuration. 

ip a or ip address: Displays all active network interfaces and assigned IP addresses.
ip route: Shows the system's current routing table.
nmcli device status: Shows the connection status of physical network devices.
nmcli connection show: Lists all configured network connection profiles.
nmcli connection up <interface>: Activates a network connection.
hostnamectl set-hostname <new-name>: Changes the system host name. 

---

5. System Monitoring & Performance

df -h: Shows disk space usage in human-readable format (GB/MB).
free -m: Displays total, used, and available RAM memory in Megabytes.
top or htop: Provides a real-time, interactive look at running processes, CPU usage, and memory.
uptime: Shows how long the system has been running and the current load average.
journalctl -u <service>: Reviews system logs specifically for a designated unit/service.

---

6. File Permissions & Navigation

ls -la: Lists all files in a directory, including hidden ones, with their permissions.
chmod <options> <file>: Modifies file permissions (read, write, execute).
chown <user>:<group> <file>: Changes the owner and group ownership of a file or directory.
tar -czvf archive.tar.gz /path: Compresses files into a backup tarball archive. 

---

7. Task Scheduling (cron)

Automating administrative scripts is handled via the time-based job scheduler. 

crontab -e: Opens your personal automation schedule in the default text editor (usually vim).
crontab -l: Lists all currently active scheduled cron jobs. 

Steps to Schedule a Daily Backup Script:

Run crontab -e in the terminal.
Add a cron expression at the bottom of the file (e.g., to run a script every night at 2:00 AM):
0 2 * * * /usr/local/bin/backup.sh
Save and close the editor (In vim, press Esc, type :wq, and hit Enter). 

---

8. System Power Controls

shutdown -h now: Powers off the machine immediately and safely.
reboot: Restarts the system safely

Oracle Linux 8 (OL8) basic administration covers NFS configuration, sudo troubleshooting, daily operations, Ansible automation, and Exadata Cloud@Customer (ExaCC) tasks.

---

NFS Mount: Commands, Considerations, and Troubleshooting

Pre-considerations

Ensure network connectivity and firewall rules allow NFS traffic on the server.
Install the required packages on both server and client: sudo dnf install -y nfs-utils.
Decide on the NFS version (NFSv3 or NFSv4/4.1/4.2). 

Essential NFS Commands

Start/Enable Server: sudo systemctl enable --now nfs-server
Export Shares (Server): sudo exportfs -arv
List Exports: showmount -e <server-ip>
Mount Share (Client): sudo mount -t nfs -o rw,sync <server-ip>:/remote/share /local/mountpoint
Permanent Mount (/etc/fstab):
<server-ip>:/remote/share /local/mountpoint nfs defaults,_netdev 0 0 

NFS Troubleshooting Steps

Check Service Status: Run systemctl status nfs-server on the server.
Verify Firewall: Run sudo firewall-cmd --permanent --zone=public --add-service=nfs followed by sudo firewall-cmd --reload.
Test Port Reachability: Use nc -zvw3 <server-ip> 2049 or telnet <server-ip> 2049 to verify TCP port 2049.
Stuck Mount Resolution: If a mount hangs, force unmount using sudo umount -f /local/mountpoint or sudo umount -l /local/mountpoint (lazy unmount). 

Sudo File: Details, Steps, and Troubleshooting

Best Practice Steps for Editing

Never edit /etc/sudoers directly with standard text editors like vi or nano without syntax verification. Always use visudo:
bash
sudo visudo


To check syntax errors on a specific file safely without breaking access:
bash
visudo -c -f /etc/sudoers
]

Sudo Troubleshooting Steps

Symptom: "sudo: parse error in /etc/sudoers near line X" or locked out of root privileges.
Resolution: Boot into single-user mode or emergency mode, correct the syntax error using visudo -c -f, or fix permissions on /etc/sudoers (must be Mode 0440, owned by root:root).

---

Daily Tasks and Ansible Commands

Typical Daily Admin Tasks

Monitor disk space (df -h) and system logs (journalctl -xe or /var/log/messages).
Manage package updates via sudo dnf update.
Check service health via systemctl.

Common Ansible Commands & Use Cases

Ad-hoc Ping Test:
bash
ansible all -m ping -i inventory


Execute Remote Shell Command:
bash
ansible web_servers -m shell -a "uptime" -i inventory


Manage Packages via Ansible:
bash
ansible db_servers -m dnf -a "name=nfs-utils state=present" --become

Exadata Cloud@Customer (ExaCC) Details & Use Cases

What it is: Oracle Exadata database hardware installed inside your corporate datacenter, managed remotely by Oracle Cloud Infrastructure (OCI) control plane via a secure tunnel. 
Admin Role: System administrators manage DomU (guest VM) OS administration, local storage mount points, patching orchestration coordination, and network integrations.
Use Cases: Running ultra-high performance Oracle Databases on-premises while keeping data localized to meet strict financial or governmental data residency laws.

---

Key Differences: Oracle Linux 8 vs. Oracle Linux 9

Base OS: OL8 is based on Red Hat Enterprise Linux (RHEL) 8 architecture, whereas OL9 is built on RHEL 9 upstream framework.
Default Kernel: OL9 ships with newer kernel iterations (UEK7 vs UEK6 on OL8 defaults).
Security & Authentication: OL9 tightens default SSH security profiles (disabling root password login strictly by default) and updates cryptographic policies (DEFAULT level is stricter).
Lifecycle: OL8 standard support runs through July 2029, whereas OL9 extends premier support support windows further out through June 2032.

---

Interview Questions & Answers

Q: How do you troubleshoot an unresponsive NFS mount point on Oracle Linux 8?
• A: First check network connectivity with ping and port status on 2049. Inspect kernel logs via dmesg | tail for RPC timeouts. If the mount is completely unresponsive, perform a lazy unmount using umount -l /mountpoint.
Q: Why should you use visudo instead of vi to edit /etc/sudoers?
• A: visudo locks the file against simultaneous edits and performs a syntax check upon saving, preventing configuration typos that could lock all users out of administrative sudo capabilities. 

Oracle Linux 8 uses standard Red Hat Enterprise Linux (RHEL) 8 commands and dnf for package management. 

Basic Administration Commands

Check OS version: cat /etc/oracle-release
Manage services: sudo systemctl [start|stop|restart|status] service_name
Update packages: sudo dnf update
Check disk usage: df -h
Check memory usage: free -m

Daily Administrative Tasks

Check system logs: sudo journalctl -xe or tail -f /var/log/messages
Monitor running processes: top or htop
Manage user accounts: sudo useradd username and sudo passwd username
Check network status: ip addr or nmcli connection show

Troubleshooting Steps

Service fails to start: Run sudo systemctl status <service> and check sudo journalctl -u <service> -n 50 for recent errors.
Disk full issues: Run du -sh /* to find large directories, then clear old logs or temporary files.
Network connectivity failure: Test local interface with ping -c 3 127.0.0.1, gateway with ip route, and DNS with nslookup oracle.com.

Ansible on Oracle Linux 8

Install Ansible Core: sudo dnf install -y ansible-core
Verify installation: ansible --version
Test ping module via ad-hoc command: ansible all -m ping -i hosts 

Exadata Cloud@Customer (ExaCC) Specifics

Check ExaCC health/status: Use Exadata-specific tools like dbnodeupdate.sh or cellcli (on storage cells).
OS monitoring: Standard commands (top, df -h, iostat -xz 2) apply to ExaCC domU guest dom0/domU environments, but avoid modifying grid/oracle user configurations or storage cell layouts manually without Oracle support guidance.

1. What tool manages RPM packages on Oracle Linux 8?

Answer: DNF (Dandified YUM) is the default package manager in Oracle Linux 8.
Details: It replaces the older YUM tool. It uses libdnf for backend operations and provides faster performance and better dependency resolution. The legacy rpm command is still available for low-level tasks.

2. How do you install a local RPM file and resolve missing dependencies automatically?

Answer: Use the dnf install command with the path to the local file.
Command: sudo dnf install /path/to/package.rpm
Details: Unlike the basic rpm -ivh command (which fails if dependencies are missing), DNF automatically checks enabled repositories and installs the required dependencies.

3. How do you find which package owns a specific file on Oracle Linux 8?

Answer: Use the dnf provides or rpm -qf command.
Command: dnf provides /etc/ssh/sshd_config or rpm -qf /etc/ssh/sshd_config
Use case: This helps when a configuration file is broken, and you need to reinstall only the package responsible for it.

---

Troubleshooting & Use Case

Scenario: Failed Transaction Due to Dependency Conflicts

The Problem: You try to update the system using sudo dnf update, but the process stops with a "Error: Transaction check error" or broken dependency conflict between two packages.

How to Troubleshoot

Check for duplicate or old packages:
• Run sudo dnf repoquery --duplicated to see if multiple versions of the same package exist.
Clean the DNF cache:
• Corrupted metadata can cause installation failures. Run sudo dnf clean all followed by sudo dnf makecache.
Use --skip-broken or handle manually:
• If a specific package is blocking the update, you can temporarily exclude it using sudo dnf update --exclude=packagename.

Real-World Use Case

A production server fails an update because an old third-party RPM package is locked to an older library version.

Resolution: Identify the conflicting package using dnf check. Remove the obsolete package using sudo rpm -e --nodeps packagename if it is not critical, then rerun sudo dnf update to complete the system sync safely.

Oracle Linux 8 (OL8) serves as a robust enterprise platform designed to maximize database and automation performance. This guide provides a comprehensive manual covering basic administration, real-world troubleshooting, daily operational checklists, Ansible tasks, and specialized Exadata Cloud at Customer (ExaCC) utilities. 

---

1. Basic Administration Commands & Step-by-Step Execution

Package Management with DNF

Oracle Linux 8 replaced yum with dnf for modern package management. 

Step-by-Step Update:
bash
# Check for available security updates
sudo dnf check-update --security
# Upgrade all packages to the latest stable release
sudo dnf upgrade -y


Enabling Repositories (e.g., EPEL or Developer repos):
bash
sudo dnf config-manager --set-enabled ol8_developer_EPEL
Service Management with Systemd

Manage active system tasks and daemons:
bash
# Enable a service to persist across reboots and start it immediately
sudo systemctl enable --now firewalld
# Query the runtime state of a background service
sudo systemctl status sshd
Storage & Filesystems (XFS/EXT4) [

Oracle Linux 8 defaults to XFS for its root filesystem structure.

Identify active mounts and partition geometries:
bash
df -hT                   # Displays file system layout and remaining free space
sudo lsblk -f            # Visualizes storage block hierarchy with UUID mappings


Extending an XFS volume safely on the fly:
bash
sudo lvextend -L +20G /dev/mapper/ol-root
sudo xfs_growfs /dev/mapper/ol-root

2. Routine Daily Admin Tasks & Health Checklists

Run these scripts or commands daily to verify host availability, mitigate performance degradation, and maintain operational stability. 

Performance Audits

bash

top -b -n 1 | head -n 20 # Captures high-overhead CPU and memory workflows instantly
free -g                  # Monitors consumption of available system RAM and SWAP allocations
iostat -xz 1 5           # Analyzes disk utilization anomalies and system I/O latency bottlenecks

Log Rotations & Kernel Health

bash

sudo journalctl -p err --since "1 day ago" # Captures modern systemd errors inside a rolling 24-hr window
sudo dmesg -T | grep -iE 'oom|segfault'   # Sweeps hardware ring buffers for fatal memory exhaustion 

Network Port Validation

bash

sudo ss -tulpn           # Lists active, unprivileged TCP/UDP network socket configurations

3. Detailed Troubleshooting & Failure Resolution

Use Case A: Rectifying File System Corruption

If a non-root mount goes offline or displays read-only behaviors, use the fsck utility. 

Unmount the corrupted logical volume immediately:
bash
sudo umount /dev/mapper/ol-data


Execute a filesystem verification process:
bash
sudo fsck -y /dev/mapper/ol-data

Use Case B: Debugging Broken DNF Stream Dependencies

OL8 leverages modular application streams that can occasionally conflict during system patching. 

Identify and reset an active module stream:

bash

sudo dnf module list python3
sudo dnf module reset python3




Synchronize transaction configurations:

bash

sudo dnf distro-sync

Use Case C: Live Diagnostics Tracking with Sosreport

When escalating server infrastructure bugs directly to Oracle Support: 

bash

sudo sos report --batch # Packages compressed system configuration metrics and system architecture data

4. Ansible Automation Orchestration for Oracle Linux 8

Oracle Linux 8 includes the Oracle Linux Automation Engine, which relies on standard ansible-core packages. 

Initialization Commands 

bash

sudo dnf install -y ansible-core
ansible --version

Playbook Use Case: Standardized Oracle Pre-Installation Sequence

This script prepares a vanilla OL8 host for an Oracle Database deployment: 

yaml

---
- name: Standardize Oracle Linux 8 Node Environment
  hosts: ol8_servers
  become: yes
  tasks:
    - name: Enable Required Software Repository Streams
      dnf:
        name: oracle-database-preinstall-19c
        state: present

    - name: Ensure Strict Kernel Parameters via Sysctl
      sysctl:
        name: fs.file-max
        value: '6815744'
        state: present
        reload: yes

    - name: Configure Secure Firewall Target Adjustments
      firewalld:
        service: oracle-db
        permanent: yes
        state: enabled
        immediate: yes

Execution Syntax: ansible-playbook -i inventory.ini prepare_host.yml

---

5. Exadata Cloud at Customer (ExaCC) Dedicated Administration

Administering an ExaCC system requires distinct commands to manage the separated database nodes (compute tier) and storage environments (cell tier) safely. 

Component	Management Tool	Primary Purpose / Role
Compute Nodes	Standard CLI / srvctl	Virtual machine lifecycle management and cluster scheduling
Local Storage Tier	ExaCLI / CellCLI	Managing flash caches, grid disks, and storage metrics safely
Multi-Node Automation	dcli (Distributed CLI)	Mass parallel command execution across multiple cells simultaneously

ExaCLI Usage & Syntax

Because root operations on the hypervisor layers of ExaCC are restricted, you must execute storage infrastructure verification queries remotely via exacli: 

bash

# General Syntax
exacli -c [username@]storage_cell_ip -e "LIST ALERTDEFINITION"

Operational ExaCC Use Cases

Use Case 1: Checking Storage Cell Alerts & Diagnostics

bash

# Check if a physical disk degradation has occurred on a storage node
exacli -c cell_admin@10.0.1.25 -e "LIST ALERTHISTORY WHERE severity='critical'"




Use Case 2: Inspecting Flash Cache Health Metrics

bash

# Verify if Exadata Smart Flash Cache hit ratios match baseline thresholds
exacli -c cell_admin@10.0.1.25 -e "LIST FLASHCACHE DETAIL"




Use Case 3: Executing Multi-Host Mass Diagnostics via dcli

bash

# Run a disk health inspection checklist across all storage cell groups in parallel
dcli -g /home/oracle/cell_group -l celladmin "cellcli -e 'LIST CELLDISK WHERE status != \"normal\"'"




Use Case 4: Running System Pre-checks Before Infrastructure Upgrades

bash

# Validate system state before rolling out Exadata bundle updates
sudo /opt/oracle.SupportTools/exachk

Oracle Linux 8 administration involves core CLI operations, package management via dnf, service control via systemd, Ansible automation, and specific considerations for environments like Exadata Cloud@Customer (ExaCC).

Basic Administration Commands & Daily Tasks

System & Service Management

Check service status: systemctl status <service_name>
Restart a service: systemctl restart <service_name>
Enable service on boot: systemctl enable --now <service_name>
View system logs: journalctl -u <service_name> -f (live tail)

Package Management (dnf)

Update all packages: dnf update -y
Install a package: dnf install -y <package_name>
Search for a package: dnf search <keyword>

Resource & Disk Monitoring

Check disk space: df -h
Check memory usage: free -m
Check top CPU/Memory processes: top or htop

---

Detailed Step: Troubleshooting High CPU/Memory

Identify the culprit: Run top and press P (for CPU) or M (for memory) to sort resource-heavy processes. Note the PID.
Inspect the process: Run ps -fp <PID> to see command details and ownership.
Check system logs: Run journalctl -xe or check /var/log/messages for Out-Of-Memory (OOM) killer events.
Gracefully stop the process: Run kill -15 <PID>. If unresponsive, use kill -9 <PID>. 

---

Ansible Commands & Use Cases on Oracle Linux 8

Oracle Linux 8 supports automation through the Oracle Linux Automation Engine or core Ansible packages. 

Ad-hoc package installation:
ansible all -m dnf -a "name=httpd state=present" --become
Service restart across a group:
ansible db_servers -m systemd -a "name=sshd state=restarted" --become
Check disk utilization on remote nodes:
ansible all -m shell -a "df -h" [

Use Case: Quickly patching 50 database nodes or verifying disk thresholds before an Oracle Grid Infrastructure upgrade.

---

Oracle Linux 8 on ExaCC (Exadata Cloud@Customer) Details

ExaCC Specifics: Exadata Cloud@Customer brings Oracle Exadata database infrastructure into your data center. The underlying Dom0 and DomU run Oracle Linux, managed strictly via ExaCC tooling (exadata.img.hwreg, oakcli, or cellcli for storage cells).
Daily Task Caution: Do not alter kernel parameters or update core InfiniBand/RDMA drivers manually using standard dnf update unless specified by Exadata quarterly patches (GI/Database and DomU image updates), as it can break high-availability clustering (RAC) and storage interconnects.

---

Key Differences: Oracle Linux 8 vs. Oracle Linux 9

Feature	Oracle Linux 9	Oracle Linux 10 / Oracle Linux 8
Default Kernel	Unbreakable Enterprise Kernel (UEK) R7 / Linux Kernel 5.14+	UEK R6 / Linux Kernel 5.4+
Python Version	Python 3.9 default	Python 3.6 / 3.8 default
Security/Crypto	OpenSSL 3.0, stricter default crypto policies	OpenSSL 1.1.1
Admin Tooling	Enhanced Web Console (Cockpit) integrations	Standard Cockpit, traditional RPM toolsets

---

Interview Questions & Answers

Q1: How do you enable the EPEL repository and install Ansible on Oracle Linux 8? 

Answer: Run # dnf install -y epel-release followed by # dnf install -y ansible. Verify the installation using ansible --version.

Q2: What is the difference between systemctl restart and systemctl reload?

Answer: restart completely stops and starts the service (causing downtime for that service), whereas reload forces the service to re-read its configuration files without dropping active connections.

Q3: Why should you avoid running an untargeted dnf update on an ExaCC DomU/Dom0 node?

Answer: It may update kernel packages or system libraries incompatible with the specific Exadata storage cell software or Grid Infrastructure stack, violating appliance support boundaries.

 Oracle Linux 8 uses the dnf package manager for RPM software management, replacing traditional yum with a faster and improved backend. 

Core DNF Commands for RPM Management

Install a package: Run sudo dnf install <package_name> to download and install an RPM along with all required dependencies.
Remove a package: Run sudo dnf remove <package_name> to delete the software safely from your system.
Update packages: Run sudo dnf update to refresh all installed packages to their latest versions.
List installed packages: Run dnf list installed to view everything currently active on your system.
Install a local RPM file: Run sudo dnf localinstall <file_name.rpm> or sudo dnf install ./<file_name.rpm> for manually downloaded packages. 

Managing Application Streams and Modules

Oracle Linux 8 splits software delivery into two primary repository components: 

BaseOS: Provides core operating system components and essential utilities as standard RPMs.
AppStream (Application Stream): Delivers additional user-space applications, databases, and programming languages using modules. 

You can manage different versions of software streams using specific module commands: 

List available modules: dnf module list
Enable a specific stream: sudo dnf module enable <module_name>:<stream>
Install a module stream: sudo dnf module install <module_name>:<stream>








You can manage SSH RSA key authentication and secure access on Exadata Cloud@Customer (ExaCC) X9M and X10M environments by configuring standard Linux OpenSSH keys for VM cluster node administration. [1, 2]
Overview
ExaCC bare-metal and virtual machine (VM) clusters run Oracle Linux where administrative access is controlled via SSH RSA/ECDSA key pairs rather than standard passwords. [1, 2, 3]
---
Example: Generating and Adding an RSA SSH Key in ExaCC X9M/X10M
Generate the RSA Key Pair on your local admin workstation or jump host:
bash
ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/exacc_rsa
Use code with caution.
[1]
View and Copy the Public Key:
bash
cat ~/.ssh/exacc_rsa.pub
Use code with caution.
[1]
Add the Public Key to the ExaCC VM Cluster via the Oracle Cloud Infrastructure Console:
• Go to Oracle Database > Exadata Database Service on Cloud@Customer > Exadata VM Clusters.
• Select your X9M or X10M VM cluster, click Add SSH Public Key, and paste the contents of exacc_rsa.pub. [1]
---
Test Case: Validating RSA Key Authentication
Objective: Verify that the Linux admin user can authenticate successfully to the ExaCC X9M/X10M database domU/VM nodes using the private RSA key.
Prerequisites: exacc_rsa (private key) placed in ~/.ssh/ with permissions set to 600.
Execution Command:
bash
ssh -i ~/.ssh/exacc_rsa -o BatchMode=yes opc@<exacc-vm-node-ip> "echo 'RSA Authentication Successful'"
Use code with caution.
Expected Result:
• Output returns: RSA Authentication Successful
• Exit status code is 0.
• No interactive password prompt appears.
---
Interview Questions and Answers
Q1: How do you enforce or add an RSA SSH key on an ExaCC X9M/X10M VM cluster node via the command line if manual access is allowed?
Answer:
Append the public key directly to the target user's authorized keys file on the specific node:
bash
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC..." >> /home/opc/.ssh/authorized_keys
chmod 600 /home/opc/.ssh/authorized_keys
chown opc:opc /home/opc/.ssh/authorized_keys
Q2: What troubleshooting steps do you take if an RSA key-based login to an ExaCC database node fails with a "Permission denied (publickey)" error?
Answer:
Check file permissions on the target server: ~/.ssh must be 700 and authorized_keys must be 600.
Verify that the correct public key is registered in the OCI console or authorized_keys file.
Run the SSH client with verbose mode enabled to trace the key negotiation failure: ssh -v -i ~/.ssh/exacc_rsa opc@<IP>.
Inspect the secure log on the ExaCC node: sudo tail -f /var/log/secure (or /var/log/auth.log depending on the Linux distribution configuration). 
Q3: Why does ExaCC prefer RSA/ECDSA key pairs over password authentication for X9M and X10M database node administration?
Answer:
Keys provide a significantly larger entropy space than human-readable passwords, neutralizing brute-force attacks over corporate or cloud networks and meeting stringent enterprise compliance mandates for engineered cloud infrastructure.


Linux Administration: OS Version Check
Example Commands
To verify the operating system and kernel version on an ExaCC or Exadata database/storage node, run:
bash
# Check OS release details
cat /etc/os-release

# Check kernel version
uname -r
Test Case Scenario
Objective: Validate that the environment has been successfully upgraded to Oracle Linux 8 (minimum requirement for modern Exadata X10M features and newer database compatibility).
Expected Output for /etc/os-release:
text
NAME="Oracle Linux Server"
VERSION="8.x"
ID="ol"
ID_LIKE="fedora"
VERSION_ID="8.x"


Expected Output for uname -r: A UEK6 or later kernel string (e.g., 5.4.32-... or higher depending on the specific Exadata software image bundle). 


Question: How do you verify the current OS and kernel versions on Exadata X9M or X10M infrastructure, and why is knowing the baseline Linux version critical when moving from X9M to X10M architectures?
Answer:
You check the OS using cat /etc/os-release and the kernel using uname -r. Knowing the baseline version is critical because Exadata X10M introduces newer AMD EPYC processor architectures that require a minimum of Oracle Linux 8 and UEK 6 for proper low-level hardware support, linear core scaling, and optimal resource management. Upgrading the system software bundle handles this OS and kernel alignment uniformly across Dom0 and DomU guests. 
Question:In an Exadata X9M or X10M environment, how do you verify and troubleshoot the RDMA over Converged Ethernet (RoCE) network fabric interface bonding and reachability at the Linux OS level if a storage-cell heartbeat warning triggers?
Context & Answer Framework
Exadata X9M/X10M systems rely on dual-port RoCE network cards for ultra-low latency cluster interconnect and storage access via the iDB protocol. Unlike standard TCP/IP networking, RoCE relies heavily on DCB (Data Center Bridging) and PFC (Priority Flow Control) configurations mapped via specific Linux network interface bonds (bondeth or native ip link). 
Check Bond and Interface Status:
Use modern iproute2 commands to check state and link layer parameters:
bash
ip -s link show
cat /proc/net/bonding/bond-roce


Verify RDMA Device Health:
Use RDMA core utilities to ensure the HCA (Host Channel Adapter) is active:
bash
ibv_devinfo -v
rdma link show


OS Version Nuance (X9M vs X10M):
• X9M typically runs Oracle Linux 7 with UEK5, utilizing legacy network.service or early NetworkManager profiles with older ibacm daemons.
• X10M runs Oracle Linux 8 with UEK6, utilizing systemd-networkd or refined NetworkManager with enhanced kernel linear scaling on high core counts. 


Scenario : An alert shows degraded RDMA throughput on database server node db01 after a switch maintenance window on the internal RoCE fabric. You need to run a validation test case to isolate whether the issue is an OS interface failure or a fabric configuration mismatch.
Step-by-Step Test Case Implementation
Test Step 1: Validate physical and bonded link state
bash
# Check if both underlying interfaces of the RoCE bond are up
ip link show up
Expected Output: Interfaces like eth5 and eth6 must show UP and master-assigned to the RDMA bond interface.
Test Step 2: Test RDMA connectivity across the fabric
Run the RDMA-core latency/bandwidth benchmark tool between the database node and a storage cell:
bash
# On the storage cell or target node, start the server component:

rping -s -v -a <local_roce_ip>

# On the database node (db01), run the client connection test:

rping -c -v -a <cell_roce_ip> -C 10
Pass Criteria: rping completes successfully with 10 iterations and zero packet/message drops, confirming that PFC and RoCE queues are passing traffic correctly across the fabric layer.
Test Step 3: Inspect kernel log for InfiniBand/RDMA stack errors
bash
dmesg | grep -iE 'rdma|roce|mlx5|bond'
Pass Criteria: Absence of mlx5_core link-down errors, driver timeouts, or local ACK timeout warnings from the Mellanox/NVIDIA adapter layer.
Question "A package installation failed with a dependency error on a production Red Hat server during a routine update. How do you troubleshoot and resolve this using RPM?"
Answer Structure
Identify the failure: Check the exact missing dependency or file conflict.
Inspect the RPM database: Query installed packages to see what version or conflicting file exists.
Resolve the issue: Use force flags carefully or let a high-level tool handle the dependency tree. 
---
Practical Example & Test Case
Scenario
You try to install a local package nginx-1.20.1-1.el7.x86_64.rpm, but the system throws a failed dependencies error or a file collision error.
1. Test Case: Simulating and Diagnosing the Issue
Run the RPM query or test install command to see what is wrong without making changes.
Check dependencies without installing (Test Mode):
bash
rpm -ivh --test nginx-1.20.1-1.el7.x86_64.rpm
Output error example:
text
error: Failed dependencies:
        libcrypto.so.10()(64bit) is needed by nginx-1:1.20.1-1.el7.x86_64


Identify which package provides the missing file/library:
bash
repoquery --whatprovides libcrypto.so.10
# Or using rpm directly if installed elsewhere:
rpm -qf /usr/lib64/libcrypto.so.10
2. Fixing the Daily Issue
Instead of using low-level rpm -ivh which fails on missing dependencies, switch to the high-level package manager (dnf or yum) which automatically pulls dependencies from repositories. 
Let DNF/YUM resolve and install the local RPM with dependencies:
bash
dnf localinstall nginx-1.20.1-1.el7.x86_64.rpm


Alternative: Handling a "file conflicts" error
If the error says a file is already owned by another package:
text
error: file /etc/nginx/nginx.conf from install of nginx conflicts with file from package old-nginx-1.18
• Solution: Replace the conflicting file or upgrade properly using the refresh flag:
  bash

  rpm -Uvh --replacefiles nginx-1.20.1-1.el7.x86_64.rpm
  

  Question: How do you use the ping command to check for network delay, and how do you handle a scenario where ping shows high delay or intermittent response?
Answer:
"I use the ping command with specific flags to control packet size and interval to measure network delay accurately. If a server shows high delay, I check packet loss, trace the network path using traceroute, and inspect interface errors using ip -s link." 
---
Example Usage with Delay Control
To add a delay (interval) between each ping packet, use the -i flag (in seconds). To set a deadline or packet count, use -c.
bash
ping -c 5 -i 2 8.8.8.8
-c 5: Send only 5 packets and stop.
-i 2: Wait 2 seconds between sending each packet (default is 1 second).
8.8.8.8: The target IP address (Google Public DNS).
Sample Output
text
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=14.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=15.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=45.1 ms  # Spike in delay
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=14.5 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=117 time=13.8 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 8011ms
rtt min/avg/max/mdev = 13.801/20.520/45.102/12.351 ms
Test Case for Troubleshooting High Ping Delay
When an application complains about lag or time-outs, execute this structured test plan.
Test Case 1: Local Loopback Test
Action: ping -c 4 127.0.0.1
Expected Result: time < 1ms
Purpose: Verifies that the local network stack and kernel interface are functioning properly. High local delay indicates a local CPU or OS resource bottleneck.
Test Case 2: Gateway Reachability Test
Action: ping -c 5 $(ip route | grep default | awk '{print $3}')
Expected Result: time < 5ms on a wired LAN.
Purpose: Isolates whether the delay originates inside your local network (switch, local router, or physical cable) or outside on the internet.
Test Case 3: Path Trace Analysis
Action: traceroute 8.8.8.8 (or mtr 8.8.8.8 for real-time diagnostics)
Expected Result: Clear sequential hops with stable response times.
Purpose: Pinpoints the exact network hop or router interface where the packet delay spikes or packet loss begins.