Create and Mount NAS on ExaCC

Project Overview & Architectural Design

1. Project Description

The goal of this project is to integrate enterprise-grade Dell EMC NAS Storage (such as Isilon/PowerScale or Unity) with an Oracle Exadata Cloud@Customer (ExaCC) 10xM X9M/X10M infrastructure.

To optimize security and network bandwidth, the storage infrastructure isolates data traffic by mounting the NAS volumes exclusively over the dedicated ExaCC Backup Network rather than routing it through the client (application) or management networks. Management and configuration validation are conducted securely via the Management IP Network.

2. Architecture & Parameters

Mount Point Name (Local target): /mnt/exacc_backup_nas
Mountpoint Source (NFS Export): 192.168.12.50:/ifs/data/exacc_backups (where 192.168.12.50 is the Dell EMC SmartConnect zone or IP assigned to the ExaCC Backup VLAN subnet)
Management IP (Dell EMC Storage): 10.20.30.100 (Used by administrators for provisioning and access control rules via CLI/UI)
ExaCC Cluster Group Name: dba_backup_grp
ExaCC Group ID (GID): 1002 (Standardized OS-level GID across all Exadata database nodes to avoid ownership conflicts)

3. Workflow

[ Dell EMC NAS Storage ] 
    │ (Provisioned via Management IP: 10.20.30.100)
    ▼
[ NFS Export: 192.168.12.50:/ifs/data/exacc_backups ]
    │
    ▼ (Traverses ExaCC Backup Network Routing)
[ ExaCC 10xM DB Nodes (Nodes 1 to N) ]
    │
    ▼ (Mounted locally using GID 1002 / UID 1001)
[ Local Mount Path: /mnt/exacc_backup_nas ]

---

Pre-Considerations & Operational Challenges

Pre-Considerations

Network Isolation: Verify that the ExaCC Backup Network routing tables (/etc/sysconfig/network-scripts/route-bond1 or equivalent depending on the OEL version) correctly route storage traffic through the backup interfaces rather than defaulting to the client interface.
Storage Exports: The Dell EMC NAS must have export rules matching the exact IP range of the ExaCC Backup Network VLAN, granting root_squash or no_root_squash permissions depending on corporate security policies.
UID/GID Uniformity: Ensure the Group Name (dba_backup_grp) and Group Number (1002) are perfectly synced across all active database nodes in the ExaCC cluster to prevent permission conflicts during concurrent backup processes.

Challenges & Mitigations

NFS Stale Mounts & Timeouts: Network blips on the backup network can cause the kernel to hang on direct I/O requests.
• Mitigation: Use the soft or intr mount flags carefully, or optimize timeo and retrans parameters so RMAN or filesystem utilities fail gracefully instead of causing kernel panics.
Silent Local Directory Fills: If the NAS share drops or disconnects, applications writing to /mnt/exacc_backup_nas will write directly to the local root file system, filling up the node's local disk.
• Mitigation: Explicitly check that the mount point is an active NFS partition before executing jobs, or configure the parent directory permissions to deny writes if unmounted.

---

Step-by-Step Technical Implementation Commands

Step 1: Storage Side Configuration (Via Management IP)

Run these commands on the Dell EMC PowerScale/Isilon CLI to create the export rule for the ExaCC backup network nodes:

bash

# Connect to EMC Management IP
ssh admin@10.20.30.100

# Create the NFS export allowing access specifically to the ExaCC Backup Network Subnet
isi nfs exports create --paths=/ifs/data/exacc_backups \
  --clients=192.168.12.0/24 \
  --root-clients=192.168.12.0/24 \
  --description="ExaCC 10xM Backup Network Storage"

Use code with caution.

Step 2: ExaCC OS Configuration (On all DB Nodes)

Log into your ExaCC database nodes as root to create groups, directories, and apply mounts:

bash

# 1. Create the dedicated group matching the requirements
groupadd -g 1002 dba_backup_grp

# 2. Create the local directory mount point 
mkdir -p /mnt/exacc_backup_nas

# 3. Mount the share over the backup network manually for validation
mount -t nfs -o rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,timeo=600,actimeo=0,vers=3 \
  192.168.12.50:/ifs/data/exacc_backups /mnt/exacc_backup_nas

# 4. Set appropriate ownership using the required group identifier
chown -R oracle:dba_backup_grp /mnt/exacc_backup_nas
chmod 775 /mnt/exacc_backup_nas

Use code with caution.

Step 3: Persistence Configuration

Add the entry to the /etc/fstab file on all cluster nodes to ensure persistent mounting on reboot:

text

192.168.12.50:/ifs/data/exacc_backups  /mnt/exacc_backup_nas  nfs  rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,timeo=600,actimeo=0,vers=3  0 0

Use code with caution.

---

Test Cases & Validation Scenarios

Test Case ID	Scenario Description	Expected Outcome	Validation Command
TC-01	Verify Mount Routing Path	Traffic must route strictly through the Backup interface (bond1 or alternative backup link).	ip route get 192.168.12.50 (Verify it returns the backup interface)
TC-02	File Write & Permissions	The oracle user belonging to dba_backup_grp must successfully write data.	su - oracle -c "touch /mnt/exacc_backup_nas/test.file"
TC-03	Failover / Unmount Safety	Writing must fail gracefully if the storage array target is unreachable.	df -h | grep /mnt/exacc_backup_nas

---

Interview Questions & Answers

Q1: Why do we isolate NAS mounts onto the Backup Network instead of using the Client Network in an Exadata environment?

A: Backup traffic involves massive, prolonged sequential I/O patterns. Running backups over the Client Network would exhaust the available bandwidth reserved for real-time application database queries (OLTP/DSS transactions). Isolating this traffic to the dedicated Backup Network ensures predictable database performance and maintains strict network perimeter security.

Q2: If an ExaCC node reboots and the NAS array is temporarily down, how do you prevent the local root partition from filling up if a backup script triggers?

A: First, mount points should be set up inside /etc/fstab using optimal timeouts (timeo, retrans). Second, backup scripts should validate whether the path is a true active mount point using commands like mountpoint -q /path or checking df -T /path before running any write operations. Lastly, setting the local directory permissions to 000 when unmounted prevents accidental writes by any non-root user.

Q3: What role does GID consistency play across an ExaCC 10xM RAC cluster when sharing a NAS mount?

A: In a Real Application Clusters (RAC) environment, multiple nodes write to the exact same shared NFS repository concurrently. If Group Names or Group IDs (GIDs) differ across the nodes (e.g., node 1 uses 1002 and node 2 uses 1005 for the same backup group name), files written by one node will display permission errors or broken ownership attributes when accessed by another node, causing database backups or restorations to fail.

---

Q: How do you mount an EMC NAS share on an ExaCC X10M cluster over a backup network while using the management IP? Explain the parameters required.

A: In an ExaCC X10M environment, traffic isolation is critical. We use the Management IP / Management Network solely to authenticate, discover, and configure the storage parameters (e.g., executing API calls or managing the EMC Unity/Isilon array). However, the actual NFS data path (read/write payload) is routed through the Backup Network (typically a dedicated high-bandwidth interface like eth1 or a specific bonded VLAN interface).

The configuration uses the following specific values:

Mount Point Name: /db_backup/emc_nas (Local path instantiated across all database virtual machines (VMs) in the cluster).
Mountpoint Source (NFS Export Path): 192.168.12.50:/ifs/data/exacc_backup (Where 192.168.12.50 represents the target IP of the EMC NAS data interface sitting squarely on the Backup Network).
Group Name: asmadmin or oracle (Depending on the backup agent or RMAN architecture).
Group Number (GID): 1001 (Standard GID for oracle/asmadmin on ExaCC VMs; it must match the UID/GID permissions assigned on the EMC NAS export).

Q: What mount options are critical when mounting EMC NAS on ExaCC for Oracle backups?

A: The mount must be configured with specific flags in /etc/fstab to ensure Oracle database stability and prevent cluster evictions during network hiccups:

bash

192.168.12.50:/ifs/data/exacc_backup  /db_backup/emc_nas  nfs  rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,noac,vers=3,suid 0 0

Use code with caution.

hard: Prevents data corruption; RMAN will retry indefinitely if a drop occurs.
noac: Disables attribute caching, ensuring all nodes see file updates instantly.

---

2. Project Architecture & Workflow

Project Description

Implementing an automated database backup infrastructure for a critical financial application hosted on Oracle Exadata Cloud@Customer (ExaCC) X10M. The database size is 150 TB, requiring daily incremental and weekly full backups. The backups must be offloaded from the primary Exadata storage to an enterprise Dell EMC PowerScale (Isilon) NAS array to comply with off-site retention policies without consuming production InfiniBand/RoCE bandwidth.

       [ Oracle ExaCC X10M VM Cluster ]
          /                        \
  (Mgmt IP: 10.x.x.x)        (Backup IP: 192.168.12.x)
        /                            \
[ Corporate Mgmt Network ]     [ Dedicated 25GbE Backup Net ]
        \                            /
  (Mgmt IP: 10.x.x.y)        (NAS Data IP: 192.168.12.50)
       [ Dell EMC PowerScale (NAS Storage System) ]

Project Requirements

Zero Production Impact: Backup traffic must be completely isolated from the client/application network and the internal RoCE fabric.
High Throughput: Utilize the dedicated 25GbE Backup Network ports on the ExaCC X10M database nodes.
Multi-Node Consistency: The NAS share must be concurrently mounted with identical privileges across all ExaCC database nodes (node1, node2, etc.).

Technical Workflow

Provisioning: The Storage Administrator provisions an NFS export on the EMC NAS and restricts access rules strictly to the ExaCC Backup Network IP range.
Routing Verification: Routing rules are implemented on the ExaCC OS layer to force all traffic bound for the EMC NAS Data IP through the backup interfaces (bond1 / ethX).
Directory Creation: Create the directory /db_backup/emc_nas across all cluster nodes and change ownership to oracle:asmadmin.
Mount Execution: Mount the storage path using the backup network IP string.
RMAN Integration: Configure Oracle Recovery Manager (RMAN) parameters to target the newly established mount path for daily backup sets.

---

3. Pre-Considerations & Challenges

Pre-Considerations

UID/GID Synchronization: The oracle user ID (typically 1001) and asmadmin group ID (1001 or 1002) on the ExaCC cluster must perfectly match the user/group permissions mapped on the EMC NAS SmartConnect zone or export rule. Mismatches will result in catastrophic Permission Denied errors during RMAN initialization.
Firewall & Security Rules: Network security groups (NSGs) or local iptables must explicitly open ports 111 (RPC) and 2049 (NFS) between the ExaCC Backup IPs and the EMC NAS Data IPs.

Challenges & Mitigations

Challenge: Asymmetric Routing. The backup request initializes over the backup network, but the OS tries to reply or send control packets via the default gateway on the Management network.
• Mitigation: Implement strict Policy-Based Routing (PBR) via ip rule and separate routing tables on the ExaCC nodes to guarantee that any traffic destined for the NAS subnet goes strictly out of the backup network interface.
Challenge: Mount Freezes During Network Blips. If a network switch on the backup line drops packets, a standard hard mount can cause the Oracle database processes interacting with it to hang permanently, potentially causing instance eviction.
• Mitigation: Use the precise mount parameters rw,bg,hard,nointr combined with customized RMAN channel configurations to ensure proper timeouts and prevent kernel locks.

---

4. Test Cases & Verification Examples

Q1: How do you mount an EMC NAS storage share on an Oracle Exadata Cloud@Customer (ExaCC) environment?

Answer: Mounting an external NAS (like Dell EMC Isilon/PowerScale) on ExaCC requires configuring the OVM/KVM guest VMs (database nodes), not the bare-metal storage cells.

Ensure the ExaCC client/backup network has a route to the EMC NAS data ports.
Create the local directory (Mount Point Name) on all DB nodes.
Configure /etc/fstab with the Mount Point Source (NAS IP/export path) and specific mount options (rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,vers=3,timeo=600).
Apply matching ownership and permissions using the specific Group Name and Group Number (GID) to ensure Oracle binaries can access it.

Q2: Why is the exact Group Name and Group Number (GID) critical when mounting NAS on ExaCC?

Answer: ExaCC environments use strictly standardized users and groups (e.g., oracle:oinstall or grid:asmadmin). If the EMC NAS export is configured with a different GID or lacks root squashing overrides, the oracle user on the ExaCC nodes will encounter Permission Denied or ORA-15124 errors. The NAS export permissions must explicitly map or allow the ExaCC oracle UID and oinstall/dba GID.

---

Project Architecture & Workflow

Project Description

The project involves expanding the staging and backup capabilities of a mission-critical Oracle 19c database running on an Exadata Cloud@Customer (ExaCC) X10M platform. Due to ExaCC local NVMe storage constraints and cost structures, cold data, data pump exports, and RMAN archivelog backups must be offloaded to an existing Dell EMC PowerScale (Isilon) NAS array.

Requirement Parameters

Mount Point Name: /mnt/exacc_nas_staging
Mount Point Source: 10.230.45.50:/ifs/data/exacc_share
Group Name: oinstall (or custom application group like nasadmin)
Group Number (GID): 1001 (Must match across all ExaCC VM cluster nodes)

Technical Workflow

[ EMC PowerScale NAS ] 
       │ (Export IP: 10.230.45.50)
       ▼ [Client/Backup Network VLAN]
[ ExaCC X10M DB Node 1 ] ───> /etc/fstab ───> Mount Point: /mnt/exacc_nas_staging (GID: 1001)
[ ExaCC X10M DB Node 2 ] ───> /etc/fstab ───> Mount Point: /mnt/exacc_nas_staging (GID: 1001)

Network Provisioning: Route the EMC NAS storage traffic through the ExaCC Client Network or a dedicated Backup Network VLAN, avoiding the internal InfiniBand/RoCE cluster interconnect.
NAS Export Setup: The storage admin provisions an NFSv3/v4 export on the EMC array, applying an export rule that trusts the ExaCC VM IP addresses and maps root to nobody while preserving standard user/group permissions.
OS Mount Configuration: The system/cloud administrator creates the directory structure on all ExaCC database nodes and updates /etc/fstab.
Database Verification: The Oracle database utilizes the mount point for direct file I/O operations (e.g., External Tables, Data Pump, or RMAN).

---

Pre-Considerations & Challenges

Pre-Considerations

Network Isolation: Ensure NAS traffic does not saturate the ExaCC client network, potentially causing application timeouts. Use the ExaCC backup network if available.
NFS Protocol Selection: NFSv3 is generally preferred for Oracle performance due to lower locking overhead, but NFSv4 may be mandated if strict security/Kerberos authentication is required.
MTU Size: Ensure MTU 9000 (Jumbo Frames) is end-to-end consistent across the ExaCC VM, network switches, and the EMC NAS front-end ports to avoid packet fragmentation.

Challenges & Mitigations

Challenge: Single Point of Failure (SPOF): If a single NAS controller goes down, database processes writing to the mount point might hang (stat command freezes).
• Mitigation: Mount with the bg (background) and hard options so that operations retry indefinitely rather than corrupting data, and utilize EMC SmartConnect DNS for multi-node load balancing.
Challenge: Oracle Cloud Control Restrictions: Cloud@Customer architectures limit root infrastructure access.
• Mitigation: Perform the mount operations inside the User-Managed Guest VMs (DomU) where you retain full root/sudo access, rather than the Oracle-managed Dom0.

---

Test Case Example

Scenario

Validate that the /mnt/exacc_nas_staging mount point is highly available, possesses the correct GID permissions, and allows the oracle user to execute a high-throughput Data Pump export.

Execution Steps

Mount Execution & Verification:
bash
sudo mkdir -p /mnt/exacc_nas_staging
sudo mount -t nfs -o rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,vers=3 10.230.45.50:/ifs/data/exacc_share /mnt/exacc_nas_staging


Permission Check:
bash
# Confirming the directory reflects the correct GID (1001 / oinstall)
ls -ld /mnt/exacc_nas_staging
# Expected Output: drwxr-xr-x 2 oracle oinstall 4096 Sep 16 00:00 /mnt/exacc_nas_staging


Functional I/O Test:
bash
# Act as the oracle user and test file creation
sudo su - oracle
touch /mnt/exacc_nas_staging/test_file.txt
dd if=/dev/zero of=/mnt/exacc_nas_staging/test_write.img bs=1M count=1000
Expected Result

The dd command completes without I/O errors and achieves speeds matching your network limitations (e.g., >800 MB/s on a 10GbE link). The file retains oracle:oinstall ownership across all nodes in the ExaCC cluster.

Project Description & Requirement

Project Description

The organization is deploying an Oracle Exadata Cloud@Customer (ExaCC) X10M environment to consolidate core database workloads. To facilitate patch management, database backups, data pump exports, and external file staging, the environment requires access to a centralized, highly available Network Attached Storage (NAS) system (such as Oracle Cloud Infrastructure File Storage Service - FSS, or on-premises ZFS/NetApp).

Project Requirement

Configure a persistent NAS mount across all database nodes (VM Cluster) in the ExaCC X10M infrastructure.

Mount Point Name: /u02/app/oracle/oradata/shared_nas
Mountpoint Source (NAS Export): 10.201.35.40:/export/exacc_shared_stage
Group Name: asmdba
Group Number (GID): 1010 (Note: The standard oracle user must be a member of this group to read/write backups/data).

---

 Architecture & Workflow

ExaCC X10M uses a highly isolated virtualized architecture. The workflow to mount external storage follows these strict infrastructural paths:

[ NAS Storage Server ] 
         │ (NFS / RPC Protocol)
         ▼
[ Customer Client Network (VLAN) ]
         │ (Through Dedicated Client Interfaces)
         ▼
[ ExaCC X10M KVM Guest VMs ] ──> [ Mount Point: /u02/app/.../shared_nas ]

Storage Provisioning: The storage administrator provisions an NFS export on the corporate network and opens access to the ExaCC Client Network IP range.
Network Routing: The NFS traffic enters the ExaCC X10M rack via the Client Network (not the private RoCE interconnect network, which is strictly reserved for internal Exadata storage and RAC cache fusion).
OS-Level Mounting: The system administrator configures rpcbind and autofs (or standard fstab) on the KVM guest VMs to attach the remote directory.
Permission Alignment: Permissions are adjusted to match the oracle:asmdba UID/GID stack so that Oracle databases can natively write to the mount.

---

 Pre-considerations & Challenges

Pre-considerations

Network Isolation: Ensure the NAS storage is accessible via the ExaCC Client Network. ExaCC explicitly blocks external traffic over the backup and management networks.
UID/GID Consistency: The group asmdba with GID 1010 must exist and be identical across all VM nodes in the cluster.
Mount Options: Use recommended Oracle NFS mount parameters (rw,bg,hard,nointr,rsize=524288,wsize=524288,tcp,vers=3,timeo=600,actimeo=0) to prevent database hangs during network hiccups.

Challenges & Mitigations

The Single Point of Failure (SPOF) Risk: If the NAS goes down, Data Pump or backup jobs will freeze.
• Mitigation: Use hard mount options with bg (background) so the VM boot process doesn't hang if the NAS is unavailable.
Firewall Blockages: ExaCC guest VMs have local iptables/firewalld rules active by default.
• Mitigation: Explicitly open standard NFS ports (111, 2049) on both the local guest VM firewall and corporate network switches.

---

 Test Cases with Examples

Test Case 1: Network Reachability & RPC Port Check

Objective: Verify the ExaCC node can talk to the NAS storage server over the client network.
Command:
bash
rpcinfo -p 10.201.35.40
showmount -e 10.201.35.40


Expected Output: A list of exported paths matching /export/exacc_shared_stage.

Test Case 2: Read/Write Permission Validation as Oracle User

Objective: Ensure the database layer can write files with the correct group ownership (asmdba).
Command:
bash
sudo su - oracle
touch /u02/app/oracle/oradata/shared_nas/test_file.txt
ls -ln /u02/app/oracle/oradata/shared_nas/test_file.txt


Expected Output: The file is successfully created showing GID 1010.

---

 ExaCC X10M Interview Questions & Answers

Q1: Can we use the ExaCC RoCE (RDMA over Converged Ethernet) network to route our custom corporate NAS traffic?

A: No. The RoCE network on ExaCC X10M is an isolated, ultra-low latency internal fabric used exclusively for communication between the database KVM guests and the Exadata Storage Cells (and inter-node RAC Cache Fusion). All external custom NAS or NFS traffic must be routed through the Customer Client Network.

Q2: If an NFS mount hangs on an ExaCC node, how do you prevent it from crashing the Oracle RAC database instances?

A: To prevent database instance crashes or severe kernel hangs, we use optimized mount flags in /etc/fstab. Specifically, actimeo=0 forces immediate attribute visibility, and using hard instead of soft guarantees data integrity while bg ensures that if a node reboots, a missing NAS mount won't stop the cluster nodes from booting up.

Q3: How do you handle a scenario where the GID for asmdba (1010) on the ExaCC nodes conflicts with a pre-existing GID on your corporate NAS server?

A: ExaCC grid infrastructure and database users are pre-created during the OCI cloud control plane provisioning. If a conflict occurs, we must employ NFS User/Group ID Mapping (idmapd) on the OS level, or configure the NAS storage controller to map the incoming UID/GID traffic to its local equivalent, ensuring that permissions remain aligned without modifying the underlying ExaCC standard deployment template.

Q4: Why is it critical to verify rpcbind status when setting up an NFS mount point on ExaCC X10M KVM guests?

A: ExaCC X10M environments use highly secure, stripped-down Oracle Linux images. Often, the rpcbind service is disabled or locked down by default security policies. Without rpcbind running, the KVM guest cannot map RPC program numbers to universal addresses, causing the NFS mount command to time out completely.

 Question :Can you describe a scenario where you had to mount an external NAS/NFS filer on an ExaCC X10M cluster? What was the business case, and how did you accomplish it safely without violating the co-managed cloud model?

Candidate Answer:

"In my previous project, we deployed an ExaCC X10M database cluster. We faced a constraint where storing daily database RMAN backups locally in the +RECO ASM diskgroup consumed too much high-performance Exadata storage. Additionally, security compliance prohibited routing backup data to the public OCI Object Storage.

To solve this, we mounted an on-premises Enterprise NAS filer via NFSv4 directly to the Guest VMs (DomU). Because ExaCC is a co-managed environment where Oracle controls the physical infrastructure (Dom0) and the customer has root access to the DomU, we configured the mounts strictly inside the DomU. We separated the backup traffic by utilizing the dedicated ExaCC Backup Network uplinks so it would not bottleneck the primary Client/Application network traffic."

---

2. Project Description, Requirements & Workflow

Project Description

The objective is to provision, secure, and attach an on-premises high-capacity NAS filer (NFS) to an ExaCC X10M system to act as a secondary storage tier for RMAN backups, Data Pump dumps, and flat-file staging.

Project Requirements

Infrastructure: Oracle ExaCC X10M (Multiple VM Nodes running Oracle Linux / Grid Infrastructure).
Network Isolation: NFS mount traffic must route through the Backup VLAN/Subnet, completely segregated from the Application Client Network.
Protocols: NFSv4 or NFSv3 with specific Oracle-optimized mount parameters (rw, bg, hard, rsize=1048576, wsize=1048576, proto=tcp).
Access Control: Mount point permissions mapped to oracle:oinstall with consistent UID/GID across all cluster nodes.

Architectural Workflow

Request Share: The storage team provisions the volume on the NAS engine and configures the /etc/exports file.
Access Control List (ACL): The NAS restricts access exclusively to the IP addresses of the ExaCC Backup Network interfaces (bondeth1 or specified backup sub-interfaces).
Routing Configuration: Static routing rules are declared on the ExaCC DomU nodes to ensure target NAS IPs are routed via the backup gateway.
Mounting: The system administrator mounts the directory to a local mount point (e.g., /u10/backup) across all cluster nodes.
Automation: The mount configuration is safely added to /etc/fstab with the _netdev option to prevent boot hanging if the network lags.

---

3. Test Cases (Example & Challenge)

Positive Test Case: Mount and Write Test

Objective: Verify standard read/write execution and performance from the database user.
Steps:
1. Execute mount -a as root.
2. Run su - oracle.
3. Execute touch /u10/backup/test_file.txt.
4. Run an RMAN validation backup script to the path.
Expected Result: Immediate success; correct file ownership (oracle:oinstall).

Edge Case / Challenge Test Case: Node Failover & Stale Mounts

Objective: Test how the cluster handles a sudden network drop or a hard reboot of the NAS controller.
The Challenge: In an Oracle RAC environment, if an NFS server drops offline unexpectedly, a standard Linux mount will experience stale mount blocks. Any df -h command or Oracle backup process attempting to touch that directory will hang indefinitely, potentially causing GI (Grid Infrastructure) evictions due to kernel thread delays.
Mitigation Test: Mount with soft,retrans=2,timeo=300 during testing, or use automation scripts to monitor mount health and immediately force unmount (umount -f -l) if a heartbeat ping to the NAS filer fails.

---

4. Pre-considerations & Pitfalls

Pre-considerations (Before implementation)

UID/GID Matching: Ensure the oracle user UID and oinstall GID match exactly between the NAS server's local authentication/LDAP and the ExaCC DomU environment. If they don't, files will write as nobody or result in Permission Denied.
OVM/KVM Network Mapping: Verify that the ExaCC VM Cluster Network configuration explicitly has the backup network enabled and connected to the corporate switch.
MTU Size: Ensure the MTU sizes match across the entire path. If the ExaCC backup network is set to Jumbo Frames (MTU 9000), the network switches and the NAS controller must also support and be configured for MTU 9000 to prevent packet fragmentation.

Key Challenges & Resolutions

Challenge 1: Cloud Automation Overwrites.
• Problem: Manually updating /etc/fstab or network files risks being overwritten during major ExaCC quarterly infrastructure updates orchestrated by Oracle Cloud Control Plane.
• Resolution: Always utilize the OCI Console / API under "Backup Destinations" to officially declare an external NFS endpoint when using it for automated cloud tool backups. For general file staging, use standard infrastructure-as-code orchestration (like Ansible) to re-verify custom local settings post-patching.
Challenge 2: Intermittent I/O Hangs Impacting Database Performance.
• Problem: A poorly optimized NAS will saturate the network cards, creating high wait times on the database if files are read/written concurrently.
• Resolution: Set precise mount parameters inside the Linux OS. Avoid using default settings. Implement DNFS (Direct NFS) within the Oracle Database layer rather than relies purely on OS-level kernel mounts for datafiles to bypass kernel-space context switching bottlenecks.

Here is a comprehensive guide to configuring an EMC NAS storage mount on a backup network within an Oracle Exadata Cloud@Customer (ExaCC) environment, structured as an interview preparation guide and project documentation.

Core Answer Summary

Mounting EMC NAS storage (via NFS) over the dedicated backup network on ExaCC isolates heavy backup traffic from client applications. This configuration requires modifying the fstab file on the ExaCC virtual machines (VMs) using specific mount options (rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,noac) to ensure Oracle database compatibility, data integrity, and optimal throughput.

---

Project Description, Requirement & Workflow

Project Description

Integration of a centralized EMC Isilon/PowerScale NAS storage cluster with an Exadata Cloud@Customer (ExaCC) Gen2 X8M/X9M/X10M platform. The storage is provisioned exclusively for database backups (RMAN backups, datapump exports, and archive log archiving) to prevent performance degradation on the primary client network.

Project Requirement

Target Infrastructure: ExaCC VM Clusters (Grid Infrastructure and Oracle Database 19c).
Network Isolation: All backup traffic must traverse the physical Backup Network (eth1/bond1 or equivalent vlan tagged interface), not the Client/Public network.
Storage Protocol: NFSv3 or NFSv4 over TCP.
Security: Access restricted via NFS export rules to the specific IPs of the ExaCC backup network subnet.

Architecture & Workflow

[ ExaCC VM Cluster ] --(Backup Subnet / bond1)--> [ Top-of-Rack Switch ] --> [ EMC NAS Storage Cluster ]

Trigger: RMAN or an automation script initiates a database backup.
Routing: The OS routing table directs traffic destined for the NAS backup IP through the dedicated backup network interface.
Writing: The Oracle database processes write backup pieces directly to the local mount point (/u02/backup).
Transport: Network packets travel over the dedicated high-bandwidth backup switches directly to the EMC storage controllers.

---

Interview Questions & Answers (ExaCC Focus)

Q1: What are the typical names, sources, and group parameters used for an EMC NAS mount on ExaCC?

Mount Point Name (Local Directory): /u02/backup or /mnt/emc_backup
Mountpoint Source (NAS Export Path): ://domain.com:/ifs/data/oracle_backups/db_name
Group Name: asmadmin or backupadmin (depending on organizational roles).
Group Number (GID): Commonly matching the grid infrastructure GID, typically 1001 (for oinstall) or 1020 (for asmadmin), but must precisely match the existing GID on the ExaCC VM.

Q2: Why must you use specific mount options for Oracle databases on an EMC NFS share?

Answer: Oracle requires specific mount flags to prevent data corruption and ensure high performance. A typical /etc/fstab entry looks like this:

text

://domain.com:/ifs/data/oracle_backups /u02/backup nfs rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,noac,vers=3 0 0

Use code with caution.

hard: If the NAS becomes unavailable, the OS will retry indefinitely rather than returning an error to Oracle (which could crash or corrupt the DB).
noac: Disables attribute caching. Mandatory if multiple nodes (RAC) are accessing the same mount point to ensure write visibility across all nodes.
rsize/wsize: Set to 1048576 (1MB) to match Oracle’s large sequential I/O patterns for backups.

Q3: How do you ensure the NAS traffic goes through the backup network and not the client network on ExaCC?

Answer: You must configure the mount using the NAS storage IP address that belongs to the backup VLAN/subnet. Additionally, ensure that the ExaCC routing table routing (/etc/sysconfig/network-scripts/route-bond1) routes traffic destined for the NAS subnet explicitly through the backup interface (bond1 or its tagged sub-interface).

---

Pre-considerations & Challenges

Pre-considerations

UID/GID Alignment: The UID of oracle/grid and GID of oinstall/asmadmin on the ExaCC nodes must match the permissions assigned on the EMC Isilon/PowerScale export. If Isilon uses Active Directory/LDAP, mapping rules must be configured.
MTU Size: Ensure the Backup Network and the EMC NAS ports are uniformly configured for Jumbo Frames (MTU 9000) to maximize throughput and reduce CPU overhead.
ExaCC Controls: Because ExaCC is a co-managed service, OS-level changes like updating /etc/fstab must be done carefully so they are not overwritten during OCI quarterly infrastructure patching.

Challenges & Troubleshooting

Performance Bottlenecks: If MTU sizes are mismatched (e.g., 9000 on ExaCC but 1500 on the switch), packet fragmentation occurs, causing massive performance drops.
Stale Mounts: If an EMC controller fails over, the NFS mount can hang. Utilizing the bg (background) and hard flags ensures the database waits for recovery without instantly crashing, though operations will pause.
Single Point of Failure (SPOF): Ensure the NAS source utilizes a SmartConnect zone (EMC Isilon) or LACP bonded ports so that the storage endpoint itself is redundant.

---

Test Case Example

Objective: Validate that the mount is operational, accessible by the database layer, and routing over the correct network.

Step	Action	Expected Result	Verification Command
1. Mount Verification	Execute mount command as root on all ExaCC nodes.	Mount succeeds without errors.	mount -a df -h /u02/backup
2. Permission Check	Switch to oracle user and attempt to create a file.	Read/Write operations succeed with correct user ownership.	su - oracle touch /u02/backup/test.txt
3. Routing Verification	Run a traceroute to the NAS IP.	The path resolves through the backup network gateway, not the public network.	traceroute -i bond1 <EMC_NAS_IP>
4. RMAN Test	Allocate a channel to the mount point and perform a test backup of a small tablespace.	The backup piece writes successfully without I/O errors.	RMAN> backup tablespace users format '/u02/backup/%U';