NODE EVICTION OVERVIEW and Troubleshooting steps
- The Process: If a node misses interconnect (network) or Voting Disk (disk) heartbeats, surviving nodes assume the non-responsive node is dead or hung. The cluster initiates fencing to isolate the node, forcing it to reboot to prevent database corruption.
- 19c FCNR Enhancement: Prior to 19c, instance recovery was a serial process driven solely by the SMON process on a surviving node. In 19c, Oracle distributes the redo logs of the failed instance to multiple surviving nodes, utilizing multiple recovery slaves to apply redo in parallel. This dramatically reduces Recovery Time Objective (RTO) and minimizes SLA breaches.
- Interconnect Issues: Network packet drops, high latency, incorrect MTU configurations, or duplex mismatches.
- Storage/Voting Disk Delays: High I/O latency or storage path flapping causing the CSS daemon to lose contact with the voting disks.
- System/OS Resource Starvation: Extreme CPU load or exhausting available RAM/Swap, causing the CSSD agent to terminate and trigger an eviction.
- Time Synchronization: Drifts in NTP or Chrony that push time differences beyond allowed tolerances between cluster nodes.
The Oracle Clusterware is designed to perform a node eviction by removing one or more nodes from the cluster if some critical problem is detected. A critical problem could be a node not responding via a network heartbeat, a node not responding via a disk heartbeat, a hung or severely degraded machine, or a hung ocssd.bin process. The purpose of this node eviction is to maintain the overall health of the cluster by removing bad members.
Starting in 11.2.0.2 RAC or above (or if you are on Exadata), a node eviction may not actually reboot the machine. This is called a rebootless restart. In this case we restart most of the clusterware stack to see if that fixes the unhealthy node.
Oracle Clusterware evicts the node when following condition occur:
- Node is not pinging via the network hearbeat
- Node is not pinging the Voting Disk
- Node is hung or busy and is unable to perform the above two tasks
What is the use of CSS Heartbeat Mechanism in Oracle RAC
The CSS of the Oracle Clusterware maintains two heartbeat mechanisms
1. The disk heartbeat to the voting device and
2. The network heartbeat across the interconnect (This establish and confirm valid node membership in the cluster).
Both of these heartbeat mechanisms have an associated timeout value. The disk heartbeat has an internal i/o timeout interval (DTO Disk TimeOut), in seconds, where an i/o to the voting disk must complete. The misscount parameter (MC), as stated above, is the maximum time, in seconds, that a network heartbeat can be missed. The disk heartbeat i/o timeout interval is directly related to the misscount parameter setting. The Disk TimeOut(DTO) = Miscount(MC) - 15 secconds (some versions are different).
1.0 - PROCESS ROLES FOR REBOOTS
OCSSD (aka CSS daemon) - This process is spawned by the cssdagent process. It runs in both vendor clusterware and non-vendor clusterware environments. OCSSD's primary job is internode health monitoring and RDBMS instance endpoint discovery. The health monitoring includes a network heartbeat and a disk heartbeat (to the voting files). OCSSD can also evict a node after escalation of a member kill from a client (such as a database LMON process). This is a multi-threaded process that runs at an elevated priority and runs as the Oracle user.
Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent --> ocssd --> ocssd.bin
CSSDAGENT - This process is spawned by OHASD and is responsible for spawning the OCSSD process, monitoring for node hangs (via oprocd functionality), and monitoring to the OCSSD process for hangs (via oclsomon functionality), and monitoring vendor clusterware (via vmon functionality). This is a multi-threaded process that runs at an elevated priority and runs as the root user.
Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent
CSSDMONITOR - This proccess also monitors for node hangs (via oprocd functionality), monitors the OCSSD process for hangs (via oclsomon functionality), and monitors vendor clusterware (via vmon functionality). This is a multi-threaded process that runs at an elevated priority and runs as the root user.
Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdmonitor
2.0 - DETERMINING WHICH PROCESS IS RESPONSIBLE FOR A REBOOT
Important files to review:
Clusterware alert log in
The cssdagent log(s)
The cssdmonitor log(s)
The ocssd log(s)
The lastgasp log(s) in /etc/oracle/lastgasp or /var/opt/oracle/lastgasp
CHM or OS Watcher data
'opatch lsinventory -detail' output for the GRID home
*Messages files:
* Messages file locations:
Linux: /var/log/messages
Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
IBM: /bin/errpt -a > messages.out
Document 1513912.1 - TFA Collector - Tool for Enhanced Diagnostic Gathering
11.2 Clusterware evictions should, in most cases, have some kind of meaningful error in the clusterware alert log. This can be used to determine which process is responsible for the reboot. Example message from a clusterware alert log:
[ohasd(11243)]CRS-8011:reboot advisory message from host: sta00129, component: cssagent, with timestamp: L-2009-05-05-10:03:25.340
[ohasd(11243)]CRS-8013:reboot advisory message text: Rebooting after limit 28500 exceeded; disk timeout 27630, network timeout 28500, last heartbeat from CSSD at epoch seconds 1241543005.340, 4294967295 milliseconds ago based on invariant clock value of 93235653
This particular eviction happened when we had hit the network timeout. CSSD exited and the cssdagent took action to evict. The cssdagent knows the information in the error message from local heartbeats made from CSSD.
If no message is in the evicted node's clusterware alert log, check the lastgasp logs on the local node and/or the clusterware alert logs of other nodes.
3.0 - TROUBLESHOOTING OCSSD EVICTIONS
If you have encountered an OCSSD eviction review common causes in section 3.1 below.
3.1 - COMMON CAUSES OF OCSSD EVICTIONS
Network failure or latency between nodes. It would take 30 consecutive missed checkins (by default - determined by the CSS misscount) to cause a node eviction.
Problems writing to or reading from the CSS voting disk. If the node cannot perform a disk heartbeat to the majority of its voting files, then the node will be evicted.
A member kill escalation. For example, database LMON process may request CSS to remove an instance from the cluster via the instance eviction mechanism. If this times out it could escalate to a node kill.
An unexpected failure or hang of the OCSSD process, this can be caused by any of the above issues or something else.
An Oracle bug.
3.2 - FILES TO REVIEW AND GATHER FOR OCSSD EVICTIONS
All files from section 2.0 from all cluster nodes. More data may be required.
Example of an eviction due to loss of voting disk:
CSS log:
2012-03-27 22:05:48.693: [ CSSD][1100548416](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 3 configured voting disks available, need 2
2012-03-27 22:05:48.693: [ CSSD][1100548416]###################################
2012-03-27 22:05:48.693: [ CSSD][1100548416]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
OS messages:
Mar 27 22:03:58 choldbr132p kernel: Error:Mpx:All paths to Symm 000190104720 vol 0c71 are dead.
Mar 27 22:03:58 choldbr132p kernel: Error:Mpx:Symm 000190104720 vol 0c71 is dead.
Mar 27 22:03:58 choldbr132p kernel: Buffer I/O error on device sdbig, logical block 0
...
4.0 - TROUBLESHOOTING CSSDAGENT OR CSSDMONITOR EVICTIONS
If you have encountered a CSSDAGENT or CSSDMONITOR eviction review common causes in section 4.1 below.
4.1 - COMMON CAUSES OF CSSDAGENT OR CSSDMONITOR EVICTIONS
An OS scheduler problem. For example, if the OS is getting locked up in a driver or hardware or there is excessive amounts of load on the machine (at or near 100% cpu utilization), thus preventing the scheduler from behaving reasonably.
A thread(s) within the CSS daemon hung.
An Oracle bug.
4.2 - FILES TO REVIEW AND GATHER FOR CSSDAGENT OR CSSDMONITOR EVICTIONS
All files from section 2.0 from all cluster nodes. More data may be required.
Cluster health check
CSS Miss-count
The Cluster Synchronization Service (CSS) Miscount is the maximum time, in seconds, that a network heartbeat can be missed before a cluster reconfiguration to evict the node
How to get the CSS Misscount value
$ crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
oracrs@node1~]$
[oracrs@node1~]$ crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
[oracrs@node1~]$
How to set the CSS Misscount value
Shut down CRS on all nodes but one as root run crsctl on that remaining node
# crsctl stop crs
#crsctl set css misscount 60
Configuration parameter misscount is now set to 60
#
2. CSS disktimeout (Default 200)
The maximum amount of time (in seconds)allowed for a voting file I/O to complete, if this time is exceeded the voting disk will be marked as offline.Note that this is also the amount of time that will be required for initial cluster formation, i.e. when no nodes have previously been up and in a cluster.
How to get the CSS disktimeout value
$ crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
[oracrs@node1l ~]$ crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
[oracrs@node1l ~]$
How to set the CSS disktimeout value
Shut down CRS on all nodes but one as root run crsctl on that remaining node
# crsctl stop crs
#crsctl set css disktimeout 300
Configuration parameter disktimeout is now set to300
#
3. CSS reboottime(Default 3 seconds)
The amount of time allowed for a node to complete a reboot after the CSS daemon has been evicted
How to get the CSS reboottime value
$crsctl get css reboottime
CRS-4678: Successful get reboottime 3 for Cluster Synchronization Services.
-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
[oracrs@node1 ~]$ crsctl get css reboottime
CRS-4678: Successful get reboottime 3 for Cluster Synchronization Services.
[oracrs@node1 ~]$
How to set the CSS reboottime value
Shut down CRS on all nodes but one as root run crsctl on that remaining node
# crsctl stop crs
#crsctl set css reboottime 10
Configuration parameter reboottime is now set to 10
#
CSS) to prevent database split-brain and subsequent data corruption. - Check Node Status: Run
crsctl check cluster -allandolsnodes -nfrom a surviving node. - Determine Eviction Type: Execute
last rebootoruptimeon the evicted host. If the OS did not restart but clusterware bounced, it is a rebootless eviction. - Locate Reconfiguration Window: Look for
CRS-1601: CSSD Reconfiguration completein the surviving nodes' clusterware alert logs to pinpoint the exact timestamp of the eviction.
[ Eviction Event Identified ]
│
┌────────────────┴────────────────┐
▼ ▼
[ Network Interconnect ] [ Storage/Voting Disk ]
- ocssd.log (misscount) - ocssd.log (disk timeout)
- /var/log/messages (dropped) - alert_+ASM.log (I/O hung)
- oswprvtnet / oswnetstat - /var/log/messages (SCSI/SAN)
- Clusterware Alert Log:
$GRID_HOME/log/<node>/alert<node>.log(Summary of CRS daemon failures). - OCSSD Log (Primary Source):
$GRID_HOME/log/<node>/cssd/ocssd.log.- Search strings:
clssnmPollingThread,misscount,Rebooting after cluster integrity check, orlost communication.
- Search strings:
- CSSDAGENT / CSSDMONITOR Logs:
$GRID_HOME/log/<node>/agent/ohasd/oracssdagent/(Tracks OS scheduler delays and node hangs). [ - System Messages:
/var/log/messagesor/var/log/syslog(Look for Out-Of-MemoryOOM-killererrors orhung_task_timeout). ]
CSS miscount threshold (typically 30 seconds). - Log Symptom:
ocssd.logshowsclssnmPollingThread: node <num> is status 0, missed heartbeats.... - Verification: Run
oifcfg getifto ensure private interfaces are assigned correctly. Usenetstat -ito verify if packet drops or errors are escalating over the interconnect interface.
disktimeout window (typically 200 seconds). - Log Symptom:
ocssd.logdisplaysclssnmWriteDiskHeartbeat: Failed to write diskorlost access to voting disk. - Verification: Run
crsctl query css votediskand check your ASM alert log (alert_+ASM.log) for disk read/write timeouts or multipath dropouts.
ocssd real-time process to get starved of CPU cycles, failing to respond to cssdmonitor. - Log Symptom:
cssdmonitor.logentries showWatchdog OS scheduler margin exceededfollowed by a forced system crash. - Verification: Review OSWatcher or
CHM(Cluster Health Monitor) metrics captured during the event window to check for heavy swapping or extreme load averages.
- Scenario: A network switch firmware update causes a transient 45-second dropout on the private VLAN.
- Log Signature (
ocssd.logon Node 2):text2026-06-24 08:12:10.104 [CSSD(1234)]CRS-1605: CSSD detected a network split. 2026-06-24 08:12:40.104 [CSSD(1234)]CRS-1612: Network heartbeat to node 1 (racnode1) missed for 50% of subnet timeout. 2026-06-24 08:12:55.104 [CSSD(1234)]CRS-1610: Node racnode2 (2) will be evicted in 5 seconds.Resolution: Coordinate with network engineers to fix VLAN tagging and implement Redundant Interconnect / HAIP to provide active-active failover paths.
- Scenario: A fiber channel controller fails on a storage array host, dropping communication to the SAN LUNs hosting the Voting Disks.
- Log Signature (
/var/log/messages):textJun 24 09:01:15 racnode1 kernel: multipath: dm-2: multipath path failure on sdg Jun 24 09:01:45 racnode1 ocssd[5543]: [CSSD]clssnmWriteDiskHeartbeat: voting disk access lost (1 out of 3 available)Resolution: Restore storage fabrics and check multipathing configurations (multipath -ll). Ensure ASM disk timeout definitions are correctly scaled to tolerate fiber-channel failovers.
CSS handles a split-brain condition properly and evicts the correct node without corrupting data. - Identify the active interconnect interface by executing:bash
oifcfg getifDrop incoming traffic on the private network interface (e.g.,eth1) of Node 2 usingiptables: - bash
iptables -A INPUT -i eth1 -j DROPExpected Result: Within 30 seconds (the defaultmiscountthreshold), Node 2 will lose network heartbeats with the rest of the cluster. It will perform a self-eviction or hard reset to protect shared storage integrity. - Remove the rule on Node 2 during reboot or startup to allow it to rejoin:bash
iptables -F
cssdmonitor cleanly fencing-off an unresponsive clusterware stack when an OS-level stall occurs. - Find the process ID (PID) of the active Cluster Synchronization Services Daemon (
ocssd.bin) on a target node:bashps -ef | grep ocssd.binSend aSIGSTOPsignal to pause the process, simulating a heavy OS kernel hang: - bash
kill -STOP <OCSSD_PID>Expected Result: The process ceases checking in with its companion threads. After the scheduler margin expires,cssdagentorcssdmonitorwill immediately step in and forcefully restart the clusterware layer or issue an immediate kernel panic/reset to fence the node.
tfactl diagcollection --from "2026-06-24 08:00:00" --to "2026-06-24 09:30:00"- Network Interconnect failure: Intermittent or total loss of communication over the private network (missed network heartbeat).
- Voting Disk I/O loss: Node cannot read/write to the voting disks over the SAN (missed disk heartbeat).
- CPU/Memory Starvation: OS processes like
ocssd.bincannot get CPU, or swapping/memory exhaustion forces a hangcheck/watchdog reboot.
- Verify Cluster State:
Executecrsctl check cluster -allandcrsctl stat res -tto evaluate the current status of cluster nodes and resources. - Check CSSD logs:
Open$GRID_HOME/log/<node_name>/cssd/ocssd.log(useadrcito parse). Search formisscountoreviction.
Example:grep -i "misscount" ocssd.log - Check ASM Alert Logs:
Navigate to$GRID_HOME/log/diag/asm/+asm/+ASM<id>/trace/alert_+ASM<id>.logto check for disk group dismounts or missing path issues that might have triggered an eviction. - Inspect OS logs:
Usegrep -i "error" /var/log/messagesordmesgaround the time of the event to catch network link down (ens5f0 NIC Link is Down) or kernel hangs (hung_task).
