Wednesday 21 February 2018

issue while applying patch on 12c clusterware


While applying patch on clusterware ,Clusterware patch had applied successfully on GRID HOME  but during post operation .it failed due to with below

2018/02/21 07:08:43 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'


Details Log

node1[/u02/oracle/12.1.0.2/grid/OPatch]# ./opatchauto apply /staging/clusterware_12cpatch/26635815  -oh /u02/oracle/12.1.0.2/grid

OPatchauto session is initiated at Wed Feb 21 06:40:59 2018
System initialization log file is /u02/oracle/12.1.0.2/grid/cfgtoollogs/opatchautodb/systemconfig2018-02-21_06-41-04AM.log.
Session log file is /u02/oracle/12.1.0.2/grid/cfgtoollogs/opatchauto/opatchauto2018-02-21_06-41-43AM.log
The id for this session is 3D7C

Executing OPatch prereq operations to verify patch applicability on home /u02/oracle/12.1.0.2/grid
Patch applicablity verified successfully on home /u02/oracle/12.1.0.2/grid
Verifying patch inventory on home /u02/oracle/12.1.0.2/grid
Patch inventory verified successfully on home /u02/oracle/12.1.0.2/grid
Bringing down CRS service on home /u02/oracle/12.1.0.2/grid
Prepatch operation log file location: /u02/oracle/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_node1_2018-02-21_06-47-45AM.log
CRS service brought down successfully on home /u02/oracle/12.1.0.2/grid
Start applying binary patch on home /u02/oracle/12.1.0.2/grid
Successfully executed command: /usr/sbin/slibclean

Binary patch applied successfully on home /u02/oracle/12.1.0.2/grid
Starting CRS service on home /u02/oracle/12.1.0.2/grid
Postpatch operation log file location: /u02/oracle/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_node1_2018-02-21_07-06-54AM.log
Failed to start CRS service on home /u02/oracle/12.1.0.2/grid

Execution of [GIStartupAction] patch action failed, check log for more details. Failures:
Patch Target : node1->/u02/oracle/12.1.0.2/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u02/oracle/12.1.0.2/grid, host: node1.
Command failed:  /u02/oracle/12.1.0.2/grid/perl/bin/perl -I/u02/oracle/12.1.0.2/grid/perl/lib -I/u02/oracle/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_node1/patchwork/crs/install /u02/oracle/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_node1/patchwork/crs/install/rootcrs.pl -postpatch
Command failure output:
Using configuration parameter file: /u02/oracle/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_node1/patchwork/crs/install/crsconfig_params
2018/02/21 07:07:12 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2018/02/21 07:08:39 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2018/02/21 07:08:43 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

2018/02/21 07:09:31 CLSRSC-400: A system reboot is required to continue installing.


After fixing the cause of failure Run opatchauto resume with session id "3D7C"

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Wed Feb 21 07:09:36 2018
Time taken to complete the session 28 minutes, 38 seconds

 opatchauto failed with error code 42
You have mail in /usr/spool/mail/root

node1[/u02/oracle/12.1.0.2/grid/OPatch]#
=====================================================
As per patch log recommendation, with help of AIX Team, server rebooted, and we executed opatchauto resume command and manually tried to start clusterware but failed due to mismatch of clusterware release patch level and software patch level


Detail log for resume command

node1[/u02/oracle/12.1.0.2/grid/OPatch]# ./opatchauto resume

OPatchauto session is initiated at Wed Feb 21 07:50:23 2018
Session log file is /u02/oracle/12.1.0.2/grid/cfgtoollogs/opatchauto/opatchauto2018-02-21_07-50-27AM.log
Resuming existing session with id 3D7C
Starting CRS service on home /u02/oracle/12.1.0.2/grid
Postpatch operation log file location: /u02/oracle/12.1.0.2/grid/cfgtoollogs/crsconfig/crspatch_node1_2018-02-21_07-50-39AM.log
Failed to start CRS service on home /u02/oracle/12.1.0.2/grid
Execution of [GIStartupAction] patch action failed, check log for more details. Failures:
Patch Target : node1->/u02/oracle/12.1.0.2/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u02/oracle/12.1.0.2/grid, host: node1.
Command failed:  /u02/oracle/12.1.0.2/grid/perl/bin/perl -I/u02/oracle/12.1.0.2/grid/perl/lib -I/u02/oracle/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_node1/patchwork/crs/install /u02/oracle/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_node1/patchwork/crs/install/rootcrs.pl -postpatch
Command failure output:
Using configuration parameter file: /u02/oracle/12.1.0.2/grid/OPatch/auto/dbtmp/bootstrap_node1/patchwork/crs/install/crsconfig_params
2018/02/21 07:50:53 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2018/02/21 07:50:54 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2018/02/21 07:51:33 CLSRSC-400: A system reboot is required to continue installing.


After fixing the cause of failure Run opatchauto resume with session id "3D7C"

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Wed Feb 21 07:51:38 2018
Time taken to complete the session 1 minute, 19 seconds

 opatchauto failed with error code 42
node1[/u02/oracle/12.1.0.2/grid/OPatch]#
/u02/oracle/12.1.0.2/grid/crs/install/rootcrs.sh -patch


buxton[/u02/oracle/12.1.0.2/grid/OPatch]# /u02/oracle/12.1.0.2/grid/crs/install/rootcrs.sh -patch
Using configuration parameter file: /u02/oracle/12.1.0.2/grid/crs/install/crsconfig_params
2018/02/21 07:15:42 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2018/02/21 07:15:42 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2018/02/21 07:15:46 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

2018/02/21 07:16:28 CLSRSC-400: A system reboot is required to continue installing.


Try to start manually but encountered below error


node1[/u02/oracle/12.1.0.2/grid/bin]# ./crsctl start crs
CRS-6706: Oracle Clusterware Release patch level ('3454717665') does not match Software patch level ('1505651481'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
buxton[/u02/oracle/12.1.0.2/grid/bin]#


Finally as work around we executed clscfg command and started clusterware manually successfully

node1[/u02/oracle/12.1.0.2/grid/bin]# ./clscfg -localpatch
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
node1[/u02/oracle/12.1.0.2/grid/bin]# id

Clusterware Status

 node1[/u02/oracle/12.1.0.2/grid/bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
You have mail in /usr/spool/mail/root
node1 [/u02/oracle/12.1.0.2/grid/bin]#


Eg
 as root from ../grid/bin

./clscfg -localpatch

./rootcrs.pl -lock

5) Kill the following process :-

ps -ef|grep ora.gpnpd

kill -9 process id

./crsctl start crs





In this Document


Description

Occurrence

Symptoms

Workaround

Patches

History

References


Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
IBM AIX on POWER Systems (64-bit)

Description

Application of 12.1.0.2 Oct PSU fails with the following error while installing ACFS driver using "acfsroot install" command in post patch step on an AIX system with large memory (128GB or more)
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9314: Removing previous ADVM/ACFS installation.
This may take several minutes. Please wait ...
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
0518-307 odmdelete: 1 objects deleted.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9359: The AIX ODM entry for 'acfsctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmctl' was successfully added.
ACFS-9359: The AIX ODM entry for 'advmvol' was successfully added.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleacfs.ext' driver.
(cfgacfsctl): ACFS-11022: failed to configure device ofsctl, errno 12 (Not enough space) <<< !! HERE
(cfgacfsctl): ACFS-11041: trying to clean up after encountering an error
ACFS-9109: oracleacfs.ext driver failed to load.
ACFS-9428: Failed to load ADVM/ACFS drivers. A system reboot is
recommended.
ACFS-9310: ADVM/ACFS installation failed.

truss of acfsroot install shows:
execve("/usr/lib/methods/cfgacfsctl.bin", 0x20093C68, 0x20093CC8) argc: 3
0x0FFFFFFFFFFFE500) = 0
11075664: 108069057: 1733.1749: thread_setmystate(0x0FFFFFFFFFFFDFA0,
0x0000000000000000) = 0
11075664: 108069057: 1733.1753: shmdt(0x0A00000000000000) = 0
11075664: 108069057: 1733.1756: close(6) = 0
11075664: 108069057: 1733.1760: sysconfig(6, 0x0FFFFFFFFFFFEBE0, 40) Err#12 ENOMEM <<--------
11075664: 108069057: 1733.3241: lseek(4, 512, 0) = 512
11075664: 108069057: 1733.3244: kread(4, "1D e + g\0\0\0\0\0\0\0\0"..,
512) = 512

Occurrence

12.1.0.2 Oct17 ACFS PSU (26392164) is applied on AIX on large memory system (128Gb or more)

Symptoms

Application of 12.1.0.2 Oct PSU fails with the below error while installing ACFS driver using "acfsroot install" command in post patch step on an AIX system with large memory (128GB or more)

Workaround

Upon patch failure, follow these steps:
As root OS user
1. Make sure all drivers are unloaded by running "acfsload stop"
cd $GI_HOME/bin
./acfsload stop
2. Create /etc/acfstunables file.
3. Edit the acfstunables file, add the following text
"OfsVmmBufCount=32768"
4. Run "acfsroot install" to install the driver.
cd $GI_HOME/bin
./acfsroot install
5. Ensure the OfsVmmBufCount tunable parameter value
"acfsutil tune OfsVmmBufCount"
6.  Re-run the postpatch patching step on the same node

If OCT ACFS PSU will be applied in the future, add below entries before applying the PSU on all nodes in a cluster:
As root OS user
1. Create /etc/acfstunables file.
2. Edit the file, add the following text "OfsVmmBufCount=32768"
  
Then apply the PSU.


Patches

This behavior is caused by code defect Bug:27105321 - ACFS-11022: FAILED TO CONFIGURE DEVICE OFSCTL, ERRNO 12 WITH OCT PSU, currently under Development Review

History

14-November-2017: Initial Document Creation
16-November-2017:  Document Published
16-November-2017:  Document modified to ALERT format
20-November-2017:  re-published as ALERT
===================


some other issue and oracle official doc to resolve issue

issue 1 


Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

Symptoms

The following opatch command fails:

$ export ORACLE_HOME=/u02/grid/11.2.0
$ $ORACLE_HOME/OPatch/opatch prereq CheckApplicable -ph /stage/patches/13653086 -oh /u02/grid/11.2.0
..

Invoking prereq "checkapplicable"

ZOP-46: The patch(es) are not applicable on the Oracle Home because some patch actions are not applicable. All required components, however, are installed.

Prereq "checkApplicable" for patch 13653086 failed.
..
Copy Action: Desctination File "/u02/grid/11.2.0/bin/crsd.bin" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'crsd.bin' to '/u02/grid/11.2.0/bin/crsd.bin'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/gnsd" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'gnsd' to '/u02/grid/11.2.0/bin/gnsd'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/gnsd.bin" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'gnsd.bin' to '/u02/grid/11.2.0/bin/gnsd.bin'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/oclskd.bin" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'oclskd.bin' to '/u02/grid/11.2.0/bin/oclskd.bin'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/octssd.bin" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'octssd.bin' to '/u02/grid/11.2.0/bin/octssd.bin'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/ohasd.bin" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'ohasd.bin' to '/u02/grid/11.2.0/bin/ohasd.bin'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/ologgerd" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'ologgerd' to '/u02/grid/11.2.0/bin/ologgerd'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/orarootagent.bin" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'orarootagent.bin' to '/u02/grid/11.2.0/bin/orarootagent.bin'
Copy Action: Desctination File "/u02/grid/11.2.0/bin/osysmond.bin" is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'osysmond.bin' to '/u02/grid/11.2.0/bin/osysmond.bin'


Or opatch auto command fails with:
8. /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /media/software/server/RAC_Servers/Patches/SYSTEST/17735354/17540582 -oh /u01/app/11.2.0/grid
2014-02-06 19:25:49: ERROR: Prereq checkApplicable failed. Refer log file for more details.


Cause

Refer to Solution section.

Solution

A. Expected behaviour if GRID_HOME has not been unlocked


If GI home has not been unlocked with "rootcrs.pl -unlock", checkapplicable will fail as many files are still owned by root user, this is expected behaviour. The solution is to use "opatch auto" or follow the patch readme step-by-step so the GI home gets unlocked first.


B. Bug 13575478

Bug 13575478 happens when running the following command while GI stack is down:
# $GRID_HOME/OPatch/opatch auto <unzipped patch location> -och <GRID_HOME>

The workaround is to keep GI up and running and execute command:
# $GRID_HOME/OPatch/opatch auto <unzipped patch location> -oh <GRID_HOME>

C. The patch is stored in a shared NFS location and there is a permission issue accessing the patch

The workaround is to move the patch to a local filesystem and unzip the patch with grid user, retry opatch auto

D. The patch is not unzipped as grid user, often it is unzipped as root user

ls -l <PATCH_UNZIPPED_PATH> will show the files are owned by root user.
The solution is to unzip the patch as grid user into an empty directory outside of GRID_HOME, then retry the patch apply.

E. The patch is unzipped inside GRID_HOME

The solution is to unzip the patch as grid user into an empty directory outside of GRID_HOME, then retry the patch apply.



on

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)