make: 1254-004 The error code from the last command is 8. make: 1254-004 The error code from the last command is 2.

After OS Upgrade (AIX from 5.2 to 5.3), the Oracle homes are relinked to bind with new OS release/version.  If you run into the following warnings, they can be safely ignored.

ld: 0711-317 ERROR: Undefined symbol: ntusini
make: 1254-004 The error code from the last command is 2.
make: 1254-004 The error code from the last command is 8.

From the Oracle metalink, you can refer below documents:

• Note 1189533.1 Relinking causes many warning on AIX
• Note 245372.1 TOC overflow Warning Can Safely Be Ignored

Script command is started on Tue Jul 9 12:02:43 CDT 2013.
oracle@gls0619{LAXPRD}/apps/oracle/product/9.2.0/bin> relink all
 /apps/oracle/product/9.2.0/bin/genclntsh
ar: Creating an archive file /apps/oracle/product/9.2.0/lib/libclntst9.a.
Created /apps/oracle/product/9.2.0/lib/libclntst9.a
ar: Creating an archive file /apps/oracle/product/9.2.0/lib32/libclntst9.a.
Created /apps/oracle/product/9.2.0/lib32/libclntst9.a
 /apps/oracle/product/9.2.0/bin/genagtsh /apps/oracle/product/9.2.0/lib/libagtsh.a 1.0
cat: 0652-050 Cannot open /apps/oracle/product/9.2.0/lib32/ldflags.
 - Linking Import utility (imp)
 rm -f /apps/oracle/product/9.2.0/rdbms/lib/imp
 ld -b64 -o /apps/oracle/product/9.2.0/rdbms/lib/imp -L/apps/oracle/product/9.2.0/rdbms/lib/ -L/apps/oracle/product/9.2.0/lib/ /apps/oracle/product/9.2.0/rdbms/lib/s0impdrv.o /apps/oracle/product/9.2.0/rdbms/lib/defopt.o -ldbtools9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lnro9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lwtc9 -lmm -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lnro9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lpls9 -ltrace9 -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 -lld -lm `cat /apps/oracle/product/9.2.0/lib/sysliblist` -lm `if [ "\`/usr/bin/uname -v\`" = "4" ]; \
 then echo "-bI:/apps/oracle/product/9.2.0/lib/pw-syscall.exp"; fi;` -lxsd9 /apps/oracle/product/9.2.0/lib/nautab.o /apps/oracle/product/9.2.0/lib/naeet.o /apps/oracle/product/9.2.0/lib/naect.o /apps/oracle/product/9.2.0/lib/naedhs.o
 mv -f /apps/oracle/product/9.2.0/bin/imp /apps/oracle/product/9.2.0/bin/impO
 mv /apps/oracle/product/9.2.0/rdbms/lib/imp /apps/oracle/product/9.2.0/bin/imp
 chmod 751 /apps/oracle/product/9.2.0/bin/imp
 - Linking Export utility (exp)
 rm -f /apps/oracle/product/9.2.0/rdbms/lib/exp
 ld -b64 -o /apps/oracle/product/9.2.0/rdbms/lib/exp -L/apps/oracle/product/9.2.0/rdbms/lib/ -L/apps/oracle/product/9.2.0/lib/ /apps/oracle/product/9.2.0/rdbms/lib/s0exudrv.o /apps/oracle/product/9.2.0/rdbms/lib/defopt.o -ldbtools9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lnro9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lwtc9 -lmm -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lnro9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lpls9 -ltrace9 -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 -lld -lm `cat /apps/oracle/product/9.2.0/lib/sysliblist` -lm `if [ "\`/usr/bin/uname -v\`" = "4" ]; \
 then echo "-bI:/apps/oracle/product/9.2.0/lib/pw-syscall.exp"; fi;` -lxsd9 /apps/oracle/product/9.2.0/lib/nautab.o /apps/oracle/product/9.2.0/lib/naeet.o /apps/oracle/product/9.2.0/lib/naect.o /apps/oracle/product/9.2.0/lib/naedhs.o
 mv -f /apps/oracle/product/9.2.0/bin/exp /apps/oracle/product/9.2.0/bin/expO
 mv /apps/oracle/product/9.2.0/rdbms/lib/exp /apps/oracle/product/9.2.0/bin/exp
 chmod 751 /apps/oracle/product/9.2.0/bin/exp

ld: 0711-317 ERROR: Undefined symbol: ntusini
ld: 0711-317 ERROR: Undefined symbol: ntpini
ld: 0711-317 ERROR: Undefined symbol: .nnflcls
ld: 0711-317 ERROR: Undefined symbol: .nnflqbf
ld: 0711-317 ERROR: Undefined symbol: .nnflinit
....
....
ld: 0711-317 ERROR: Undefined symbol: .nnflsai
ld: 0711-317 ERROR: Undefined symbol: .nnflobc
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
make: 1254-004 The error code from the last command is 8.

Stop.
make: 1254-004 The error code from the last command is 2.

Stop.
 rm -f sqlplus
 /bin/chmod 755 /apps/oracle/product/9.2.0/bin/demobld
 /bin/chmod 755 /apps/oracle/product/9.2.0/bin/demodrop
 /bin/chmod 755 /apps/oracle/product/9.2.0/bin/helpins
 /bin/chmod 755 /apps/oracle/product/9.2.0/bin/pupbld
Linking sqlplus
 rm -f sqlplus
 ld -b64 -o sqlplus -L/apps/oracle/product/9.2.0/sqlplus/lib/ -L/apps/oracle/product/9.2.0/lib/ /apps/oracle/product/9.2.0/sqlplus/lib/s0afimai.o -lsqlplus `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lnro9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lwtc9 -lmm -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lnro9 `cat /apps/oracle/product/9.2.0/lib/ldflags` -lnsslb9 -lncrypt9 -lnsgr9 -lnzjs9 -ln9 -lnl9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lpls9 -ltrace9 -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 -lclient9 -lvsn9 -lwtc9 -lcommon9 -lgeneric9 -lnls9 -lcore9 -lnls9 -lcore9 -lnls9 -lxml9 -lcore9 -lunls9 -lnls9 -lld -lm `cat /apps/oracle/product/9.2.0/lib/sysliblist` -lm -lpthreads `if [ "\`/usr/bin/uname -v\`" = "4" ]; \
 then echo "-bI:/apps/oracle/product/9.2.0/lib/pw-syscall.exp"; fi;` -lxsd9 /apps/oracle/product/9.2.0/lib/nautab.o /apps/oracle/product/9.2.0/lib/naeet.o /apps/oracle/product/9.2.0/lib/naect.o /apps/oracle/product/9.2.0/lib/naedhs.o /apps/oracle/product/9.2.0/rdbms/lib/defopt.o -lpls9 -lsql9
 /bin/chmod 755 sqlplus
 mv -f /apps/oracle/product/9.2.0/bin/sqlplus /apps/oracle/product/9.2.0/bin/sqlplusO
 mv -f /apps/oracle/product/9.2.0/sqlplus/lib/sqlplus /apps/oracle/product/9.2.0/bin/sqlplus
 /bin/chmod 751 /apps/oracle/product/9.2.0/bin/sqlplus
Target "install" is up to date.
oracle@gls0619{LAXPRD}/apps/oracle/product/9.2.0/bin> exit

ORA-00338: log 32 of thread 3 is more recent than control file

The solution is to apply patch 12770551.

We got this error ORA-338 in the Data Guard environment with ASYNC redo transport.  The connection with the remote database is closed and will not reopen until the next log switch.  Although it's not a critical situation and is not affect the database operation, applying the patch 12770551 is simple to get rid of the error.

Summary steps to apply PSU 11.2.0.3.2

It’s recommended to patch the PSU manually  on IBM Linux on z and other platform.  The opatch auto is still testing by Oracle support. and it won’t be stable (at least from what we’ve been tested)  until July 2013 PSU patchsets.  There are couple note that worth to mention: 

·         Before applying GI home, you must run rootcrs.pl to unlock the permissions.  Otherwise, the patching process will be failed with Opatch failed with error code 39.

·         When applying a GI patch to either the Grid Home or Database Home, you use "napply"

·         When applying a Database patch to either Database Home or Grid Home, you use "apply"

1)  export PATH=$ORACLE_HOME/OPatch:$PATH

2)  opatch version

     OPatch Version: 11.2.0.3.4 ß-------- Make sure you have the latest opatch version.

3) opatch prereq CheckconflictAgainstOHWithDetail -phBaseDir ./13696251

     Oracle Interim Patch Installer version 11.2.0.3.4

    Copyright (c) 2012, Oracle Corporation.  All rights reserved.

…….

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

4)  sudo -u root $ORACLE_HOME/crs/install/rootcrs.pl –unlock Don’t forget this step!

[sudo] password for oracle:

Using configuration parameter file: /u01/app/oracle/11.2.0.3/grid/crs/install/crsconfig_params

Grid Home Patch:

5)  opatch napply -oh /u01/app/oracle/11.2.0.3/grid -local /data/oracle/bkup1/software/psu11.2.0.3.2/13696251

6) opatch apply -oh /u01/app/oracle/11.2.0.3/grid -local /data/oracle/bkup1/software/psu11.2.0.3.2/13696216

Database Patch:

7) opatch napply -oh /u01/app/oracle/11.2.0.3/racdb -local /data/oracle/bkup1/software/psu11.2.0.3.2/13696251/custom/server/13696251

8) opatch apply -oh /u01/app/oracle/11.2.0.3/racdb -local /data/oracle/bkup1/software/psu11.2.0.3.2/13696216

9) /data/oracle/bkup1/software/psu11.2.0.3.2/13696251/custom/server/13696251/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/11.2.0.3/racdb

10) as root:  /u01/app/oracle/11.2.0.3/grid/rdbms/install/rootadd_rdbms.sh

11) as root :  /u01/app/oracle/11.2.0.3/grid/crs/install/rootcrs.pl -patch

OPatch failed to locate Central Inventory

Problem:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo
OPatch failed with error code 73

Solution:

+ASM2 > /aps/oracle/product/11.2.0.3/grid/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/aps/oracle/product/11.2.0.3/grid ORACLE_HOME_NAME=Ora11g_gridinfrahome1 CLUSTER_NODES=glscind01,glscind02 "INVENTORY_LOCATION=/aps/oracle/oraInventory" -invPtrLoc "/aps/oracle/product/11.2.0.3/grid/oraInst.loc" LOCAL_NODE=glscind02   
Starting Oracle Universal Installer...
   
    Checking swap space: must be greater than 500 MB.   Actual 7844 MB    Passed
    The inventory pointer is located at /aps/oracle/product/11.2.0.3/grid/oraInst.loc
    The inventory is located at /apps/oracle/oraInventory
    'AttachHome' was successful.

ORA-00312, ORA-00338

ORA-00312: online log 5 thread 3: '+DATA_CD501/oltp/onlinelog/group_5.264.799435405'
ORA-00338: log 5 of thread 3 is more recent than control file
ORA-00312: online log 5 thread 3: '+DATA_CD501/oltp/onlinelog/group_5.265.799435401'
ORA-00338: log 5 of thread 3 is more recent than control file
ORA-00312: online log 5 thread 3: '+DATA_CD501/oltp/onlinelog/group_5.264.799435405'
ORA-00338: log 5 of thread 3 is more recent than control file

Bug 12770551.
Please go through below doc for reference :
Bug 12770551 - Frequent ORA-338 during controlfile restore with ASYNC Data Guard (Doc ID 12770551.8)

Check out the patch for download

https://updates.oracle.com/Orion/PatchDetails/process_form?patch_num=12770551

Apply Database PSU 11.2.0.3.2

Download the latest OPatch version Version: 11.2.0.3.0

opatch versionOPatch Version: 11.2.0.3.0

Create OCM configuration file $ <ORACLE_HOME>/OPatch/ocm/bin/emocmrsp

sudo opatch auto /apps/oracle/software/11.2.0.3/psu -oh /apps/oracle/product/11.2.0.3/RACDB -ocmrf /tmp/ocm.rsp

Note:  /apps/oracle/software/11.2.0.3/psu is the PSU software location.

[sudo] password for oracle:
Executing /usr/bin/perl /apps/oracle/product/11.2.0.3/RACDB/OPatch/crs/patch112.pl -patchdir /apps/oracle/software/11.2.0.3 -patchn psu -oh /apps/oracle/product/11.2.0.3/RACDB -ocmrf /tmp/ocm.rsp -paramfile /apps/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /apps/oracle/product/11.2.0.3/RACDB/OPatch/crs/../../cfgtoollogs/opatchauto2012-07-15_14-26-50.log
Detected Oracle Clusterware install
Using configuration parameter file: /apps/11.2.0/grid/crs/install/crsconfig_params
patch /apps/oracle/software/11.2.0.3/psu/13696251/custom/server/13696251  apply successful for home  /apps/oracle/product/11.2.0.3/RACDB
patch /apps/oracle/software/11.2.0.3/psu/13696216  apply successful for home  /apps/oracle/product/11.2.0.3/RACDB

Apply 11.2.0.3.1 Grid Infrastructure Patch Set Update (GI PSU)

+ASM1 > $ORACLE_HOME/OPatch/ocm/bin/emocmrsp

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name: ashley.nguyen@dbalab.com

Provide your My Oracle Support password to receive security updates via your My Oracle Support account.
Password (optional):          
The OCM configuration response file (ocm.rsp) was successfully created.

mv ocm.rsp /tmp

sudo -u root opatch auto /apps/oracle/software/psu.11.2.0.3.1 -oh /apps/11.2.0/grid -ocmrf /tmp/ocm.rsp

<===Failed ==>

If the opatch auto failed, you may need to re-run the command again.  See the example below

Re-run it and enter yes if this is not a shared home.

sudo -u root opatch auto /apps/oracle/software/psu.11.2.0.3.1 -oh /apps/11.2.0/grid -ocmrf /tmp/ocm.rsp

Executing /usr/bin/perl /apps/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /apps/oracle/software -patchn psu.11.2.0.3.1 -oh /apps/11.2.0/grid -ocmrf /tmp/ocm.rsp -paramfile /apps/11.2.0/grid/crs/install/crsconfig_params

opatch auto log file location is /apps/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-06-04_17-41-29.log

Detected Oracle Clusterware install

Using configuration parameter file: /apps/11.2.0/grid/crs/install/crsconfig_params

Unable to determine if /apps/11.2.0/grid is shared oracle home

Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

Successfully unlock /apps/11.2.0/grid

patch /apps/oracle/software/psu.11.2.0.3.1/13348650  apply successful for home  /apps/11.2.0/grid

patch /apps/oracle/software/psu.11.2.0.3.1/13343438  apply successful for home  /apps/11.2.0/grid

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9312: Existing ADVM/ACFS installation detected.

ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9154: Loading 'oracleoks.ko' driver.

ACFS-9154: Loading 'oracleadvm.ko' driver.

ACFS-9154: Loading 'oracleacfs.ko' driver.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.

ACFS-9156: Detecting control device '/dev/ofsctl'.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4123: Oracle High Availability Services has been started.

ORA-00600: internal error code, arguments: [ktbesc_plugged],

Recently we migrated RAC databases from 10.2.0.4 AIX  to 11.2.0.3 PSU1 on Redhat using transport tablespaces.  As part of our tests, we setup flashback to a gurantee restore point.  After several cycle of testing, we flashed back the database to the original save point, and it completed sucessfully.  However, when users inserted or deleted on tables data belonged to the tablespace that we transported,  application showed error ORA-00600: internal error code, arguments: [ktbesc_plugged], [], [], [], [], [], [], [], [], [], [],

It was confirmed that we hit bug 1291956 .  There are 3 solutions for this error.

1)  Temporary workaround of moving the problem table to another tablespace.

alter table <table name> move tablespace <tablespace name>;
2)  There is an additional workaround of setting "_fastpin_enable"=0 which is discussed in Document 12919564.8.

3)  Long term resolution would be to apply one-off Patch:12919564.
We choose #3 and apply this one-off Patch.  This patch overlays the PSU patch 13343438 .  After applying the patch to the database home, it fixed insert/delete issues.

opatch lsinv | grep ^Patch
Patch  12919564     : applied on Mon Apr 16 13:32:53 CDT 2012
Patch  13343438     : applied on Wed Feb 22 15:56:35 CST 2012
Patch Description:  "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch  13348650     : applied on Wed Feb 22 15:56:14 CST 2012
Patch Description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.1 (13348650)"

opatch lsinv -bugs_fixed | grep -i 12919564
12919564   12919564  Mon Apr 16 13:32:53 CDT 2012   ENCOUNTERED ORA-600 [KTBESC_PLUGGED] WHILE

OS upgrade and oracleasm packages

After upgrade OS, for example RHEL 5.6 to 5.7, you will need to download and install the oracleasm packages that match with your new kernel.  There is no need to relink oracle or grid binaries after installing or upgrading ASMlib packages. 


You need to run the "uname -r" to determine your kernel version  and the corresponding package has the name oracleasm-<kernel_version>.   For more information, please follow this reference link..
  
http://www.oracle.com/technetwork/server-storage/linux/downloads/rhel5-084877.html

11.2.0.3 Grid Infrastructure and RDBMS Patch Set Notes and Upgrade

11.2.0.3 is a full installation of the Oracle software.  This means that yo don't need to install 11.2.0.1 before installing 11.2.0.3.  The new installation consists of the most recent patch set (not install a base release then upgrade).  The direct upgrades from previous release or in-place upgrade are supported, but not recommended.  Out-out-place patch set upgrades recommended with new separate oracle home.

Below are all the details you need for the 11.2.0.3 upgrade:

1. The PSU should be applied to 11.2.0.2 before upgrading to 11.2.0.3 is PSU1
11.2.0.2 GI PSU1(which includes database PSU1) - PATCH 12539000

Reference Doc:

Things to Consider Before Upgrading to 11.2.0.3 Grid Infrastructure/ASM (Doc ID 1363369.1)

2. The link to download the above PSU.

https://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=12539000

3. The link to download 11.2.0.3GI

https://updates.oracle.com/download/10404530.html

Reference Doc:
11.2.0.3 Patchset Location For Grid Infrastructure And RDBMS. (Doc ID 1362393.1)

4. The link to download the latest 11.2.0.3GI PSU
Patch 13348650 - 11.2.0.3.1 Grid Infrastructure Patch Set Update (GI PSU)

https://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=13348650

Reference Doc:
11.2.0.3 Patch Set - Availability and Known Issues (Doc ID 1348336.1)

Installation Types and Associated Zip Files

Installation Type	Zip File
Oracle Database (includes Oracle Database and Oracle RAC) Note: you must download both zip files to install Oracle Database.	p10404530_112030_platform_1of7.zip p10404530_112030_platform_2of7.zip
Oracle Grid Infrastructure (includes Oracle ASM, Oracle Clusterware, and Oracle Restart)	p10404530_112030_platform_3of7.zip
Oracle Database Client	p10404530_112030_platform_4of7.zip
Oracle Gateways	p10404530_112030_platform_5of7.zip
Oracle Examples	p10404530_112030_platform_6of7.zip
Deinstall	p10404530_112030_platform_7of7.zip