Bug 12876128: ORA-600 KCLCHKBLK_4/2662 ERRORS IN THE LOG AFTER DOING A FLASHBACK.


Currently Oracle Development is working on the patch fix.  This issue occurs on 11.2.0.2 Linux when multiple restore points created while the database is opened.

DIAGNOSTIC ANALYSIS
  ===================
  Errors ORA-600 [kclchkblk_4] and ORA-600 [2662] After Recovery of
  Database (Doc ID 275902.1)
 
  BUG: 3517013  10.1.0.2 RDBMS 10.1.0.2 BUFFER CACHE PRODID-5 PORTID-23
  ORA-600
  Abstract: GSIST: OPEN DB RESETLOG AFTER FLASHBACK DB FAILS ORA-600
  [KCLCHKBLK_4], [1904]

Initially, Oracle Support suggested us to drop and re-create the temp since the symptom seems to be similar to the below description, but it doesn't work for us.  Oracle Developer is working on the fix patch..


Internal Problem Description:
  Flashback will move back the recent scn, but it will not touch temporary
  files, so it is possible to read a temporary file, and find a block
  scn which is greater than the recent scn.

ipcs, pmap, lsof

There are several useful Linux performance monitoring commands like uptime, last, ps, top, pgrep -flu oracle, free, etc. the ipcs, pmap, lsof I use them often to get the detail of memory utilization.

The ipcs command with the -m argument can be used to display the configured shared memory segments on the system.
ORATOOLS1 - oracle: ipcs -m
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status     
0x00000000 3571714    root      644        80         2                      
0x00000000 3604484    root      644        16384      2                      
0x00000000 3637253    root      644        280        2                      
0xf14e8d68 4096007    oracle    660        4096       0                      
0x148fbb84 4227085    oracle    660        4096       0     

The command ipcmr with the -M argument can be used to manually delete shared memory segments by a user.  However, you should use pmap and lsof beforehand to identify the processes using the shared memory segment
The pmap Command - details the memory mapped by that particial process including the total memory utilization. 
pmap -x

ORATOOLS1 - oracle: pmap -x 44234423:   ora_pmon_ORATOOLS1
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000  180232   12692       0 r-x--  oracle
000000000b602000    1820     280      72 rwx--  oracle
000000000b7c9000     300     204     196 rwx--    [ anon ]
000000000f707000     576     512     512 rwx--    [ anon ]
0000000060000000       4       0       0 r-xs-  ora_ORATOOLS1_4227085_0
0000000060001000    4092     356     164 rwxs-  ora_ORATOOLS1_4227085_0
0000000060400000    4096       0       0 rwxs-  ora_ORATOOLS1_4227085_1
0000000078800000    4096      28      20 rwxs-  ora_ORATOOLS1_4227085_98
0000000079800000    4096      16       8 rwxs-  ora_ORATOOLS1_4227085_102
0000000079c00000    4096      16      16 rwxs-  ora_ORATOOLS1_4227085_103
000000007a000000    4096     384     136 rwxs-  ora_ORATOOLS1_4227085_104
000000007f000000    4096      80      36 rwxs-  ora_ORATOOLS1_4227085_124
000000007f400000    4096    1836      48 rwxs-  ora_ORATOOLS1_4227085_125
000000007f800000    4096    1084      92 rwxs-  ora_ORATOOLS1_4227085_126
000000007fc00000    4096      76      28 rwxs-  ora_ORATOOLS1_4227085_127
0000000080000000    4096       4       4 rwxs-  ora_ORATOOLS1_4227085_128
00000035b1600000     112     104       0 r-x--  ld-2.5.so
00000035b5600000      84      24       0 r-x--  libnsl-2.5.so
00000035b5615000    2044       0       0 -----  libnsl-2.5.so
00000035b5814000       4       4       4 r-x--  libnsl-2.5.so
00000035b5815000       4       4       4 rwx--  libnsl-2.5.so
00000035b5816000       8       0       0 rwx--    [ anon ]
00002b01b37e1000       8       8       8 rwx--    [ anon ]
00002b01b37e3000       4       4       0 r-x--  libodmd11.so
00002b01b37e4000    1024       0       0 -----  libodmd11.so
00002b01b58f0000      64      64      24 rwx--  zero
00002b01b5900000     144     144       0 rwx--  zero
00002b01b5924000       8       8       8 rwx--    [ anon ]
00002b01b5926000       4       4       0 rwxs-  hc_ORATOOLS1.dat
00002b01b5927000      40      16       0 r-x--  libnque11.so
00002b01b5931000    1020       0       0 -----  libnque11.so
00002b01b5a30000       4       4       4 rwx--  libnque11.so
00002b01b5a31000    1048      24      24 rwx--    [ anon ]
00007fff35e2b000      84      52      52 rwx--    [ stack ]
ffffffffff600000    8192       0       0 -----    [ anon ]
----------------  ------  ------  ------
total kB          770108   22116    3248


The lsof command:  is an extensive command that lists the open files on the system.  It can be used for diagnosing connectivity to a number of resource.  It provides on the usage of standard files, shared memory segments,
and network ports.

 lsof -u oracle | grep 4006
ORATOOLS1 - oracle: lsof -u oracle | grep 4006oracle     4006 oracle  cwd       DIR               8,17      4096               950323 /apps/grid/11.2.0/grid/dbs
oracle     4006 oracle  rtd       DIR              253,0      4096                    2 /
oracle     4006 oracle  txt       REG               8,17 200678430               952608 /apps/grid/11.2.0/grid/bin/oracle
oracle     4006 oracle  mem       REG               0,20   4194304                98378 /dev/shm/ora_+ASM1_4096007_21
oracle     4006 oracle  mem       REG               0,20   4194304                98379 /dev/shm/ora_+ASM1_4096007_22
oracle     4006 oracle  mem       REG               0,20   4194304                98416 /dev/shm/ora_+ASM1_4096007_59
...
oracle     4006 oracle  mem       REG              253,0    139416               786005 /lib64/ld-2.5.so
oracle     4006 oracle  mem       REG              253,0   1717800               786006 /lib64/libc-2.5.so
oracle     4006 oracle  mem       REG              253,0    615136               786018 /lib64/libm-2.5.so
oracle     4006 oracle  mem       REG              253,0     23360               786008 /lib64/libdl-2.5.so
oracle     4006 oracle  mem       REG              253,0    145824               786009 /lib64/libpthread-2.5.so
oracle     4006 oracle  mem       REG              253,0     53448               786010 /lib64/librt-2.5.so
oracle     4006 oracle  mem       REG              253,0    114352               786023 /lib64/libnsl-2.5.so
oracle     4006 oracle  mem       REG               8,17     12315               958409 /apps/grid/11.2.0/grid/lib/libodmd11.so
oracle     4006 oracle  mem       REG               8,17    532417               958508 /apps/grid/11.2.0/grid/lib/libcell11.so
oracle     4006 oracle  mem       REG               8,17    993144               958469 /apps/grid/11.2.0/grid/lib/libskgxp11.so
oracle     4006 oracle  mem       REG               8,17   7898628               958504 /apps/grid/11.2.0/grid/lib/libnnz11.so
oracle     4006 oracle   10r      CHR                1,3                           1449 /dev/null
oracle     4006 oracle   11r      CHR                1,3                           1449 /dev/null
oracle     4006 oracle   12w      REG               8,17    284821              1261703 /apps/grid/11.2.0/grid/log/si01/alertsi01.log
oracle     4006 oracle   13u     IPv4             103693                            UDP 169.254.101.90:59753
oracle     4006 oracle   14u      REG               8,17      1544               958679 /apps/grid/11.2.0/grid/dbs/hc_+ASM1.dat
oracle     4006 oracle   15r      DIR                0,3         0            262537225 /proc/4006/fd
oracle     4006 oracle   16u     unix 0xffff81004225da00                          84931 socket
oracle     4006 oracle   17u     unix 0xffff81004225dcc0                          84932 socket
oracle     4006 oracle   18u     unix 0xffff81004223a100                          84933 socket
oracle     4006 oracle   19u     unix 0xffff81004225cf00                          84934 socket
oracle     4006 oracle   20r      CHR                1,5                           1451 /dev/zero
oracle     4006 oracle   21u     IPv4             103694                            UDP localhost.localdomain:40395
oracle     4006 oracle   22r      CHR                1,5                           1451 /dev/zero
oracle     4006 oracle   23r      REG               8,17   1092096              1179709 /apps/grid/11.2.0/grid/rdbms/mesg/oraus.msb
oracle     4006 oracle   24u      REG               8,17      1544               958679 /apps/grid/11.2.0/grid/dbs/hc_+ASM1.dat
oracle     4006 oracle   25u     IPv4             103695                            UDP 169.254.101.90:51252
oracle     4006 oracle   26u     IPv6             103696                            UDP localhost6.localdomain6:9648
oracle     4006 oracle   28u     IPv4             328057                            TCP si01-vip.an.com:33874->si01-vip.an.com:ncube-lm (ESTABLISHED)

Changing the Broker Configuration Properties

You can modify the configuration, database, and instance using the EDIT command.  See several examples below

Changing the Broker Configuration Properties
DGMGRL> show database verbose 'DBATOOLS'
Database - DBATOOLS
  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    DBATOOLS
  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=pi01.an.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBATOOLS)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = ''
    LogFileNameConvert              = ''
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    SidName                         = 'DBATOOLS'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=pi01.an.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBATOOLS_DGMGRL)(INSTANCE_NAME=DBATOOLS)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'
Database Status:
SUCCESS
DGMGRL> show database 'DBATOOLS' statusreport
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT
      
DGMGRL> show database 'DBATOOLS' logshipping;
  LogShipping = 'OFF' 
DGMGRL> edit database 'DBATOOLS' set Property Logshipping='ON';
Property "logshipping" updated
DGMGRL> show database 'DBATOOLS' logshipping;
  LogShipping = 'ON'
DGMGRL> show database 'DBATOOLS' lOgXptMode;
  LogXptMode = 'ASYNC'
DGMGRL> edit database 'DBATOOLS' set Property LogXptMode='SYNC';
Property "logxptmode" updated
DGMGRL> show database 'DBATOOLS' LogXptMode
  LogXptMode = 'SYNC'

To turn transport on / off using TRANSPORT-ON/TRANSPORT-OFF
DGMGRL> edit database 'DBATOOLS' SET STATE=TRANSPORT-OFF;
ALTER SYSTEM SET log_archive_dest_state_2='RESET' SCOPE=BOTH;

DGMGRL> edit database 'DBATOOLS' SET STATE=TRANSPORT-ON';
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
To turn the apply services on or off on the standby database
DGMGRL> show configuration
Configuration - DBATOOLS
  Protection Mode: MaxPerformance
  Databases:
    DBATOOLS - Primary database
    STBYTOOL - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

Changing The State of A Database
DGMGRL> edit database 'DBATOOLS' set STATE=APPLY-OFF;
Error: ORA-16516: current state is invalid for the attempted operation
Failed.
DGMGRL> edit database 'STBYTOOL' set STATE=APPLY-OFF;
In the alert log, you should see the below line
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL

SQL> alter database open read only;
Database altered.

DGMGRL> edit database 'STBYTOOL' set STATE=APPLY-ON
In the alert log, you should see this line
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

Online Patches for the RDBMS instances

An online patch is a special kind of patch that can be applied to a live, running RDBMS instance.  It contains a single shared library and does not required to shutdown the instance or relink the oracle binary.   It utilizes the oradebug interface to install and enable the patches and is required additional memory consumption overhead.

Online Patches installation:

1)  opatch query -is_online_patch
--------------------------------------------------------------------------------
 Patch is an online patch: true

2)  opatch apply online -connectString LAXCNT:sys:Im1laxcnt
Installing and enabling the online patch 'bug9407198.pch', on database 'LAXCNT'

3)  opatch util enableOnlinePatch -connectString SANCNT:sys:Im1sancnt: -id 9407198
Invoking utility "enableonlinepatch"
Installing and enabling the online patch 'bug9407198.pch', on database 'SANCNT'.

alert_SANCNT.log
Patch bug9407198.pch Installed - Update #1
Patch bug9407198.pch Enabled - Update #2
Tue Jan 25 15:26:03 2011
Online patch bug9407198.pch has been installed
Online patch bug9407198.pch has been enabled

Online Patching Best Practices (metalink note:  761111.1)
  • It is strongly recommended to rollback all online patches and replace them with regular (offline) patches on next instance shutdown
  • Online patches should be used when the patch needs to be applied urgently and a downtime cannot be scheduled. IMPORTANT: It is strongly recommended to rollback all online patches and replace them with regular (offline) patches on next instance shutdown
  • Apply one instance at a time
  • When rolling back online patches, ensure all patched instances are included
    • Avoids the dangerous and confusing situation of having different software across instances using the same $ORACLE_HOME
  • Assess memory impact on a test system before deploying to production
    • Example: pmap command
  • Never remove $ORACLE_HOME/hpatch directory

ORA-600 [17147] during gathering table statistics with BMS_STATS.auto_degree

SQL> begin
  2        DBMS_STATS.gather_table_stats
  3                             (ownname =>               'DS_APP',
  4                              tabname =>               'TD_ITEM_INV',
  5                              estimate_percent =>      DBMS_STATS.auto_sample_size,
  6                              method_opt =>            'FOR ALL INDEXED COLUMNS SIZE AUTO',
  7                              CASCADE =>               TRUE,
  8                              DEGREE =>               
DBMS_STATS.auto_degree  9                             );
10  end;
11  /
begin
*
ERROR at line 1:
ORA-20011: Approximate NDV failed: ORA-12801: error signaled in parallel queryserver P008
ORA-00600: internal error code, arguments: [17147], [0x085241FA8], [], [], [],
[], [], [], [], [], [], []
ORA-06512: at "SYS.DBMS_STATS", line 23112
ORA-06512: at "SYS.DBMS_STATS", line 23205
ORA-06512: at line 2


This is an Oracle bug on Linux with 11.2.   A workarround is to change DEGREE = 1 or apply Patch #10013177

ORA-28579: network error during callback from external procedure

One of our application used EXTPROC_LISTENER/ external procedure, and the application displayed two errors when the SQL statement ran 120 seconds or more.


ORA-28576: lost RPC connection to external procedure agent
ORA-28579: network error during callback from external procedure



Further investigation, the SQLNET.INBOUND_CONNECT_TIMEOUT was a suspect.  It was set to 120 seconds in the sqlnet.ora


NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
SQLNET.INBOUND_CONNECT_TIMEOUT = 120



SQLNET.INBOUND_CONNECT_TIMEOUT parameter specifies the time (60 seconds is the default) that would be allowed for a client to connect with the database server and provide necessary authentication information.  If the client fails to establish a connection and complete authentication in the time specified, then the database server terminates the connection.  Oracle recommends setting this parameter in combination with the INBOUND_CONNECT_TIMEOUT_<listener_name> parameter in the listener.ora file.

Setup 10046 and 28579 errorstack at database level

 SQL> alter session set tracefile_identifier='10046';
SQL> alter session set timed_statistics = true;
SQL> alter session set statistics_level=all;
SQL> alter session set max_dump_file_size = unlimited;
SQL> alter session set events '10046 trace name context forever,level 12';

SQL> alter system set events ‘28579 trace name errorstack level 3’;


To unset it after trace file is produced:
SQL> alter system set events ‘28579 trace name errorstack off’;


Setup sqlnet and Extproc listener Traces
1. Enable the client trace in client side sqlnet.ora.
(If you are running from database server itself,then you need to set the following in server sqlnet.ora)
TRACE_LEVEL_CLIENT = 16
TRACE_FILE_CLIENT = Client
TRACE_DIRECTORY_CLIENT= <valid directory path that exists>
TRACE_TIMESTAMP_ CLIENT = ON
TRACE_UNIQUE_CLIENT = ON
DIAG_ADR_ENABLED =OFF

2.Enable the server trace in server side sqlnet.ora
TRACE_LEVEL_SERVER = 16
TRACE_FILE_SERVER = Server
TRACE_DIRECTORY_SERVER = <valid directory path that exists>
TRACE_TIMESTAMP_SERVER=ON
DIAG_ADR_ENABLED =OFF

3. Enable the listener tracing for Extproc listener (Listener needs to be restart to enable / disable the trace)

TRACE_FILE_<LISTENER NAME>=Listener
TRACE_LEVEL_<LISTENER NAME>=SUPPORT
TRACE_TIMESTAMP_<LISTENER NAME>=TRUE
TRACE_DIRECTORY_<LISTENER NAME>=<valid directory path>
DIAG_ADR_ENABLED_listener_name>=off

4.Enable the tracing for Extproc Agent (in Server side sqlnet.ora)
TRACE_LEVEL_AGENT= SUPPORT
TRACE_DIRECTORY_AGENT =<valid directory path that exists>
TRACE_FILE_AGENT=agent
TRACE_TIMESTAMP_AGENT =ON

Conclusion:
The issue occured in our application when SQLNET.INBOUND_CONNECT_TIMEOUT = 120, the   parameter was "explicitly" set to a hard coded value, it forced the External Procedure callout execution to that limited value. The problem was resolved either changing SQLNET.INBOUND_CONNECT_TIMEOUT  = 0 (client connection to the Database server can stay open  indefinitely without authentication) or comment it from sqlnet.ora

enq: HW contention - Bug 9407198

enq: HW contention  occured at the top wait event as a result of multiple sessions insert data into the same table.  This event is the competition for high water mark lock.  Oracle Support confirmed that we were hitting a bug 9407198.  The bug's summary is below..

Bug 9407198  "LOG ERRORS INTO" can cause OERI[kcb***] or hang scenarios
 This note gives a brief overview of bug 9407198.
 The content was last updated on: 23-FEB-2011
 Click here for details of each of the sections below.

Affects:

Product (Component)Oracle Server (Rdbms)
Range of versions believed to be affectedVersions BELOW 12.1
Versions confirmed as being affected
Platforms affectedGeneric (all / most platforms affected)

Fixed:

This issue is fixed in

Symptoms:

Related To:

  • DBMS_ERRLOG

Description

"LOG ERRORS INTO" can cause various ORA-600 errors, typically 
beginning ORA-600 [kcb*], or hangs, if an ORA-1578 or ORA-1115
error occurs during processing of the SQL statement.

Workaround:
 Address the cause of the triggering error (ie: ORA-1578 or ORA-1115).
 or
 Do not use the error logging clause on SQL statements
 

Note: 
 This fix only addresses the scenario of ORA-1578 and 
 ORA-1115 causing a subsequent problem during error logging.
 See bug 9150510 for a similar issue if the triggering error
 is an ORA-3113 error.
 
SQLTXPLAIN is known as SQLT.  It’s a tool provided by Oracle Support Center of Expertise CoE.  This tool helps to diagnose SQL statements performing poorly by inputs one SQL statements and outputs set of diagnostics files. 

Download and Installation:  You can get sqlt.zip from the Oracle metalink and unzip it to a local file system on your server.

SQL> start sqdrop.sql
... uninstalling SQLT, please wait

SQL> start sqcreate.sql
During the installation you will be asked these questions:
Optional Connect Identifier.
SQLTXPLAIN password.
SQLTXPLAIN Default Tablespace.
SQLTXPLAIN Temporary Tablespace.
Application User –the main application schema
Licensed Oracle Pack.
"T" if you have license for Diagnostic and Tuning
"D" if you have license only for Oracle Diagnostic
"N" if you do not have these two license

Some examples to use this tool:

SQL> connect rodba/readdba
SQL> run sqltxplain.sql /apps/oracle/sql1.sql (sql1.sql is my text file)
SQL> start sqltxtract.sql 1z2putj05jwfd
SQL > start sqltxecute.sql /apps/oracle/sql1.sq
SQL> start sqltcompare.sql (to compare CBO environment and sys stats, statistics, execution plans, tables, indexes, partitions, etc.)
SQL > start sqltprofile.sql;

OVM-1004 XML – RPC Client Call Oracle VM Agent API ‘check_sys_passwd’ error

When creating a server pool with the Oracle VM Manager, I got this error  OVM-1004 XML – RPC Client Call Oracle VM Agent API ‘check_sys_passwd’ error.  After viewing logs from /var/log/ovs-agent/*, and /var/log/ovm-manager/*, I fixed this problem by correcting 3 points below:
1)  Checked all the hosts and see if there was firewall blocking between the VM Manager and VM servers
2)  Checked  /etc/hosts to make sure host names / IP addresses were correct
3)  Checked and corrected the system time to make sure they were in sync. 

Monitor Standby Site

**SCN and additional information stored in the Standby's data file headers


SQL>  select status, to_char(checkpoint_change#, '999999999999999') as checkpoint_change#, to_char(checkpoint_time, 'DD-MON-YYYY HH24:MI:SS') as checkpoint_time,count(*), fuzzy 
from v$datafile_header
group by status,checkpoint_change#,checkpoint_time, fuzzy;

STATUS  CHECKPOINT_CHANG CHECKPOINT_TIME        COUNT(*) FUZ
------- ---------------- -------------------- ---------- ---
ONLINE     5976304015881 07-AUG-2011 04:31:06         44 NO


**Distinct SCNs stored in the Standby's data file headers

SQL> select distinct to_char(fhscn, '99999999999999999999') as Stby_datafile_hdr_SCN
from X$KCVFH;
STBY_DATAFILE_HDR_SCN
---------------------
        5976304015881


**Next archived log sequence number the Standby needs to apply

SQL> select distinct inst_id, fhrba_seq, count(*) from X$KCVFH group by INST_ID,FHRBA_SEQ;
   INST_ID  FHRBA_SEQ   COUNT(*)
---------- ---------- ----------
         1      16683         44


**Use the LOWEST SCNs stored in the Standby's data file headers to extract this backup from the Primary database

SQL> select min(to_char(fhscn, '99999999999999999999')) as SCN_to_be_used from X$KCVFH;
SCN_TO_BE_USED
---------------------
        5976304015881

Monitor Logs From Primary and Standby Databases

Query 1:  - Primary

SQL> select thread#, max(sequence#) "Last Primary Seq Generated"
  2  from v$archived_log val, v$database vdb
  3  where val.resetlogs_change# = vdb.resetlogs_change#
  4  group by thread# order by 1;

   THREAD# Last Primary Seq Generated
---------- --------------------------
         1                      21893
         2                      16200
         3                      14331
         4                      16682

Query2:  - Physical Standby:

SQL> select thread#, max(sequence#) "Last Standby Seq Received"
  2  from v$archived_log val, v$database vdb
  3  where val.resetlogs_change# = vdb.resetlogs_change#
  4  group by thread# order by 1;

   THREAD# Last Standby Seq Received
---------- -------------------------
         1                     21893
         2                     16200
         3                     14331
         4                     16682
Compare the values from Query 1 and Query 2, if there is a difference between the values of "Last Primary Seq Generated" and "Last Standby Seq Received”, the Primary site has isses of shipping logs to the Standby server.

Query 3:  - Physical Standby:

SQL> select thread#, max(sequence#) "Last Standby Seq Applied"
  2  from v$archived_log val, v$database vdb
  3  where val.resetlogs_change# = vdb.resetlogs_change#
  4  and val.applied='YES'
  5  group by thread# order by 1;

   THREAD# Last Standby Seq Applied
---------- ------------------------
         1                    21892
         2                    16199
         3                    14330
         4                    16682

Compare the values from Query 2 and Query 3, if there is a difference between the values of "Last Standby Seq Received" and "Last Standby Seq Applied", the standby has issue or logs have not yet applied.

AWR scripts

awrrpt.sql  -- Displays statistics for a range of snapshot Ids.

awrrpti.sql -- Displays statistics for a range of snapshot Ids on a specified database and instance.

awrsqrpt.sql -- Displays statistics of a particular SQL statement for a range of snapshot Ids. Run this report to investigate the performance of a SQL statement.

awrsqrpi.sql -- Displays statistics of a particular SQL statement for a range of snapshot Ids on a specified database and instance. Run this to investigate performance of a SQL statement on a specific database and instance.

awrddrpt.sql -- Compares detailed performance attributes and configuration settings between two selected time periods.

awrddrpi.sql --Compares detailed performance attributes and configuration settings between two selected time periods on a specific database and instance.

awrsqrpt.sql -- SQL performance report

OPATCH_DEBUG

When applying a patch, you could run into failure during prerequisite checks..
Running prerequisite checks...
Prerequisite check "CheckApplicable" failed.
The details are:
To debug the issue, you can run opatch with debug option as
export OPATCH_DEBUG=TRUE

opatch lsinv
opatch apply

ORA-01110

ORA-01110: data file 130: '/data/apps/PROD/data1/lob12.dbf'
: INSERT_MAP generic failure:
file  130 cannot be read at this time
The file was there, but it was offline. 
recover datafile 130;
alter database datafile 130 online;

SQL*NET Tracing

If you run into connection timeout issue, firewall, etc..between client and server, you should enable SQL*NET tracing from both client and server.  

To enable client tracing, you can follow the below steps:

(1) Open the SQLNET.ORA file typically found in the following location:
ORACLE_BASE\ORACLE_HOME\Network\Admin

(2) Add the following parameters at the end of the file:
#CLIENT-SIDE SQL*NET TRACE PARAMETERS
#====================================
TRACE_UNIQUE_CLIENT = ON
TRACE_LEVEL_CLIENT = 16
TRACE_DIRECTORY_CLIENT = C:\temp
TRACE_FILE_CLIENT = SQLNetTrace
TRACE_TIMESTAMP_CLIENT = ON
#TRACE_FILELEN_CLIENT = 2048
#TRACE_FILENO_CLIENT = 2

 

To enable the server side tracing, please follow the below steps:
(1) Open the SQLNET.ORA file typically found in the following location:
     ORACLE_BASE\ORACLE_HOME\Network\Admin
(2) Add the following parameters at the end of the file:

#SERVER-SIDE SQL*NET TRACE PARAMETERS
#====================================
TRACE_LEVEL_SERVER = 16
TRACE_DIRECTORY_SERVER = C:\temp
TRACE_FILE_SERVER = SQLNetTrace
TRACE_TIMESTAMP_SERVER = ON

Installing Oracle VM Manager 2.2

You will need OracleVM Server and Oracle VM Manager ISO files.

[root@lrac01 ~]# cd Desktop
[root@lrac01 Desktop]# ls
OracleVM-Manager-2.2.0.iso  OracleVM-Server-2.2.2.iso
[root@lrac01 Desktop]# mkdir ovm
[root@lrac01 Desktop]# ls
OracleVM-Manager-2.2.0.iso  OracleVM-Server-2.2.2.iso  ovm
[root@lrac01 Desktop]# mount -t iso9660 -o loop OracleVM-Manager-2.2.0.iso ovm[root@lrac01 Desktop]# cd ovm
[root@lrac01 ovm]# ls
EULA  LICENSE  readme.txt  runInstaller.sh  scripts  source  TRANS.TBL
[root@lrac01 ovm]# ls -ltr
[root@lrac01 ovm]# sh runInstaller.shWelcome to Oracle VM Manager 2.2
Please enter the choice: [1|2|3]
1. Install Oracle VM Manager
2. Uninstall Oracle VM Manager
3. Upgrade Oracle VM Manager
1Starting Oracle VM Manager 2.2 installation …
Do you want to install a new database or use an existing one? [1|2]
1. Install a new Oracle XE database on lrac01
2. Use an existing Oracle database in my network
1Prepare to install the Oracle XE database ...
Checking the supported platforms ... Done
Checking the prerequisite packages are installed ... Done
Checking the available disk space ... Done
Installing the oracle-xe-univ package (rpm) now ...
 Done
Oracle Database 10g Express Edition Configuration
-------------------------------------------------
This will configure on-boot properties of Oracle Database 10g Express Edition.  The following questions will determine whether the database should be starting upon system boot, the ports it will use, and the passwords that will be used for database accounts.  Press <Enter> to accept the defaults. 
Ctrl-C will abort.
Specify the HTTP port that will be used for Oracle Application Express [8080]:
Specify a port that will be used for the database listener [1521]:
Specify a password to be used for database accounts.  Note that the same password will be used for SYS and SYSTEM.  Oracle recommends the use of different passwords for each database account.
This can be done after initial configuration:
Confirm the password:
Do you want Oracle Database 10g Express Edition to be started on boot (y/n) [y]:yStarting Oracle Net Listener...Done
Configuring Database...Done
Starting Oracle Database 10g Express Edition Instance...Done
Installation Completed Successfully.
To access the Database Home Page go to "http://10.112.5.1:8080/apex"
Checking the availability of the database ...
Set default database schema to 'OVS'.
Please enter the password for account 'OVS':
Confirm the password:
Creating the Oracle VM Manager database schema ...Done
Installing the ovs-manager package (rpm) ...
Done
Installing the oc4j package (rpm) ...
Done
Please enter the password for account 'oc4jadmin':
Confirm the password:
Starting OC4J ... Done.
To access the OC4J Home Page and change the password go to http://10.112.5.1:8888/em

Deploying Oracle VM Manager application to OC4J container.
Creating connection pool ... Done
Creating data source ... Done
Deploying application help ... Done
Deploying application ... Done
Please enter the keystore password for the Web Service:
Confirm the password:
Setting keystore password for Web Service ... Done
Do you want to use HTTPS access for Oracle VM Manager (Y|n)?nConfiguring OC4J to use HTTP ... Done
Stopping OC4J ... Done
Starting OC4J ... Done
Please enter the password for the default account 'admin':
Confirm the password:
Configuring SMTP server ...
Please enter the outgoing SMTP mail server(e.g. - mail.abc.com, mail.abc.com:25): rac01.an.com
Mail server checking, may need some time, please wait ...
Setting the SMTP server to rac01.an.com ...
Done
Please enter an e-mail address for account 'admin': root@rac01.an.com

Confirm the e-mail address : root@rac01.an.com

Updating e-mail address for account 'admin' to 'root@rac01.an.com' ...
Done
The console feature is not enabled by default.
For detailed setup, refer to Oracle VM Manager User's Guide
Installation of Oracle VM Manager completed successfully.
To access the Oracle VM Manager 2.2 home page go to:  http://10.112.5.1:8888/OVS

To access the Oracle VM Manager web services WSDL page go to:
  http://10.112.5.1:8888/OVSWS/LifecycleService.wsdl

  http://10.112.5.1:8888/OVSWS/ResourceService.wsdl

  http://10.112.5.1:8888/OVSWS/PluginService.wsdl

  http://10.112.5.1:8888/OVSWS/ServerPoolService.wsdl

  http://10.112.5.1:8888/OVSWS/VirtualMachineService.wsdl

  http://10.112.5.1:8888/OVSWS/AdminService.wsd

To access the Oracle VM Manager help page go to:
  http://10.112.5.1:8888/help/help

How Numa Allocates Memory

According to MOS, In a NUMA system, processors, memory and I/O are group together into nodes so that each processor is bound to a specific memory address.  By default a NUMA system will chose the local node to allocate memory from and totally exhaust all of the memory from that node before deciding to allocate memory from other remote NUMA nodes.  While this results in holding an object that will fit in a single NUMA node and avoid fragmentation, it can also result in aggressive swapping on one node while there is plenty of memory on other nodes.

It's strongly recommended to evaluate the performance and perform sufficient testing on the NUMA settings.  In our environment, we disable it. If NUMMA is disabled, it's noted by the kernel:

dmesg | grep -i numa

Command line: ro root=/dev/sysVG/rootLV numa=off crashkernel=128M@16M
NUMA turned off
Kernel command line: ro root=/dev/sysVG/rootLV numa=off crashkernel=128M@16M


Alternatively,

 - root: cat /proc/cmdline
ro root=/dev/sysVG/rootLV numa=off crashkernel=128M@16M

cat /etc/grub.conf
 kernel /vmlinuz-2.6.18-238.9.1.el5 ro root=/dev/sysVG/rootLV numa=off crashkernel=128M@16M


  - root: numactl --hardware
 available: 1 nodes (0)
 node 0 size: 145416 MB
 node 0 free: 46749 MB
 node distances:
 node   0
  0:  10

Starting 11g Release 2, _enable_NUMA_support default is set to FALSE.

  1  select a.KSPPINM "Parameter", b.KSPPSTVL "Session Values"
  2  , c.KSPPSTVL "Instance Value"
  3  from x$ksppi a, x$ksppcv b, x$ksppsv c
  4  where a.INDX = b.INDX
  5* and a.KSPPINM = '_enable_NUMA_support'

_enable_NUMA_support
FALSE


TCPDUMP to capture Netowrk Traffic

tcpdump is a network utility that listeners and capture network traffic.  It can be useful in investigating network problem.  Running tcpdump needs to be run as root in able to sniff network packets.
tcpdump -D to get a list of network interfaces on the system
1.eth0
2.eth1
3.eth2
4.any (Pseudo-device that captures on all interfaces)
5.lo

tcpdump -i eth0 -s 65535 -W dump
65535:  max size allowed
w:  outputs captured information to the specified file.  dump is the filename

To show the packets being transferred across the private interconnect:
tcpdump -i 2 | more