dig, nslookup to determine scan name is mapped to the correct IP addresses

With 11gR2 Grid Infrastructure  we need 3 SCAN IPs on the same subnet with the Virtual IPs  resolving the same name with DNS round-robin resolution.  The SCAN name and IP addresses are not listed in the /etc/hosts file and not ping-able unit the Grid Infrastructural is configured.

How to determine that our SCAN name is mapped to the right IP addresses?  2 methods

Method 1:

srvctl config scan

SCAN name: lapqa-scan, Network: 1/10.112.14.0/255.255.255.0/bond0
SCAN VIP name: scan1, IP: /lapqa-scan/10.112.14.101
SCAN VIP name: scan2, IP: /lapqa-scan/10.112.14.102
SCAN VIP name: scan3, IP: /lapqa-scan/10.112.14.103

nslookup lapqa-scan
Server:         10.210.121.15
Address:        10.210.121.15#53

Name:   lapqa-scan.ctt.com
Address: 10.112.14.103
Name:   lapqa-scan.ctt.com
Address: 10.112.14.101
Name:   lapqa-scan.ctt.com
Address: 10.112.14.102


Method 2:

crsctl query dns –servers

CRS-10018: the following configuration was found on the system:
CRS-10019: There are 1 domains in search order. They are:
ctt.com
CRS-10022: There are 2 name servers. They are:
10.210.121.15
10.101.11.109
CRS-10020: number of retry attempts for name lookup is: 4
CRS-10021: timeout for each name lookup is: 5

Alternatively,

cat /etc/resolv.conf
domain ctt.com
nameserver 10.210.121.15
nameserver 10.101.11.109

nslookup 10.210.121.15
Server:         10.210.121.15
Address:        10.210.121.15#53

79.129.250.10.in-addr.arpa      name = xy1.cttxyx.local.


+ASM1 - oracle: dig @xy1.cttxyx.local lapqa-scan.ctt.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> @xy1.cttxyx.local lapqa-scan.ctt.com
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50505
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;lapqa-scan.ctt.com.           IN      A

;; ANSWER SECTION:
lapqa-scan.ctt.com.    30      IN      A       10.112.14.102
lapqa-scan.ctt.com.    3600    IN      A       10.112.14.103
lapqa-scan.ctt.com.    30      IN      A       10.112.14.101

;; AUTHORITY SECTION:
ctt.com.                3600    IN      NS      xy1.cttxyx.local.
ctt.com.                3600    IN      NS      xy2.cttxyx.local.

;; ADDITIONAL SECTION:
xy1.cttxyx.local.      86400   IN      A       10.210.121.15
xy2.cttxyx.local.      86400   IN      A       10.101.11.109

;; Query time: 20 msec
;; SERVER: 10.210.121.15#53(10.210.121.15)
;; WHEN: Tue Mar  5 11:46:24 2013
;; MSG SIZE  rcvd: 166

Deinstalling the Grid Infrastructure Software

You don’t have to manually remove directories.  There is a deinstall script in the grid home directory.  The deinstall checks and prompt questions to confirm before it performs a clean deinstallation (see below).

cd /apps/grid/11.2.0/grid/deinstall

LADB1 - oracle: ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2012-05-23_02-25-20PM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /apps/11.2.0/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is: /apps/oracle
Checking for existence of central inventory location /apps/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /apps/11.2.0/grid
The following nodes are part of this cluster: dbsrvl100,dbsrvl101
Checking for sufficient temp space availability on node(s) : 'dbsrvl100,dbsrvl101'

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2012-05-23_02-25-20PM/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/netdc_check2012-05-23_02-25-48-PM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/asmcadc_check2012-05-23_02-37-14-PM.log

Automatic Storage Management (ASM) instance is detected in this Oracle home /apps/11.2.0/grid.
ASM Diagnostic Destination : /apps/oracle
ASM Diskgroups : +GI
ASM diskstring : /dev/oracleasm/disks
Diskgroups will be dropped
De-configuring ASM will drop all the diskgroups and it's contents at cleanup time. This will affect all of the databases and ACFS that use this ASM instance(s).
If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'. Do you  want to modify above information (y|n) [n]: y
Specify the ASM Diagnostic Destination [/apps/oracle]:
Specify the diskstring [/dev/oracleasm/disks]:
Specify the diskgroups that are managed by this ASM instance [+GI]:

De-configuring ASM will drop the diskgroups at cleanup time. Do you want deconfig tool to drop the diskgroups y|n [y]: y


######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /apps/11.2.0/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:dbsrvl100,dbsrvl101
Oracle Home selected for deinstall is: /apps/11.2.0/grid
Inventory Location where the Oracle home registered is: /apps/oraInventory
Following RAC listener(s) will be de-configured: LISTENER
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2012-05-23_02-25-20PM/logs/deinstall_deconfig2012-05-23_02-25-25-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2012-05-23_02-25-20PM/logs/deinstall_deconfig2012-05-23_02-25-25-PM.err'

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/asmcadc_clean2012-05-23_02-38-32-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/netdc_clean2012-05-23_02-40-15-PM.log

De-configuring RAC listener(s): LISTENER

De-configuring listener: LISTENER
    Stopping listener: LISTENER
    Listener stopped successfully.
    Unregistering listener: LISTENER
    Listener unregistered successfully.
Listener de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END


---------------------------------------->

The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on  the local node after the execution completes on all the remote nodes.

Run the following command as the root user or the administrator on node "dbsrvl101".

/tmp/deinstall2012-05-23_02-25-20PM/perl/bin/perl -I/tmp/deinstall2012-05-23_02-25-20PM/perl/lib -I/tmp/deinstall2012-05-23_02-25-20PM/crs/install /tmp/deinstall2012-05-23_02-25-20PM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2012-05-23_02-25-20PM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Run the following command as the root user or the administrator on node "dbsrvl100".

/tmp/deinstall2012-05-23_02-25-20PM/perl/bin/perl -I/tmp/deinstall2012-05-23_02-25-20PM/perl/lib -I/tmp/deinstall2012-05-23_02-25-20PM/crs/install /tmp/deinstall2012-05-23_02-25-20PM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2012-05-23_02-25-20PM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode

Press Enter after you finish running the above commands

<----------------------------------------

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/apps/11.2.0/grid' from the central inventory on the local node : Done

Delete directory '/apps/11.2.0/grid' on the local node : Done

Delete directory '/apps/oraInventory' on the local node : Done

The Oracle Base directory '/apps/oracle' will not be removed on local node. The directory is not empty.

Detach Oracle home '/apps/11.2.0/grid' from the central inventory on the remote nodes 'dbsrvl101' : Done

Delete directory '/apps/11.2.0/grid' on the remote nodes 'dbsrvl101' : Done

Delete directory '/apps/oraInventory' on the remote nodes 'dbsrvl101' : Done

The Oracle Base directory '/apps/oracle' will not be removed on node 'dbsrvl101'. The directory is not empty.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2012-05-23_02-25-20PM' on node 'dbsrvl100'
Clean install operation removing temporary directory '/tmp/deinstall2012-05-23_02-25-20PM' on node 'dbsrvl101'

## [END] Oracle install clean ##


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
ASM instance was de-configured successfully from the Oracle home
Following RAC listener(s) were de-configured successfully: LISTENER
Oracle Clusterware is stopped and successfully de-configured on node "dbsrvl101"
Oracle Clusterware is stopped and successfully de-configured on node "dbsrvl100"
Oracle Clusterware is stopped and de-configured successfully.
Successfully detached Oracle home '/apps/11.2.0/grid' from the central inventory on the local node.
Successfully deleted directory '/apps/11.2.0/grid' on the local node.
Successfully deleted directory '/apps/oraInventory' on the local node.
Successfully detached Oracle home '/apps/11.2.0/grid' from the central inventory on the remote nodes 'dbsrvl101'.
Successfully deleted directory '/apps/11.2.0/grid' on the remote nodes 'dbsrvl101'.
Successfully deleted directory '/apps/oraInventory' on the remote nodes 'dbsrvl101'.
Oracle Universal Installer cleanup was successful.


Run 'rm -rf /etc/oraInst.loc' as root on node(s) 'dbsrvl100,dbsrvl101' at the end of the session.

Run 'rm -rf /opt/ORCLfmap' as root on node(s) 'dbsrvl100,dbsrvl101' at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

11.2.0.3 cluvfy comp healthcheck

Starting with Oracle Grid Infrastructure 11g release 2 (11.2.0.3) and later, you can use the CVU healthcheck command to check your Oracle Clusterware and Oracle Database installations for their compliance with mandatory requirements and best practices guidelines, and to ensure that they are functioning properly.  In addition, If you want to perform best practices checks on these databases, then you must create the cvusys user on each database and grant that user the CVUSAPP role with the SELECT privileges needed to perform the best practice checks. The SQL script is cvusys.sql in the CVU_home/cv/admin/ directory to facilitate the creation of this user. Use this SQL script to create the cvusys user on all the databases that you want to verify using CVU.

-rw-r--r-- 1 oracle oinstall 2841 Apr 27  2011 cvusys.sql
-rwxr-xr-x 1 oracle oinstall 1011 Apr 27  2011 cvu_config*

SQL> @cvusys.sql
DROP USER cvusys
          *
ERROR at line 1:
ORA-01918: user 'CVUSYS' does not exist

DROP ROLE cvusapp
          *
ERROR at line 1:
ORA-01919: role 'CVUSAPP' does not exist

Enter password for user cvusys
'Creating user cvusys...'
User created.

Grant succeeded.

Role created.

Grant succeeded.
Grant succeeded.
Grant succeeded.
Grant succeeded.
................
................
................
................
Grant succeeded.


Usage:

cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name] [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]
cluvfy comp healthcheck
Verifying OS mandatory requirements
Verifying OS Best Practice
Verifying Clusterware mandatory requirements
Verifying Clusterware Best Practice
Verifying Database "wesbdr"
Please specify password for user "cvusys" : 

******************************************************************************************
Summary of environment
******************************************************************************************
System recommendations
******************************************************************************************
******************************************************************************************
Clusterware requirements
******************************************************************************************
******************************************************************************************
Clusterware recommendations
******************************************************************************************
Database recommendation checks for "wesbdr"
******************************************************************************************

TCP and UDP parameters

The  default maximum Linux TCP buffer sizes are too small.Oracle recommendations for related TCP and UDP parameters are following:

net.core.rmem_max = 4194304 

net.core.rmem_default=262144
net.core.rmem_max= for 11g: 4194304 For 10g: 2097152 (With RDS use at least 4194304)
net.core.wmem_default=262144
net.core.wmem_max=1048576 (With RDS use at least 2097152)
ip_local_port_range 9000 65500

Prerequisite Steps for Additing cluster Nodes

Below are the steps to prepare a node to be added to the cluster:

1.  Server build out and establish physical connection:  Build out servers, OS, confiure network, interconnect, share disks, etc..

2.  Install OS:  Clone a image of OS that matches with other nodes in your cluster including patches updates, drivers, etc..

3.  Create Oracle users and groups:  Make sure all matches up

4.  Setup SSH

Use cluvfy to validate the hardware and OS installation.

cluvfy stage -post hwos -n all -verbose
cluvfy comp peer -refnode RAC01 -n RAC08,RAC09 -orainv oinstall -osdba asmdba -verbose
cluvfy stage -pre nodeadd -n RAC08,RAC09 -fixup

TAF - Transparent Application Failover

TAF is a feature of OCI.  It allows applications to reconnect if the initial connection fails.  TAF can resume the SELECT operation but all active transactions are rolled back.  TAF supports 2 types of failover methods:

1.  BASIC:  Reconnection is established at failover time.

TAF without FAN (fast application notification)
srvctl add service -d DBALAB -s DBSRV -r dbsrv01,dbsrv02 -P BASIC

TAF with FAN:
srvctl add service -d DBALAB -s DBSRV -r dbsrv01,dbsrv02 -P BASIC
srvctl start service -d DBALAB -s DBSRV
srvctl modify service -d DBALAB -s DBSRV -q TRUE -P BASIC -e SELECT -z 180 -w 5 -j LONG

2.  PRECONNECT:  In addition to the intial connection, the shadow connection is also created.

Note about Services and Load Balancing Advisory

There are two types of load balancing that you can implement:

1.  Client-side:  Balance the connections across the listeners
2.  Server-side:  The listener directs the connection requests to the best instance based on the load balance advisory (LBA).

You can use LONG or SHORT load balancing methods.  Just like its name, LONG is suitable for long-lived connections like connection pools and SQL*Forms.  SHORT is suitable for short-lived connections like ORDER service type..

ex:  srvctl modify service -d labbdr -s labbpoc -q TRUE -P BASIC -e SELECT -z 180 -w 5 -j LONG
(e:  failover method, z:  number of retries, w:  delay between, j:  load balancing goal

Load Balancing Advisory (LBA):  Advisory to send works across instance and is available for applications JDBC, ODP.  There are two types of service level goals:

1.  SERVICE_TIME:  based on the elapse time for work to be done where the rate of demands change.  Open orders, internet shopping type of workloads

ex:    srvctl modify service -d DBAPOC -s DBA_RAC -B SERVICE_TIME -j SHORT

2.  THROUGHPUT:  direct works according to throughput and suitable for workload like batch processing..
ex:    srvctl modify service -d DBAPOC -s dbatplt_rac -B THROUGHPUT -j LONG

Monitor service
select machine, failover_method, failover_type, failed_over, service_name, count(*) from v$session
group by machine, failover_method, failover_type, failed_over, service_name

Create Server Pool with crsctl or srvctl

srvctl add srvpool -h
 Adds a server pool to the Oracle Clusterware.
 
  Usage: srvctl add srvpool -g <pool_name> [-l <min>] [-u <max>] [-i <importance>] [-n "<server_list>"] [-f]
      -g <pool_name>           Server pool name
      -l <min>                 Minimum size of the server pool (Default value is 0)
      -u <max>                 Maximum size of the server pool (Default value is -1 for unlimited maximum size)
      -i <importance>          Importance of the server pool (Default value is 0)
      -n "<server_list>"       Comma separated list of candidate server names
      -f                       Force the operation even though some resource(s) will be stopped
    -h                       Print usage
   
   
 crsctl add serverpool -h
 Usage:
   crsctl add serverpool <spName> [[-file <filePath>] | [-attr "<attrName>=<value>[,...]"]] [-i]
 where
      spName          Add named server pool
      filePath        Attribute file
      attrName        Attribute name
      value           Attribute value
      -i              Fail if request cannot be processed immediately
     -f              Force option
    
    
   srvctl add srvpool -g ashley_sp -l 2 -u 5 -i 3 -n "ljtcax101, ljtbab102" 
 
  crsctl status serverpool -p
 
  NAME=ora.ashley_sp
  IMPORTANCE=3
  MIN_SIZE=2
  MAX_SIZE=5
  SERVER_NAMES=ljtcax101, ljtbab102
  PARENT_POOLS=
  EXCLUSIVE_POOLS=
  ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--

  To modify server pool
  srvctl modify serverpool ashley_sp -attr "MIN_SIZE=1,MAX_SIZE=3, IMPORTANCE=7"
 
 
  To delete remove server pool
  srvctl remove srvpool -g SP1

To check permission

  crsctl getperm serverpool ora.ashley_sp
  Name: ora.ashley_sp
  owner:oracle:rwx,pgrp:oinstall:rwx,other::r--

Resource Parameters

crsctl status resource ora.dbapoc.db -f

ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r—
Owner and access privileges granted to OS users and groups.  Oracle and oinstall have all privileges, but other users can only view the resource.

ACTIVE_PLACEMENT=1
When set to 1, Oracle Clusterware uses this attribute to reevaluate the placement of a resource during addition or restart of a cluster server. For resources where PLACEMENT=favored, Oracle Clusterware may relocate running resources if the resources run on a non-favored server when a favored one joins the cluster.

AUTO_START=restore
·         always: always restart
·         restore: restore the resource to the same start before the server is stopped.
·         never: never restarts

CARDINALITY=3
The number of servers on which a resource can run, simultaneously.

CHECK_INTERVAL=1
The time interval, in seconds, between repeated executions of the check action

RESTART_ATTEMTS = 2
The number of times that Oracle Clusterware attempts to restart a resource on the resource's current server before attempting to relocate it.

FAILURE_INTERVAL=60
The interval, in seconds, before which Oracle Clusterware stops a resource if the resource has exceeded the number of failures specified by the FAILURE_THRESHOLD attribute. If the value is zero (0), then tracking of failures is disabled.

START_DEPENDENCIES=hard(ora.DATAPD101.dg,ora.DATAPF101.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns) pullup(ora.DATAPD101.dg,ora.DATAPF101.dg)

START_TIMEOUT=600
The maximum time (in seconds) in which a start action can run. Oracle Clusterware returns an error message if the action does not complete within the time specified.

STOP_TIMEOUT=600
Maximum time in seconds a stop action can run

STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATAPD101.dg,shutdown:ora.DATAPF101.dg)
dependency: The only possible value is hard.

START_DEPENDENCIES=hard(ora.DATAPD101.dg,ora.DATAPF101.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns) pullup(ora.DATAPD101.dg,ora.DATAPF101.dg)
Specifies a set of relationships that Oracle Clusterware considers when starting a resource. You can specify a space-separated list of dependencies on several resources and resource types on which a particular resource can depend.  Dependency possible values are hard, weak, attraction, pullup, and dispersion. You can specify each dependency only once, except for pullup, which you can specify multiple times.

Working with OCRDUMP

Dump the contents of OCR and count the number of lines.  If you dump the OCR as root, you will see more information than grid/oracle user because information in the OCR is organized by keys that are associated with privileges.  See examples below:

As oracle or grid user:
ocrdump -stdout | wc -l
489

As root:
ocrdump -stdout | wc -l
2993

To dump the OCR content using XML format for the first 50 lines
              ocrdump -stdout -xml | head -50

To dump current ocr
                ocrdump -xml /apps/oracle/dump_current_ocr.xml

Dump the backup contents of a OCR in XML format, then compare it with the current OCR to detect any changes..

              ocrconfig -showbackup
si01.an.com     2010/11/17 11:40:39     /dba/backup00.ocr
si01.an.com     2010/11/17 07:40:37     /dba/backup01.ocr
si01.an.com     2010/11/17 03:40:35     /dba/backup02.ocr
si01.an.com     2010/11/16 03:40:27     /dba/day.ocr

           ocrdump -xml -backupfile /dba/day.ocr previous_day.ocr

 Compare: 
          diff dump_current_ocr.xml previous_day.ocr
3,4c3,5
< <TIMESTAMP>11/17/2011 13:57:05</TIMESTAMP>
< <COMMAND>/apps/oracle/product/11.2.0.2/grid/bin/ocrdump.bin -xml /apps/oracle/dump_current_ocr.xml </COMMAND>
---
> <TIMESTAMP>11/17/2011 14:00:46</TIMESTAMP>
> <DEVICE>/dba/day.ocr</DEVICE>
> <COMMAND>/apps/oracle/product/11.2.0.2/grid/bin/ocrdump.bin -xml -backupfile /dba/day.ocr previous_day.ocr </COMMAND>
879c880
< <VALUE><![CDATA[83]]></VALUE>
---
> <VALUE><![CDATA[75]]></VALUE>

Troubleshooting 11.2 Clusterware Node Evictions (Note 1050693.1)

Starting 11.2.0.2, a node eviction may not actually reboot the machine.  This is called a rebootless restart.

To identify which process initiates a reboot, you need to review below are important files

• Clusterware alert log in <GRID_HOME>/log/<nodename>alertnodename
• The cssdagent log(s) in <GRID_HOME>/log/<nodename>/agent/ohasd/oracssdagent_root
• The cssdmonitor log(s) in <GRID_HOME>/log/<nodename>/agent/ohasd/oracssdmonitor_root
• The ocssd log(s) in <GRID_HOME>/log/<nodename>/cssd
• The lastgasp log(s) in /etc/oracle/lastgasp or /var/opt/oracle/lastgasp
• IPD/OS or OS Watcher data.  IPD/OS is an old name for the Cluster Health Monitor.  The names can be used interchaneably although Oracle now calls the tool Cluster Health Monitor
• 'opatch lsinventory -detail' output for the GRID home
• Message files /var/log/message

Common Causes of eviction:

OCSSD Eviction: 1) Network failure or latencies issue between nodes.  It takes 30 consecutive missed checkins to cause a node eviction.  2)  Problem writing / reading the voting disk  3) A member kill escallation like the LMON process may request CSS to remove an instance from the cluster via the instance eviction mechanisim.  If this times out, it could escalate to a node evict.

CSSDAGENT or CSSDMONITOR Eviction:  1) OS Scheduler problem as a result of OS is locked upor execsive amounts of load on the server such as CPU utilization is as high as 100% 2) CSS process is hung 3) Oracle bug

Enable Trace / Debug

Below are several ways to enable tracing and debugging for Oracle RAC resources.

1)  SRVM_TRACE:  Enable Tracing for cluvfy, netca, and srvctl

      export SRVM_TRACE=TRUE

      srvctl config database -d db11g1

You can dynamically enable logging (level 1 to 5)

2)  Enable debug to capture clusterware resource:  crsctl set log res "<resource name">:1"
     crsctl set log res "ora.registry.acfs=1"
     Set Resource ora.registry.acfs Log Level: 1

After you collect all the trace, disable the trace:  crsctl set log res "<resource name >:0"

crsctl set log res "ora.registry.acfs=0"

Set Resource ora.registry.acfs Log Level: 0

3)  You can enable dynamic debugging for CRS, CSS, EVM, and other clusterware sub components.  The  crsctl lsmodules css, crs, evm commands use to list the module's components. 

crsctl lsmodules crs
List CRSD Debug Module: AGENT
List CRSD Debug Module: AGFW
List CRSD Debug Module: CLSFRAME
List CRSD Debug Module: CLSVER
List CRSD Debug Module: CLUCLS
List CRSD Debug Module: COMMCRS
List CRSD Debug Module: COMMNS
List CRSD Debug Module: CRSAPP
List CRSD Debug Module: CRSCCL
List CRSD Debug Module: CRSCEVT
List CRSD Debug Module: CRSCOMM
List CRSD Debug Module: CRSD
List CRSD Debug Module: CRSEVT
List CRSD Debug Module: CRSMAIN
List CRSD Debug Module: CRSOCR
List CRSD Debug Module: CRSPE
List CRSD Debug Module: CRSPLACE
List CRSD Debug Module: CRSRES
List CRSD Debug Module: CRSRPT
List CRSD Debug Module: CRSRTI
List CRSD Debug Module: CRSSE
List CRSD Debug Module: CRSSEC
List CRSD Debug Module: CRSTIMER
List CRSD Debug Module: CRSUI
List CRSD Debug Module: CSSCLNT
List CRSD Debug Module: OCRAPI
List CRSD Debug Module: OCRASM
List CRSD Debug Module: OCRCAC
List CRSD Debug Module: OCRCLI
List CRSD Debug Module: OCRMAS
List CRSD Debug Module: OCRMSG
List CRSD Debug Module: OCROSD
List CRSD Debug Module: OCRRAW
List CRSD Debug Module: OCRSRV
List CRSD Debug Module: OCRUTL
List CRSD Debug Module: SuiteTes
List CRSD Debug Module: UiServer


As root, crsctl set log crs "CRSEVT=1","CRSAPP=1","OCRASM=2"
Set CRSD Module: CRSAPP  Log Level: 1
Set CRSD Module: CRSEVT  Log Level: 1
Set CRSD Module: OCRASM  Log Level: 2

Clusterware logs

Oracle clusterwares store its log files in the following locations:

• Oracle clusterware alert:  GRID_HOME/log/hostname/alert<nodename>.log
• CRS logs (Cluster Ready Service):  GRID_HOME/log/hostname/crsd/ .  The crsd.log file is archived every 10MB
• CSS logs (Cluster Synchonization Service):  GRID_HOME/log/hostname/cssd/.  The cssd.log is archived every 20MB
• EVM (Event Manager):  GRID_HOME/log/hostname/evmd
• SRVM (srvctl) and OCR (ocrdump, ocrconfig, ocrcheck) logs:  GRID_HOME/log/hostname/client and ORACLE_HOME/log/hostname/client
•  diagcollection.pl:  $GRID_HOME/bin/
• ASM:  GRID_BASE/diag/asm/+asm/+ASMn

What's New for Grid Infrastructure in 11.2.0.2

1.  Software Update before install or upgrade - new installer screen will allow you to download software updates, bug fixes, and Oracle patches from My Oracle Support.

2.  Automatic NIC Bonding for Private Networks:  The clusterware has the ability to automatically bond private interconnect interfaces

3.  Cluster Health Monitor:  It's now delivered with Grid Infrastructure

4.  Grid Installation Owner Group Requirements:  The grid installation owner no longer has to be a member of the group designated for ASMOPER if one has been created.

5.  Out of Place Install for Patchset Releases:  New installation and out of place so it can be patched easily prior to being applied to reduce downtime.  The Grid home at least must have 5.5 GB.

6.  Clusterware configuration Wizard

7.  Automatic CVU checks:  The CVU is managed by CRSD to report any problems if sees in the cluster setup.  You can check the frequency via "srvctl config cvu"

OCR v.s OLR

The OCR holds the metadata and wallets for all the resounces under the CRSD statck and its agents.  OCR still hol information about local resources, but it's not required to be accesible to join the cluster.  It includes information about the state of resouces, how to stop/start resources.  At least one OCR must exist, but you can have it up to five copies. In 11gR2, OCR can be stored in ASM.  The OCR location can be found in /etc/oracle/ocr.loc

+ASM1 - oracle: ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3528
         Available space (kbytes) :     258592
         ID                       : 1710014838
         Device/File Name         :        +OV
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check bypassed due to non-privileged user

From the output above, there are 5 slots, but one is used. 

The OLR holds metadata for the local nodeand GPnP profile.  It's managed by the OHASD and also manages the low-level processes required to allow a node to join a cluster.  None of the OLR data is shared.

 - root: ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2572
         Available space (kbytes) :     259548
         ID                       :  272464046
         Device/File Name         : /apps/grid/11.2.0/grid/cdata/dbplno155.olr
                                    Device/File integrity check succeeded
         Local registry integrity check succeeded
         Logical corruption check succeeded

 or you can view it in /etc/oracle/olr.loc

SCAN - Sample EZConnect and Thin JDBC Connect Strings

The main benefit of using SCAN is that you don't need to change connect string when adding or removing nodes in the 11gR2 Oracle cluster.  Below are few examples of how to connect to a database using SCAN.

sqlplus anguyen/an123@//lab-scan:1521/TOOLS_RAC

si01.an.com> oracle: sqlplus
SQL*Plus: Release 11.2.0.2.0 Production on Fri Jul 22 10:40:24 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Enter user-name: anguyen/an123@//lab-scan:1521/TOOLS_RAC

sqlplus anguyen@\"lab-scan:1521/TOOLS_RAC\"


JDBC thin URL

jdbc:oracle:thin:@mydr-scan:1521/TOOLS_RAC

CRS-2675

crsctl stop cluster -all

CRS-2675: Stop of 'ora.DATA.dg' on 'si01' failed

Checking the ASM alert log

NOTE: [emcrsp.bin@si01.an.com (TNS V1-V3) 20111] opening OCR file

NOTE: [emcrsp.bin@si01.an.com (TNS V1-V3) 20115] opening OCR file

NOTE: [emcrsp.bin@si01.an.com (TNS V1-V3) 20119] opening OCR file

Some notes about this issue:

1. This Bug is fixed in 11.2.0.3 and above.  If patch 9897335 is unavailable, the warning can be   ignored as it posts no harm.

2. You can't connect as sysasm and shut it down

SQL> connect / as sysasm

Connected.

SQL> shutdown immediate;

ORA-15097: cannot SHUTDOWN ASM instance with connected client (process 10871)

SQL> set linesize 132

SQL> select * from v$asm_client;

GROUP_NUMBER INSTANCE_NAME                                                    DB_NAME  STATUS

------------ ---------------------------------------------------------------- -------- ------------

SOFTWARE_VERSION                                             COMPATIBLE_VERSION

------------------------------------------------------------ ---------------------------------------

           1 +ASM1                                                            +ASM     CONNECTED

11.2.0.2.0                                                   11.2.0.2.0

           3 +ASM1                                                            asmacfs   CONNECTED

11.2.0.2.0                                                   11.2.0.2.0

NEVER To shutdown ASM or kill -9.  CRS need to be shutdown: crsctl stop crs

Remove failed node from 11gR2 RAC cluster

To delete a failed node and de-registered it from the cluster:

crsctl unpin css -n si01

crsctl delete node -n si01

crsctl stat res -t | grep si01

crsctl remove vip -i si01-vip -f

crsctl stat res -t

crsctl stat res -t | grep si01

 

Validate current stage before adding the node back:

cluvfy stage -pre nodeadd -n si01 -fixup -fixupdir /tmp

cluvfy stage -post hwos -n si01 -verbose

 

export IGNORE_PREADDNODE_CHECKS=Y

si02:/apps/grid/11.2.0/grid/oui/bin

./runInstaller -updateNodelist ORACLE_HOME='/apps/grid/11.2.0/grid' "CLUSTER_NODES=si02,si03" CRS=TRUE

Add node back to the cluster:

./addNode.sh -silent "CLUSTER_NEW_NODES={si01}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={si01-vip}"

  Starting Oracle Universal Installer...

 

  Checking swap space: must be greater than 500 MB.   Actual 32767 MB    Passed

  Oracle Universal Installer, Version 11.2.0.2.0 Production

  Copyright (C) 1999, 2010, Oracle. All rights reserved.

 

  

  Performing tests to see whether nodes si01,si03,si01 are available

  ............................................................... 100% Done.

WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.

To register the new inventory please run the script at '/apps/oraInventory/orainstRoot.sh' with root privileges on nodes 'si01'.

If you do not register the inventory, you may not be able to update or patch the products you installed.

The following configuration scripts need to be executed as the "root" user in each cluster node.

/apps/oraInventory/orainstRoot.sh #On nodes si01

/apps/grid/11.2.0/grid/root.sh #On nodes si01

To execute the configuration scripts:

    1. Open a terminal window

    2. Log in as "root"

    3. Run the scripts in each cluster node

   

The Cluster Node Addition of /apps/grid/11.2.0/grid was successful.

Please check '/tmp/silentInstall.log' for more details.

- root: /apps/oraInventory/orainstRoot.sh

Changing permissions of /apps/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /apps/oraInventory to oinstall.

The execution of the script is complete.

:/etc/rc.d/init.d

- root: /apps/grid/11.2.0/grid/root.sh

Running Oracle 11g root script...

The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /apps/grid/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /apps/grid/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

CRS-4046: Invalid Oracle Clusterware configuration.

CRS-4000: Command Create failed, or completed with errors.

Failure initializing entries in /etc/oracle/scls_scr/si01

/apps/grid/11.2.0/grid/perl/bin/perl -I/apps/grid/11.2.0/grid/perl/lib -I/apps/grid/11.2.0/grid/crs/install /apps/grid/11.2.0/grid/crs/install/rootcrs.pl execution failed

To fix this CRS-4046, CRS-4000 issue:

/apps/grid/11.2.0/grid/crs/install

./rootcrs.pl -verbose -deconfig –force

Then re-run root.sh

- root: ./root.sh

…

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Change Scan Port

srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is enabled

SCAN listener LISTENER_SCAN1 is running on node srvdb02

SCAN Listener LISTENER_SCAN2 is enabled

SCAN listener LISTENER_SCAN2 is running on node srvdb01

SCAN Listener LISTENER_SCAN3 is enabled

SCAN listener LISTENER_SCAN3 is running on node srvdb01

lsnrctl status LISTENER_SCAN1

lsnrctl status LISTENER_SCAN2

lsnrctl status LISTENER_SCAN3

crsctl stat res ora.LISTENER_SCAN1.lsnr

NAME=ora.LISTENER_SCAN1.lsnr

TYPE=ora.scan_listener.type

TARGET=ONLINE

STATE=ONLINE on srvdb02

ps -ef | grep -i scan

srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP: 1521

SCAN Listener LISTENER_SCAN2 exists. Port: TCP: 1521

SCAN Listener LISTENER_SCAN3 exists. Port: TCP: 1521

srvctl stop scan_listener

srvctl modify scan_listener -p 1972

srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1972

SCAN Listener LISTENER_SCAN2 exists. Port: TCP: 1972

SCAN Listener LISTENER_SCAN3 exists. Port: TCP: 1972

srvctl start scan_listner

SQLPLUS> alter system set remote_listener='racpoc-scan:1972';

System altered.

Restore OCR and Vote after corrupted ASM CRS diskgroup

From the below alert logs, the CRS (OCR, VOTE in ASM diskgroup) was corrupted.

WARNNING: cache read a corrupted block group=1(GRID) fn=6 blk=2 from disk 1(APP_PV_DISK1)
Errors in file /apps/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8335.trc:
ORA-15196: invalid ASM block header [kfc.c:25316] [endian_kfbh] [6] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:25165] [endian_kfbh] [6] [2] [0 != 1]
NOTE: a corrupted block from group GRID was dumped to /apps/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8335.trc
WARNNING: cache read(retry) a corrupted block group=1(GRID) fn=6 blk=2 from disk 1(APP_PV_DISK1)
Errors in file /apps/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8335.trc:
ORA-15196: invalid ASM block header [kfc.c:25316] [endian_kfbh] [6] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:25165] [endian_kfbh] [6] [2] [0 != 1]

The fastest way to fix this problem is to recreate and recover the ocr and vote ASM diskgroup by following the below steps:

1) Stop crs from all nodes

                crsctl stop crs –f

2) To clear out ASM header of the corrupted ASM that contains OCR and vote files

sudo -u root dd if=/dev/zero of=/dev/emcpowerb9 bs=8192 count=12800

sudo -u root dd if=/dev/zero of=/dev/emcpowerc9 bs=8192 count=12800

sudo -u root dd if=/dev/zero of=/dev/emcpoweri9 bs=8192 count=12800

3)  Label the CRS disks

/etc/init.d/oracleasm deletedisk APP_PV_DISK1

/etc/init.d/oracleasm createdisk APP_PV_DISK1 /dev/emcpowerb9

/etc/init.d/oracleasm deletedisk APP_PV_DISK2

/etc/init.d/oracleasm createdisk APP_PV_DISK2 /dev/emcpowerc9

/etc/init.d/oracleasm deletedisk APP_PV_DISK3

/etc/init.d/oracleasm createdisk APP_PV_DISK3 /dev/emcpoweri9

4)  Rescan disks from all nodes as root

/usr/sbin/oracleasm scandisks

5)  Start CRS in exclusive mode to prevent ora.crsd to start up

[root@rac01bin]# ./crsctl start crs -excl -nocrs

6)  Create CRS diskgroup

SQL> create diskgroup GRID normal redundancy disk 'ORCL:APP_PV_DISK1','ORCL:APP_PV_DISK2','ORCL:APP_PV_DISK3' attribute=’11.2’;

Diskgroup created.

SQL> select name, state from v$asm_diskgroup;

NAME                           STATE

------------------------------ -----------

GRID                             MOUNTED

7)  Restore OCR from good / latest backup

cd /u01/grid/11.2.0/grid/cdata/appdev

sudo -u root /u01/grid/11.2.0/grid/bin/ocrconfig -restore backup00.ocr

8)  Recreate the vote disk

sudo -u root /u01/grid/11.2.0/grid/bin/crsctl replace votedisk +GRID

Successful addition of voting disk a47251fdbb914f2bbfe0abbfac752af0.

Successful addition of voting disk 9e34471ba0134f67bf90e922d1af2e3e.

Successful addition of voting disk 433dcfc8a0664f5bbf1a7b264ab108ce.

Successfully replaced voting disk group with +GRID.

CRS-4266: Voting file(s) successfully replaced

9)  Stop and restart crs

crsctl stop crs –f

crsctl start crs