Infiniband verification

Verify IB cards are present
# lspci | grep Infi
Should be 1 card per system.

Verify IB ports are UP
# ibstatus | grep phys
        phys state:      5: LinkUp
        phys state:      5: LinkUp

Should show 2 ports up per system.

Verify IB ports are at QDR
# ibstatus | grep rate
        rate:            40 Gb/sec (4X QDR)
        rate:            40 Gb/sec (4X QDR)
Should show 2 ports at 40 Gb/sec (4x QDR) per system.

ibcheckerrors (to check network error)


ibdiagnet -r (to check cable)

List Flash Storage on Exadata Servers

You can query SCSI flash device information using lsscsi or list your PCI flash module using flash_dom to understand how flash storage is configured and presented on an Exadata Storage Server

lsscsi -v | grep MARVEL

Note:  Flash devices are show four 8, 9, 10, 11.  This is because of four flash cards have four FMods.  Therefore there are 16 flash devices and you can use flash_dom to display the details for the PCI flash devices

flash_dom –l

HBA# Port Name         Chip Vendor/Type/Rev    MPT Rev  Firmware Rev  IOC     WWID                 Serial Number

1.  /proc/mpt/ioc0    LSI Logic SAS1068E C0     105      011b5c00     0       50310020001116a80     465769T+1209A4ADFA

        Current active firmware version is 011b5c00 (1.27.92)
        Firmware image's version is MPTFW-01.27.92.00-IT
        x86 BIOS image's version is MPTBIOS-6.26.00.00 (2008.10.14)
        FCode image's version is MPT SAS FCode Version 1.00.49 (2007.09.21)


          D#  B___T  Type       Vendor   Product          Rev    Operating System Device Name
          1.  0   0  Disk       ATA      MARVELLXXXXXXXD21Y   /dev/sdn    [8:0:0:0]
          2.  0   1  Disk       ATA      MARVELLXXXXXXXD21Y   /dev/sdo    [8:0:1:0]
          3.  0   2  Disk       ATA      MARVELLXXXXXXXD21Y   /dev/sdp    [8:0:2:0]
          4.  0   3  Disk       ATA      MARVELLXXXXXXXD21Y   /dev/sdq    [8:0:3:0]


Validate FMods

You can see the flash devices grouped into sets of four on PCI slots 1, 2, 4, 5.  Each device per PCI slot residing in FDOM 0, 1, 2,OR 3.  Each FMOD is 24GB slide of storage x 16 = 284GB.  In X2 models, there are 4x 96GB pci FLASH cards per storage cell.  Each PCI flash car has a dvice partitioned per FDOM so there are 16 flash devices.  These flash devices  are the flash disks and used for Smart flash cache and smart flash logging. 

You should see 16 FMODs.  To validate all FMODs are present at the OS level, you can perform the following steps:

cellcli > list physicaldisk where disktype='FlashDisk' attributes name, disktype, physicalSize, slotNumber

flash_dom –l

cellcli > list flashcache detail
         id:                     cf20a022-a381-428e-bf6f-ee54dbcdcb56
         size:                   364.75G
         status:                 normal

cellcli > list flashlog detail

id:                     e7cbe8c7-d2f3-4d01-9f87-da83861c21d1
         size:                   512M
         status:                 normal

To verify hardware profile
/opt/oracle.SupportTools/CheckHWnFWProfile -d


Flash Storage

You can query SCSI flash device using lsscsi, list your PCI flash modedule using flash_dom

lsscsi -v | grep MARVEL                 (to view your flash devices)

 [8:0:0:0]    disk    ATA      MARVELL SD88SA02 D20Y  /dev/sdn
[11:0:3:0]   disk    ATA      MARVELL SD88SA02 D20Y  /dev/sdac

flash_dom  -l      (flash devices are split into groups of four:  8, 9, 10, 11).  Each of flash cars have 4 FMods.  Therefore each cell storage have 16 flash devices (4 x 4)

1.  /proc/mpt/ioc0    LSI Logic SAS1068E C0     105      011b5c00     0       5080020001116a80     465769T+1209A40TXF

        Current active firmware version is 011b5c00 (1.27.92)
        Firmware image's version is MPTFW-01.27.92.00-IT
        x86 BIOS image's version is MPTBIOS-6.26.00.00 (2008.10.14)
        FCode image's version is MPT SAS FCode Version 1.00.49 (2007.09.21)


          D#  B___T  Type       Vendor   Product          Rev    Operating System Device Name
          1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y   /dev/sdn    [8:0:0:0]
          2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y   /dev/sdo    [8:0:1:0]
          3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y   /dev/sdp    [8:0:2:0]
          4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y   /dev/sdq    [8:0:3:0]


Display physical disk, Megaraid device information, mdadm software RAID configuration

lsscsi      can show both physical SAS drives and flash devices.  There are 12 SAS devices and
lsscsi -v | grep MARVEL

[8:0:0:0]    disk    ATA      MARVELL SD88SA02 D20Y  /dev/sdn
[8:0:1:0]    disk    ATA      MARVELL SD88SA02 D20Y  /dev/sdo
….
[11:0:3:0]   disk    ATA      MARVELL SD88SA02 D20Y  /dev/sdac

lsscsi -v | grep LSI             show 12 physical devices

/opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary –aALL  (you should see 12 disks)


                Connector          : Port 0 - 3<Internal><Encl Pos 0 >: Slot 0
                Vendor Id          : HITACHI
                Product Id         : HQWERASDFA.0T
                State              : Online
                Disk Type          : SAS,Hard Disk Device
                Capacity           : 2.727 TB
                Power State        : Active

Each Exadata Storage Server has 12 physical SAS disks and 4 96GB PCIe Sun Flash Accelerator flash cards.  Each card partition into 4 x 24 GB partitions.

On Compute Node, there are 4 x 300 GB SAS – 1 hot spare (slot 3) and 3 active (slot 0,1,2)
/opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary –aALL


[root@r22dwdbs06h-adm MegaCli]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               VGExaDb
  PV Size               834.89 GB / not usable 3.84 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              213731
  Free PE               153827
  Allocated PE          59904
  PV UUID               TMKLyW-2e2u-2BFq-xf4l-jdne-Vnb7-LOmcHd
  
[root@r22dwdbs06h-adm MegaCli]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/VGExaDb/LVDbSys1
  VG Name                VGExaDb
  --- Logical volume ---
  LV Name                /dev/VGExaDb/LVDbSys2
  VG Name                VGExaDb
 
  --- Logical volume ---
  LV Name                /dev/VGExaDb/LVDbSwap1
  VG Name                VGExaDb

  --- Logical volume ---
  LV Name                /dev/VGExaDb/LVDbOra1
  VG Name                VGExaDb
 
  --- Logical volume ---
  LV Name                /dev/VGExaDb/sysback
  VG Name                VGExaDb
  LV UUID                kAKBpc-2I2n-xeJh-8sZy-OPxD-hklZ-ykaBsv

lvdisplay -v /dev/VGExaDb/LVDbSys1 (shows attributes of that logical volume)


ls -ltr /dev/VGExaDb/LVDb* (show the map to /dev/mapper devices)

Display Storage and Compute Server architecture

You’re assigned to manage the Exadata Machine.  You want to display overall architecture, storage configuration, network, and OS.
uname –a            (To display the kernel version for Oracle enterprise linux)
dmidecode –s system-product-name (To display the server model and serial number)
[root@glscell01 ~]# dmidecode -s system-product-name
SUN FIRE X4170 M2 SERVER
[root@glscell01 ~]# dmidecode -s system-serial-number
1308FMM0MG 
imageinfo            (To see cell version, kernel version, active system partition)
imagehistory      (image installed/patched overtime)
[root@r22dwdbs06h-adm ~]# imagehistory
Version                              : 11.2.3.2.0.120713
Image activation date                : 2012-1-17 15:23:49 -0500
Imaging mode                         : fresh
Imaging status                       : success

Version                              : 11.2.3.2.1.130302
Image activation date                : 2013-12-04 21:02:09 -0600
Imaging mode                         : patch
Imaging status                       : success

Each storage cell contains either: 
12 physical 600GB 15000 RPM high performance of delivering up to 1.8 GB per second of raw per cell or
3 TB 7200 RPM High Capacity SAS disk of delivery up to 1.3GB per second of raw data bandwidth

Compute server:
NET0      allow ssh, it uses eth0 interface
NET1, NET2, NET1-2        RAC VIP and SCAN
IB            Connects two ports on the compute servers to both of the InfiniBand leaf switches in the rac.  All storage servers and RAC interconnect use this network

NET3 is built on eth3       Use for Backup/external traffic

Transaction Guard and Application Continuity

Transaction Guard:  A new feauture in version 12c.  Transaction guard provides a way to keep track of transaction state by using a logical transaction identifier which maps to a dataabase transactio_id.  If there is a failure, the applciation connection can query the transaction status using logical transaction_id to determine if a transaction is commited or not.  There are two parameters when creating a service as transaciton guard.  commit_outcome:  to determine if a logical transaction is tracked or not.  retention how long to retend in the database (default 86400 seconds = 1 day).

srvctl add service -d dba12c -service TGDBA12C -preferred DBA12C1 -available DBA12C2 -commit_outcome TRUE -retention 86400

Application Continuity:  
Application Continuity is a new feature in 12c using the transaction guard.  It attempts to mask the outage by rebuilding the database session and resubmitting the pending work.  It used client-side JDBC driver 12c, Universal connection Pool 12c, and Oracle WeblOGIC sERVER 12.1.2 class.

-failovertype TRANSACTION
-commit_outcome  TRUE

srvctl add service -d dba12c -service ACDBA12C -preferred DBA12C1 -available DBA12C2 -commit_outcome TRUE -retention 86400 -failovertype TRANSACTION -failoverretry 10 -failoverdelay 5 -replay_int_time 1200

Creating Services with Pluggable database in RAC


For applications to access the Pluggable database, you need to create services for the PDB.  

Example:  Create order_oltp service associated with pocpdb1 (PDB) prefered on poccdb1(Container DB instance 1) and available on poccdb2 (Container DB instance 2)

srvctl add service -db poccdb -service order_oltp -pdb pocpdb1 -preferred poccdb1 -available poccdb2

srvctl start service -d poccdb -s order_oltp

SQL> connect sys@pocpdb1 as sysdba
Enter password:
Connected.
SQL> show con_name
POCPDB1

To check services of the pluggable database

select name, pdb from all_services;

From clusterware, you can check the service configuration of a container database using srvctl command

poccdb1 > srvctl config service -db poccdb
Service name: order_oltp
Service is enabled
Server pool: poccdb_order_oltp
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
TAF failover retries:
TAF failover delay:
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pocpdb1
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Session State Consistency:
Preferred instances: poccdb1
Available instances: poccdb2

Creating order_oltp Net Service name for Pluggale database

After define the service with property of the pluggable database, you can create the net service name order_oltp in the tnsnames.ora file for applications to access the PDBs.  The connection string should look like this..

ORDER =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 12cracpoc-scan)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = order_oltp)
    )
  )

sqlplus > connect sys@ORDER as sysdba
Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

Oracle 12c ASM New Features Notes

ASM Fast Mirror Resync
Enabled when COMPATIBLE.RDBMS >=11.1
ASM fast mirror resync is used to deal with transient disk failures.  When a disk goes offline, ASM tracks the extents that are modified during the outage.  When the transient failure is repaired, ASM can quickly resynchronize only the disk extents that have been modified during the outage.  Note that the tracking mechanism uses one bit for each modified extent and is very efficient.

When using ASM fast mirror resync, the failed disk is taken offline and if you set the DISK_REPAIRE_TIME attribute for the correponding disk group, the disk won't be dropped.  The setting for this attribute determines the duration of disk outages that ASM will tolerate while still being able to resynchronize after the failed disk is repaired.  The default setting for the DISK_REPAIR_TIME attribute is 3.6 hours.  If a disk remains offline longer than the DISK_REPAIR_TIME setting, the disk is dropped from the diskgroup and the diskgroup is rebalanced.

Controlling the Resources Used by Resync

Power limit an set for disk resync operations:  It is conceptually similar to the power limit setting for disk group rebalance.  The Range is 1 to 1024, and if not specified the default setting s 1

alter diskgroup DATA ONLINE DISK data_000 POWER 99;
asmcmd > online -g DATA -D data_000 --power 99
alter diskgroup DATA ONLINE ALL POWER 500

More Efficient Disk Replacement
Administrators have the option to replace disk as a fast and efficient operation.  The replacement disk is populated with mirror copies of the ASM extents from from other disks and there is no need for reorganization or rebalancing of the disk group.  The replacement disk takes the same name as the original disk and becomes part of the same failure group as the original disk.

alter diskgroup GLS_DATA replace disk DATA_0001 with '/dev/disk002';

Dealing with Transient Falure on a Failure Group
SQL> alter diskgroup DBA_GLS set attribute 'failgroup_repair_time'='48h';
Diskgroup altered.
With Oracle 12c, ASM provides administrators with the ability to specify a failure group repair time in addition to the disk repair time.  The failure group repair time can be set by using a new disk group attribuute, FAILGROUP_REPAIRT_TIME.  The default failure group repair time is 24 hours.  Now by default, if all disks in a failure group fail simultaneously, ASM will not drop the disks when the disk repair time expires after 3.6 hrs.  Rather, the administrator will have 24 hours to correct the failure.  If the failure is corrected before the time expires, the disks are updated wusing a fast mirror resync.  If the failure is not corrected in time, ASM must automatically drop all disks in the failure group.

Resync Time Estimate

Check v$ASM_OPERTION for resync operation estimate
select pass, state, est_minutes from v$asm_operation;

Resync Checkpoint and Auto-Restart
With Oracle 12c ASM interrupted resync operations are automatically restarted.  The resync operations are broken into phases:  A checkpoint marks the end of each resync phase, and stale extent metadata is cleared.  Interrupted resync operations restart from the last checkpoint prior to the interruption.

Rebalance Work Estimates
Oracle 12c ASM, a more detailed and more accurate work plan is created at the begining of each rebalance operation.  The admin can generate the work plan by using the EXPLAN WORK command and the work can be viewed in v$ASM_ESTIMATE view (EST_WORK column shows the ASM allocateion units to be moved).

SQL> explain work set statement_id='Drop DATA_DISK15' for alter diskgroup DBA_DATA501 drop disk DATA_DISK15;
Explained.

SQL> select est_work from v$asm_estimate where statement_id='Drop DATA_DISK15';
  EST_WORK
----------
    111168

Priority Ordered Rebalance
 With Oracle 12c ASM, priority ordered rebalance is implemented.  This capability helps to restoring the redundancy of critical files like control files and online redo logs to ensure that they're protected in case of a secondary failure occurs.

Proactively Validating Data Integrity
ASM 12c provides proactive scrubbing capabilities that check for logical corruptions and automatically repair them where possible.  Two ways of scrubbing:  On-demand  by administrator on specific area, Occur as part of rebalance operation

SQL> alter diskgroup DBA_GLS501 scrub repair;
Diskgroup altered.

SQL> alter diskgroup DBA_DATA501 scrub file '+DATA_DISK/GLSLAB/DATAFILE/system.553.826023617' repair wait;
Diskgroup altered.

SQL> alter diskgroup DBA_DATA501 scrub disk DATA_DISK15 repair power max force;
Diskgroup altered

REPAIR:  If the repair option is not specified, ASM only check and report logical corruption
POWER:  LOW, HIGH, or MAX.  If power is not specified, the scrubbing power is controlled based on the system I/O load
FORCE:  Command is processed immediately regarless of system load

Proactive Content Checking during rebalance
Checks are automatic with automatic error correction for mirrored data and checks are enabled with the disk group attribute content.check

SQL> alter diskgroup DBA_DATA501 set attribute 'content.check' = 'TRUE';
Diskgroup altered.

Managing Password Files in ASM

Create ASM password file
+ASM1 > orapwd asm=y file='+GLS_DATA501' password='Password4dba'

Create database password file
ASMCMD> pwcreate --dbuniquename GLSLAB  +DBA_DATA laga4me

Locate password file
+ASM1 > crsctl stat res ora.glslab.db -f | grep PWFILE
PWFILE=+DATA_DISK/GLSLAB/PASSWORD/pwdglslab.464.827084721

Specifying Content Type for a Disk Group
Possible values:  data, recovery or system

Flex ASM - Migrate standard ASM to Flex ASM

Perform the following steps to migrate a standard ASM to flex ASM

export DISPLAY=glsdb01.global:1
 
asmca -> 
 
Under ASM Istances ->ASM listener Port, Select the network(s) to be used for ASM, Click OK, Click Convert to Oracle Flex ASM
 
gls01:/apps/oracle/grid/cfgtoollogs/asmca/scripts

-rwxr-xr-x 1 oracle oinstall 620 Sep 24 13:19 converttoFlexASM.sh

+ASM1 > cat converttoFlexASM.sh | more
#!/bin/sh

su oracle -c "/apps/oracle/product/12.1.0.1/grid/bin/srvctl add asm -proxy"
/apps/oracle/product/12.1.0.1/grid/srvm/admin/srvmhelper -updateDepASM
for node in glsdb01 glsdb02
do
        /apps/oracle/product/12.1.0.1/grid/bin/crsctl stop cluster -n $node
        /apps/oracle/product/12.1.0.1/grid/bin/crsctl start cluster -n $node

        stat=1
        cmd="/apps/oracle/product/12.1.0.1/grid/bin/crsctl status server $node"
        while [ $stat = 1 ]
        do
                sleep 5
                op=`$cmd`
                op=`echo $op | /bin/cut -d' ' -f2  | /bin/cut -d'=' -f2`
                if [ "$op" = "ONLINE" ]
                then
                        stat=0
                        echo "Cluster bounced in node $node"
                fi
        done
done

gls01:/apps/oracle/grid/cfgtoollogs/asmca/scripts 
 
As root:
 
/apps/oracle/grid/cfgtoollogs/asmca/scripts/converttoFlexASM.sh

CRS-2673: Attempting to stop 'ora.crsd' on 'glsdb01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'glsdb01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dba12c.db' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'glsdb01'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.glsdb01.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.db' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dbatools.db' on 'glsdb01'
CRS-2677: Stop of 'ora.scan1.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.glsdb01.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.glsdb01.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.dbatools.db' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.DBA_DATA501.dg' on 'glsdb01'
CRS-2676: Start of 'ora.scan1.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'glsdb02'
CRS-2677: Stop of 'ora.DBA_DATA501.dg' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.glsdb01.vip' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.GI.dg' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'glsdb01'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'glsdb02'
CRS-2677: Stop of 'ora.GI.dg' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'glsdb02'
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error: 
CRS-2679: Attempting to clean 'ora.mgmtdb' on 'glsdb02'
CRS-2681: Clean of 'ora.mgmtdb' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb01'
CRS-2677: Stop of 'ora.asm' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb01'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'glsdb01'
CRS-2677: Stop of 'ora.ons' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'glsdb01'
CRS-2677: Stop of 'ora.net1.network' on 'glsdb01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'glsdb01' has completed
CRS-2677: Stop of 'ora.crsd' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.evmd' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.storage' on 'glsdb01'
CRS-2677: Stop of 'ora.storage' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb01'
CRS-2677: Stop of 'ora.ctssd' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.asm' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'glsdb01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'glsdb01'
CRS-2677: Stop of 'ora.cssd' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'glsdb01'
CRS-2676: Start of 'ora.evmd' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'glsdb01'
CRS-2676: Start of 'ora.cssdmonitor' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'glsdb01'
CRS-2672: Attempting to start 'ora.diskmon' on 'glsdb01'
CRS-2676: Start of 'ora.diskmon' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.cssd' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'glsdb01'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'glsdb01'
CRS-2676: Start of 'ora.ctssd' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'glsdb01'
CRS-2676: Start of 'ora.asm' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'glsdb01'
CRS-2676: Start of 'ora.storage' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'glsdb01'
CRS-2676: Start of 'ora.crsd' on 'glsdb01' succeeded
Cluster bounced in node glsdb01
CRS-2673: Attempting to stop 'ora.crsd' on 'glsdb02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'glsdb02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.cvu' on 'glsdb02'
CRS-2677: Stop of 'ora.cvu' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'glsdb01'
CRS-2676: Start of 'ora.cvu' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.scan2.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.scan3.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'glsdb01'
CRS-2676: Start of 'ora.scan2.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dba12c.db' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.dbatools.db' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.oggdb.db' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.glsdb02.vip' on 'glsdb02'
CRS-2676: Start of 'ora.scan3.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'glsdb01'
CRS-2677: Stop of 'ora.glsdb02.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.glsdb02.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.db' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.dbatools.db' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.glsdb02.vip' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.oggdb.db' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.DBA_DATA501.dg' on 'glsdb02'
CRS-2677: Stop of 'ora.DBA_DATA501.dg' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.GI.dg' on 'glsdb02'
CRS-2677: Stop of 'ora.GI.dg' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.dba12c.dba12c_rac.svc' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dba12c.dba12c_rpt.svc' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.dba12c_rac.svc' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.dba12c.dba12c_rpt.svc' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.dba12c.db' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.db' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb02'
CRS-2677: Stop of 'ora.asm' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.oc4j' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb02'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'glsdb02'
CRS-2677: Stop of 'ora.ons' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'glsdb02'
CRS-2677: Stop of 'ora.net1.network' on 'glsdb02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'glsdb02' has completed
CRS-2677: Stop of 'ora.crsd' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.evmd' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.storage' on 'glsdb02'
CRS-2677: Stop of 'ora.storage' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb02'
CRS-2677: Stop of 'ora.ctssd' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.asm' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'glsdb02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'glsdb02'
CRS-2677: Stop of 'ora.cssd' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'glsdb02'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'glsdb02'
CRS-2676: Start of 'ora.evmd' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'glsdb02'
CRS-2672: Attempting to start 'ora.diskmon' on 'glsdb02'
CRS-2676: Start of 'ora.diskmon' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.cssd' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'glsdb02'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'glsdb02'
CRS-2676: Start of 'ora.ctssd' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'glsdb02'
CRS-2676: Start of 'ora.asm' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'glsdb02'
CRS-2676: Start of 'ora.storage' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'glsdb02'
CRS-2676: Start of 'ora.crsd' on 'glsdb02' succeeded
Cluster bounced in node glsdb02

real    7m3.449s
user    0m1.587s
sys     0m0.435s
You have new mail in /var/spool/mail/oracle